Backfilling Features and Training Data

Backfills rebuild historical feature rows from retained raw or intermediate data. They are useful after correcting a bug, changing a definition, repairing missed events, or creating training data for a new time range. Before running one, determine which feature version and source snapshots it should use, which partitions and entities are affected, and whether it should replace or append data. A backfill may be appropriate for a training dataset, an online store, or both, but the destination and historical semantics differ. Each training example should use values available at its prediction time. Preserve event timestamps and, where relevant, the time a value became available; otherwise a late-arriving event or retroactive correction may rewrite what the model appears to have known. Feast’s event-time point-in-time join and optional created-time filtering can help for supported sources, but neither repairs a feature that was computed using future observations. Replaying raw event logs through new logic can also produce a different historical answer than the value served online at the time, so store code/config versions and compare known examples. Retention planning affects whether a feature can be rebuilt, but retaining raw data is subject to privacy, security, contractual, and legal policies. Make a backfill idempotent or explicitly versioned, test a small range, compare counts and values, and retain rollback information. Do not assume every feature change requires rewriting all history; decide based on the new definition, the model’s training window, and the intended serving behavior.