Streaming Features with Kafka and Flink

A streaming feature pipeline usually separates transport from computation. Kafka stores ordered records within each partition and allows consumers to replay retained events. A processor such as Flink can key state by entity, compute windows or rolling aggregates, and write results to an online feature store. This supports low-latency features such as recent transaction counts when the model needs fresher values than a batch schedule provides. A feature definition should specify the entity key, window, default for missing history, and late-event policy so serving behavior is explicit.

Event time is when the event happened; processing time is when the job handled it. Events can arrive out of order, so Flink watermarks estimate event-time progress and allow windows to close while accounting for expected lateness. A watermark is a progress signal, not proof that no older event can ever arrive; late-event policy determines whether to update, route, or drop those records. Bad timestamps, duplicate events, or incorrectly keyed state can corrupt a rolling feature even when the job has no downtime.

Flink checkpoints can recover managed state and source positions. But exactly-once state recovery is not the same as end-to-end exactly-once output: source participation and a compatible sink or transaction/idempotence strategy matter. Kafka’s own idempotent and transactional producer semantics have defined scope. Test replay, restart, late data, and sink behavior against the exact connector versions in use before promising delivery guarantees.