Data Flywheels: Turning Production Data into Training Data

A data flywheel connects model use with future model development. A production system generates signals such as corrections, appeals, clicks, reformulations, exceptions or expert reviews. The team selects some records, verifies or annotates them, adds them to a curated dataset, trains a candidate and evaluates whether it improves the intended task. If deployed, the updated model may then produce new cases and feedback. Raw interactions are not automatically high-quality labels. A click can reflect position rather than preference, an agent correction can contain personal information, and user abandonment can stem from a broken interface. Define what signal means, who can label it and how disagreements are resolved. Keep provenance, timestamp and policy context. Remove or protect sensitive details and honor consent, retention and deletion obligations. Sampling affects what the flywheel learns. If the system captures only high-confidence successes, it may miss failures. Selecting uncertain or diverse cases for review can improve coverage, but reviewers need clear instructions and quality checks. Log exposure and selection probabilities where feasible to understand biases. Avoid training on model-generated labels without validation, because repeated self-labeling can reinforce errors. New data should enter a versioned dataset with deduplication, leakage checks and separation between training and evaluation. Preserve an untouched or appropriately refreshed evaluation set. Train candidate models, compare against the current model and simple baselines, inspect slices and safety outcomes, then promote through a controlled release. Measure the flywheel itself: useful label yield, reviewer burden, performance change, group coverage and privacy incidents. The loop creates value only if new feedback is trustworthy and the release process can reject harmful regressions. More production data do not guarantee better models when the data-generation process is selective or labels are noisy.