Data Quality Monitoring in Production

A deployed model relies on inputs arriving with expected names, types, units and semantics. Upstream systems can change schemas, stop refreshing a field, add new categories, shift units or introduce missing values. Data-quality monitoring checks these conditions at ingestion or inference time, before failures appear as model metric changes. Typical checks include schema compatibility, null rates, ranges, uniqueness, freshness, categorical vocabularies, volume and cross-field consistency. Data validation tools can infer or apply schemas and compare incoming batches with known expectations. A schema should distinguish hard constraints from statistical expectations. A missing required identifier may block processing, while a moderate distribution shift may trigger review rather than hard failure. Great expectations need context: legitimate seasonality can change value distributions, and an overly strict rule can cause outages. Define owners and actions for each alert. Train-serving skew occurs when training and inference compute features differently. For example, timezone handling, default values or category mapping may differ between pipelines. Shared transformation code or comparison tests can reduce the risk, while live monitoring tracks whether the served feature representation resembles the training reference. Distribution similarity is not proof of correctness, but it can expose changes. Checks should respect latency budgets and privacy. Some validation can occur in batch or asynchronously, while critical schema checks run inline. Log aggregate failures and sampled examples under access and retention controls. Decide whether invalid inputs are rejected, repaired, routed to fallback or sent to manual review. Data-quality monitoring detects violations and shifts; it does not establish model accuracy, fairness or causal validity. Pair it with delayed-label evaluation and system health monitoring. When a check fires, trace the field to its source, determine whether the change is expected and update contracts only after validation. Otherwise, changing the schema simply to make alerts pass can hide a real defect.