SLOs and Alerting for ML Services

An SLO states a target for a service-level indicator (SLI) over a time window. Common SLIs include availability, latency, throughput and error rate. ML services may also monitor prediction freshness, feature availability, queue delay or quality once outcomes arrive. The SLO should reflect what users need and what the service can measure reliably. A clear denominator matters: availability might be the fraction of eligible requests that succeed, with exclusions defined.

Latency objectives often focus on a percentile rather than the mean because a small slow tail can affect users even when average latency is low. Batch inference may use deadlines, completion rates and data freshness instead of request-level latency. A feature pipeline can have its own SLO if stale features make predictions less useful. Model accuracy usually cannot be measured immediately without labels, so quality SLOs may lag or use carefully validated proxies.

Alerts should prompt action. Page for urgent user-impacting conditions, such as sustained errors or exhausted capacity; use tickets or dashboards for slower trends. Error budgets quantify how much unreliability is allowed under a target and can guide release pace, but they do not replace product judgment. Alerts need windows, thresholds, routing and runbooks. A noisy alert creates fatigue, while a poorly chosen indicator can stay green as user experience declines.

Separate service SLOs from model-quality goals. A service can be available but serve stale or low-quality predictions. Conversely, model metrics may shift for benign population reasons while requests remain healthy. Define data sources, ownership, exclusions, aggregation and burn-rate behavior. Review incidents and user feedback to update SLOs. Monitoring should support service reliability and model oversight without turning every statistical fluctuation into an emergency page.