ML Monitoring with Prometheus and Grafana

Prometheus is a monitoring system that collects numeric time-series data, commonly by scraping an HTTP metrics endpoint exposed by a service. An inference server can export counters for requests and failures, gauges for queue depth or resource state, and histograms for latency and request sizes. Labels allow grouping by bounded attributes such as endpoint, status code or model version. The exposition format and metric semantics should be consistent so queries mean what operators expect. Grafana can query Prometheus and display time-series panels, tables and alerts in dashboards. A useful ML service dashboard combines service health with model-relevant signals: request rate, p50/p95 latency, error codes, saturation, batch size, GPU memory and coarse prediction summaries. Service and model signals answer different questions. A healthy latency chart does not prove predictions are useful; prediction drift does not necessarily mean the service is down. Label cardinality requires care. Labels such as user ID, raw prompt, request ID or unbounded item ID can create a huge number of time series, increasing memory and query costs. Sensitive values should not be put in metrics labels. Use logs or traces with access controls and sampling for high-cardinality context, and metrics for aggregated counts and distributions. Histograms support aggregation across instances, while client-side summary quantiles may not aggregate in the same way. Define alerts around actionable service objectives and include a time window to avoid reacting to brief noise. Route alerts to owners and test them. Model-quality alerting often depends on delayed labels or careful proxy metrics, and should be managed separately from uptime alarms. Dashboards need clear units, model versions, deployment markers and documented thresholds. Prometheus and Grafana provide collection and visualization components; they do not automatically define meaningful ML metrics or explain why a prediction changed. Privacy, retention and access policies apply to every signal emitted by the service.