LegközelebbKövetkező útmutató
Z-Loss és edzési stabilitás
Műszaki
Műszaki ÚTMUTATÓ
Population Stability Index (PSI) summarizes how binned feature distributions differ between a reference population and a comparison population.
Hallgassa meg ezt az útmutatótAI-narrált · ~3 min
It can flag shifts for investigation, but binning choices, zero-count handling, sample size and context-specific thresholds limit its value as a stand-alone drift or model-performance signal.
Population Stability Index compares distributions by dividing a variable into bins and measuring changes in the proportion of observations in each bin. For reference share E_i and comparison share A_i, a common formula is PSI = sum_i (A_i-E_i)*ln(A_i/E_i). A contribution is zero when the shares match, and larger differences generally add more to the total. PSI is often applied to score or feature distributions in monitoring, particularly where labels arrive late. The result depends on how bins are defined. Fixed bins make comparison across time interpretable, while quantile bins based only on a reference sample can preserve baseline balance. Recomputing bins separately for each sample can make shifts harder to see. Coarse bins may hide changes within a bin; too many bins can create sparse counts and unstable ratios. If any bin has zero share, the logarithm is undefined, so implementations use smoothing or minimum proportions. Report that convention and test whether conclusions change. PSI measures distributional difference, not whether the change harms predictions. A feature can shift due to seasonality or a legitimate population change while model performance remains stable. Conversely, conditional relationships can change without a large marginal feature PSI. A score-distribution shift may indicate changed inputs, changed policy or changed model version. None of these possibilities is distinguished by the scalar index. Some domains use rule-of-thumb bands for interpreting PSI, but thresholds vary and should not be treated as universal statistical guarantees. The index is sensitive to sample size, bin count and chosen reference period. Use it as one alert signal alongside schema checks, slice metrics and delayed-label evaluation. Investigate which bins contribute to the score, whether the shift persists and whether a downstream decision changes. PSI is a compact summary for prioritizing review, not a substitute for ground-truth performance measurement or causal diagnosis.
A hypothetical feature has reference bin shares 0.6 and 0.4, while the new sample has 0.5 and 0.5. PSI sums each bin's share difference times the log ratio of new to reference shares.
A reference bin has no observations, making the log ratio undefined. An analyst applies a documented smoothing convention and checks sensitivity instead of silently assigning an arbitrary score.
A feature's distribution shifts substantially but remains within the same coarse bins, so PSI is small. Finer bins or another test may reveal detail, illustrating dependence on binning.
A model's PSI for score distribution rises after deployment. The team investigates input shifts, policy changes and seasonality; it does not conclude from PSI alone that accuracy declined.
PSI monitoring can be improved by fixing reference periods and bin definitions, reporting per-bin contributions and tracking both statistical and operational context. Teams should set alert thresholds using historical variation and known changes rather than importing a universal cutoff. When labels become available, compare PSI alerts with actual performance changes to learn which shifts matter. Pair distribution monitoring with data-quality checks and conditional performance analysis. This helps teams use PSI to prioritize investigation without turning a coarse histogram comparison into a pass/fail judgment about a model.
Ingyenes hírlevél
Minden hétköznap reggel három ellenőrzött AI történet, egyszerű angol nyelven. Örökké ingyen, hirdetések nélkül.
Minden hétköznap egy e-mail. Egy kattintással lemondhat. Soha nem értékesítjük vagy osztjuk meg az Ön címét.
Teszteld magad
Azonnali visszajelzés minden válaszra, és megosztható tanúsítvány egy ellenőrizhető azonosítóval, miután sikeresen teljesít egy kurzust.
Támogassa az ingyenes AI oktatást. AI Understanding egy 501(c)(3) nonprofit – nincs hirdetés, nincs fizetőfal, soha. Tegyen adományt
Population Stability Index (PSI) summarizes how binned feature distributions differ between a reference population and a comparison population. It can flag shifts for investigation, but binning choices, zero-count handling, sample size and context-specific thresholds limit its value as a stand-alone drift or model-performance signal.
Each contribution is (A-E) times ln(A/E), where A and E are comparison and reference proportions.
A zero denominator makes the logarithm undefined, so implementations need a documented treatment.
Fixed bins let analysts compare shares for the same value ranges over time.
A coarse partition may place different distributions into the same bin proportions.
PSI measures a distribution difference, not the cause or its effect on predictive performance.
Tanulj tovább
További útmutatók készültek ehhez a témához
LegközelebbKövetkező útmutató
Z-Loss és edzési stabilitás
Műszaki