Z-Score and IQR Outlier Detection

An outlier is an observation unusually far from much of a selected dataset, but unusual does not mean erroneous. It might be a measurement mistake, a rare valid event, a data-entry mismatch or a sign that several populations were mixed. First define the reference group. Comparing a child's measurement with an adult reference, for example, may manufacture an apparent anomaly. A z-score subtracts the reference mean from an observation and divides by the reference standard deviation. A value three standard deviations above that mean has z = 3 under the stated reference. Rules such as flagging values with absolute z above three are heuristics, not universal laws. They work best when mean and standard deviation summarize a reasonably stable, appropriate distribution. Extreme points can pull both quantities, and a heavily skewed distribution can produce many legitimate high values. NIST's outlier guidance cautions that familiar z-score tests can be misleading in small samples. The interquartile range, IQR, is Q3 minus Q1, spanning the middle half of values. A common box-plot rule places inner fences at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR. For illustrative quartiles Q1 = 10 and Q3 = 18, IQR is 8 and the upper fence is 30. A value of 32 exceeds that fence, while 29 does not. This is a useful screen, not a probability that the value is false. Quartile calculations can differ slightly by convention on small datasets. Both methods depend on the data period and population. Compare flags with source records, repeat measurements and domain knowledge. In a machine-learning pipeline, derive any threshold from training data rather than from the future validation or test set. Keep a record of what was flagged, how it was handled and whether a rule disproportionately affects a relevant group. Recheck thresholds when the environment changes, and consider more suitable methods when the distribution is skewed or multimodal.