Evaluating Recommender Systems: NDCG, Hit Rate and More

Ranking metrics are summaries of a particular evaluation setup, not universal descriptions of recommendation quality. Precision@K is the fraction of the top K recommendations that are relevant under the chosen labels. Recall@K is the fraction of relevant items in the evaluation set that appear in the top K. Hit Rate@K typically records whether a user’s top-K list contains at least one relevant item and averages that indicator across users; implementations differ, so state the convention. A hit does not reveal how many relevant items were retrieved or whether they ranked first.

Discounted Cumulative Gain (DCG) rewards relevant results while discounting lower-ranked positions. Normalized DCG (NDCG) divides DCG by the ideal DCG for the same relevance judgments, making comparisons easier across queries with different ideal gains. Implementations can differ in gain and discount formulas, treatment of ties, and cutoff. For example, scikit-learn describes NDCG as summing true scores in the predicted ranking after logarithmic discount and dividing by the best possible score. A metric is only meaningful alongside a clear definition of ground truth: clicks, purchases, ratings, and survey judgments measure different behaviors and carry biases.

Offline evaluation on historical logs is fast and reproducible, but it cannot alone predict live impact. Exposure and selection effects shape observed interactions, and optimizing one metric can reduce diversity, catalog coverage, satisfaction, or a business outcome. Google’s ML project guidance explicitly cautions that strong model metrics do not guarantee business success and recommends tracking focused business metrics. Teams should report metric conventions, baselines, segments, uncertainty, and guardrails, then validate consequential changes with a suitable online experiment where ethical and practical. There is no single best recommender metric for every objective.