Guide-linked quiz · Medium Level

Evaluating Recommender Systems: NDCG, Hit Rate and More Quiz

Tests top-K ranking metric definitions, what each measure captures, how labels affect interpretation, and why offline scores need live outcome validation.

Question 1 of 8

A list has 10 recommendations, of which 4 are relevant under the evaluation labels. What is Precision@10?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. A list has 10 recommendations, of which 4 are relevant under the evaluation labels. What is Precision@10?
    • 0.4, because 4 of the first 10 items are relevant
    • 0.6, because 6 items are irrelevant
    • 4, because precision is the relevant-item count
    • 0.1, because the cutoff is 10
  2. What does Recall@K measure in a recommender evaluation?
    • Whether at least one item in the list was clicked
    • The average position of every catalog item
    • The share of all labeled relevant items that appears in the top K
    • The share of the top K list that is relevant
  3. Under the common Hit Rate@K convention, when does a user count as a hit?
    • Every item in the top-K list is relevant
    • The user clicks every recommendation
    • The first result has a higher score than the second
    • At least one held-out relevant item appears in the user’s top-K list
  4. What does NDCG add compared with an unordered count of relevant recommendations?
    • It computes whether any one item appears anywhere in the catalog
    • It accounts for ranking position by discounting lower-ranked relevant results and normalizing by an ideal ranking
    • It measures only catalog size and item coverage
    • It replaces relevance labels with click-through revenue
  5. Why should a report state its relevance labels and metric convention?
    • All relevance labels are interchangeable once K is selected
    • A single standard convention is mandated across every recommender
    • Metric labels do not affect reported results
    • Clicks, purchases, and ratings represent different behaviors, and metric implementations can vary
  6. An offline NDCG score improves. What can the team safely conclude?
    • The model ranked the chosen evaluation labels better under that offline setup; live impact still needs validation
    • The product will necessarily earn more revenue
    • Users will definitely be more satisfied
    • The model is ready to launch without guardrails
  7. Why can historical click logs be a biased relevance source?
    • Historical logs contain no ranking information
    • User activity can only bias Recall, never Precision or NDCG
    • The previous ranker influenced which items users had a chance to see
    • Clicks always reveal objective relevance without exposure effects
  8. Which evaluation plan is most informative before a major recommender launch?
    • Compare global click totals without checking user-level exposure
    • Report offline metrics with definitions and guardrails, then test live outcomes where suitable
    • Choose the model with the highest Hit Rate and skip product metrics
    • Optimize a single offline score until it reaches 1.0

Keep testing yourself

More quizzes picked for your level.

View all quizzes