Benchmark Contamination Quiz

Answers and explanations appear as you take the quiz.

1. What is benchmark contamination?
   • When a benchmark contains incorrect answer keys
   • When a model is tested on too few questions
   • When benchmark test items or their answers appear in a model's training data
   • When two benchmarks measure the same skill
2. What is the purpose of the canary string included in BIG-bench tasks?
   • To let data teams filter those documents from training data and test whether a model saw them
   • To encrypt the answers
   • To mark which tasks are the hardest
   • To identify which model generated each answer
3. What did the GSM1k study suggest?
   • All models scored higher on the new problems
   • GSM8K has no errors
   • Contamination cannot be measured without training data
   • Some models scored noticeably lower on fresh, matched problems than on GSM8K
4. What overlap analysis did the GPT-3 paper run?
   • Embedding similarity on full documents
   • 13-gram overlap between benchmarks and training data
   • Human review of every training document
   • Min-K% Prob on each benchmark
5. What is the core idea of Min-K% Prob?
   • Removing the least common k percent of words from training data
   • Training a second model to classify contaminated text
   • Text the model has seen tends to have few very low-probability tokens
   • Measuring the length of the model's answers
6. How does a time-based split, as in LiveCodeBench, help detect contamination?
   • It compares performance on problems published before and after the model's training cutoff
   • It measures how long the model takes to answer
   • It splits the model into older and newer layers
   • It only counts problems answered within a time limit
7. Why does finding no exact overlap not prove a benchmark is clean?
   • Exact matching is too slow to run
   • Benchmarks change their answers over time
   • Models never memorize exact text
   • Paraphrased or translated copies can slip past string matching
8. In n-gram decontamination, what happens if n is chosen too small?
   • Near-copies with small edits are missed
   • Many false positives from common phrases are flagged
   • The filter stops working entirely
   • Only answers, not questions, are matched