Bài kiểm tra có liên kết hướng dẫn · Trung bình Cấp độ

Bài kiểm tra ô nhiễm điểm chuẩn

Kiểm tra xem các mục điểm chuẩn rò rỉ vào dữ liệu đào tạo như thế nào, các phương pháp phát hiện như chồng chéo n-gram và Kiểm tra Min-K% cũng như các biện pháp giảm nhẹ như canary và phân chia thời gian.

Đường dẫn hướng dẫn liên quanÔ nhiễm điểm chuẩn
Câu hỏi 1 của 8

What is benchmark contamination?

Xem trước mọi câu hỏi8 câu hỏi

Câu trả lời và giải thích xuất hiện khi bạn làm bài kiểm tra.

  1. What is benchmark contamination?
    • When a benchmark contains incorrect answer keys
    • When a model is tested on too few questions
    • When benchmark test items or their answers appear in a model's training data
    • When two benchmarks measure the same skill
  2. What is the purpose of the canary string included in BIG-bench tasks?
    • To let data teams filter those documents from training data and test whether a model saw them
    • To encrypt the answers
    • To mark which tasks are the hardest
    • To identify which model generated each answer
  3. What did the GSM1k study suggest?
    • All models scored higher on the new problems
    • GSM8K has no errors
    • Contamination cannot be measured without training data
    • Some models scored noticeably lower on fresh, matched problems than on GSM8K
  4. What overlap analysis did the GPT-3 paper run?
    • Embedding similarity on full documents
    • 13-gram overlap between benchmarks and training data
    • Human review of every training document
    • Min-K% Prob on each benchmark
  5. What is the core idea of Min-K% Prob?
    • Removing the least common k percent of words from training data
    • Training a second model to classify contaminated text
    • Text the model has seen tends to have few very low-probability tokens
    • Measuring the length of the model's answers
  6. How does a time-based split, as in LiveCodeBench, help detect contamination?
    • It compares performance on problems published before and after the model's training cutoff
    • It measures how long the model takes to answer
    • It splits the model into older and newer layers
    • It only counts problems answered within a time limit
  7. Why does finding no exact overlap not prove a benchmark is clean?
    • Exact matching is too slow to run
    • Benchmarks change their answers over time
    • Models never memorize exact text
    • Paraphrased or translated copies can slip past string matching
  8. In n-gram decontamination, what happens if n is chosen too small?
    • Near-copies with small edits are missed
    • Many false positives from common phrases are flagged
    • The filter stops working entirely
    • Only answers, not questions, are matched