GUIDE bu am solo

Kontaminasioŋ Referansi

Benchmark contamination dafay am su laaj wala tontu yu bawoo ci benn benchmark evaluation mujjee nekk ci done yiñ tàggat ci model bi, kon poñ yu bari mën nañu wane li ñuy jàng ci xel, du mën mën dëgg.

Déglul tegtal biiIA-nettali · ~3 min

Ci xët wii4 simili jàng
  1. Résumé
  2. Plongeur bu xóot
  3. Doxal ci àdduna dëgg
  4. Ëlëgu Kontaminasioŋ Benchmark
  5. Teg ko ci jëf
  6. Weyal di banneexu
  7. Laaj yi ñuy faral di laaj

Résumé

Dafa am solo ndax nimero yi ci tablo bi dañuy yóbbe gëstukat yi ak dogal yi ñuy jënd, ba noppi poñ yu baaxul yi dañuy gëna fësal ni model bi di doxee ci jafe-jafe yi mu musul gis.

Plongeur bu xóot

Large language models are trained on web-scale text that includes GitHub, arXiv papers, forums and tutorial sites. Popular benchmarks are public by design, so their questions, answers and discussions of them get copied across the web and swept into training data. Contamination can be direct, where the exact test item appears, or indirect, through paraphrases, translations, solution write-ups, or synthetic data generated by another model that had seen the benchmark. It can also enter during fine-tuning if instruction datasets were assembled from benchmark-like sources. A useful distinction is between seeing only the question and seeing the question with its answer. The second is more damaging, because the model can recall the answer instead of working it out. The result is an inflated score that does not transfer to new problems. Lab awareness is not new. The GPT-3 paper (2020) ran a 13-gram overlap analysis between its benchmarks and training data and acknowledged that a bug left some overlaps in place. When training data is available, overlap search is the most direct check. When it is not, researchers use indirect methods: prompting the model with the start of a test item to see if it completes the rest verbatim; membership inference methods such as Min-K% Prob (Shi and colleagues, 2023), which look at how confidently a model predicts a text's tokens; comparing scores on original items against rewritten or newly written equivalents; and time-based splits that compare problems created before and after the training cutoff. Two misconceptions are common. First, overlap does not always raise scores much; some studies find modest effects on certain tasks. Second, finding no exact overlap does not prove a benchmark is clean, because paraphrased or translated copies slip past string matching. Mitigations include private held-out test sets, canary strings, regularly refreshed benchmarks such as LiveBench and LiveCodeBench, decontamination filters, and publishing contamination analyses alongside results.

Doxal ci àdduna dëgg

Jafe-jafe yi ci benn benchmark ci math dañu leen di publie ci forum yi ak ay pexe yuñ liggéey; web crawl dafay seddale xët yooyu, ba noppi ab model buñu tàggat ci moom mooy génne tontu yu gëna jub, boole ci ab quirk ci kàddu yu njëkk ya.

Gëstukat yi ci Scale AI bind nañu GSM1k, jafe-jafe math bu bees bu méngoo ak GSM8K ci stil ak jafe-jafe, ba noppi ñu gis ni yenn model yi ñoo gëna néew poñ ci set bu bees bi, muy luy màndargaal ni dafa méngoo bu baax ak li ñépp bokk.

Benchmark codage biy date jafe-jafe bu nekk dafay wane ab model buy saafara jafe-jafe yu bari yuñ siiwal balaa ñuy dagg tàggat-yaram bi, te loolu mooy motif bi LiveCodeBench defar ngir wane.

Liggéeyu BIG-bench yi dañuy àndaale ak benn string canary suko defee ekipu done yi mëna segg bépp këyit bu ci nekk ci corpus tàggat, ginaaw ga xool ndax benn model mën nako génne.

Ëlëgu Kontaminasioŋ Benchmark

Jëfandikoo jàngat mingi toxu ci ensemble test yu privé wala yu prive, ay benchmark yuy yokk mbir yu bees ci diir bu gàtt, ak jàngat yu nekk ci digante yeen ak ñoom ñooy saytu done test yi. Loolu moo jur jafe-jafe dëgg ak ubbeeku, ndax benchmarks publik yi yomb nañu joxe ak saytu. Leer gu gëna mag ci done yiñ tàggat mooy gëna yombal saytu kontaminasioŋ bi, waaye defarkati model yu bari duñu siiwal done yépp. Jàngatkat yi dañu wara jëfandikoo bépp poñ buñ joxe ci benchmark static ak xel mu dal, ba noppi seet ci resultaa yi ci done yu bees wala yuñ tëye balaa ñuy jël doggal yu dëgër.

Teg ko ci jëf

Lu tax mu am solo

  • dogal yu gëna leer. Daf lay jàppale nga tàqale kàddu yu leer ci wàllu xarala ak làkku fësal njaay.
  • Njëgg ak budget. Mën nga laaj laaj yu gëna baax ci samp gi balaa ngay dugal xaalis wala sa jotu liggéey.
  • Ekip ak def liggéey. Ekip yi bokk xam-xam ñoo gëna mëna jël yenn dogal ci wàllu produit, politik ak jàng.

Moytul

  • Ekip yu bari mën nañu jëfandikoo benn baat ci anam wu wuute, kon teela leeral yaatuwaayam.
  • Benchmark yi mën nañu nuru lu am doole waaye performance yi ci àdduna bi duñu tolloo.
  • Bëgg kalite done ak palaŋu jàngat dafay faral di jur njariñ yu yomba dagg.

Jéego yi ci topp

  1. Tàmbaleel ci joxe leeral ci làkk wu leer ci njariñ li nga soxla.
  2. Tannal benn metric bu baax ak benn anam bu baaxul balaa ngay saytu.
  3. Doxal ab pilote bu ndaw ak ay done yu representatif, du ab demo bu leer.
  4. Bindal fi Contamination Benchmark di jàppale ak fi pexe yu gëna yomba gëna baax.

Weyal di banneexu

Bulletin bu amul fayda

Jàngal xibaar AI bu bes

Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.

Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.

Nattal sa bopp

Defal quiz bi tuddu Kontaminasioŋ

Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.

Tambalil quiz

Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax

Laaj yi ñuy faral di laaj

Luy kontaminasioŋ référence?

Benchmark contamination dafay am su laaj wala tontu yu bawoo ci benn benchmark evaluation mujjee nekk ci done yiñ tàggat ci model bi, kon poñ yu bari mën nañu wane li ñuy jàng ci xel, du mën mën dëgg. Dafa am solo ndax nimero yi ci tablo bi dañuy yóbbe gëstukat yi ak dogal yi ñuy jënd, ba noppi poñ yu baaxul yi dañuy gëna fësal ni model bi di doxee ci jafe-jafe yi mu musul gis.

Luy kontaminasioŋ référence?

Kontaminasioŋ dafay tekki ni model bi mën na gis macceer test ci diiru tàggat yaram, kon poñ yi mu am mën nañu wane limu jàng ci xel.

Lan mooy njariñu string canary biñ boole ci liggéey BIG-bench?

Benn ràññeekaay bu amul fenn mooy yombal gis ak dindi mbindu benchmark ci corpus ak xool ndax model bi daf ko xam.

Lu gëstu GSM1k bi digle?

Wàññeeku ci jafe-jafe yu bees yu méngoo ci stil ak jafe-jafe dafay wane ni dafa méngoo bu baax ak référence publik bi moo gën xam-xam math bu ñépp bokk.

Lan mooy jàngat biy jaxasoo ci këyit GPT-3 bi?

Auteur yu GPT-3 xool nañu 13-gram overlap ba noppi nangu ni amna bug bu bàyyi yenn overlap yuñu dindiwul.

Lan mooy xalaatu Min-K% Prob?

Bokk na ci li gëna ndaw k pursaa ci probabilite log yi; poñ yiñ tëral ci bind yi dañu yéeg lool ndax du bari luñu ciy yéem ci model bi.