Chinchilla - Tàggat yaram bu gëna baax
Chinchilla ab DeepMind bu 2022 gisna ni modelu lakk yu mag yi bariwoon nañu ci anam wu baaxul: ngir budget ordinatër bu takku danga wara scale parametre yi ak done yi ci anam wu tolloo, baña tabax model bu gëna mag rek.
Résumé
Dafa soppi anam wi liggéeyukaay bi di ekilibre dayo model bi ak done yiñ tàggat.
Plongeur bu xóot
Këyitu Chinchilla bu DeepMind xoolaatna eskalaasioŋ ba noppi tàggat lu ëpp 400 model ngir gis ekilibre bi gëna baax ci xayma. Regle bu am solo bi mooy: dayo model bi ak jeton tàggat yi dañu wara màgg ci lockstep, lu tollu ci 20 jeton tàggat ci parametre bu nekk. Ngir firnde ko, dañu tàggat Chinchilla, benn model bu am 70 milyaar ciy parametre ci 1.4 trillion ciy jeton, ñu jëfandikoo benn calcul bi Gopher bu am 280 milyaar ciy parametre tàggat ci jeton yu néew lool. Chinchilla, ak limu gëna ndaw ñeenti yoon, dafa raw Gopher, GPT-3, ak yeneen mag ci daanaka bépp référence. Njàngale mi dafa soppi OpenAI conclusion bi ñu njëkka wax ni dafa taamu dayo ci done, di wane ni model flagship yu bari dañu bàyyi performance ci kaw taabal ji ndax dañu yaatu lool te xiif lool ci done.
Gis-gis xarala
Chinchilla mën na ñàkk L(N,D) = E + A·N^(-α) + B·D^(-β), α ak β ñoom ñaar jege nañu 0.34, loolu dafay tekki ni parametre yi ak done yi dañuy bokk ci anam wu tolloo. Optimiser lii ci biir benn constraint calcul fixe (calcul ≈ 6·N·D ngir transformateur yi) dafay joxe resultaa biy méngoo. Benn model bu gëna ndaw, bari ay done moo gëna xéewale ci inference, kon njariñam mingi aju ci jëfandikoo gi, du tàggat rek.
njeextalu pexe
dogal yu gëna leer
Daf lay jàppale nga tàqale kàddu yu leer ci wàllu xarala ak làkku fësal njaay.
Njëgg ak budget
Mën nga laaj laaj yu gëna baax ci samp gi balaa ngay dugal xaalis wala sa jotu liggéey.
Ekip ak def liggéey
Ekip yi bokk xam-xam ñoo gëna mëna jël yenn dogal ci wàllu produit, politik ak jàng.
Ëlëgu Chinchilla ci tàggat yaram
Modèle yu bees yu melni Llama 3 dañuy puus fu sori ci Chinchilla 20-jetons-parametre bu nekk, di tàggat model yu ndaw ci ay trillion jetons ngir mëna def inference bu yomb, nangu tàggat ordinatër bu baaxul. Ginaaw done yu baax dañuy gëna néew, nit ñi dañuy gëna bëgg ay epoch yu bari, ay done yuñ defaree ay done, ak ay segg yu baax. Chinchilla mingi wéy di nekk royuwaay bi, waaye li gëna baax mingi aju ci njëgu dundu gi, te baña aju ci budget tàggat yaram bu benn yoon kese.
Doxal ci àdduna dëgg
Tann ngir tàggat xeetu parametre 7-milyaar ci 2 trillion jetons moo gën xeetu 30-milyaar ci done yu néew lool ngir benn budget.
Xayma nañu ni benn model bu am 10 milyaar ci paramet yi defa bëgg lu tollu ci 200 milyaar ciy jeton ngir mëna dem ba ci barab bi gëna baax ci xayma.
Justifier model bu gëna ndaw biñ dugal ngir wàññi njëgu inference bu laaj bu nekk boole ci méngoo ak kalite rival bu gëna mag.
Saytu ab xeetu model bu nekk te gis ni deful luñu ko tàggat, ba noppi waajal ab tàggat bu gëna gudd ci barabu yokk ay parametre.
Risk yi ak balustrade yi
Ekip yu bari mën nañu jëfandikoo benn baat ci anam wu wuute, kon teela leeral yaatuwaayam.
Benchmark yi mën nañu nuru lu am doole waaye performance yi ci àdduna bi duñu tolloo.
Bëgg kalite done ak palaŋu jàngat dafay faral di jur njariñ yu yomba dagg.
Roadmap ngir samp gi
Tàmbaleel ci joxe leeral ci làkk wu leer ci njariñ li nga soxla.
Tannal benn metric bu baax ak benn anam bu baaxul balaa ngay saytu.
Doxal ab pilote bu ndaw ak ay done yu representatif, du ab demo bu leer.
Dokument fu Chinchilla Compute-Optimal Tàggat di jàppale ak fi pexe yu gëna yomba gëna baax.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Chinchilla Compute-Optimal Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Taggat ci waxtu test
Laaj yi ñuy faral di laaj
Luy tàggat yaram bu baax ci Chinchilla?
Chinchilla ab DeepMind bu 2022 gisna ni modelu lakk yu mag yi bariwoon nañu ci anam wu baaxul: ngir budget ordinatër bu takku danga wara scale parametre yi ak done yi ci anam wu tolloo, baña tabax model bu gëna mag rek. Dafa soppi anam wi liggéeyukaay bi di ekilibre dayo model bi ak done yiñ tàggat.
Lan mooy sàrt bu siiw bi Chinchilla tëral ngir tàggat yaram ci ordinatër?
DeepMind gisna ni parametre yi ak jeton yi dañu wara boole lu tollu ci 20 jeton ci parametre bu nekk ngir benn budget buñ jox.
Lan moo méngoo ak Chinchilla bi am 70B ak Gopher bi am 280B?
Chinchilla dafa tàggat ci jeton yu bari ak benn ordinatër, moo daan Gopher bi gëna mag ci benchmark yu bari.
Ban jafe-jafe bu mag la këyitu Chinchilla wane ci model yu mag yi njëkka am?
Modèle yu melni GPT-3 ak Gopher dañu yaatuwoon lool buñu ko méngale ak seeni done yuñ tàggat, moo tax ñu baña amal seen performance.
Lu tollu ci ñaata jeton la sàrtu Chinchilla digle ngir xeetu 10-milyaar ciy parametre?
Ak 20 jetons ci parametre bu nekk, 10 milyaari parametre dafay tekki luy tollu ci 200 milyaari jetons tàggat.
Lu weesu njariñu tàggat yaram, lu tax xeetu model bu gëna ndaw te bari ay done di xëcc nit ñi ci jëfandikoo gi?
Parametre yu néew ñooy gëna néew xayma ci laaj bu nekk, kon sakkanal bi dafay yokk saa yu ñuy jëfandikoo model bi.