Yoon yiy yamale Chinchilla
Yoon scaling Chinchilla, yu bawoo ci DeepMind ci 2022, dafa wane ni modelu làkk yu mag yi bari wuñu woon luñu tàggat bu baax: ngir am budget ordinatër bu takku, danga wara scaleer dayo model bi ak done yiñ tàggat ci anam wu tolloo.
Résumé
It matters because it redefined what 'optimal' model size means and reshaped how labs spend compute.
Plongeur bu xóot
Laata Chinchilla, tendaas bi mooy tabax model yu gëna rëy (lu melni 175B-parametre GPT-3) ci di tàggat ci ay done yu néew. DeepMind tàggat na lu ëpp 400 xeetu ci dayo yu bari ak budget done, ba noppi ñu méngale ay courbe yuy wax luy ñàkk ci fonction parametre ak jeton ci suufu budget fixe (FLOP). Seenug gis-gis: parametre yi ak token tàggat yi dañu wara bokk scale, lu tollu ci 1-ci-1 ratio, loolu dafay tekki lu tollu ci 20 token ci done tàggat ci parametre bu nekk. Ngir firnde ko, ñu tàggat Chinchilla, xeetu 70B-paramet ci 1.4 trillion jetons, mu raw Gopher bu gëna mag 280B-paramet doonte dafa jëfandikoo benn ordinatër, ndax dañu ko tàggat ci done yu bari.
Gis-gis xarala
Yoon yi dañu bawoo ci méngale ab fonction perte parametrik L(N, D) fu N nekk ay parametre ak D ay jeton, boole ci perte buñu mënul wàññi, dayo model, ak terme dayo done. Wàññi perte bi aju ci benn tënk ci xayma (jëfandikoo lu tollu ci N yoon D) dafay jur njariñu N ak D yi gëna baax ñoom ñaar ñuy màgg ni dooley xayma ak exponent yu noonu, kon ratio xayma-gëna des ci wetu 20 jetons ci parametre bu nekk.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu Chinchilla ci yooni eskalaasioŋ
Chinchilla dafa soppi wàll wi ci topp lim parametre yi ngir joxe model yu gëna am kalite, te model yu bees yi dañuy faral di tàggat bu baax ba romb poñ 'compute-optimal' ngir gëna xéewale inference. Bi mbindu web yu baax di gëna néew, nit ñi dañuy wëlbatiku ci curation done, done synthetik, epoch yu bari, ak done multimodal ngir wéy di scaling. The core lesson endures: data and parameters must be balanced, and raw size alone is no longer the goal.
Doxal ci àdduna dëgg
DeepMind's 70B-parametre Chinchilla moo raw Gopher 280B ci ay référence di jëfandikoo ordinatër buy méngoo, ci di tàggat ci ay done yu bari
Di jàppale ekip yi ñu budget lu tollu ci 20 jetons tàggat ci parametre bu nekk suñuy waajal ab model bu tàmbalee ci noonu
Justifier model yu gëna ndaw, yu bari ay done yu melni LLaMA yu gëna xéewale ci diiru inference
Xayma ndax xeetu waajal bi 'ñu tàggatul bu baax' te dina gëna am njariñ ci ay done yu gëna bari ay parametre yu gëna bari
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Chinchilla Scaling Laws quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Yoon yiy yamale ngir reso neuronal
Laaj yi ñuy faral di laaj
What is Chinchilla Scaling Laws?
Yoon scaling Chinchilla, yu bawoo ci DeepMind ci 2022, dafa wane ni modelu làkk yu mag yi bari wuñu woon luñu tàggat bu baax: ngir am budget ordinatër bu takku, danga wara scaleer dayo model bi ak done yiñ tàggat ci anam wu tolloo. Dafa am solo ndax dafa joxe leeral yu bees ci li 'optimal' model size tekki ba noppi soppi anam wi lab yi di jëfandikoo ordinatër.
Lan mooy liñu gis ci yooni eskaalu Chinchilla?
Chinchilla dafa wane ni ngir am budget ordinatër bu takku, paramet yi ak jeton tàggat yi dañu wara bokk màgg, lu tollu ci 1-ci-1.
Lu tollu ci ñaata jeton tàggat ci parametre bu nekk la Chinchilla digle ni moo gëna baax ci xayma?
Rasio xayma bi gëna baax mingi tollu ci 20 jeton tàggat ci parametru model bu nekk.
Ban xeetu Chinchilla moo gëna am doole doonte moo gëna ndaw?
Chinchilla bi am paramet 70B moo raw Gopher bi am paramet 280B bu DeepMind ci jëfandikoo benn ordinatër bi, ndax dafa tàggat ci done yu bari.
Lan la Chinchilla bëgga wax ci model yu melni GPT-3 ci jamono jooju?
Modèle yu mag yu bari ci jamono jooju deñu leen tàggatoon bu baax, loolu dafay tekki ni suñu amee ay done yu bari ci seeni parametre, ñoo gënoon mëna def seen liggéey.
Ci gàttal, naka lañu xaymaa xayma ci jàngat bu Chinchilla bi?
calcul de formation (FLOPs) mingi méngoo ak limu parametre yi ñu yokk ko ci limu jetons de formation yi.