Yoon yiy yamale ngir reso neuronal
Yooni eskalaasioŋ formul empirik lañu yuy wane ni ñàkka am reso neuronal dafay wàññeeku bu baax suñu màggee dayo model bi, dayo done yi ak xayma yi.
Résumé
They matter because they let researchers forecast performance before spending millions on training a giant model.
Plongeur bu xóot
Yooni scaling, siiwal ci OpenAI's këyitu 2020 bi Kaplan ak ay naataango bind, dafa wane ni test perte dafay wàññeeku ni yoon bu yomb ci ñatti anam: lim parametre (N), jetons tàggat (D), ak lim total (C). Buñu ko rësee ci axes log-log, perte ak facteur bu nekk dafay forme ligne bu jub buy jaar ci rang magnitude yu bari. Relasioŋ yi dañu am formu Perte ≈ a + b·X^(-c), fu X nekk facteur de échelle. Li gëna am solo mooy liggéey bu njëkk bi dafa wane ni dayo model bi moo gëna am solo ci done yi, loolu moo waral ñuy wut model yu gëna mag yu melni 175 milyaari parametre yu GPT-3. Yooni scaling yi dañu soppi jàng bu xóot bi juge ci guesswork mu nekk disipline ingénieur buñ mëna seentu, may ekip yi ñu mëna seentu resultaa yu yaatu yi ci ay experience yu ndaw te yomb.
Gis-gis xarala
Formu yoon bu am doole bi dafay tekki ni yokkute bu nekk ci xayma bi dafay jur daanu buy yokk ci perte bi. Ñu ngi koy natt ci nats wala bit ci token bu nekk ci entropie croix. Ndax exponent c dafa tuuti (dafay faral di tollu ci 0.05-0.1), benefiis yi dañu am waaye dañuy wàññeeku: ñaari yoon xayma dafay gëna néew lu muy jàppale ñaari yoon yu njëkk yi. Li am solo mooy yoon yii dañuy fësal ñàkka mëna wàññeeku-plus-mën wàññeeku, fu ab term budul soppeeku di jàpp entropi intrinsèque bu done yi te benn model mënu ko raw.
njeextalu pexe
dogal yu gëna leer
Daf lay jàppale nga tàqale kàddu yu leer ci wàllu xarala ak làkku fësal njaay.
Njëgg ak budget
Mën nga laaj laaj yu gëna baax ci samp gi balaa ngay dugal xaalis wala sa jotu liggéey.
Ekip ak def liggéey
Ekip yi bokk xam-xam ñoo gëna mëna jël yenn dogal ci wàllu produit, politik ak jàng.
Ëlëgu yooni scaling ngir reso neuronal
Gëstukat yi dañuy yaatal yooni scaling bi weesu ñàkka tàggat ci liggéey bu gëna jub, xeetu multimodal, ak xayma ci diiru inference, fu xeetu xalaat yi di gëna yàgg di xalaat ci laaj bu nekk. Ginaaw mbind yu am kalite di gëna néew, nit ñi dañuy bàyyi seen xel ci kalite done yi, done yuñ defaree ay done, ak sàrti eskaalu done yu bari. Ñenn ñi dañu jàpp ni scaling raw mingi dem ba yegg ci xaalis, energie, ak bind yiñ mëna am, di puus barab bi ci efficacité algorithmic ak architectures yu bees moo gën ñu tabax lu gëna rëy.
Doxal ci àdduna dëgg
Xayma luy waaja ñàkk ci modelu 70 milyaar ciy paramet ci ay test yu ndaw yu am 100 milioŋ ciy paramet balaa ñuy jël budget GPU.
Xalaatal ñaata trillion ci jetons yi nga wara dajale suko defee budget bu takku bi baña yàqu ci model buñu tàggatul.
Teg ñaari architecture ci anam wu yomb, boole seeni courbe scaling ci escale bu ndaw, moo gën ñu tàggat ñoom ñaar ci dayo bu mat sëkk.
Teg ay xaar-xaar yu dëggu ngir dugal xaalis wala xoolaatkat yi ci extrapoler courbe perte ci niveau calcul biñ bëgga am.
Risk yi ak balustrade yi
Ekip yu bari mën nañu jëfandikoo benn baat ci anam wu wuute, kon teela leeral yaatuwaayam.
Benchmark yi mën nañu nuru lu am doole waaye performance yi ci àdduna bi duñu tolloo.
Bëgg kalite done ak palaŋu jàngat dafay faral di jur njariñ yu yomba dagg.
Roadmap ngir samp gi
Tàmbaleel ci joxe leeral ci làkk wu leer ci njariñ li nga soxla.
Tannal benn metric bu baax ak benn anam bu baaxul balaa ngay saytu.
Doxal ab pilote bu ndaw ak ay done yu representatif, du ab demo bu leer.
Dokument fu Scaling Laws ngir Reseau Neural di jàppale ak fu pexe yu gëna yomba gëna baax.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Scaling Laws for Neural Networks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Reseau neuronal konvolusionel
Laaj yi ñuy faral di laaj
What is Scaling Laws for Neural Networks?
Yooni eskalaasioŋ formul empirik lañu yuy wane ni ñàkka am reso neuronal dafay wàññeeku bu baax suñu màggee dayo model bi, dayo done yi ak xayma yi. Dañu am solo ndax dañu may gëstukat yi ñu seetlu li ñuy def balaa ñuy dugal ay milioŋ ci tàggat model bu mag.
Ci axes log-log, naka la test loss di doxee ginaaw bi calcul bi yokkee ci wàllu escalier?
Perte ak xayma, dayo model bi, wala done yi dañuy forme liiñ bu jege jub ci kaw parsel log-log, siife relasioŋ yoon-power.
Yan ci ñatti limu yooni scaling yi njëkka wax ci perte?
Kaplan ak ñeneen. jàngat ñàkk ni yoon doole ci lim parametre (N), jeton tàggat (D), ak lim total (C).
Lan moo waral loi scaling am solo lool ci labo IA yi?
Su ekip yi boole ay courbe ci escale bu ndaw, dañuy seetlu model bu mag bi balaa ñuy dugal xaalis bu bari.
Lan mooy terme constant (bu mënul wàññeeku) ci formul perte ci loi scaling?
Loss amna wàll wuñu mëna wàññi buy wàññeeku ak eskaal bi boole ci dër bu mënul wàññeeku buñu tëral ci randomness bi ci done yi.
Lan moo waral ñuy wax ni njariñu scaling dafay 'wàññeeku'?
Ndax exponent yoon-power bi dafa tuuti, yokkuteg calcul multiplicatif bu tolloo dafay gëna néew luñuy wàññi absolu perte.