Chinchilla Scaling Mitemo
Mitemo yeChinchilla yekuyera, kubva kuDeepMind muna 2022, yakaratidza kuti mhando dzemitauro mikuru mikuru yakanga isina kudzidziswa zvakanyanya: kune yakamisikidzwa komputa bhajeti, iwe unofanirwa kuyera saizi yemuenzaniso uye data rekudzidzisa zvakaenzana muchikamu chakaenzana.
Pfupiso
It matters because it redefined what 'optimal' model size means and reshaped how labs spend compute.
Kudzika Kwakadzika
Pamberi peChinchilla, maitiro aive ekuvaka mamodheru anogara akakura (seiyo 175B-parameter GPT-3) uchidzidzira nezve data ine mwero. DeepMind yakadzidzisa anopfuura mazana mana emodhi muhukuru hwakawanda uye mabhajeti edatha, yobva yakwana macurves kufanotaura kurasikirwa sebasa rema paramita uye tokens pasi peyakagadziriswa komputa (FLOP) bhajeti. Kuwana kwavo: maparamendi uye ma tokeni ekudzidzisa anofanirwa kuyera pamwe chete, ingangoita 1-kusvika-1 reshiyo, zvichireva nezve makumi maviri tokeni edhata rekudzidzisa paparameta. Kuti azviratidze, vakadzidzisa Chinchilla, 70B-parameter womuenzaniso pa 1,4 tiririyoni zviratidzo, izvo kupfuura zvikuru guru 280B-parameter Gopher pasinei kushandisa komputa imwe chete, nokuti akadzidziswa pamusoro kure zvikuru mashoko.
Technical Insight
Mitemo inobva pakukodzera parametric kurasikirwa basa L (N, D) uko N iri paramita uye D iri tokens, kusanganisira irreducible-kurasikirwa, modhi-saizi, uye data-saizi mazwi. Kuderedza kurasikirwa zvichienderana necompute constraining (compute inenge yakaenzana neN nguva D) inopa mhedzisiro yekuti iyo yakakwana N uye D zvese zvinokura sesimba rekombuta ine maexponents akafanana, saka compute-optimal reshiyo inogara pedyo ne20 tokens paparameter.
Strategic Impact
Kumhanya uye chiyero
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Svika uye svika
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Sarudzo dzakajeka
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Ramangwana reChinchilla Kuyera Mitemo
Chinchilla yakashandura munda kubva pakudzingirira kuverenga kweparameta kuenda kumhando dzekudyisa data dzemhando yepamusoro, uye mhando dzemazuva ano dzinowanzo dzidzisa kupfuura iyo 'compute-optimal' poindi kuita kuti inferensi idhure. Sezvo zvinyorwa zvemhando yepamusoro zvewebhu zvave kushomeka, kutarisisa kuri kutendeukira kune data curation, synthetic data, akawanda epochs, uye multimodal data kuti irambe ichikwira. Chidzidzo chepakati chinotsungirira: data uye paramita zvinofanirwa kuve zvakaenzana, uye saizi yakasvibira chete haisisiri chinangwa.
Real-World Implementation
DeepMind's 70B-parameter Chinchilla ichirova 280B Gopher pamabenchmarks vachishandisa compute yakaenzana, nekudzidziswa pane mamwe data.
Kutungamira zvikwata kubhajeti inosvika makumi maviri ekudzidziswa tokeni paparameta kana uchironga kubva-mukutanga modhi
Kururamisa madiki, ane data-akapfuma modhi seLLaMA ayo akachipa kumhanya panguva yekufungidzira
Kufungidzira kana modhi yakarongwa 'yakadzidziswa' uye ingabatsirike zvakanyanya kubva kune yakawedzera data pane yakawedzera paramita
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Chinchilla Scaling Laws quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Kuyera Mitemo yeNeural Networks
Mibvunzo inowanzo bvunzwa
What is Chinchilla Scaling Laws?
Mitemo yeChinchilla yekuyera, kubva kuDeepMind muna 2022, yakaratidza kuti mhando dzemitauro mikuru mikuru yakanga isina kudzidziswa zvakanyanya: kune yakamisikidzwa komputa bhajeti, iwe unofanirwa kuyera saizi yemuenzaniso uye data rekudzidzisa zvakaenzana muchikamu chakaenzana. Izvo zvine basa nekuti yakatsanangudza kuti 'yakakwana' modhi inorevei uye yakagadziridza mashandisiro anoita maLab compute.
Chii chaive chikuru chakawanikwa chemitemo yekuyera yeChinchilla?
Chinchilla yakaratidza kuti kune yakatarwa compute bhajeti, parameters uye zviratidzo zvekudzidzisa zvinofanira kukura pamwe chete, zvinenge 1-to-1.
Angangoita mangani matokeni ekudzidziswa paparameta yakataurwa naChinchilla kuti ndeye compute-yakanyanya?
Iyo compute-optimal reshiyo inoshanda kune angangoita makumi maviri ekudzidziswa tokens yeese modhi parameter.
Ndeipi modhi iyo Chinchilla yakapfuura kunyangwe yaive idiki zvakanyanya?
Iyo 70B-parameter Chinchilla yakarova DeepMind's yega 280B-parameter Gopher ichishandisa komputa imwe chete, nekuti yakadzidzira pane yakawanda data.
Chii chairehwa naChinchilla nezvemhando dzakaita seGPT-3 panguva iyoyo?
Mazhinji mahombe emhando dzenguva iyoyo aive akadzidziswa, zvichireva kuti vangadai vakaita zvirinani nedata rakawanda rekuverenga kwavo parameter.
Zvichida kuti compute yakafananidzwa sei muongororo yeChinchilla?
Kudzidzira compute (FLOPs) inenge yakaenzana nenhamba yezvikamu zvakapetwa nehuwandu hwema tokens ekudzidzira.