Chinchilla Compute-Optimal Training
Chinchilla ndeye 2022 DeepMind ichiwana kuti mizhinji mizhinji yemitauro mikuru yakanga isina kudzidziswa zvakanyanya: kune yakamisikidzwa komputa bhajeti iwe unofanirwa kuyera maparamita nedata zvakaenzana, kwete kungovaka modhi hombe.
Pfupiso
It reshaped how the industry balances model size against training data.
Kudzika Kwakadzika
DeepMind's Chinchilla bepa rakadzokorodza kuyera uye kudzidzisa anopfuura mazana mana modhi kuti uwane iyo compute-yakanyanya chiyero. Musoro wekutonga wechigunwe: saizi yemuenzaniso uye tokeni dzekudzidzisa dzinofanirwa kukura mukuvhara, dzinenge makumi maviri ekudzidzisa tokeni paparamita. Kuti vazviratidze, vakadzidzisa Chinchilla, 70-bhiriyoni-parameter modhi pa 1.4 tiririyoni tokens, vachishandisa komputa yakafanana neiyo 280-bhiriyoni-parameter Gopher akadzidziswa pane mashoma ma tokens. Chinchilla, kunyangwe idiki zvakapetwa kana, yakapfuura Gopher, GPT-3, nemamwe mahofori paanenge ese mabhenji. Chidzidzo chakapidigura mhedziso yepakutanga OpenAI iyo yaifarira saizi pane data, zvichiratidza mhando zhinji dzemhando yepamusoro dzaive dzichisiya kuita patafura nekukurisa uye kuziya nenzara.
Technical Insight
Chinchilla fit loss as L(N,D) = E + A·N^(-α) + B·D^(-β), ine α ne β zvese zviri pedyo ne0.34, zvichireva maparamita nedata zvinopa zvakada kuita symmetrically. Kugadzirisa izvi pasi pezvakagadziriswa compute constraint (compute ≈ 6 · N·D yevashanduri) inobereka mhedzisiro yakaenzana. Iyo diki, ine data-yakapfuma modhi zvakare yakachipa kumhanya pakunongedza, saka mukana wayo unobatanidza mukutumirwa, kwete kungodzidziswa.
Strategic Impact
Sarudzo dzakajeka
Inokubatsira kuparadzanisa zvakajeka zvichemo zvehunyanzvi kubva mumutauro wekushambadzira.
Mutengo uye bhajeti
Iwe unogona kubvunza zvirinani kuita mibvunzo usati washandisa mari kana nguva.
Team uye workflow
Zvikwata zvine nzwisiso yakagovaniswa inoita zvirinani chigadzirwa, mutemo, uye sarudzo dzekudzidza.
Ramangwana reChinchilla Compute-Optimal Training
Mhando dzemazuva ano dzakaita seLlama 3 dzinosundidzira nemaune kupfuura Chinchilla's 20-tokens-per-parameter ratio, kudzidzisa mamodhi madiki pamatiririyoni ematokeni kuita kufungidzira kuchipa, kubvuma suboptimal kudzidziswa komputa. Sezvo data yakanaka ichikura kushomeka, kufarira kuri kuwedzera mukudzokororwa epochs, synthetic data, uye kusefa kwemhando. Chinchilla inoramba iri nzvimbo yereferensi, asi iyo optimum inowedzera zvinoenderana nehupenyu hwekufungidzira mutengo, kwete chete bhajeti yenguva imwe yekudzidzisa.
Real-World Implementation
Kusarudza kudzidzisa 7-bhiriyoni-parameter modhi pa 2 tiririyoni tokens pane 30-bhiriyoni modhi pane yakawandisa data kune imwechete bhajeti.
Kufungidzira kuti 10-bhiriyoni-parameter modhi inoda angangoita 200 bhiriyoni tokens kurova compute-optimal inotapira nzvimbo.
Kururamisa modhi diki yakaisirwa kudzikisa mutengo wemubvunzo wega wega uchifananidza hunhu hwemukwikwidzi.
Kuongorora iyo iripo modhi uye nekuipedzisa yaive undertrained, wozoronga kureba kwekudzidzira kumhanya pane kuwedzera parameter.
Njodzi & Guardrails
Zvikwata zvakasiyana zvinogona kushandisa izwi rimwechete zvakasiyana, saka tsanangura nzvimbo nekukurumidza.
Benchmarks inogona kutaridzika yakasimba nepo chaiyo-yenyika kuita isina kuenzana.
Kuregeredza mhando yedata uye zvirongwa zvekuongorora zvinowanzogadzira mhedzisiro isina kusimba.
Implementation Roadmap
Tanga netsanangudzo yemutauro wakajeka yemhedzisiro yaunoda.
Sarudza metric imwe yekubudirira uye imwe yekutadza mamiriro usati waedzwa.
Mhanya mutyairi mudiki ane data remumiriri, kwete demo rakakwenenzverwa.
Gwaro uko Chinchilla Compute-Optimal Training inobatsira uye uko nzira dzakareruka dziri nani.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Chinchilla Compute-Optimal Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Kudzidzira Nguva Yekuedza
Mibvunzo inowanzo bvunzwa
What is Chinchilla Compute-Optimal Training?
Chinchilla ndeye 2022 DeepMind ichiwana kuti mizhinji mizhinji yemitauro mikuru yakanga isina kudzidziswa zvakanyanya: kune yakamisikidzwa komputa bhajeti iwe unofanirwa kuyera maparamita nedata zvakaenzana, kwete kungovaka modhi hombe. Yakagadzirisazve kuti indasitiri inoyera sei saizi yemuenzaniso inopesana nedata rekudzidzisa.
Ndeupi mutemo wakakurumbira weChinchilla wekombuta-yakanyanya kudzidziswa?
DeepMind yakawana paramita uye tokeni dzinofanirwa kuyera pamwe chete painoda kusvika makumi maviri tokens paparameta kune yakapihwa compute bhajeti.
Ko 70B-parameter Chinchilla yakaenzaniswa sei ne280B-parameter Gopher?
Akadzidziswa pane mamwe ma tokens ane komputa imwe chete, Chinchilla yakakunda Gopher yakakura kupfuura mamwe mabhenji.
Nderipi dambudziko guru rakaburitswa nebepa reChinchilla pamusoro pemhando dzakakura dzakapfuura?
Mamodheru akaita seGPT-3 neGopher aive akakura zvakanyanya kune data ravo rekudzidziswa, achisiya kuita kusingaite.
Anenge mangani ma tokens ayo mutemo weChinchilla unoratidza kune gumi-bhirioni-parameter modhi?
Pa 20 tokens per parameter, 10 bhiriyoni parameters inoreva nezve 200 bhiriyoni ekudzidziswa tokens.
Kunze kwekuita basa rekudzidzisa, nei idiki, ine data-yakapfuma modhi ichikwezva mukutumirwa?
Mashoma ma paramita anoreva kudzika pa-mubvunzo komputa, saka iyo yekuchengetedza inosanganiswa pese paishandiswa modhi.