Overview
Chinchilla ndeye 2022 DeepMind ichiwana kuti mizhinji mizhinji yemitauro mikuru yakanga isina kudzidziswa zvakanyanya: kune yakamisikidzwa komputa bhajeti iwe unofanirwa kuyera maparamita nedata zvakaenzana, kwete kungovaka modhi hombe. Yakagadzirisazve kuti indasitiri inoyera sei saizi yemuenzaniso inopesana nedata rekudzidzisa.
Chinchilla Compute-Optimal Training inogara mukati meiyo AI toolkit. Paunonzwisisa, mamwe maAI misoro inova nyore kuongorora uye kuenzanisa.
Deep Dive
DeepMind's Chinchilla bepa rakadzokorodza kuyera uye kudzidzisa anopfuura mazana mana modhi kuti uwane iyo compute-yakanyanya chiyero. Musoro wekutonga wechigunwe: saizi yemuenzaniso uye tokeni dzekudzidzisa dzinofanirwa kukura mukuvhara, dzinenge makumi maviri ekudzidzisa tokeni paparamita. Kuti vazviratidze, vakadzidzisa Chinchilla, 70-bhiriyoni-parameter modhi pa 1.4 tiririyoni tokens, vachishandisa komputa yakafanana neiyo 280-bhiriyoni-parameter Gopher akadzidziswa pane mashoma ma tokens. Chinchilla, kunyangwe idiki zvakapetwa kana, yakapfuura Gopher, GPT-3, nemamwe mahofori paanenge ese mabhenji. Chidzidzo chakapidigura mhedziso yepakutanga OpenAI iyo yaifarira saizi pane data, zvichiratidza mhando zhinji dzemhando yepamusoro dzaive dzichisiya kuita patafura nekukurisa uye kuziya nenzara.
Technical Insight
Chinchilla fit loss as L(N,D) = E + A·N^(-α) + B·D^(-β), ine α ne β zvese zviri pedyo ne0.34, zvichireva maparamita nedata zvinopa zvakada kuita symmetrically. Kugadzirisa izvi pasi pezvakagadziriswa compute constraint (compute ≈ 6 · N·D yevashanduri) inobereka mhedzisiro yakaenzana. Iyo diki, ine data-yakapfuma modhi zvakare yakachipa kumhanya pakunongedza, saka mukana wayo unobatanidza mukutumirwa, kwete kungodzidziswa.
Mastering Chinchilla Compute-Optimal Training
Chinchilla ndeye 2022 DeepMind ichiwana kuti mizhinji mizhinji yemitauro mikuru yakanga isina kudzidziswa zvakanyanya: kune yakamisikidzwa komputa bhajeti iwe unofanirwa kuyera maparamita nedata zvakaenzana, kwete kungovaka modhi hombe. Yakagadzirisazve kuti indasitiri inoyera sei saizi yemuenzaniso inopesana nedata rekudzidzisa. Chinchilla Compute-Optimal Training inogara mukati meiyo AI toolkit. Paunonzwisisa, mamwe maAI misoro inova nyore kuongorora uye kuenzanisa. Kuti uvake kunzwisisa kwakadzama, tora Chinchilla Compute-Optimal Training semuenzaniso wekushanda, kwete chinhu chimwe chete: tsanangura zvinodikanwa, kujekesa fungidziro, uye patsanura zvinogona kuitwa nehurongwa hwakavimbika kubva kune izvo zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa Chinchilla Compute-Optimal Training zvinovaka mamodheru akasimba ekutanga, obva anyora iwo modhi kune zvipingaidzo chaizvo zvekugadzira. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Inokubatsira kuparadzanisa zvakajeka zvichemo zvehunyanzvi kubva mumutauro wekushambadzira. Panguva imwecheteyo, Zvikwata zvakasiyana zvinogona kushandisa izwi rimwechete zvakasiyana, saka tsanangura nzvimbo nekukasira. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Inokubatsira kuparadzanisa zvakajeka zvichemo zvehunyanzvi kubva mumutauro wekushambadzira.
Inokubatsira kuparadzanisa zvakajeka zvichemo zvehunyanzvi kubva mumutauro wekushambadzira. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Iwe unogona kubvunza zvirinani kuita mibvunzo usati washandisa mari kana nguva.
Iwe unogona kubvunza zvirinani kuita mibvunzo usati washandisa mari kana nguva. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Zvikwata zvine nzwisiso yakagovaniswa inoita zvirinani chigadzirwa, mutemo, uye sarudzo dzekudzidza.
Zvikwata zvine nzwisiso yakagovaniswa inoita zvirinani chigadzirwa, mutemo, uye sarudzo dzekudzidza. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Kusarudza kudzidzisa 7-bhiriyoni-parameter modhi pa 2 tiririyoni tokens pane 30-bhiriyoni modhi pane yakawandisa data kune imwechete bhajeti.
Kufungidzira kuti 10-bhiriyoni-parameter modhi inoda angangoita 200 bhiriyoni tokens kurova compute-optimal inotapira nzvimbo.
Kururamisa modhi diki yakaisirwa kudzikisa mutengo wemubvunzo wega wega uchifananidza hunhu hwemukwikwidzi.
Kuongorora iyo iripo modhi uye nekuipedzisa yaive undertrained, wozoronga kureba kwekudzidzira kumhanya pane kuwedzera parameter.
Maitiro Ekuita
Chinchilla Compute-Yakanyanya Kudzidziswa mukuita
Kusarudza kudzidzisa 7-bhiriyoni-parameter modhi pa 2 tiririyoni tokens pane 30-bhiriyoni modhi pane yakawandisa data kune imwechete bhajeti.
Kusarudza kudzidzisa 7-bhiriyoni-parameter modhi pa 2 tiririyoni modhi pane 30-bhiriyoni modhi pane yakawandisa dhata kune imwechete bhajeti Matimu anowanzo kuwana mhedzisiro iri nani kana vachitsanangudza hunhu hwepamberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Chinchilla Compute-Yakanyanya Kudzidziswa mukuita
Kufungidzira kuti 10-bhiriyoni-parameter modhi inoda angangoita 200 bhiriyoni tokens kurova compute-optimal inotapira nzvimbo.
Kufungidzira kuti bhiriyoni gumi-mabhirioni-parameta modhi inoda angangoita mabhiriyoni mazana maviri ematokeni kurova iyo compute-yakakwana inotapira nzvimbo Matimu anowanzo kuwana mibairo iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Chinchilla Compute-Yakanyanya Kudzidziswa mukuita
Kururamisa modhi diki yakaisirwa kudzikisa mutengo wemubvunzo wega wega uchifananidza hunhu hwemukwikwidzi.
Kururamisa diki rakaisirwa modhi kudzikisa mutengo we-mubvunzo wekufungidzira uchifananidza mukwikwidzi akakura emhando Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Chinchilla Compute-Yakanyanya Kudzidziswa mukuita
Kuongorora iyo iripo modhi uye nekuipedzisa yaive undertrained, wozoronga kureba kwekudzidzira kumhanya pane kuwedzera parameter.
Kuongorora iyo iripo modhi uye nekugumisa kuti yakanga isina kudzidziswa, wozoronga kudzidziswa kwakareba pachinzvimbo chekuwedzera paramende Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura mhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Zvikwata zvakasiyana zvinogona kushandisa izwi rimwechete zvakasiyana, saka tsanangura nzvimbo nekukurumidza.
Benchmarks inogona kutaridzika yakasimba nepo chaiyo-yenyika kuita isina kuenzana.
Kuregeredza mhando yedata uye zvirongwa zvekuongorora zvinowanzogadzira mhedzisiro isina kusimba.
Implementation Roadmap
Tanga netsanangudzo yemutauro wakajeka yemhedzisiro yaunoda.
Tanga netsanangudzo yemutauro wakajeka yemhedzisiro yaunoda. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Sarudza metric imwe yekubudirira uye imwe yekutadza mamiriro usati waedzwa.
Sarudza metric imwe yekubudirira uye imwe yekutadza mamiriro usati waedzwa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Mhanya mutyairi mudiki ane data remumiriri, kwete demo rakakwenenzverwa.
Mhanya mutyairi mudiki ane data remumiriri, kwete demo rakakwenenzverwa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Gwaro uko Chinchilla Compute-Optimal Training inobatsira uye uko nzira dzakareruka dziri nani.
Gwaro uko Chinchilla Compute-Optimal Training inobatsira uye uko nzira dzakareruka dziri nani. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.