Kuyera Mitemo yeNeural Networks
Mitemo yezviyereso maitiro eempirical anoratidza kuti kurasikirwa kweneural network kunowira kufanofungidzira paunokura modhi saizi, saizi yedataset, uye compute.
Pfupiso
They matter because they let researchers forecast performance before spending millions on training a giant model.
Kudzika Kwakadzika
Mitemo yekuyera, yakakurumbira ne OpenAI's 2020 bepa naKaplan nevamwe vaaishanda navo, vakaona kuti kurasikirwa kwebvunzo kunodzikira semutemo wakatsetseka wesimba muzvikamu zvitatu: parameter count (N), tokens yekudzidzira (D), uye yakazara compute (C). Yakarongerwa palog-log-axes, kurasikirwa maringe nechinhu chimwe nechimwe kunoumba mutsara wakatwasuka unotora akawanda maodha ehukuru. Hukama hunotora fomu Kurasika ≈ a + b · X^(-c), apo X ndiyo inoyera chinhu. Nehurombo, iro basa rekutanga rakakurudzira saizi yemhando yakakosha kupfuura data, zvichiita kuti mujaho uende kumhando dzakakura seGPT-3's 175 bhiriyoni paramita. Mitemo yekuyera yakashandura kudzidza kwakadzama kubva mukufungidzira kuita chirango cheinjiniya, zvichiita kuti zvikwata zvifungidzire mibairo mikuru kubva mukuedza kudiki, kwakachipa.
Technical Insight
Iyo simba-mutemo fomu inoreva imwe neimwe yakamisikidzwa yakawedzera kuwanda kwekuwedzera mukombuta inobereka zvinogara zvichiwedzera kuwedzera kudonha mukurasikirwa. Kurasika kunopimwa nema nats kana mabits pa tokeni ye cross-entropy. Nekuti iyo exponent c idiki (kazhinji yakatenderedza 0.05-0.1), mibairo ndeyechokwadi asi inoderera: kaviri komputa inobatsira zvishoma pane yekutanga kupetwa. Zvakakosha, iyi mitemo inotsanangura kurasikirwa kusingadzoreki-pamwe-kudzokororwa, uko izwi rinogara richitora iyo data yemukati entropy iyo isina modhi inogona kukunda.
Strategic Impact
Sarudzo dzakajeka
Inokubatsira kuparadzanisa zvakajeka zvichemo zvehunyanzvi kubva mumutauro wekushambadzira.
Mutengo uye bhajeti
Iwe unogona kubvunza zvirinani kuita mibvunzo usati washandisa mari kana nguva.
Team uye workflow
Zvikwata zvine nzwisiso yakagovaniswa inoita zvirinani chigadzirwa, mutemo, uye sarudzo dzekudzidza.
Ramangwana reKuyera Mitemo yeNeural Networks
Vatsvagiri vari kuwedzera mitemo yekuyera kupfuura kudzidzira kurasikirwa kusvika pakudzika kwebasa, mamodhi akawanda, uye inference-nguva compute, uko mamodheru ekufunga anopedza kufunga kwakawanda pamubvunzo wega wega. Sezvo zvinyorwa zvemhando yepamusoro zvave kushomeka, kutarisisa kuri kuchinjika kumhando yedata, data rekugadzira, uye inodzokororwa-data yekuyera mitemo. Vamwe vanopokana kuti kuyera kwakashata kuri kurova miganhu inoshanda yemari, simba, uye zvinyorwa zviripo, kusundira munda kune algorithmic kunyatsoshanda uye zvivakwa zvitsva pane kungovaka hombe.
Real-World Implementation
Kufanotaura kurasikirwa kwekupedzisira kweyakarongwa 70-bhiriyoni-parameta modhi kubva kune yakatevedzana madiki 100-miriyoni-parameter bvunzo inomhanya isati yaita bhajeti reGPU.
Kusarudza kuti mangani matrillion ematokeni ekuunganidza kuitira kuti bhajeti rekombuta rakagadziriswa rirege kuraswa pamuenzaniso usina kudzidziswa.
Kuenzanisa zvivakwa zviviri zvakachipa nekukodzera macurves avo pachiyero chidiki pane kudzidzisa ese ari maviri saizi yakazara.
Kuisa tarisiro yechokwadi kune vanoisa mari kana kupa vanoongorora nekuwedzera iyo yekurasikirwa curve kune inotangwa compute level.
Njodzi & Guardrails
Zvikwata zvakasiyana zvinogona kushandisa izwi rimwechete zvakasiyana, saka tsanangura nzvimbo nekukurumidza.
Benchmarks inogona kutaridzika yakasimba nepo chaiyo-yenyika kuita isina kuenzana.
Kuregeredza mhando yedata uye zvirongwa zvekuongorora zvinowanzogadzira mhedzisiro isina kusimba.
Implementation Roadmap
Tanga netsanangudzo yemutauro wakajeka yemhedzisiro yaunoda.
Sarudza metric imwe yekubudirira uye imwe yekutadza mamiriro usati waedzwa.
Mhanya mutyairi mudiki ane data remumiriri, kwete demo rakakwenenzverwa.
Gwaro uko Kuyera Mitemo yeNeural Networks kunobatsira uye uko nzira dzakareruka dziri nani.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Scaling Laws for Neural Networks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Convolutional Neural Networks
Mibvunzo inowanzo bvunzwa
What is Scaling Laws for Neural Networks?
Mitemo yezviyereso maitiro eempirical anoratidza kuti kurasikirwa kweneural network kunowira kufanofungidzira paunokura modhi saizi, saizi yedataset, uye compute. Izvo zvine basa nekuti vanorega vaongorori vachifanotaura kuita vasati vashandisa mamirioni pakudzidzisa hofori modhi.
Palog-log axes, kurasikirwa kwebvunzo kunowanzoita sei sezvo compute inowedzera pasi pemitemo yekuyera?
Kurasikirwa maringe nemakomputa, saizi yemhando, kana data inoumba mutsara wepedyo-wakatwasuka pane log-log plots, siginicha yesimba-mutemo hukama.
Ndezvipi zviyero zvitatu izvo mitemo yekutanga yekuyera ine chekuita nekurasikirwa?
Kaplan et al. akadzidza kurasikirwa semutemo wesimba mukuverenga parameter (N), tokens yekudzidzira (D), uye yakazara compute (C).
Nei mitemo yekuyera yakakosha kuAI labs?
Nekukodzera macurves padiki, zvikwata zvinofanotaura kuita kwemodhi hombe zvisati zvashandisa mari hombe.
Ko izwi risingaperi (risingadzoreki) muchiyero chekurasikirwa kwemutemo formula rinomiririrei?
Kurasika kune chikamu chinodzikisika chinodzikira nechikero pamwe nepasi risingadzoreki rakagadzikwa neiyo data yezvakasikwa zvisina tsarukano.
Sei kukwidziridzwa kuchitsanangurwa se'kuderera'?
Nekuti simba-mutemo exponent idiki, yakaenzana compute compute inowedzera goho zvishoma nezvishoma kuderera kwekurasikirwa zvachose.