I-Chinchilla Compute-Optimal Training
I-Chinchilla iwuhlelo lwango-2022 lwe-DeepMind oluthola ukuthi amamodeli amaningi ezilimi ayengafundelwanga kahle kakhulu: kubhajethi yekhompiyutha engaguquki kufanele ukale amapharamitha nedatha cishe ngokulinganayo, ungagcini nje ukwakha imodeli enkulu.
Uhlolojikelele
It reshaped how the industry balances model size against training data.
I-Deep Dive
Iphepha le-DeepMind's Chinchilla liphinde lavakashela ukukala futhi laqeqeshelwa amamodeli angaphezu kuka-400 ukuze lithole ibhalansi yekhompyutha elungile. Umthetho wesihloko sesithupha: usayizi wemodeli namathokheni okuqeqesha kufanele akhule ku-lockstep, cishe amathokheni okuqeqesha angama-20 ngepharamitha ngayinye. Ukufakazela lokho, baqeqeshe i-Chinchilla, imodeli ye-70-billion-parameter kumathokheni we-1.4 trillion, besebenzisa ikhompuyutha efanayo ne-280-billion-parameter Gopher eqeqeshwe ngamathokheni ambalwa kakhulu. I-Chinchilla, naphezu kokuba yincane ngokuphindwe kane, i-Gopher, i-GPT-3, namanye ama-giants yaphumelela cishe kuwo wonke ama-benchmark. Isifundo siguqule isiphetho sangaphambili OpenAI sokuthi usayizi othandwayo kunedatha, okubonisa amamodeli amaningi aphambili ayeshiya ukusebenza etafuleni ngokuba makhulu kakhulu futhi abulawa yidatha kakhulu.
I-Technical Insight
I-Chinchilla fit loss as L(N,D) = E + A·N^(-α) + B·D^(-β), no-α kanye no-β kokubili eduze kuka-0.34, okusho ukuthi amapharamitha nedatha kunikela cishe ngokulinganayo. Ukuthuthukisa lokhu ngaphansi kwesithiyo sekhompiyutha esingaguquki (bale ≈ 6·N·D kuma-transformer) kunikeza umphumela wokulinganisa okulinganayo. Imodeli encane, enothile ngedatha nayo ishibhile ukuyisebenzisa ngokuqonda, ngakho-ke inzuzo yayo ihlanganisa ukuthunyelwa, hhayi nje ukuqeqeshwa.
I-Strategic Impact
Izinqumo ezicacile
Kukusiza ukuthi uhlukanise izimangalo ezicacile zobuchwepheshe kusukela olimini lokumaketha.
Izindleko kanye nesabelomali
Ungabuza imibuzo yokusebenzisa kangcono ngaphambi kokusebenzisa imali noma isikhathi.
Ithimba kanye nokusebenza komsebenzi
Amaqembu anokuqonda okwabiwe enza izinqumo ezingcono zomkhiqizo, inqubomgomo, nokufunda.
Ikusasa Le-Chinchilla Compute-Optimal Training
Amamodeli esimanje afana ne-Llama 3 aphusha ngamabomu isilinganiso samathokheni angama-20 ngepharamitha ye-Chinchilla, eqeqesha amamodeli amancane kumathriliyoni amathokheni ukuze enze inkambiso eshibhile, amukele ikhompuyutha yokuqeqeshwa ephansi. Njengoba idatha enhle iyancipha, intshisekelo iyakhula kuma-epoch aphindaphindiwe, idatha yokwenziwa, nokuhlunga kwekhwalithi. I-Chinchilla isalokhu iyiphoyinti eliyinkomba, kodwa okuhle kuya ngokuya ngokuya kuncika ezindlekweni zokuphila konke, hhayi nje isabelomali sokuqeqeshwa sesikhathi esisodwa.
Ukuqaliswa Komhlaba Wangempela
Ukukhetha ukuqeqesha imodeli yepharamitha eyizigidi eziyizinkulungwane ezingu-7 kumathokheni angu-2 trillion kunokuba imodeli yezigidigidi ezingu-30 kudatha encane kakhulu yesabelomali esifanayo.
Ilinganisela ukuthi imodeli yepharamitha yebhiliyoni eyi-10 ifuna amathokheni acishe abe yizigidi eziyizinkulungwane ezingama-200 ukuze afinyelele endaweni emnandi yekhompiyutha.
Ukuthethelela imodeli encane esetshenzisiwe ukuze kuncishiswe izindleko ze-inference yombuzo ngamunye kuyilapho kuqhathaniswa nekhwalithi yembangi enkulu.
Ukuhlola imodeli ekhona nokuyiphetha akuqeqeshelwe phansi, bese kuhlelwa ukuqeqeshwa okude esikhundleni sokukhushulwa kwepharamitha.
Izingozi & Guardrails
Amaqembu ahlukene angasebenzisa igama elifanayo ngokuhlukile, ngakho chaza ububanzi kusenesikhathi.
Amabhentshimakhi angabukeka eqinile kuyilapho ukusebenza komhlaba wangempela kungalingani.
Ukuziba ikhwalithi yedatha nezinhlelo zokuhlaziya kuvame ukudala imiphumela entekenteke.
Ukuqalisa Umhlahlandlela
Qala ngencazelo yolimi olulula yomphumela oyidingayo.
Khetha imethrikhi eyodwa yempumelelo nesimo esisodwa sokuhluleka ngaphambi kokuhlolwa.
Qalisa umshayeli omncane onedatha emele, hhayi isethi yedemo ephucuziwe.
Bhala lapho i-Chinchilla Compute-Optimal Training isiza khona nalapho izindlela ezilula zingcono.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Chinchilla Compute-Optimal Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Ukuqeqeshwa Kwesikhathi Sokuhlola
Imibuzo evame ukubuzwa
What is Chinchilla Compute-Optimal Training?
I-Chinchilla iwuhlelo lwango-2022 lwe-DeepMind oluthola ukuthi amamodeli amaningi ezilimi ayengafundelwanga kahle kakhulu: kubhajethi yekhompiyutha engaguquki kufanele ukale amapharamitha nedatha cishe ngokulinganayo, ungagcini nje ukwakha imodeli enkulu. Ilungise kabusha ukuthi imboni ibhalansisa kanjani usayizi wemodeli ngokumelene nedatha yokuqeqeshwa.
Imuphi umthetho wesithupha odumile we-Chinchilla wokubala ukuqeqeshwa okungcono kakhulu?
I-DeepMind ithole amapharamitha namathokheni kufanele alinganisele ndawonye cishe kumathokheni angama-20 ngepharamitha ngayinye yesabelomali sekhompiyutha esinikeziwe.
I-70B-parameter Chinchilla iqhathanise kanjani ne-280B-parameter Gopher?
Uqeqeshwe ngamathokheni amaningi kakhulu ngekhompuyutha efanayo, u-Chinchilla wehlula i-Gopher enkulu kakhulu kuwo wonke amabhentshimakhi amaningi.
Iyiphi inkinga esemqoka iphepha leChinchilla eladalula mayelana namamodeli amakhulu angaphambili?
Amamodeli afana ne-GPT-3 ne-Gopher ayemakhulu kakhulu uma kuqhathaniswa nedatha yawo yokuqeqeshwa, eshiya ukusebenza kungenzeki.
Cishe mangaki amathokheni umthetho we-Chinchilla ophakamisa imodeli yepharamitha yezigidigidi eziyi-10?
Kumathokheni angama-20 ngepharamitha ngayinye, amapharamitha ayizigidi eziyizinkulungwane eziyi-10 asho amathokheni okuqeqesha ayizigidi eziyizinkulungwane ezingama-200.
Ngaphandle kokuqeqeshwa kahle, kungani imodeli encane, enothile ngedatha iheha ekusetshenzisweni?
Amapharamitha ambalwa asho ukubala okuphansi kombuzo ngamunye, ngakho-ke inhlanganisela yokonga njalo uma imodeli isetshenziswa.