Ngwakọta nke Omimi
Ngwakọta nke Omimi (MoD) na-enye ohere ka onye na-atụgharị ihe na-emefu ego dị iche iche na akara dị iche iche, na-ebugharị naanị akara 'dị mkpa' site na mgbakọ dị arọ nke oyi akwa ọ bụla.
Nchịkọta
It cuts the cost of processing easy tokens while keeping a fixed, predictable compute budget.
Ime miri emi
Ihe ngbanwe ọkọlọtọ na-etinye akwa oyi akwa ọ bụla na akara ngosi ọ bụla, ọbụlagodi ndị na-adịchaghị mkpa dị ka akara edemede. Ngwakọta nke omimi, nke Google DeepMind webatara na 2024, na-agbakwunye obere rawụta na ngọngọ nke ọ bụla na-ahọrọ obere oke-k nke akara akara iji nweta nlebara anya onwe ya zuru oke na mkpokọta MLP; ndị fọdụrụ na-awụpụ ngọngọ site na njikọ fọdụrụnụ. N'ihi na ọ bụ naanị k token ka a na-ahazi n'otu oyi akwa, a na-ekpuchi mkpokọta mkpokọta (FLOPs) ma mara ya tupu oge eruo, n'adịghị ka ụzọ omimi siri ike na mbụ nke dịgasị iche na-enweghị atụ. Nke a na-eme batching na ngwaike itinye n'ọrụ nke ọma. Ụdị a zụrụ nke MoD nwere ike dakọbara n'ịdịmma transformer baseline site na iji FLOP ole na ole n'ihu gafere, ma ọ bụ rute ogo dị elu n'otu mgbakọ ahụ, echiche a na-ejikọta ya na Mixture-of-Experts iji nye 'MoDE' ụdị nke na-aga na omimi na obosara.
Nghọta nka nka
N'ime ngọngọ MoD ọ bụla, onye na-ahụ maka ihe ntanetị amụtara na-enyocha akara ọ bụla ma na-edobe top-k site na akara; akara ngosi ahọpụtara na-agafe nlebara anya na MLP, ebe akara ndị a na-ahọrọghị na-aga n'ihu na-agbanweghị site na ụzọ fọdụrụnụ. Iji top-k edoziri (kama akara nrịbama nke ọ bụla) na-eme ka eserese kọmpụta static na tensor na-akpụzi mgbe niile, nke bụ enyi na enyi. A zụrụ rawụta ya na netwọkụ ndị ọzọ, na ọgbọ na-akpata na-eji amụma inyeaka ka mkpebi ntụgharị ghara ileba anya na akara ngosi n'ọdịnihu.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke ngwakọta nke omimi
Mgbakọ ọnọdụ bụ isi ihe eji arụ ọrụ dị ka ọnụ ọgụgụ ụdị, yana MoD bụ ihe atụ mbụ, dị ọcha. Na-atụ anya mwekota miri emi na Mixture-of-Experts (routing na ma omimi na ndị ọkachamara), mmefu ego na-agbanwe agbanwe nke na-adaba maka ntinye dị mfe, na ndị na-anya ụgbọ njem mmụta nke na-achọpụta nke ọma nke tokens chọrọ nhazi miri emi. Dị ka ọnụ ahịa ntinye aka na-achịkwa akụ na ụba mbugharị, usoro ndị na-ahapụ ụdị 'iche echiche siri ike' naanị ebe ọ dị mkpa, ebe ha na-edobe latency a na-atụghị anya ya, nwere ike bụrụ ọkọlọtọ na nnukwu ụlọ ọrụ.
Mmejuputa n'ezie n'ụwa
Mbelata FLOP ndị achọrọ iji hazie akwụkwọ dị ogologo site n'ịwụpụ ngụkọ miri emi na token filler
Ọzụzụ ihe nlereanya dabara na ịdịmma ntọala na mgbakọ dị ala, na-eweda ọnụ ahịa ozi
Na-ejikọta ya na Mixture-of-Experts (MoDE) na-aga n'ihu na omimi oyi akwa yana nhọrọ ọkachamara
Idobe amụma, nke edobere anya n'otu akara n'ihi na a na-edozi mmefu ego kwa-layer n'ihu
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture of Depths quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Ngwakọta-nke ndị nnọchi anya
Ajụjụ a na-ajụkarị
What is Mixture of Depths?
Ngwakọta nke Omimi (MoD) na-enye ohere ka onye na-atụgharị ihe na-emefu ego dị iche iche na akara dị iche iche, na-ebugharị naanị akara 'dị mkpa' site na mgbakọ dị arọ nke oyi akwa ọ bụla. Ọ na-ebelata ọnụ ahịa nke nhazi akara ngosi dị mfe ka ị na-edobe mmefu ego gbakọọ, nke a na-atụghị anya ya.
Kedu ihe ngwakọta nke omimi dị iche n'ofe akara?
MoD na-aga naanị ụfọdụ akara site na ngụkọ dị arọ nke oyi akwa ọ bụla, yabụ akara ngosi dị iche iche na-enweta omimi dị iche iche nke ọma.
Kedu otu MoD ga-esi edobe ngụkọ mmefu ego ya?
Ịhọrọ top-k nke tokens nke ọ bụla n'otu ngọngọ na-adaba ma na-eme ka ọkpụkpụ tensor na-adị mgbe niile, nke bụ enyi na enyi.
Kedu ihe na-eme akara ngosi na rawụta anaghị ahọrọ na ngọngọ enyere?
Ihe nrịbama ndị ahọpụtaraghị ga-agafe mgbako ngọngọ ma na-aga n'ihu na-agbanweghị site na ụzọ fọdụrụnụ.
Which lab published the Mixture of Depths technique?
Ọ bụ Google DeepMind webatara ngwakọta nke omimi n'ime akwụkwọ 2024 na mgbako mgbanwe mgbanwe.
Kedu ihe ijikọ MoD na Mixture-of-Experts na-emepụta?
Ijikọta ụzọ omimi na ụzọ ndị ọkachamara na-arụpụta 'MoDE' ụdị nke na-agbakọ na ọkwa abụọ na ndị ọkachamara.