Ingxube Yochwepheshe
I-Mixture of Experts (i-MoE) iwumklamo oyimodeli ehlukanisa inethiwekhi ibe amanethiwekhi amancane amaningi akhethekile futhi ivule ambalwa kuphela okokufaka ngakunye.
Uhlolojikelele
It lets models hold enormous knowledge while keeping each prediction fast and cheap.
I-Deep Dive
Isiguquli esijwayelekile sisebenzisa konke okokufaka ngezandlalelo eziminyene ezifanayo, ngakho ukwenza imodeli ihlakaniphe ngokuvamile kusho ukwenza konke ukubala kubize kakhulu. Ingxube Yochwepheshe iphula leso sixhumanisi. Ingena esikhundleni sesendlalelo esikhulu sokuphakelayo ngamanethiwekhi amaningi amancane 'ochwepheshe' kanye 'nerutha' encane enquma ukuthi yimuphi ochwepheshe abaphatha ithokheni ngayinye. Ngokuvamile kuphela ochwepheshe abaphezulu abangu-1 noma abangu-2 abadubulayo, ngakho imodeli ingaba namakhulu ezigidigidi zamapharamitha aphelele kodwa ivule ingxenye encane yethokheni ngayinye. Yingakho amamodeli afana ne-Mixtral 8x7B kanye nezakhiwo ezinamahemuhemu ngemuva kwe-GPT-4 zifinyelela ikhwalithi ephezulu ngaphandle kwezindleko ezilinganiselwe eziphezulu. Ukuhwebelana kuyinkimbinkimbi: bonke ochwepheshe kusafanele bangene ememori, futhi i-router ingasebenzisa kabi noma ilayishe ochwepheshe abathile, ngakho ukuqeqeshwa kudinga ukulinganisa ngokucophelela.
I-Technical Insight
Inhliziyo ye-MoE inethiwekhi yokusangweni, isendlalelo esincane esifundiwe esikala uchwepheshe ngamunye ngethokheni engenayo futhi iqondise ithokheni kwabathole amaphuzu aphezulu kakhulu (ngokuvamile k=1 noma 2). Ukumisa umzila ekuthumeleni yonke into kochwepheshe abambalwa abayintandokazi, ukuqeqeshwa kwengeza 'ukulahlekelwa kokulinganisela komthwalo' okuyisizayo okujezisa ukusetshenziswa okungalingani. Ngenxa yokuthi ochwepheshe abangu-k kuphela abasebenzisa ithokheni ngayinye, ikhomputha (i-FLOPs) ihlala ifana ngokufanayo njengoba wengeza ochwepheshe abengeziwe, ngakho-ke amapharamitha aphelele kanye nesilinganiso sezindleko zethokheni ngayinye ngokuzimela.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Lengxube Yezazi
I-MoE isiba yithuluzi elizenzakalelayo lamamodeli esikali somngcele ngoba yehlisa umthamo ezindlekweni. Lindela ochwepheshe abacokeme, umzila ohlakaniphile ocabangela umongo owengeziwe, kanye namasu angcono okunikeza amamodeli amakhulu agqagqene kuhadiwe elinganiselwe. Ucwaningo luphinde lubhekane nenkinga yenkumbulo, njengoba bonke ochwepheshe kufanele balayishwe nakuba bembalwa abagijimayo, ngokukhipha uchwepheshe kanye nokulinganisa. Njengamamodeli avulekile afana ne-Mixtral ne-DeepSeek-MoE evuthiwe, izakhiwo ezincane zizonika amandla abasizi abaphumelelayo kumabhajethi amancane we-GPU.
Ukuqaliswa Komhlaba Wangempela
I-Mixtral 8x7B isebenzisa ochwepheshe abayisi-8 futhi yenza kusebenze okungu-2 ngethokheni ngayinye, inikeze cishe amapharamitha aphelele angu-47B kodwa kuphela ~13B esebenzayo ithokheni ngayinye ukuze uthole ukucatshangelwa okusheshayo, okushibhile.
I-DeepSeek ne-Qwen ithumela amamodeli amakhulu olimi lwe-MoE afana namamodeli aminyene kumabhentshimakhi kuyilapho isebenza ngekhompuyutha yethokheni ephansi.
Abahlinzeki be-Cloud LLM basebenzisa i-MoE ukuze imodeli eyodwa enkulu ikwazi ukusiza abasebenzisi abaningi ngokuthengekayo, njengoba isicelo ngasinye sikhanyisa ochwepheshe abambalwa.
Google I-Switch Transformer yangaphambilini ifinyelele kumapharamitha angaphezu kwesigidintathu isebenzisa umzila ophezulu ongu-1 ukuze kugcinwe ikhompuyutha yokuqeqesha ilawuleka.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture of Experts quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Ingxube Yochwepheshe be-LoRA
Imibuzo evame ukubuzwa
What is Mixture of Experts?
I-Mixture of Experts (i-MoE) iwumklamo oyimodeli ehlukanisa inethiwekhi ibe amanethiwekhi amancane amaningi akhethekile futhi ivule ambalwa kuphela okokufaka ngakunye. Ivumela amamodeli ukuthi abambe ulwazi olukhulu ngenkathi egcina ukuqagela ngakunye kushesha futhi kushibhile.
Kusendlalelo Sengxube Yochwepheshe, yini enqumayo ukuthi yibaphi ochwepheshe abacubungula ithokheni enikeziwe?
Inethiwekhi encane yokugenca ifunda ukuthola amaphuzu uchwepheshe ngamunye wethokheni futhi iyihambise kwabanamaphuzu aphezulu. Umzila uyafundwa, awunqunyelwe noma awuhleliwe.
Kungani imodeli ye-MoE ingaba namapharamitha esewonke amaningi kakhulu kunemodeli eminyene ngaphandle kokukhuphuka okufanayo kwezindleko ngethokheni ngayinye?
Ngoba ochwepheshe be-top-k kuphela badubula ithokheni ngayinye, ikhompuyutha esebenzayo ihlala cishe ingashintshi njengoba isibalo sepharamitha sikhula ngokungeza ochwepheshe abengeziwe.
Iyiphi inkinga ikheli lokulahlekelwa komthwalo (okusizayo) ngesikhathi sokuqeqeshwa kwe-MoE?
Ngaphandle kokulinganisa, i-router ivame ukuthanda idlanzana lochwepheshe. Ukulahlekelwa okuyisizayo kujezisa ukusetshenziswa okungalingani ukuze bonke ochwepheshe baqeqeshwe.
I-Mixtral 8x7B yenza kusebenze ochwepheshe bayo abangu-2 kwabangu-8 ithokheni ngayinye. Kusho ukuthini lokhu ngekhompyutha yayo uma iqhathaniswa nobukhulu bayo?
Ngochwepheshe abangu-2 kwabangu-8 kuphela abasebenzayo, amapharamitha asebenzayo ngethokheni ngayinye (~13B) mancane kakhulu kunengqikithi engu-~47B, ehlisa izindleko zokucabanga.
Iluphi uhlangothi lwangempela lwamamodeli we-MoE uma kuqhathaniswa namamodeli aminyene anekhono ngokulinganayo?
Noma ochwepheshe abambalwa kuphela bebala ithokheni ngayinye, izisindo zochwepheshe ngamunye kufanele zilayishwe ngenkumbulo, okwenza amamodeli e-MoE alambele inkumbulo ukuwapha.