Uruvange rwinzobere
Uruvange rwinzobere (MoE) nigishushanyo mbonera kigabanya umuyoboro mubice byinshi byihariye-bikora kandi bigakora bike gusa mubitekerezo.
Incamake
It lets models hold enormous knowledge while keeping each prediction fast and cheap.
Kwibira cyane
Impinduka isanzwe ikora ibyinjijwe binyuze murwego rumwe, bityo gukora moderi ubwenge mubisanzwe bisobanura gukora buri kubara bihenze. Uruvange rwinzobere ruhagarika iyo link. Irasimbuza ibiryo binini-bigana imbere hamwe nudusanduku twinshi 'impuguke' hiyongereyeho 'router' ntoya ihitamo abahanga bakora buri kimenyetso. Mubisanzwe abahanga 1 cyangwa 2 bambere bonyine barashya, kuburyo moderi ishobora kugira miliyari amagana yibipimo byose ariko igakora agace gato kuri buri kimenyetso. Niyo mpamvu moderi nka Mixtral 8x7B hamwe nubwubatsi buvugwa inyuma ya GPT-4 bugera ku rwego rwo hejuru nta giciro cyo hejuru cyagereranijwe. Gucuruza-bigoye biragoye: abahanga bose bagomba gukomeza guhuza nibuka, kandi router irashobora kuyobora nabi cyangwa kurenza urugero abahanga bamwe, bityo amahugurwa akenera gushyira mu gaciro.
Ubushishozi
Umutima wa MoE numuyoboro winjira, urwego ruto rwize rutanga amanota buri mpuguke kubimenyetso byinjira kandi ikayobora icyerekezo kuri top-k amanota menshi (akenshi k = 1 cyangwa 2). Kugirango uhagarike router yohereza ibintu byose kubahanga bake bakunda, amahugurwa yongeramo 'igihombo-kiringaniye igihombo' gihana imikoreshereze idahwitse. Kuberako k impuguke zonyine zikoresha kuri buri kimenyetso, kubara (FLOPs) bigumaho hafi nubwo wongeyeho abahanga benshi, bityo ibipimo byose hamwe nigipimo cyibiciro byigenga.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Igihe kizaza cyo kuvanga abahanga
MoE ihinduka igikoresho kidasanzwe kumipaka-yerekana urugero kuko ikuramo ubushobozi kubiciro. Witege impuguke zifite ubuhanga, inzira zubwenge zitekereza cyane, hamwe nubuhanga bwiza bwo gutanga moderi nini zidasanzwe kubikoresho bike. Ubushakashatsi nabwo bukemura ikibazo cyo kwibuka, kubera ko abahanga bose bagomba kuremererwa nubwo ari bake biruka, binyuze mubuhanga bwo gupakurura no kubara. Nka moderi ifunguye nka Mixtral na DeepSeek-MoE ikuze, imyubakire idahwitse irashobora guha imbaraga abafasha bakora neza kuri bije nto ya GPU.
Gushyira mu bikorwa Isi
Mixtral 8x7B ikoresha abahanga 8 kandi ikora 2 kuri token, itanga ibipimo hafi 47B byose ariko ~ 13B gusa ikora kuri token kugirango yihute, ihendutse.
DeepSeek na Qwen bohereza imiterere nini yururimi rwa MoE ihuza imiterere yuzuye kubipimo mugihe ikora hamwe na compte yo hasi kuri token.
Abatanga Cloud LLM bakoresha MoE kuburyo moderi imwe nini ishobora guha abakoresha benshi ku buryo buhendutse, kubera ko buri cyifuzo kimurikira abahanga bake.
__AIU_PROTECTED_11 __ 'mbere yo Guhindura Transformer yapimye kugera kuri tiriyari zirenga miriyoni ukoresheje top-1 inzira kugirango ikomeze kubara compte.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture of Experts quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Uruvange rwinzobere za LoRA
Ibibazo bikunze kubazwa
What is Mixture of Experts?
Uruvange rwinzobere (MoE) nigishushanyo mbonera kigabanya umuyoboro mubice byinshi byihariye-bikora kandi bigakora bike gusa mubitekerezo. Ireka abanyamideli bafite ubumenyi bukomeye mugihe buri guhanura byihuse kandi bihendutse.
Muruvange rwinzobere, niki cyemeza abahanga batunganya ikimenyetso runaka?
Umuyoboro muto winjira wiga gutanga amanota buri mpuguke kuri token kandi akayerekeza kuri top-amanota menshi. Inzira irigwa, ntabwo ikosowe cyangwa idasanzwe.
Ni ukubera iki moderi ya MoE ishobora kugira ibipimo byinshi birenze urugero byuzuye bitarinze kwiyongera kubiciro kuri buri kimenyetso?
Kuberako impuguke zo hejuru-k zirasa kuri buri kimenyetso, compte ikora igumaho hafi nubwo ibipimo byose bibara byiyongera mukongeramo abahanga benshi.
Ni ikihe kibazo gikemura ibibazo (auxiliary) igihombo mugihe cy'amahugurwa ya MoE?
Hatabayeho kuringaniza, router ikunda gutonesha inzobere. Igihombo cyabafasha gihana imikoreshereze idahwitse kuburyo abahanga bose bahugurwa.
Mixtral 8x7B ikora 2 muri 8 zinzobere zayo kuri buri kimenyetso. Ibi bivuze iki kuri compte yayo nubunini bwayo?
Hamwe ninzobere 2 kuri 8 gusa zikora, ibipimo bifatika kuri token (~ 13B) ni bito cyane kurenza ~ 47B byose, bigabanya igiciro cyo gufata umwanzuro.
Nibihe bibi byukuri bya moderi ya MoE ugereranije na moderi yuzuye yuzuye?
Nubwo impuguke nke zibara kuri buri kimenyetso, uburemere bwinzobere bugomba kuba bwuzuye mububiko, bigatuma moderi ya MoE yibuka-ishonje kugirango ikore.