njaxasu kàngam
Njaxasu Ekspeer yi (MoE) xeetu jëmmal la buy xaaj reso bi ci ay sous-reseau yu yam, ba noppi yenn ci dugal yi kese lañuy aktive.
Résumé
It lets models hold enormous knowledge while keeping each prediction fast and cheap.
Plongeur bu xóot
Transformatër buñ miin dafay doxal bepp dugg ci benn couche dense, kon def model bi gëna xarañ dafay tekki ni bepp calcul dafay gëna seer. Njaxasu Ekspeer yi dafay dog lëkkaloo bi. Dafay wecci feed-forward layer bu mag bi ak reso 'expert' yu ndaw yu bari boole ci 'routeur' bu ndaw buy xool ban expert mooy jëfandikoo token bu nekk. Daanaka 1 wala 2 kàngam yi gëna mag ñooy génne, kon benn model mën na am téemeeri milyaari parametre ci lëmm waaye du aktive benn wàll bu ndaw ci token bu nekk. Lii moo tax model yu melni Mixtral 8x7B ak architecture bi ñuy wax ci ginaaw GPT-4 yegg ci kalite bu kawe te du am njëgu inference bu yéeg. Kompromis bi mooy jafe-jafe bi: bépp expert dafa wara nekk ci memory bi, te routeur bi mën na juum ci yoon wi wala mu yab yenn expert yi, kon tàggat yaram dafay laaj equilibre bu baax.
Gis-gis xarala
Xolu MoE mooy reso gating bi, ab layer bu ndaw buñu jàng buy jox poñ expert bu nekk ci benn token buy duggu ba noppi yóbbu token bi ci top-k bi gëna am poñ (dafay faral di nekk k = 1 wala 2). Ngir tere routeur bi yónnee lépp yenn expert yuñ taamu, tàggat yaram dafay yokk 'load-balancing loss' buy daanel jëfandikoo bu jaarul yoon. Ndax k expert kese ñooy daw ci token bu nekk, calcul (FLOPs) dafay des ci anam wu soppeeku doonte dangay yokk yeneen expert, kon parametre yu mat yi ak njëgu token bu nekk dañuy wuute.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu njaxasu kàngam yi
MoE mingi nekk jumtukaay biñ jagleel xeetu frontiere yi ndax dafay dindi kàttan ak njëg. Xaarandil ay kàngam yu xarañ, yoon wu gëna am xel, ak pexe yu gëna baax ngir liggéey ci model yu bari te wuute ci hardware bu néew. Gëstu itam dafay xeex jafe-jafe mémoire, ndax expert yépp dañu wara charger doonte ñu néew lañuy daw, jaaraleko ci expert offloading ak quantization. Ginaaw model yu ubbeeku yu melni Mixtral ak DeepSeek-MoE dañuy màgg, architecture yu bari yi dina ñu am doole ci assistant yu gëna am doole ci budget GPU yu gëna ndaw.
Doxal ci àdduna dëgg
Mixtral 8x7B dafay jëfandikoo 8 kàngam ba noppi di aktive 2 ci token bu nekk, di joxe lu tollu ci 47B ci parametre yi waaye ~ 13B kese lañuy aktive ci token bu nekk ngir gëna gaaw, gëna xéewale.
DeepSeek ak Qwen dañuy yónnee xeetu làkk MoE yu mag yu méngoo ak xeetu làkk yu dëgër yi ci benchmark yi, fekk ñuy daw ak xayma bu gëna ndaw ci token bu nekk.
Cloud LLM fournisër yi jëfandikoo MoE suko defee benn model bu rëy mëna liggéey jëfandikukat yu bari ci njëg yu yomb, ndax laaj bu nekk dafay leeral yenn kàngam.
Google Transformatëru kommutatër bi njëkka am dafa yokk ba ëpp benn bilioŋ ciy parametre di jëfandikoo yoon wi gëna kawe ngir mëna wéy di tàggat ordinatër bi.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture of Experts quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
njaxasu kàngam yi ci LoRA
Laaj yi ñuy faral di laaj
What is Mixture of Experts?
Njaxasu Ekspeer yi (MoE) xeetu jëmmal la buy xaaj reso bi ci ay sous-reseau yu yam, ba noppi yenn ci dugal yi kese lañuy aktive. Dafay may model yi ñu mëna tëye xam-xam bu bari te ba noppi di tëye bépp wax luy waaja am lu gaaw te yomb.
Ci biir njaxasu Expert yi, lan mooy dogal ban expert mooy doxal benn token buñ jox?
Benn reso gating bu ndaw dafay jàng poñ expert bu nekk ci token bi ba noppi yóbbu ko ci ñi gëna am poñ. Routing dañu koy jàng, du fixe wala random.
Lan moo waral benn xeetu MoE mëna am ay parametre yu ëpp benn xeetu dense te amul yokkute bu méngoo ci njëgu token bu nekk?
Ndax top-k experts yi kese ñooy génne token bu nekk, calcul biy dox dafay des ci anam wu soppeeku doonte limu parametre yi dafay gëna yokk ndax dañuy yokk yeneen experts.
Lan mooy jafe-jafe bi charge-balancing (auxiliaire) di saafara ci diiru tàggat-yaram MoE?
Ku amul ekilibre, routeur bi dafay bëgg ñu bari ñu xarañ. Perte auxiliaire bi dafay yar jëfandikoo bu jaarul yoon, moo tax eksper yépp ñu tàggat leen.
Mixtral 8x7B dafay aktive 2 ci 8 kàngam yi ci jeton bu nekk. Lu loolu di tekki ci xaymaam ak dayoom?
Ak 2 ci 8 kàngam yu am doole, parametre yi am ci token bu nekk (~ 13B) ñoo gëna ndaw ~ 47B ci lëmm, wàññi njëgu inference.
Lan mooy gëna baaxul ci xeetu MoE yi sunu ko méngale ak xeetu dense yu mëna tolloo?
Doonte yenn kàngam yu néew ñooy xayma token bu nekk, diisaayu kàngam bu nekk dafa wara sarse ci memër bi, moo tax modelu MoE yi dañu xiif ci memër ngir liggéey.