Parallelism eksper ngir liggéey MoE
Parallelism expert dafay xaaj xeetu 'expert' yu bari yu feed-forward ci GPU yu wuute suko defee aparey bu nekk tëye benn wàll ci parametre yi.
Résumé
It is the key to serving trillion-parameter MoE models cheaply, since only a few experts run per token.
Plongeur bu xóot
Benn layer Mixture-of-Experts (MoE) dafay wecci benn reso feed-forward bu mag ak yeneen yu gëna ndaw (expert) boole ci routeur buy tànn top-k (dafay faral di nekk 1 wala 2) eksper ci token bu nekk. Parallelism eksper (EP) dafay dugal ay eksper yu wuute ci GPU yu wuute. Ci inference, routeur bi mooy tànn ban expert la token bu nekk soxla, ci ganaw ga jéego jokkoo bu mat sëkk dafay jaxase token yi ci GPU yi yor seen eksper yiñ tànn, doxal FFN, ba noppi jaxase resultaa yi. Loolu dafay may benn model mu am ay parametre yu bari (sparse) fekk muy aktive benn fraction bu ndaw ci token bu nekk (FLOPs yu woyof). Modèle yu melni Mixtral 8x7B, DeepSeek-V3, ak GPT-OSS dañu koy jëfandikoo. Wàll yu dëgër yi ñooy balance charge ci diggante eksper yi ak ñaari hops yu seer yi ci couche bu nekk.
Gis-gis xarala
Mekanisien core bi mooy ñaari collectives yu ñépp bokk ci MoE layer bu nekk: yónnee (yónnee ay jeton seeni kàngam) ak boole (dajale ay génne). Ndax yoon wi dafay aju ci done, limu jeton yiy dóor eksper bu nekk dafay wuute, moo waral desekilibre sargal ak 'stragglers.' Sistem yiy serwiis dañuy yokk fakteer yu kàttan, tampon yu xarañ, ak daaneel token wala padding ngir tëye GEMMs (matrix multiplies) yu wuute, ba noppi ñuy faral di jaxasoo jokkoog ñépp ak xayma yu xarañ ngir nëbb latency.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu Parallelism Expert ngir MoE di liggéey
Xaarandil jëmmal bu gëna dëgër ci wàllu yoon ak jumtukaay: kernel yuñ boole ngir yónnee-jëmmal-jëmmal, GEMM yu boole ay kàngam yu bari, ak NVLink/InfiniBand-xam lépp-ci-ñépp. Pexem DeepSeek yu melni balance bu amul perte ak routage bu yam ci node yi dañuy wàññi dem bi ak dikk bi ci diggante node yi. Disaggregated serving dina dedike 'expert' GPUs tàqaloo ak GPUs attention, ak lim expert yu gëna mag (téemeeri) ak top-k bu gëna baax dina puus MoE ci sparsity bu tar ci di tëye njëgu token bu nekk.
Doxal ci àdduna dëgg
Mixtral 8x7B di liggéey ci 2-4 GPUs ci def 2-4 ci 8 kàngam yi ci aparey bu nekk
DeepSeek-V3 dafay jëfandikoo yoon wu yam ci node ngir tënk ñaata node la kàngam yi ci token bi, dagg diggante node yépp-ci-ñépp
Jëfandikoo vLLM wala SGLang anam eksper-paralel ngir dalal ab xeetu 200B+ bu bariwul ci benn node 8-GPU
boole paralelismu eksper ak paralelismu tensor ci wàllu bàyyi xel ci jëfandikoo EP+TP hybrid
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Expert Parallelism for MoE Serving quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Prefill ak decode buñ xaaj
Laaj yi ñuy faral di laaj
What is Expert Parallelism for MoE Serving?
Parallelism expert dafay xaaj xeetu 'expert' yu bari yu feed-forward ci GPU yu wuute suko defee aparey bu nekk tëye benn wàll ci parametre yi. Mooy caabi ngir joxe xeetu MoE yu am ay trillion ci njëg yu yomb, ndax yenn kàngam yu néew rek ñooy doxal token bu nekk.
Lan la parallelism expert bi di séddale ci GPU yi?
Parallelism expert dafay def ay expert yu wuute (sous-reseau FFN yi ci benn couche MoE) ci GPU yu wuute, kon aparey bu nekk dafay tëye ay expert yu ndaw.
Ban xeetu jokkoo mooy gëna am solo ci paralelismu kàngam yi ci MoE?
Bépp MoE layer dafay soxla all-to-all ngir yónnee ay token seen expert ak beneen all-to-all ngir boole génne yi.
Lan moo waral MoE di tëye xayma ci token bu nekk doonte ay parametre yu bari lañu?
Routeur bi du def ludul top-k expert (1-2) ci token bu nekk, kon FLOPs yi dañuy des yu ndaw doonte model bi amna expert yu bari ci lëmm.
Ban jafe-jafe mooy xew ndax routing mingi aju ci done?
Ginaaw tànneefi routeur a ngi aju ci liñu dugal, yenn expert yi dañuy jot ay token yu bari yeneen yi, moo waral désequilibre charge ak stragglers.
Lan mooy pexem gëna bari luñu koy jëfandikoo ngir matrix expert yi di yokk seen dayo?
Stacks yiy liggéey dañuy tëral kàttanu eksper bu nekk (limu jeton max) ak pad wala drop jetons ginaaw loolu suko defee GEMM eksper bu nekk am jëmm juñu mëna wax.