Ikibazo gihuza ibibazo · Biragoye Urwego
Uruvange rwinzobere Ikibazo
Witoze gake ya MoE inzira, impuguke zipakurura impuguke, hamwe no kugurisha ibicuruzwa, harimo itandukaniro riri hagati yimirimo ikora kuri buri kimenyetso hamwe nibintu byose byabitswe.
Ikibazo 1 Bya 8
Muruvange rwinzobere, niki cyemeza abahanga batunganya ikimenyetso runaka?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Muruvange rwinzobere, niki cyemeza abahanga batunganya ikimenyetso runaka?
- Gitoya yize ya router (umuyoboro wa gating) utanga amanota
- Imbonerahamwe ihamye yo gushakisha yashyizweho mbere yimyitozo
- Uburebure bwumukoresha
- Igishushanyo kidasanzwe kuri buri ntambwe
- Ni ukubera iki moderi ya MoE ishobora kugira ibipimo byinshi birenze urugero byuzuye bitarinze kwiyongera kubiciro kuri buri kimenyetso?
- Ihagarika ibipimo muburemere bumwe busangiwe
- Isimbuka ibyerekezo byuzuye
- Gusa abahanga bake (top-k) bakora kuri buri kimenyetso
- Ikoresha umuhanga ariko ugereranije ibisubizo
- Ni ikihe kibazo gikemura ibibazo (auxiliary) igihombo mugihe cy'amahugurwa ya MoE?
- Tokens kuba ndende cyane kumadirishya yimiterere
- Router ikoresha abahanga bake kandi inzara isigaye
- Gradients izimira mumutwe
- Kurenza urugero rwo kwemeza
- Mixtral 8x7B ikora 2 muri 8 zinzobere zayo kuri buri kimenyetso. Ibi bivuze iki kuri compte yayo nubunini bwayo?
- Ikoresha compte zose zinzobere 8 kuri buri kimenyetso
- Ifite ibipimo bike ugereranije na 7B yuzuye
- Ntishobora gukora kuri GPU rwose
- Igikorwa gikora kuri buri kimenyetso ni gito cyane kuruta ibipimo rusange byerekana
- Why can a sparse MoE still have substantial memory or storage requirements?
- Every token necessarily activates all experts
- Inactive experts cease to have parameters
- Inactive expert weights still need accessible storage even when not selected for a token
- The router removes all attention layers
- Two selected MoE expert outputs are 2 and 6, with combination weights 0.25 and 0.75. What is their weighted output?
- 5
- 8
- 2
- 0.5
- Increasing an MoE’s expert count keeps the same top-k and expert size. What can still increase serving cost?
- The selected expert count must automatically become the total count
- The model must stop using attention
- Every inactive weight is automatically deleted
- Router or communication overhead and the storage required for more experts
- A sparse MoE calls a learned feed-forward subnetwork an expert. What does that name establish?
- It is certified to answer only one academic discipline
- It is one of the networks selected by routing, without proving a named subject specialty
- It must be a human-written collection of rules
- It contains no learned parameters