Idanwo ti o sopọ mọ itọsọna · Lile Ipele

Adalu ti Amoye adanwo

Practice sparse MoE routing, expert load balance, and memory tradeoffs, including the difference between active per-token work and total stored parameters.

Jẹmọ itọsọna awọn ọnaAdalu Of Amoye
Ibeere 1 ti 8

Ni a Adalu ti Amoye Layer, ohun ti pinnu eyi ti amoye ilana a fi fun àmi?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Ni a Adalu ti Amoye Layer, ohun ti pinnu eyi ti amoye ilana a fi fun àmi?
    • A kekere kẹkọọ olulana (gating nẹtiwọki) ti o ikun amoye
    • Tabili wiwa ti o wa titi ti a ṣeto ṣaaju ikẹkọ
    • Awọn olumulo ká ipari ipari
    • Iyaworan laileto ni gbogbo igbesẹ
  2. Kini idi ti awoṣe MoE le ni awọn aye lapapọ lapapọ ju awoṣe ipon laisi ilosoke ibaramu ni idiyele fun ami kan?
    • O compresses paramita sinu kan nikan pín àdánù
    • O fo awọn ipele akiyesi patapata
    • Nikan kan diẹ amoye (top-k) ti wa ni mu šišẹ fun kọọkan àmi
    • O gbalaye gbogbo amoye ṣugbọn awọn aropin awọn esi
  3. Isoro wo ni adiresi pipadanu fifuye-iwọntunwọnsi (oluranlọwọ) lakoko ikẹkọ MoE?
    • Awọn ami ti o gun ju fun ferese ọrọ ọrọ
    • Awọn olulana overusing kan diẹ amoye ati ki o nki awọn iyokù
    • Gradients ti sọnu ni awọn ori akiyesi
    • Overfitting to afọwọsi ṣeto
  4. Mixtral 8x7B mu 2 ti awọn amoye 8 ṣiṣẹ fun ami kan. Kini eyi tumọ si nipa iṣiro rẹ dipo iwọn rẹ?
    • O nlo gbogbo iṣiro awọn amoye 8 lori gbogbo aami
    • O ni awọn paramita lapapọ ti o dinku ju awoṣe ipon 7B kan
    • Ko le ṣiṣẹ lori GPU kan rara
    • Iṣiro ti nṣiṣe lọwọ fun ami kan kere pupọ ju kika paramita lapapọ rẹ daba
  5. Why can a sparse MoE still have substantial memory or storage requirements?
    • Every token necessarily activates all experts
    • Inactive experts cease to have parameters
    • Inactive expert weights still need accessible storage even when not selected for a token
    • The router removes all attention layers
  6. Two selected MoE expert outputs are 2 and 6, with combination weights 0.25 and 0.75. What is their weighted output?
    • 5
    • 8
    • 2
    • 0.5
  7. Increasing an MoE’s expert count keeps the same top-k and expert size. What can still increase serving cost?
    • The selected expert count must automatically become the total count
    • The model must stop using attention
    • Every inactive weight is automatically deleted
    • Router or communication overhead and the storage required for more experts
  8. A sparse MoE calls a learned feed-forward subnetwork an expert. What does that name establish?
    • It is certified to answer only one academic discipline
    • It is one of the networks selected by routing, without proving a named subject specialty
    • It must be a human-written collection of rules
    • It contains no learned parameters