Kedis ku xidhan hagaha · Adag Heerka

SwiGLU iyo Kediska Dhaqdhaqaaqa Gated

Wuxuu tijaabiyaa saadaasha albaabbada loo laabay, albaabka SiLU, isweydaarsiga-qiimaha-qiimaynta iyo sababta ballaca qarsooni isu beddelo marka SwiGLU uu beddelo MLP-ga caadiga ah.

Wadooyinka haga ee la xidhiidhaSwiglu iyo Dhaqdhaqaaqyada Gateed
Su'aal 1 ee 8

Hawlgalkee xudunta u ah ayaa qeexaya Unugga toosan ee Gated?

Horudhac su'aal kasta8 su'aalo

Jawaabaha iyo sharraxaadaha waxay soo baxayaan markaad qaadanayso kediska.

  1. Hawlgalkee xudunta u ah ayaa qeexaya Unugga toosan ee Gated?
    • Isku dhufashada curiyaha-xikmadaysan ee hal odoroska iyadoo la adeegsanayo odoroska labaad ee firfircoon
    • Isku-darka laba saadaaliye oo toosan
    • Madax dareenka isku dhafan
    • U qaybinta firfircoonida habkooda caadiga ah
  2. Shaqadee hawl qabad ayay SwiGLU u isticmaashaa laanteeda albaabka?
    • ReLU
    • Iswiish (SiLU)
    • Tanh
    • Softmax
  3. Immisa xabbo oo miisaan ah ayaa SwiGLU quudin-horumarintu isticmaashaa?
    • Mid
    • Laba
    • Saddex
    • Afar
  4. Waa maxay sababta cabbirka dahsoon sida caadiga ah loogu qiyaaso saddex-meeloodlaba laba lakabyada SwiGLU?
    • Si loo hagaajiyo saxnaanta tirooyinka
    • Si loo sameeyo qaabka qoto dheer
    • Si meesha looga saaro baahida caadiga ah
    • Si aad u ilaaliso wadarta cabbirada iyo FLOP-yada isbarbardhigga MLP-ga caadiga ah inkasta oo shax dheeri ah
  5. Hanti gaar ah oo Swish ah marka la barbar dhigo ReLU waa in ay tahay:
    • Si fudud oo aan monotonic ahayn, taas oo u oggolaanaysa diidmo yaryar
    • Si adag u kordhaya oo lagu soo koobay 1
    • La mid ah shaqada sigmoid
    • Kaliya waxa lagu qeexay wax gelinta togan
  6. Yaa ku caan baxay SwiGLU lakabyada quudinta gudbinta beddelka sanadka 2020?
    • Geoffrey Hinton
    • Noam Shazeer
    • Yann LeCun
    • Andrej Karpathy
  7. In the SwiGLU product Swish(xW) multiplied by xV, what happens when one gate value is near zero?
    • The entire Transformer layer is skipped
    • The output projection is replaced by a sigmoid
    • The matching value-branch dimension contributes little to the output
    • The value branch is permanently frozen
  8. Why can keeping a standard MLP’s hidden width unchanged increase SwiGLU cost?
    • The SiLU gate removes the output projection
    • SwiGLU computes a full Hessian
    • The value branch has no learned weights
    • SwiGLU uses an extra projection matrix, so the comparable-width design would have more parameters and FLOPs