Ikibazo gihuza ibibazo · Biragoye Urwego

Ikibazo cya SwiGLU na Gate Ibikorwa

Gerageza amarembo yerekana amarembo, irembo rya SiLU, ibipimo-bicuruza-ibicuruzwa n'impamvu ubugari bwihishe buhinduka mugihe SwiGLU isimbuye MLP isanzwe.

Inzira zijyanye no kuyoboraIbikorwa bya Swiglu
Ikibazo 1 Bya 8

Ni ikihe gikorwa cy'ibanze gisobanura umurongo wa Gateur?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Ni ikihe gikorwa cy'ibanze gisobanura umurongo wa Gateur?
    • Ikintu-cyiza cyo kugwiza icyerekezo kimwe cyakoreshejwe kabiri
    • Ongeramo ibice bibiri byerekanwe hamwe
    • Guhuriza hamwe imitwe
    • Kugabana ibikorwa ukurikije amahame yabo
  2. Nibihe bikorwa byo gukora SwiGLU ikoresha ishami ryayo?
    • ReLU
    • Swish (SiLU)
    • Tanh
    • Softmax
  3. Ni bangahe matrices yuburemere SwiGLU igaburira-imbere ikumira?
    • Imwe
    • Babiri
    • Bitatu
    • Bane
  4. Ni ukubera iki igipimo cyihishe gisanzwe gipima hafi bibiri bya gatatu mubice bya SwiGLU?
    • Kunoza imibare
    • Kugirango icyitegererezo cyimbitse
    • Gukuraho ibikenewe bisanzwe
    • Kugirango ugumane ibipimo byose hamwe na FLOPs ugereranije na MLP isanzwe nubwo matrix yinyongera
  5. Umutungo wihariye wa Swish ugereranije na ReLU nuko ari:
    • Byoroheje kandi bitari monotonic, byemerera ibibi bito
    • Kwiyongera cyane no gufatwa kuri 1
    • Bisa na sigmoid imikorere
    • Gusa byasobanuwe kubintu byiza byinjira
  6. Ninde wamamaje SwiGLU ya transfert yo kugaburira imbere muri 2020?
    • Geoffrey Hinton
    • Noam Shazeer
    • Yann LeCun
    • Andrej Karpathy
  7. In the SwiGLU product Swish(xW) multiplied by xV, what happens when one gate value is near zero?
    • The entire Transformer layer is skipped
    • The output projection is replaced by a sigmoid
    • The matching value-branch dimension contributes little to the output
    • The value branch is permanently frozen
  8. Why can keeping a standard MLP’s hidden width unchanged increase SwiGLU cost?
    • The SiLU gate removes the output projection
    • SwiGLU computes a full Hessian
    • The value branch has no learned weights
    • SwiGLU uses an extra projection matrix, so the comparable-width design would have more parameters and FLOPs