Ikibazo gihuza ibibazo · Biragoye Urwego
Ikibazo cya SwiGLU na Gate Ibikorwa
Gerageza amarembo yerekana amarembo, irembo rya SiLU, ibipimo-bicuruza-ibicuruzwa n'impamvu ubugari bwihishe buhinduka mugihe SwiGLU isimbuye MLP isanzwe.
Ikibazo 1 Bya 8
Ni ikihe gikorwa cy'ibanze gisobanura umurongo wa Gateur?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Ni ikihe gikorwa cy'ibanze gisobanura umurongo wa Gateur?
- Ikintu-cyiza cyo kugwiza icyerekezo kimwe cyakoreshejwe kabiri
- Ongeramo ibice bibiri byerekanwe hamwe
- Guhuriza hamwe imitwe
- Kugabana ibikorwa ukurikije amahame yabo
- Nibihe bikorwa byo gukora SwiGLU ikoresha ishami ryayo?
- ReLU
- Swish (SiLU)
- Tanh
- Softmax
- Ni bangahe matrices yuburemere SwiGLU igaburira-imbere ikumira?
- Imwe
- Babiri
- Bitatu
- Bane
- Ni ukubera iki igipimo cyihishe gisanzwe gipima hafi bibiri bya gatatu mubice bya SwiGLU?
- Kunoza imibare
- Kugirango icyitegererezo cyimbitse
- Gukuraho ibikenewe bisanzwe
- Kugirango ugumane ibipimo byose hamwe na FLOPs ugereranije na MLP isanzwe nubwo matrix yinyongera
- Umutungo wihariye wa Swish ugereranije na ReLU nuko ari:
- Byoroheje kandi bitari monotonic, byemerera ibibi bito
- Kwiyongera cyane no gufatwa kuri 1
- Bisa na sigmoid imikorere
- Gusa byasobanuwe kubintu byiza byinjira
- Ninde wamamaje SwiGLU ya transfert yo kugaburira imbere muri 2020?
- Geoffrey Hinton
- Noam Shazeer
- Yann LeCun
- Andrej Karpathy
- In the SwiGLU product Swish(xW) multiplied by xV, what happens when one gate value is near zero?
- The entire Transformer layer is skipped
- The output projection is replaced by a sigmoid
- The matching value-branch dimension contributes little to the output
- The value branch is permanently frozen
- Why can keeping a standard MLP’s hidden width unchanged increase SwiGLU cost?
- The SiLU gate removes the output projection
- SwiGLU computes a full Hessian
- The value branch has no learned weights
- SwiGLU uses an extra projection matrix, so the comparable-width design would have more parameters and FLOPs