SwiGLU ati Idanwo Awọn iṣẹ ṣiṣe Gated

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

1. Iṣe mojuto wo ni asọye Ẹka Linear Gated kan?
   • Ilọpo-ọlọgbọn eroja ti isọtẹlẹ kan nipasẹ isọtẹlẹ keji ti mu ṣiṣẹ
   • Fifi awọn asọtẹlẹ laini meji pọ
   • Concatenating akiyesi olori
   • Pipin awọn iṣẹ ṣiṣe nipasẹ iwuwasi wọn
2. Iṣẹ imuṣiṣẹ wo ni SwiGLU lo fun ẹka ẹnu-ọna rẹ?
   • ReLU
   • Siwiṣi (SiLU)
   • Tanh
   • Softmax
3. Awọn matiri iwuwo iwuwo melo ni idinaki ifunni siwaju SwiGLU lo?
   • Ọkan
   • Meji
   • Mẹta
   • Mẹrin
4. Kini idi ti iwọn ti o farapamọ ṣe deede iwọn si bii ida meji ninu mẹta ni awọn fẹlẹfẹlẹ SwiGLU?
   • Lati mu imudara iye to
   • Lati ṣe awoṣe jinle
   • Lati yọ iwulo fun isọdọtun
   • Lati tọju awọn paramita lapapọ ati awọn FLOPs ni afiwe si MLP boṣewa laibikita matrix afikun
5. Ohun-ini iyasọtọ ti Swish ni akawe si ReLU ni pe o jẹ:
   • Dan ati ti kii-monotonic, gbigba awọn odi kekere
   • Ti n pọ si ni ilodi si ati pe ni 1
   • Aami si iṣẹ sigmoid
   • Nikan ni asọye fun awọn igbewọle rere
6. Tani o gbakiki SwiGLU fun awọn ipele ifunni-ilọsiwaju ni 2020?
   • Geoffrey Hinton
   • Noam Shazeer
   • Yan LeCun
   • Andrej Karpathy
7. In the SwiGLU product Swish(xW) multiplied by xV, what happens when one gate value is near zero?
   • The entire Transformer layer is skipped
   • The output projection is replaced by a sigmoid
   • The matching value-branch dimension contributes little to the output
   • The value branch is permanently frozen
8. Why can keeping a standard MLP’s hidden width unchanged increase SwiGLU cost?
   • The SiLU gate removes the output projection
   • SwiGLU computes a full Hessian
   • The value branch has no learned weights
   • SwiGLU uses an extra projection matrix, so the comparable-width design would have more parameters and FLOPs