Guide-yakabatana mibvunzo · Zvakaoma Nhanho

SwiGLU uye Gated Activations Quiz

Inoedza fungidziro yegedhi, gedhi reSiLU, parameter-mutengo wekutengesa uye nei yakavanzika hupamhi ichichinja kana SwiGLU ichitsiva yakajairwa MLP.

Nzira dzekutungamira dzinoenderanaSwiglu Uye Gated Activations
Mubvunzo 1 ye 8

Ndeipi yakakosha kushanda inotsanangura Gated Linear Unit?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Ndeipi yakakosha kushanda inotsanangura Gated Linear Unit?
    • Element-huchenjeri kuwanda kwefungidziro imwechete neyakagadziriswa yechipiri fungidziro
    • Kuwedzera mitsara miviri yekufungidzira pamwe chete
    • Concatenating attention heads
    • Kupatsanura activation nemaitiro avo
  2. Ndeipi activation basa rinoshandiswa neSwiGLU kune rayo gedhi bazi?
    • ReLU
    • Swish (SiLU)
    • Tanh
    • Softmax
  3. Mangani ehuremu matrices anoshandiswa neSwiGLU feed-forward block block?
    • Imwe
    • Vaviri
    • Vatatu
    • Zvina
  4. Nei iyo yakavanzika dimension ichiwanzo kuyerwa kusvika pazvikamu zviviri muzvitatu muSwiGLU layer?
    • Kuvandudza nhamba chaiyo
    • Kuita kuti muenzaniso uwedzere
    • Kubvisa kudiwa kwe normalization
    • Kuchengeta yakazara paramita uye FLOPs inofananidzwa neyakajairwa MLP kunyangwe yakawedzera matrix
  5. Chinhu chakasarudzika cheSwish chichienzaniswa neReLU ndechekuti:
    • Smooth uye kwete-monotonic, kubvumira zviduku zvisina kunaka
    • Kuwedzera zvakanyanya uye kuvharirwa pa1
    • Kufanana nebasa re sigmoid
    • Inotsanangurwa chete kune zvinovaka
  6. Ndiani akakurudzira SwiGLU ye transformer feed-mberi maseru muna 2020?
    • Geoffrey Hinton
    • Noam Shazeer
    • Yann LeCun
    • Andrej Karpathy
  7. In the SwiGLU product Swish(xW) multiplied by xV, what happens when one gate value is near zero?
    • The entire Transformer layer is skipped
    • The output projection is replaced by a sigmoid
    • The matching value-branch dimension contributes little to the output
    • The value branch is permanently frozen
  8. Why can keeping a standard MLP’s hidden width unchanged increase SwiGLU cost?
    • The SiLU gate removes the output projection
    • SwiGLU computes a full Hessian
    • The value branch has no learned weights
    • SwiGLU uses an extra projection matrix, so the comparable-width design would have more parameters and FLOPs