Ikibazo gihuza ibibazo · Biragoye Urwego

Urutonde Ruringaniza hamwe nimpeta Yitondewe Ikibazo

Imyitozo Impeta Kwitondera guhagarika kwimurwa, softmax kumurongo, hamwe no kuringaniza imizigo, harimo imipaka yitumanaho rihuza ibikoresho.

Ikibazo 1 Bya 8

Ni mu buhe buryo uburinganire bukurikirana butandukanya amakuru?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Ni mu buhe buryo uburinganire bukurikirana butandukanya amakuru?
    • Igipimo cyihishe / ibiranga
    • Ikimenyetso (urutonde)
    • Urwego rwicyiciro gusa
    • Umubare w'ibyiciro
  2. Ni ubuhe butumwa Impeta itambuka hagati ya GPU itunganijwe mu mpeta?
    • Ibipimo byuzuye by'icyitegererezo
    • Optimizer ivuga
    • Inyandiko zanyuma
    • Urufunguzo / agaciro bahagaritse
  3. How can Ring Attention combine partial attention blocks without approximating the full attention pattern?
    • Online softmax with a running maximum and normalizer
    • Randomly dropping unseen key blocks
    • Truncating every distant key
    • Replacing all attention scores with zeros
  4. Kuberiki Impeta idasaba GPU imwe kugirango ibike K / V yuzuye?
    • Ijugunya urufunguzo rwinshi
    • Ikoresha icyitegererezo gito
    • K / V ikurikiranwa mubice kandi ibisubizo byigice gusa byegeranijwe
    • Ibyitonderwa biragereranijwe, ntabwo aribyo
  5. When can Ring Attention reduce the exposed time spent transferring K/V blocks?
    • Whenever the model has any number of parameters
    • Only when every device stores the full sequence
    • By eliminating all transfers between devices
    • When useful block computation overlaps those transfers
  6. Ni izihe ngorane ziterwa no guhishira impanvu zitangiza impeta Kuringaniza imitwaro iringaniye?
    • Bituma icyitegererezo kidatandukana
    • Nyuma ibimenyetso byitabira urufunguzo rwinshi, bitera akazi kangana kuri GPU
    • Bikuraho gukenera impeta
    • Bikubye kabiri ibipimo bibarwa
  7. A new Ring Attention block has larger attention scores than previous blocks. Why rescale the earlier softmax sums?
    • To discard the earlier keys permanently
    • To replace all token embeddings
    • To keep accumulated terms consistent with the updated running maximum
    • Kugabanya ingano yamagambo
  8. Under causal masking with contiguous sequence shards, which tokens can require more attention work?
    • Later tokens that can attend to more preceding positions
    • All tokens require exactly one key
    • Only padding tokens at the start
    • Earlier tokens because they can see the future