quiz bu lëkkale ak tegtal · Dëgër Niveau

Paralelism ci toppalante ak quiz ci bàyyi xel ci ring

Jàngale Ring Attention bloke toxal, softmax ci net bi, ak balans charge causal, boole ci yam ci jokkoo biy jaxasoo ci aparey yi.

Yooni njiit yu jëm ciParalelism ci toppalante ak bàyyi xel ci ring
Laaj 1 de 8

Ci ban dimension la parallelism bu toppalante di xaaj done yi?

Xoolaatal laaj bu nekk8 laaj

Sooy def quiz bi, tontu yi ak leeral yi dañuy feeñ.

  1. Ci ban dimension la parallelism bu toppalante di xaaj done yi?
    • Dimension bu nëbbu/man-man
    • Token (sequence) dimension
    • Dimension lote bi kese
    • Limu diisaay yi
  2. Lan mooy Ring Attention di jaar ci digganté GPU yuñ raññe ci benn ring?
    • Poids model bu mat
    • Optimiser bi
    • Logits yu mujj
    • Bloku caabi/valeur
  3. How can Ring Attention combine partial attention blocks without approximating the full attention pattern?
    • Online softmax with a running maximum and normalizer
    • Randomly dropping unseen key blocks
    • Truncating every distant key
    • Replacing all attention scores with zeros
  4. Lu tax Ring Attention soxlawul benn GPU ngir denc K/V bi yépp?
    • Dafay sànni caabi yu bari
    • Dafay jëfandikoo model bu gëna ndaw
    • K/V dañu koy stream ci ay bloc te yenn resultaa yi kese lañuy dajale
    • Foofu dañuy xayma, du ndànk
  5. When can Ring Attention reduce the exposed time spent transferring K/V blocks?
    • Whenever the model has any number of parameters
    • Only when every device stores the full sequence
    • By eliminating all transfers between devices
    • When useful block computation overlaps those transfers
  6. Lan mooy jafe-jafe bi maskeer sabab yi di indi ci balansu sargu Ring Attention?
    • Dafay tax model bi mënul wuute
    • Jetons yu ci topp yi deñuy toppatoo caabi yu bari, di waral liggéey bu tolloowul ci GPU bu nekk
    • Dina dindi soxla bague
    • Dafay yokk limu parametre yi ñaari yoon
  7. A new Ring Attention block has larger attention scores than previous blocks. Why rescale the earlier softmax sums?
    • To discard the earlier keys permanently
    • To replace all token embeddings
    • To keep accumulated terms consistent with the updated running maximum
    • Ngir wàññi limu vocabulaire bi
  8. Under causal masking with contiguous sequence shards, which tokens can require more attention work?
    • Later tokens that can attend to more preceding positions
    • All tokens require exactly one key
    • Only padding tokens at the start
    • Earlier tokens because they can see the future