Maswali yanayounganishwa na mwongozo · Ngumu Kiwango

Sambamba na Maswali ya Kuzingatia Pete

Fanya mazoezi ya kuzuia uhamishaji wa Usikivu wa Pete, uhamishaji wa laini mtandaoni, na kusawazisha upakiaji wa sababu, ikijumuisha vikomo vya mawasiliano hupishana kwenye vifaa.

Njia za mwongozo zinazohusianaMfuatano wa Usambamba na Usikivu wa Pete
Swali 1 ya 8

Usambamba wa mfuatano hugawanya data katika mwelekeo gani?

Hakiki kila swali8 maswali

Majibu na maelezo huonekana unapojibu maswali.

  1. Usambamba wa mfuatano hugawanya data katika mwelekeo gani?
    • Kipimo kilichofichwa/kipengele
    • Kipimo cha ishara (mlolongo).
    • Kipimo cha batch pekee
    • Idadi ya tabaka
  2. Je! Uangalifu wa Pete hupita nini kati ya GPU zilizopangwa kwa pete?
    • Uzito wa mfano kamili
    • Majimbo ya kiboreshaji
    • Kumbukumbu za mwisho
    • Vizuizi vya ufunguo/thamani
  3. How can Ring Attention combine partial attention blocks without approximating the full attention pattern?
    • Online softmax with a running maximum and normalizer
    • Randomly dropping unseen key blocks
    • Truncating every distant key
    • Replacing all attention scores with zeros
  4. Kwa nini Usikivu wa Mlio hauhitaji GPU yoyote ili kuhifadhi K/V kamili?
    • Inatupa funguo nyingi
    • Inatumia mfano mdogo
    • K/V inatiririshwa kwa vizuizi na matokeo kidogo tu ndio hukusanywa
    • Uangalifu ni wa kukadiria, sio kamili
  5. When can Ring Attention reduce the exposed time spent transferring K/V blocks?
    • Whenever the model has any number of parameters
    • Only when every device stores the full sequence
    • By eliminating all transfers between devices
    • When useful block computation overlaps those transfers
  6. Je, ni tatizo gani ambalo ufichaji uso wa sababu huleta kwa kusawazisha upakiaji wa Usikivu wa Pete?
    • Inafanya mfano kuwa usio tofauti
    • Baadaye tokeni hushughulikia funguo zaidi, na kusababisha kazi isiyo sawa kwa kila GPU
    • Huondoa hitaji la pete
    • Inaongeza hesabu ya parameta mara mbili
  7. A new Ring Attention block has larger attention scores than previous blocks. Why rescale the earlier softmax sums?
    • To discard the earlier keys permanently
    • To replace all token embeddings
    • To keep accumulated terms consistent with the updated running maximum
    • Ili kupunguza ukubwa wa msamiati
  8. Under causal masking with contiguous sequence shards, which tokens can require more attention work?
    • Later tokens that can attend to more preceding positions
    • All tokens require exactly one key
    • Only padding tokens at the start
    • Earlier tokens because they can see the future