Guide-yakabatana mibvunzo · Pakati Nhanho

FlashAttention Quiz

Tarisa kuti FlashAttention inoderedza sei yakakwirira-bandwidth-memory traffic paunenge uchigadzira kutarisisa chaiko, uye chii hardware chinoshandura FlashAttention-3 zvinangwa.

Nzira dzekutungamira dzinoenderanaFlashattention
Mubvunzo 1 ye 8

Chii chikuru bhodhoro FlashAttention tarisiro mukutarisisa kwakajairwa?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Chii chikuru bhodhoro FlashAttention tarisiro mukutarisisa kwakajairwa?
    • Kuverenga nekunyora iyo yakakura yekutarisisa matrix kuti inonoke GPU ndangariro
    • Huwandu hwekuyangarara-nzvimbo kuwanda
    • Mutengo wekuisa chiratidzo chemavara ekupinda
    • Network latency pakati peGPUs
  2. Kubuda kweFlashAttention kunofananidzwa sei nekutarisisa kwakajairwa?
    • Iyo fungidziro inotengesa kurongeka kwekukurumidza
    • Zvakafanana nemasvomhu (chaizvo)
    • Inongoshanda kune pfupi dzakatevedzana
    • Inoshandura huremu hwemuenzaniso
  3. Ndeipi ndangariro yeGPU iyo FlashAttention inoshandisa nekugadzirisa data mumataira?
    • Iyo hombe asi inononoka yakakwira-bandwidth memory (HBM)
    • Iyo RAM yeCPU
    • Iyo diki asi inokurumidza kwazvo pa-chip SRAM
    • Disk kuchengetedza
  4. Ndeipi nzira inobvumira FlashAttention kuverengera softmax pasina kuona zvibodzwa zvese kamwechete?
    • Kusvetuka iyo softmax zvachose
    • Kutenderedza zvibodzwa zvese kusvika zero
    • Precomputing softmax paCPU
    • Pamhepo (inowedzerwa) softmax ine kumhanya kudzoreredza
  5. Nekuti iyo yakazara yekutarisisa matrix haina kumbochengetwa muHBM, ndangariro dzekushandisa zviyero zvakangoita:
    • Linearly nekutevedzana kureba
    • Quadratically nehurefu hwekutevedzana
    • Exponentially nehurefu hwekutevedzana
    • Kuzvimiririra kureba kwekutevedzana
  6. Which numeric format does FlashAttention-3 use in its Hopper-focused experiments to accelerate attention?
    • Only 64-bit floating point
    • FP8, with techniques to manage its numerical error
    • Integer text tokens
    • No reduced-precision format
  7. What work-partitioning change does FlashAttention-2 introduce to improve GPU parallelism?
    • It stores the complete attention matrix in HBM
    • It replaces attention with convolution
    • It reduces non-matrix-multiplication work and improves parallelization across thread blocks and warps
    • It uses only CPU execution
  8. In FlashAttention-3, what lets data movement overlap with computation?
    • Replacing attention with a lookup table
    • Computing all scores in host memory
    • Disabling the softmax step
    • Asynchronous execution on Hopper hardware