Idanwo ti o sopọ mọ itọsọna · Alabọde Ipele

FlashAttention adanwo

Check how FlashAttention reduces high-bandwidth-memory traffic while computing exact attention, and what hardware changes FlashAttention-3 targets.

Jẹmọ itọsọna awọn ọnaIfarabalẹ
Ibeere 1 ti 8

Kini awọn ibi-afẹde FlashAttention akọkọ igo ni akiyesi boṣewa?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Kini awọn ibi-afẹde FlashAttention akọkọ igo ni akiyesi boṣewa?
    • Kika ati kikọ matrix akiyesi nla lati fa fifalẹ iranti GPU
    • Nọmba awọn isodipupo-ojuami lilefoofo
    • Awọn iye owo ti àmi ọrọ input
    • Lairi nẹtiwọki laarin GPUs
  2. Bawo ni iṣelọpọ FlashAttention ṣe afiwe si akiyesi boṣewa?
    • O jẹ isunmọ ti o ṣowo deede fun iyara
    • O jẹ aami mathematiki (gangan)
    • O nikan ṣiṣẹ fun kukuru lesese
    • O ayipada awọn awoṣe ká òṣuwọn
  3. Iranti GPU wo ni FlashAttention nilokulo nipasẹ sisẹ data ni awọn alẹmọ?
    • Iranti bandiwidi giga ti o tobi ṣugbọn o lọra (HBM)
    • Sipiyu ká Ramu
    • Awọn kekere sugbon gan sare on-chip SRAM
    • Ibi ipamọ disk
  4. Ilana wo ni o jẹ ki FlashAttention ṣe iṣiro softmax laisi ri gbogbo awọn ikun ni ẹẹkan?
    • Foju softmax naa patapata
    • Yika gbogbo awọn ikun si odo
    • Precomputing softmax lori Sipiyu
    • Online (afikun) softmax pẹlu ṣiṣe atunṣe
  5. Nitori matrix akiyesi ni kikun ko ni ipamọ rara ni HBM, awọn iwọn lilo iranti ni aijọju:
    • Linearly pẹlu ọkọọkan ipari
    • Quadratically pẹlu ọkọọkan ipari
    • Exponentially pẹlu ọkọọkan ipari
    • Ni ominira ti ipari ọkọọkan
  6. Which numeric format does FlashAttention-3 use in its Hopper-focused experiments to accelerate attention?
    • Only 64-bit floating point
    • FP8, with techniques to manage its numerical error
    • Integer text tokens
    • No reduced-precision format
  7. What work-partitioning change does FlashAttention-2 introduce to improve GPU parallelism?
    • It stores the complete attention matrix in HBM
    • It replaces attention with convolution
    • It reduces non-matrix-multiplication work and improves parallelization across thread blocks and warps
    • It uses only CPU execution
  8. In FlashAttention-3, what lets data movement overlap with computation?
    • Replacing attention with a lookup table
    • Computing all scores in host memory
    • Disabling the softmax step
    • Asynchronous execution on Hopper hardware