Flash Attention Quiz

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

1. Ndeipi bhodhoro guru rinotariswa neFlash Attention?
   • Kuverenga nekunyora iyo yakakura yekutarisisa matrix kuti inonoke GPU ndangariro
   • Huwandu hweinoyangarara-nzvimbo yekuwedzera mukutarisa
   • Mutengo wekurodha modhi uremu kubva kudhisiki
   • Network latency pakati pemaGPU akawanda
2. Flash Attention inodzivirira sei kuchengetedza iyo yakazara N-by-N yekutarisisa matrix?
   • Inofananidzira kutarisisa nekufungidzira zvisina kujairika
   • Inogadzirisa mibvunzo, makiyi, uye kukosha mumatairi madiki uye inosanganisa mhedzisiro
   • Inodarika ma token pairs zvachose
   • Inomanikidza matrix ne quantization
3. Ndeipi nzira inobvumira Flash Attention kuverengera softmax nenzira kwayo pasina kuona mutsara wese kamwechete?
   • A precomputed lookup table of exponentials
   • Kudonhedza iyo softmax uye kushandisa mutsara kufungidzira
   • Inomhanya (pamhepo) softmax inoteedzera max uye sum
   • Kuronga zvibodzwa usati wajaira
4. Kuenzaniswa nekutarisa kwakajairwa, kuburitswa kweFlash Attention ndiko:
   • A rough approximation inotsinhanisa huchokwadi hwekumhanya
   • Zvakanyanya kurongeka nekuda kwehutsanana hwepamusoro
   • Inoshanda chete kune pfupi dzakatevedzana
   • Pamasvomhu zvakafanana, zvakangoitwa nemakomputa zvakanyanya
5. Nei Flash Attention ichibatsira zvakanyanya nemateedzana akareba?
   • Nekuti kutarisisa ndangariro inokura quadratically nekutevedzana kureba
   • Nekuti kutevedzana kwakareba kunoda matinji mashoma
   • Nekuti softmax inova isingakodzeri kune marefu ekuisa
   • Nekuti maGPU anomhanya zvishoma pane mapfupi akateedzana
6. Chii chinonzi 'IO-aware' dhizaini yeFlash Attention inotungamira kudzikisira?
   • Nhamba yemuenzaniso parameters
   • Memory inoverenga uye inonyora pakati peSRAM neHBM
   • Kudzika kweneural network
   • Saizi yemazwi eiyo tokenizer
7. What remains computationally quadratic for dense exact self-attention even when FlashAttention reduces memory traffic?
   • The number of model layers must double
   • Every token is compared with only one other token
   • The number of query-key pair interactions grows with the square of sequence length
   • The number of model parameters becomes zero
8. Why can FlashAttention’s results differ slightly from a reference implementation despite computing the same mathematical attention?
   • It intentionally drops all long-distance tokens
   • It changes the model weights during inference
   • It replaces softmax with a random sample
   • Floating-point operations can occur in a different order during tiled accumulation