Cross-Attention Quiz

Answers and explanations appear as you take the quiz.

1. What is the key difference between self-attention and cross-attention?
   • In cross-attention, queries come from one sequence and keys/values from another
   • Cross-attention uses no softmax
   • Self-attention works only on images
   • Cross-attention cannot be trained with gradient descent
2. In an encoder-decoder Transformer, where do the queries for cross-attention come from?
   • The encoder's outputs
   • The decoder's current state
   • The raw input embeddings only
   • A separate random matrix
3. Why can the two sequences in cross-attention have different lengths or modalities?
   • Because attention requires equal-length inputs
   • Because it ignores one of the sequences entirely
   • Because queries and keys come from different, independent sources
   • Because it pads everything to a fixed size first
4. How does Stable Diffusion use cross-attention?
   • To compress images into smaller files
   • To translate text between languages
   • To remove noise from audio
   • To condition generated image regions on the text prompt
5. What math operation does cross-attention share with self-attention?
   • Scaled dot-product attention with a softmax
   • Convolution over pixels
   • Recurrent state updates
   • Hard-coded lookup tables
6. What gate does Flamingo use to control information flow through its cross-attention layers?
   • A hard threshold initialized to one
   • A tanh gate initialized to zero
   • A fixed residual weight of 0.5
   • A learned vocabulary lookup table
7. What input sequence structure does Flamingo’s paper say its visual-language model can handle?
   • Only one image followed by a fixed-length label
   • Only text tokens with no visual inputs
   • Visual and textual data interleaved in arbitrary sequences
   • Only video frames with no text
8. What is one purpose of cross-attention in Flamingo’s visual-language architecture?
   • It replaces all text tokens with pixels
   • It prevents any information from passing between modalities
   • It performs only image resizing
   • It lets language-model layers incorporate visual features from image/video inputs