Idanwo ti o sopọ mọ itọsọna · Alabọde Ipele

Idanwo Ṣiṣayẹwo Gradient

Explains activation rematerialization, memory savings, schedule-dependent compute overhead, and combinations with sharding and mixed precision.

Jẹmọ itọsọna awọn ọnaṢiṣayẹwo Gradient
Ibeere 1 ti 8

Kini ayẹwo ayẹwo gradient ni akọkọ ṣe iṣowo lati le fipamọ iranti?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Kini ayẹwo ayẹwo gradient ni akọkọ ṣe iṣowo lati le fipamọ iranti?
    • Iṣiro afikun (iṣiro) fun lilo iranti kekere
    • Awoṣe deede fun iyara
    • Aaye disk fun bandiwidi nẹtiwọki
    • Konge fun ipele iwọn
  2. Kini idi ti awọn iṣẹ ṣiṣe ṣe deede ni ipamọ lakoko gbigbe siwaju?
    • Lati ṣafihan wọn si olumulo
    • Nitori ifakalẹ nilo wọn lati ṣe iṣiro awọn gradients
    • Lati mu iyara ikojọpọ data
    • Wọn ti wa ni ko kosi nilo
  3. In a simplified chain of N similarly sized layers, what can checkpoint boundaries spaced about sqrt(N) layers apart achieve in the textbook schedule?
    • It stores no activations and performs no recomputation
    • It guarantees constant memory for every model graph
    • Activation storage can scale on the order of sqrt(N) checkpoints
    • It makes optimizer-state memory scale as sqrt(N)
  4. What most determines gradient-checkpointing runtime overhead in a practical model?
    • Only the model’s file extension
    • A fixed 20–30% slowdown for every model
    • The number of labels in the dataset
    • Which operations are recomputed, checkpoint placement, graph structure, and hardware
  5. Ni PyTorch, iru ohun elo wo ni a lo nigbagbogbo lati lo ibi-iyẹwo gradient si module kan?
    • ògùṣọ.utils.checkpoint
    • ògùṣọ.optim.SGD
    • ògùṣọ.nn.Dropout
    • ògùṣọ.cuda.empty_cache
  6. What happens to operations inside a checkpointed region during backward?
    • Their gradients are skipped entirely
    • Their forward computations are rerun to regenerate needed activations
    • Their weights are permanently discarded
    • They are executed only during inference
  7. Which training techniques can be combined with activation checkpointing to reduce different memory costs?
    • Only increasing the number of labels
    • Turning off backpropagation
    • FSDP for sharded model state and mixed precision for lower-precision tensors
    • Replacing the model with a tokenizer
  8. What is selective activation checkpointing intended to choose?
    • Which training labels are removed
    • Which model outputs are published
    • Which GPU becomes the tokenizer
    • Which operations to save and which to recompute based on their memory and runtime costs