Idanwo ti o sopọ mọ itọsọna · Alabọde Ipele

Ibere ise Iṣiro Tradeoffs adanwo

Test activation checkpointing memory tradeoffs, selective recomputation, and replay correctness, including why extra compute does not fix all memory costs.

Jẹmọ itọsọna awọn ọnaImuṣiṣẹ atunkọ Iṣiro Tradeoffs
Ibeere 1 ti 8

Kini atunwi imuṣiṣẹ ṣe iṣowo kuro lati fi iranti pamọ?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Kini atunwi imuṣiṣẹ ṣe iṣowo kuro lati fi iranti pamọ?
    • Awoṣe deede
    • Iṣiro afikun (FLOPs)
    • Bandiwidi nẹtiwọki
    • Ipeye iye
  2. Kini idi ti awọn imuṣiṣẹ siwaju-kọja ni deede ti o tọju rara?
    • Lati ṣafihan wọn si olumulo
    • Lati compress awoṣe
    • Wọn ti wa ni ti beere fun tokenization
    • Nitori ifakalẹ nilo wọn lati ṣe iṣiro awọn gradients
  3. Why should activation checkpointing overhead be measured on the actual model?
    • Checkpoint placement, operators, and hardware determine the cost of replay
    • It always adds exactly forty percent to wall-clock time
    • It guarantees a tenfold speed improvement
    • It removes all forward computation
  4. For an idealized N-layer chain with comparable activations, what storage scaling can square-root checkpoint placement achieve?
    • O(N squared) for all model parameters
    • O(1) for all training memory regardless of N
    • O(sqrt(N)) for activation storage
    • O(N squared) for optimizer state
  5. Kini imọran lẹhin yiyan (kii ṣe kikun) atunlo?
    • Recompute gbogbo isẹ lemeji
    • Maṣe ṣe atunṣe ohunkohun
    • Recompute nikan ni ifibọ Layer
    • Ṣe atunṣiro iranti nikan-eru ṣugbọn awọn ops ti ko gbowolori, awọn kaṣe gbowolori
  6. Iru ilana ibaramu wo ni igbagbogbo ni idapo pẹlu iṣiro lati fipamọ paapaa iranti diẹ sii?
    • Nmu iwọn ipele pọ si
    • Awọn iṣẹ ṣiṣe piparẹ si Sipiyu tabi NVMe
    • Yiyọ gbogbo normalization fẹlẹfẹlẹ
    • Yipada si ipo itọkasi
  7. A checkpointed block contains dropout. Why does consistent RNG state matter during replay?
    • Dropout becomes a model-weight quantizer
    • The optimizer no longer needs gradients
    • Backward recomputation should reproduce the intended forward randomness
    • The batch size must become one
  8. Checkpointing reduces stored activations, but optimizer states still exceed GPU memory. Why is the job still out of memory?
    • Activation recomputation does not remove optimizer-state storage
    • Checkpointing increases every optimizer tensor to FP64
    • Backward no longer consumes any memory
    • The model cannot use checkpointing with an optimizer