Maswali yanayounganishwa na mwongozo · Kati Kiwango

Maswali ya Marekebisho ya Uamilisho wa Recomputation

Jaribio la kuwezesha mabadiliko ya kukagua kumbukumbu, hesabu iliyochaguliwa, na usahihi wa kucheza tena, ikijumuisha kwa nini hesabu ya ziada haisuluhishi gharama zote za kumbukumbu.

Njia za mwongozo zinazohusianaUamilisho Recomputation Tradeoffs
Swali 1 ya 8

Je, hesabu ya uanzishaji inauza nini ili kuokoa kumbukumbu?

Hakiki kila swali8 maswali

Majibu na maelezo huonekana unapojibu maswali.

  1. Je, hesabu ya uanzishaji inauza nini ili kuokoa kumbukumbu?
    • Usahihi wa mfano
    • Kokotoo la ziada (FLOPs)
    • Bandwidth ya mtandao
    • Usahihi wa nambari
  2. Kwa nini uanzishaji wa kupita-mbele kawaida huhifadhiwa kabisa?
    • Ili kuzionyesha kwa mtumiaji
    • Ili kubana mfano
    • Wanahitajika kwa ishara
    • Kwa sababu uenezaji wa nyuma unazihitaji ili kukokotoa viwango vya juu
  3. Why should activation checkpointing overhead be measured on the actual model?
    • Checkpoint placement, operators, and hardware determine the cost of replay
    • It always adds exactly forty percent to wall-clock time
    • It guarantees a tenfold speed improvement
    • It removes all forward computation
  4. For an idealized N-layer chain with comparable activations, what storage scaling can square-root checkpoint placement achieve?
    • O(N squared) for all model parameters
    • O(1) for all training memory regardless of N
    • O(sqrt(N)) for activation storage
    • O(N squared) for optimizer state
  5. Je, ni wazo gani la urejeshaji wa kuchagua (sio kamili)?
    • Rejesha kila operesheni mara mbili
    • Kamwe usirudishe chochote
    • Rejesha safu ya upachikaji pekee
    • Rekebisha ops zenye kumbukumbu-nzito lakini hesabu-nafuu, akiba za gharama kubwa
  6. Ni mbinu gani ya ziada ambayo mara nyingi hujumuishwa na hesabu ili kuokoa kumbukumbu zaidi?
    • Kuongeza ukubwa wa kundi
    • Inapakia uanzishaji kwa CPU au NVMe
    • Kuondoa tabaka zote za urekebishaji
    • Inabadilisha hadi modi ya uelekezaji
  7. A checkpointed block contains dropout. Why does consistent RNG state matter during replay?
    • Dropout becomes a model-weight quantizer
    • The optimizer no longer needs gradients
    • Backward recomputation should reproduce the intended forward randomness
    • The batch size must become one
  8. Checkpointing reduces stored activations, but optimizer states still exceed GPU memory. Why is the job still out of memory?
    • Activation recomputation does not remove optimizer-state storage
    • Checkpointing increases every optimizer tensor to FP64
    • Backward no longer consumes any memory
    • The model cannot use checkpointing with an optimizer