Ikibazo gihuza ibibazo · Hagati Urwego
Gukora Ibarura ry'ubucuruzi Ikibazo
Ikizamini cyo gukora igenzura ryerekana ububiko bwibicuruzwa, guhitamo gutoranya, no gusubiramo ukuri, harimo n'impamvu compte yinyongera idakemura ibiciro byose byo kwibuka.
Ikibazo 1 Bya 8
Ni ubuhe buryo bwo gusubiramo ibicuruzwa bigurisha kure kugirango ubike kwibuka?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Ni ubuhe buryo bwo gusubiramo ibicuruzwa bigurisha kure kugirango ubike kwibuka?
- Icyitegererezo
- Kubara ibirenze (FLOPs)
- Umuyoboro mugari
- Umubare wuzuye
- Ni ukubera iki ibikorwa-byimbere-bikorwa bisanzwe bibikwa muri byose?
- Kugaragaza kubakoresha
- Guhagarika icyitegererezo
- Barasabwa kubimenyetso
- Kuberako backpropagation ibakeneye kubara gradients
- Why should activation checkpointing overhead be measured on the actual model?
- Checkpoint placement, operators, and hardware determine the cost of replay
- It always adds exactly forty percent to wall-clock time
- It guarantees a tenfold speed improvement
- It removes all forward computation
- For an idealized N-layer chain with comparable activations, what storage scaling can square-root checkpoint placement achieve?
- O(N squared) for all model parameters
- O(1) for all training memory regardless of N
- O(sqrt(N)) for activation storage
- O(N squared) for optimizer state
- Ni ikihe gitekerezo kiri inyuma yo gutoranya (kutuzuye)?
- Ongera ibikorwa byose kabiri
- Ntuzigere ubara ikintu na kimwe
- Ongera usubiremo gusa
- Ongera wibuke gusa-uburemere ariko kubara-bihendutse ops, cache ihenze
- Ni ubuhe buryo bwuzuzanya bukunze guhuzwa no kubara kugirango ubike kwibuka cyane?
- Kongera ingano yicyiciro
- Kurekura ibikorwa kuri CPU cyangwa NVMe
- Kuraho ibice byose bisanzwe
- Guhindura uburyo bwo gufata umwanzuro
- A checkpointed block contains dropout. Why does consistent RNG state matter during replay?
- Dropout becomes a model-weight quantizer
- The optimizer no longer needs gradients
- Backward recomputation should reproduce the intended forward randomness
- The batch size must become one
- Checkpointing reduces stored activations, but optimizer states still exceed GPU memory. Why is the job still out of memory?
- Activation recomputation does not remove optimizer-state storage
- Checkpointing increases every optimizer tensor to FP64
- Backward no longer consumes any memory
- The model cannot use checkpointing with an optimizer