Ikibazo gihuza ibibazo · Biragoye Urwego

Gukwirakwiza Leta Kurekura CPU na NVMe Ikibazo

Koresha optimizer yikuramo ibaruramari ryibikoresho, iyakirwa ryavuguruwe, hamwe no kohereza ibicuruzwa, harimo n'impamvu gukuramo bidakuraho GPU yose.

Ikibazo 1 Bya 8

Niyihe ntego yibanze yo gupakurura optimizer leta kuri CPU cyangwa NVMe?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Niyihe ntego yibanze yo gupakurura optimizer leta kuri CPU cyangwa NVMe?
    • Kugirango icyitegererezo kibe cyiza
    • Kugabanya umubare wamahugurwa akenewe
    • Kurandura ibikenewe bya gradients burundu
    • Guhuza moderi nini mukurekura ububiko bwa GPU
  2. For an Adam setup with two FP32 moments and an FP32 master weight, how much memory do those three values use per parameter?
    • Two bytes for all three values
    • Twelve bytes, excluding model weights and gradients
    • Sixteen bytes for the three values alone
    • Zero bytes because they are implicit
  3. Nibihe bikoresho biranga abantu benshi bakwirakwiza optimizer?
    • Igikorwa gishya cyo gukora
    • JPEG yogushushanya
    • Umuvuduko Wihuse ZeRO-Kurekura na ZeRO-Infinity
    • Gukata buhoro buhoro
  4. Ni ikihe giciro nyamukuru cyibikorwa byo gupakurura kuri CPU cyangwa NVMe?
    • Amakuru agomba kugenda hejuru yihuta nka PCIe cyangwa kuri disiki, wongeyeho ubukererwe
    • Icyitegererezo cyibagiwe imyitozo mbere
    • Optimizer ihinduka imibare itari yo
    • Ubushyuhe bwa GPU bugabanuka cyane
  5. Nigute sisitemu yo gupakurura ihisha byinshi mubukererwe?
    • Mugusiba inzira yinyuma
    • Muguhuza amakuru mbere yo kubara hamwe na GPU ikomeje
    • Muguhugura ikintu kimwe gusa icyarimwe
    • Muguhagarika kuvanga neza
  6. Muri ZeRO-Offload, ni hehe intambwe ya Adam optimizer ikorera?
    • Byose kumugenzuzi wa NVMe
    • Kwerekana bitandukanye GPU
    • Kuri CPU, ukoresheje uburemere bwibihe nibihe bibitswe aho
    • Imbere ya enterineti
  7. Optimizer states are offloaded, but GPU weights and activations still exceed capacity. What follows?
    • The job can still run out of GPU memory
    • Offloading guarantees the job fits any GPU
    • The optimizer moments automatically become model activations
    • The model no longer needs forward tensors
  8. A CPU-offloaded optimizer becomes the training bottleneck. Which measurement is most relevant?
    • Only the inference accuracy
    • The UI font used for training logs
    • Only the number of dataset classes
    • Host optimizer-update time and host-device transfer time