Guide-yakabatana mibvunzo · Zvakaoma Nhanho

ZeRO uye Sharded Optimizers Quiz

Covers ZeRO stages, Adam state memory, parameter gathering, gradient reduction, offload options, and communication costs.

Nzira dzekutungamira dzinoenderanaZero Uye Sharded Optimizers
Mubvunzo 1 ye 8

Ndeipi redundancy iyo ZeRO inobvisa ichienzaniswa neyakajeka data parallelism?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Ndeipi redundancy iyo ZeRO inobvisa ichienzaniswa neyakajeka data parallelism?
    • Dzokorora makopi akazara e optimizer state, gradients, uye paramita paGPU yega yega
    • Duplicate makopi eiyo data yekudzidziswa pane yega GPU
    • Redundant network tambo pakati node
    • Inodzokororwa kumberi inopfuura pamusoro pebatch imwechete
  2. Nei optimizer state kazhinji iri hombe yekurangarira hogi naAdam?
    • Adhamu haachengeti imwe imwe mamiriro zvachose
    • Adamu anochengeta ekuwedzera per-parameta makoshero (senge simba uye musiyano), kazhinji kakawanda saizi yemuenzaniso
    • Adamu anodzokorora dataset rese
    • Adamu anoda kuchengetedza gradient yese yakapfuura zvachose
  3. Chii chinoita ZeRO Stage 3 shard isingaite Stage 1 ne2?
    • Iyo data yekudzidziswa
    • Basa rekurasikirwa
    • The modhi parameters pachavo
    • The kudzidza rate purogiramu
  4. MuZeRO Stage 3, iyo GPU inowana sei maparamendi akazara ainoda kuti ipfuure yekumberi?
    • Hazvimbodi zvizere parameters
    • Inoadhaunirodha kubva painternet
    • Inovadzokorodza kubva pakatanga nguva imwe neimwe
    • Inoita zvese-kuunganidza kuunganidza zvimedu zvisipo, yobva yazvirasa mushure
  5. Ndeipi PyTorch inoratidzira natively inoshandisa ZeRO-maitiro sharding?
    • FSDP (Fully Shared Data Parallel)
    • DataLoader
    • TorchScript
    • Autograd
  6. Chii chinonzi core tradeoff ZeRO inoita kuchengetedza ndangariro?
    • Inopira kurongeka kwemuenzaniso wekumhanya
    • Inowedzera kutaurirana kuunganidza uye kuparadzira shards sezvinodiwa
    • Inobvisa zvidimbu kubva pamuenzaniso
    • Inoshanda chete nemhando shoma shoma
  7. Which state is partitioned in ZeRO Stage 1?
    • Only parameters, with optimizer states replicated
    • Parameters, gradients, and optimizer states all sharded
    • Optimizer states, while parameters and gradients remain replicated
    • Only input activations and labels
  8. What is a practical cost of ZeRO-style sharding?
    • It removes the need to calculate gradients
    • It makes every optimizer update local with no data exchange
    • It requires the model to be retrained from scratch for every step
    • Additional communication to gather parameters and distribute gradients