Kuiz berkaitan panduan · Sederhana Tahap

Toleransi Kesalahan dan Kegagalan GPU pada Kuiz Skala

Reka bentuk latihan teragih berdaya tahan dengan pusat pemeriksaan tahan lama, isyarat kesihatan, dasar mulakan semula dan tingkah laku pemulihan yang disahkan.

soalan 1 daripada 8

Why can a single worker failure interrupt a distributed training job?

Pratonton setiap soalan8 soalan

Jawapan dan penjelasan muncul semasa anda mengambil kuiz.

  1. Why can a single worker failure interrupt a distributed training job?
    • Every rank can continue an update after any peer disappears
    • A process exit never affects other workers
    • Workers coordinate computation and synchronization across devices
    • Distributed updates require no communication between participants
  2. Which state may be needed to resume training faithfully?
    • Model weights without optimizer or scheduler state
    • Model, optimizer, scheduler, progress, and relevant random state
    • Epoch count without data progress
    • Train-test split metadata without model state
  3. Where should recovery checkpoints be stored for node-loss recovery?
    • Ephemeral node disk
    • GPU memory
    • A host page cache
    • Durable storage that survives the failed node
  4. Why should restart policies distinguish infrastructure faults from deterministic errors?
    • Blind retries can repeat a code or data failure indefinitely
    • A transient timeout can never recover
    • Every process crash corrupts all training data
    • Restart repairs malformed input records
  5. What can change when elastic training adjusts the number of workers?
    • The number of data-loader workers only
    • The GPU model changes when workers are removed
    • Effective batch, data sharding, and training schedule
    • The parameter count changes with rank assignment
  6. Why test recovery by terminating a worker in staging?
    • A staging run proves that hardware failures cannot recur
    • A controlled worker termination verifies checkpoint consistency and restart behavior
    • A restart test removes the need to keep durable checkpoints
    • A recovery test confirms convergence without restoring state
  7. What can GPU health monitoring reveal?
    • Class-specific prediction errors
    • Data split leakage
    • Optimizer convergence speed
    • Thermal, memory, power, driver, or interconnect issues
  8. What should be checked after a job resumes from a checkpoint?
    • Training progress and evaluation behavior remain consistent
    • A single rank restarted while others retained stale state
    • The checkpoint timestamp changed after loading
    • The worker count remains identical after a membership change