গাইড লিঙ্ক ক্যুইজ · মাঝারি স্তর
স্কেল কুইজে ফল্ট টলারেন্স এবং GPU ব্যর্থতা
টেকসই চেকপয়েন্ট, স্বাস্থ্য সংকেত, পুনঃসূচনা নীতি এবং যাচাইকৃত পুনরুদ্ধার আচরণ সহ স্থিতিস্থাপক বিতরণকৃত প্রশিক্ষণ ডিজাইন করুন।
প্রশ্ন 1 এর 8
Why can a single worker failure interrupt a distributed training job?
প্রতিটি প্রশ্নের পূর্বরূপ দেখুন8 প্রশ্ন
আপনি কুইজ নেওয়ার সাথে সাথে উত্তর এবং ব্যাখ্যাগুলি উপস্থিত হয়।
- Why can a single worker failure interrupt a distributed training job?
- Every rank can continue an update after any peer disappears
- A process exit never affects other workers
- Workers coordinate computation and synchronization across devices
- Distributed updates require no communication between participants
- Which state may be needed to resume training faithfully?
- Model weights without optimizer or scheduler state
- Model, optimizer, scheduler, progress, and relevant random state
- Epoch count without data progress
- Train-test split metadata without model state
- Where should recovery checkpoints be stored for node-loss recovery?
- Ephemeral node disk
- GPU memory
- A host page cache
- Durable storage that survives the failed node
- Why should restart policies distinguish infrastructure faults from deterministic errors?
- Blind retries can repeat a code or data failure indefinitely
- A transient timeout can never recover
- Every process crash corrupts all training data
- Restart repairs malformed input records
- What can change when elastic training adjusts the number of workers?
- The number of data-loader workers only
- The GPU model changes when workers are removed
- Effective batch, data sharding, and training schedule
- The parameter count changes with rank assignment
- Why test recovery by terminating a worker in staging?
- A staging run proves that hardware failures cannot recur
- A controlled worker termination verifies checkpoint consistency and restart behavior
- A restart test removes the need to keep durable checkpoints
- A recovery test confirms convergence without restoring state
- What can GPU health monitoring reveal?
- Class-specific prediction errors
- Data split leakage
- Optimizer convergence speed
- Thermal, memory, power, driver, or interconnect issues
- What should be checked after a job resumes from a checkpoint?
- Training progress and evaluation behavior remain consistent
- A single rank restarted while others retained stale state
- The checkpoint timestamp changed after loading
- The worker count remains identical after a membership change