引導式測驗 · 硬 等級
估算 LLM 訓練 FLOPs 測驗
使用 6ND 估計密集的 LLM 訓練操作,將吞吐量轉換為運行時間,區分利用率指標,並檢查規劃假設。
問題 1 的 8
In the dense-model approximation C ≈ 6ND, which quantities do N and D represent?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- In the dense-model approximation C ≈ 6ND, which quantities do N and D represent?
- Parameter count and total processed training tokens
- Number of devices and unique documents
- Network layers and batch size
- Numerical precision and model file size
- A ten-billion-token corpus is processed twice. Which D belongs in the 6ND estimate?
- Ten billion, because only unique tokens count
- Twenty billion, because both passes perform training work
- Five billion, because repeated data halves the cost
- Two, because D means the number of passes
- Which distinction between FLOPs and FLOP/s is needed when estimating a training schedule?
- Both are memory capacities
- FLOPs is speed and FLOP/s is total work
- FLOPs counts operations; FLOP/s is an operation rate
- Both directly specify the monetary cost
- For one billion parameters and twenty billion processed tokens, what does 6ND estimate?
- 1.2 × 10¹⁰ FLOPs
- 6 × 10²⁰ FLOPs
- 2 × 10¹⁹ FLOPs
- 1.2 × 10²⁰ FLOPs
- Eight accelerators each have a relevant peak of 100 TFLOP/s and assumed model FLOPs utilization of 50%. What effective model throughput is used?
- 400 TFLOP/s
- 800 TFLOP/s
- 4 TFLOP/s
- 40,000 TFLOP/s
- The worked run takes about 83.3 elapsed hours on eight accelerators. Approximately how many accelerator-hours does it consume?
- About 10.4
- About 667
- About 83.3
- About 300,000
- Why can hardware FLOPs utilization exceed model FLOPs utilization for a run that recomputes activations?
- Recomputation creates new unique training tokens
- Model utilization counts storage instead of operations
- Hardware accounting can include recomputation that is excluded from useful model-work accounting
- The two measures always use different device counts
- Before treating a 6ND runtime estimate as a practical schedule, which check does the guide recommend?
- Assume the model fits if peak FLOP/s is high
- Reuse any published utilization percentage unchanged
- Count only model weights and ignore training buffers
- Measure a representative pilot and account for memory and activities outside its timing scope