引導式測驗 · 硬 等級

使用 Graders Quiz 進行強化微調

測試評分者如何對模型輸出進行評分、哪些任務適合強化微調,以及獎勵駭客和弱訓練訊號是如何出現的。

問題 1 的 8

How does reinforcement fine-tuning differ from supervised fine-tuning?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. How does reinforcement fine-tuning differ from supervised fine-tuning?
    • It trains the model to imitate exact target outputs
    • It rewards sampled outputs a grader scores highly instead of copying examples
    • It needs no training prompts
    • It only changes the system prompt
  2. Which grader type would you use to check that generated code actually works?
    • A programmatic grader that runs unit tests
    • A string-match grader comparing to one reference function
    • A grader that counts words
    • No grader is needed for code
  3. Which task is the poorest fit for RFT according to the guide?
    • Classifying items into defined category codes
    • Solving maths problems with known answers
    • Judging subjective quality where expert graders disagree
    • Structured data extraction with a reference answer
  4. In Reinforcement Fine-Tuning with Graders: what is reward hacking?
    • Stealing another team's grader
    • The model learning to satisfy the grader rather than the real intent
    • Using too little compute during training
    • A grader giving partial credit
  5. Why do prompts where every sample gets the same score provide little learning signal in group-relative methods like GRPO?
    • Because the grader crashes on them
    • Because learning comes from score differences within the group
    • Because they are too long
    • Because GRPO ignores scores entirely
  6. What is the purpose of a KL penalty against the original model during RFT?
    • To speed up grading
    • To limit how far the model drifts from its starting behaviour
    • To increase output length
    • To replace the grader
  7. A model grader over-rewards confident, lengthy replies. What is the likely outcome of training against it?
    • Outputs drift toward longer, more confident wording
    • 模型停止產生輸出
    • Accuracy is guaranteed to improve
    • The grader becomes stricter automatically
  8. Which is the best way to check that rising training reward reflects real improvement?
    • Trust the training reward curve
    • Increase the learning rate
    • Spot-check outputs by hand or with an independent grader on held-out data
    • Remove the validation set