Quiz połączony z przewodnikiem · Ciężko Poziom

Dopasowywanie wzmocnienia za pomocą quizu dla równiarek

Testuje, w jaki sposób oceniający oceniają wyniki modelu, które zadania odpowiadają dostrojeniu wzmocnienia oraz w jaki sposób pojawiają się hakowanie nagród i słabe sygnały szkoleniowe.

Powiązane ścieżki prowadząceDostrajanie zbrojenia za pomocą równiarek
Pytanie 1 z 8

How does reinforcement fine-tuning differ from supervised fine-tuning?

Podgląd każdego pytania8 pytania

Odpowiedzi i wyjaśnienia pojawiają się w miarę rozwiązywania quizu.

  1. How does reinforcement fine-tuning differ from supervised fine-tuning?
    • It trains the model to imitate exact target outputs
    • It rewards sampled outputs a grader scores highly instead of copying examples
    • It needs no training prompts
    • It only changes the system prompt
  2. Which grader type would you use to check that generated code actually works?
    • A programmatic grader that runs unit tests
    • A string-match grader comparing to one reference function
    • A grader that counts words
    • No grader is needed for code
  3. Which task is the poorest fit for RFT according to the guide?
    • Classifying items into defined category codes
    • Solving maths problems with known answers
    • Judging subjective quality where expert graders disagree
    • Structured data extraction with a reference answer
  4. In Reinforcement Fine-Tuning with Graders: what is reward hacking?
    • Stealing another team's grader
    • The model learning to satisfy the grader rather than the real intent
    • Using too little compute during training
    • A grader giving partial credit
  5. Why do prompts where every sample gets the same score provide little learning signal in group-relative methods like GRPO?
    • Because the grader crashes on them
    • Because learning comes from score differences within the group
    • Because they are too long
    • Because GRPO ignores scores entirely
  6. What is the purpose of a KL penalty against the original model during RFT?
    • To speed up grading
    • To limit how far the model drifts from its starting behaviour
    • To increase output length
    • To replace the grader
  7. A model grader over-rewards confident, lengthy replies. What is the likely outcome of training against it?
    • Outputs drift toward longer, more confident wording
    • Model przestaje generować dane wyjściowe
    • Accuracy is guaranteed to improve
    • The grader becomes stricter automatically
  8. Which is the best way to check that rising training reward reflects real improvement?
    • Trust the training reward curve
    • Increase the learning rate
    • Spot-check outputs by hand or with an independent grader on held-out data
    • Remove the validation set