引導式測驗 · 硬 等級

收集人類偏好資料以進行RLHF測驗

了解成對偏好收集、評估者說明、獎勵模型、PPO、DPO、位置偏差和偏好標籤的限制。

問題 1 的 8

What kind of label does the InstructGPT preference-collection stage record?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. What kind of label does the InstructGPT preference-collection stage record?
    • Which of two candidate responses a rater prefers under task instructions
    • A universal quality score for all possible responses
    • The model’s next-token probabilities only
    • A transcription of the rater’s own ideal response
  2. What typically happens when agreement checks reveal persistently low agreement between raters on certain prompts?
    • The raters are immediately fired
    • The labeling guidelines are usually revised, since the disagreement often points to ambiguity in the instructions
    • The prompts are deleted from the dataset with no further action
    • The reward model is retrained without any human data
  3. In the traditional RLHF pipeline, what is a reward model trained to do?
    • Generate the original candidate responses
    • Translate responses into other languages
    • Assign a scalar score to any response, trained so the human-chosen response in each pair scores higher than the rejected one
    • Replace the need for a language model entirely
  4. How does direct preference optimization (DPO) differ from the traditional reward-model-plus-PPO approach?
    • DPO requires no preference data at all
    • DPO only works for image models, not language models
    • DPO eliminates the need for human raters entirely
    • DPO fine-tunes the language model directly on preference pairs, skipping the separate reward model training step
  5. Why do rater interfaces typically randomize whether a response appears as 'A' or 'B'?
    • To prevent raters from developing a positional bias toward always picking the first option shown
    • To make the interface look more colorful
    • Because response A is always the correct answer
    • To speed up model inference time
  6. In the classic RLHF sequence described in the guide, what does PPO update?
    • The preference rubric alone
    • The policy model, using the reward signal while constraining drift from a reference
    • The raw annotation interface
    • The human raters’ demographic labels
  7. Why is it a misconception that RLHF preference data reflects one single universal notion of a good response?
    • Because preference data is generated entirely by the model with no human involvement
    • Because RLHF has never been used by any real company
    • Because the data actually reflects the specific guidelines and rater pool chosen, which can shift model behavior in different directions
    • Because reward models are incapable of learning from human data
  8. Why might comparing two model responses become harder for human raters as underlying models improve?
    • Because better models produce identical responses every time
    • Because rater training becomes unnecessary once models improve
    • Because improved models no longer require any preference data
    • Because both candidate responses may already be fluent and largely correct, making meaningful differences subtler to detect