引導式測驗 · 硬 等級
收集人類偏好資料以進行RLHF測驗
了解成對偏好收集、評估者說明、獎勵模型、PPO、DPO、位置偏差和偏好標籤的限制。
問題 1 的 8
What kind of label does the InstructGPT preference-collection stage record?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- What kind of label does the InstructGPT preference-collection stage record?
- Which of two candidate responses a rater prefers under task instructions
- A universal quality score for all possible responses
- The model’s next-token probabilities only
- A transcription of the rater’s own ideal response
- What typically happens when agreement checks reveal persistently low agreement between raters on certain prompts?
- The raters are immediately fired
- The labeling guidelines are usually revised, since the disagreement often points to ambiguity in the instructions
- The prompts are deleted from the dataset with no further action
- The reward model is retrained without any human data
- In the traditional RLHF pipeline, what is a reward model trained to do?
- Generate the original candidate responses
- Translate responses into other languages
- Assign a scalar score to any response, trained so the human-chosen response in each pair scores higher than the rejected one
- Replace the need for a language model entirely
- How does direct preference optimization (DPO) differ from the traditional reward-model-plus-PPO approach?
- DPO requires no preference data at all
- DPO only works for image models, not language models
- DPO eliminates the need for human raters entirely
- DPO fine-tunes the language model directly on preference pairs, skipping the separate reward model training step
- Why do rater interfaces typically randomize whether a response appears as 'A' or 'B'?
- To prevent raters from developing a positional bias toward always picking the first option shown
- To make the interface look more colorful
- Because response A is always the correct answer
- To speed up model inference time
- In the classic RLHF sequence described in the guide, what does PPO update?
- The preference rubric alone
- The policy model, using the reward signal while constraining drift from a reference
- The raw annotation interface
- The human raters’ demographic labels
- Why is it a misconception that RLHF preference data reflects one single universal notion of a good response?
- Because preference data is generated entirely by the model with no human involvement
- Because RLHF has never been used by any real company
- Because the data actually reflects the specific guidelines and rater pool chosen, which can shift model behavior in different directions
- Because reward models are incapable of learning from human data
- Why might comparing two model responses become harder for human raters as underlying models improve?
- Because better models produce identical responses every time
- Because rater training becomes unnecessary once models improve
- Because improved models no longer require any preference data
- Because both candidate responses may already be fluent and largely correct, making meaningful differences subtler to detect