Kvíz spojený s průvodcem · Těžko úroveň
Collecting Human Preference Data for RLHF Quiz
Understand pairwise preference collection, rater instructions, reward models, PPO, DPO, position bias, and limits of preference labels.
Otázka 1 z 8
Jaký druh štítku zaznamenává fáze sběru preferencí InstructGPT?
Náhled každé otázky8 otázky
Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.
- Jaký druh štítku zaznamenává fáze sběru preferencí InstructGPT?
- Which of two candidate responses a rater prefers under task instructions
- A universal quality score for all possible responses
- The model’s next-token probabilities only
- A transcription of the rater’s own ideal response
- Co se obvykle stane, když kontroly shody odhalí trvale nízkou shodu mezi hodnotiteli u určitých promptů?
- The raters are immediately fired
- The labeling guidelines are usually revised, since the disagreement often points to ambiguity in the instructions
- The prompts are deleted from the dataset with no further action
- The reward model is retrained without any human data
- Co se v tradiční pipeline RLHF učí model odměn dělat?
- Generate the original candidate responses
- Translate responses into other languages
- Assign a scalar score to any response, trained so the human-chosen response in each pair scores higher than the rejected one
- Replace the need for a language model entirely
- Jak se přímá optimalizace preferencí (DPO) liší od tradičního přístupu modelu odměn plus PPO?
- DPO requires no preference data at all
- DPO only works for image models, not language models
- DPO eliminates the need for human raters entirely
- DPO fine-tunes the language model directly on preference pairs, skipping the separate reward model training step
- Proč rozhraní hodnotitelů obvykle náhodně přiřazují, zda se odpověď zobrazí jako „A“ nebo „B“?
- To prevent raters from developing a positional bias toward always picking the first option shown
- To make the interface look more colorful
- Because response A is always the correct answer
- To speed up model inference time
- In the classic RLHF sequence described in the guide, what does PPO update?
- The preference rubric alone
- The policy model, using the reward signal while constraining drift from a reference
- The raw annotation interface
- The human raters’ demographic labels
- Why is it a misconception that RLHF preference data reflects one single universal notion of a good response?
- Because preference data is generated entirely by the model with no human involvement
- Because RLHF has never been used by any real company
- Because the data actually reflects the specific guidelines and rater pool chosen, which can shift model behavior in different directions
- Because reward models are incapable of learning from human data
- Why might comparing two model responses become harder for human raters as underlying models improve?
- Because better models produce identical responses every time
- Because rater training becomes unnecessary once models improve
- Because improved models no longer require any preference data
- Because both candidate responses may already be fluent and largely correct, making meaningful differences subtler to detect