Kvíz spojený s průvodcem · Těžko úroveň

Collecting Human Preference Data for RLHF Quiz

Understand pairwise preference collection, rater instructions, reward models, PPO, DPO, position bias, and limits of preference labels.

Otázka 1 z 8

Jaký druh štítku zaznamenává fáze sběru preferencí InstructGPT?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. Jaký druh štítku zaznamenává fáze sběru preferencí InstructGPT?
    • Which of two candidate responses a rater prefers under task instructions
    • A universal quality score for all possible responses
    • The model’s next-token probabilities only
    • A transcription of the rater’s own ideal response
  2. Co se obvykle stane, když kontroly shody odhalí trvale nízkou shodu mezi hodnotiteli u určitých promptů?
    • The raters are immediately fired
    • The labeling guidelines are usually revised, since the disagreement often points to ambiguity in the instructions
    • The prompts are deleted from the dataset with no further action
    • The reward model is retrained without any human data
  3. Co se v tradiční pipeline RLHF učí model odměn dělat?
    • Generate the original candidate responses
    • Translate responses into other languages
    • Assign a scalar score to any response, trained so the human-chosen response in each pair scores higher than the rejected one
    • Replace the need for a language model entirely
  4. Jak se přímá optimalizace preferencí (DPO) liší od tradičního přístupu modelu odměn plus PPO?
    • DPO requires no preference data at all
    • DPO only works for image models, not language models
    • DPO eliminates the need for human raters entirely
    • DPO fine-tunes the language model directly on preference pairs, skipping the separate reward model training step
  5. Proč rozhraní hodnotitelů obvykle náhodně přiřazují, zda se odpověď zobrazí jako „A“ nebo „B“?
    • To prevent raters from developing a positional bias toward always picking the first option shown
    • To make the interface look more colorful
    • Because response A is always the correct answer
    • To speed up model inference time
  6. In the classic RLHF sequence described in the guide, what does PPO update?
    • The preference rubric alone
    • The policy model, using the reward signal while constraining drift from a reference
    • The raw annotation interface
    • The human raters’ demographic labels
  7. Why is it a misconception that RLHF preference data reflects one single universal notion of a good response?
    • Because preference data is generated entirely by the model with no human involvement
    • Because RLHF has never been used by any real company
    • Because the data actually reflects the specific guidelines and rater pool chosen, which can shift model behavior in different directions
    • Because reward models are incapable of learning from human data
  8. Why might comparing two model responses become harder for human raters as underlying models improve?
    • Because better models produce identical responses every time
    • Because rater training becomes unnecessary once models improve
    • Because improved models no longer require any preference data
    • Because both candidate responses may already be fluent and largely correct, making meaningful differences subtler to detect