Útmutatóhoz kapcsolódó kvíz · Kemény Szint

Collecting Human Preference Data for RLHF Quiz

Understand pairwise preference collection, rater instructions, reward models, PPO, DPO, position bias, and limits of preference labels.

Kapcsolódó útmutató utakCollecting Human Preference Data For Rlhf
kérdés 1 -ból 8

What kind of label does the InstructGPT preference-collection stage record?

Tekintse meg az összes kérdést8 kérdéseket

A válaszok és magyarázatok megjelennek a kvíz kitöltésekor.

  1. What kind of label does the InstructGPT preference-collection stage record?
    • Which of two candidate responses a rater prefers under task instructions
    • A universal quality score for all possible responses
    • The model’s next-token probabilities only
    • A transcription of the rater’s own ideal response
  2. What typically happens when agreement checks reveal persistently low agreement between raters on certain prompts?
    • The raters are immediately fired
    • The labeling guidelines are usually revised, since the disagreement often points to ambiguity in the instructions
    • The prompts are deleted from the dataset with no further action
    • The reward model is retrained without any human data
  3. In the traditional RLHF pipeline, what is a reward model trained to do?
    • Generate the original candidate responses
    • Translate responses into other languages
    • Assign a scalar score to any response, trained so the human-chosen response in each pair scores higher than the rejected one
    • Replace the need for a language model entirely
  4. How does direct preference optimization (DPO) differ from the traditional reward-model-plus-PPO approach?
    • DPO requires no preference data at all
    • DPO only works for image models, not language models
    • DPO eliminates the need for human raters entirely
    • DPO fine-tunes the language model directly on preference pairs, skipping the separate reward model training step
  5. Why do rater interfaces typically randomize whether a response appears as 'A' or 'B'?
    • To prevent raters from developing a positional bias toward always picking the first option shown
    • To make the interface look more colorful
    • Because response A is always the correct answer
    • To speed up model inference time
  6. In the classic RLHF sequence described in the guide, what does PPO update?
    • The preference rubric alone
    • The policy model, using the reward signal while constraining drift from a reference
    • The raw annotation interface
    • The human raters’ demographic labels
  7. Why is it a misconception that RLHF preference data reflects one single universal notion of a good response?
    • Because preference data is generated entirely by the model with no human involvement
    • Because RLHF has never been used by any real company
    • Because the data actually reflects the specific guidelines and rater pool chosen, which can shift model behavior in different directions
    • Because reward models are incapable of learning from human data
  8. Why might comparing two model responses become harder for human raters as underlying models improve?
    • Because better models produce identical responses every time
    • Because rater training becomes unnecessary once models improve
    • Because improved models no longer require any preference data
    • Because both candidate responses may already be fluent and largely correct, making meaningful differences subtler to detect