Quiz collegato alla guida · Difficile Livello
Collecting Human Preference Data for RLHF Quiz
Understand pairwise preference collection, rater instructions, reward models, PPO, DPO, position bias, and limits of preference labels.
Domanda 1 di 8
Che tipo di etichetta registra la fase di raccolta delle preferenze di InstructGPT?
Anteprima di ogni domanda8 domande
Le risposte e le spiegazioni vengono visualizzate mentre rispondi al quiz.
- Che tipo di etichetta registra la fase di raccolta delle preferenze di InstructGPT?
- Which of two candidate responses a rater prefers under task instructions
- A universal quality score for all possible responses
- The model’s next-token probabilities only
- A transcription of the rater’s own ideal response
- Cosa succede tipicamente quando i controlli di accordo rivelano un persistente basso accordo tra i valutatori su certi prompt?
- The raters are immediately fired
- The labeling guidelines are usually revised, since the disagreement often points to ambiguity in the instructions
- The prompts are deleted from the dataset with no further action
- The reward model is retrained without any human data
- Nella pipeline tradizionale RLHF, a cosa è addestrato un modello di ricompensa?
- Generate the original candidate responses
- Translate responses into other languages
- Assign a scalar score to any response, trained so the human-chosen response in each pair scores higher than the rejected one
- Replace the need for a language model entirely
- In che modo l'ottimizzazione diretta delle preferenze (DPO) differisce dall'approccio tradizionale modello‑di‑ricompensa‑più‑PPO?
- DPO requires no preference data at all
- DPO only works for image models, not language models
- DPO eliminates the need for human raters entirely
- DPO fine-tunes the language model directly on preference pairs, skipping the separate reward model training step
- Perché le interfacce di valutazione di solito randomizzano se una risposta appare come 'A' o 'B'?
- To prevent raters from developing a positional bias toward always picking the first option shown
- To make the interface look more colorful
- Because response A is always the correct answer
- To speed up model inference time
- In the classic RLHF sequence described in the guide, what does PPO update?
- The preference rubric alone
- The policy model, using the reward signal while constraining drift from a reference
- The raw annotation interface
- The human raters’ demographic labels
- Why is it a misconception that RLHF preference data reflects one single universal notion of a good response?
- Because preference data is generated entirely by the model with no human involvement
- Because RLHF has never been used by any real company
- Because the data actually reflects the specific guidelines and rater pool chosen, which can shift model behavior in different directions
- Because reward models are incapable of learning from human data
- Why might comparing two model responses become harder for human raters as underlying models improve?
- Because better models produce identical responses every time
- Because rater training becomes unnecessary once models improve
- Because improved models no longer require any preference data
- Because both candidate responses may already be fluent and largely correct, making meaningful differences subtler to detect