Guide-yakabatana mibvunzo · Pakati Nhanho

Yakananga Preference Optimization Quiz

Review DPO preference pairs, reference policy, implicit rewards, beta, variants, and limits of direct preference training.

Nzira dzekutungamira dzinoenderanaDirect Preference Optimization
Mubvunzo 1 ye 8

Chii chinobvisa DPO zvichienzaniswa neyechinyakare RLHF?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Chii chinobvisa DPO zvichienzaniswa neyechinyakare RLHF?
    • Iyo yakaparadzana mubairo modhi uye yekusimbisa yekudzidza loop
    • Kudiwa kwechero data yekuda kwevanhu
    • The base pretrained mutauro modhi
    • Kushandiswa kwe log-zvinogoneka
  2. Ndeipi data iyo imwe DPO yekudzidzisa muenzaniso ine?
    • Kukurumidza uye nhamba yemubairo zvibodzwa
    • Kurumidziro ine yakasarudzwa uye yakarambwa mhinduro
    • Mhinduro yekukurumidza uye imwe yechokwadi
    • Chete mubairo modhi inobuda
  3. Ndeipi basa rinoitwa nereferenzi modhi muDPO?
    • Inoburitsa mhinduro dzakarambwa
    • Inowana mhinduro nemibairo yakajeka
    • Ndiyo yekutanga yakaomeswa iyo mutemo unoyerwa nawo kudzikamisa kudonha
    • Inotsiva iyo data yekudzidziswa
  4. How does the DPO loss use a labeled preference pair?
    • It regresses both responses to the same scalar target
    • It trains a separate value critic to predict token-level returns
    • It maximizes the rejected response without a reference
    • It increases the chosen response’s policy-versus-reference log-ratio relative to the rejected response
  5. Chii chinonzi beta (tembiricha) paramende muDPO kutonga?
    • Iyo yakasimba sei iyo policy inochengetwa padyo nereferensi modhi
    • Chiyero chekudzidza che optimizer
    • Huwandu hwemapeya ekuda
    • Saizi yemazwi
  6. Which alignment objective can learn from binary feedback that an output is desirable or undesirable?
    • PPO, which requires a value critic for every label
    • KTO, a prospect-theoretic objective for binary desirability feedback
    • DARTS, which searches differentiable network cells
    • Beam search, which changes model weights from rankings
  7. In a standard DPO preference example, which response is preferred?
    • The response with more tokens by definition
    • The response with the lowest reference likelihood
    • The response selected by the preference labeler for the given prompt
    • A random response unrelated to the prompt
  8. In DPO, what expression serves as the implicit reward for a response, up to an input-dependent constant?
    • A separately trained scalar reward-model head
    • A human-written point score for each token
    • The average sequence length of preference data
    • Beta times the policy/reference log-probability ratio