引導式測驗 · 中等 等級
提示版本控制和回歸測試測驗
測試版本化提示配置中的內容、回歸套件如何處理不確定性,以及為什麼模型快照和 LLM 法官需要固定。
問題 1 的 8
哪一組項目應作為一個提示配置一起進行版本控制?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 哪一組項目應作為一個提示配置一起進行版本控制?
- Only the prompt wording
- 僅型號名稱
- Prompt template, model identifier, sampling parameters and tool schemas
- Only the user inputs from production
- 為什麼要固定過時的模型快照而不是通用的最新別名?
- So model changes happen only when you deliberately switch and re-test
- Because aliases are always slower
- Because snapshots are free
- Because aliases do not support system prompts
- 指南对温度 0 有何说明?
- It guarantees identical outputs every time
- It makes regression testing unnecessary
- It only affects the length of outputs
- It reduces randomness but does not guarantee identical outputs across runs
- 回归套件应如何处理不确定性输出?
- Require exact string matches
- Use thresholds and repeated samples instead of exact equality
- Ignore any failing cases
- Only test with a single example
- 黃金組新測試案例的強大來源是什麼?
- Past production failures turned into test cases
- Random text from the web
- Only cases the current prompt already passes
- Outputs copied from a competitor
- If an LLM judge grades outputs, what should you do?
- Let the judge model change freely
- Trust every verdict without review
- Pin the judge's model and prompt and check it against human labels
- Use the same prompt under test as the judge
- Why log the prompt version ID with each production request?
- To reduce token costs
- To improve model accuracy
- To satisfy the model provider's terms
- To tie problems to a specific deploy and roll back quickly
- Why list individual cases that flipped from pass to fail rather than only the average score?
- Averages are always wrong
- An unchanged average can hide new breakage offset by gains elsewhere
- Flipped cases reduce API costs
- Only flipped cases are graded