Ikibazo gihuza ibibazo · Hagati Urwego
Kwihutisha Guhindura no Kwipimisha Ikizamini
Gerageza ikiri muburyo bwihuse bwiboneza, uburyo suite yo gusubira inyuma ikora nondeterminism, nimpamvu snapshots yicyitegererezo hamwe nabacamanza ba LLM bakeneye pinning.
Ikibazo 1 Bya 8
Ni ubuhe bwoko bw'ibintu bugomba guhindurwamo hamwe nk'ibikoresho bimwe byihuse?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Ni ubuhe bwoko bw'ibintu bugomba guhindurwamo hamwe nk'ibikoresho bimwe byihuse?
- Only the prompt wording
- Only the model name
- Prompt template, model identifier, sampling parameters and tool schemas
- Only the user inputs from production
- Kuberiki pin yerekana itariki yerekana ifoto aho kuba rusange alias?
- So model changes happen only when you deliberately switch and re-test
- Because aliases are always slower
- Because snapshots are free
- Because aliases do not support system prompts
- Ubuyobozi buvuga iki kubushyuhe 0?
- It guarantees identical outputs every time
- It makes regression testing unnecessary
- It only affects the length of outputs
- It reduces randomness but does not guarantee identical outputs across runs
- Nigute suite yo kwisubiraho igomba gufata ibisubizo bya nondeterministic?
- Require exact string matches
- Use thresholds and repeated samples instead of exact equality
- Ignore any failing cases
- Only test with a single example
- Niyihe soko ikomeye yimanza nshya zipimisha kumurongo wa zahabu?
- Past production failures turned into test cases
- Random text from the web
- Only cases the current prompt already passes
- Outputs copied from a competitor
- If an LLM judge grades outputs, what should you do?
- Let the judge model change freely
- Trust every verdict without review
- Pin the judge's model and prompt and check it against human labels
- Use the same prompt under test as the judge
- Why log the prompt version ID with each production request?
- To reduce token costs
- To improve model accuracy
- To satisfy the model provider's terms
- To tie problems to a specific deploy and roll back quickly
- Why list individual cases that flipped from pass to fail rather than only the average score?
- Averages are always wrong
- An unchanged average can hide new breakage offset by gains elsewhere
- Flipped cases reduce API costs
- Only flipped cases are graded