Building an LLM Eval Dataset

Start with real usage. Logs, support tickets, search queries and user interviews show what people actually ask, including the messy phrasing that invented examples miss. Before launch, have domain experts write inputs. You can add synthetic cases generated by an LLM, as long as a person reviews them and removes unrealistic or duplicate items. Next, define what correct means for each case. Some tasks have one right answer, such as classification, extraction or a factual lookup. For these, store a golden answer that can be compared exactly or after light normalization. For open-ended tasks, write a rubric of specific, checkable criteria, such as 'mentions the refund deadline' or 'does not recommend a competitor'. Vague criteria like 'is helpful' lead to inconsistent grading, whether the grader is a person or a judge model. Then cover the range of real inputs on purpose. Tag cases by intent, difficulty, language and user type, and check that the important groups are represented. Add edge cases and adversarial inputs: - ambiguous questions - requests with missing information - very long inputs - out-of-scope requests that should be declined - prompt-injection attempts Keep a regression slice made of past failures. Size the set to the decision it has to support. A few dozen well-chosen cases can reveal large problems early. Detecting small differences reliably takes more. With 100 pass/fail cases and a pass rate near 80 percent, the margin of error is roughly plus or minus 8 percentage points at 95 percent confidence. A two-point improvement is indistinguishable from noise at that size. Report results for each slice, not only an overall average. A common mistake is to treat the dataset as finished. Products and users change, so refresh it from new logs. Keep a held-out portion that you never tune against, so your prompts do not overfit to the test.