Prompt Versioning and Regression Testing

Prompts contain logic: instructions, constraints, examples and output formats that determine what an application does. Treating them as casual text strings leads to untracked edits and surprises in production. Treating them like code means version control, review, automated tests and controlled releases. What needs versioning is more than the prompt wording. Output depends on the full configuration: the system prompt, the template and its variables, few-shot examples, tool or function schemas, the model identifier, sampling parameters such as temperature and maximum tokens, and retrieval settings if documents are injected. Changing any one can change behavior, so teams version them together, often identified by a content hash, and store them in git or a prompt registry. Regression testing needs an evaluation dataset, often called a golden set: representative real inputs, known edge cases, and past production failures turned into test cases. Graders range from strict to fuzzy: exact match, JSON schema validation, regular expressions, code-based checks, an LLM judge scoring against a rubric, and periodic human review of samples. The suite runs automatically on every change and compares results to the last accepted baseline. Model changes are as risky as prompt edits. Provider aliases such as a generic latest model name can point to new versions over time, and older snapshots get deprecated. Pinning a dated snapshot and re-running the suite before switching turns silent drift into a deliberate decision. Several misconceptions cause trouble. A few manual spot checks miss rare failures. Temperature 0 reduces randomness but does not guarantee identical outputs across runs. A higher LLM-judge score is not automatically better, because judges have biases such as favoring longer answers. Because outputs vary, good suites use thresholds and repeated samples rather than demanding exact string equality. Open-source tools such as promptfoo, along with commercial evaluation platforms, support these workflows, but a simple script in CI can also work.