Simulated User Testing for Chatbots

Single-turn tests miss failures that emerge over a conversation: a chatbot may forget a constraint, misunderstand a correction or fail when a user repeats a request. In simulated-user testing, a second model or scripted agent generates user turns from a scenario, persona or goal while the chatbot responds. Review the transcript against specific criteria, such as whether the bot retained an earlier preference or completed a task safely. Simulation can support inexpensive exploration, repeatable regressions and tedious scenarios. Specify the goal, information available to the simulated user, what to reveal and when, and how much behavior may vary. Ground scenarios in observed product workflows instead of stereotypes. Use multiple simulator configurations and inspect transcripts; a judge model can help triage but its scores also need validation. Do not treat synthetic conversations as evidence of what real users will do or prefer. A 2026 study compared LLM-simulated users with participants on tau-Bench retail tasks and found simulator-dependent outcomes and subgroup calibration differences. Those results apply to that study’s tasks, but illustrate why simulation needs validation. Use it to generate hypotheses and find repeatable failures, then validate usability, accessibility, language variation and user priorities with representative people. Record the conversation state, simulator prompt, system version and criterion for each failure. Avoid demographic labels that stand in for stereotypes. For sensitive topics, use a sandbox and ensure no real external action occurs. Log the starting state, simulator model, persona instructions and transcript for each run. Avoid demographic stereotypes as shortcuts for behavior. For sensitive scenarios, isolate the system and prevent the simulated conversation from triggering real messages, purchases or account changes.