Mi történt
Researchers have introduced a hypothesis-driven simulation workflow designed to screen customer experience (CX) AI agents before they are deployed to production environments. By utilizing synthetic customers and simulated tool outputs, the framework allows developers to evaluate multi-step agentic workflows without the risks associated with live testing or the limitations of manual end-to-end evaluation. The study validated this approach using Nubank’s high-volume chat-support agents, demonstrating a strong correlation between simulated performance scores and actual production outcomes.
The proposed workflow uses the 'Snowglobe' simulator to create synthetic customer interactions. These synthetic users to agent responses, allowing the system to test multi-step workflows without needing to invoke actual production backends.
The researchers tested the framework across four versions of Nubank's chat-support agents. They found a high correlation between binary evaluator scores generated in the simulation and those observed in production.
In a live A/B test, the simulation-guided iteration resulted in a 36.69-point increase in transactional net promoter score (tNPS).
The team also used the simulator to screen open- model configurations across 16,000 conversations. The selected model achieved an 8.82 percentage point increase in self-service rate (SSR) in subsequent live testing, reaching the highest level recorded for the company.
Miért számít
This simulation framework addresses a critical bottleneck in AI deployment: the trade-off between rigorous testing and customer safety. In regulated industries, live A/B testing can expose users to failures that damage trust, while manual testing often lacks the necessary scale. By enabling the evaluation of over 16,000 simulated conversations, this method allows for the safe exploration of various model configurations, reasoning settings, and prompts. The practical impact is significant, as evidenced by the researchers' ability to increase transactional net promoter scores (tNPS) by 36.69 points and improve self-service rates (SSR) by 8.82 percentage points in live environments following simulation-guided iterations.
The framework provides a safer alternative to live experimentation, which is often the only way to verify agent performance at scale but carries the risk of eroding customer trust through errors.
By decoupling testing from production backends, organizations can iterate on prompts and model configurations much faster than traditional methods allow.
The ability to simulate complex, multi-step agentic workflows is particularly valuable for agents that must adhere to strict operational policies in regulated industries.
The documented improvements in tNPS and SSR suggest that simulation-based screening can lead to tangible business outcomes while minimizing the 'trial and error' phase of production deployment.
Interaktív mechanizmus: Hogyan működik valójában
Fedezze fel interaktívan a fejlesztés mögött meghúzódó technológiát.
An agent must create a draft calendar event for Tuesday at 2 p.m. Which evidence would establish the requested result?
Mit nézzünk ezután
The researchers demonstrated the framework's efficacy on Nubank's 'Card Delivery' and 'Card Management' agents. Future developments will likely focus on whether this simulation-first approach can be generalized to other domains beyond customer support, particularly in sectors with higher regulatory complexity or more unpredictable user interactions. Additionally, it remains unknown how well this specific simulation workflow scales to agents that require integration with highly dynamic or real-time external data sources that are difficult to replicate in a synthetic environment.
The study focuses on chat-support agents; it is unclear how the Snowglobe simulator would perform for agents with different modalities or those requiring more complex, non-conversational .
The researchers do not specify the cost or technical overhead required to maintain the simulation environment, which may be a barrier for smaller organizations.
The long-term reliability of synthetic customers in predicting human behavior as AI agents become more sophisticated remains an open question for the field.