Scenario

The decision

A customer-facing agent is ready to go live and someone has to sign off on it, so what should that person ask to see, and who owns it once it's out there? A demo that went well doesn't tell you much on its own.

The problem it removes

Agents fail in the conversations nobody wrote down, like the customer who says the parcel never arrived and then gets impatient, or the one who asks for a person and then changes their mind. Scripted tests replay the same lines every time, so they only find the failures someone already imagined, and the rest shows up as refunds, escalations and the odd court case, as when an airline was held to a refund policy its chatbot made up.

What I ran

I pointed Scenario at a support agent I built for a fictional bookshop, once at an early version that passed all 13 of its own scripted tests and once at the current one, with five simulated customers and one attacker each run three times against each version, for 41 conversations in total.

The early version failed 6 of its 15 customer conversations, refunding the "my book never arrived" claim every time even though the carrier said delivered, once on the very first message, and none of that was in its scripted tests. The current version failed one.

The most useful failure was one I hadn't planned for: a customer asked for a person, and the agent asked one question first, solved the problem and refunded correctly, while my test assumed an immediate handover. Was the agent wrong? Nobody had decided.

Each conversation cost about $0.24, and most took under a minute.

Where it sits in the stack

Between the agent and the release. It talks to your agent the way a customer would, through a thin adapter of about twenty lines, so it tests the whole thing at once, prompts, tools, business rules and guardrails included, and it works with any model provider for the simulated customer and the judge and runs in CI like any other test suite.

The first 30 days

Week 1. Sit the support operations lead down with the AI team and list the ten conversations that would be most expensive to get wrong, like refunds, cancellations, complaints and requests for a person, and turn each one into a scenario with a plain-language description and three or four pass criteria.

Weeks 2 to 4. Add hard checks on what the agent actually did (refunds issued, tickets opened) next to the judge's verdict, run every scenario three times before each release against a pass threshold, and add one red-team scenario for the outcome that would make the news.

Skills. Someone who knows your customers and an engineer for the adapter, with no new platform needed.

Running cost. Ten scenarios with three repeats comes to about $7 a release at my prices.

Working at day 30 means there's a scenario report on the release checklist and support operations owns the scenario list.

Risk

Verdicts vary. The simulated customer words things differently each run, so one scenario passed twice and failed once on the same code, which is why you gate on three repeats and a pass threshold.

The judge gets it wrong both ways. It failed a correct refund because my criteria assumed a handover, and it also caught a failure that my own hard check passed because the check was vacuously true, so use both and read the transcript of every failure.

The attacker model matters. One model returned nothing as the attacker in two runs out of six while another ran every time, so pick the red-team model on purpose.

Maturity. It's seventeen months old with frequent releases, 194 open issues and two maintainers at the company behind it, which also sells a hosted platform, but the library is Apache-2.0 and runs fully without that platform, so pin the version and you're fine.

The organisational risk is that nobody owns the scenario list and it slowly stops describing your real customers.

Build, adopt or buy

Build. A simulated customer and a judge are a few prompts, but the months go into everything around them, from turn control and scripted steps to red-team strategies, voice adapters and a report someone is willing to sign off on, and rebuilding all of that gets you nothing.

Buy. Hosted evaluation platforms add dashboards, history across releases and a way for non-engineers to take part, which is worth it once several teams are shipping agents. The customer-service agent vendors also run their own simulations inside their products, so ask to see them before you sign.

Adopt the library as the release gate, and keep the scenario list in plain text so it outlives whichever tool runs it.

Verdict

I'd adopt it. It found the failure my early version shipped with in every single run, it raised a policy question I'd never answered, and it costs cents a conversation, so make support operations the owner of the scenario list, require three repeats and a threshold before release, and when a failure surprises you, take it to the business first, because it might be a policy question before it's a bug.