A voice agent handles the demonstration perfectly. A week later, someone adds a new service, updates the instructions, or changes a tool. The new request works, but an existing booking flow now ends without a confirmed appointment.
That is a regression: a change breaks behavior that previously worked. Agencies maintaining voice AI need a repeatable way to notice it before the client’s customers do.
On October 8, LiveKit announced general availability of Agent Simulations for LiveKit Cloud projects. The feature uses simulated callers and an automated judge to exercise conversations. For delivery teams, its value is the ability to revisit the same business expectations after each change.
Turn the client’s promise into a testable outcome
Begin with the service the client bought. If the agent schedules consultations, a useful test asks whether the correct consultation exists in the booking system. “The agent sounded helpful” is not enough.
Write the initial conditions, caller goal, approved boundaries, and expected end state. Use fictional information. For example, a caller requests an unavailable afternoon appointment, declines the alternatives, and asks for a follow-up. The expected outcome could be one assigned follow-up request and no appointment confirmation.
This scenario is our proposed example, not a vendor result. Its usefulness comes from the distinction between the desired booking and the outcome the business can actually support.
| Scenario | Expected evidence | Failure to catch |
|---|---|---|
| Requested slot is unavailable | Only available alternatives are offered. | Invented availability. |
| Caller corrects a detail | The final record contains the corrected value. | Stale information carried forward. |
| Scheduling system fails | An honest fallback is offered without a booking claim. | False confirmation. |
| Caller repeats a request | No duplicate booking or follow-up record. | Duplicate action. |
| Caller requests an unsupported exception | The agreed boundary and escalation route are followed. | An unauthorized promise. |
Use text runs and audio runs for different questions
LiveKit’s announcement distinguishes text simulations, which bypass speech models to exercise conversation logic, from audio simulations, which run through the audio pipeline. An LLM-driven caller pursues a scenario, and an LLM judge evaluates the resulting conversation against expectations.
Our suggested release process uses quick text checks during editing and selected audio checks before a release. A logic-only pass cannot tell you whether a spoken correction was heard accurately or whether the agent talked over the caller.
Use real calls through the intended phone route as a further check where telephony matters. Audio simulation is useful evidence, but it is not by itself a validation of your carrier routing, destination phone queue, or the whole customer journey outside the simulated environment.
Keep the scenario suite separate from the latest prompt
If the tests are rewritten every time the agent changes, a failure can disappear simply because the expectation moved. Keep a stable core of business outcomes, then add cases for new behavior and previously discovered problems.
LiveKit documents a versioned scenarios.yaml workflow. Its current simulation documentation explains that generated scenarios can be created from agent source; that generation uploads code to LiveKit Cloud and asks for confirmation. Review the code-sharing implications before using generation on a client project.
A generated suite can help discover paths. It still needs someone who understands the business to review what success means. The same code that omits an important rule may be a weak starting point for discovering that missing requirement.
A transcript verdict is not the final business record
LiveKit’s documentation explicitly supports adding final-state checks alongside the simulated judge’s verdict. It also describes using simulation context and controlled tool behavior to seed repeatable scenarios. These capabilities matter because a conversation can describe a correct action while leaving the wrong record.
For your implementation, use isolated test data and a test backend or appropriate tool mocks. Confirm those boundaries before running a suite that can call real tools. Do not let a simulated caller book a real customer slot or trigger an actual notification unintentionally.
Check the resulting state directly: which record exists, what changed, and whether an unwanted duplicate was created. Keep that evidence with the conversation and the version being reviewed. A screenshot of a green verdict alone is a weak handover artifact.
Illustrative plan · not measured results
A 23-check release plan
One proposed workload for the five-scenario starter suite above.
Budget for evidence, not an assumed free test
The October 8 announcement says the simulation service is free through October, while normal language-model and, in audio mode, speech-model charges still apply. Treat that as a dated provider statement, not a permanent pricing promise. Confirm current terms before estimating ongoing usage.
For planning, count scenarios, repeats, approximate conversation length, and the models exercised. Use a clearly labelled forecast if you model cost. This article does not claim a universal test-suite price or return on investment.
Repeat important scenarios when behavior is inconsistent. An automated caller and judge can vary, and a single successful attempt may hide that variation. Investigate whether the agent is inconsistent, the expectation is ambiguous, or the test setup differs before changing a release decision.
What the agency should hand over after a change
- The agent and tool version reviewed, with a plain-language description of the change.
- The scenario set, expected outcomes, and test environment.
- Separate text, audio, and phone-route evidence where each was used.
- Failures, unresolved limitations, and the owner of each follow-up.
- The release decision and a defined recovery or rollback route.
This adds depth to the launch planning in our Voice AI pilot checklist. The next useful step is to choose one existing workflow, define five failure-sensitive scenarios, and run them before and after the next proposed update. Sell the maintenance process with clear evidence of what was checked.
Sources
- LiveKit: Testing voice agents with Agent Simulations, October 8, 2026. Announcement, text/audio distinction, and dated introductory pricing statement.
- LiveKit: Agent Simulations documentation, reviewed October 20, 2026. Scenario workflow, code-upload notice, simulated state, and final-state checks.
Editorial note: Researched October 20, 2026. Vendor announcements and documentation establish what their providers describe, not independently verified customer performance. Business Meets AI did not conduct product testing for this article. Implementation recommendations and fictional examples are editorial analysis, not measured results. Images are AI-generated editorial illustrations or fictional scenes. See our editorial standards and publisher disclosures.