Testing a conversational agent before it meets customers

Why it matters

Gartner found in 2026 that only 27% of customers would try a chatbot again after a negative experience.¹ A launch that goes wrong loses customers from the channel, and many of them will not come back to it.

The frame: two IMDA starter kits

IMDA’s Starter Kit for Testing LLM-Based Applications, version 1.0 from January 2026, sets out pre-deployment testing for baseline safety and reliability using benchmarks, red teaming and tests of outputs and components.² Several of its tests run in Project Moonshot, the open-source toolkit from the AI Verify Foundation.³

The Starter Kit for Testing Agentic Systems followed in October 2026. It describes a structured approach that starts with identifying the risks relevant to the agent and ends with continuous monitoring and re-testing, and it covers designing tests for erroneous or unauthorised actions without causing real-world damage.⁴⁵

Build the test set from real conversations

1. Intents. Sample real transcripts for each of your top intents, including the messy ones.
2. Languages. Match the share of each language and of mixed-language conversations to your traffic.
3. Policy questions. Refunds, fees and eligibility, where a wrong answer costs money.
4. Adversarial cases. Attempts to make the agent ignore its instructions or reveal other customers’ data.
5. Handoff cases. Customers who ask for a person, repeat themselves or are upset.

Test actions in a sandbox

Run every action the agent can take against test accounts and a copy of the systems it uses. Check that limits hold, that irreversible actions ask for confirmation and that every action is logged. This is the part of testing the agentic kit is most concerned with.⁴

Keep testing after launch

Re-run the full set whenever the model, the prompt, the knowledge base or the tools change. In production, track resolution, recontact within a set window, escalation rate and complaints, by intent and by language, and review a sample of conversations every week.