How to test an AI customer support chatbot before launch

Test an AI customer support chatbot with realistic questions and compare each answer against your approved business information before launch. Include questions it should answer, questions it cannot answer and requests that need a person, then check the whole conversation rather than only the first reply.

Write a small test sheet with a question, the approved source, the expected outcome, the actual reply and a pass or fail result. Use anonymised examples from your support queue or write synthetic questions that reflect your policies. Keep customer details, payment information and credentials out of the test sheet. The checklist below is a suggested manual process, not a claim that Chat Support supplies an automated evaluation tool.

Start with a documented question, such as 'What is your returns window?' Compare the reply with your current returns policy, including conditions and exceptions. Ask the same question in different words and follow it with a clarification. Pass it only when the answer stays within the source and makes sense in the conversation. A fluent answer is not evidence that the policy is correct.

Next, test missing and conflicting information. Ask about a delivery region absent from your policy or an exception your team has not approved. The chatbot should acknowledge the gap or route the request for review instead of inventing a promise. If two source pages disagree, correct the underlying information before repeating the test. Grounding helps the bot find information; your team still owns its accuracy.

Test the boundary between explaining a policy and taking an action. A general refund explanation does not prove a bot can issue a refund or access an order. In a safe test environment, ask for an account-specific change and check that the reply does not claim an action succeeded without a confirmed result. Use synthetic accounts and avoid triggering real customer notifications, payments or account changes.

Ask to speak to a person, then try a question your business reserves for human judgement. Check the customer-facing response, the queue your team uses and the conversation context available to the next teammate. If notifications are part of your setup, confirm delivery separately. Also test outside your staffed hours, when a handoff message must not promise an immediate response your team cannot provide.

Fix the source or workflow responsible for each failure and rerun both the failed question and the questions that passed earlier. Keep the test sheet so you can repeat it after policy or configuration changes. Intercom's batch-testing documentation describes using customer questions to check coverage and diagnose gaps before deployment. That is evidence for testing as a practice, not evidence that another product has the same testing features.

Choose launch criteria before you review the results. For example, require every policy answer in your chosen test set to match its approved source, every unsupported request to avoid an invented answer and every required handoff to reach the right queue. These are suggested criteria, not a measured accuracy rate or a guarantee about future conversations. Review live failures after launch and add new examples to the test sheet.

Compare Chat Support plans, create an agent with your business information and use the checklist to evaluate the setup you intend to run. If your workflow needs a specific integration or handoff arrangement, ask about it before relying on it. The related guides explain source grounding and human handoff in more detail.