Your customers are trying to escape your AI support bot. Here's how to build one they don't.

Most AI support bots fail the same way. They answer from the model's memory instead of from your own documents, and the first time a customer is told the wrong return window, the whole thing loses the room. This is how to build one that quotes your systems, hands over to a person cleanly, and gets checked every week.

Two things are true at once, and the gap between them is where most AI support projects die.

Business owners want this badly. A small-business owner on Reddit put it plainly: "between Instagram DMs, WhatsApp and emails I'm basically glued to my phone all day answering the exact same questions on repeat." A SaaS team in the same thread reckoned "60-70% of tickets are the same handful of questions." That is a real problem and AI genuinely solves part of it.

Their customers, meanwhile, are actively working out how to escape. The highest-voted thread we found on the topic is a tip for getting past support bots: swear at it, because the escalation rules assume angry customers need a human. Six hundred and seventy people upvoted that.

Both groups are right. The technology works. Most implementations of it do not.

The short answer: automate only the questions whose answer already exists in writing, make the bot answer from those documents and nothing else, have it say so and hand over when it finds nothing, and check a sample of real conversations against the source every week. Skip any one of those four and you get the transcripts people share.

Jump to:

• Which questions are safe to automate
• Work out how your bot should handle a question
• What to ask a vendor before building
• Common questions

Which support questions are safe to automate?

The ones whose correct answer already exists in writing. Start here, because it decides everything else.

Pull your last two hundred tickets and sort them. Almost every business finds the same shape: a small number of questions make up most of the volume. Where is my order. How do I reset my password. What is your return window. Do you service my postcode. How much does X cost.

These are the questions worth automating, and they share a useful property. Every one of them has a correct answer that already exists in writing somewhere in your business. That is not a coincidence. It is the test for whether a question is safe to automate at all, and it is the same test we use for automating our own admin.

The questions in the bottom two rows are where the screenshots come from. Pretending otherwise is what produces the transcripts people share.

What is the failure that actually costs you?

A confident wrong answer, in your brand's voice, that a customer then acts on. Not rudeness. Not a clumsy sentence. Someone running AI support for e-commerce sellers described exactly this:

Read what that costs. The customer now believes something false, because your website told them. When they try to return an item on day 60 they are not confused, they are angry, and they have a screenshot. You are now choosing between honouring a policy you never offered and arguing with someone who has your own bot in writing.

Someone working in support at a food delivery company put it more bluntly: "the hallucinations are getting out of hand."

This is a different class of problem from the ones support teams are used to. A human agent who does not know your return policy says so and asks. A language model does not have that instinct. It produces the most plausible-sounding answer, and a plausible return policy sounds exactly like a real one.

How do you stop an AI support bot making things up?

Make it answer from your documents, not from the model's memory. The fix is architectural rather than a matter of better prompting.

A general-purpose model answering from memory is guessing, however fluent it sounds. What you want instead is retrieval-augmented generation: before the model writes anything, the system searches your own material, retrieves the passages that actually address the question, and instructs the model to answer from those and nothing else.

The difference in practice is that the bot's answer about your return window comes from your returns page, because that page was retrieved and put in front of it. Change the page and the answer changes with it. Nobody retrains anything.

That is the whole argument, and it has three consequences worth being honest about.

Your documentation becomes the product. If your returns policy is ambiguous, the bot will be ambiguous. Most of the work in a project like this is not machine learning, it is finding that the real policy lives in three contradictory places and one person's head, and fixing that first. Teams are consistently surprised by how much of the effort lands here, and it is the part a web development agency building the thing should be warning you about in the first meeting.

It should refuse. A grounded system that finds nothing relevant needs to say it does not know and hand over. That feels like a failure and it is the opposite. A bot that answers everything is a bot that invents things.

Retrieval quality is the ceiling. If the search step returns the wrong passage, the model faithfully answers from the wrong passage. Grounding removes invention, not the need to test.

A common question is whether the bot should cite the docs or be allowed to search the internet. Cite the docs. The moment it can reach the open web, you have reintroduced everything grounding was there to prevent, and your support bot is quoting a competitor's blog post about your product.

Why is the route to a human a feature, not a failure?

Because customers are already better at finding it than you are at hiding it, and the attempt is what generates the contempt. Every instinct in a cost-saving project pushes toward burying the escape hatch. Resist it.

Consider what people report. One described forty-five minutes in a loop with a bot that kept telling them to restart their router, for a physical product that had arrived damaged. Another asked whether the bot "just resets and gets amnesia when you ask a different question." A third described a refund bot that offered a discount coupon instead of the refund, and reaching a person took a lot of back and forth. That thread drew 423 upvotes.

None of those are hallucination problems. They are design problems, and they share a cause: the system was built to contain the conversation rather than to resolve it.

Three rules we hold to:

• The handover is always one step away and always visible. Not buried in a menu, not gated behind three clarifying questions.
• The bot escalates on its own when it is out of depth. Low retrieval confidence, a repeated question, or visible frustration should all trigger a handover without the customer asking.
• The context travels. Nothing burns goodwill faster than being passed to a human and made to explain the whole thing again. The transcript goes with the customer.

Get this right and the bot stops being an obstacle and becomes triage, which is what it is good at.

How do you measure whether AI support is working?

Containment and correctness together, and never containment alone. Containment rate, the share of conversations resolved without a human, is the number every vendor leads with. On its own it is dangerous, because the cheapest way to raise it is to make escaping harder.

Track it against correctness, and treat the pair as the real result. Sample real transcripts every week and check the answers against the source documents. Not a spot check before launch. Every week, because your policies change and the retrieval that worked in March quietly stops matching in June.

The article on automating our own agency covers the check every one of our automations ships with: it fails loudly when a run produces nothing. In this kind of failure nothing crashes and everything reports success, so only a check that expects an answer will catch it.

What should you ask before building one?

Five questions worth more than any demo. Ask them of a vendor, or of your own team.

• Where does the answer come from? If the response is anything other than "your documents", stop there.
• What happens when it cannot find an answer? You want "it says so and hands over", not "it does its best".
• How does a customer reach a person? Time it yourself, and count the steps.
• What happens when a policy page changes? If the answer involves retraining, the maintenance cost is much higher than quoted.
• Can I read last week's transcripts? If not, you cannot measure correctness, only containment.

Specific answers to all five are rarer than they should be. They are the whole difference between the tool that saves your team an afternoon a day and the one that ends up as a screenshot in a thread about support bots going wrong, and they are the questions we would want a client to ask us about AI automation.

Common questions about AI customer service

Will it still be right in year two?

Build the support bot your team can still trust in year two, once the policies have changed twice, the person who set it up has moved on, and nobody has read a transcript since launch.

Every vendor demo shows the same thing: a question typed, a correct answer returned, a delighted customer. None of them show the Tuesday in month nine when the returns page was updated, the retrieval quietly stopped matching it, and the bot has been confidently quoting the old policy to every customer who asked.

That is the day worth planning for. Ground it, let it refuse, keep the door to a person open, and read the transcripts every week. Do that and the same five questions stop arriving in your inbox, which was the point.