Customer replies
How to stop AI inventing answers to your customers
Ask a general AI tool what your refund policy is and it will tell you. Fluently. It has never seen your refund policy. What it gives you is a plausible one — assembled from thousands of other businesses, formatted like yours would be, and wrong in ways you cannot spot unless you already know the answer.
The problem is not that it lies
A model does not know the difference between recalling something and constructing something. Both feel identical from the inside, and both come out equally confident. There is no hesitation in the output to warn you.
So if a customer asks whether you deliver to Scotland and your AI has never been told, it will not say "I don't know". It will produce a sentence that sounds exactly like the answer, because sentences that sound like answers are what it was built to produce.
If that reaches the customer, you have made a commitment you never made. A delivery date. A price. A refund you do not offer.
The fix is not a better prompt
People try to solve this by asking the model to be accurate, or to only say true things, or not to make things up. None of it works reliably, because the model has no way to tell which of its outputs are made up.
What works is giving it your actual answers and forbidding it from going beyond them. The technique is called grounding, and it is the difference between a support assistant that is useful and one that is a liability.
Step one: write fifteen answers
This is the work, and no automation replaces it. A spreadsheet, two columns: the question as customers actually ask it, and your answer in your words.
Start with fifteen. You already know them — they are the ones you have answered so often you are bored of them. Do you deliver to X. How long does it take. Can I change my order. What if it does not fit.
Write them as you would actually type them to a customer, not as policy prose. The model matches the register of whatever you give it, so formal source material produces stiff, corporate-sounding drafts.
An hour spent here is worth more than any amount of prompt tuning. The same fifteen answers become your website FAQ and your own copy-paste replies — one source, three uses, no drift between them.
Step two: the grounded prompt
Hand over the whole sheet alongside the question. Fifteen short answers is a small amount of text — do not over-engineer this until it is large enough to be a problem.
Then the instruction that does the actual work:
- Below are the only facts you may use about this business. Everything else is unknown to you.
- [your approved answers]
- Customer question: [the message]
- Write a reply using ONLY the information above.
- If the question is not covered by that information, do not answer it. Reply with exactly: NOT COVERED — followed by a one-line summary of what they asked.
- Do not combine two answers to infer a third. Do not estimate. Do not say what is 'usually' or 'typically' the case. If a price or a date is not written above, you do not know it.
That last paragraph matters more than it looks. Inference is the failure mode nobody anticipates. A model told that delivery takes three days, and that you are closed at weekends, will happily calculate a Monday arrival — which sounds reasonable, follows logically, and may be completely wrong.
The one test that proves it works
Ask it something you have definitely not written down.
That is the whole test. Does it say NOT COVERED, or does it improvise? Everything else about your setup can be right and this can still be broken.
Three more worth running:
- A question you HAVE answered, worded differently from your sheet. Does it still match, or does it decline something it should know?
- A question that invites inference. Combine two facts from your sheet and ask for the conclusion. It should refuse.
- Two questions in one message, one covered and one not. Most setups answer the first and quietly ignore the second — which is worse than declining both, because nobody notices.
What usually goes wrong
It answers everything, confidently
Grounding is not holding. Usually there is a competing instruction elsewhere in the prompt, or the refusal rule is buried in the middle instead of at the end.
It declines things you clearly answered
The question is worded very differently from your sheet. Add the alternative phrasings customers actually use — your gap list will show you them.
The drafts sound stiff
Your approved answers are written as policy. Rewrite two of them the way you would say them out loud and compare the difference.
Nobody maintains the sheet
This is the slow failure, and the one that actually kills these systems. The gap list grows, nobody writes the new answers, and the assistant declines more every month until people stop trusting it. Ten minutes a month keeps it alive.
The part that surprises people
Every NOT COVERED goes on a list. That list turns out to be the most valuable thing this automation produces — more valuable than the drafted replies.
It is your customers telling you, in their own words, exactly which answer you should write next. The first month of it will teach you more about what people actually want to know than a survey would.
One line to keep
A support assistant that says "I'll check that for you" is useful. One that invents a delivery date is a liability — and the difference between them is not the model you chose or how clever your prompt is.
It is whether you told it what it is allowed to say.