Chatbot Grounding: Answer From Your Documents, Not the Web

The answer that cost a refund

A small dental practice put a chat widget on its site to handle the questions its receptionist answered forty times a day. It worked well for nine weeks. Then a prospective patient asked whether a particular treatment was covered, and the bot said yes, confidently, with a friendly paragraph about how most plans include it.

It was not covered. The patient booked, arrived, and was told the cost at the desk. The practice waived the fee to avoid a scene, then spent a fortnight wondering why the bot had lied.

The bot had not lied. It had improvised. It had been given the practice website, which is marketing copy, and asked questions whose answers live in a fee schedule, an insurance policy and a treatment protocol. None of those had ever been in front of it.

That is the whole problem with most small business chatbots, and it is a retrieval problem far more often than a model problem.

What grounding actually means

Grounding, or retrieval augmented generation, is a two step process that gets described in one breath and then skipped.

Retrieval: your question is matched against a searchable index of your own documents, and the three to five most relevant passages are pulled out.

Generation: the language model writes an answer using those passages, and only those passages, with an instruction to say it does not know when they do not contain the answer.

Everything that makes a grounded bot trustworthy happens in the first step. Everything that makes it untrustworthy is usually blamed on the second.

Where the common advice is wrong

A newer, bigger model will fix wrong answers. No model knows your fee schedule, your cancellation window, or which of your three price lists is current. Swapping models changes the tone of an invented answer, not whether the answer is invented.

Upload your website and you are done. Websites are written to persuade. Policy documents are written to bind. A bot grounded only in marketing copy will answer policy questions by inventing a plausible policy.

Grounding means the content never needs maintaining. A grounded bot is exactly as current as its documents. When the fee schedule changes and the indexed copy does not, the bot answers yesterday with total confidence. This is the most common failure I see after month three.

Give it everything, it will find what matters. Long context is not a substitute for good retrieval. A stack of unfiltered documents dilutes relevance, and relevance is the entire mechanism.

How the retrieval layer is built, in plain terms

Chunking. Documents are split into passages of roughly 200 to 500 words with some overlap, because the model receives passages, not files. Too large and the relevant sentence is buried. Too small and a condition loses the sentence it depends on.

Embeddings and keywords together. Passages are indexed by meaning and by exact term. You need both. Meaning search understands that a patient asking about a crown wants restorative work. Keyword search is what finds SKU 4471 or clause 8.2, which meaning search will happily smear into something nearby.

Metadata. Every chunk carries its source, its page or section, its effective date, and the location it applies to. This is what lets the bot answer for the right branch and decline when the only document it found expired last year.

A relevance threshold. Below a score, the bot does not answer at all. It says it does not have that information and offers a route to a person. A refusal is a feature, and it is the one setting most projects leave at the default.

Visible citations. Show the reader which document the answer came from. It builds trust, and it is the fastest way for your own team to spot a bad chunk.

An unanswered question log. Every refusal is a content gap with a customer attached. This log becomes your writing list.

The order to do this in

1. Inventory what you have. Fee schedule, policies, service descriptions, opening hours, warranty terms, delivery rules. One document per topic, in one place.
2. Choose a single source of truth for prices. If two documents disagree, the bot will find the wrong one first.
3. Clean before indexing. Remove out of date pages, merge duplicates, date every document. Scanned PDFs need text extraction first, or retrieval silently returns nothing.
4. Write thirty test questions, half of them questions the bot should refuse. Ten of those should be deliberately unfair.
5. Set the refusal behaviour and confirm it with the test set before anyone outside sees the bot.
6. Launch on the lowest risk intents. Hours, services, where to park, how to reschedule. Not coverage, not pricing.
7. Review transcripts weekly for a month, then monthly. Every wrong answer is either a missing document or a retrieval miss, and the two have different fixes.
8. Put document updates on a calendar. If the price changes, the index changes the same day.

What to say no to

• A bot with write access to your booking system or CRM on day one. Read only until you have watched it answer a few hundred questions.
• Scanned contracts as the knowledge base. Without text extraction you have built an expensive random answer generator.
• Promising refunds, discounts or coverage. The bot should state what the document says and hand anything transactional to a person.
• Auto generated answers on a topic nobody owns. If no human is accountable for the correct answer, the bot cannot be either.
• A knowledge base copied from the vendor demo. Generic content produces generic answers and hides the gaps you needed to find.

Key takeaways

• Wrong answers are usually a retrieval problem, not a model problem. Check what the bot was actually given.
• Marketing copy is not policy. Index the documents that bind you, not the ones that sell you.
• Hybrid search, metadata and a relevance threshold do most of the work.
• A refusal with a route to a person is a feature. Log it, and write the missing page.
• Grounded bots go stale with their documents. Put updates on the same calendar as the fee change.

Where to start

Take your last fifty chat transcripts, find the ten questions the bot answered incorrectly, and check whether the correct answer exists in a document you gave it. If it does, you have a retrieval problem and a fortnight of tuning. If it does not, you have a writing problem, and no amount of model upgrading will solve it.

A free digital presence audit on Webcheckhub examines website health and conversion readiness, which shows where a chat layer would actually earn its place. Our playbook for what to do after a chatbot gives a confidently wrong answer covers the recovery side, and building an AI agent knowledge base covers the content side in more depth.

When you want the knowledge base structured and the retrieval layer tuned properly, Solutions Elites builds grounded assistants for small businesses.