Grounding AI: Why Your Chatbot Needs an Open-Book Exam
We’ve all seen it. A chatbot confidently invents a fact, makes a questionable judgment, or bungles a citation. Hallucinations are a fact of life with LLMs, despite efforts to engineer around them.
The main solution right now is grounding, which means requiring an AI to use verified, real-world sources rather than letting it rely on its internal training data. Without this, an LLM is essentially making a probabilistic guess based on patterns it ingested and derived during training.
Think of it like an open-book exam. Without an external source of truth, the model relies on memory. It might remember the right answer, or it might confuse it with a similar-sounding fact. By requiring the AI to pull from external materials, you shift the burden from memory to retrieval. It doesn’t need to "know" the answer from memory; it just needs to find it in the provided text.
However, this isn’t a magic switch, it simply lowers the error and hallucination rates. Even the most advanced systems still have a “last mile” issue: the AI can retrieve the right document but still misinterpret the content. This is why human judgement always needs to be the final arbiter of any workflow.
The Problem: Why AI Guesses
To understand grounding, you first have to understand why AI hallucinates. It’s not because the AI is “lying.” It’s because of how it’s trained.
OpenAI’s September 2025 paper touches on an uncomfortable truth: Most benchmarks are akin to a multiple-choice exam where both wrong answers and “I don’t know” score zero. If admitting uncertainty isn’t rewarded, guessing is always the superior option.
The “Confident Guesser” Bias
Current Large Language Models (LLMs) are probabilistic engines. They predict the next word in a sequence based on patterns learned from massive datasets. Crucially, their training and evaluation processes often reward confidence over accuracy. If an AI says “I don’t know,” it might receive a lower score in some training regimes than if it makes a plausible-sounding guess.
Research from OpenAI and subsequent studies in Nature suggest that the model is graded like a test-taker who is penalized for blanking out but rewarded for filling in the blanks with a convincing narrative. This creates a systematic bias toward confabulation; making things up to satisfy the prompt.
The Cost of Guessing
This isn’t just an academic quirk. It’s a real risk:
Legal: Sullivan & Cromwell, a prestigious Wall Street law firm, recently apologized to a federal judge for submitting a court filing with more than 35 AI-generated errors, including hallucinated citations.
Professional: Deloitte had to partially refund the Australian government’s AU $440,000 ($290,000) contract after their AI-generated report contained fabricated references and a made-up court quote.
Everyday: In our own research, an LLM cited a lone user review on a Google forum as an “official Google response” because the link tied to Google, not because the source was authoritative.
These aren’t glitches. They are inherent features of an AI trying to be helpful without grounding their responses with concrete external checks.
Grounding with Prompts and Documents
The first layer to a well-grounded pipeline is often just asking the right questions and providing the right materials.
1. Prompting Hygiene: Make the AI Cite
The simplest way to ground an LLM is to change how you ask it questions. Instead of asking for an answer, ask for evidence.
"Only Use The Attached Files": When you paste a document into a chat, explicitly tell the AI: "Based only on the provided text, answer the following question. If the answer is not in the text, state that you do not know."
Citation Requirements: Ask the AI to quote the source. "Provide your answer and include the page number or paragraph where you found this information." for provided documents. For web search, make it explicitly cite which part of the text it pulled along with the website it pulled from. Don’t let it summarize and research within the same task. This is a recipe for confabulation. Ask it to research first, then digest the research once you have the concrete source list.
Why this works: It forces the model to look backward at the provided context rather than forward into its training data. It creates a "paper trail" that allows you to verify the claim.
2. Web Search and Document Grounding
Continuing this thread, a common mistake is expecting LLMs to reliably read directly from the source. The AI might find a relevant-looking article, but it can struggle with noisy or low-volume searches. This is especially true for subjects where training data is thin. The more an LLM tries to approximate, the worse its judgement on a topic will be.
A. Web Search and LLMs
When you rely on an AI’s built-in web search, it is polling the slice of the internet that the cloud provider is able to access via their platform. The problem is twofold. First, the information available to it is more limited than a human using a browser. Second, AI can make mistakes on domain authority, or even miss crucial context with
The Fix: Always be skeptical of the output. We recommend the following:
Curate: Upload specific PDFs, internal memos, or verified industry reports. If you must use web search, restrict the AI to trusted domains whenever possible. (e.g., "Only search within nytimes.com and reuters.com").
Verify: Always assume the AI will prioritize relevance over authority. Just because a source appears at the top of a search result doesn’t mean it’s the most accurate or official one. Treat every retrieved source as a claim that needs verification, not a fact to be accepted.
B. Document Uploads
When you upload PDFs, manuals, or internal memos, you are creating a private, static knowledge base. This is more controlled but has different limits. Using good prompt hygiene and forcing citations are big here.
Prompt: Be as specific and clear as possible on which materials you want referenced.
Constrain: "Based only on the provided text, answer the following question. If the answer is not in the text, state that you do not know."
Citations: "Provide your answer and include the page number or paragraph where you found this information."
3. Static Documents: Your Private Knowledge Base
For recurring tasks, you can create a "Reference Document." This is a single file (PDF, DOCX, or TXT) containing:
Standard operating procedures (SOPs)
Brand voice guidelines
Specific technical specifications
How to use it: Instead of typing out instructions every time, upload this document and say: "Follow the steps in the attached SOP to draft this email."
The Limitation of Static Documents:
Static documents are "frozen in time." They cannot update themselves. If a policy changes, you must manually update the PDF. Furthermore, if your knowledge base is large (e.g., 500 PDFs), you can’t manually upload them all into a chat window. This is where we need more sophisticated systems.
Advanced Solutions: RAG and Beyond
When your knowledge base grows beyond a single PDF, or when you need real-time information, simple prompting isn’t enough. You need a system that can search and retrieve specific information before the AI answers.
1. RAG (Retrieval-Augmented Generation): The Open-Book Exam
RAG is the industry standard for grounding.
How it works: Instead of forcing the AI to memorize your entire company library, RAG allows the AI to "look up" the answer in a specific database before answering.
Retrieve: When you ask a question, the system searches your knowledge base for relevant documents.
Augment: It feeds those specific documents into the AI’s prompt.
Generate: The AI answers the question using only that retrieved context.
Why it matters: It solves the "scale" problem. You can now ground an AI in 500 PDFs, 10,000 emails, or a live database, and it will find the relevant pieces automatically. It also helps with live data: if your RAG system is connected to a live news feed or a real-time API, the AI can retrieve current events, bypassing its frozen training cutoff.
2. Knowledge Graphs: Connecting the Dots
While RAG retrieves text snippets, Knowledge Graphs retrieve relationships.
How it works: Instead of just storing facts as sentences, a Knowledge Graph maps out entities (people, companies, products) and the relationships between them (e.g., "Company A owns Company B," "CEO X works for Company Y").
Why it matters: This is crucial for complex, multi-step questions. If you ask, "Who is the CEO of the company that acquired our main competitor?", a simple text search might miss the connection. A Knowledge Graph can trace the path: Competitor → Acquired by → Company X → CEO → Person Y.
The Trade-off: Knowledge Graphs are harder to build. They require extracting structured data from unstructured text, but they offer a deeper, more logical form of grounding.
The Limitations: Managing Expectations
Even with the best grounding systems, AI is not a replacement for human verification. Grounding reduces the probability of error, but it does not eliminate it.
1. The "Last Mile"
Misinterpretation: The biggest misconception about grounding is that if the AI cites a source, the answer must be correct. This is false.
The Issue: The AI can retrieve the right document but still misinterpret the text. It might miss a crucial caveat, misread a numerical value, or conflate two similar sections.
The Reality: You are still working with a probabilistic engine. The LLM is "guessing" how to interpret the retrieved text, not just retrieving it. If the source text is ambiguous, the AI will likely guess wrong.
2. Verification is Still Human Work
You cannot automate trust. Even in a RAG system, the user must perform a "sanity check."
The Check: Don’t just read the AI’s answer. Click the citation. Read the actual paragraph. Did the AI quote it accurately? Did it leave out a key condition?
The Rule: If you are using AI for high-stakes decisions (legal, medical, financial), you must treat the AI as a junior researcher who needs supervision, not a senior expert who needs no oversight.
3. Hallucinations Will Persist
Grounding helps, but it is not a silver bullet.
The Stat: A study by Stanford’s Regulatory Laboratory tested premium legal AI tools from major providers (LexisNexis, Thomson Reuters) that were marketed as "hallucination-free" and grounded in authoritative legal databases. The result? These RAG-based tools still hallucinated 17–33% of the time.
The Takeaway: If the most sophisticated, expensive, and legally grounded AI systems in the world still fail 1 in 3 times, you cannot assume your simple PDF-based grounding is perfect.
4. The Base Model is Biased
Finally, remember that the AI’s underlying personality and biases are baked into its training data. Even if you ground it in a neutral, factual document, the tone of the response, the structure of the argument, and the implicit assumptions may still reflect the biases of the model’s training data. Grounding controls the facts, but it doesn’t always control the framing.
The Bottom Line
Grounding is not about making AI perfect. It’s about making AI verifiable.
Without Grounding: You are trusting the AI’s memory.
With Grounding: You are trusting the AI’s ability to find and cite evidence, which you can then verify by checking the sources yourself.
The goal isn’t to eliminate human effort; it’s to shift that effort from "checking if the answer makes sense" to "checking if the source supports the answer." That is a much more manageable and safer workflow.