A caller asks whether their plan covers water damage. The AI call agent says yes, in a calm, certain voice. It’s wrong. Nobody catches it until the claim is filed weeks later, against a policy that says otherwise.
Why an AI Call Agent’s Wrong Answers Are Worse Out Loud
On a screen, a wrong answer sits there. The user can scroll back, re-read it, click a source link, or search the sentence to check it. Text invites scrutiny by default: it persists, it’s inspectable, it waits. That’s the whole premise behind a text-first widget, and half the argument for moving callers into chat wherever the call reason allows it. A spoken answer from an AI call agent gets none of that. Said once, in a confident voice, the caller acts on it before anyone can question it. No scrollback on an AI phone call. No copy-paste. No “wait, let me re-read that.”
That asymmetry is the real problem with automated phone calling, and it isn’t a voice problem. Text-to-speech is solved: a modern AI caller can sound perfectly natural. What’s still unsolved is whether the thing being said is true. Provision an AI phone number yourself or buy an artificial intelligence phone number from a vendor, and the knowledge problem underneath stays identical. We cover the rest of what breaks in voice deployments elsewhere on the getagent.chat blog. This piece is only about the knowledge layer: why an AI call bot confidently states things that aren’t in its documentation, and what fixes that, not a better voice. Everything below applies just as directly to a text widget as it does to a phone line.
Three Different Sources of AI Call Agent Errors
”The AI got it wrong” does a lot of work in most post-mortems, and it hides three separate failures that need three separate fixes. Take one customer question and watch it break three different ways.
A caller asks: “Do you cover water damage on the plan with the higher deductible?”
Mis-transcription. Speech-to-text hears “the plan with the higher deductible” as “the plan with the higher deducted bill” — a phrase that means nothing. The system searches for that garbled string, finds nothing close, and defaults to a generic answer about the base plan. The knowledge base was fine. The question that reached it wasn’t. Our piece on what actually breaks in production calls goes deep on transcription failure rates and why they’re bigger than most teams expect.
Retrieval miss. The transcription is clean this time. The system searches for “water damage higher deductible plan” and comes back with the base plan’s water-damage clause, because that’s the closer semantic match — the higher-deductible plan’s exception is buried in a different document and never reaches the top of the list. The right answer exists. Retrieval never found it, so the model never saw it.
Generation hallucination. Retrieval works this time too. The correct clause (the higher-deductible plan excludes water damage) sits right there in context. The model answers yes anyway, because “covers water damage” is a more common completion for insurance questions than a careful no. The right text was in front of it. It said something else.
Same caller, same question, three unrelated root causes. Most teams diagnose all three as “the AI is dumb” and respond by rewriting the system prompt — the one lever that fixes none of them. A transcription problem needs better audio handling and query rewriting. A retrieval problem needs better search. A generation problem needs a stricter grounding instruction and proof the model is being fed the right text.
Why Naive Vector Search Breaks on Spoken Queries
Vector search assumes the query is a well-formed sentence that says what it means. A typed search mostly meets that bar. A spoken utterance mid-call usually doesn’t.
Real call transcripts are short, elliptical, and full of pronouns with no clear referent outside the conversation. “What about the second one?” “Does that include tax?” “Can I switch to that instead?” None of those carry enough meaning on their own to embed into anything useful: the meaning lives in the two or three preceding turns, not the utterance itself. Embed “does that include tax” on its own and you get a vector sitting close to every FAQ entry that mentions tax, and far from the one thing the caller actually asked about.
Then the transcription layer poisons the query before search even runs. A product name the AI caller mishears (“ProGuard” transcribed as “pro guard”) doesn’t embed anywhere near the real product’s documentation. Search returns a confident top match that has nothing to do with what the caller said. Nobody sees an error; the system just answers based on the wrong retrieval — and the same mangled token survives into the stored transcript, which puts a floor under post-call analysis long before anyone reads the summary.
Put those two problems together, a short, referent-dependent utterance and a name mangled on the way in, and running top-3 vector search directly on a raw transcript is close to the worst input you could hand a retrieval system. Vector search isn’t bad technology; this is just a hard case for it.
Query Rewriting: Turning an Utterance Into a Search
The fix for the elliptical-query problem is a step most naive RAG builds skip: before anything gets embedded or searched, condense the conversation so far plus the latest turn into one standalone query, the sentence the caller would have typed if the whole exchange were one clean question.
The same technique applies whether the turn started as speech or as typed text. In AI Chat Agent’s pipeline this rewriting runs on every message, not just the first: it resolves the pronoun, folds in missing context, and preserves the user’s language rather than translating mid-conversation. A short worked example, using the water-damage call from earlier:
Turn 1: "Do you cover water damage?"
Turn 2: "What about the plan with the higher deductible?"
Rewritten search query:
"Does the higher-deductible plan cover water damage?"
The second turn alone would have embedded as a fragment about deductibles and matched nothing useful. Rewritten against the conversation, it becomes a query retrieval can actually work with.
The other half of this step is knowing when not to search at all. A caller who says “hi” or “thanks, that’s everything” doesn’t need a knowledge-base lookup: retrieving anything for a greeting adds noise and a real chance of surfacing an irrelevant chunk that then gets treated as relevant. The rewriting step detects that case and emits a sentinel that skips retrieval entirely. It sounds small, but it removes an entire category of spurious retrieval: the kind where the system answers a question nobody asked because it retrieved something for “OK, thank you” and felt obligated to use it.
Hybrid Retrieval: Dense Plus Lexical
Dense vector search is good at paraphrase — “cancel my subscription” and “how do I stop being billed” land close together in embedding space despite sharing almost no words. It’s bad at exact tokens. A SKU, an error code, a plan name like “Tier 3 Pro”: embeddings blur those into their semantic neighborhood instead of matching them exactly, which is backwards for a query where the caller read a code off a screen and needs the one document that mentions it.
Lexical full-text search is the mirror image: it nails exact tokens and misses paraphrase entirely. Search “stop being billed” against an index built for “cancel subscription” and a pure keyword match finds nothing, even though it’s obviously the same question.
Running both and fusing the results is the fix: Reciprocal Rank Fusion runs the dense arm and the lexical arm independently, each returning a wide candidate pool, then combines by rank position instead of trying to reconcile two incompatible scoring scales. A chunk ranked second by one arm and fifth by the other outranks a chunk that’s first in only one. RRF rewards results both methods agree on, without needing cosine similarity and a keyword score to live on the same axis. This retrieval layer runs as a single SQL query: a pgvector-cosine dense arm and a Postgres full-text arm, each pulling 40 candidates, fused by RRF. The mechanics of setting this up are covered in our RAG knowledge base guide.
The real limitation worth knowing: a language-agnostic lexical arm (one running Postgres’s ‘simple’ text-search configuration instead of a language-specific one) does no stemming. It matches “cancel” but not “cancelling” unless the exact form appears in the document. For English that’s a minor loss the dense arm recovers. For morphologically rich languages, where a single word carries case, number, and tense as inflections, the lexical arm contributes far less and the dense arm ends up doing most of the real work. See our multilingual chatbot guide for what changes across languages more broadly.
Reranking and the Relevance Gate
Hybrid retrieval gets you a wide, reasonably good candidate pool. It doesn’t get you a short, precise one: RRF is good at “these are plausible,” not “these actually answer the question.” The second step is a reranker that looks at the full pool against the real query and orders candidates by relevance rather than rank-fusion math. A well-tuned pipeline caps the reranker’s input at the top 16 fused candidates and trims to a final 6 that reach the model as context. The reranker runs on the bot’s own configured provider and model — no separate reranking vendor, no extra API key.
The part that matters most isn’t the trimming. It’s that the reranker can return a verdict of “none of these are actually relevant.” That verdict is the refusal signal — the thing that tells the system to say “I don’t have that information” instead of stretching a mediocre match into an answer.
Earlier versions of this pipeline used a fixed cosine-similarity threshold for the same job: below a set score, treat nothing as relevant. It’s intuitive, but weak in practice. Similarity scores aren’t calibrated across corpora, languages, or embedding models: a 0.72 cosine score means something different on a technical manual than on a two-sentence FAQ, and a threshold tuned for one collapses on the other. AI Chat Agent removed the fixed RAG_MIN_SCORE threshold in v1.8.0 and replaced it with the reranker’s relevance judgment, which generalizes across corpora far better than a single numeric cutoff. If the reranker itself times out, the system falls back to plain retrieval order instead of breaking the call. The accuracy gain is additive, not a single point of failure. If you’re evaluating retrieval-heavy platforms, ask specifically how each one handles the no-match case.
The Confabulation Trap: When the Bot Invents Facts About Its Own Knowledge Base
One failure mode deserves its own callout, because it’s stranger than a normal wrong answer and does more damage to trust once a customer notices it. The bot doesn’t get a fact wrong. It gets facts about itself wrong — telling someone “your file isn’t loaded” when it is, or “the knowledge base only has three fragments” when it has hundreds.
We shipped this bug. A deployed bot started confabulating about its own knowledge base, confidently describing what it did and didn’t contain based on nothing but the handful of chunks retrieved for the current question. The cause, once traced, was almost embarrassingly simple: the grounding prompt handed the model retrieved excerpts and never told it those excerpts were a sample, not the whole library. Left to fill that gap itself, the model treated six chunks as the complete contents of the knowledge base and answered questions about its own scope accordingly.
The fix, shipped in v1.8.0, is a framing change more than an architecture change. Context handed to the model is now explicit: these are excerpts retrieved for this specific question, not the complete knowledge base, and the model is instructed not to speculate about what else the base does or doesn’t contain. Context blocks are formatted as plain markdown — ## Source: <name> (Chunk N) — so the model has a structural cue that it’s looking at labeled fragments, not a table of contents. Small instruction. It closed a failure mode that, from the outside, looked like the AI had opinions about its own storage, an unsettling thing for a customer to read mid-conversation.
Neighbour Context and Chunk Granularity
Retrieval doesn’t return documents. It returns fragments, sized for embedding quality, not readability. That’s fine for finding the right needle. It’s a problem when the needle turns out to be the middle of a sentence, and the clause that actually answers the question sits in the chunk right before or after the one that got retrieved.
A caller asks about a refund exception, retrieval correctly finds the paragraph describing it, and the chunk boundary cuts it off exactly where the actual condition (“unless the item was opened”) would appear. The model answers with the general rule and misses the exception, not because retrieval failed, but because the chunk it retrieved was truncated mid-thought.
Pulling the neighbouring chunk from the same source (one before, one after) fixes most of this cheaply. A well-built pipeline expands every retrieved chunk with its adjacent ±1 chunks from the same document, deduplicates overlap, and caps total context at roughly 8,000 characters to keep the prompt from ballooning. The chunking that produces these fragments matters too: markdown-aware chunking that respects heading and list boundaries keeps a numbered exception from splitting across two chunks, and language-aware chunking avoids cutting mid-word in non-whitespace scripts. None of this is exotic. It’s the unglamorous part of RAG that decides whether the sentence that finishes the thought is actually in front of the model when it answers.
Source Attribution When You Can’t Show a Citation
Text has an attribution mechanism voice doesn’t: a citation. A chat widget can render a link, a footnote, a highlighted excerpt next to the answer, so the user can check the claim without a follow-up question. A phone call has no equivalent — you can’t hand a caller a link mid-sentence.
What’s left is weaker, but real. An AI call agent can speak the source out loud — “according to our returns policy” — which at least names the document behind the claim. A transcript with sources can go out by SMS or email after the call, though by then the decision is already made. Or the system can recognize the question needs a citation and hand it off to a channel that can provide one.
That last option is where a text interface earns its keep for anything document-heavy. In AI Chat Agent’s admin panel, every conversation is stored with per-conversation detail and exportable to CSV (the actual transcript, not a lossy guess at what was said), and a chat answer can point straight at the excerpt it came from because the excerpt is on screen. See how a resolution-billed platform frames the same problem in our Intercom comparison. The pattern holds across the category: text is the channel built for showing its work. Voice never was, and no amount of prompt engineering changes that.
Measuring Knowledge Accuracy, Not CSAT
CSAT tells you the caller felt fine about the interaction. It doesn’t tell you whether the information was correct. A caller can rate a warm, confident, wrong answer five stars and only find out it was wrong weeks later. The complaint shows up as something else entirely, disconnected from the call that actually caused it.
The fix is a golden question set built from real call transcripts, not invented test cases: pull a sample of actual questions callers asked, including the messy, elliptical, mis-transcribed ones, because those are exactly what a synthetic test set tends to skip. Score retrieval and generation separately; a good score on one tells you nothing about the other.
| What to measure | What it tells you | What it misses if you skip it |
|---|---|---|
| Retrieval recall@k | Was the chunk that actually answers the question inside the top-k retrieved candidates? | A good answer built on a lucky retrieval that won’t repeat |
| Unsupported-claim rate | Did the answer assert something the retrieved context didn’t actually support? | Hallucination hiding behind a confident, well-formed sentence |
| Refusal accuracy | Did the system say “I don’t know” exactly when the knowledge base genuinely lacked the answer? | Silent over-confidence on out-of-scope questions |
Retrieval recall@k catches the case from earlier in this piece: the right chunk existed and never made the candidate pool. Unsupported-claim rate catches the case where retrieval worked and the model said something else anyway. Neither shows up in a CSAT score, and neither shows up in a deflection rate. Our guide to evaluating AI agents walks through building a set like this, including how to keep it current as the knowledge base changes.
When an AI Call Agent Should Hand the Query to Chat
None of this argues that voice is bad. It argues that some questions are structurally mismatched to a channel with no scrollback, no citation, and one shot at getting a spoken string of characters right.
Sort by query type and the line gets clear fast. Multi-step instructions — do X, then Y, then check Z — need re-reading, not memorizing after one pass of synthesized speech. Anything with an identifier, an order number, a policy number, an alphanumeric code, needs to be seen and confirmed character by character. Anything the caller needs to keep — a confirmation number, a policy detail they’ll reference next week — needs to persist after the call ends. Anything that needs a citation, per the point above, needs a channel where a citation is possible at all.
Voice keeps its advantage where none of that applies: short conversational intent, hands-busy situations, a fast yes or no without opening an app. Real, durable use case — just a different one than “answer a knowledge-heavy question accurately over a phone line.”
This is exactly where a text-first tool like AI Chat Agent fits: not as a replacement for an AI call agent, but as the better vehicle for the slice of contact volume that’s actually knowledge-heavy. Route the question, not the whole caller, and each channel does what it’s structurally suited for. If you’re weighing a voice-first product against a text-first one for this specific slice of volume, our Voiceflow comparison covers the tradeoff directly.
If your AI call agent — or your chat bot — gives confidently wrong answers because the knowledge layer underneath was never built for this, the fix isn’t a better voice or a longer system prompt. It’s query rewriting, hybrid retrieval, reranking with a real relevance gate, and knowledge grounding that stops the model from guessing about its own knowledge base. AI Chat Agent ships all of it out of the box. Try the pipeline yourself at the live demo, or get the full source for a one-time €79.
Frequently Asked Questions
What is the difference between a hallucination and a retrieval failure in an AI call agent?
A retrieval failure means the right answer existed in the knowledge base but search never surfaced it, so the model never saw it. A hallucination means the correct text was sitting in context and the model asserted something else anyway. They need different fixes: better search for the first, stricter grounding for the second.
Why do AI call agents misunderstand what customers say?
Two things break the question before search even runs. Speech-to-text garbles phrases and product names, so the query embeds nowhere near the real documentation, and spoken turns are short and full of pronouns whose meaning lives in the previous turns rather than the utterance itself. Rewriting every turn into a standalone question against the conversation history removes most of the damage.
Can vector search alone power an AI call agent knowledge base?
No. Dense vector search handles paraphrase well but blurs exact tokens like SKUs, error codes and plan names into their semantic neighbourhood. Pairing it with a lexical full-text arm and fusing both by rank position, using Reciprocal Rank Fusion, covers what each method misses on its own.
How do you know if an AI call agent is using its knowledge base correctly?
Not from CSAT, because a caller will happily rate a warm, confident, wrong answer five stars. Build a golden question set from real call transcripts, including the messy and mis-transcribed ones, then score retrieval and generation separately. The three numbers that matter are retrieval recall@k, unsupported-claim rate and refusal accuracy.
What should an AI call agent do when it does not have the answer?
Refuse plainly and say the information is not available, instead of stretching a mediocre match into an answer. The reliable trigger is a reranker that can return a verdict of none of these are relevant. That generalises far better than a fixed cosine-similarity threshold, since similarity scores are not calibrated across corpora, languages or embedding models.
Is AI chat or an AI phone agent better for complex questions?
Text wins on anything knowledge-heavy: multi-step instructions that need re-reading, identifiers that must be confirmed character by character, details the customer will need later, and answers that require a citation. Voice keeps its advantage for short conversational intent, hands-busy moments and a fast yes or no. Route the question by type rather than sending the whole caller down one channel.