Using AI to Analyze Phone Calls: The Accuracy Floor

Using AI to analyze phone calls sounds like a solved problem. Upload a recording, get a transcript, get a summary, get a QA score, feed it into your CRM. Vendors demo it flawlessly — clean audio, articulate speakers, a dashboard that lights up green. We build a chat widget for a living, not a call center tool, which is exactly why we can say this without a sales quota riding on the answer: the demo is real, but the accuracy floor underneath it is lower than almost anyone selling conversation intelligence will quote you.

This isn’t another post about whether your AI voice agent should route the call correctly or disclose itself properly — that ground is already covered elsewhere on this blog. This one is about what happens after the call ends: the transcript sitting in your call analysis platform, and whether the sentiment score, the QA rubric, and the extracted “next steps” built on top of it are actually true.

What AI Call Analysis Actually Does

Every AI call analysis product, from a $30/seat SaaS dashboard to an in-house pipeline built on open models, runs the same five stages: capture, transcribe, diarize, extract, score. Capture records the call and hands off raw audio. Transcribe converts audio to text with an automatic speech recognition (ASR) model. Diarize labels who said what — customer versus agent. Extract pulls structured facts out of the text: names, dates, amounts, objections, commitments. Score applies a rubric or sentiment model on top of the extracted facts to produce the number a manager actually looks at.

The part that gets glossed over in every pitch deck: each stage doesn’t just add its own error, it multiplies the error already baked in from the stage before. A word the ASR model got wrong doesn’t get corrected downstream — it gets diarized to the wrong speaker, extracted as the wrong entity, and scored with false confidence. By the time a QA dashboard tells a manager “this rep confirmed the refund amount,” that claim has passed through four independent points of failure, and none of them show their work. This is the accuracy floor: not a single error rate you can look up, but a compounding one that gets worse the further downstream you read.

It is also the reason two vendors can quote the same 90%+ transcription accuracy and still produce wildly different QA scores on the same recording — the number on the transcription slide describes stage two of five, and says nothing about what happens in the other four.

The Benchmark-to-Production Gap

Every ASR vendor leads with a benchmark number, and the benchmark number is real. Whisper Large-v3 scores roughly 2.7% word error rate (WER) on LibriSpeech, a clean, read-aloud audiobook corpus recorded in a studio. That number gets printed on slide one. What doesn’t get printed: Whisper’s reported WER on real-world English audio — call center recordings, accents, crosstalk, background noise — runs 8–12%, three to four times worse.

Deepgram, which sells transcription for a living, has been unusually candid about this in its own published ASR buyer’s guide: production accuracy degrades roughly 7.5×–16× from benchmark figures across use cases generally — and for phone calls specifically, about 2.8×–5.7×. That’s a vendor publishing a number that makes its own product look worse than the marketing page — which is exactly why it’s worth trusting. Deepgram’s own production reporting puts accuracy at roughly 92% on clean headset audio, dropping to about 78% in conference-room conditions, and down to roughly 65% on mobile audio with background noise — a call taken from a car, a warehouse floor, or a spotty cell connection.

We’ve written before about how this same acoustic degradation breaks live call handling in real time. Post-call analysis runs on the identical recordings, so it inherits the identical gap — it just has the luxury of not needing to be fast about being wrong.

Why Named Entities Break First

Aggregate word error rate is a bad proxy for what actually matters in a call transcript analysis, because errors aren’t distributed evenly across the text. Filler words, articles, and common verbs get transcribed correctly almost all the time — the language model backing the ASR has seen “the” and “okay” a billion times. Names, account numbers, SKUs, dollar amounts, and dates get transcribed correctly far less often, because they’re exactly the tokens an ASR model has the least statistical grip on: low-frequency, out-of-vocabulary, or acoustically ambiguous under noise — is that “fifteen” or “fifty”? “Sarah” or “Zara”?

A number spoken as a run of digits — an order ID, a phone number read back for confirmation — is arguably the single hardest thing for an ASR model to get exactly right, because there is no surrounding sentence context to disambiguate a mis-heard digit the way there is for an ordinary word.

Deepgram has argued publicly for measuring “semantic error rate” instead of WER for precisely this reason — a transcript can hit 95% word accuracy and still get the one number that matters, the refund amount or the appointment date, completely wrong. Research on Answer Error Rate — accuracy on the specific extracted answer rather than the surrounding words — has reported it running 10–30 percentage points worse than raw WER on entity extraction and sentiment tasks. In plain terms: if your call analysis tool reports 90% transcription accuracy, treat that as a description of the sentence, not a guarantee about the one fact inside it you actually needed.

Speaker Diarization and the Stereo Fix

Diarization — deciding which words belong to the customer and which belong to the agent — is a separate model with its own error rate, and it’s a worse one than most buyers expect. Speaker diarization error rates on spontaneous, overlapping conversational speech typically run 10–20%. NVIDIA has published figures showing roughly 13.24% diarization error rate on calls with fewer than five speakers, climbing to about 42.56% once a call involves more than five speakers — a conference call, a warm transfer, a supervisor joining mid-call.

Every diarization error is also an attribution error one layer up: a QA score that penalizes the agent for a sentence the customer actually said, or a commitment logged as customer-made when the agent said it. The practical fix is not a better diarization model — it’s not needing one. Capture the caller and the agent on separate stereo channels at the point of recording, and diarization becomes a solved problem by construction rather than an inference the system has to guess at afterward. Most call analysis vendors support dual-channel ingestion; most phone systems don’t turn it on by default. It’s the cheapest accuracy fix in the entire pipeline, and it happens before a single AI model runs.

How Errors Propagate Into LLM Analysis

Once a transcript exists, most modern call analysis tools hand it to an LLM for the interesting part: summarization, objection tagging, next-step extraction, coaching notes. This is where the compounding problem gets invisible, because LLMs are extremely good at producing confident, well-formatted prose regardless of whether the input underneath it was correct. An LLM asked to summarize “the customer confirmed the refund of fifteen dollars” when the actual utterance was “fifty dollars” doesn’t flag uncertainty — it has no way to know the number is wrong, because the transcript is the only reality it has access to.

This is fundamentally the same blind spot that shows up in ungrounded retrieval — the model produces fluent, plausible output whether or not the underlying material was accurate. In post-call analysis, there’s usually no source of truth to check against; the transcript is treated as ground truth by default, errors and all. The result is a summary, an action item, or a sentiment tag that reads as authoritative and carries none of the uncertainty it inherited three stages back. If that output then syncs onward into a deal record, the error stops being a transcript problem and becomes a data-quality problem inside your CRM, where nobody will ever trace it back to a mis-heard digit.

The AI Call QA Scores Nobody Validates

Here’s the honest gap in the entire category: there are no published studies correlating ASR word error rate with the accuracy of downstream sentiment analysis or AI call QA scoring. None of the major platforms — Gong, Chorus, Observe.AI — publish data showing how their WER affects the reliability of their own scorecards, coaching flags, or sentiment dashboards. That’s not a criticism of any one vendor’s engineering; it’s a statement about the state of the evidence for an entire product category.

The absence is the finding. You can buy a transcription engine on a published WER number. You cannot buy a QA scoring product on published evidence that its scores track human judgment, because that evidence isn’t published anywhere, by anyone, for any of the incumbent platforms. Buyers are left inferring reliability from demo calls and case studies — exactly the materials least likely to expose edge cases.

If a category can’t show you the correlation between input accuracy and output accuracy, the honest response isn’t to assume it’s fine; it’s to ask the vendor directly, in the sales call, whether that number exists internally at all, and watch how the conversation goes. A vendor with a genuine answer will have a validation methodology to point to; most will pivot to a customer logo instead.

What Call Transcript Analysis Genuinely Does Well

None of this means AI call analysis is worthless — it means it’s being sold for the wrong use case. Individual-call accuracy is where the accuracy floor bites hardest, because a single wrong entity or misattributed sentence is exactly the kind of error nobody catches. Aggregate analysis is a different problem entirely, and it’s one these systems are genuinely good at.

Run the same imperfect transcription across ten thousand calls, and the individual errors — some in each direction, on different words, in different calls — wash out at the population level the way measurement noise always does with enough samples. Topic volume across a month of calls, keyword flagging for compliance language, coaching trends across a rep’s call history, or emerging complaint patterns across a product line: these hold up even at a 15% WER, because you’re reading a trend, not a transcript.

The same logic applies to teams running AI voice calls for lead qualification at volume — no single call’s score should be trusted blindly, but the aggregate pattern of which scripts produce qualified leads is a real, usable signal. The failure mode is treating one call’s AI-generated score as if it carries the reliability of the aggregate trend it’s part of. It doesn’t, and no vendor’s dashboard makes that distinction visible.

The Consent and Retention Bill

Before any of the accuracy questions matter, there’s a legal one: are you allowed to record and analyze this call at all? In the United States, two-party (all-party) consent recording laws apply in California, Delaware, Florida, Illinois, Maryland, Massachusetts, Montana, Nevada, New Hampshire, Pennsylvania, and Washington, with Connecticut, Michigan, Oregon, and Vermont treated as contested or partial depending on the specifics. Laws change and vary by call direction and industry — this is a starting point for a conversation with counsel, not a substitute for one. We’ve covered what an AI caller is required to disclose before the recording even starts in more depth.

Once you’re allowed to record, you still own the retention decision, and it’s a bigger liability surface than most teams price in. Under GDPR, exposure for mishandled call data runs up to €20M or 4% of global annual revenue — whichever is larger. In practice, most sales orgs keep recordings for 6–24 months; regulated industries often keep them for 7 years or more. Every month a recording sits in storage after it’s stopped being useful is a month of exposure with no offsetting benefit. The decision that gets skipped most often isn’t “should we record” — it’s “when do we delete,” and it should be set before the first call is analyzed, not after a subject access request forces the question.

What It Costs to Use AI to Analyze Phone Calls at Scale

Transcription pricing gets quoted per minute, and in 2026 it’s genuinely cheap: AssemblyAI’s Universal-3.5 Pro runs around $0.0035/minute, Deepgram Nova-3 around $0.0043/minute, Google Cloud’s dynamic batch tier around $0.004/minute with a 24-hour turnaround, OpenAI’s Whisper API around $0.006/minute, and AWS Transcribe’s first tier around $0.0078/minute. Self-hosting faster-whisper drops the per-minute fee to zero, in exchange for owning a GPU and its uptime.

Run the math on a mid-size support line — 10,000 calls a month, six minutes average — and transcription turns out to be a rounding error next to everything built on top of it:

Volume:          10,000 calls/mo x 6 min avg = 60,000 minutes/mo

Transcription:   60,000 min x $0.0043/min (Nova-3)       ~ $258/mo
LLM analysis:    10,000 calls x ~$0.08/call (summary+QA) ~ $800/mo
Human QA audit:  5% sample (500 calls) x 8 min/call      ~ 67 hrs/mo
Storage:         2-yr retention, audio + text            ~ low, non-zero

Transcription is the part everyone negotiates over, and it’s the part that barely moves the total. LLM analysis costs two to three times more per month at this volume, and it scales linearly with call count in a way transcription pricing tiers don’t always match. The line item nobody puts in the vendor comparison spreadsheet is human review time — because if you take the QA-scores-nobody-validates problem seriously, someone on your team needs to spot-check a sample against the AI’s output regularly, and that’s a headcount cost, not a per-minute one.

The Channel With No Transcription Problem

There’s a version of this whole problem that doesn’t exist for text. A chat conversation has no acoustic signal to degrade, no accent for an ASR model to mishear, no overlapping speakers for a diarization model to untangle, no word error rate at all — because the customer’s words arrive as the words they typed. Human-written text is, by nature, its own record. Conversational speech transcription, by contrast, lands somewhere in the 10–25% WER range depending on acoustic conditions, before any downstream analysis even starts.

This is the honest reason we think moving suitable call volume into chat is worth evaluating even outside the analysis question. In AI Chat Agent, every session and every message — user, assistant, and operator turns — is stored verbatim, with rating, comment, page URL, a salted IP hash, visitor identity, and UTM parameters attached. The admin gives you paginated history, filters by period and rating, case-insensitive substring search across everything a visitor typed, and a per-session view showing exactly which knowledge-base chunks were retrieved for each answer and how well they matched. Retention auto-deletes at 90 or 365 days, so the decision from the previous section gets made once, in a setting, rather than argued about later.

To be precise about what that is and isn’t: it’s a verbatim, searchable log — not a conversation-intelligence suite. There’s no built-in sentiment dashboard, no topic clustering, no automated call-style QA scoring layered on top. If you want that, you build it on your own database, which is a very different proposition than building it on a black box, because you own the schema and can verify every number it produces against the transcript sitting right next to it. If you’re weighing a self-hosted chat widget against an all-in-one helpdesk that bundles its own analytics, our comparison with Intercom and the Zendesk breakdown both cover how the underlying data ownership differs.

Running Call Analysis Honestly

None of the above is an argument to abandon AI call analysis — it’s an argument to buy and operate it with the accuracy floor priced in rather than assumed away. A short checklist, in the order we’d actually apply it:

• Capture dual-channel, always. Separate the caller and agent onto distinct stereo channels at the point of recording. This eliminates diarization error by construction instead of paying an ML model to guess at it afterward.
• Measure entity-level accuracy, not WER. Ask your vendor for accuracy on names, amounts, dates, and SKUs specifically — the tokens your business decisions actually depend on — not the aggregate word error rate that hides them.
• Human-validate a sample of QA scores. Since no vendor publishes a correlation between transcription accuracy and scoring accuracy, build your own: have a human rater score 5–10% of calls independently and compare against the AI’s output every month, not once at onboarding.
• Keep the raw transcript next to every AI claim. A sentiment score or coaching note without a one-click path back to the exact sentence it came from isn’t an insight — it’s an unverifiable assertion in a confident font.
• Set retention before you start, not after a complaint. Decide the deletion window against your actual regulatory exposure, write it into the tool’s configuration, and treat every extra month of storage as a cost with no matching benefit. The same discipline applies to text conversation data under GDPR.

Most of this is process, not technology — which is also why most vendors won’t bring it up first. For more on where AI holds up under production conditions versus where it doesn’t, our blog covers the rest of the stack, voice and text both.

If what you take from all this is that text beats voice for anything you actually need to audit later, that’s a reasonable read — a chat transcript is accurate by construction in a way a phone call recording never will be, no matter how good the model gets. AI Chat Agent won’t score your calls, because it doesn’t take calls; what it gives you is a self-hosted widget with a verbatim, searchable log of every conversation, at a one-time €79, with the source code and the database both yours to inspect. You can see the transcript view yourself in the live demo, or go straight to checkout if you already know you’d rather own the record than trust someone else’s dashboard. Read a transcript, and you already know exactly how accurate it is — because you wrote every word of your half of it, and the AI never had to guess at the other half.

Frequently Asked Questions