# Whisper Hallucinations: Why One AI Pass Should Never Become Evidence
OpenAI's Whisper is the most widely used automatic speech recognition (ASR) model in the world. It is also confidently wrong on faint audio — and it does not tell you when.
What is a hallucination?
In ASR, a hallucination is invented speech. The audio is unclear — noise, overlap, silence — and the model fills the gap with plausible words. The output looks fluent. The confidence score may be high. The words were never spoken.
On clear audio, Whisper is excellent. On difficult audio — jail calls, bodycam wind, voicemails from moving cars — it guesses, and those guesses become your transcript.
The research
A Cornell University study measured Whisper's hallucination rate at approximately 1.4% of transcribed content. That sounds small. On a 30-minute recording, 1.4% is 25 seconds of invented speech.
The University of Michigan analyzed public meeting recordings and found errors in 8 out of 10 samples — including fabricated medical terms, nonexistent speaker names, and phrases that contradicted the audio.
The Careless Whisper paper (FAccT 2024) examined harmful hallucinations specifically: invented violence, fabricated statements, and demographic errors. The finding: 38% of Whisper's hallucinations were harmful — not benign filler, but content that could change the meaning of a recording.
Why confidence scores do not help
ASR confidence scores measure fluency, not accuracy. A model can be 95% confident in a hallucination because the invented words form a grammatically plausible sentence.
Whisper does not return per-word confidence by default. Third-party tools estimate it, but those estimates are derived from the same model that hallucinated — you are asking the guesser to rate its own guess.
Confidence scores also cannot distinguish between different types of errors:
- Substitution errors — wrong word, plausible context ("thank you" instead of "can't do")
- Insertion errors — words added that were never spoken
- Deletion errors — words omitted, leaving gaps in meaning
A high-confidence substitution error is worse than a low-confidence [inaudible] marker — because the former looks authoritative and the latter signals uncertainty.
The legal problem
In Clarke v Guardian News (UK, 2024), questions about AI transcription reliability surfaced in court. The issue was not whether the tool was popular or well-reviewed — the issue was whether its output could be verified when the process was opaque.
The RCMP External Review Committee has issued guidance on the use of AI-assisted evidence in tribunal and court proceedings. The guidance emphasizes that AI tools used in preparing evidence must have their methods documented and their limitations disclosed. A tool's reputation does not make its output admissible. The method must be reproducible and the uncertainty must be stated.
When a transcript is challenged, the burden is on the proponent to demonstrate reliability. Understanding what makes a transcript admissible is the first step — but avoiding hallucinations is the practical foundation.
What happens when you submit a single-pass transcript
Opposing counsel cross-examines the transcript, not the recording. The questions:
- Did a person review this?
- How do you know the AI did not invent anything?
- Were alternative transcriptions considered?
- Can you explain why the AI chose these specific words?
If your answer is "Whisper is very accurate," you have no defense. The model does not document its uncertainty. A single pass gives you no second opinion.
Multi-pass consensus as the fix
One pass is a guess. Five passes vote.
VeriVox runs multiple transcription models across multiple enhancement recipes. Each pass produces word-level timestamps. The passes are aligned and voted.
Where five passes agree, the line is trustworthy. Where they scatter — one says "thank you," another says "can't do," a third marks it inaudible — the disagreement is the signal. That line goes to a human ear.
Hallucinations do not survive a vote they have to win five times.
The disagreement itself becomes part of the record. When you present a verified correction and opposing counsel challenges it, you can show: five models disagreed, here are the five outputs (preserved), and a qualified person resolved the conflict by ear. That is defensible. A single-pass output with no alternatives is not.
A real pattern from a real matter
In a contested recording used in a legal matter, Whisper produced this on the faintest section:
[22:03] "Thank you." — confidence 0.34
The audio was almost inaudible. The model invented polite filler. Confidence was low, but the transcript presented it as fact.
When the same audio was run through VeriVox's multi-pass pipeline:
[22:03.1–22:04.6] ⚑ SUBJECT-A: [grave excerpt — operator-verified]
agree 0/5 — machine could not resolve
Five passes could not agree. The line went to the ear queue. A human verified it. The verified words were nothing like "thank you" — and they mattered to the case.
The single-pass transcript would have been a gift to opposing counsel.
What to do instead
Run multiple passes. Different models make different mistakes. Agreement is signal. Disagreement surfaces the marginal sections.
Preserve the scatter. When passes disagree, do not pick one and delete the others. The disagreement tells you which lines need human review.
Queue by importance. Not every word matters equally. Disagreements on faint speech in critical moments — threats, admissions, contested facts — go to the top of the verification queue.
Label what was verified. The export must say which lines were reviewed by a person and which were machine-only. Unlabeled machine output is assumed to be unverified.
The honesty ledger approach
VeriVox exports do not hide uncertainty. Every line carries its provenance:
[03:12.4–03:16.1] ✓ SPEAKER-A: You were never supposed to be here.
agree 5/5 · verified by ear
[07:41.8–07:44.0] ✎ SPEAKER-A: The paperwork was already gone by then.
agree 3/5 · corrected
(ASR original: "the paper it was already gone been")
[14:22.0–14:23.5] · UNATTRIBUTED: [background media audible]
excluded from excerpts · machine-only
High agreement earns trust. Low agreement earns scrutiny. Machine-only lines are labeled as such. Nothing is taken on faith.
When Whisper works
Whisper works well on clear audio: podcasts, interviews, lectures. It is fast, multilingual, and often accurate.
It fails on the audio that matters in legal cases: jail calls with compression artifacts, bodycam recordings in wind, voicemails from moving vehicles, threats recorded through walls.
If you are transcribing difficult evidence audio, one pass is not enough.
The pipeline that catches hallucinations
VeriVox's five-stage pipeline:
- Ingest — SHA-256 hash, custody manifest
- Enhance ×N — scored, reproducible recipes
- Transcribe ×N — multiple models, multiple passes
- Consensus — word-level voting, agreement scoring
- Verify — human ear on disagreements, enrolled voice scoring
Personal use is free forever. See the full pipeline →
For deeper understanding of how disagreement between passes surfaces the need for human review, read about the inaudible problem — what transcripts hide when models fail to converge.
FAQ
Does VeriVox use Whisper?
Yes — as one model in a multi-pass ensemble. Whisper's errors are caught by disagreement with other models and flagged for review.
What if all the passes hallucinate the same thing?
Rare, but possible on very faint audio. That is why VeriVox queues low-agreement sections for human ear review — sorted by importance, not just by score.
Can I just run Whisper multiple times myself?
Running the same model on the same audio will produce the same output. Multi-pass consensus requires different models or different enhancement recipes to produce independent opinions.
How do I know which lines to review?
VeriVox sorts the disagreement queue by agreement score and flags lines based on context (speaker transitions, long pauses, elevated volume). Start with the flagged sections.
Is Whisper unusable for legal transcription?
No — but it should never be the only opinion. Treat any single-pass output as a draft, not a record.
What about other AI transcription services like Otter or Descript?
They all use similar underlying models (often Whisper or variants). The hallucination problem is not unique to Whisper — it is inherent in single-pass ASR on difficult audio. Multi-pass voting mitigates it regardless of which models you use.
How do I explain this to a judge who is not technical?
Use the voting analogy: one person's opinion can be wrong. Five independent opinions that agree are harder to dismiss. When they disagree, that disagreement tells you where a human needs to decide. Courts understand jury deliberation — this is the same principle applied to speech recognition.