Investigating Hallucinations in Whisper ASR

Overview

The research problem I would like to explore is hallucination in automatic speech recognition (ASR) models, particularly Whisper. In some situations, especially when an audio clip is very short or contains substantial background noise, Whisper can produce words or entire phrases that were never spoken.

I have worked on this problem previously. That work led to a practical audio-pipeline-level mitigation that reduced hallucinations by improving how audio was handled before transcription. Although it was useful in real-world applications, it did not address the underlying behavior of the ASR model itself. I now want to revisit the problem at the model level, with the goal of understanding its causes and investigating a more fundamental solution.

Materials Shared

I have shared the following materials to illustrate the problem:

• Experiment notebook: Runs Whisper on our recorded evaluation dataset and captures its outputs.
• Recorded audio dataset: Contains short clips in which a single word is spoken with background noise.
• Results CSV: Contains the transcription results exported from the notebook and uploaded to the shared Hugging Face bucket/repository.

What the Experiment Shows

In the notebook, I ran Whisper on the recorded single-word audio clips. In some cases, rather than transcribing the word that was actually spoken, the model generated additional or unrelated text. I exported the resulting transcriptions to a CSV so that these errors can be inspected and compared systematically.

This is an initial demonstration of the problem, not yet a controlled study of its root cause. The current dataset provides a starting point for designing more rigorous experiments.

Why Might Whisper Hallucinate?

Whisper was trained on a large collection of web audio and transcripts. Repeated expressions found in online videos, such as “thank you” or “thank you very much,” may influence what the model predicts when the audio signal is ambiguous.

Answer to Question 3

My first proposed approach is to fine-tune Whisper using carefully labeled noisy speech data, then assess whether hallucinations decrease compared with the original model.

I would start by:

1. Preparing a more representative dataset: Expand the existing single-word noisy recordings to cover different speakers, noise types, and signal-to-noise ratios. Add silence-only and noise-only samples with appropriate no-speech labels, as well as clean speech examples to help preserve ordinary ASR performance.
2. Establishing a baseline: Evaluate the original Whisper model on a held-out test set, recording expected transcripts, generated transcripts, and common hallucination patterns.
3. Fine-tuning Whisper: Train on the curated noisy examples with accurate transcripts, taking care not to make the model simply suppress difficult but valid speech.
4. Evaluating the results: Compare the baseline and fine-tuned models using hallucination frequency, false transcriptions on non-speech audio, word/phrase accuracy on short utterances, and overall ASR quality on clean and noisy speech.
5. Testing generalization: Evaluate on unseen speakers and noise conditions to determine whether the improvement extends beyond the particular recordings used for training.

The objective is not merely to reduce the amount of text the model outputs, but to make its transcriptions more faithful to the actual audio, particularly under noisy and uncertain conditions.