harphool17/whisper-noise-adapter

🎧 Whisper Noise Adapter (ASR) Created by: harphool17

Base Model: openai/whisper-large-v3 (OpenAI Whisper)

Architecture: LoRA (Low-Rank Adaptation) PEFT Adapter

Task: Automatic Speech Recognition (ASR)

Language: English

📖 Overview: What is this model? Standard OpenAI Whisper models are state-of-the-art at turning speech into text. However, in real-world environments—such as busy streets, windy days, or crowded rooms—the base model can become confused by background noise, leading to dropped words or AI "hallucinations."

This repository contains a Parameter-Efficient Fine-Tuned (PEFT) LoRA Adapter for the Whisper model. Think of this adapter as "noise-canceling headphones" for the AI. It was specifically engineered to help the base model ignore background static, chatter, and environmental noise, allowing it to focus strictly on transcribing the human voice accurately.

Instead of retraining the massive 3-Gigabyte base model, this lightweight 35-Megabyte adapter plugs directly into the original Whisper architecture, altering its attention mechanisms to filter out noise without losing its mastery of the English language.

📊 Training Data & Performance To teach the model how to ignore background noise, it was trained on a highly challenging custom dataset.

The Dataset Total Training Data: Approximately [Insert Number, e.g., 50] hours of audio data.

The "Noise" Strategy: To simulate real-world environments, clean speech samples were artificially degraded by overlaying various noise profiles (e.g., street traffic, room echo, static, and background chatter).

The model was trained across various Signal-to-Noise Ratios (SNRs) to learn voice extraction regardless of the background volume.

What We Achieved By targeting the cross-attention modules with LoRA, the model learned to mathematically "cancel out" noise signatures before the text-generation phase. Compared to the base Whisper Large-v3 model, this adapter achieved:

A [Insert %, e.g., 25%] reduction in Word Error Rate (WER) on highly noisy audio files.

A massive reduction in AI "hallucinations" (where the base model invents words to compensate for static).

Maintained baseline accuracy on clean audio, successfully avoiding catastrophic forgetting.

📂 File Breakdown (What's Inside?) If you are new to PEFT and LoRA, here is exactly what the files in this repository do:

🧠 adapter_model.safetensors: The actual compiled learning. It contains the new "knowledge" the model learned about ignoring noise. The .safetensors format ensures fast, secure loading.

📐 adapter_config.json: The architectural blueprint. It tells the base Whisper model exactly where to connect these new weights (Rank=16, targeting Attention modules).

👂 processor_config.json: The acoustic instructions. It tells the model how to convert an audio file into the specific mathematical format (spectrograms) this fine-tune expects.

🗣️ tokenizer.json & tokenizer_config.json: The dictionary used to translate the AI's final mathematical predictions back into readable English text.

💻 How to Use This Model (Step-by-Step) Because this is an "Adapter," it cannot run by itself. You must load the Base Whisper model first, and then fuse this adapter to it.

Prerequisites

pip install transformers peft torch torchaudio librosa

Inference Code Use the following Python script to automatically download the models, fuse them together, and transcribe an audio file.

Python import torch import librosa from transformers import WhisperProcessor, WhisperForConditionalGeneration from peft import PeftModel

--- 1. SET UP THE PATHS ---

base_model_name = "openai/whisper-large-v3" adapter_name = "harphool17/whisper-noise-adapter"

--- 2. LOAD THE MODELS ---

print("⏳ Downloading the Base Whisper Model...") processor = WhisperProcessor.from_pretrained(base_model_name)

Load the base model in 16-bit precision to save GPU memory

base_model = WhisperForConditionalGeneration.from_pretrained( base_model_name, torch_dtype=torch.float16, device_map="auto" )

print("📥 Snapping on the Noise-Canceling Adapter...")

Automatically downloads the safetensors file and applies it

model = PeftModel.from_pretrained(base_model, adapter_name) model.eval() print("✅ AI is ready to listen!")

--- 3. TRANSCRIBE YOUR AUDIO ---

Replace "your_audio_file.wav" with your actual audio file

audio_path = "your_audio_file.wav"

Whisper requires audio to be exactly 16000 Hz. Librosa resamples automatically.

audio_array, sampling_rate = librosa.load(audio_path, sr=16000)

Process the audio into AI-readable tensors

inputs = processor(audio_array, sampling_rate=sampling_rate, return_tensors="pt") inputs = inputs.to("cuda", torch.float16)

Generate the prediction

with torch.no_grad(): predicted_ids = model.generate(**inputs)

Decode the prediction into English text

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

print("\n🎙️ FINAL TRANSCRIPTION:") print(transcription) 🛠️ Engineering Details For machine learning engineers, building this required specific hardware constraint resolutions:

Precision Management: Whisper's Encoder-Decoder architecture is highly susceptible to Type Mismatch crashes during training when cross-attention layers attempt to combine Float32 and BFloat16 calculations. This was resolved by forcing the entire PyTorch loop into strict bfloat16 precision.

Target Modules: To prevent catastrophic forgetting, we isolated the LoRA targets strictly to the Attention Projections (q_proj, v_proj).

Hardware: Trained utilizing NVIDIA Native Data Collators to bypass high-level library abstraction errors during Mixture of Experts (MoE) routing.

⚠️ Limitations Audio Length: The Whisper architecture processes audio in 30-second chunks. If you pass an audio file longer than 30 seconds into the standard generate function, it will only transcribe the first 30 seconds. For longer files, you must implement chunking or use Hugging Face's pipeline class.

Extreme Distortion: While highly robust, audio where the human voice is completely overpowered by noise (a negative Signal-to-Noise Ratio) may still result in inaccuracies. This adapter enhances the signal but cannot recover audio that was never captured by the microphone.