kulsoom-abdullah/Qwen2-Audio-7B-Transcription

Qwen2-VL-Audio-Adapter

Multimodal Fusion: Integrating Whisper Audio Encoder with Qwen2-VL for Production-Grade Speech Recognition

Achieves commercial-grade ASR quality (WER 3.6% on Train, 7.3% on Unseen Test) by fusing a Whisper-Large-v3-Turbo encoder onto Qwen2-VL-7B-Instruct using a two-stage training pipeline.

🎯 Performance Highlights

Evaluation Context: Tested on a held-out subset of 100 samples from the SpeechBrain test partition (English Parliamentary speech).

Metric	Training Set	Test Set (Unseen)	Industry Standard
Word Error Rate (WER)	3.6%	7.3%	5-10%
True WER (Label-Corrected)	-	~14%	-
Character Error Rate (CER)	2.5%	2.5%	3-5%
Label Correction Rate	-	36%	-

Novel Finding: On completely unseen test data, the model corrected ground truth annotations in 36% of disagreement cases, demonstrating super-human labeling performance through context-aware semantic reasoning.

🏗️ Architecture

┌─────────────────────────────────────────────────┐
│  Whisper-Large-v3-Turbo Encoder (Frozen)        │
│  1.5B params → 1280-dim audio features          │
└────────────────┬────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────┐
│  Audio Projector (Trainable)                    │
│  Linear: 1280 → 3584 dims (4.6M params)         │
└────────────────┬────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────┐
│  Qwen2-VL-7B LLM (QLoRA Fine-tuned)             │
│  7B params with rank-64 LoRA adapters           │
└─────────────────────────────────────────────────┘

🔬 Rigorous Audit: Label Noise & Semantic Bias

To validate model quality on truly unseen data, we conducted a blind manual audit of 100 samples from the SpeechBrain test partition.

🔎 Audit Visualizer

1. Label Noise & Entity Resolution The model (Green) correctly identified "Mr. Šefčovič" (Maroš Šefčovič, EU Commissioner), correcting the ground truth "Mr. Efovi" (Red).

2. Semantic Bias & Long-Range Context The model "hallucinated" the word "Malta" (Green) in the first sentence because it attended to the context provided later in the audio, proving editorial reasoning.

Quantitative Analysis (N=100)

Category	Count	Description
✅ Label Noise (Model Correct)	36%	Model outperformed ground truth annotations
❌ True Model Errors	14%	Model genuinely misheard or hallucinated
⚠️ Ambiguous	11%	Heavy accents or unclear audio
✓ Perfect Matches	37%	Exact agreement

💻 Usage

Important: This model requires a modified transformers library (included in the repo files).

Installation

Method 1: Git Clone (Recommended)

# Clone the model repo (includes transformers fork)
git clone [https://huggingface.co/kulsoom-abdullah/Qwen2-VL-Audio-Adapter](https://huggingface.co/kulsoom-abdullah/Qwen2-VL-Audio-Adapter)
cd Qwen2-VL-Audio-Adapter

# Install dependencies
pip install torch transformers librosa soundfile accelerate

Basic Inference

import sys
import torch
import librosa

# Load modified transformers from repo
sys.path.insert(0, "./transformers_fork/src")

from transformers import (
    Qwen2VLForConditionalGeneration,
    AutoTokenizer,
    WhisperFeatureExtractor
)

# Load model
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "kulsoom-abdullah/Qwen2-VL-Audio-Adapter",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "kulsoom-abdullah/Qwen2-VL-Audio-Adapter",
    trust_remote_code=True
)

feature_extractor = WhisperFeatureExtractor.from_pretrained(
    "openai/whisper-large-v3-turbo"
)

# Load and prepare audio
audio_path = "your_audio.wav"
y, sr = librosa.load(audio_path, sr=16000, mono=True)
inputs = feature_extractor(y, sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to(model.device).to(torch.bfloat16)

# Build prompt
AUDIO_TOKEN_ID = 151657
NUM_AUDIO_TOKENS = 1500
audio_tokens = [AUDIO_TOKEN_ID] * NUM_AUDIO_TOKENS
input_ids_audio = torch.tensor([audio_tokens], device=model.device)

p1 = tokenizer.encode("<|im_start|>user\n<|audio_bos|>", add_special_tokens=False, return_tensors="pt").to(model.device)
p2 = tokenizer.encode("<|audio_eos|>\nTranscribe this audio.<|im_end|>\n<|im_start|>assistant\n", add_special_tokens=False, return_tensors="pt").to(model.device)
input_ids = torch.cat([p1, input_ids_audio, p2], dim=1)

# Generate
with torch.no_grad():
    generated_ids = model.generate(
        input_ids=input_ids,
        input_features=input_features,
        max_new_tokens=128
    )

print(tokenizer.decode(generated_ids[0][input_ids.shape[1]:], skip_special_tokens=True))

📝 Citation

@misc{qwen2-vl-audio-adapter,
  author = {Kulsoom Abdullah},
  title = {Qwen2-VL-Audio-Adapter: Multimodal Projection Alignment for Speech Recognition},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{[https://huggingface.co/kulsoom-abdullah/Qwen2-VL-Audio-Adapter](https://huggingface.co/kulsoom-abdullah/Qwen2-VL-Audio-Adapter)}}
}

📄 License

Apache 2.0 (inherits from Qwen2-VL and Whisper)

---

Kulsoom Abdullah | GitHub