Whisper Fine-Tuned Automatic Speech Recognition Model

Model Details

Model Description

This model is a fine-tuned version of OpenAI's Whisper model for Automatic Speech Recognition (ASR). It has been trained on a custom speech dataset to improve transcription accuracy for domain-specific speech and conversational audio.

The model leverages the strong multilingual capabilities of Whisper while adapting to the characteristics of the target dataset through supervised fine-tuning.

Developed by: Gad Amr , Osama Mohamed , Rojeh Wael
Funded by: Self-funded Academic Graduation Project
Shared by: Gad Amr
Model type: Automatic Speech Recognition (ASR)
Base model: OpenAI Whisper Medium
Language(s): Arabic / English
Framework: Hugging Face Transformers
License: Apache-2.0
Fine-tuned from model: openai/whisper-Medium

Direct Use

This model is designed for automatic speech recognition tasks, including:

Speech-to-text transcription

Downstream Use

The model can be integrated into:

Meezan Graduation Project

Out-of-Scope Use

Bias, Risks, and Limitations

Like all speech recognition models, performance depends on:

Audio quality
Background noise
Microphone quality
Accent diversity
Speaking speed
Domain mismatch

Potential limitations include:

Reduced accuracy on unseen accents but covered Egyptian ones .
Difficulty with overlapping speakers.

Recommendations

How to Get Started with the Model

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="DBGad/Meezan-ASR"
)

result = pipe("sample.wav")

print(result["text"])

Preprocessing

The following preprocessing steps were applied:

Resampling audio to 16 kHz
Removing corrupted samples
Feature extraction using WhisperFeatureExtractor
Text normalization
Tokenization using WhisperTokenizer
Dynamic padding with DataCollatorSpeechSeq2SeqWithPadding

LoRA Configuration

Parameter	Value
Rank (r)	8
Alpha	16
Dropout	0.1

Target Modules:

encoder_attn.q_proj
encoder_attn.v_proj
encoder_attn.out_proj

Training Hyperparameters

Hyperparameter	Value
Base Model	openai/whisper-medium
Epochs	12
Batch Size (per device)	16
Gradient Accumulation	8
Effective Batch Size	128
Learning Rate	5e-4
Warmup Steps	150
Evaluation Strategy	Every 200 Steps
Checkpoint Saving	Every 200 Steps
Logging Steps	Every 50 Steps
Best Model Selection Metric	Word Error Rate (WER)
Load Best Model	Enabled

Optimization

The model was trained using:

Mixed Precision (FP16/BF16)
Gradient Checkpointing
LoRA Parameter-Efficient Fine-Tuning

These techniques significantly reduced GPU memory consumption while maintaining strong model performance.

Evaluation

Validation was performed every 200 training steps.

The best checkpoint was automatically selected using the lowest Word Error Rate (WER).

Evaluation Metric:

Word Error Rate (WER)

Hardware & Software

Hardware

Training was performed on an NVIDIA GPU with CUDA support.

Software

Python
PyTorch
Hugging Face Transformers
PEFT (LoRA)
Datasets
Evaluate
Accelerate

Model Objective

The objective of the model is to convert spoken English audio into accurate text transcriptions.

The model minimizes sequence-to-sequence cross-entropy loss while optimizing transcription quality measured by Word Error Rate (WER).

Model Card Authors

Gad Amr

Model Card Contact

For questions, suggestions, or collaboration:

Downloads last month: 55

Safetensors

Model size

0.8B params

Tensor type

F16