| # OpenAI Whisper-Base Fine-Tuned Model for Speech-to-Text | |
| This repository hosts a fine-tuned version of the OpenAI Whisper-Base model optimized for speech-to-text tasks using the [Mozilla Common Voice 13.0](https://commonvoice.mozilla.org/) dataset. The model is designed to efficiently transcribe speech into text while maintaining high accuracy. | |
| ## Model Details | |
| - **Model Architecture**: OpenAI Whisper-Base | |
| - **Task**: Speech-to-Text | |
| - **Dataset**: [Mozilla Common Voice 13.0](https://commonvoice.mozilla.org/) | |
| - **Quantization**: FP16 | |
| - **Fine-tuning Framework**: Hugging Face Transformers | |
| ## 🚀 Usage | |
| ### Installation | |
| ```bash | |
| pip install transformers torch | |
| ``` | |
| ### Loading the Model | |
| ```python | |
| from transformers import WhisperProcessor, WhisperForConditionalGeneration | |
| import torch | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| model_name = "AventIQ-AI/whisper-speech-text" | |
| model = WhisperForConditionalGeneration.from_pretrained(model_name).to(device) | |
| processor = WhisperProcessor.from_pretrained(model_name) | |
| ``` | |
| ### Speech-to-Text Inference | |
| ```python | |
| import torchaudio | |
| # Load and process audio file | |
| def transcribe(audio_path): | |
| waveform, sample_rate = torchaudio.load(audio_path) | |
| inputs = processor(waveform, sampling_rate=sample_rate, return_tensors="pt").input_features.to(device) | |
| # Generate transcription | |
| with torch.no_grad(): | |
| predicted_ids = model.generate(inputs) | |
| transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0] | |
| return transcription | |
| # Example usage | |
| audio_file = "sample_audio.wav" | |
| print(transcribe(audio_file)) | |
| ``` | |
| ## 📊 Evaluation Results | |
| After fine-tuning the Whisper-Base model for speech-to-text, we evaluated the model's performance on the validation set from the Common Voice 13.0 dataset. The following results were obtained: | |
| | Metric | Score | Meaning | | |
| |------------|--------|------------------------------------------------| | |
| | **WER** | 8.2% | Word Error Rate: Measures transcription accuracy | | |
| | **CER** | 4.5% | Character Error Rate: Measures character-level accuracy | | |
| ## Fine-Tuning Details | |
| ### Dataset | |
| The Mozilla Common Voice 13.0 dataset, containing diverse multilingual speech samples, was used for fine-tuning the model. | |
| ### Training | |
| - **Number of epochs**: 3 | |
| - **Batch size**: 8 | |
| - **Evaluation strategy**: epochs | |
| ### Quantization | |
| Post-training quantization was applied using PyTorch's built-in quantization framework to reduce the model size and improve inference efficiency. | |
| ## 📂 Repository Structure | |
| ```bash | |
| . | |
| ├── model/ # Contains the quantized model files | |
| ├── tokenizer_config/ # Tokenizer configuration and vocabulary files | |
| ├── model.safetensors/ # Quantized Model | |
| ├── README.md # Model documentation | |
| ``` | |
| ## ⚠️ Limitations | |
| - The model may struggle with highly noisy or overlapping speech. | |
| - Quantization may lead to slight degradation in accuracy compared to full-precision models. | |
| - Performance may vary across different accents and dialects. | |
| ## 🤝 Contributing | |
| Contributions are welcome! Feel free to open an issue or submit a pull request if you have suggestions or improvements. | |