README.md

---
license: mit
base_model:
- facebook/wav2vec2-large-robust
- aadel4/Wav2vec_Classroom
pipeline_tag: automatic-speech-recognition
library_name: transformers
language: en
tags:
  - audio
  - automatic-speech-recognition
  - wav2vec2
---
## Model Card: Wav2vec_Classroom_WSP_FT

### Model Overview
**Model Name:** Wav2vec_Classroom_WSP_FT  
**Version:** 1.0  
**Developed By:** Ahmed Adel Attia (University of Maryland)  
**Date:** 2025  

**Description:**  
Wav2vec_Classroom_WSP_FT is an automatic speech recognition (ASR) model trained specifically for classroom speech transcription using a weakly supervised pretraining (WSP) approach. The model first undergoes supervised pretraining on weakly transcribed classroom data (NCTE-Weak) and is then fine-tuned using a small amount of human-verified gold-standard data (NCTE-Gold). This methodology allows the model to generalize well despite the scarcity of precisely transcribed classroom speech.  

This model is adapted from **[Wav2vec-Classroom](https://huggingface.co/aadel4/Wav2vec_Classroom)**, which was trained using continued pretraining (CPT) on large-scale unlabeled classroom speech data. The adaptation involves further fine-tuning to leverage weak transcriptions before final refinement on high-quality annotations.

This model was originally trained using the fairseq library then ported into Huggingface.

The model should be run with n-gram LM beamsearch decoding for best results. We got our best results using [this](https://drive.google.com/drive/u/0/folders/1yAFXcbozqDUFZu-hnnzFP_8SAzDYT2JJ) 5-gram LM we trained on classroom speech text.

**Use Case:**  
- Speech-to-text transcription for classroom environments.
- Forced allignment of transcription with audio to provide character and word level boundaries. 
- Educational research and analysis of classroom discourse.
- Low-resource ASR applications where gold-standard labels are limited.

### Model Details
**Architecture:** Wav2vec2.0-based model fine-tuned with Fairseq  

**Training Data:**  
- **NCTE-Weak:** 5000 hours of weak transcriptions from the NCTE dataset. 
- **NCTE-Gold:** 13 hours of manually transcribed classroom recordings.

**Training Strategy:**  
1. **Weakly Supervised Pretraining (WSP):** The model is first trained using NCTE-Weak transcripts, which contain alignment errors and omissions but provide useful weak supervision.
2. **Precise Fine-tuning:** The pretrained model is fine-tuned on NCTE-Gold, ensuring it adapts to high-quality transcriptions.

### Evaluation Results
**Word Error Rate (WER) comparison on NCTE and MPT test sets:**

| Training Data | NCTE WER | MPT WER |
|--------------|----------|---------|
| **Baseline (TEDLIUM-trained ASR)** | 55.82 / 50.56 | 55.11 / 50.50 |
| **NCTE-Weak only** | 36.23 / 32.30 | 50.84 / 46.09 |
| **NCTE-Gold only** | 21.12 / 16.47 | 31.52 / 27.93 |
| **Self-training** | 17.45 / 15.09 | 27.42 / 26.24 |
| **NCTE-WSP-ASR (NCTE-Weak → NCTE-Gold)** | **16.54 / 13.51** | **25.07 / 23.70** |

### Limitations
- The model relies on weak supervision, and transcription quality is dependent on the balance between weak and gold-standard data.
- Classroom noise, overlapping speech, and spontaneous interactions may still lead to recognition errors.
- The model was trained specifically on elementary math classrooms and may not generalize well to other educational settings without further adaptation.

### Usage Request
If you use the NCTE-WSP-ASR model in your research, please acknowledge this work and refer to the original paper submitted to Interspeech 2025.

For inquiries or collaborations, don't hesitate to contact me at aadel@umd.edu or ahmadadelattia@gmail.com