README.md

---
license: apache-2.0
datasets:
- AImpower/MandarinStutteredSpeech
language:
- zh
metrics:
- cer
base_model:
- openai/whisper-large-v2
pipeline_tag: automatic-speech-recognition
---
# Model Card: AImpower/StutteredSpeechASR

This model is a version of OpenAI's `whisper-large-v2` fine-tuned on the **AImpower/MandarinStutteredSpeech** dataset, a grassroots-collected corpus of Mandarin Chinese speech from people who stutter (PWS).

## Model Details

* **Base Model:** `openai/whisper-large-v2`
* **Language:** Mandarin Chinese
* **Fine-tuning Dataset:** [AImpower/MandarinStutteredSpeech](https://huggingface.co/datasets/AImpower/MandarinStutteredSpeech)
* **Fine-tuning Method:** The model was fine-tuned using the LoRA adapter (AdaLora) methodology to preserve speech disfluencies in its transcriptions.
* **Paper:** [Our Collective Voices: The Social and Technical Values of a Grassroots Chinese Stuttered Speech Dataset](https://doi.org/10.1145/3715275.3732179)

## Model Description

This model is specifically adapted to provide more accurate and authentic transcriptions for Mandarin-speaking PWS.  
Standard Automatic Speech Recognition (ASR) models often exhibit "fluency bias," where they "smoothen" out or delete stuttered speech patterns like repetitions and interjections.  
This model was fine-tuned on literal transcriptions that intentionally preserve these disfluencies.

The primary goal is to create a more inclusive ASR system that recognizes and respects the natural speech patterns of PWS, reducing deletion errors and improving overall accuracy.

## Intended Uses & Limitations

### Intended Use

This model is intended for transcribing conversational Mandarin Chinese speech from individuals who stutter. It's particularly useful for:
* Improving accessibility in speech-to-text applications.
* Linguistic research on stuttered speech.
* Developing more inclusive voice-enabled technologies.

### Limitations

* **Language Specificity:** The model is fine-tuned exclusively on Mandarin Chinese and is not intended for other languages.
* **Data Specificity:** Performance is optimized for speech patterns present in the AImpower/MandarinStutteredSpeech dataset. It may not perform as well on other types of atypical speech or in environments with significant background noise.
* **Variability:** Stuttering is highly variable. While the model shows significant improvements across severity levels, accuracy may still vary between individuals and contexts.

---

## How to Use

You can use the model with the `transformers` library. Ensure you have `torch`, `transformers`, and `librosa` installed.

```python
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch
import librosa

# Load the fine-tuned model and processor
model_path = "AImpower/StutteredSpeechASR"
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_path)
processor = AutoProcessor.from_pretrained(model_path)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

# Load an example audio file (replace with your audio file)
audio_input_name = "example_stuttered_speech.wav"
waveform, sampling_rate = librosa.load(audio_input_name, sr=16000)

# Process the audio and generate transcription
input_features = processor(waveform, sampling_rate=sampling_rate, return_tensors="pt").input_features
input_features = input_features.to(device)

predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

print(f"Transcription: {transcription}")
```

-----

## Training Data

The model was fine-tuned on the **[AImpower/MandarinStutteredSpeech](https://huggingface.co/datasets/AImpower/MandarinStutteredSpeech)** dataset.  
This dataset was created through a community-led, grassroots effort with StammerTalk, an online community for Chinese-speaking PWS.

  * **Size:** The dataset contains nearly 50 hours of speech from 72 adults who stutter.
  * **Content:** It includes both unscripted, spontaneous conversations between two PWS and the dictation of 200 voice commands.
  * **Transcription:** The training was performed on verbatim (literal) transcriptions that include disfluencies such as word repetitions and interjections, which was a deliberate choice by the community to ensure their speech was represented authentically.

## Training Procedure

  * **Data Split:** A three-fold cross-validation approach was used, with data split by participant to ensure robustness. Each fold had a roughly 65:10:25 split for train/dev/test sets, with a balanced representation of mild, moderate, and severe stuttering levels. This model card represents the best-performing fold.
  * **Hyperparameters:**
      * **Epochs:** 3
      * **Learning Rate:** 0.001
      * **Optimizer:** AdamW
      * **Batch Size:** 16
      * **Fine-tuning Method:** AdaLora
  * **GPU:** Four NVIDIA A100 80G GPUs
-----

## Evaluation Results

The fine-tuned model demonstrates a substantial improvement in transcription accuracy across all stuttering severity levels compared to the baseline `whisper-large-v2` model.  
The key metric used is Character Error Rate (CER), evaluated on literal transcriptions to measure the model's ability to preserve disfluencies.

| Stuttering Severity | Baseline Whisper CER | Fine-tuned Model CER |
| :------------------ | :------------------- | :------------------- |
| Mild                | 16.34%               | **5.80%**            |
| Moderate            | 21.72%               | **9.03%**            |
| Severe              | 49.24%               | **20.46%**           |

*(Results from Figure 3 of the paper)*

Notably, the model achieved a significant reduction in **deletion errors (DEL)**, especially for severe speech (from 26.56% to 2.29%), indicating that it is much more effective at preserving repeated words and phrases instead of omitting them.

## Citation

If you use this model, please cite the original paper:

```bibtex
@inproceedings{li2025collective,
  author = {Li, Jingjin and Li, Qisheng and Gong, Rong and Wang, Lezhi and Wu, Shaomei},
  title = {Our Collective Voices: The Social and Technical Values of a Grassroots Chinese Stuttered Speech Dataset},
  year = {2025},
  isbn = {9798400714825},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3715275.3732179},
  booktitle = {The 2025 ACM Conference on Fairness, Accountability, and Transparency},
  pages = {2768–2783},
  location = {Athens, Greece},
  series = {FAccT '25}
}
```