|
|
--- |
|
|
datasets: |
|
|
- slprl/TinyStress-15K |
|
|
license: cc-by-nc-4.0 |
|
|
library_name: transformers |
|
|
pipeline_tag: automatic-speech-recognition |
|
|
--- |
|
|
|
|
|
# WhiStress Model |
|
|
|
|
|
This is the official model checkpoint for [***WhiStress***](https://arxiv.org/abs/2505.19103) — introduced in our paper: |
|
|
**WhiStress: Enriching Transcriptions with Sentence Stress Detection** (Interspeech 2025). |
|
|
|
|
|
- 🔗 Project Page: [pages.cs.huji.ac.il/adiyoss-lab/whistress](https://pages.cs.huji.ac.il/adiyoss-lab/whistress) |
|
|
- 📚 Code: [github.com/slp-rl/WhiStress](https://github.com/slp-rl/WhiStress) |
|
|
- 📦 Dataset: [slprl/TinyStress-15K](https://huggingface.co/datasets/slprl/TinyStress-15K) |
|
|
|
|
|
--- |
|
|
|
|
|
## Overview |
|
|
|
|
|
**WhiStress** extends OpenAI's [Whisper](https://huggingface.co/openai/whisper-small.en) ASR model with a decoder-based classifier that predicts **token-level sentence stress**. This allows models not only to transcribe speech but also to detect which words are emphasized. |
|
|
|
|
|
This checkpoint is based on the `whisper-small.en` variant and adds two stress-specific modules: |
|
|
- `additional_decoder_block.pt` |
|
|
- `classifier.pt` |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔧 How to Use |
|
|
|
|
|
You can use the weights in your own pipeline by cloning our codebase and loading the components: |
|
|
|
|
|
```bash |
|
|
git clone https://github.com/slp-rl/WhiStress.git |
|
|
cd WhiStress |
|
|
pip install -r requirements.txt |
|
|
``` |
|
|
|
|
|
Then, either download the weights manually from this Hugging Face repo or use our script: |
|
|
|
|
|
```bash |
|
|
python download_weights.py |
|
|
``` |
|
|
|
|
|
The weights should be placed in the following directory structure: |
|
|
|
|
|
``` |
|
|
whistress/ |
|
|
├── weights/ |
|
|
│ ├── additional_decoder_block.pt |
|
|
│ ├── classifier.pt |
|
|
│ └── metadata.json |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 🗣️ Inference Example |
|
|
|
|
|
```python |
|
|
from whistress import WhiStressInferenceClient |
|
|
|
|
|
|
|
|
whistress_client = WhiStressInferenceClient(device="cuda") # or "cpu" |
|
|
|
|
|
pred_transcription, pred_stresses = whistress_client.predict( |
|
|
audio=sample['audio'], # (sr, np.ndarray) |
|
|
transcription=None, # predict directly from audio both transcription and stress, pass transcription to predict stress only. |
|
|
return_pairs=False # set to True if you a list want a list of (word, binary_label) pairs. |
|
|
) |
|
|
print(pred_transcription) # e.g., "I didn’t say she stole my money." |
|
|
print(pred_stresses) # e.g., ['my'] |
|
|
``` |
|
|
|
|
|
Each prediction includes: |
|
|
- `transcription`: full text output |
|
|
- `emphasis_indices`: list of stressed token indices |
|
|
- `emphasized_tokens`: list of corresponding words |
|
|
|
|
|
--- |
|
|
|
|
|
## Notes |
|
|
|
|
|
The model is intended for research purposes only. |
|
|
|
|
|
## 📜 Citation |
|
|
If you use our model, please cite our work: |
|
|
|
|
|
```bibtex |
|
|
@misc{yosha2025whistress, |
|
|
title={WHISTRESS: Enriching Transcriptions with Sentence Stress Detection}, |
|
|
author={Iddo Yosha and Dorin Shteyman and Yossi Adi}, |
|
|
year={2025}, |
|
|
eprint={2505.19103}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL}, |
|
|
url={https://arxiv.org/abs/2505.19103}, |
|
|
} |
|
|
``` |
