README.md

---
library_name: transformers
tags:
- AVSR
- AVHuBERT
language:
- ja
pipeline_tag: automatic-speech-recognition
base_model:
- enactic/japanese-avhubert-base_noise_pt
metrics:
- cer
---

<div align="center">
    <video width="80%" controls>
        <source src="https://huggingface.co/datasets/enactic/assets/resolve/main/AVista%20demo.mp4" type="video/mp4">
        Your browser does not support the video tag.
    </video>
</div>

# AVista Base+ 🐦‍🔥

This is AVHuBERT (Audio-Visual Hidden Unit BERT) Base model for AVSR (Audio-Visual Speech Recognition) task, derived from [`enactic/japanese-avhubert-base_noise_pt`](https://huggingface.co/enactic/japanese-avhubert-base_noise_pt).

This model is fine-tuned on approximately 1,300h of Japanese audio-visual dataset.

## Usage

Please install dependencies first.
```bash
$ pip install git+https://github.com/reazon-research/ReazonSpeech.git#subdirectory=pkg/avsr
```

### Using `transformers` directly

You can load AVSR models by directly using Hugging Face transformers if you trust our remote code.

```python
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

processor = AutoProcessor.from_pretrained("enactic/avista-base-plus", trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained("enactic/avista-base-plus", trust_remote_code=True)

inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model.generate(**inputs, num_beams=5, max_new_tokens=256)
transcription = processor.decode(outputs[0], skip_special_tokens=True)
```

### Using `reazonspeech.avsr` package

You can also load AVSR models by using reazonspeech.avsr. If you don't want to use remote code for security reasons for example, you can use the following code.

```python
from reazonspeech.avsr import AVHubertProcessor, AVHubertForConditionalGeneration

processor = AVHubertProcessor.from_pretrained("enactic/avista-base-plus")
model = AVHubertForConditionalGeneration.from_pretrained("enactic/avista-base-plus")

inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = processor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model.generate(**inputs, num_beams=5, max_new_tokens=256)
transcription = processor.decode(outputs[0], skip_special_tokens=True)
```

## Test Results

We report the Character Error Rate (CER) on an out-of-domain evaluation dataset that was internally collected for AVSR benchmarking.

The following table presents the benchmark results of this model and Japanese ASR models under different noise levels and noise types.

Details of the dataset and the complete benchmark results can be found [here](https://huggingface.co/datasets/enactic/avsr-leaderboard).

**+ ReazonSpeech Speech**

| Model               | #Params |  N/A   | SNR=10 |  SNR=5 |   SNR=0 |  SNR=-5 |
| :------------------ | ------: | -----: | -----: | -----: | ------: | ------: |
| AVista Base+        |   156M | 26.88% | 33.24% | 38.13% |  47.64% |  63.60% |
| reazonspeech k2     |   159M |  7.42% |  9.13% | 19.47% |  71.61% | 104.15% |
| reazonspeech nemo   |   619M |  8.50% | 11.74% | 25.38% |  77.65% | 103.42% |
| reazonspeech espnet |   118M |  7.44% |  9.20% | 16.58% |  69.34% | 103.22% |
| whisper large-v3    | 1,550M |  7.75% |  8.70% | 12.81% |  49.34% | 100.53% |
| whisper medium      |   769M | 10.07% | 13.23% | 19.21% |  50.56% |  99.27% |
| whisper small       |   244M | 10.82% | 19.82% | 28.98% |  69.69% | 108.56% |

**+ JSUT Speech**

| Model               | #Params |  N/A   | SNR=10 |  SNR=5 |   SNR=0 |  SNR=-5 |
| :------------------ | ------: | -----: | -----: | -----: | ------: | ------: |
| AVista Base+        |   156M | 26.88% | 31.56% | 34.03% |  38.85% |  47.72% |
| reazonspeech k2     |   159M |  7.42% |  8.49% | 21.94% |  70.81% |  93.04% |
| reazonspeech nemo   |   619M |  8.50% | 10.93% | 29.06% |  83.77% |  98.76% |
| reazonspeech espnet |   118M |  7.44% |  8.30% | 14.45% |  66.15% |  69.34% |
| whisper large-v3    | 1,550M |  7.75% |  8.69% | 13.03% |  60.24% |  98.67% |
| whisper medium      |   769M | 10.07% | 12.27% | 18.80% |  58.00% |  97.35% |
| whisper small       |   244M | 10.82% | 19.44% | 26.75% |  71.33% | 101.84% |

**+ Babble**

| Model               | #Params |  N/A   | SNR=10 |  SNR=5 |   SNR=0 |  SNR=-5 |
| :------------------ | ------: | -----: | -----: | -----: | ------: | ------: |
| AVista Base+        |   156M | 26.88% | 30.02% | 36.83% |  54.02% |  82.00% |
| reazonspeech k2     |   159M |  7.42% |  8.24% | 10.17% |  21.65% |  61.57% |
| reazonspeech nemo   |   619M |  8.50% | 10.40% | 14.83% |  31.74% |  77.29% |
| reazonspeech espnet |   118M |  7.44% |  8.85% | 11.75% |  24.59% |  67.27% |
| whisper large-v3    | 1,550M |  7.75% |  8.95% | 12.50% |  30.09% |  81.60% |
| whisper medium      |   769M | 10.07% | 12.52% | 18.18% |  42.27% |  95.43% |
| whisper small       |   244M | 10.82% | 19.72% | 28.24% |  56.72% | 109.61% |

**+ Music**

| Model               | #Params |  N/A   | SNR=10 |  SNR=5 |  SNR=0 |  SNR=-5 |
| :------------------ | ------: | -----: | -----: | -----: | -----: | ------: |
| AVista Base+        |   156M | 26.88% | 27.91% | 31.29% | 41.01% |  56.38% |
| reazonspeech k2     |   159M |  7.42% |  7.69% |  8.33% |  9.49% |  16.90% |
| reazonspeech nemo   |   619M |  8.50% |  9.28% |  9.97% | 13.65% |  24.61% |
| reazonspeech espnet |   118M |  7.44% |  7.86% |  8.57% | 10.41% |  16.62% |
| whisper large-v3    | 1,550M |  7.75% |  8.16% |  9.01% | 11.23% |  21.26% |
| whisper medium      |   769M | 10.07% | 11.13% | 12.97% | 16.45% |  31.62% |
| whisper small       |   244M | 10.82% | 18.02% | 19.86% | 26.82% |  47.69% |

**+ Environmental Noise**

| Model               | #Params |  N/A   | SNR=10 |  SNR=5 |  SNR=0 | SNR=-5 |
| :------------------ | ------: | -----: | -----: | -----: | -----: | -----: |
| AVista Base+        |   156M | 26.88% | 28.65% | 30.60% | 35.43% | 44.16% |
| reazonspeech k2     |   159M |  7.42% |  8.07% |  8.68% | 10.32% | 15.53% |
| reazonspeech nemo   |   619M |  8.50% |  9.31% | 10.16% | 12.71% | 18.32% |
| reazonspeech espnet |   118M |  7.44% |  8.00% |  8.63% | 10.06% | 14.54% |
| whisper large-v3    | 1,550M |  7.75% |  8.46% |  9.17% | 11.98% | 19.36% |
| whisper medium      |   769M | 10.07% | 11.77% | 13.06% | 17.04% | 24.83% |
| whisper small       |   244M | 10.82% | 17.62% | 19.84% | 25.55% | 33.77% |

## Citation

```
@misc{enactic/avista-base-plus,
  title={avista-base-plus},
  author={Sasaki, Yuta},
  url = {https://huggingface.co/enactic/avista-base-plus},
  year = {2025}
}

@article{shi2022avhubert,
    author  = {Bowen Shi and Wei-Ning Hsu and Kushal Lakhotia and Abdelrahman Mohamed},
    title = {Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction},
    journal = {arXiv preprint arXiv:2201.02184}
    year = {2022}
}

@article{shi2022avsr,
    author  = {Bowen Shi and Wei-Ning Hsu and Abdelrahman Mohamed},
    title = {Robust Self-Supervised Audio-Visual Speech Recognition},
    journal = {arXiv preprint arXiv:2201.01763}
    year = {2022}
}
```

## License

[Apache License 2.0](https://choosealicense.com/licenses/apache-2.0/)