README.md

---
language:
- ja
- multilingual
tags:
- voice-activity-detection
- vad
- whisper
- onnx
- speech-detection
- audio-classification
- asmr
- japanese
- whispered-speech
license: mit
base_model: openai/whisper-base
library_name: transformers
pipeline_tag: audio-classification
---

# Whisper-base Voice Activity Detection (VAD) for Japanese ASMR - ONNX

## Model Description

This is a refined Whisper-based Voice Activity Detection (VAD) model that leverages the pre-trained Whisper encoder with a lightweight non-autoregressive decoder for high-precision speech activity detection. While fine-tuned on Japanese ASMR content for optimal performance on soft speech and whispers, the model retains Whisper's robust multilingual foundation, enabling effective speech detection across diverse languages and acoustic conditions. It has been optimized and exported to ONNX format for efficient inference across different platforms. For the full training code, configs, and ONNX export utilities, see the GitHub repository: [TransWithAI/whisper-vad](https://github.com/TransWithAI/whisper-vad).

This work builds upon recent research demonstrating the positive transfer of Whisper's speech representations to VAD tasks, as shown in [WhisperSeg](https://github.com/nianlonggu/WhisperSeg) and related work.

### Key Features

- **Architecture**: Encoder-Decoder model based on whisper-base
- **Frame Resolution**: 20ms per frame for precise temporal detection
- **Input Duration**: Processes 30-second audio chunks
- **Output**: Frame-level speech/non-speech predictions
- **Optimized**: ONNX format for cross-platform deployment
- **Real-time capable**: Fast non-autoregressive inference

### Model Architecture Details

- **Base Model**: OpenAI whisper-base encoder (frozen during training)
- **Decoder**: 2-layer transformer decoder with 8 attention heads
- **Processing**:
  - Input: 30-second audio chunks (480,000 samples @ 16kHz)
  - Features: 80-channel log-mel spectrogram
  - Output: 1500 frame predictions (20ms per frame)

## Performance

- **Frame Duration**: 20ms per frame for precise temporal detection
- **Processing Speed**: ~100x real-time on CPU (single-sample processing)
- **Batch Processing**: Currently limited to batch size of 1 due to ONNX export constraints, but single-sample inference is extremely fast
- **Specialized Training**: Japanese ASMR and whispered speech
- **Generalization**: Despite being fine-tuned on Japanese ASMR, the model inherits Whisper's strong multilingual capabilities and can effectively detect speech in various languages and acoustic environments

### Advantages over Native Whisper VAD

- **No hallucinations**: Discriminative model cannot generate spurious text
- **Much faster**: Single forward pass, non-autoregressive inference
- **Higher precision**: 20ms frame-level temporal resolution vs Whisper's 30s chunks
- **Robust**: Focal loss training handles speech/silence imbalance effectively
- **Lightweight**: Decoder adds minimal parameters to base Whisper encoder

## Usage

### Quick Start with ONNX Runtime

```python
import numpy as np
import onnxruntime as ort
from transformers import WhisperFeatureExtractor
import librosa

# Load model
session = ort.InferenceSession("model.onnx")
feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-base")

# Load and preprocess audio
audio, sr = librosa.load("audio.wav", sr=16000)
audio_chunk = audio[:480000]  # 30 seconds

# Extract features
inputs = feature_extractor(
    audio_chunk,
    sampling_rate=16000,
    return_tensors="np"
)

# Run inference
outputs = session.run(None, {session.get_inputs()[0].name: inputs.input_features})
predictions = outputs[0]  # Shape: [1, 1500] - 1500 frames of 20ms each

# Apply threshold
speech_frames = predictions[0] > 0.5
```

### Using the Provided Inference Script

The model repository includes a comprehensive `inference.py` script with advanced features:

```python
from inference import WhisperVADInference

# Initialize model
vad = WhisperVADInference(
    model_path="model.onnx",
    threshold=0.5,           # Speech detection threshold
    min_speech_duration=0.25,  # Minimum speech segment duration
    min_silence_duration=0.1   # Minimum silence between segments
)

# Process audio file
segments = vad.process_audio("audio.wav")

# Segments format: List of (start_time, end_time) tuples
for start, end in segments:
    print(f"Speech detected: {start:.2f}s - {end:.2f}s")
```

### Streaming/Real-time Processing

```python
# Process audio stream in chunks
vad = WhisperVADInference("model.onnx", streaming=True)

for audio_chunk in audio_stream:
    speech_active = vad.process_chunk(audio_chunk)
    if speech_active:
        # Handle speech detection
        pass
```

## Input/Output Specifications

### Input
- **Audio Format**: 16kHz mono audio
- **Chunk Size**: 30 seconds (480,000 samples)
- **Feature Type**: 80-channel log-mel spectrogram
- **Shape**: `[1, 80, 3000]` (batch size fixed to 1 - see note below)

### Output
- **Type**: Frame-level probabilities
- **Shape**: `[1, 1500]` (batch size fixed to 1)
- **Frame Duration**: 20ms per frame
- **Range**: [0, 1] probability of speech presence

**Note on Batch Processing**: Currently, the ONNX model only supports batch size of 1 due to export limitations between PyTorch transformers and ONNX. However, single-sample inference is highly optimized and runs extremely fast (~100x real-time on CPU), making sequential processing still very efficient for most use cases.

## Training Details

### Training Configuration
- **Dataset**: ~500 hours Japanese ASMR audio recordings with accurate speech timestamps
- **Loss Function**: Focal Loss (α=0.25, γ=2.0) for class imbalance
- **Optimizer**: AdamW with learning rate 1.5e-3
- **Batch Size**: 128
- **Training Duration**: 5 epochs
- **Hardware**: Single GPU training with mixed precision (bf16)

### Data Processing
- Audio segmented into 30-second chunks
- Frame-level labels generated from word-level timestamps
- Augmentation: None (relying on Whisper's pre-training robustness)

## Limitations and Considerations

1. **Fixed Duration**: Model expects 30-second chunks; shorter audio needs padding
2. **Training Specialization**: While the model performs well across languages and environments due to Whisper's strong multilingual foundation, it excels particularly at:
   - Japanese ASMR content (primary training data)
   - Whispered and soft speech detection
   - Quiet, intimate audio environments
3. **Generalization**: The model can effectively handle various languages and normal speech volumes, though performance may be slightly better on content similar to the training data
4. **Background Noise**: Performance may degrade in very noisy conditions
5. **Music/Singing**: Primarily trained on speech; may have variable performance on singing

## Model Files

- `model.onnx`: ONNX model file
- `model_metadata.json`: Model configuration and parameters
- `inference.py`: Ready-to-use inference script with post-processing
- `requirements.txt`: Python dependencies

## Installation

```bash
pip install onnxruntime  # or onnxruntime-gpu for GPU support
pip install librosa transformers numpy
```

## Applications

- **ASMR Content Processing**: Detect whispered speech and subtle vocalizations in ASMR recordings
- **Japanese Audio Processing**: Optimized for Japanese language content, especially soft speech
- **Transcription Pre-processing**: Filter out silence before ASR, particularly effective for whispered content
- **Audio Indexing**: Identify speech segments in long recordings
- **Real-time Communication**: Detect active speech in calls/meetings
- **Audio Analytics**: Speech/silence ratio analysis for ASMR and meditation content
- **Subtitle Alignment**: Accurate timing for subtitles, including whispered dialogue

## Citation

If you use this model, please cite:

```bibtex
@misc{whisper-vad,
  title={Whisper-VAD: Whisper-based Voice Activity Detection},
  author={Grider},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/TransWithAI/Whisper-Vad-EncDec-ASMR-onnx}}
}
```

## References

- Original Whisper Paper: [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)
- WhisperSeg: [Positive Transfer of the Whisper Speech Transformer to Human and Animal Voice Activity Detection](https://doi.org/10.1101/2023.09.30.560270)
  - GitHub: [https://github.com/nianlonggu/WhisperSeg](https://github.com/nianlonggu/WhisperSeg)

## License

MIT License

## Acknowledgments

This model builds upon OpenAI's Whisper model and implements architectural refinements for efficient voice activity detection.