File size: 3,971 Bytes
df8717e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | ---
license: apache-2.0
base_model: FireRedTeam/FireRedVAD
tags:
- voice-activity-detection
- vad
- coreml
- apple
- ios
- macos
- streaming
- real-time
- dfsmn
- firered
pipeline_tag: voice-activity-detection
library_name: coremltools
language:
- multilingual
---
# FireRedVAD-CoreML
Core ML conversion of [FireRedVAD](https://huggingface.co/FireRedTeam/FireRedVAD) Stream-VAD for real-time voice activity detection on Apple platforms (iOS 16+ / macOS 13+). Converted from the original PyTorch model by [FireRedTeam/FireRedVAD](https://huggingface.co/FireRedTeam/FireRedVAD).
## Model Description
- **Original model:** FireRedVAD by Xiaohongshu (小红书) FireRedTeam
- **Architecture:** DFSMN (Deep Feedforward Sequential Memory Network) — 8 DFSMN blocks + 1 DNN layer
- **Variant:** Stream-VAD (causal, lookahead=0), suitable for real-time streaming
- **Parameters:** ~568K (extremely lightweight)
- **Model size:** 2.2 MB (FP32)
- **Input:** 80-dim log-Mel filterbank features (16kHz, 25ms frame, 10ms shift)
- **Output:** Speech probability [0, 1] per frame
- **Language support:** 100+ languages, 20+ Chinese dialects
## Performance
Results from the FLEURS-VAD-102 benchmark (102 languages, 9,443 audio clips):
| Metric | FireRedVAD | Silero-VAD | TEN-VAD | FunASR-VAD | WebRTC-VAD |
|--------|-----------|-----------|---------|-----------|-----------|
| AUC-ROC | **99.60** | 97.99 | 97.81 | - | - |
| F1 Score | **97.57** | 95.95 | 95.19 | 90.91 | 52.30 |
| False Alarm | **2.69%** | 9.41% | 15.47% | 44.03% | 2.83% |
| Miss Rate | 3.62% | 3.95% | 2.95% | 0.42% | 64.15% |
## Core ML Model Specification
### Inputs
| Name | Shape | Type | Description |
|------|-------|------|-------------|
| `feat` | `[1, 1..512, 80]` | Float32 | Log-Mel filterbank features (dynamic time axis) |
| `cache_0` ~ `cache_7` | `[1, 128, 19]` | Float32 | FSMN lookback cache for each of the 8 layers |
### Outputs
| Name | Type | Description |
|------|------|-------------|
| `probs` | Float32 | Speech probability, shape `[1, T, 1]` |
| `new_cache_0` ~ `new_cache_7` | Float32 | Updated lookback cache |
- **Minimum deployment target:** iOS 16 / macOS 13
- **Compute units:** CPU + Neural Engine
## Conversion
Converted from PyTorch using [coremltools](https://github.com/apple/coremltools) via the export script in [FireRedASR2S](https://github.com/FireRedTeam/FireRedASR2S). The Stream-VAD variant was selected for its causal (no lookahead) property, making it suitable for real-time streaming applications.
## Usage
```swift
import CoreML
// Load model
let model = try FireRedVAD(configuration: .init())
// Initialize caches (8 layers x [1, 128, 19])
var caches = (0..<8).map { _ in
try! MLMultiArray(shape: [1, 128, 19], dataType: .float32)
}
// Process audio frame by frame
let input = FireRedVADInput(
feat: fbankFeatures, // [1, T, 80]
cache_0: caches[0], cache_1: caches[1],
cache_2: caches[2], cache_3: caches[3],
cache_4: caches[4], cache_5: caches[5],
cache_6: caches[6], cache_7: caches[7]
)
let output = try model.prediction(input: input)
let speechProb = output.probs // [1, T, 1]
// Update caches for next frame
caches = [
output.new_cache_0, output.new_cache_1,
output.new_cache_2, output.new_cache_3,
output.new_cache_4, output.new_cache_5,
output.new_cache_6, output.new_cache_7
]
```
For a complete implementation with feature extraction, CMVN normalization, and speech state machine, see [FireRedASRKit](https://github.com/leaker/firered_asr).
## References
- [FireRedVAD (Original Model)](https://huggingface.co/FireRedTeam/FireRedVAD)
- [FireRedASR2S GitHub](https://github.com/FireRedTeam/FireRedASR2S)
- [FireRedASR Paper (arXiv:2501.14350)](https://arxiv.org/abs/2501.14350)
- [DFSMN Paper (arXiv:1803.05030)](https://arxiv.org/abs/1803.05030)
## License
Apache 2.0, following the original [FireRedVAD](https://huggingface.co/FireRedTeam/FireRedVAD) license.
|