Restore the card after the sync flattened it again
Browse files
README.md
CHANGED
|
@@ -40,6 +40,7 @@ tags:
|
|
| 40 |
pipeline_tag: automatic-speech-recognition
|
| 41 |
---
|
| 42 |
|
|
|
|
| 43 |
# Voz
|
| 44 |
|
| 45 |
Every word, with the time it was said.
|
|
@@ -47,3 +48,203 @@ Every word, with the time it was said.
|
|
| 47 |
On-device speech recognition: transcripts with word-level timestamps, 25 languages.
|
| 48 |
|
| 49 |
- **SDKs, install and examples:** https://github.com/Desert-Ant-Labs/desert-ant-core/blob/main/docs/models/voz.md
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
pipeline_tag: automatic-speech-recognition
|
| 41 |
---
|
| 42 |
|
| 43 |
+
<!-- card-header:start (generated from manifest.json, edit below this block) -->
|
| 44 |
# Voz
|
| 45 |
|
| 46 |
Every word, with the time it was said.
|
|
|
|
| 48 |
On-device speech recognition: transcripts with word-level timestamps, 25 languages.
|
| 49 |
|
| 50 |
- **SDKs, install and examples:** https://github.com/Desert-Ant-Labs/desert-ant-core/blob/main/docs/models/voz.md
|
| 51 |
+
|
| 52 |
+
<!-- card-header:end -->
|
| 53 |
+
|
| 54 |
+
Transcribes speech to text with word-level timestamps, in 25 languages, with the
|
| 55 |
+
whole graph resident on the Neural Engine. Half an hour of audio takes about
|
| 56 |
+
seven seconds on an M3 Ultra, and peak memory does not grow with the length of
|
| 57 |
+
the recording.
|
| 58 |
+
|
| 59 |
+
> `"We made the bet to build for iOS"` - each word with a start and an end, so a
|
| 60 |
+
> range is enough to cut on.
|
| 61 |
+
|
| 62 |
+
## Try it
|
| 63 |
+
|
| 64 |
+
Ships as an Apple SwiftPM package: **[Desert-Ant-Labs/desert-ant-core](https://github.com/Desert-Ant-Labs/desert-ant-core)**.
|
| 65 |
+
|
| 66 |
+
- **iOS / iPadOS / Mac Catalyst / macOS / tvOS / visionOS:** the Swift SDK (Swift
|
| 67 |
+
Package Manager). The models below are downloaded on demand and cached, so
|
| 68 |
+
nothing is bundled into your app.
|
| 69 |
+
- Apple-only. The SDK drives Core ML directly to keep the graph on the Neural
|
| 70 |
+
Engine, which has no equivalent on the other backends.
|
| 71 |
+
|
| 72 |
+
```swift
|
| 73 |
+
let voz = try await Voz()
|
| 74 |
+
let result = try await voz.transcribe(url)
|
| 75 |
+
result.text // "We made the bet to build for iOS..."
|
| 76 |
+
result.words.first?.start // 0.32
|
| 77 |
+
result.words.first?.end // 0.58
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
## Files
|
| 81 |
+
|
| 82 |
+
| File | What it is |
|
| 83 |
+
|---|---|
|
| 84 |
+
| `encoder.mlmodelc` | Acoustic encoder |
|
| 85 |
+
| `mel.mlmodelc` | Audio frontend |
|
| 86 |
+
| `decoder.mlmodelc` | Decoder |
|
| 87 |
+
| `meta.json` | Geometry the runtime reads instead of hardcoding |
|
| 88 |
+
| `vocab.json` | SentencePiece vocabulary |
|
| 89 |
+
| `embedding.f16` | Token embedding table |
|
| 90 |
+
|
| 91 |
+
Artifact names describe roles rather than the network behind them, so replacing
|
| 92 |
+
the recogniser is a new upload rather than an SDK change.
|
| 93 |
+
|
| 94 |
+
The models are shipped **compiled** (`.mlmodelc`). Keep them that way: an
|
| 95 |
+
`.mlpackage` is recompiled on every launch and loads far more slowly.
|
| 96 |
+
|
| 97 |
+
## Architecture
|
| 98 |
+
|
| 99 |
+
A three-stage Core ML cascade over a log-mel spectrogram, dispatched from Swift:
|
| 100 |
+
|
| 101 |
+
- **Frontend**: a log-mel spectrogram computed inside Core ML, normalized over the frames
|
| 102 |
+
that hold audio rather than the whole padded window.
|
| 103 |
+
- **Encoder**: a conformer-style acoustic encoder over a fixed 15 s window, producing one
|
| 104 |
+
frame every 80 ms.
|
| 105 |
+
- **Decoder**: a transducer that emits a token and a duration at each step, run with sixteen
|
| 106 |
+
independent windows batched into the lanes of a single dispatch.
|
| 107 |
+
|
| 108 |
+
Longer audio is cut into consecutive windows at pauses, transcribed independently, and
|
| 109 |
+
joined on the longest run of words two neighbouring windows agree on. Every stage runs on
|
| 110 |
+
the Neural Engine with no CPU or GPU fallback.
|
| 111 |
+
|
| 112 |
+
## Inputs and outputs
|
| 113 |
+
|
| 114 |
+
- **Input:** mono audio at any sample rate; the SDK resamples and downmixes.
|
| 115 |
+
- **Output:** the transcript, plus every word with a start and an end in seconds.
|
| 116 |
+
|
| 117 |
+
## Accuracy
|
| 118 |
+
|
| 119 |
+
Ten minutes of audio on an M3 Ultra, release build, warm:
|
| 120 |
+
|
| 121 |
+
| | |
|
| 122 |
+
|---|---|
|
| 123 |
+
| Speed | **2.1 s for 611 s of audio (~290x real time)** on long files; 50-62x transcribing single short utterances, where every clip pays for a full 15 s window |
|
| 124 |
+
| Word error rate | **7.40%** averaged over six Open ASR Leaderboard sets, against 7.00% for Whisper large-v3-turbo |
|
| 125 |
+
| Long-form word error rate | **2.83%** on half an hour of narration scored against the book, against 2.72% for Whisper large-v3-turbo on the same span |
|
| 126 |
+
| Word timestamps | starts 83 ms, ends 95 ms mean absolute error against a forced aligner |
|
| 127 |
+
| Neural Engine residency | 100%, with no CPU or GPU fallback |
|
| 128 |
+
| Size on disk | 467 MB |
|
| 129 |
+
| Load | ~0.2 s warm; ~20 s once per install while Core ML specializes |
|
| 130 |
+
|
| 131 |
+
### English, on the Open ASR Leaderboard
|
| 132 |
+
|
| 133 |
+
Scored on the [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
|
| 134 |
+
datasets with its own text normalizer. Whisper's figures are the leaderboard's,
|
| 135 |
+
on the same dataset configurations.
|
| 136 |
+
|
| 137 |
+
| dataset | Voz | Whisper large-v3-turbo |
|
| 138 |
+
|---|---:|---:|
|
| 139 |
+
| LibriSpeech test-clean | 2.19% | 2.13% |
|
| 140 |
+
| LibriSpeech test-other | 3.86% | 3.70% |
|
| 141 |
+
| GigaSpeech | 9.70% | 8.47% |
|
| 142 |
+
| SPGISpeech | 3.86% | 2.79% |
|
| 143 |
+
| Earnings-22 | 12.97% | 11.07% |
|
| 144 |
+
| AMI | 11.84% | **13.87%** |
|
| 145 |
+
| **average** | **7.40%** | **7.00%** |
|
| 146 |
+
|
| 147 |
+
Close overall, two points better on meetings, behind on prepared and read
|
| 148 |
+
speech. It gets there in 467 MB entirely on the Neural Engine, against 1.6 GB
|
| 149 |
+
for Whisper large-v3-turbo.
|
| 150 |
+
|
| 151 |
+
**Expect the conversational figures, not the LibriSpeech one.** Read speech in a
|
| 152 |
+
clean recording scores around 2%; meetings, earnings calls and podcast audio
|
| 153 |
+
score 10-13%, and most real material is nearer the second group. Roughly one word
|
| 154 |
+
in ten wanting a look is the honest expectation for a podcast.
|
| 155 |
+
|
| 156 |
+
VoxPopuli and TEDLIUM are omitted: the leaderboard's scripts name a 628-utterance
|
| 157 |
+
shard of the former, which is not the audio its published figure measures, and
|
| 158 |
+
the latter's config carries no rows.
|
| 159 |
+
|
| 160 |
+
### Word timestamps
|
| 161 |
+
|
| 162 |
+
Every word carries a start and an end, scored against [torchaudio's MMS_FA
|
| 163 |
+
forced aligner](https://pytorch.org/audio/stable/generated/torchaudio.pipelines.MMS_FA.html):
|
| 164 |
+
|
| 165 |
+
| | words | start | end | ends within 80 ms | within 200 ms |
|
| 166 |
+
|---|---:|---:|---:|---:|---:|
|
| 167 |
+
| English, LibriSpeech | 6295 | 83 ms | 95 ms | 60% | 90% |
|
| 168 |
+
| German, FLEURS | 939 | 80 ms | 92 ms | 62% | 92% |
|
| 169 |
+
|
| 170 |
+
Ends are the harder of the two. The recogniser reports how far to skip after
|
| 171 |
+
each token rather than where a word stops, which overshoots into the pause that
|
| 172 |
+
follows it, so ends are trimmed back using the audio. Forced alignment is itself
|
| 173 |
+
approximate at the tens of milliseconds level, so treat these as agreement with a
|
| 174 |
+
good aligner rather than absolute truth. Much of the residual is the 80 ms frame
|
| 175 |
+
resolution, which is the floor for any word time this export produces.
|
| 176 |
+
|
| 177 |
+
### Every supported language, on long audio
|
| 178 |
+
|
| 179 |
+
Ten minutes per language, built by concatenating [FLEURS](https://huggingface.co/datasets/google/fleurs)
|
| 180 |
+
test utterances so that each file crosses about forty analysis boundaries. A
|
| 181 |
+
single FLEURS utterance is shorter than one window and so measures nothing about
|
| 182 |
+
how consecutive windows are joined, which is most of what happens on real
|
| 183 |
+
material. 4.2 hours in total.
|
| 184 |
+
|
| 185 |
+
| | WER | | WER | | WER |
|
| 186 |
+
|---|---:|---|---:|---|---:|
|
| 187 |
+
| it | 3.31% | cs | 14.27% | hu | 21.26% |
|
| 188 |
+
| pt | 6.08% | sk | 15.42% | et | 21.37% |
|
| 189 |
+
| uk | 6.40% | hr | 16.92% | sv | 21.67% |
|
| 190 |
+
| ru | 6.57% | fi | 18.17% | mt | 22.01% |
|
| 191 |
+
| en | 7.36% | ro | 20.95% | da | 24.32% |
|
| 192 |
+
| de | 8.12% | | | lt | 26.89% |
|
| 193 |
+
| es | 9.01% | | | lv | 30.57% |
|
| 194 |
+
| nl | 9.84% | | | sl | 33.85% |
|
| 195 |
+
| pl | 9.99% | | | el | 39.46% |
|
| 196 |
+
| bg | 12.39% | | | | |
|
| 197 |
+
| fr | 12.78% | | | | |
|
| 198 |
+
|
| 199 |
+
Aggregate 16.58%, median 283x real time. Throughput varies only 12% across
|
| 200 |
+
languages, because cost follows how much audio there is rather than what is in
|
| 201 |
+
it.
|
| 202 |
+
|
| 203 |
+
Read the spread before choosing a language. The nine best are usable as-is; the
|
| 204 |
+
tail above 20% will cost more to correct than to retype for many uses, and
|
| 205 |
+
reflects where the underlying recogniser is already known to be weak rather than
|
| 206 |
+
anything specific to this export. This is read speech in clean recordings, so
|
| 207 |
+
treat it as a ranking rather than as a number to expect on your own audio.
|
| 208 |
+
|
| 209 |
+
## Languages
|
| 210 |
+
|
| 211 |
+
Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek,
|
| 212 |
+
Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian,
|
| 213 |
+
Slovak, Slovenian, Spanish, Swedish, and Ukrainian.
|
| 214 |
+
|
| 215 |
+
Accuracy varies widely across them; see the per-language table above before choosing one.
|
| 216 |
+
|
| 217 |
+
## Built on
|
| 218 |
+
|
| 219 |
+
- [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) - NVIDIA
|
| 220 |
+
(CC BY 4.0): the base recogniser, converted to Core ML and compressed for this export.
|
| 221 |
+
Weight values are otherwise unchanged.
|
| 222 |
+
- [FLEURS](https://huggingface.co/datasets/google/fleurs) (CC BY 4.0): evaluation audio for
|
| 223 |
+
the per-language table above.
|
| 224 |
+
|
| 225 |
+
See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). FLEURS is not redistributed here.
|
| 226 |
+
|
| 227 |
+
<!-- card-footer:start (generated from manifest.json, edit above this block) -->
|
| 228 |
+
## License
|
| 229 |
+
|
| 230 |
+
[Desert Ant Labs Source-Available License](https://license.desertant.com/1.0). Free for most
|
| 231 |
+
apps, and a commercial license is required at scale. Full terms are at the link.
|
| 232 |
+
Licensing: <licensing@desertant.com>.
|
| 233 |
+
|
| 234 |
+
See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md).
|
| 235 |
+
|
| 236 |
+
## Citation
|
| 237 |
+
|
| 238 |
+
```bibtex
|
| 239 |
+
@software{voz_2026,
|
| 240 |
+
title = {Voz: On-device speech recognition: transcripts with word-level timestamps, 25 languages},
|
| 241 |
+
author = {Desert Ant Labs},
|
| 242 |
+
year = {2026},
|
| 243 |
+
url = {https://huggingface.co/desert-ant-labs/voz},
|
| 244 |
+
}
|
| 245 |
+
```
|
| 246 |
+
|
| 247 |
+
---
|
| 248 |
+
|
| 249 |
+
© 2026 Desert Ant Labs · <https://desertant.com>
|
| 250 |
+
<!-- card-footer:end -->
|