pveugen commited on
Commit
ca2eae2
·
verified ·
1 Parent(s): ce46c49

Restore the card after the sync flattened it again

Browse files
Files changed (1) hide show
  1. README.md +201 -0
README.md CHANGED
@@ -40,6 +40,7 @@ tags:
40
  pipeline_tag: automatic-speech-recognition
41
  ---
42
 
 
43
  # Voz
44
 
45
  Every word, with the time it was said.
@@ -47,3 +48,203 @@ Every word, with the time it was said.
47
  On-device speech recognition: transcripts with word-level timestamps, 25 languages.
48
 
49
  - **SDKs, install and examples:** https://github.com/Desert-Ant-Labs/desert-ant-core/blob/main/docs/models/voz.md
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  pipeline_tag: automatic-speech-recognition
41
  ---
42
 
43
+ <!-- card-header:start (generated from manifest.json, edit below this block) -->
44
  # Voz
45
 
46
  Every word, with the time it was said.
 
48
  On-device speech recognition: transcripts with word-level timestamps, 25 languages.
49
 
50
  - **SDKs, install and examples:** https://github.com/Desert-Ant-Labs/desert-ant-core/blob/main/docs/models/voz.md
51
+
52
+ <!-- card-header:end -->
53
+
54
+ Transcribes speech to text with word-level timestamps, in 25 languages, with the
55
+ whole graph resident on the Neural Engine. Half an hour of audio takes about
56
+ seven seconds on an M3 Ultra, and peak memory does not grow with the length of
57
+ the recording.
58
+
59
+ > `"We made the bet to build for iOS"` - each word with a start and an end, so a
60
+ > range is enough to cut on.
61
+
62
+ ## Try it
63
+
64
+ Ships as an Apple SwiftPM package: **[Desert-Ant-Labs/desert-ant-core](https://github.com/Desert-Ant-Labs/desert-ant-core)**.
65
+
66
+ - **iOS / iPadOS / Mac Catalyst / macOS / tvOS / visionOS:** the Swift SDK (Swift
67
+ Package Manager). The models below are downloaded on demand and cached, so
68
+ nothing is bundled into your app.
69
+ - Apple-only. The SDK drives Core ML directly to keep the graph on the Neural
70
+ Engine, which has no equivalent on the other backends.
71
+
72
+ ```swift
73
+ let voz = try await Voz()
74
+ let result = try await voz.transcribe(url)
75
+ result.text // "We made the bet to build for iOS..."
76
+ result.words.first?.start // 0.32
77
+ result.words.first?.end // 0.58
78
+ ```
79
+
80
+ ## Files
81
+
82
+ | File | What it is |
83
+ |---|---|
84
+ | `encoder.mlmodelc` | Acoustic encoder |
85
+ | `mel.mlmodelc` | Audio frontend |
86
+ | `decoder.mlmodelc` | Decoder |
87
+ | `meta.json` | Geometry the runtime reads instead of hardcoding |
88
+ | `vocab.json` | SentencePiece vocabulary |
89
+ | `embedding.f16` | Token embedding table |
90
+
91
+ Artifact names describe roles rather than the network behind them, so replacing
92
+ the recogniser is a new upload rather than an SDK change.
93
+
94
+ The models are shipped **compiled** (`.mlmodelc`). Keep them that way: an
95
+ `.mlpackage` is recompiled on every launch and loads far more slowly.
96
+
97
+ ## Architecture
98
+
99
+ A three-stage Core ML cascade over a log-mel spectrogram, dispatched from Swift:
100
+
101
+ - **Frontend**: a log-mel spectrogram computed inside Core ML, normalized over the frames
102
+ that hold audio rather than the whole padded window.
103
+ - **Encoder**: a conformer-style acoustic encoder over a fixed 15 s window, producing one
104
+ frame every 80 ms.
105
+ - **Decoder**: a transducer that emits a token and a duration at each step, run with sixteen
106
+ independent windows batched into the lanes of a single dispatch.
107
+
108
+ Longer audio is cut into consecutive windows at pauses, transcribed independently, and
109
+ joined on the longest run of words two neighbouring windows agree on. Every stage runs on
110
+ the Neural Engine with no CPU or GPU fallback.
111
+
112
+ ## Inputs and outputs
113
+
114
+ - **Input:** mono audio at any sample rate; the SDK resamples and downmixes.
115
+ - **Output:** the transcript, plus every word with a start and an end in seconds.
116
+
117
+ ## Accuracy
118
+
119
+ Ten minutes of audio on an M3 Ultra, release build, warm:
120
+
121
+ | | |
122
+ |---|---|
123
+ | Speed | **2.1 s for 611 s of audio (~290x real time)** on long files; 50-62x transcribing single short utterances, where every clip pays for a full 15 s window |
124
+ | Word error rate | **7.40%** averaged over six Open ASR Leaderboard sets, against 7.00% for Whisper large-v3-turbo |
125
+ | Long-form word error rate | **2.83%** on half an hour of narration scored against the book, against 2.72% for Whisper large-v3-turbo on the same span |
126
+ | Word timestamps | starts 83 ms, ends 95 ms mean absolute error against a forced aligner |
127
+ | Neural Engine residency | 100%, with no CPU or GPU fallback |
128
+ | Size on disk | 467 MB |
129
+ | Load | ~0.2 s warm; ~20 s once per install while Core ML specializes |
130
+
131
+ ### English, on the Open ASR Leaderboard
132
+
133
+ Scored on the [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
134
+ datasets with its own text normalizer. Whisper's figures are the leaderboard's,
135
+ on the same dataset configurations.
136
+
137
+ | dataset | Voz | Whisper large-v3-turbo |
138
+ |---|---:|---:|
139
+ | LibriSpeech test-clean | 2.19% | 2.13% |
140
+ | LibriSpeech test-other | 3.86% | 3.70% |
141
+ | GigaSpeech | 9.70% | 8.47% |
142
+ | SPGISpeech | 3.86% | 2.79% |
143
+ | Earnings-22 | 12.97% | 11.07% |
144
+ | AMI | 11.84% | **13.87%** |
145
+ | **average** | **7.40%** | **7.00%** |
146
+
147
+ Close overall, two points better on meetings, behind on prepared and read
148
+ speech. It gets there in 467 MB entirely on the Neural Engine, against 1.6 GB
149
+ for Whisper large-v3-turbo.
150
+
151
+ **Expect the conversational figures, not the LibriSpeech one.** Read speech in a
152
+ clean recording scores around 2%; meetings, earnings calls and podcast audio
153
+ score 10-13%, and most real material is nearer the second group. Roughly one word
154
+ in ten wanting a look is the honest expectation for a podcast.
155
+
156
+ VoxPopuli and TEDLIUM are omitted: the leaderboard's scripts name a 628-utterance
157
+ shard of the former, which is not the audio its published figure measures, and
158
+ the latter's config carries no rows.
159
+
160
+ ### Word timestamps
161
+
162
+ Every word carries a start and an end, scored against [torchaudio's MMS_FA
163
+ forced aligner](https://pytorch.org/audio/stable/generated/torchaudio.pipelines.MMS_FA.html):
164
+
165
+ | | words | start | end | ends within 80 ms | within 200 ms |
166
+ |---|---:|---:|---:|---:|---:|
167
+ | English, LibriSpeech | 6295 | 83 ms | 95 ms | 60% | 90% |
168
+ | German, FLEURS | 939 | 80 ms | 92 ms | 62% | 92% |
169
+
170
+ Ends are the harder of the two. The recogniser reports how far to skip after
171
+ each token rather than where a word stops, which overshoots into the pause that
172
+ follows it, so ends are trimmed back using the audio. Forced alignment is itself
173
+ approximate at the tens of milliseconds level, so treat these as agreement with a
174
+ good aligner rather than absolute truth. Much of the residual is the 80 ms frame
175
+ resolution, which is the floor for any word time this export produces.
176
+
177
+ ### Every supported language, on long audio
178
+
179
+ Ten minutes per language, built by concatenating [FLEURS](https://huggingface.co/datasets/google/fleurs)
180
+ test utterances so that each file crosses about forty analysis boundaries. A
181
+ single FLEURS utterance is shorter than one window and so measures nothing about
182
+ how consecutive windows are joined, which is most of what happens on real
183
+ material. 4.2 hours in total.
184
+
185
+ | | WER | | WER | | WER |
186
+ |---|---:|---|---:|---|---:|
187
+ | it | 3.31% | cs | 14.27% | hu | 21.26% |
188
+ | pt | 6.08% | sk | 15.42% | et | 21.37% |
189
+ | uk | 6.40% | hr | 16.92% | sv | 21.67% |
190
+ | ru | 6.57% | fi | 18.17% | mt | 22.01% |
191
+ | en | 7.36% | ro | 20.95% | da | 24.32% |
192
+ | de | 8.12% | | | lt | 26.89% |
193
+ | es | 9.01% | | | lv | 30.57% |
194
+ | nl | 9.84% | | | sl | 33.85% |
195
+ | pl | 9.99% | | | el | 39.46% |
196
+ | bg | 12.39% | | | | |
197
+ | fr | 12.78% | | | | |
198
+
199
+ Aggregate 16.58%, median 283x real time. Throughput varies only 12% across
200
+ languages, because cost follows how much audio there is rather than what is in
201
+ it.
202
+
203
+ Read the spread before choosing a language. The nine best are usable as-is; the
204
+ tail above 20% will cost more to correct than to retype for many uses, and
205
+ reflects where the underlying recogniser is already known to be weak rather than
206
+ anything specific to this export. This is read speech in clean recordings, so
207
+ treat it as a ranking rather than as a number to expect on your own audio.
208
+
209
+ ## Languages
210
+
211
+ Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek,
212
+ Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian,
213
+ Slovak, Slovenian, Spanish, Swedish, and Ukrainian.
214
+
215
+ Accuracy varies widely across them; see the per-language table above before choosing one.
216
+
217
+ ## Built on
218
+
219
+ - [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) - NVIDIA
220
+ (CC BY 4.0): the base recogniser, converted to Core ML and compressed for this export.
221
+ Weight values are otherwise unchanged.
222
+ - [FLEURS](https://huggingface.co/datasets/google/fleurs) (CC BY 4.0): evaluation audio for
223
+ the per-language table above.
224
+
225
+ See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). FLEURS is not redistributed here.
226
+
227
+ <!-- card-footer:start (generated from manifest.json, edit above this block) -->
228
+ ## License
229
+
230
+ [Desert Ant Labs Source-Available License](https://license.desertant.com/1.0). Free for most
231
+ apps, and a commercial license is required at scale. Full terms are at the link.
232
+ Licensing: <licensing@desertant.com>.
233
+
234
+ See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md).
235
+
236
+ ## Citation
237
+
238
+ ```bibtex
239
+ @software{voz_2026,
240
+ title = {Voz: On-device speech recognition: transcripts with word-level timestamps, 25 languages},
241
+ author = {Desert Ant Labs},
242
+ year = {2026},
243
+ url = {https://huggingface.co/desert-ant-labs/voz},
244
+ }
245
+ ```
246
+
247
+ ---
248
+
249
+ © 2026 Desert Ant Labs · <https://desertant.com>
250
+ <!-- card-footer:end -->