File size: 18,501 Bytes
a739ab0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
505d1ed
a739ab0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
d422588
a739ab0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
505d1ed
a739ab0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
505d1ed
a739ab0
505d1ed
a739ab0
505d1ed
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
---
license: other
license_name: webai-non-commercial-license-ver.-1.0
license_link: https://huggingface.co/webAI-Official/Parakeet-Air/blob/main/LICENSE.md
base_model: nvidia/parakeet-tdt-0.6b-v3
base_model_relation: quantized
pipeline_tag: automatic-speech-recognition
library_name: gguf
language:
 - bg
 - cs
 - da
 - de
 - el
 - en
 - es
 - et
 - fi
 - fr
 - hr
 - hu
 - it
 - lt
 - lv
 - mt
 - nl
 - pl
 - pt
 - ro
 - ru
 - sk
 - sl
 - sv
 - uk
tags:
 - gguf
 - automatic-speech-recognition
 - speech-recognition
 - parakeet
 - transcribe.cpp
 - multilingual
 - on-device
---

# Parakeet-Air

A 4-bit build of NVIDIA's [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) speech
recognition model for 25 European languages, in GGUF for [transcribe.cpp](https://github.com/handy-computer/transcribe.cpp).
It has the same size (485,425,504 bytes) and the same tensor types as handy-computer's widely used
`parakeet-tdt-0.6b-v3-Q4_K_M.gguf`, and it loads in unmodified transcribe.cpp. In a pre-registered test on 22,766
utterances (FLEURS in 25 languages plus LibriSpeech), its word error rate was **within 0.5 points of full precision
(+0.21 points)** where handy-computer's Q4_K_M file is +0.47, and its transcripts were **26% closer to full
precision's** than that file's.

> **Model Released under the webAI Non-Commercial License ver. 1.0; see** [License and attribution](#license-and-attribution).

### At a glance

Passes all four rules of the pre-registered test.

| | F32 (full precision) | handy-computer Q4_K_M | **Parakeet-Air** |
|---|---:|---:|---:|
| File size | 2,508,435,616 B | 485,425,504 B | **485,425,504 B** |
| Word error rate, all 22,766 utterances | 11.59% | 12.06% | **11.80%** |
| Word error rate, FLEURS test, 25 languages | 12.80% | 13.33% | **13.04%** |
| Word error rate, LibriSpeech test-clean | 1.96% | 1.98% | **1.94%** |
| Words that differ from F32's transcript, per 100 words | 0 (itself) | 4.81 | **3.54** |
| Test sets where it beats handy-computer's Q4_K_M (95% interval below zero) | | | **8 of 26, worse in none** |

<sub>transcribe.cpp `5700f127`, CUDA backend on an NVIDIA H200, greedy decoding, timestamps and inverse text
normalization off; Whisper text normalizers; one run per file. Word error rates are pooled (total errors over total
words). See [Evaluation protocol](#evaluation-protocol).</sub>

![Word error rate for F32, handy-computer's Q4_K_M and Parakeet-Air](figures/benchmarks.png)

## Highlights

* **Within half a point of full precision.** Pooled word error rate 11.80% vs F32's 11.59%, +0.21 points (95% CI
  +0.16 to +0.26), inside the pre-registered 0.5-point margin.
* **Lower word error than handy-computer's Q4_K_M.** βˆ’0.26 points (95% CI βˆ’0.32 to βˆ’0.20) overall, about 55% of that
  file's increase over F32 removed. It is significantly lower in 8 test sets (Czech, Danish, Croatian, Lithuanian,
  Latvian, Maltese, Dutch, Romanian) and significantly higher in none.
* **Closer to full precision's transcripts.** 3.54 words per 100 differ from F32's transcript, against 4.81 for
  handy-computer's Q4_K_M: βˆ’26% (word edits per sentence βˆ’0.70, 95% CI βˆ’0.74 to βˆ’0.66), lower in all 26 test sets.
* **Same size, same runtime.** Same bytes and tensor types as handy-computer's Q4_K_M; it loads in stock
  transcribe.cpp with no code changes.

It is **not identical to full precision**: its word error rate is still significantly higher than F32's overall and
in 10 of the 26 test sets, and handy-computer's 740 MB Q8_0 shows no measurable loss. See [Limitations](#limitations).

## Model Details

| Property | Value |
|---|---|
| Model ID | `webAI-Official/Parakeet-Air` (proposed; not yet published) |
| Base model | [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), 0.6B parameters (FastConformer encoder, TDT decoder), via handy-computer's F32 GGUF conversion in [`handy-computer/parakeet-tdt-0.6b-v3-gguf`](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf) |
| Model file | `Parakeet-Air-Q4_K_M.gguf`, 485,425,504 bytes (sha256 `14c12436640ccdd6…`) |
| Format | GGUF, file type Q4_K_M: 194 matrices in Q4_K, 31 in Q8_0 (attention output, prediction network, joint network), the 48 pointwise convolutions in F16 and the remaining small tensors in F32, exactly as in handy-computer's Q4_K_M. The file's metadata is the same as handy-computer's, so it reports its name as "Parakeet TDT 0.6B v3" |
| Runs on | transcribe.cpp. Evaluated with the CUDA backend (commit `5700f127`, NVIDIA H200); also run with the Metal backend on Apple silicon |
| Input / output | 16 kHz mono audio in any of the 25 languages / text with punctuation and capitalization; language detected automatically |
| License | webAI Non-Commercial License ver. 1.0; based on NVIDIA's CC-BY-4.0 weights |

## Results

Every result comes from one test, run once per file: each file transcribed the same 22,766 utterances (FLEURS test in
25 languages, 20,146 utterances, plus LibriSpeech test-clean, 2,620) with stock transcribe.cpp on the same GPU.
Verdicts pair the same utterances and use a 95% bootstrap interval that resamples groups of related utterances
(FLEURS by sentence, since several speakers read each sentence; LibriSpeech by speaker and chapter), 8,586 groups in
all. "F32" is handy-computer's full-precision conversion of NVIDIA's weights; "handy-computer Q8_0" and
"handy-computer Q4_K_M" are files from the same repository; "transcribe.cpp quantizer, Q4_K_M" is a second same-size
file, made from the same F32 file with transcribe.cpp's standard quantizer. A second F32 run gave identical
transcripts, so runtime noise is zero.

### Pre-registered test

Parakeet-Air passes all four rules, which were written down before the test ran.

| Check | Rule | Result (95% CI) | Pass |
|---|---|---|---|
| Harness: F32 and handy-computer Q4_K_M on LibriSpeech test-clean | within 0.10 points of handy-computer's published 1.95% and 1.98% | 1.96% and 1.98% | yes |
| Primary: Parakeet-Air vs F32, word error rate | upper end of the 95% interval below +0.5 points | 11.59% β†’ 11.80%, +0.21 (+0.16 to +0.26) | yes |
| Primary: Parakeet-Air vs handy-computer Q4_K_M, word edits from F32's transcript per sentence | upper end of the 95% interval below 0 | 2.65 β†’ 1.95, βˆ’0.70 (βˆ’0.74 to βˆ’0.66) | yes |
| Guardrail: Parakeet-Air vs handy-computer Q4_K_M, word error rate | not significantly worse | 12.06% β†’ 11.80%, βˆ’0.26 (βˆ’0.32 to βˆ’0.20) | yes |

These four results are confirmatory. Everything else on this page (per-language rows and the other files) is
supporting detail and should be read as preliminary.

### Word error rate

Parakeet-Air is between full precision and the other two 4-bit files; the 8-bit file matches full precision.

| File | Size | Word error rate, all 22,766 | Change from F32 (95% interval) | FLEURS, 25 languages | LibriSpeech test-clean |
|---|---:|---:|---|---:|---:|
| F32 (full precision) | 2,508 MB | 11.59% | reference | 12.80% | 1.96% |
| handy-computer Q8_0 | 740 MB | 11.58% | βˆ’0.01 points (βˆ’0.03 to +0.01), no clear change | 12.79% | 1.94% |
| handy-computer Q4_K_M | 485 MB | 12.06% | +0.47 points (+0.41 to +0.53) | 13.33% | 1.98% |
| transcribe.cpp quantizer, Q4_K_M | 485 MB | 12.07% | +0.48 points (+0.42 to +0.54) | 13.34% | 1.94% |
| **Parakeet-Air** | 485 MB | **11.80%** | +0.21 points (+0.16 to +0.26) | **13.04%** | **1.94%** |

Rates are pooled: total word errors over total reference words. On FLEURS alone, Parakeet-Air is +0.24 points from
F32 (95% CI +0.19 to +0.29). The mean of the 25 per-language FLEURS rates is 12.96% for F32, 13.48% for
handy-computer's Q4_K_M and 13.19% for Parakeet-Air.

### Per language

Against handy-computer's Q4_K_M, Parakeet-Air is lower in 8 of 26 test sets and higher in none; the other 18 show no
clear change.

![Change in word error rate from F32 per test set, for handy-computer's Q4_K_M and Parakeet-Air](figures/languages.png)

| Test set | F32 | handy-computer Q4_K_M | **Parakeet-Air** | Parakeet-Air vs handy-computer Q4_K_M (95% interval) |
|---|---:|---:|---:|---|
| FLEURS Bulgarian | 12.77% | 13.10% | **13.11%** | +0.01 (βˆ’0.28 to +0.32), no clear change |
| FLEURS Czech | 12.38% | 13.09% | **12.66%** | βˆ’0.43 (βˆ’0.79 to βˆ’0.07), lower |
| FLEURS Danish | 18.61% | 19.83% | **19.26%** | βˆ’0.57 (βˆ’0.93 to βˆ’0.19), lower |
| FLEURS German | 5.22% | 5.25% | **5.25%** | +0.00 (βˆ’0.19 to +0.20), no clear change |
| FLEURS Greek | 35.32% | 35.81% | **35.52%** | βˆ’0.29 (βˆ’0.71 to +0.11), no clear change |
| FLEURS English | 4.86% | 4.90% | **4.93%** | +0.03 (βˆ’0.14 to +0.21), no clear change |
| FLEURS Spanish | 3.74% | 3.84% | **3.70%** | βˆ’0.14 (βˆ’0.30 to +0.00), no clear change |
| FLEURS Estonian | 17.98% | 18.33% | **18.10%** | βˆ’0.23 (βˆ’0.68 to +0.20), no clear change |
| FLEURS Finnish | 13.49% | 14.03% | **13.79%** | βˆ’0.24 (βˆ’0.62 to +0.11), no clear change |
| FLEURS French | 5.26% | 5.60% | **5.50%** | βˆ’0.10 (βˆ’0.32 to +0.12), no clear change |
| FLEURS Croatian | 12.60% | 13.44% | **12.92%** | βˆ’0.52 (βˆ’0.84 to βˆ’0.21), lower |
| FLEURS Hungarian | 16.23% | 16.93% | **16.80%** | βˆ’0.13 (βˆ’0.52 to +0.26), no clear change |
| FLEURS Italian | 3.01% | 3.02% | **3.03%** | +0.01 (βˆ’0.15 to +0.18), no clear change |
| FLEURS Lithuanian | 22.27% | 23.77% | **23.04%** | βˆ’0.73 (βˆ’1.14 to βˆ’0.34), lower |
| FLEURS Latvian | 23.94% | 25.46% | **24.24%** | βˆ’1.22 (βˆ’1.71 to βˆ’0.76), lower |
| FLEURS Maltese | 20.49% | 21.31% | **20.77%** | βˆ’0.54 (βˆ’0.95 to βˆ’0.16), lower |
| FLEURS Dutch | 7.65% | 8.03% | **7.54%** | βˆ’0.49 (βˆ’0.85 to βˆ’0.12), lower |
| FLEURS Polish | 7.41% | 7.67% | **7.41%** | βˆ’0.26 (βˆ’0.51 to +0.01), no clear change |
| FLEURS Portuguese | 4.90% | 4.90% | **4.90%** | +0.00 (βˆ’0.15 to +0.16), no clear change |
| FLEURS Romanian | 12.67% | 13.67% | **13.03%** | βˆ’0.63 (βˆ’0.95 to βˆ’0.34), lower |
| FLEURS Russian | 6.48% | 6.59% | **6.67%** | +0.09 (βˆ’0.13 to +0.28), no clear change |
| FLEURS Slovak | 10.07% | 10.73% | **10.61%** | βˆ’0.12 (βˆ’0.46 to +0.24), no clear change |
| FLEURS Slovenian | 24.44% | 24.65% | **24.44%** | βˆ’0.21 (βˆ’0.68 to +0.26), no clear change |
| FLEURS Swedish | 15.23% | 15.81% | **15.57%** | βˆ’0.24 (βˆ’0.62 to +0.13), no clear change |
| FLEURS Ukrainian | 6.88% | 7.20% | **6.95%** | βˆ’0.25 (βˆ’0.50 to +0.00), no clear change |
| LibriSpeech test-clean (English) | 1.96% | 1.98% | **1.94%** | βˆ’0.04 (βˆ’0.09 to +0.02), no clear change |

Differences are in percentage points. With 26 test sets, about one false "lower" or "higher" is expected by chance at
this interval width. Against F32, Parakeet-Air is significantly higher in 10 test sets (Bulgarian, Danish, French,
Croatian, Hungarian, Lithuanian, Romanian, Russian, Slovak, Swedish); handy-computer's Q4_K_M is significantly higher
in 16.

### Closeness to full precision's transcripts

Parakeet-Air's transcripts are 26% closer to F32's than handy-computer's Q4_K_M's, at the identical file size.

| File | Words that differ from F32's transcript, per 100 reference words |
|---|---:|
| handy-computer Q8_0 | 0.91 |
| handy-computer Q4_K_M | 4.81 |
| transcribe.cpp quantizer, Q4_K_M | 4.78 |
| **Parakeet-Air** | **3.54** |

This counts word edits (substitutions, insertions, deletions) between each file's normalized transcript and F32's
transcript of the same audio, so it measures change from full precision regardless of whether the reference text
agrees. Parakeet-Air is lower than handy-computer's Q4_K_M in all 26 test sets.

![Words that differ from F32's transcript per 100 words, for every quantized file](figures/fidelity.png)

## Files

| File | What it is |
|---|---|
| `Parakeet-Air-Q4_K_M.gguf` | Parakeet-Air (485,425,504 bytes, sha256 `14c12436640ccdd6f6890e791751454728897b7494ccbc0adb4a358a290e4fdb`); modified from handy-computer's F32 GGUF of NVIDIA's weights |
| `LICENSE.md` | webAI Non-Commercial License ver. 1.0 |
| `cc-by-4.0-LICENSE.txt` | Creative Commons Attribution 4.0 International, the license of NVIDIA's Parakeet TDT 0.6B v3 and of handy-computer's GGUF conversion |
| `figures/` | The charts in this card |

Check the download with `shasum -a 256 Parakeet-Air-Q4_K_M.gguf` against the hash above.

## Quick start

```bash
git clone https://github.com/handy-computer/transcribe.cpp && cd transcribe.cpp
git checkout 5700f127
cmake -B build && cmake --build build -j
hf download webAI-Official/Parakeet-Air Parakeet-Air-Q4_K_M.gguf --local-dir .
ffmpeg -i input.mp3 -ar 16000 -ac 1 input.wav
build/bin/transcribe-cli -m Parakeet-Air-Q4_K_M.gguf input.wav
```

* transcribe.cpp expects 16 kHz mono audio; the `ffmpeg` line converts other formats.
* The default build uses Metal on Apple silicon and the CPU elsewhere. For NVIDIA GPUs, the backend this file was
  evaluated on, configure with `cmake -B build -DTRANSCRIBE_CUDA=ON` and run with `--backend cuda`.
* The evaluation used greedy decoding with timestamps and inverse text normalization off. The command line prints
  word and token timestamps by default; those were not evaluated.

## Limitations

* **Not identical to full precision.** Its word error rate is +0.21 points above F32's overall (significant) and
  significantly higher in 10 of the 26 test sets. If accuracy matters more than size, handy-computer's Q8_0 (740 MB)
  showed no measurable loss on this test.
* **Moderate gains.** The improvement over handy-computer's Q4_K_M is βˆ’0.26 points of word error rate overall; in 18
  of 26 test sets the difference was not significant.
* **One run.** Each file was measured once on one test set. Greedy decoding on this backend is deterministic (a
  second F32 run gave identical transcripts), so the intervals reflect the choice of utterances, not runtime noise.
* **Scope.** Read speech only: FLEURS (read Wikipedia sentences) and LibriSpeech test-clean (read audiobooks), one
  sentence or short passage per clip. Conversational, noisy, accented, long-form and streaming audio were not evaluated, nor
  were timestamps or inverse text normalization.
* **Backends.** Results hold for transcribe.cpp's CUDA backend. Metal on Apple silicon was smoke-tested only; CPU,
  Vulkan and HIP were not tested with this file.
* **Speed was not measured** under controlled conditions.One example on an Apple silicon Mac with the Metal
  backend: a 4.5 s clip transcribed in 218 ms of processing, 4.3 s in total including model loading.
* **Which file.** Results hold for the included `Parakeet-Air-Q4_K_M.gguf` (sha256 above), byte-identical to the
  file that was evaluated.

## Evaluation protocol

* **Runtime:** stock transcribe.cpp at commit `5700f127`, CUDA backend, one NVIDIA H200 (shared with another job),
  batch transcription through transcribe.cpp's own `scripts/wer/run.py` with `--backend cuda --timestamps none`;
  greedy decoding; inverse text normalization off (the default for this model). The harness passes each test set's
  language code, which transcribe.cpp's Parakeet implementation does not use: the model detects the language itself.
* **Files:** F32, Q8_0 and Q4_K_M from `handy-computer/parakeet-tdt-0.6b-v3-gguf` (sha256 `bbd36acd8c6fbb81…`,
  `5859f77944efcd8e…`, `b68557be1e3c4020…`), which handy-computer converted from NVIDIA's revision `6d590f7`.
* **Test data:** FLEURS test split from `google/fleurs` (revision `70bb2e84`), 25 languages, 20,146 utterances, with
  FLEURS's normalized transcriptions as references; LibriSpeech test-clean, 2,620 utterances, from transcribe.cpp's
  WER manifest. Spanish and Portuguese use the FLEURS configurations `es_419` and `pt_br`.
* **Word error rate:** Whisper's English text normalizer for English (FLEURS English and LibriSpeech) and Whisper's
  basic normalizer for the other languages, applied to reference and transcript; word errors counted with jiwer;
  pooled over words. Utterances whose reference is empty after normalization are skipped (none in this test).
* **Closeness to F32:** word edits between each file's normalized transcript and F32's normalized transcript of the
  same utterance.
* **Statistics:** paired comparisons over the same 22,766 utterances, with a 95% interval from 2,000 bootstrap
  resamples of 8,586 groups (8,499 FLEURS sentences, each read by one or more speakers in one language, and 87
  LibriSpeech speaker-chapters); differences in errors per group are divided by mean words per group to give
  word-error-rate points. "No clear change" means the interval includes zero.
* **Pre-registration:** the question, primary measures, the 0.5-point margin against F32, the harness check and the
  pass rules were written down before the test ran.
* **Comparability:** NVIDIA's and handy-computer's published word error rates use their own normalization and data
  versions. The harness check above reproduces handy-computer's LibriSpeech test-clean numbers within 0.01 points; other
  published numbers are not directly comparable with this page.

## License and attribution

Released under the webAI Non-Commercial License ver. 1.0 (LICENSE.md). This distribution incorporates a modified version of NVIDIA’s Parakeet TDT 0.6B v3.

The original NVIDIA Parakeet TDT 0.6B v3 model is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0).

Original model: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
License: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md

The F32 GGUF conversion of the original NVIDIA Parakeet TDT 0.6B v3 model by handy-computer is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0).
Original https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf
License: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md


The F32 GGUF conversion of the original NVIDIA Parakeet TDT 0.6B v3 model by handy-computer has been modified using quantization technology developed by webAI to 4-bit (Q4_K_M) at reduced numeric precision. This modified version is made available under the webAI Non-Commercial License Version 1.0.

The webAI license does not replace or restrict rights granted by NVIDIA or handy-computer under CC-BY-4.0 with respect to NVIDIA’s or handy-computer’s original licensed material. No NVIDIA or handy-computer sponsorship, affiliation, or endorsement is implied.

Please follow NVIDIA’s model card for intended use and its notes on bias, privacy and safety.