|
Download README.md from webAI-Official/Parakeet-Air: direct link, hf CLI and curl.
- Browser
- Download file 18.5 kB
-
https://huggingface.co/webAI-Official/Parakeet-Air/resolve/main/README.md
- Command line
-
hf download hf://webAI-Official/Parakeet-Air/README.md
-
curl -L -o README.md https://huggingface.co/webAI-Official/Parakeet-Air/resolve/main/README.md
18.5 kB
| license: other | |
| license_name: webai-non-commercial-license-ver.-1.0 | |
| license_link: https://huggingface.co/webAI-Official/Parakeet-Air/blob/main/LICENSE.md | |
| base_model: nvidia/parakeet-tdt-0.6b-v3 | |
| base_model_relation: quantized | |
| pipeline_tag: automatic-speech-recognition | |
| library_name: gguf | |
| language: | |
| - bg | |
| - cs | |
| - da | |
| - de | |
| - el | |
| - en | |
| - es | |
| - et | |
| - fi | |
| - fr | |
| - hr | |
| - hu | |
| - it | |
| - lt | |
| - lv | |
| - mt | |
| - nl | |
| - pl | |
| - pt | |
| - ro | |
| - ru | |
| - sk | |
| - sl | |
| - sv | |
| - uk | |
| tags: | |
| - gguf | |
| - automatic-speech-recognition | |
| - speech-recognition | |
| - parakeet | |
| - transcribe.cpp | |
| - multilingual | |
| - on-device | |
| # Parakeet-Air | |
| A 4-bit build of NVIDIA's [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) speech | |
| recognition model for 25 European languages, in GGUF for [transcribe.cpp](https://github.com/handy-computer/transcribe.cpp). | |
| It has the same size (485,425,504 bytes) and the same tensor types as handy-computer's widely used | |
| `parakeet-tdt-0.6b-v3-Q4_K_M.gguf`, and it loads in unmodified transcribe.cpp. In a pre-registered test on 22,766 | |
| utterances (FLEURS in 25 languages plus LibriSpeech), its word error rate was **within 0.5 points of full precision | |
| (+0.21 points)** where handy-computer's Q4_K_M file is +0.47, and its transcripts were **26% closer to full | |
| precision's** than that file's. | |
| > **Model Released under the webAI Non-Commercial License ver. 1.0; see** [License and attribution](#license-and-attribution). | |
| ### At a glance | |
| Passes all four rules of the pre-registered test. | |
| | | F32 (full precision) | handy-computer Q4_K_M | **Parakeet-Air** | | |
| |---|---:|---:|---:| | |
| | File size | 2,508,435,616 B | 485,425,504 B | **485,425,504 B** | | |
| | Word error rate, all 22,766 utterances | 11.59% | 12.06% | **11.80%** | | |
| | Word error rate, FLEURS test, 25 languages | 12.80% | 13.33% | **13.04%** | | |
| | Word error rate, LibriSpeech test-clean | 1.96% | 1.98% | **1.94%** | | |
| | Words that differ from F32's transcript, per 100 words | 0 (itself) | 4.81 | **3.54** | | |
| | Test sets where it beats handy-computer's Q4_K_M (95% interval below zero) | | | **8 of 26, worse in none** | | |
| <sub>transcribe.cpp `5700f127`, CUDA backend on an NVIDIA H200, greedy decoding, timestamps and inverse text | |
| normalization off; Whisper text normalizers; one run per file. Word error rates are pooled (total errors over total | |
| words). See [Evaluation protocol](#evaluation-protocol).</sub> | |
|  | |
| ## Highlights | |
| * **Within half a point of full precision.** Pooled word error rate 11.80% vs F32's 11.59%, +0.21 points (95% CI | |
| +0.16 to +0.26), inside the pre-registered 0.5-point margin. | |
| * **Lower word error than handy-computer's Q4_K_M.** β0.26 points (95% CI β0.32 to β0.20) overall, about 55% of that | |
| file's increase over F32 removed. It is significantly lower in 8 test sets (Czech, Danish, Croatian, Lithuanian, | |
| Latvian, Maltese, Dutch, Romanian) and significantly higher in none. | |
| * **Closer to full precision's transcripts.** 3.54 words per 100 differ from F32's transcript, against 4.81 for | |
| handy-computer's Q4_K_M: β26% (word edits per sentence β0.70, 95% CI β0.74 to β0.66), lower in all 26 test sets. | |
| * **Same size, same runtime.** Same bytes and tensor types as handy-computer's Q4_K_M; it loads in stock | |
| transcribe.cpp with no code changes. | |
| It is **not identical to full precision**: its word error rate is still significantly higher than F32's overall and | |
| in 10 of the 26 test sets, and handy-computer's 740 MB Q8_0 shows no measurable loss. See [Limitations](#limitations). | |
| ## Model Details | |
| | Property | Value | | |
| |---|---| | |
| | Model ID | `webAI-Official/Parakeet-Air` (proposed; not yet published) | | |
| | Base model | [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), 0.6B parameters (FastConformer encoder, TDT decoder), via handy-computer's F32 GGUF conversion in [`handy-computer/parakeet-tdt-0.6b-v3-gguf`](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf) | | |
| | Model file | `Parakeet-Air-Q4_K_M.gguf`, 485,425,504 bytes (sha256 `14c12436640ccdd6β¦`) | | |
| | Format | GGUF, file type Q4_K_M: 194 matrices in Q4_K, 31 in Q8_0 (attention output, prediction network, joint network), the 48 pointwise convolutions in F16 and the remaining small tensors in F32, exactly as in handy-computer's Q4_K_M. The file's metadata is the same as handy-computer's, so it reports its name as "Parakeet TDT 0.6B v3" | | |
| | Runs on | transcribe.cpp. Evaluated with the CUDA backend (commit `5700f127`, NVIDIA H200); also run with the Metal backend on Apple silicon | | |
| | Input / output | 16 kHz mono audio in any of the 25 languages / text with punctuation and capitalization; language detected automatically | | |
| | License | webAI Non-Commercial License ver. 1.0; based on NVIDIA's CC-BY-4.0 weights | | |
| ## Results | |
| Every result comes from one test, run once per file: each file transcribed the same 22,766 utterances (FLEURS test in | |
| 25 languages, 20,146 utterances, plus LibriSpeech test-clean, 2,620) with stock transcribe.cpp on the same GPU. | |
| Verdicts pair the same utterances and use a 95% bootstrap interval that resamples groups of related utterances | |
| (FLEURS by sentence, since several speakers read each sentence; LibriSpeech by speaker and chapter), 8,586 groups in | |
| all. "F32" is handy-computer's full-precision conversion of NVIDIA's weights; "handy-computer Q8_0" and | |
| "handy-computer Q4_K_M" are files from the same repository; "transcribe.cpp quantizer, Q4_K_M" is a second same-size | |
| file, made from the same F32 file with transcribe.cpp's standard quantizer. A second F32 run gave identical | |
| transcripts, so runtime noise is zero. | |
| ### Pre-registered test | |
| Parakeet-Air passes all four rules, which were written down before the test ran. | |
| | Check | Rule | Result (95% CI) | Pass | | |
| |---|---|---|---| | |
| | Harness: F32 and handy-computer Q4_K_M on LibriSpeech test-clean | within 0.10 points of handy-computer's published 1.95% and 1.98% | 1.96% and 1.98% | yes | | |
| | Primary: Parakeet-Air vs F32, word error rate | upper end of the 95% interval below +0.5 points | 11.59% β 11.80%, +0.21 (+0.16 to +0.26) | yes | | |
| | Primary: Parakeet-Air vs handy-computer Q4_K_M, word edits from F32's transcript per sentence | upper end of the 95% interval below 0 | 2.65 β 1.95, β0.70 (β0.74 to β0.66) | yes | | |
| | Guardrail: Parakeet-Air vs handy-computer Q4_K_M, word error rate | not significantly worse | 12.06% β 11.80%, β0.26 (β0.32 to β0.20) | yes | | |
| These four results are confirmatory. Everything else on this page (per-language rows and the other files) is | |
| supporting detail and should be read as preliminary. | |
| ### Word error rate | |
| Parakeet-Air is between full precision and the other two 4-bit files; the 8-bit file matches full precision. | |
| | File | Size | Word error rate, all 22,766 | Change from F32 (95% interval) | FLEURS, 25 languages | LibriSpeech test-clean | | |
| |---|---:|---:|---|---:|---:| | |
| | F32 (full precision) | 2,508 MB | 11.59% | reference | 12.80% | 1.96% | | |
| | handy-computer Q8_0 | 740 MB | 11.58% | β0.01 points (β0.03 to +0.01), no clear change | 12.79% | 1.94% | | |
| | handy-computer Q4_K_M | 485 MB | 12.06% | +0.47 points (+0.41 to +0.53) | 13.33% | 1.98% | | |
| | transcribe.cpp quantizer, Q4_K_M | 485 MB | 12.07% | +0.48 points (+0.42 to +0.54) | 13.34% | 1.94% | | |
| | **Parakeet-Air** | 485 MB | **11.80%** | +0.21 points (+0.16 to +0.26) | **13.04%** | **1.94%** | | |
| Rates are pooled: total word errors over total reference words. On FLEURS alone, Parakeet-Air is +0.24 points from | |
| F32 (95% CI +0.19 to +0.29). The mean of the 25 per-language FLEURS rates is 12.96% for F32, 13.48% for | |
| handy-computer's Q4_K_M and 13.19% for Parakeet-Air. | |
| ### Per language | |
| Against handy-computer's Q4_K_M, Parakeet-Air is lower in 8 of 26 test sets and higher in none; the other 18 show no | |
| clear change. | |
|  | |
| | Test set | F32 | handy-computer Q4_K_M | **Parakeet-Air** | Parakeet-Air vs handy-computer Q4_K_M (95% interval) | | |
| |---|---:|---:|---:|---| | |
| | FLEURS Bulgarian | 12.77% | 13.10% | **13.11%** | +0.01 (β0.28 to +0.32), no clear change | | |
| | FLEURS Czech | 12.38% | 13.09% | **12.66%** | β0.43 (β0.79 to β0.07), lower | | |
| | FLEURS Danish | 18.61% | 19.83% | **19.26%** | β0.57 (β0.93 to β0.19), lower | | |
| | FLEURS German | 5.22% | 5.25% | **5.25%** | +0.00 (β0.19 to +0.20), no clear change | | |
| | FLEURS Greek | 35.32% | 35.81% | **35.52%** | β0.29 (β0.71 to +0.11), no clear change | | |
| | FLEURS English | 4.86% | 4.90% | **4.93%** | +0.03 (β0.14 to +0.21), no clear change | | |
| | FLEURS Spanish | 3.74% | 3.84% | **3.70%** | β0.14 (β0.30 to +0.00), no clear change | | |
| | FLEURS Estonian | 17.98% | 18.33% | **18.10%** | β0.23 (β0.68 to +0.20), no clear change | | |
| | FLEURS Finnish | 13.49% | 14.03% | **13.79%** | β0.24 (β0.62 to +0.11), no clear change | | |
| | FLEURS French | 5.26% | 5.60% | **5.50%** | β0.10 (β0.32 to +0.12), no clear change | | |
| | FLEURS Croatian | 12.60% | 13.44% | **12.92%** | β0.52 (β0.84 to β0.21), lower | | |
| | FLEURS Hungarian | 16.23% | 16.93% | **16.80%** | β0.13 (β0.52 to +0.26), no clear change | | |
| | FLEURS Italian | 3.01% | 3.02% | **3.03%** | +0.01 (β0.15 to +0.18), no clear change | | |
| | FLEURS Lithuanian | 22.27% | 23.77% | **23.04%** | β0.73 (β1.14 to β0.34), lower | | |
| | FLEURS Latvian | 23.94% | 25.46% | **24.24%** | β1.22 (β1.71 to β0.76), lower | | |
| | FLEURS Maltese | 20.49% | 21.31% | **20.77%** | β0.54 (β0.95 to β0.16), lower | | |
| | FLEURS Dutch | 7.65% | 8.03% | **7.54%** | β0.49 (β0.85 to β0.12), lower | | |
| | FLEURS Polish | 7.41% | 7.67% | **7.41%** | β0.26 (β0.51 to +0.01), no clear change | | |
| | FLEURS Portuguese | 4.90% | 4.90% | **4.90%** | +0.00 (β0.15 to +0.16), no clear change | | |
| | FLEURS Romanian | 12.67% | 13.67% | **13.03%** | β0.63 (β0.95 to β0.34), lower | | |
| | FLEURS Russian | 6.48% | 6.59% | **6.67%** | +0.09 (β0.13 to +0.28), no clear change | | |
| | FLEURS Slovak | 10.07% | 10.73% | **10.61%** | β0.12 (β0.46 to +0.24), no clear change | | |
| | FLEURS Slovenian | 24.44% | 24.65% | **24.44%** | β0.21 (β0.68 to +0.26), no clear change | | |
| | FLEURS Swedish | 15.23% | 15.81% | **15.57%** | β0.24 (β0.62 to +0.13), no clear change | | |
| | FLEURS Ukrainian | 6.88% | 7.20% | **6.95%** | β0.25 (β0.50 to +0.00), no clear change | | |
| | LibriSpeech test-clean (English) | 1.96% | 1.98% | **1.94%** | β0.04 (β0.09 to +0.02), no clear change | | |
| Differences are in percentage points. With 26 test sets, about one false "lower" or "higher" is expected by chance at | |
| this interval width. Against F32, Parakeet-Air is significantly higher in 10 test sets (Bulgarian, Danish, French, | |
| Croatian, Hungarian, Lithuanian, Romanian, Russian, Slovak, Swedish); handy-computer's Q4_K_M is significantly higher | |
| in 16. | |
| ### Closeness to full precision's transcripts | |
| Parakeet-Air's transcripts are 26% closer to F32's than handy-computer's Q4_K_M's, at the identical file size. | |
| | File | Words that differ from F32's transcript, per 100 reference words | | |
| |---|---:| | |
| | handy-computer Q8_0 | 0.91 | | |
| | handy-computer Q4_K_M | 4.81 | | |
| | transcribe.cpp quantizer, Q4_K_M | 4.78 | | |
| | **Parakeet-Air** | **3.54** | | |
| This counts word edits (substitutions, insertions, deletions) between each file's normalized transcript and F32's | |
| transcript of the same audio, so it measures change from full precision regardless of whether the reference text | |
| agrees. Parakeet-Air is lower than handy-computer's Q4_K_M in all 26 test sets. | |
|  | |
| ## Files | |
| | File | What it is | | |
| |---|---| | |
| | `Parakeet-Air-Q4_K_M.gguf` | Parakeet-Air (485,425,504 bytes, sha256 `14c12436640ccdd6f6890e791751454728897b7494ccbc0adb4a358a290e4fdb`); modified from handy-computer's F32 GGUF of NVIDIA's weights | | |
| | `LICENSE.md` | webAI Non-Commercial License ver. 1.0 | | |
| | `cc-by-4.0-LICENSE.txt` | Creative Commons Attribution 4.0 International, the license of NVIDIA's Parakeet TDT 0.6B v3 and of handy-computer's GGUF conversion | | |
| | `figures/` | The charts in this card | | |
| Check the download with `shasum -a 256 Parakeet-Air-Q4_K_M.gguf` against the hash above. | |
| ## Quick start | |
| ```bash | |
| git clone https://github.com/handy-computer/transcribe.cpp && cd transcribe.cpp | |
| git checkout 5700f127 | |
| cmake -B build && cmake --build build -j | |
| hf download webAI-Official/Parakeet-Air Parakeet-Air-Q4_K_M.gguf --local-dir . | |
| ffmpeg -i input.mp3 -ar 16000 -ac 1 input.wav | |
| build/bin/transcribe-cli -m Parakeet-Air-Q4_K_M.gguf input.wav | |
| ``` | |
| * transcribe.cpp expects 16 kHz mono audio; the `ffmpeg` line converts other formats. | |
| * The default build uses Metal on Apple silicon and the CPU elsewhere. For NVIDIA GPUs, the backend this file was | |
| evaluated on, configure with `cmake -B build -DTRANSCRIBE_CUDA=ON` and run with `--backend cuda`. | |
| * The evaluation used greedy decoding with timestamps and inverse text normalization off. The command line prints | |
| word and token timestamps by default; those were not evaluated. | |
| ## Limitations | |
| * **Not identical to full precision.** Its word error rate is +0.21 points above F32's overall (significant) and | |
| significantly higher in 10 of the 26 test sets. If accuracy matters more than size, handy-computer's Q8_0 (740 MB) | |
| showed no measurable loss on this test. | |
| * **Moderate gains.** The improvement over handy-computer's Q4_K_M is β0.26 points of word error rate overall; in 18 | |
| of 26 test sets the difference was not significant. | |
| * **One run.** Each file was measured once on one test set. Greedy decoding on this backend is deterministic (a | |
| second F32 run gave identical transcripts), so the intervals reflect the choice of utterances, not runtime noise. | |
| * **Scope.** Read speech only: FLEURS (read Wikipedia sentences) and LibriSpeech test-clean (read audiobooks), one | |
| sentence or short passage per clip. Conversational, noisy, accented, long-form and streaming audio were not evaluated, nor | |
| were timestamps or inverse text normalization. | |
| * **Backends.** Results hold for transcribe.cpp's CUDA backend. Metal on Apple silicon was smoke-tested only; CPU, | |
| Vulkan and HIP were not tested with this file. | |
| * **Speed was not measured** under controlled conditions.One example on an Apple silicon Mac with the Metal | |
| backend: a 4.5 s clip transcribed in 218 ms of processing, 4.3 s in total including model loading. | |
| * **Which file.** Results hold for the included `Parakeet-Air-Q4_K_M.gguf` (sha256 above), byte-identical to the | |
| file that was evaluated. | |
| ## Evaluation protocol | |
| * **Runtime:** stock transcribe.cpp at commit `5700f127`, CUDA backend, one NVIDIA H200 (shared with another job), | |
| batch transcription through transcribe.cpp's own `scripts/wer/run.py` with `--backend cuda --timestamps none`; | |
| greedy decoding; inverse text normalization off (the default for this model). The harness passes each test set's | |
| language code, which transcribe.cpp's Parakeet implementation does not use: the model detects the language itself. | |
| * **Files:** F32, Q8_0 and Q4_K_M from `handy-computer/parakeet-tdt-0.6b-v3-gguf` (sha256 `bbd36acd8c6fbb81β¦`, | |
| `5859f77944efcd8eβ¦`, `b68557be1e3c4020β¦`), which handy-computer converted from NVIDIA's revision `6d590f7`. | |
| * **Test data:** FLEURS test split from `google/fleurs` (revision `70bb2e84`), 25 languages, 20,146 utterances, with | |
| FLEURS's normalized transcriptions as references; LibriSpeech test-clean, 2,620 utterances, from transcribe.cpp's | |
| WER manifest. Spanish and Portuguese use the FLEURS configurations `es_419` and `pt_br`. | |
| * **Word error rate:** Whisper's English text normalizer for English (FLEURS English and LibriSpeech) and Whisper's | |
| basic normalizer for the other languages, applied to reference and transcript; word errors counted with jiwer; | |
| pooled over words. Utterances whose reference is empty after normalization are skipped (none in this test). | |
| * **Closeness to F32:** word edits between each file's normalized transcript and F32's normalized transcript of the | |
| same utterance. | |
| * **Statistics:** paired comparisons over the same 22,766 utterances, with a 95% interval from 2,000 bootstrap | |
| resamples of 8,586 groups (8,499 FLEURS sentences, each read by one or more speakers in one language, and 87 | |
| LibriSpeech speaker-chapters); differences in errors per group are divided by mean words per group to give | |
| word-error-rate points. "No clear change" means the interval includes zero. | |
| * **Pre-registration:** the question, primary measures, the 0.5-point margin against F32, the harness check and the | |
| pass rules were written down before the test ran. | |
| * **Comparability:** NVIDIA's and handy-computer's published word error rates use their own normalization and data | |
| versions. The harness check above reproduces handy-computer's LibriSpeech test-clean numbers within 0.01 points; other | |
| published numbers are not directly comparable with this page. | |
| ## License and attribution | |
| Released under the webAI Non-Commercial License ver. 1.0 (LICENSE.md). This distribution incorporates a modified version of NVIDIAβs Parakeet TDT 0.6B v3. | |
| The original NVIDIA Parakeet TDT 0.6B v3 model is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). | |
| Original model: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 | |
| License: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md | |
| The F32 GGUF conversion of the original NVIDIA Parakeet TDT 0.6B v3 model by handy-computer is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). | |
| Original https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf | |
| License: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md | |
| The F32 GGUF conversion of the original NVIDIA Parakeet TDT 0.6B v3 model by handy-computer has been modified using quantization technology developed by webAI to 4-bit (Q4_K_M) at reduced numeric precision. This modified version is made available under the webAI Non-Commercial License Version 1.0. | |
| The webAI license does not replace or restrict rights granted by NVIDIA or handy-computer under CC-BY-4.0 with respect to NVIDIAβs or handy-computerβs original licensed material. No NVIDIA or handy-computer sponsorship, affiliation, or endorsement is implied. | |
| Please follow NVIDIAβs model card for intended use and its notes on bias, privacy and safety. | |