README.md

---
license: cc-by-4.0
language:
- en
- es
- fr
- de
- bg
- hr
- cs
- da
- nl
- et
- fi
- el
- hu
- it
- lv
- lt
- mt
- pl
- pt
- ro
- sk
- sl
- sv
- ru
- uk
pipeline_tag: automatic-speech-recognition
library_name: nemo
base_model: nvidia/parakeet-tdt-0.6b-v3
base_model_relation: quantized
tags:
- automatic-speech-recognition
- speech
- audio
- Transducer
- TDT
- FastConformer
- Conformer
- pytorch
- NeMo
- bf16
- fp16
- nvfp4
- fp4
- 4-bit
- quantization
- safetensors
---

# parakeet-tdt-0.6b-v3 — half-precision and 4-bit checkpoints

Reduced-precision builds of NVIDIA's [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3),
packaged for direct GPU loading. Three artifacts, same model:

| Artifact | Format | Size | For |
|---|---|---|---|
| `parakeet-tdt-0.6b-v3-bf16.nemo` | bfloat16 | 1.25 GB | Ampere and newer (compute capability 8.0+) |
| `parakeet-tdt-0.6b-v3-fp16.nemo` | float16 | 1.25 GB | anything with CUDA, including pre-Ampere (sm_75: GTX 16-series, RTX 20-series, Tesla T4) |
| `parakeet-tdt-0.6b-v3-nvfp4/` | NVFP4 weights, bf16 activations | 0.49 GB | Ampere and newer, when VRAM is the constraint |

The two `.nemo` files are pure casts of the fp32 weights — identical to upstream
apart from the conversion. Restoring from either loads straight onto a CUDA GPU
without ever materialising an fp32 copy, which is faster to start (~13 s vs ~22 s)
and roughly halves peak load-time VRAM.

The NVFP4 directory is **not** a cast: 4-bit scales are chosen by calibrating on
real audio, so it ships pre-quantized. It halves VRAM again at no accuracy cost
this evaluation can detect, and costs 2.4× the decode latency.

**Which one?**

- **bf16** — the default. Fastest, and what the accuracy of the others is measured against.
- **fp16** — if your GPU predates Ampere, where bf16 is not merely slower but unsupported. On a card that does both, the two are statistically indistinguishable and fp16 is marginally closer to fp32 numerically.
- **nvfp4** — only when you are short of VRAM. It is the smallest by a wide margin and the slowest of the three; it buys memory, not throughput.

This repository redistributes the model under its original **CC BY 4.0** license.
See the [upstream model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)
for architecture, training data, supported languages, evaluation, and intended use.

## Usage

### bf16 / fp16 (NeMo)

```python
import nemo.collections.asr as nemo_asr
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("lightware-dev/parakeet-tdt-0.6b-v3",
                       "parakeet-tdt-0.6b-v3-bf16.nemo")   # or -fp16.nemo
model = nemo_asr.models.ASRModel.restore_from(ckpt, map_location="cuda")
model.eval()
print(model.transcribe(["audio.wav"]))
```

### NVFP4

Needs `nvidia-modelopt` alongside NeMo. The loader lives in
[`lightware-dev/blurt`](https://github.com/lightware-dev/blurt) as
`server/nvfp4.py`:

```python
import os
from huggingface_hub import snapshot_download
from server import nvfp4          # from github.com/lightware-dev/blurt

root = snapshot_download("lightware-dev/parakeet-tdt-0.6b-v3",
                         allow_patterns="parakeet-tdt-0.6b-v3-nvfp4/*")
model = nvfp4.load_snapshot(os.path.join(root, "parakeet-tdt-0.6b-v3-nvfp4"),
                            device="cuda")
print(model.transcribe(["audio.wav"]))
```

In `blurtd` itself this is just `PARAKEET_DTYPE=nvfp4`.

The load path builds the module graph on the CPU, replays the quantization recipe
onto it, then reads the packed tensors straight onto the GPU — so **the device
never holds a bf16 copy**. That is the entire point: quantizing at startup instead
would require the GPU to hold the full bf16 model first (peaking at 2.53 GB), which
a card small enough to need 4-bit cannot do.

## What NVFP4 is here

NVFP4 is **E2M1** — four bits per weight (1 sign, 2 exponent, 1 mantissa, so eight
representable magnitudes) — grouped in blocks of 16, each block carrying its own
**E4M3 fp8 scale**. The block scale is what makes four bits survivable: precision
is allocated locally, so a block of small weights is not crushed by a large outlier
elsewhere in the row. It is also why the packed model is 0.49 GB rather than the
0.27 GB a naive 4-bit count would suggest.

Only the **217 encoder Linear layers** are quantized — 533M of 627M parameters
(86%). The convolutions, LSTM decoder and joint network stay bf16, and so do all
activations (**W4A16**). The joint network's vocabulary projection is deliberately
excluded: it runs inside the TDT decode loop, where an extra unpack per step would
cost latency and feed the argmax directly.

Contents of the directory:

| File | Size | Holds |
|---|---|---|
| `weights.safetensors` | 488.6 MB | 217 packed `uint8` weights (two 4-bit values per byte), 217 `float8_e4m3fn` block scales, 701 `bfloat16` tensors |
| `recipe.json` | 1.8 MB | quantizer layout and calibrated amax values |
| `model_config.yaml` | 0.2 MB | NeMo config, tokenizer paths localized |
| tokenizer files | 0.5 MB | SentencePiece model and vocabs |

**No pickles.** `torch.load` reconstructs arbitrary Python objects, so loading one
is equivalent to running its author's code — a poor property for a file fetched at
service startup. Weights use safetensors (a length-prefixed JSON header over raw
tensor bytes, structurally unable to carry code), the recipe is JSON, the config is
YAML. The single class the recipe names is resolved by attribute lookup against a
`modelopt.*` allowlist, never called.

## Evaluation

All figures below come from one corpus of **208 clips / 25.8 minutes**: 100
LibriSpeech `test-clean` utterances, the same utterances degraded with white noise
at 10 dB and 5 dB SNR, babble at 5 dB and a near-clipping loud variant, plus 8
synthetic clips. Greedy TDT decoding, WER after text normalisation, on an RTX 5090.
Differences against bf16 are paired bootstrap estimates — paired because both
models decode identical audio, so per-utterance noise cancels instead of being
counted twice.

NVFP4 was calibrated on 62 clips drawn from **different** LibriSpeech rows than the
208 scored ones, asserted disjoint at build time. Calibrating on the evaluation set
would tune the quantizer to its own test.

### NVFP4 vs bf16

| | bf16 | NVFP4 |
|---|---|---|
| WER, all 208 clips | 2.99 % | **2.86 %** |
| WER, clean real speech | 2.40 % | 2.40 % |
| Difference vs bf16 | — | −0.128 pp, 95 % CI [−0.693, +0.295], p = 0.69 |
| Identical transcripts | — | 179 / 208 (86.1 %) |
| Encoder output vs fp32, mean relative L2 | 0.0436 | 0.2257 |
| Encoder output vs fp32, worst cosine | 0.8351 | 0.8261 |
| Non-finite activations | 0 | 0 |
| Empty transcripts | 3 / 208 | 1 / 208 |
| Weights | 1.31 GB | **0.51 GB** |
| Real-time factor | 0.0134 | 0.0316 |

**NVFP4 is not better than bf16.** Its point estimate is lower and its
babble-noise slice looks notably better, but the confidence interval spans
−0.69 to +0.30 pp — that is noise landing favourably. At 208 clips and a ~3 % base
rate the corpus cannot resolve a difference below roughly 0.3 pp, so the honest
claim is "no cost we can measure", not "identical".

What *is* visible: four bits changes the exact text of 14 % of clips while leaving
the aggregate untouched, and pushes the encoder output about 5× further from fp32
than bf16 does. The greedy decoder absorbs most of that drift.

Per slice:

| Slice | n | bf16 | NVFP4 |
|---|---|---|---|
| Clean real speech | 100 | 2.40 % | 2.40 % |
| White noise, 10 dB SNR | 25 | 1.90 % | 2.61 % |
| White noise, 5 dB SNR | 25 | 3.32 % | 4.50 % |
| Babble, 5 dB SNR | 25 | 8.29 % | 6.16 % |
| Near-clipping loud | 25 | 1.42 % | 1.42 % |
| Synthetic | 8 | 2.66 % | 0.53 % |
| All | 208 | 2.99 % | 2.86 % |

The babble and synthetic slices move in NVFP4's favour and the white-noise slices
against it, on 25 and 8 clips respectively. Do not read a denoising story into
that — these are small slices and the aggregate CI already says the corpus cannot
separate the two models.

### Memory

Measured per precision in a fresh process, loading the published artifact the way a
service would:

| | bf16 `.nemo` | NVFP4 snapshot |
|---|---|---|
| File on disk | 1255 MB | **491 MB** |
| Load time | 20.3 s | 20.9 s |
| Peak CUDA while loading | 2.532 GB | **0.776 GB** |
| Resident after load | 1.276 GB | 0.776 GB |
| Peak during decode | 1.430 GB | 0.931 GB |
| `nvidia-smi` after load | 3032 MB | **1370 MB** |

The load-time peak is the figure that decides whether a card can run the model at
all, and it is where shipping pre-quantized weights pays: 0.78 GB against 2.53 GB.
Quantizing a bf16 checkpoint at startup would produce the same 0.51 GB of weights
but still need 2.53 GB to get there.

Load time is a wash. Of the ~21 s, only **0.3 s** is device work — 0.2 s to read
489 MB of packed tensors onto CUDA and 0.1 s to bind them. The rest is CPU
overhead that should not need to exist: NeMo cannot construct this model on the
meta device (`ConformerEncoder.__init__` calls `.item()`), so a full module graph
is built and zero-filled, and ModelOpt's restore replays the compression on those
throwaway weights rather than installing the packed layout directly.

### Speed

NVFP4 decodes **2.4× slower** than bf16 (RTF 0.0316 vs 0.0134). Encoder forward
time is 95.6 ms against 27.0 ms.

The reason is that Parakeet's encoder is bound by **kernel launches, not
arithmetic**. Encoder forward time barely responds to how much audio it is given:

| Clip length | bf16 encoder forward |
|---|---|
| 2.2 s | 27.2 ms |
| 5.4 s | 27.6 ms |
| 35.6 s | 28.9 ms |

Sixteen times the audio for 6 % more time means the GPU spends its life launching
kernels across 24 conformer layers and 220 Linears. Quantization makes the
multiplies cheaper — the part that was never the bottleneck — while adding an
unpack per layer. It lands net negative. At RTF 0.032 the model is still ~30×
faster than real time, which is imperceptible for dictation, but NVFP4 should not
be chosen expecting throughput.

### fp16 vs bf16 vs fp32

| | fp32 | bf16 | fp16 |
|---|---|---|---|
| WER, all clips | 2.96 % | 2.99 % | 2.96 % |
| WER, clean real speech | 2.40 % | 2.40 % | 2.40 % |
| Encoder output vs fp32, mean relative L2 | — | 0.0436 | 0.0052 |
| Encoder output vs fp32, worst cosine similarity | — | 0.8351 | 0.9948 |
| Weights on disk / in VRAM | 2.5 GB | 1.31 GB | 1.31 GB |
| Real-time factor | 0.0117 | 0.0134 | 0.0118 |

fp16 and bf16 produced **identical transcripts on 207 of 208 clips**. The WER
difference between them is +0.026 pp (95 % bootstrap CI [0.000, +0.077],
p = 0.74) — statistically indistinguishable, and exactly zero on real speech.

fp16 tracks fp32 roughly 8× more closely than bf16 does at the activation level,
which is what you would expect from its 10 mantissa bits against bf16's 7. No
overflow was observed: zero non-finite activations anywhere, including on the
deliberately near-clipping loud slice. The narrower fp16 exponent range is not a
practical problem for this model at inference.

### Other quantizations evaluated, and why they are not here

The same corpus and method were used to test five other post-training schemes.
None is published — this table is here so the choice of NVFP4 is checkable rather
than asserted.

| Scheme | WER, all | vs bf16 | Weights | RTF | Identical transcripts |
|---|---|---|---|---|---|
| bf16 (reference) | 2.99 % | — | 1.31 GB | 0.0134 | — |
| **NVFP4 W4A16** (published) | 2.86 % | −0.128 pp [−0.69, +0.30], p = 0.69 | **0.51 GB** | 0.0316 | 179 / 208 |
| INT4 AWQ, weight-only | 3.19 % | +0.204 pp [−0.08, +0.52], p = 0.19 | 0.49 GB | 0.0328 | 176 / 208 |
| NVFP4 W4A4 (4-bit activations too) | 3.50 % | **+0.510 pp [+0.14, +0.96], p = 0.006** | 0.51 GB | 0.0384 | 169 / 208 |
| INT8 weight-only | 2.96 % | −0.026 pp [−0.18, +0.12], p = 0.88 | 0.74 GB | 0.0161 | 203 / 208 |
| INT8 dynamic activations | 2.99 % | +0.000 pp [−0.21, +0.22] | 0.74 GB | 0.0608 | 189 / 208 |
| FP8 dynamic activations | 3.04 % | +0.051 pp [−0.08, +0.20], p = 0.61 | 0.74 GB | 0.0312 | 200 / 208 |

Two results shaped the choice:

- **Weights tolerate four bits; activations do not.** NVFP4 W4A4 is the only
  variant with a statistically significant regression (CI excludes zero,
  p = 0.006), and it is worst on babble noise (10.43 % against bf16's 8.29 %).
  Quantizing weights only is what keeps 4-bit free.
- **Four bits is a better deal than eight here.** NVFP4 saves 38 % more memory
  than INT8 weight-only for the same undetectable accuracy cost, which inverts the
  usual expectation that fewer bits means more risk.

Nothing was faster than bf16 at any width, for the launch-bound reason above.

### Caveats

- **English-only corpus.** These figures say nothing directly about the other 24
  languages the base model supports. For the pure casts (bf16/fp16) there is a
  strong prior that a weight cast behaves the same everywhere; for NVFP4 that prior
  is weaker, because the scales were calibrated on English speech. If you use it on
  another language, measure.
- **208 clips bounds the resolution.** Differences below roughly 0.3 pp are beyond
  what this corpus can see.
- **NVFP4 is tied to its tooling.** Built with ModelOpt 0.45.0 / PyTorch 2.12.1 /
  NeMo 2.7.3; `recipe.json` records the ModelOpt version. Cross-version restore is
  untested — treat the snapshot as rebuildable from source rather than archival.
- **Speed figures are RTX 5090 (sm_120).** The quality figures carry across
  hardware; the timings do not.
- Both `.nemo` files were cast from the upstream fp32 weights — the fp16 one is not
  a bf16 → fp16 round trip, which would inherit bf16's coarser mantissa for no
  benefit. NVFP4 was quantized from the bf16 checkpoint.

## Provenance

- **Base model:** `nvidia/parakeet-tdt-0.6b-v3`
- **bf16 / fp16:** fp32 weights cast, re-saved as `.nemo` (701 float tensors,
  627,090,582 parameters, unchanged key set)
- **NVFP4:** 217 encoder Linear layers quantized to NVFP4 W4A16 with NVIDIA
  TensorRT Model Optimizer 0.45.0, calibrated on 62 LibriSpeech clips held out
  from the evaluation set, compressed to packed 4-bit and serialized as
  safetensors + JSON
- **License:** CC BY 4.0 (inherited from the base model)
- **Built and evaluated with:** [`lightware-dev/blurt`](https://github.com/lightware-dev/blurt) —
  `scripts/build_bf16_ckpt.py`, `scripts/build_nvfp4_snapshot.py`,
  `scripts/make_eval_corpus.py` and `scripts/compare_precision.py`

Each snapshot build records the transcripts it produced in memory; the loader is
verified by reproducing them exactly from disk (20 / 20), since a mis-bound scale
buffer still yields fluent English.

Previously published as `lightware-dev/parakeet-tdt-0.6b-v3-bf16` and
`lightware-dev/parakeet-tdt-0.6b-v3-fp16`; both are now consolidated here.