add 44k-class bitrate comparison table
Browse files
README.md
CHANGED
|
@@ -38,6 +38,35 @@ Load instructions (canonical):
|
|
| 38 |
| `pytorch_model.bin` | decoder weights for **inference** (load with `NeuCodec._from_pretrained(..., decoder_depth=20)`) |
|
| 39 |
| `last.ckpt` | full PyTorch-Lightning checkpoint **with optimizer states + LR schedulers** for **resuming training** |
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## Training stages / which revision to pull
|
| 42 |
|
| 43 |
This model is trained in two stages. `main` always points at the **latest** stage.
|
|
|
|
| 38 |
| `pytorch_model.bin` | decoder weights for **inference** (load with `NeuCodec._from_pretrained(..., decoder_depth=20)`) |
|
| 39 |
| `last.ckpt` | full PyTorch-Lightning checkpoint **with optimizer states + LR schedulers** for **resuming training** |
|
| 40 |
|
| 41 |
+
## How WideCodec compares (44 kHz-class, by bitrate)
|
| 42 |
+
|
| 43 |
+
Offline benchmark on a **9,291-clip wideband set** (188 clean ≥44.1 kHz datasets),
|
| 44 |
+
faster-UTMOSv2 + spectral vs ground truth. **Read this by bitrate, not raw MOS** — these
|
| 45 |
+
codecs span a ~30× bitrate range. DAC/NeMo/EnCodec are high-bitrate multi-codebook RVQ
|
| 46 |
+
*acoustic* codecs; WideCodec is a **single-codebook, 50-tokens/s** code at **~0.8 kbps** —
|
| 47 |
+
the only one here whose token stream an autoregressive LLM/TTS model can practically predict.
|
| 48 |
+
|
| 49 |
+
| codec | native SR | tokens/s | codebooks | ~bitrate | UTMOSv2 | mel-L1 ↓ | HF≥11k | rolloff |
|
| 50 |
+
|---|--:|--:|--:|--:|--:|--:|--:|--:|
|
| 51 |
+
| ground truth | — | — | — | — | 2.822 | — | 0.00244 | 8263 |
|
| 52 |
+
| nvidia nemo44k | 44.1k | ~86 | RVQ (many) | ~6–9 kbps | **2.903** | 0.379 | 0.00196 | 8372 |
|
| 53 |
+
| neucodec (24k anchor) | 24k | 50 | 1 | ~0.8 kbps | 2.848 | 0.706 | 0.00005 | 6214 |
|
| 54 |
+
| dac | 44.1k | 86 | 9 | ~8 kbps | 2.672 | **0.341** | 0.00156 | 8185 |
|
| 55 |
+
| **WideCodec (ours)** | 44.1k | **50** | **1** | **~0.8 kbps** | 2.590 ∗ | 0.655 | 0.00229 | 8546 |
|
| 56 |
+
| snac44k | 44.1k | multi-scale | 3–4 | ~2.6 kbps | 2.340 | 0.493 | 0.00158 | 8402 |
|
| 57 |
+
| encodec48k | 48k | 150 | RVQ | 24 kbps | 2.042 | 0.458 | 0.00169 | 8800 |
|
| 58 |
+
|
| 59 |
+
**Bitrate = frame-rate × Σ log₂(codebook size).** WideCodec's FSQ is `levels=[4]×8,
|
| 60 |
+
num_quantizers=1` → 8·log₂4 = **16 bits/frame** × 50 frames/s = **800 bps = 0.8 kbps**
|
| 61 |
+
(vs DAC's 9 codebooks × 10 bits × 86 fps ≈ 8 kbps).
|
| 62 |
+
|
| 63 |
+
∗ The WideCodec row is the **pre-cleanft step-1.58M** checkpoint; the current published
|
| 64 |
+
checkpoint (gs-1.875M, clean-finetuned) is **not yet re-scored** on this benchmark. At
|
| 65 |
+
~0.8 kbps / 1 codebook it matches DAC's naturalness class using ~10× fewer bits, and unlike
|
| 66 |
+
the 24 kHz anchor it carries real high-band content (rolloff 8.5 kHz; HF energy ≈ 94% of GT
|
| 67 |
+
vs the anchor's ~0). Full report:
|
| 68 |
+
[evaluation/REPORT_44k_class_codecs.md](https://github.com/Scicom-AI-Enterprise-Organization/WideCodec/blob/main/evaluation/REPORT_44k_class_codecs.md).
|
| 69 |
+
|
| 70 |
## Training stages / which revision to pull
|
| 71 |
|
| 72 |
This model is trained in two stages. `main` always points at the **latest** stage.
|