Duplicate from nineninesix/diamond-1.0

Co-authored-by: Den Pavloff <Simonlob@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+TECH_REPORT.pdf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,202 @@
+---
+license: apache-2.0
+pipeline_tag: audio-to-audio
+tags:
+- speech-restoration
+- speech-enhancement
+- audio-to-audio
+- autoregressive
+- neural-audio-codec
+- audio
+---
+
+<div align="center">
+
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/65ae44614b5cc40be0e4fe72/qovLywMpRwPbWSyiCTbY-.png" alt="logo" width="800">
+
+  [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Tech Report](https://img.shields.io/badge/📄-Tech%20Report-red.svg)](https://huggingface.co/nineninesix/diamond-1.0/resolve/main/TECH_REPORT.pdf) [![Model](https://img.shields.io/badge/🤗-Demo-yellow.svg)](https://huggingface.co/spaces/nineninesix/diamond) [![Website](https://img.shields.io/badge/⚡-nineninesix.ai-brightgreen.svg)](https://www.nineninesix.ai)
+
+</div>
+
+# **DIAMOND**
+
+**A sequence-to-sequence model for speech restoration via an autoregressive RQ-Transformer over neural audio codec tokens**
+
+[**Diamond**](https://huggingface.co/nineninesix/diamond-1.0/resolve/main/TECH_REPORT.pdf) turns degraded audio into near-studio 44.1 kHz speech. A bidirectional Transformer encodes the degraded mel-spectrogram; an autoregressive decoder with cross-attention predicts the tokens of a frozen neural audio codec (Descript Audio Codec, 9-book RVQ), which synthesizes the restored waveform.
+
+Restoration is not denoising. A codec cuts the band above 8 kHz, clipping cuts the peaks, a lost packet zeroes the frame — no mask can return what the observation no longer contains. The missing content has to be **generated** from the surviving formants, which is why Diamond is a generative seq2seq model rather than a filter.
+
+Diamond is trained **from scratch** — no pretrained backbone — and reaches the level of the strongest open restorers while remaining the best from-scratch restorer in its class.
+
+## Highlights
+
+- **Two-transformer read-out.** A *time-transformer* models the frame sequence; a compact *depth-transformer* walks the 9 RVQ codebooks inside each frame. This restores the RVQ causal chain and distributes the output projection — where earlier models read all nine books out of a single frame vector with parallel heads.
+- **From scratch, 166.6M trainable params.** No pretrained speech backbone, 63 GPU-hours of training for the released checkpoint.
+- **44.1 kHz output.** The frozen DAC decoder synthesizes full-band audio; sibilants and "air" above 8 kHz are regenerated, not merely passed through.
+- **Content is measured, not assumed.** A generative restorer can sound clean while smearing words, so CER against the ground-truth transcript is a mandatory second axis alongside DNSMOS.
+- **Calibrated degradation.** Inputs come from a DSP augmentor whose codec palette is fitted per-sample to a real degraded-speech distribution, not from a bag of random effects.
+
+## Hear the difference
+
+Real, heavily degraded speech restored to 44.1 kHz. DNSMOS-P.835 OVRL rises by more
+than a full point on each clip — audible even without headphones. Try it live in the
+[**Diamond Space**](https://huggingface.co/spaces/nineninesix/Diamond).
+
+<table>
+  <thead>
+    <tr><th>#</th><th>Degraded input</th><th>Restored — 44.1&nbsp;kHz</th><th>DNSMOS&nbsp;OVRL</th></tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>1</td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example1_degraded.wav"></audio></td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example1_restored.wav"></audio></td>
+      <td>2.16&nbsp;→&nbsp;<b>3.50</b> &nbsp;(+1.34)</td>
+    </tr>
+    <tr>
+      <td>2</td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example2_degraded.wav"></audio></td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example2_restored.wav"></audio></td>
+      <td>2.83&nbsp;→&nbsp;<b>3.59</b> &nbsp;(+0.76)</td>
+    </tr>
+    <tr>
+      <td>3</td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example3_degraded.wav"></audio></td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example3_restored.wav"></audio></td>
+      <td>2.56&nbsp;→&nbsp;<b>3.65</b> &nbsp;(+1.10)</td>
+    </tr>
+    <tr>
+      <td>4</td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example4_degraded.wav"></audio></td>
+      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example4_restored.wav"></audio></td>
+      <td>3.32&nbsp;→&nbsp;<b>3.89</b> &nbsp;(+0.57)</td>
+    </tr>
+  </tbody>
+</table>
+
+## Model Details
+
+- **Developed by:** [nineninesix.ai](https://www.nineninesix.ai)
+- **Model type:** Autoregressive sequence-to-sequence speech restoration (encoder + RQ-Transformer decoder over codec tokens)
+- **Encoder:** Bidirectional Transformer, no downsampling — 20 layers, 512d, 8 heads (63.9M)
+- **Decoder / time-transformer:** causal self-attention + cross-attention — 20 layers, 512d, 8 heads (88.7M)
+- **Codebook read-out / depth-transformer:** local AR over the 9 RVQ codes of a frame — 4 layers, 384d, 6 heads (14.0M)
+- **Parameters:** ≈ 166.6M trainable (the DAC codec, ~74M, is frozen and downloaded at runtime)
+- **Audio codec:** [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec) — 44.1 kHz, 9-codebook RVQ, 86 frames/s
+- **Primitives:** RMSNorm, learnable per-layer RoPE, QK-Norm, LayerScale, GELU FFN
+- **Input / output sample rate:** any input, resampled to 24 kHz internally → 44,100 Hz output
+- **Training data:** 681.5 h of studio speech, ~2.5k speakers, 28% natively wide-band.
+- **Languages:** English
+- **License:** Apache 2.0
+
+## Benchmarks
+
+Measured on 750 real degraded recordings — identical clips for every model. DNSMOS-P.835 (raw `sig_bak_ovr.onnx`) for perceptual quality, CER against the ground-truth transcript for content preservation.
+
+| Model | DNSMOS OVRL ↑ | SIG ↑ | BAK ↑ | CER (median) ↓ |
+|---|---|---|---|---|
+| Sidon | **3.923** | **4.129** | **4.416** | 0.016 |
+| **Diamond (ours)** | 3.829 | 4.056 | 4.348 | 0.028 |
+| RE-USE | 3.789 | 4.003 | 4.344 | **0.014** |
+| Resemble Enhance | 3.764 | 3.994 | 4.301 | 0.027 |
+| UniSE | 3.752 | 4.014 | 4.261 | 0.027 |
+| VoiceFixer | 3.566 | 3.790 | 4.226 | 0.042 |
+| input (degraded) | 3.575 | 3.890 | 4.082 | 0.014 |
+
+Diamond improves ≈88% of clips (mean ΔOVRL +0.255) and places second of six on perceptual quality — ahead of RE-USE, which trains on roughly four times the data. Both systems that beat it on CER are **non-autoregressive**: the gap is the exposure bias inherent to AR decoding, not a capacity limit. The one other autoregressive system here, UniSE, lands on exactly Diamond's mean CER (0.131) by a different route — which suggests the tail belongs to the paradigm, not to this particular recipe.
+
+## Links
+
+- **Checkpoint:** [huggingface.co/nineninesix/diamond-1.0](https://huggingface.co/nineninesix/diamond-1.0)
+- **Technical Report:** [TECH_REPORT.pdf](https://huggingface.co/nineninesix/diamond-1.0/resolve/main/TECH_REPORT.pdf)
+- **Training:** [github.com/nineninesix-ai/diamond-train](https://github.com/nineninesix-ai/diamond-train)
+- **Website:** [nineninesix.ai](https://www.nineninesix.ai)
+
+## What it's good for
+
+Offline restoration and **dataset cleansing** — turning large volumes of degraded recordings (podcasts, interviews, archival and user-generated audio that has passed through lossy codecs) into material clean enough to train on. This is the task Diamond was built for and the one it is measured on.
+
+**Keep in mind:**
+
+- **A thin CER tail.** As an autoregressive decoder, Diamond occasionally smears words on hard clips. Inference safeguards (chunked decoding, repetition penalty) hold it down, but the tail is real — check the transcript when content fidelity is critical.
+- **Decoding is serial, not real-time.** One frame at a time, 86 frames per second of audio, plus the depth pass. This is offline restoration, not a live filter.
+- **English-centric.** Training data is English; other languages are untested.
+- **It generates, it does not filter.** Content above the codec's cutoff is invented from context — plausible, not recovered. Do not treat the output as forensic evidence of what was said.
+
+> **Use it responsibly.** Restored speech is synthesized speech. Don't present it as an unaltered recording, and don't use it to fabricate or misattribute what someone said.
+
+## Acknowledgments
+
+Built on the [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec) for the frozen token space. Trained on [LibriTTS-R](https://www.openslr.org/141/), [Hi-Fi TTS](https://www.openslr.org/109/), [VCTK](https://doi.org/10.7488/ds/2645). Evaluated with [DNSMOS P.835](https://github.com/microsoft/DNS-Challenge) and [faster-whisper](https://github.com/SYSTRAN/faster-whisper).
+
+## Citation
+
+If you use this work in your research, please cite:
+
+```bibtex
+@software{diamond_2026,
+  author = {Almaz Zholdoshbek uulu, Ulanbek Abdurazakov, Denis Pavlov and Nursultan Bakashov},
+  title = {Diamond: A Sequence-to-Sequence Model for Speech Restoration via an Autoregressive RQ-Transformer over Neural Audio Codec Tokens},
+  year = {2026},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/nineninesix/diamond-1.0}},
+  note = {Trained from scratch; no pretrained backbone}
+}
+```
+
+## References
+
+```bibtex
+@inproceedings{kumar2023dac,
+  title={High-Fidelity Audio Compression with Improved RVQGAN},
+  author={Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan},
+  booktitle={NeurIPS},
+  year={2023},
+  note={arXiv:2306.06546}
+}
+
+@inproceedings{lee2022rqtransformer,
+  title={Autoregressive Image Generation using Residual Quantization},
+  author={Lee, Doyup and Kim, Chiheon and Kim, Saehoon and Cho, Minsu and Han, Wook-Shin},
+  booktitle={CVPR},
+  year={2022},
+  note={arXiv:2203.01941}
+}
+
+@article{defossez2024moshi,
+  title={Moshi: a speech-text foundation model for real-time dialogue},
+  author={D{\'e}fossez, Alexandre and Mazar{\'e}, Laurent and Orsini, Manu and Royer, Am{\'e}lie and P{\'e}rez, Patrick and J{\'e}gou, Herv{\'e} and Grave, Edouard and Zeghidour, Neil},
+  journal={arXiv preprint arXiv:2410.00037},
+  year={2024}
+}
+
+@inproceedings{copet2023musicgen,
+  title={Simple and Controllable Music Generation},
+  author={Copet, Jade and Kreuk, Felix and Gat, Itai and Remez, Tal and Kant, David and Synnaeve, Gabriel and Adi, Yossi and D{\'e}fossez, Alexandre},
+  booktitle={NeurIPS},
+  year={2023},
+  note={arXiv:2306.05284}
+}
+
+@inproceedings{koizumi2023librittsr,
+  title={LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus},
+  author={Koizumi, Yuma and Zen, Heiga and Karita, Shigeki and Ding, Yifan and Yatabe, Kohei and Morioka, Nobuyuki and Bacchiani, Michiel and Zhang, Yu and Han, Wei and Bapna, Ankur},
+  booktitle={Interspeech},
+  year={2023},
+  note={arXiv:2305.18802}
+}
+
+@inproceedings{reddy2022dnsmos,
+  title={DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors},
+  author={Reddy, Chandan K. A. and Gopal, Vishak and Cutler, Ross},
+  booktitle={ICASSP},
+  year={2022},
+  note={arXiv:2110.01763}
+}
+```
+
+## License
+
+Apache 2.0 — this model and its weights are released under the [Apache License 2.0](https://opensource.org/licenses/Apache-2.0).
+
+The frozen [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec) is downloaded at runtime and carries its own license (MIT).

TECH_REPORT.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35eac5ff11270f30fc2178c7c470e4a58234d7dc0114dea1fdc578fce822351d
+size 1231121

diamond.json

@@ -0,0 +1,72 @@
+{
+  "encoder": {
+    "n_mels": 80,
+    "d_model": 512,
+    "n_heads": 8,
+    "n_layers": 20,
+    "ff_dim": 2048,
+    "conv_channels": [
+      512,
+      512
+    ],
+    "conv_kernel": 3,
+    "conv_stride": [
+      1,
+      1
+    ],
+    "pos_encoding": "rope",
+    "norm": "rms",
+    "activation": "gelu",
+    "dropout": 0.1,
+    "bidirectional": true,
+    "layer_scale_init": 0.05,
+    "rope_learnable": true,
+    "qk_norm": true
+  },
+  "decoder": {
+    "d_model": 512,
+    "n_heads": 8,
+    "n_layers": 20,
+    "ff_dim": 2048,
+    "n_codebooks": 9,
+    "vocab_size": 1024,
+    "pad_token_id": 1024,
+    "bos_token_id": 1024,
+    "pos_encoding": "rope",
+    "norm": "rms",
+    "activation": "gelu",
+    "dropout": 0.1,
+    "cross_attn_every_layer": true,
+    "rope_learnable": true,
+    "qk_norm": true,
+    "layer_scale_init": 0.05
+  },
+  "depth": {
+    "d_model": 384,
+    "n_heads": 6,
+    "n_layers": 4,
+    "ff_dim": 1536,
+    "dropout": 0.1,
+    "qk_norm": true,
+    "layer_scale_init": 0.05
+  },
+  "mel": {
+    "sample_rate": 24000,
+    "n_fft": 1024,
+    "hop_length": 280,
+    "win_length": 1024,
+    "n_mels": 80,
+    "fmin": 0,
+    "fmax": 8000,
+    "log_clamp_db": -80,
+    "normalization": "minmax"
+  },
+  "dac": {
+    "model": "44khz",
+    "sample_rate": 44100,
+    "n_codebooks": 9,
+    "vocab_size": 1024,
+    "frame_rate": 86.13,
+    "checkpoint_path": null
+  }
+}

diamond.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6703b9dc88844722e45980e26b06a35f7167bb6d012c3b642dea6f3500bd06f7
+size 666592168

samples/example1_degraded.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b39e5b969cc7ee9f45600d7b33cab00c22db81ce343f6ca5da46f9454864be17
+size 498812

samples/example1_restored.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bc0c0981c6c6fcd6a6a4ea2472c9c09c4408e1f4456f1f76385ead177afb8ce
+size 916530

samples/example2_degraded.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fab31faca02e79be0f07ab646dda7d68fb82e53f295509f7b040a55efa015e8
+size 229052

samples/example2_restored.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ddc60a40d86bf5c8fa1a8ffde0aa102a4fda333abc8d12dc46cc564c8f936713
+size 420846

samples/example3_degraded.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ba005eed89c186ee4cd265959d1db115bd9afd79eec0f3b2794c296fc3a61e2
+size 535484

samples/example3_restored.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e669fdeb47db0bebc8ab54fcac7eb78604688b4c26796aa0b84ee9fd593d10be
+size 983916

samples/example4_degraded.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bfa561451972f91c339777680123db23b8c2ac18fdc4435d80f711755e14ec9
+size 555884

samples/example4_restored.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f4e42e41ee2d3f7ca39d1604582bb21dd7138c89024f18ad8ccc9a08a517e09
+size 1021400