File size: 11,713 Bytes

0db3d22

---
license: apache-2.0
pipeline_tag: audio-to-audio
tags:
- speech-restoration
- speech-enhancement
- audio-to-audio
- autoregressive
- neural-audio-codec
- audio
---

<div align="center">

  <img src="https://cdn-uploads.huggingface.co/production/uploads/65ae44614b5cc40be0e4fe72/qovLywMpRwPbWSyiCTbY-.png" alt="logo" width="800">

  [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Tech Report](https://img.shields.io/badge/📄-Tech%20Report-red.svg)](https://huggingface.co/nineninesix/diamond-1.0/resolve/main/TECH_REPORT.pdf) [![Model](https://img.shields.io/badge/🤗-Demo-yellow.svg)](https://huggingface.co/spaces/nineninesix/diamond) [![Website](https://img.shields.io/badge/⚡-nineninesix.ai-brightgreen.svg)](https://www.nineninesix.ai)

</div>

# **DIAMOND**

**A sequence-to-sequence model for speech restoration via an autoregressive RQ-Transformer over neural audio codec tokens**

[**Diamond**](https://huggingface.co/nineninesix/diamond-1.0/resolve/main/TECH_REPORT.pdf) turns degraded audio into near-studio 44.1 kHz speech. A bidirectional Transformer encodes the degraded mel-spectrogram; an autoregressive decoder with cross-attention predicts the tokens of a frozen neural audio codec (Descript Audio Codec, 9-book RVQ), which synthesizes the restored waveform.

Restoration is not denoising. A codec cuts the band above 8 kHz, clipping cuts the peaks, a lost packet zeroes the frame — no mask can return what the observation no longer contains. The missing content has to be **generated** from the surviving formants, which is why Diamond is a generative seq2seq model rather than a filter.

Diamond is trained **from scratch** — no pretrained backbone — and reaches the level of the strongest open restorers while remaining the best from-scratch restorer in its class.

## Highlights

- **Two-transformer read-out.** A *time-transformer* models the frame sequence; a compact *depth-transformer* walks the 9 RVQ codebooks inside each frame. This restores the RVQ causal chain and distributes the output projection — where earlier models read all nine books out of a single frame vector with parallel heads.
- **From scratch, 166.6M trainable params.** No pretrained speech backbone, 63 GPU-hours of training for the released checkpoint.
- **44.1 kHz output.** The frozen DAC decoder synthesizes full-band audio; sibilants and "air" above 8 kHz are regenerated, not merely passed through.
- **Content is measured, not assumed.** A generative restorer can sound clean while smearing words, so CER against the ground-truth transcript is a mandatory second axis alongside DNSMOS.
- **Calibrated degradation.** Inputs come from a DSP augmentor whose codec palette is fitted per-sample to a real degraded-speech distribution, not from a bag of random effects.

## Hear the difference

Real, heavily degraded speech restored to 44.1 kHz. DNSMOS-P.835 OVRL rises by more
than a full point on each clip — audible even without headphones. Try it live in the
[**Diamond Space**](https://huggingface.co/spaces/nineninesix/Diamond).

<table>
  <thead>
    <tr><th>#</th><th>Degraded input</th><th>Restored — 44.1&nbsp;kHz</th><th>DNSMOS&nbsp;OVRL</th></tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example1_degraded.wav"></audio></td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example1_restored.wav"></audio></td>
      <td>2.16&nbsp;→&nbsp;<b>3.50</b> &nbsp;(+1.34)</td>
    </tr>
    <tr>
      <td>2</td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example2_degraded.wav"></audio></td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example2_restored.wav"></audio></td>
      <td>2.83&nbsp;→&nbsp;<b>3.59</b> &nbsp;(+0.76)</td>
    </tr>
    <tr>
      <td>3</td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example3_degraded.wav"></audio></td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example3_restored.wav"></audio></td>
      <td>2.56&nbsp;→&nbsp;<b>3.65</b> &nbsp;(+1.10)</td>
    </tr>
    <tr>
      <td>4</td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example4_degraded.wav"></audio></td>
      <td><audio controls src="https://huggingface.co/nineninesix/diamond-1.0/resolve/main/samples/example4_restored.wav"></audio></td>
      <td>3.32&nbsp;→&nbsp;<b>3.89</b> &nbsp;(+0.57)</td>
    </tr>
  </tbody>
</table>

## Model Details

- **Developed by:** [nineninesix.ai](https://www.nineninesix.ai)
- **Model type:** Autoregressive sequence-to-sequence speech restoration (encoder + RQ-Transformer decoder over codec tokens)
- **Encoder:** Bidirectional Transformer, no downsampling — 20 layers, 512d, 8 heads (63.9M)
- **Decoder / time-transformer:** causal self-attention + cross-attention — 20 layers, 512d, 8 heads (88.7M)
- **Codebook read-out / depth-transformer:** local AR over the 9 RVQ codes of a frame — 4 layers, 384d, 6 heads (14.0M)
- **Parameters:** ≈ 166.6M trainable (the DAC codec, ~74M, is frozen and downloaded at runtime)
- **Audio codec:** [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec) — 44.1 kHz, 9-codebook RVQ, 86 frames/s
- **Primitives:** RMSNorm, learnable per-layer RoPE, QK-Norm, LayerScale, GELU FFN
- **Input / output sample rate:** any input, resampled to 24 kHz internally → 44,100 Hz output
- **Training data:** 681.5 h of studio speech, ~2.5k speakers, 28% natively wide-band.
- **Languages:** English
- **License:** Apache 2.0

## Benchmarks

Measured on 750 real degraded recordings — identical clips for every model. DNSMOS-P.835 (raw `sig_bak_ovr.onnx`) for perceptual quality, CER against the ground-truth transcript for content preservation.

| Model | DNSMOS OVRL ↑ | SIG ↑ | BAK ↑ | CER (median) ↓ |
|---|---|---|---|---|
| Sidon | **3.923** | **4.129** | **4.416** | 0.016 |
| **Diamond (ours)** | 3.829 | 4.056 | 4.348 | 0.028 |
| RE-USE | 3.789 | 4.003 | 4.344 | **0.014** |
| Resemble Enhance | 3.764 | 3.994 | 4.301 | 0.027 |
| UniSE | 3.752 | 4.014 | 4.261 | 0.027 |
| VoiceFixer | 3.566 | 3.790 | 4.226 | 0.042 |
| input (degraded) | 3.575 | 3.890 | 4.082 | 0.014 |

Diamond improves ≈88% of clips (mean ΔOVRL +0.255) and places second of six on perceptual quality — ahead of RE-USE, which trains on roughly four times the data. Both systems that beat it on CER are **non-autoregressive**: the gap is the exposure bias inherent to AR decoding, not a capacity limit. The one other autoregressive system here, UniSE, lands on exactly Diamond's mean CER (0.131) by a different route — which suggests the tail belongs to the paradigm, not to this particular recipe.

## Links

- **Checkpoint:** [huggingface.co/nineninesix/diamond-1.0](https://huggingface.co/nineninesix/diamond-1.0)
- **Technical Report:** [TECH_REPORT.pdf](https://huggingface.co/nineninesix/diamond-1.0/resolve/main/TECH_REPORT.pdf)
- **Training:** [github.com/nineninesix-ai/diamond-train](https://github.com/nineninesix-ai/diamond-train)
- **Website:** [nineninesix.ai](https://www.nineninesix.ai)

## What it's good for

Offline restoration and **dataset cleansing** — turning large volumes of degraded recordings (podcasts, interviews, archival and user-generated audio that has passed through lossy codecs) into material clean enough to train on. This is the task Diamond was built for and the one it is measured on.

**Keep in mind:**

- **A thin CER tail.** As an autoregressive decoder, Diamond occasionally smears words on hard clips. Inference safeguards (chunked decoding, repetition penalty) hold it down, but the tail is real — check the transcript when content fidelity is critical.
- **Decoding is serial, not real-time.** One frame at a time, 86 frames per second of audio, plus the depth pass. This is offline restoration, not a live filter.
- **English-centric.** Training data is English; other languages are untested.
- **It generates, it does not filter.** Content above the codec's cutoff is invented from context — plausible, not recovered. Do not treat the output as forensic evidence of what was said.

> **Use it responsibly.** Restored speech is synthesized speech. Don't present it as an unaltered recording, and don't use it to fabricate or misattribute what someone said.

## Acknowledgments

Built on the [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec) for the frozen token space. Trained on [LibriTTS-R](https://www.openslr.org/141/), [Hi-Fi TTS](https://www.openslr.org/109/), [VCTK](https://doi.org/10.7488/ds/2645). Evaluated with [DNSMOS P.835](https://github.com/microsoft/DNS-Challenge) and [faster-whisper](https://github.com/SYSTRAN/faster-whisper).

## Citation

If you use this work in your research, please cite:

```bibtex
@software{diamond_2026,
  author = {Almaz Zholdoshbek uulu, Ulanbek Abdurazakov, Denis Pavlov and Nursultan Bakashov},
  title = {Diamond: A Sequence-to-Sequence Model for Speech Restoration via an Autoregressive RQ-Transformer over Neural Audio Codec Tokens},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nineninesix/diamond-1.0}},
  note = {Trained from scratch; no pretrained backbone}
}
```

## References

```bibtex
@inproceedings{kumar2023dac,
  title={High-Fidelity Audio Compression with Improved RVQGAN},
  author={Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan},
  booktitle={NeurIPS},
  year={2023},
  note={arXiv:2306.06546}
}

@inproceedings{lee2022rqtransformer,
  title={Autoregressive Image Generation using Residual Quantization},
  author={Lee, Doyup and Kim, Chiheon and Kim, Saehoon and Cho, Minsu and Han, Wook-Shin},
  booktitle={CVPR},
  year={2022},
  note={arXiv:2203.01941}
}

@article{defossez2024moshi,
  title={Moshi: a speech-text foundation model for real-time dialogue},
  author={D{\'e}fossez, Alexandre and Mazar{\'e}, Laurent and Orsini, Manu and Royer, Am{\'e}lie and P{\'e}rez, Patrick and J{\'e}gou, Herv{\'e} and Grave, Edouard and Zeghidour, Neil},
  journal={arXiv preprint arXiv:2410.00037},
  year={2024}
}

@inproceedings{copet2023musicgen,
  title={Simple and Controllable Music Generation},
  author={Copet, Jade and Kreuk, Felix and Gat, Itai and Remez, Tal and Kant, David and Synnaeve, Gabriel and Adi, Yossi and D{\'e}fossez, Alexandre},
  booktitle={NeurIPS},
  year={2023},
  note={arXiv:2306.05284}
}

@inproceedings{koizumi2023librittsr,
  title={LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus},
  author={Koizumi, Yuma and Zen, Heiga and Karita, Shigeki and Ding, Yifan and Yatabe, Kohei and Morioka, Nobuyuki and Bacchiani, Michiel and Zhang, Yu and Han, Wei and Bapna, Ankur},
  booktitle={Interspeech},
  year={2023},
  note={arXiv:2305.18802}
}

@inproceedings{reddy2022dnsmos,
  title={DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors},
  author={Reddy, Chandan K. A. and Gopal, Vishak and Cutler, Ross},
  booktitle={ICASSP},
  year={2022},
  note={arXiv:2110.01763}
}
```

## License

Apache 2.0 — this model and its weights are released under the [Apache License 2.0](https://opensource.org/licenses/Apache-2.0).

The frozen [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec) is downloaded at runtime and carries its own license (MIT).