Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/architecture.png filter=lfs diff=lfs merge=lfs -text
+assets/codec_firing_examples.png filter=lfs diff=lfs merge=lfs -text
+assets/english_reconstruction.png filter=lfs diff=lfs merge=lfs -text
+assets/multilingual_reconstruction.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,227 @@
+---
+license: apache-2.0
+language:
+  - en
+  - ja
+  - zh
+  - multilingual
+datasets:
+  - amphion/Emilia-Dataset
+  - nvidia/hifitts-2
+  - reazon-research/reazonspeech
+  - MLCommons/peoples_speech
+  - facebook/voxpopuli
+  - facebook/multilingual_librispeech
+  - facebook/omnilingual-asr-corpus
+  - ai4bharat/IndicVoices
+pipeline_tag: audio-to-audio
+tags:
+  - audio
+  - speech
+  - neural-audio-codec
+  - speech-tokenizer
+  - variable-frame-rate
+  - streaming
+  - text-to-speech
+---
+
+# RubatoCodec
+
+[![Code: GitHub](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/third-intelligence/RubatoCodec)
+![Paper: coming soon](https://img.shields.io/badge/Paper-coming%20soon-lightgrey)
+[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue)](https://www.apache.org/licenses/LICENSE-2.0)
+
+RubatoCodec is a neural speech codec that divides speech into a fixed number of
+variable-duration events. Each event carries discrete code indices and an
+explicit duration. Event placement adapts to speech content, while the number of
+events is determined exactly by the utterance length, so the token sequence
+length a speech generator must produce is predictable. A causal decoder renders
+completed events incrementally, which makes the codec suitable for streaming
+text-to-speech.
+
+The name comes from the musical term *rubato*: local timing flexes while the
+overall tempo is preserved.
+
+- **Adaptive Firing with an exact event budget.** Multi-scale latent
+  dissimilarity and waveform spectral flux guide cumulative allocation on a
+  50-Hz latent grid. For `T` frames the event count is exactly
+  `max(1, min(T, round(0.25 · T)))`, giving 12.5 events/s. No learned boundary
+  predictor, boundary labels, or rate loss is used.
+- **Streaming decoding.** Events can be decoded one at a time or in chunks.
+  Event-by-event decoding takes about 11 ms per event on an NVIDIA H200, well
+  below the ~80 ms of audio each event represents on average.
+- **Hybrid residual quantizer with depth control.** A semantically supervised
+  FSQ first layer is followed by 15 DAC-style residual VQ layers, each with
+  4,096 entries. Any prefix depth from 1 to 16 can be decoded, giving code-index
+  bitrates of 0.15–2.4 kbit/s plus duration information.
+- **Multilingual.** Trained on approximately 375,000 hours of speech, mostly
+  English, Japanese, and Chinese, with 16-kHz input and 44.1-kHz output.
+
+![RubatoCodec architecture](assets/architecture.png)
+
+## Model details
+
+| | |
+|---|---|
+| Input / output sample rate | 16 kHz / 44.1 kHz |
+| Latent grid | 50 Hz |
+| Event rate | 12.5 events/s (retention ratio 0.25) |
+| Quantizer | 16 layers × 4,096 entries; layer 0 FSQ with semantic distillation, layers 1–15 residual VQ |
+| Bitrate | 1.23 kbit/s at depth 8, 2.43 kbit/s at depth 16 (LibriSpeech, including duration information) |
+| Decoder | Fully causal Transformer + ISTFT head, streaming capable |
+| Parameters | 246M codec (72M encoder, 173M decoder, 1M quantizer) + frozen [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) (580M, encoding only) |
+| License | Apache 2.0 |
+
+## Usage
+
+Install the package from the [GitHub repository](https://github.com/third-intelligence/RubatoCodec):
+
+```bash
+git clone https://github.com/third-intelligence/RubatoCodec
+cd RubatoCodec
+uv sync
+```
+
+Encode and decode:
+
+```python
+import soundfile as sf
+import torch
+
+from rubatocodec import RubatoCodec
+
+codec = RubatoCodec.from_pretrained("third-intelligence/RubatoCodec", device="cuda")
+
+data, sr = sf.read("speech.wav", dtype="float32")  # mono waveform
+wav = torch.from_numpy(data)
+
+enc = codec.encode(wav, sr=sr)       # {"codes": (16, K) int64, "durs": (K,) int64}
+audio = codec.decode(enc)            # (1, 1, T_audio) at 44.1 kHz, full depth
+audio = codec.decode(enc, depth=8)   # lower bitrate with the first 8 layers
+
+sf.write("recon.wav", audio[0, 0].cpu().numpy(), codec.output_sr)
+```
+
+`durs` gives each event's duration in 20-ms grid frames. Loading also downloads
+`facebook/w2v-bert-2.0`, which is used only for encoding.
+
+Streaming decoding, inference settings, the code format, and training are
+documented in the repository:
+[inference guide](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/inference.md),
+[architecture](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/architecture.md),
+[training](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/training.md).
+
+## Evaluation
+
+RubatoCodec is evaluated on LibriSpeech test-clean, VCTK, JVS nonpara30, JVNV,
+and seed-tts-eval test-zh against 13 other codecs. At comparable bitrates, it
+shows the best intelligibility preservation among the compared codecs on the
+English and Japanese benchmarks and high predicted quality, while the strongest
+comparators achieve higher PESQ-WB and STOI.
+
+dWER/dCER compare ASR transcripts of the reconstructed and original audio
+(Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is
+ECAPA-TDNN speaker similarity. Bitrates include duration information.
+
+The tables below show a subset of the compared codecs. **Bold** and
+<ins>underline</ins> mark the best and second-best scores within each bitrate
+band.
+
+**LibriSpeech test-clean (English)**
+
+| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dWER (%) ↓ | SIM ↑ |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| *~1.2 kbit/s* | | | | | | | | |
+| FlexiCodec, τ=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | <ins>0.83</ins> | 0.805 |
+| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 | <ins>0.879</ins> |
+| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.885** |
+| DTM-Codec, 80-Hz target | 81.5 | 1.28 | <ins>3.421</ins> | **3.334** | <ins>2.971</ins> | <ins>0.947</ins> | 1.41 | 0.845 |
+| **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | <ins>3.327</ins> | 2.684 | 0.931 | **0.75** | 0.862 |
+| *~2.4 kbit/s* | | | | | | | | |
+| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | <ins>3.177</ins> | <ins>0.958</ins> | <ins>0.53</ins> | 0.931 |
+| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | <ins>3.288</ins> | 3.094 | 0.947 | 0.67 | <ins>0.933</ins> |
+| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | <ins>3.239</ins> | 3.203 | **3.536** | **0.968** | 0.62 | **0.943** |
+| **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.920 |
+
+**JVNV (Japanese emotional speech with nonverbal vocalizations)**
+
+On expressive speech, RubatoCodec keeps dCER below 2% at both bitrates, while
+every other compared codec exceeds 3%.
+
+| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dCER (%) ↓ | SIM ↑ |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| *~1.2 kbit/s* | | | | | | | | |
+| FlexiCodec, τ=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.771 |
+| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.860 |
+| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | <ins>5.44</ins> | **0.909** |
+| DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 |
+| **RubatoCodec, depth 8** | 12.5 | 1.23 | <ins>2.628</ins> | <ins>3.164</ins> | <ins>2.650</ins> | <ins>0.907</ins> | **1.92** | <ins>0.903</ins> |
+| *~2.4 kbit/s* | | | | | | | | |
+| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | <ins>3.041</ins> | <ins>0.931</ins> | <ins>3.01</ins> | <ins>0.945</ins> |
+| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.927 |
+| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.953** |
+| **RubatoCodec, depth 16** | 12.5 | 2.43 | <ins>2.651</ins> | <ins>3.138</ins> | 3.032 | 0.929 | **1.54** | 0.941 |
+
+**Quality versus bitrate** (RubatoCodec is marked with a red star)
+
+![English reconstruction versus bitrate](assets/english_reconstruction.png)
+
+![Japanese and Mandarin reconstruction versus bitrate](assets/multilingual_reconstruction.png)
+
+**Adaptive event placement.** Each rectangle in the lower strip is one event.
+Longer events cover pauses and low-energy regions, while speech receives
+shorter events, with the same 12.5 events/s budget in every utterance.
+
+![Adaptive event placement on English, Japanese, and Mandarin speech](assets/codec_firing_examples.png)
+
+The full protocol, all compared codecs and operating points, and throughput
+measurements are in the
+[evaluation document](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/evaluation.md)
+and the paper.
+
+## Training data
+
+Approximately 375,000 hours of public and internal speech. Public sources include
+[Emilia-YODAS](https://huggingface.co/datasets/amphion/Emilia-Dataset),
+[HiFiTTS-2](https://huggingface.co/datasets/nvidia/hifitts-2) (44-kHz subset),
+[ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech),
+[The People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech)
+(excluding ShareAlike-licensed data),
+[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli),
+[Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech)
+(excluding English),
+[Omnilingual ASR Corpus](https://huggingface.co/datasets/facebook/omnilingual-asr-corpus),
+and [IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices). English, Japanese,
+and Chinese account for approximately 44%, 27%, and 13% of the data, and other
+languages for the remaining 16%.
+
+## Limitations
+
+- RubatoCodec is trained on speech only and is intended for speech. It is not
+  designed for music or general audio.
+- Encoding requires the whole utterance, because the event budget is defined at
+  the utterance level. Only decoding is streaming.
+- Signal-fidelity metrics (PESQ-WB, STOI) are below those of the strongest
+  compared codecs at similar bitrates.
+- Evaluation covers English, Japanese, and Mandarin, all of which appear in the
+  training data. Performance on other languages has not been evaluated.
+- Like other speech codecs, RubatoCodec can be used in speech generation systems
+  capable of voice cloning. Do not use it to impersonate people or to create
+  deceptive content without consent.
+
+## Acknowledgements
+
+RubatoCodec builds on [X-Codec-2.0](https://github.com/zhenye234/X-Codec-2.0)
+and uses [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) as its
+frozen semantic encoder. It also adapts components from
+[BigCodec](https://github.com/Aria-K-Alethia/BigCodec),
+[DAC](https://github.com/descriptinc/descript-audio-codec),
+[Inworld TTS](https://github.com/inworld-ai/tts), and
+[BigVGAN](https://github.com/NVIDIA/BigVGAN). See the repository for the full
+list and their licenses.
+
+## License
+
+The model weights and code are released under the
+[Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). Third-party
+components retain their respective licenses.

config.json

@@ -0,0 +1,91 @@
+{
+  "semantic_model": "facebook/w2v-bert-2.0",
+  "semantic_layer_index": 16,
+  "sr": 16000,
+  "output_sr": 44100,
+  "codec_encoder": {
+    "ngf": 48,
+    "hidden_dim": 1024,
+    "up_ratios": [
+      2,
+      2,
+      4,
+      4,
+      5
+    ],
+    "dilations": [
+      1,
+      3,
+      9
+    ],
+    "activation": "snake_beta"
+  },
+  "codec_decoder": {
+    "hidden_dim": 1024,
+    "activation": "snake_beta",
+    "conv_padding_mode": "causal_full",
+    "resnet_kernel_size": 1,
+    "embed_kernel_size": 3,
+    "depth": 12,
+    "heads": 16,
+    "pos_emb_dim": 64,
+    "vq_dim": 2048,
+    "attn_causal": true,
+    "mlp_expansion": 4,
+    "upsampler_factors": [
+      3
+    ],
+    "upsampler_kernel_sizes": [
+      9
+    ],
+    "output_n_fft": 1176,
+    "output_hop_length": 294,
+    "quantizer": {
+      "mode": "rvq",
+      "num_quantizers": 16,
+      "codebook_size": 4096,
+      "fsq_levels": [
+        8,
+        8,
+        8,
+        8
+      ],
+      "cb1_type": "fsq",
+      "residual_type": "vq",
+      "vq_codebook_dim": 8
+    }
+  },
+  "firing": {
+    "enabled": true,
+    "mode": "adaptive",
+    "target_ratio": 0.25,
+    "max_gap": 64,
+    "decision": "clock",
+    "hnet_ms_offsets": [
+      [
+        1,
+        0
+      ],
+      [
+        1,
+        1
+      ],
+      [
+        2,
+        1
+      ]
+    ],
+    "head_in_budget": true,
+    "aggregate": "mean",
+    "signals": [
+      "hnet_ms",
+      "flux"
+    ]
+  },
+  "inference": {
+    "attention_window_seconds": 6.0,
+    "decode_window_frames": 300,
+    "decode_stride_frames": 100,
+    "decode_crossfade_ms": 10.0
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edaa53e8263134625f69c721ffe168ba15659eefe76fb81f8ce6c755620ccb34
+size 984379032