Update model card with paper metadata, authors, and tags

README.md

@@ -1,18 +1,27 @@
 ---
-pipeline_tag: text-to-speech
 datasets:
 - facebook/multilingual_librispeech
 language:
 - en
+pipeline_tag: text-to-speech
+tags:
+- audio-codec
+- neural-audio-codec
 ---
+
 # FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
 
-[![Demo Page](https://img.shields.io/badge/GitHub.io-Demo_Page-blue?logo=Github&style=flat-square)](https://flexicodec.github.io/)
 [![ArXiv](https://img.shields.io/badge/arXiv-PDF-green?logo=arxiv&style=flat-square)](https://arxiv.org/abs/2510.00981)
+[![Demo Page](https://img.shields.io/badge/GitHub.io-Demo_Page-blue?logo=Github&style=flat-square)](https://flexicodec.github.io/)
+[![OpenReview](https://img.shields.io/badge/OpenReview-ICLR2026-red?logo=OpenReview&style=flat-square)](https://openreview.net/forum?id=kYkfCs4ZAH)
 
+This is the official Hugging Face repository for **FlexiCodec**, a dynamic neural audio codec designed for speech language models.
+
+**Authors:** Jiaqi Li, Yao Qian, Yuxuan Hu, Leying Zhang, Xiaofei Wang, Heng Lu, Manthan Thakker, Jinyu Li, Sheng Zhao, Zhizheng Wu.
 
 ## Abstract
 Neural audio codecs are foundational to speech language models. It is expected to have a low frame rate and decoupled semantic and acoustic information. A lower frame rate codec can reduce the computational cost of speech language models by shortening the sequence length. Recent studies have developed 12.5Hz low-frame-rate audio codecs, but even lower frame rate codecs remain underexplored. We find that a major challenge for very low frame rate tokens is missing semantic information. This paper introduces FlexiCodec to address this limitation. FlexiCodec improves semantic preservation with a dynamic frame rate approach and introduces a novel architecture featuring an ASR feature-assisted dual stream encoding and Transformer bottlenecks. With dynamic frame rates, it uses less frames at information-sparse regions through adaptively merging semantically similar frames. A dynamic frame rate also allows FlexiCodec to support inference-time controllable frame rates between 3Hz and 12.5Hz. Experiments on 6.25Hz, 8.3Hz and 12.5Hz average frame rates confirm that FlexiCodec excels over baseline systems in semantic information preservation and delivers a high audio reconstruction quality. We also validate the effectiveness of FlexiCodec in language model-based TTS.
+
 ![](.github/flexicodec.png)
 
 ## Installation
@@ -21,11 +30,8 @@ git clone https://github.com/amphionspace/FlexiCodec.git
 cd FlexiCodec
 pip install -r requirements.txt
 ```
-<!-- # pip install -e . -->
-
-## FlexiCodec
-Code is available under [`flexicodec/modeling_flexicodec.py`](flexicodec/modeling_flexicodec.py). 
 
+## FlexiCodec Usage
 To run inference (automatically downloads checkpoint from huggingface):
 ```python
 import torch
@@ -55,17 +61,9 @@ print(f"This sample avg frame rate: {encoded_output['token_lengths'].shape[-1] /
 ```
 
 Notes:
-- You may tune the `num_quantizers=xxx` (maximum 24), `merging_threshold=xxx` (maximum 1.0) parameters. If you set `merging_threshold=1.0`, it will be a standard 12.5Hz neural audio codec. All of its `token_lengths` items will be 1. 
-
-- For mainland China users, you might need to execute `export HF_ENDPOINT=https://hf-mirror.com` in terminal, before running the code. If you don't want to automatically download from huggingface, you can manually specify your downloaded checkpoint paths [![Huggingface](https://img.shields.io/badge/huggingface-yellow?logo=huggingface&style=flat-square)](https://huggingface.co/jiaqili3/flexicodec/tree/main) in `prepare_model`. 
-
-
-- Batched input is supported. You can directly pass audios shaped [B,T] to the script above, but the audio length information will be unavailable.
-To resolve this, you can additionally pass an `audio_lens` parameter to `encode_flexicodec`, and you can crop the output for each audio in `encoded_output[speech_token_len]`. 
-
-- If you want to use the above code elsewhere, you might want to add `sys.path.append('/path/to/FlexiCodec')` to find the code.
-
-- To extract continuous features from the semantic tokens, use:
+- You may tune the `num_quantizers` (max 24) and `merging_threshold` (max 1.0) parameters. Setting `merging_threshold=1.0` results in a standard 12.5Hz neural audio codec.
+- Batched input is supported. You can pass audios shaped `[B, T]`, but audio length information will be unavailable unless you pass an `audio_lens` parameter to `encode_flexicodec`.
+- To extract continuous features from the semantic tokens:
   ```python
   feat = model_dict['model'].get_semantic_feature(encoded_output['semantic_codes'])
   ```
@@ -77,68 +75,8 @@ sudo apt install espeak-ng
 pip install cached_path phonemizer openai-whisper
 ```
 
-### FlexiCodec-based Voicebox NAR Inference
-The VoiceBox NAR system can decode FlexiCodec's RVQ-1 tokens into speech. It is used as the second stage in FlexiCodec-TTS, but can also be used standalone.
-To run NAR TTS inference using FlexiCodec-Voicebox:
-
-```python
-import torch
-import torchaudio
-from flexicodec.nar_tts.inference_voicebox import (
-    prepare_voicebox_model, 
-    infer_voicebox_tts
-)
-import cached_path
-# Prepare model (loads model and vocoder)
-checkpoint_path = cached_path('hf://jiaqili3/flexicodec/nartts.safetensors')
-model_dict = prepare_voicebox_model(checkpoint_path)
-
-# Option 1: Inference with audio file paths
-gt_audio_path = "audio_examples/61-70968-0000_gt.wav"  # Target content. Example GT audio
-ref_audio_path = "audio_examples/61-70968-0000_ref.wav"     # Reference voice/style. 
-
-output_audio, output_sr = infer_voicebox_tts(
-    model_dict=model_dict,
-    gt_audio_path=gt_audio_path,
-    ref_audio_path=ref_audio_path,
-    n_timesteps=15,          # Number of diffusion steps (default: 15)
-    cfg=2.0,                 # Classifier-free guidance scale (default: 2.0)
-    rescale_cfg=0.75,        # CFG rescaling factor (default: 0.75)
-    merging_threshold=1.0    # Merging threshold for frame rate control (default: 1.0, max: 1.0)
-)
-
-# Save output
-torchaudio.save("output.wav", output_audio.unsqueeze(0) if output_audio.dim() == 1 else output_audio, output_sr)
-
-# Option 2: Inference with audio tensors
-gt_audio, gt_sr = torchaudio.load("path/to/ground_truth.wav")
-ref_audio, ref_sr = torchaudio.load("path/to/reference.wav")
-
-output_audio, output_sr = infer_voicebox_tts(
-    model_dict=model_dict,
-    gt_audio=gt_audio,
-    ref_audio=ref_audio,
-    gt_sample_rate=gt_sr,
-    ref_sample_rate=ref_sr,
-    n_timesteps=15,
-    cfg=2.0,
-    rescale_cfg=0.75,
-    merging_threshold=1.0
-)
-```
-
-**Notes:**
-- The model automatically detects and uses CUDA, MPS (Apple Silicon), or CPU devices
-- Ground truth audio (`gt_audio`) determines the semantic content of the output
-- Reference audio (`ref_audio`) determines the voice/style characteristics
-- Output sample rate is typically 16000 Hz or 24000 Hz depending on the model configuration
-- You can reuse `model_dict` for multiple inference calls to avoid reloading the model
-- `merging_threshold` controls FlexiCodec's dynamic frame rate: lower values (e.g., 0.87, 0.91) enable merging for lower average frame rates, while 1.0 disables merging (standard 12.5Hz)
-
-### FlexiCodec-based AR+NAR TTS Inference
-The AR+NAR TTS system generates speech tokens from text using an autoregressive transformer model, and then uses the Voicebox NAR system to decode the tokens into audio.
-
-To perform complete text-to-speech with both AR generation and NAR decoding:
+### AR+NAR TTS Inference
+The AR+NAR TTS system generates speech tokens from text and then uses the Voicebox NAR system to decode them into audio.
 
 ```python
 import torch
@@ -146,7 +84,7 @@ import torchaudio
 from flexicodec.ar_tts.inference_tts import tts_synthesize
 from flexicodec.ar_tts.modeling_artts import prepare_artts_model
 from flexicodec.nar_tts.inference_voicebox import prepare_voicebox_model
-import cached_path
+from cached_path import cached_path
 
 # Prepare both AR and NAR models
 ar_checkpoint = cached_path('hf://jiaqili3/flexicodec/artts.safetensors')
@@ -156,46 +94,26 @@ ar_model_dict = prepare_artts_model(ar_checkpoint)
 nar_model_dict = prepare_voicebox_model(nar_checkpoint)
 
 # Full TTS synthesis
-output_audio, output_sr = tts_synthesize(
+output_audio, output_sr, duration_classes = tts_synthesize(
     ar_model_dict=ar_model_dict,
     nar_model_dict=nar_model_dict,
-    text="Hello, this is a complete text-to-speech example.",
+    text="Hello, this is a complete text to speech example.",
     language="en",
-    ref_audio_path="audio_examples/61-70968-0000_ref.wav",  # Reference voice
-    ref_text="bear us escort so far as the Sheriff's house",  # Optional reference text
-    merging_threshold=0.91,  # Frame rate control (used for both AR and NAR)
-    beam_size=1,
-    top_k=25,
-    temperature=1.0,
-    predict_duration=True,
-    duration_top_k=1,
-    n_timesteps=15,  # NAR diffusion steps
-    cfg=2.0,  # NAR classifier-free guidance
-    rescale_cfg=0.75,  # NAR CFG rescaling
-    use_nar=True,  # Set to False for AR-only decoding
+    ref_audio_path="./audio_examples/1089-134686-0030.flac",  # Reference voice
+    ref_text="be ware of making that mistake",  # Optional reference text
+    merging_threshold=0.91,  # 0.91 for ~8.3Hz, 0.86 for ~6.25Hz
+    use_nar=True,
 )
 
 # Save output
 torchaudio.save("output.wav", output_audio.unsqueeze(0) if output_audio.dim() == 1 else output_audio, output_sr)
 ```
 
-**Notes:**
-- `tts_synthesize` performs the full pipeline: AR generation + NAR decoding to audio
-- Reference audio (`ref_audio_path`) provides the voice/style characteristics
-- Reference text (`ref_text`) is optional and can help with prosody alignment
-- Set `use_nar=False` in `tts_synthesize` to use AR-only decoding (faster but lower quality)
-
-### Training reference implementations
-Inside `flexicodec/ar_tts/modeling_artts.py` and `flexicodec/nar_tts/modeling_voicebox.py` there are `training_forward` methods that receive audios and prepared sensevoice-small input "FBank" features. (`dl_output` dictionary containing `x` (the [`feature_extractor`](flexicodec/infer.py#L50) output), `x_lens` (length of each x before padding), `audio` (the 16khz audio tensor)). 
-Training can be replicated by passing the same data to the `training_forward` methods. 
-
-
-
 ## Acknowledgements & Citation
 - Our codebase setup is based on [DualCodec](https://github.com/jiaqili3/DualCodec)
 - We thank the [Mimi Codec](https://github.com/kyutai-labs/moshi) for transformer implementations
 
-If you find our works useful, please consider citing as:
+If you find our works useful, please consider citing:
 ```biblatex
 @article{li2025flexicodec,
   title={FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates},
@@ -203,11 +121,4 @@ If you find our works useful, please consider citing as:
   journal={arXiv preprint arXiv:2510.00981},
   year={2025}
 }
-
-@article{li2025dualcodec,
-  title={Dualcodec: A low-frame-rate, semantically-enhanced neural audio codec for speech generation},
-  author={Li, Jiaqi and Lin, Xiaolong and Li, Zhekai and Huang, Shixi and Wang, Yuancheng and Wang, Chaoren and Zhan, Zhenpeng and Wu, Zhizheng},
-  journal={Interspeech 2025},
-  year={2025}
-}
 ```