README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: text-to-speech
tags:
  - speech
  - spoken-language-model
  - audio
  - tts
  - asr
  - flexislm
---

# FlexiSLM-7B-Stage2

Stage 2 checkpoint of **FlexiSLM-7B**, a spoken language model with dynamic and controllable frame rates on both speech input and output.

- Paper: [arXiv:2606.31247](https://arxiv.org/abs/2606.31247)
- Demo: [flexislm.github.io](https://flexislm.github.io)
- Code: [AmphionTeam/FlexiSLM](https://github.com/AmphionTeam/FlexiSLM)
- Sibling checkpoint: [FlexiSLM-0_5B-Stage2](https://huggingface.co/FlexiSLM/FlexiSLM-0_5B-Stage2)

## Quick start (auto-download)

Install from the code repo, then run with `auto_download=True`. On first run this downloads **this** Stage 2 checkpoint plus the shared Qwen2.5-Omni audio encoder, SenseVoice, FlexiCodec, flow-matching decoder, and vocoder into `models/`.

```bash
git clone --recurse-submodules https://github.com/AmphionTeam/FlexiSLM.git
cd FlexiSLM
pip install -r requirements.txt
```

```python
from pathlib import Path

import soundfile as sf
import torch

from src.inference_flexislm import FlexiSLMInferenceConfig, FlexiSLMInference

config = FlexiSLMInferenceConfig(
    auto_download=True,
    checkpoint="stage2_7B",  # this repo
    use_flow_matching_decoder=True,
    flow_matching_prompt_audio_path=str(Path("examples/input.wav").resolve()),
    enable_flexible_framerate=True,
    input_framerate=8.0,
    default_framerate=8.0,
    decode_audio=True,
    torch_dtype="bfloat16",
    attn_implementation="flash_attention_2",
)
engine = FlexiSLMInference(config, device="cuda:0")


def save_audio(result, output_path):
    waveform = result.get("audio")
    if waveform is None:
        raise RuntimeError("The model did not return decoded audio")
    if torch.is_tensor(waveform):
        waveform = waveform.detach().float().cpu().numpy()
    sf.write(Path(output_path), waveform.squeeze(), 16_000)


# Text-to-speech
result = engine.generate_tts(
    sentence="FlexiSLM supports controllable speech generation.",
    framerate=8.0,
)
save_audio(result, "tts.wav")

# ASR
result = engine.generate_from_audio(
    audio_path="examples/input.wav",
    text_query="Please transcribe the audio.",
    framerate=8.0,
    output_text_only=True,
)
print(result["text"])

# Audio QA
result = engine.generate_from_audio(
    audio_path="examples/question.wav",
    text_query="",
    framerate=8.0,
    output_text_only=True,
)
print(result["text"])

# Speech-to-speech
result = engine.generate_from_audio(
    audio_path="examples/input.wav",
    text_query="",
    framerate=8.0,
    output_text_only=False,
)
save_audio(result, "s2s.wav")
```

## Manual download

```bash
MODEL_ROOT="$PWD/models"

hf download FlexiSLM/FlexiSLM-7B-Stage2 --local-dir "$MODEL_ROOT/FlexiSLM-7B-Stage2"

# Shared auxiliary files (required for inference)
hf download FlexiSLM/Qwen2_5-Omni-Audio_Encoder --local-dir "$MODEL_ROOT/Qwen2_5-Omni-Audio_Encoder"
hf download FunAudioLLM/SenseVoiceSmall --local-dir "$MODEL_ROOT/SenseVoiceSmall"
hf download jiaqili3/flexicodec \
  12hz_v1_half_config.yaml \
  nartts_flexicodec_only.safetensors \
  nartts.safetensors \
  --local-dir "$MODEL_ROOT/FlexiCodec"
hf download amphion/dualcodec-tts vocos_emilia.safetensors \
  --local-dir "$MODEL_ROOT/FlexiCodec"
```

Then point `FlexiSLMInferenceConfig` at the local paths (`checkpoint="stage2_7B"`, `model_path=models/FlexiSLM-7B-Stage2`, plus encoder/codec paths). See the [code README Inference section](https://github.com/AmphionTeam/FlexiSLM#inference) for the full config block.

## Batch inference

After downloading weights, use the committed examples:

```bash
python -m src.infer examples/infer_7b.yaml
```

Set `engine.config.checkpoint: stage2_7B` and `engine.config.model_path: models/FlexiSLM-7B-Stage2`, or use `auto_download: true`.

## Controllable frame rate

A single Stage 2 model can be steered between about **12.5 Hz** and **4.0 Hz** without retraining via `input_framerate` / `default_framerate` / per-call `framerate`.

## Citation

```bibtex
@misc{li2026flexislmdynamiccontrollableframe,
      title={FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model},
      author={Jiaqi Li and Chaoren Wang and Xiaohai Tian and Mingjie Chen and Xinyu Liang and Xu Li and Yufan Lin and Junwen Qiu and Jun Zhang and Lu Lu and Haizhou Li and Zhizheng Wu},
      year={2026},
      eprint={2606.31247},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.31247},
}
```