CoSTA-MIX

CoSTA-MIX is the Chinese and English dialogue-adapted checkpoint from CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models. It is based on Step-Audio-2-mini.

Paper · Code and full instructions

Model

CoSTA compresses the speech-token sequences generated by the language model using a learned BPE vocabulary. Generated tokens are expanded losslessly to the original speech-token space before waveform synthesis. The original speech tokenizer and waveform decoder are retained.

This is the final checkpoint after vocabulary extension, pronunciation adaptation, and dialogue adaptation. It contains BF16 weights with 238,352 text-and-speech token IDs, including 80,000 added speech tokens. The matching speech BPE vocabulary has 86,656 entries: 6,656 original units and 80,000 merged units.

Files

• model-*.safetensors and model.safetensors.index.json: complete model weights.
• config.json, configuration_step_audio_2.py, and modeling_step_audio_2.py: model configuration and custom Transformers implementation.
• tokenizer.json and companion tokenizer files: the LLM tokenizer.
• speech_bpe_tokenizer.json: the matching speech BPE tokenizer. This is distinct from the LLM tokenizer.json.
• speech_bpe_vocab_int.jsonl: BPE-token expansions to original speech-token IDs.
• audio_bpeid_to_llmid.json: mapping from speech BPE IDs to global LLM token IDs.

Download and inference

Use the inference code and dependency versions in the CoSTA repository. The supported workflow uses Transformers 4.49.0 and the custom model implementation; a generic Transformers pipeline is not the speech-dialogue inference entry point.

After cloning the repository and installing its dependencies, run from its root:

export COSTA_LANGUAGE=mix
source scripts/env.sh

hf download D-YZ/CoSTA-MIX --local-dir "checkpoints/CoSTA-MIX"
hf download stepfun-ai/Step-Audio-2-mini \
  --include "token2wav/*" --local-dir "$BASE_MODEL_PATH"

export MODEL_PATH="$PWD/checkpoints/CoSTA-MIX"
export MODEL_NAME=CoSTA-MIX
export BPE_TOKENIZER_PATH="$MODEL_PATH/speech_bpe_tokenizer.json"
export BPE_MAPPING_PATH="$MODEL_PATH/audio_bpeid_to_llmid.json"
export DATASET_NAME=AlpacaEval-zh
export JSONL_PATH="$DATA_ROOT/eval/$DATASET_NAME/test_norm.jsonl"

CUDA_VISIBLE_DEVICES=0 bash inference/infer_costa.sh

Prepare the evaluation JSONL and referenced audio files as described in the data and inference instructions. Prepared training and evaluation data are released separately and are not included in this model repository. The external token2wav decoder is obtained from the base model as shown above. For the bilingual checkpoint, select the appropriate input manifest for the language being evaluated while retaining this checkpoint's own BPE files.

Training and evaluation

The adaptation sequence is mean-initialized vocabulary extension, pronunciation adaptation with an embedding/LM-head stage followed by a full-LLM stage, and speech-to-speech dialogue adaptation. Pronunciation data is based on Emilia with Seed-VC voice conversion; dialogue preparation follows OpenS2S with CosyVoice 2 assistant-speech synthesis. The paper reports evaluation on URO-Bench.

See the paper for the experimental settings, metrics, results, and limitations. In particular, much of the adaptation data is synthetic, and evaluation focuses on general speech dialogue. Model outputs can contain factual errors and depend on the input language and acoustic conditions.

License and attribution

This checkpoint is released under Apache-2.0. See LICENSE and NOTICE. The custom model implementation derives from Step-Audio-2-mini and contains CoSTA changes for training loss, embedding resizing, and inference timing.

Citation

@misc{deng2026costa,
  title={{CoSTA}: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models},
  author={Yingzhuo Deng and Chong Li and Bingxin Liu and Yi Shu and Tianyu Peng and Can Xie and Siyao Song and Shuo Ren and Jiajun Zhang},
  year={2026},
  eprint={2610.21143},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.21143}
}