Step-Audio (models: Step-Audio-AQAA)

models/Step-Audio-AQAA/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/Step-Audio-AQAA/README.md

@@ -0,0 +1,66 @@
+---
+license: apache-2.0
+---
+# Step-Audio-AQAA: A Fully End-to-End Expressive Large Audio Language Model
+**📚 Paper:** [Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model](https://arxiv.org/abs/2506.08967)
+
+**🚀 Live Demo:** [![Try the Demo](https://img.shields.io/badge/StepFun-Audio-AQAA)](https://www.stepfun.com/docs/zh/step-audio-aqaa?studio_code=step-audio-aqaa&studio_id=121368403356246016&studio_type=1)
+
+## Model Overview
+Step-Audio-AQAA is a fully end-to-end Large Audio-Language Model (LALM) designed for Audio Query-Audio Answer (AQAA) tasks. It directly processes audio inputs and generates natural, accurate speech responses without relying on traditional ASR and TTS modules, eliminating cascading errors and simplifying the system architecture.
+
+## Key Capabilities
+- **Fully End-to-End Audio Interaction**: Generates speech outputs directly from raw audio inputs without ASR/TTS intermediates.
+- **Fine-Grained Voice Control**: Supports sentence-level adjustments of emotional tone, speech rate, and other vocal features.
+- **Multilingual & Dialect Support**: Covers Chinese (including Sichuanese, Cantonese), English, Japanese, etc.
+- **Complex Task Handling**: Excels in speech emotion control, role-playing, logical reasoning, and other complex audio interactions.
+
+## Model Architecture
+Step-Audio-AQAA consists of three core modules:
+
+### Dual-Codebook Audio Tokenizer
+- **Linguistic Tokenizer**: Based on Paraformer encoder, extracts phonemic and linguistic attributes with a 1,024-codebook at 16.7Hz.
+- **Semantic Tokenizer**: References CosyVoice 1.0, captures acoustic features with a 4,096-codebook at 25Hz.
+- **Temporal Alignment**: Uses a 2:3 interleaving ratio to ensure temporal consistency between token types.
+
+### Backbone LLM
+- **Parameter Scale**: 130-billion-parameter multi-modal LLM (Step-Omni).
+- **Architecture**: Decoder-only with Transformer blocks, RMSNorm layers, and grouped query attention.
+- **Vocabulary Expansion**: Incorporates 5,120 audio tokens into the text vocabulary for text-audio interleaved output.
+
+### Neural Vocoder
+- **Architecture**: Flow-matching model based on CosyVoice, using U-Net and ResNet-1D layers.
+- **Conditional Generation**: Generates high-fidelity speech waveforms conditioned solely on audio tokens.
+
+## Training Approach
+### Multi-Stage Training Pipeline
+1. **Pretraining**: Multi-modal pretraining on text, audio, and image data.
+2. **Supervised Fine-Tuning (SFT)**:
+   - Stage 1: Full-parameter update on AQTA and AQTAA datasets.
+   - Stage 2: Optimizes specific capabilities with high-quality AQTAA data.
+3. **Direct Preference Optimization (DPO)**: Uses audio token masking to avoid degradation of speech generation.
+4. **Model Merging**: Weighted combination of SFT and DPO models to enhance overall performance.
+
+### Training Data
+- **Multi-Modal Pretraining Data**: 800 billion text tokens and audio-text interleaved data.
+- **AQTA Dataset**: Audio query-text answer pairs.
+- **AQTAA Dataset**: Audio query-text answer-audio answer triplets generated from AQTA.
+
+
+## Citation
+```bibtex
+@misc{huang2025stepaudioaqaa,
+      title={Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model}, 
+      author={Ailin Huang and Boyong Wu and Bruce Wang and Chao Yan and Chen Hu and Chengli Feng and Fei Tian and Feiyu Shen and Jingbei Li and Mingrui Chen and et al.},
+      year={2025},
+      eprint={2506.08967},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD}
+}
+```
+
+## Team & Contributions
+Step-Audio-AQAA is developed by the StepFun team, with contributions from multiple researchers and engineers. For technical support or collaboration, contact the corresponding authors: Daxin Jiang (djiang@stepfun.com), Shuchang Zhou (scotzhou@stepfun.com), Chen Hu (hatcher@stepfun.com).
+
+## License
+This model is released under the Apache 2.0 license. For more details, please refer to the license file.

models/Step-Audio-AQAA/config.json

@@ -0,0 +1,33 @@
+{
+  "architectures": [
+    "MMGPTStep1ForCausalLMV4"
+  ],
+  "model_type": "mmgpt_step1_v2",
+  "hidden_size": 12288,
+  "intermediate_size": 31232,
+  "num_attention_heads": 96,
+  "num_attention_groups": 8,
+  "num_hidden_layers": 88,
+  "max_seq_len": 999999,
+  "vocab_size": 74752,
+  "rms_norm_eps": 1e-05,
+  "torch_dtype": "bfloat16",
+  "im_end_token": "<im_end>",
+  "im_patch_token": "<im_patch>",
+  "im_start_token": "<im_start>",
+  "image_token_len": 169,
+  "use_im_start_end": true,
+  "vision_select_layer": -1,
+  "understand_projector_stride": 2,
+  "vit_scale": 1.0,
+  "projector_bias": false,
+  "vision_tower_config": {
+    "hidden_size": 1792,
+    "output_hidden_size": 4096,
+    "image_size": 728,
+    "intermediate_size": 15360,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 63,
+    "patch_size": 14
+  }
+}

models/Step-Audio-AQAA/model-00001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:401119d809d338614caa2adabe6e55f7d5aad1a7395bf624bdcdc14605de6181
+size 9434733376

models/Step-Audio-AQAA/model-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:799f7fad3eb75123046be5a822337ed28a405c8136c5243e380d59241a6cd85b
+size 9940651088

models/Step-Audio-AQAA/model-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb21dfe9f67bde6d78db684419372fd245dcea5e167ad7b2b94ba613db78303b
+size 9940676016

models/Step-Audio-AQAA/model-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3f6caa5db5606bed09b08a53b51fac44f94701880d5d0cf3cdda42c0370c46b
+size 9991007912

models/Step-Audio-AQAA/model-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:660af7ae96e6369b92a987327c343adc88706047d991338d890ab3fdde174850
+size 9638661344

models/Step-Audio-AQAA/model-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2434a2806cc231904ee07bc7575c49c28e4f74f318fb041e37fb8bf2d540b1af
+size 9940676048

models/Step-Audio-AQAA/model-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:205072bb59dafe89f248968a26298b7f91a06e0150a66690ce99cbcd8bae686b
+size 9991007944

models/Step-Audio-AQAA/model-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c476603a3cd719dfc2832cfe9c312c459260f90dd6af8125b75e45a0604c8ae7
+size 9638661352

models/Step-Audio-AQAA/model-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d29cbfce6f0738a4431c6a885ddc4a01ca0831f56d7748d8277971ba268efda0
+size 9940676048

models/Step-Audio-AQAA/model-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f14f17011d5c36e84b0393611d62c9a26e6fb3a403bae1e26028b6377c183f63
+size 9991007944

models/Step-Audio-AQAA/model-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ca4a5d29e943b4b4a7e6c11de86e44d40940c739f0fb8801030e86ac54f55d7
+size 9638661352

models/Step-Audio-AQAA/model-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb1c87ec0c8bc78feba6bb39432bb009b040d2e9eb15d068fd96e824cb88e957
+size 9940676048

models/Step-Audio-AQAA/preprocessor_config.json

@@ -0,0 +1,19 @@
+{
+  "do_center_crop": false,
+  "do_normalize": true,
+  "do_resize": true,
+  "feature_extractor_type": "CLIPFeatureExtractor",
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "size": 728,
+  "mode": "bilinear"
+}

models/Step-Audio-AQAA/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/stepfun-ai/Step-Audio-AQAA

models/Step-Audio-AQAA/step2_tokenizer_241028.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22f3dc190340b02713be9cb43d5fcf7e3c01c302d407c4470def4bac5eac9fdc
+size 1264015