Add files, models, and assets

.gitattributes

@@ -33,3 +33,16 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+qwen3-tts_onnx/codec_decoder_model.onnx filter=lfs diff=lfs merge=lfs -text
+qwen3-tts_onnx/speaker_encoder_model.onnx filter=lfs diff=lfs merge=lfs -text
+qwen3-tts_onnx/talker_codec_embed_model.onnx filter=lfs diff=lfs merge=lfs -text
+qwen3-tts_onnx/talker_local_model.onnx filter=lfs diff=lfs merge=lfs -text
+qwen3-tts_onnx/talker_model.onnx filter=lfs diff=lfs merge=lfs -text
+qwen3-tts_onnx/text_embed_proj_model.onnx filter=lfs diff=lfs merge=lfs -text
+audio_ref/female_shadowheart.flac filter=lfs diff=lfs merge=lfs -text
+audio_ref/male_old_movie.flac filter=lfs diff=lfs merge=lfs -text
+audio_ref/male_petergriffin.wav filter=lfs diff=lfs merge=lfs -text
+audio_ref/male_stewie.mp3 filter=lfs diff=lfs merge=lfs -text
+audio_ref/rick-sanchez.mp3 filter=lfs diff=lfs merge=lfs -text
+audio_ref/david-attenborough.mp3 filter=lfs diff=lfs merge=lfs -text
+audio_synth/output_1775946408.2838778.wav filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,2 @@
+__pycache__
+.*swp

README.md

@@ -1,3 +1,249 @@
 ---
+language:
+  - ru
+  - zh 
+  - en 
+  - de
+  - es
+  - fr
+  - ja
+  - it
+  - pt
+  - ko
+tags:
+  - text-to-speech
+  - TTS
+  - ONNX
+  - qwen3-tts
+  - voice-clone
+  - streaming
+  - qwen3
+  - rvq
+  - multilingual
+pipeline_tag: text-to-speech
 license: apache-2.0
+base_model: Qwen/Qwen3-TTS-12Hz-0.6B-Base
 ---
+
+# Qwen3-TTS-Realtime ONNX Inference
+
+Pure ONNX Runtime inference pipeline for [Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base), enabling **streaming text-to-speech** without PyTorch dependency at runtime.
+
+## Overview
+
+This repository provides:
+
+- **`qwen3_tts_inferencer_onnx.py`** — Core streaming TTS engine that orchestrates six ONNX models (talker LLM, local talker transformer, codec decoder, speaker encoder, talker codec embedding, text embedding projection) using only NumPy and ONNX Runtime.
+- **`test_qwen3-tts-streaming_onnx.py`** — End-to-end test script that simulates LLM streaming text and produces a WAV file.
+
+## Architecture
+
+```
+Reference Audio ──► Speaker Encoder ──► Speaker Embedding Vector (voice clone context)
+                                           │
+                                           ▼
+Text Deltas ──► Talker LLM (Qwen3-0.6B) ──► [Hidden States, VQ Token]
+                                                          │
+                                                          ▼
+                                                Local Transformer ──► 15-codebook RVQ Tokens
+                                                                            │
+                                                                            ▼
+                                                          VQ Token ──► Codec Decoder ──► 24 kHz Waveform
+```
+
+| Component | ONNX Model | Description |
+|-----------|------------|-------------|
+| Talker LLM | `talker_model.onnx` | Qwen3-based talker LM mapping interleaved text+audio tokens embeddings to hidden states and VQ. Maintains a growing KV-cache across the entire generation. |
+| Local Talker | `talker_local_model.onnx` | Depth-wise decoder generating 15 RVQ codebook entries per frame from talker hidden states and VQ. Creates and discards a fresh KV-cache per frame. |
+| Codec Decoder | `codec_decoder_model.onnx` | Decodes VQ+RVQ audio codes back to 24 kHz waveform. Maintains KV-caches and convolutional caches for streaming decode. |
+| Speaker Encoder | `speaker_encoder_model.onnx` | ECAPA-TDNN-based speaker encoder. Produces a 1024-dim speaker embedding vector for voice identity cloning. |
+| Talker Codec Embed | `talker_codec_embed_model.onnx` | VQ embedding for the talker model. Consists of 2048 token vocabs. |
+| Text Embed Projection | `text_embed_proj_model.onnx` | Text embedding and projection for the talker model. Text embedding consists of 151,936 token vocabs. |
+
+## Requirements
+
+```
+librosa
+numpy
+onnxruntime
+python-box
+soundfile
+transformers==4.57.3
+```
+
+Example installation with conda env:
+
+```bash
+conda create --name qwen3-tts-streaming-onnx-1 python=3.12
+conda activate qwen3-tts-streaming-onnx-1
+pip install -r requirements.txt
+```
+
+## Directory Structure
+
+```
+.
+├── test_qwen3-tts-streaming_onnx.py        # End-to-end test script
+├── README.md
+├── requirements.txt
+├── qwen3-tts_onnx/  # FP32
+│   ├── talker_model.onnx
+│   ├── talker_local_model.onnx
+│   ├── codec_decoder_model.onnx
+│   ├── speaker_encoder_model.onnx
+│   ├── talker_codec_embed_model.onnx
+│   └── text_embed_proj_model.onnx
+├── configs/
+│   ├── config.json                         # Talker, Local Talker, Speaker Encoder config
+│   ├── speech_tokenizer_config.json        # Codec config
+│   ├── preprocessor_config.json            # Text Processor configs
+│   ├── tokenizer_config.json
+│   ├── vocab.json
+│   └── merges.txt
+├── src/
+│   ├── core/
+│   │   ├── configuration_qwen3_tts.py
+│   │   └── processing_qwen3_tts.py
+│   ├── inference/
+│   │   └── qwen3_tts_inferencer_onnx.py    # Core ONNX inference engine 
+│   └── utils/
+│       └── audio_utils.py
+├── logs/
+│   └── <log_synth>.txt
+├── audio_ref/
+│   └── <reference_speaker>.[wav|mp3|flac]
+└── audio_synth/
+    └── <synthesized_example>.wav
+```
+
+## Usage
+
+### Basic streaming TTS usage
+
+```bash
+python -u test_qwen3-tts-streaming_onnx.py >& logs/log_test-streaming-onnx-1.txt
+# audio automatically saved in audio_synth/ with default parameters, text, language.
+```
+
+### Usage with parameters
+
+```
+python test_qwen3-tts-streaming_onnx.py \
+    --talker_model_path qwen3-tts_onnx/talker_model.onnx \
+    --talker_local_model_path qwen3-tts_onnx/talker_local_model.onnx \
+    --codec_decoder_model_path qwen3-tts_onnx/codec_decoder_model.onnx \
+    --speaker_encoder_model_path qwen3-tts_onnx/speaker_encoder_model.onnx \
+    --talker_codec_embed_model_path qwen3-tts_onnx/talker_codec_embed_model.onnx \
+    --text_embed_proj_model_path qwen3-tts_onnx/text_embed_proj_model.onnx \
+    --model_config_path configs/config.json \
+    --codec_config_path configs/tokenizer_config.json \
+    --backbone_config_path configs/config_backbone.json \
+    --preprocessor_config_dir configs/ \
+    --temperature 0.85 \
+    --top_p 0.8 \
+    --top_k 50 \
+    --repetition_penalty 1.9 \
+    --repetition_window 50 \
+    --num_threads 4 \
+    --chunk_frames 4 \
+    --prompt_wav audio_ref/speaker.[wav|flac|mp3] \
+    --out_wav output.wav \
+    --text "Text to be synthesized" \
+    --language "english"
+```
+
+### Available Languages
+```
+"chinese", "english", "german", "italian", "portuguese",
+"spanish", "japanese", "korean", "french", "russian"
+```
+
+### Programmatic Usage
+
+```python
+from src.inference import Qwen3TTSInferencerONNX
+
+# Create inferencer
+inferencer = Qwen3TTSInferencerONNX(
+    talker_llm, talker_local, codec_decoder,
+    speaker_encoder, talker_codec_embed, text_embed_proj,
+    preprocessor_config_dir, model_config, codec_config,
+    audio_ref_path, language,
+)
+inferencer.reset_turn(reset_cache=True)
+
+# Stream text and collect audio
+for delta in your_llm_stream():
+    audio_frames = inferencer.push_text(delta)
+    ...
+    for audio_tokens in audio_frames:
+        ...
+        inferencer.push_tokens(audio_tokens)
+        for wav in inferencer.audio_chunks():
+            ...
+            yield wav
+```
+
+### Command-Line Arguments
+
+| Argument | Type | Default | Description |
+|----------|------|---------|-------------|
+| `--talker_model_path` | str | "qwen3-tts_onnx/talker_model.onnx" | Path to talker LLM model |
+| `--talker_local_model_path` | str | "qwen3-tts_onnx/talker_local_model.onnx" | Path to local talker transformer model |
+| `--codec_decoder_model_path` | str | "qwen3-tts_onnx/codec_decoder_model.onnx" | Path to codec decoder model |
+| `--speaker_encoder_model_path` | str | "qwen3-tts_onnx/speaker_encoder_model.onnx" | Path to speaker encoder model |
+| `--talker_codec_embed_model_path` | str | "qwen3-tts_onnx/talker_codec_embed_model.onnx" | Path to talker codec embedding |
+| `--text_embed_proj_model_path` | str | "qwen3-tts_onnx/text_embed_proj_model.onnx" | Path to text embedding and projection |
+| `--preprocessor_config_dir` | str | "configs/" | Directory path to configuration files for the Qwen3 text tokenizer |
+| `--model_config_path` | str | "configs/config.json" | Path to original model configuration file for the Qwen3-TTS-12Hz-0.6B-Base |
+| `--codec_config_path` | str | "configs/speech_tokenizer_config.json" | Path to original model configuration file for the codec of Qwen3-TTS-12Hz-0.6B-Base |
+| `--temperature` | float | `0.85` | Sampling temperature |
+| `--top_p` | float | `0.8` | Nucleus sampling threshold |
+| `--top_k` | int | `50` | Top-k sampling cutoff |
+| `--repetition_penalty` | float | `1.9` | Repetition penalty coefficient |
+| `--repetition_window` | int | `50` | Window for repetition penalty |
+| `--delta_chunk_chars` | int | `1` | Characters per simulated LLM delta |
+| `--delta_delay_s` | float | `0.0` | Delay between simulated deltas (seconds) |
+| `--num_threads` | int | `4` | Number of threads used in sess.intra_op_num_threads of the onnxruntime session options |
+| `--chunk_frames` | int | `4` | Number of chunk frames to be passed on to the codec decoder forward each time [default 4 frame is 0.32 s as token rate is 12.5 Hz] |
+| `--prompt_wav` | str | audio_ref/female_shadowheart.flac | Reference speaker audio for voice cloning |
+| `--out_wav` | str | `out_streaming.wav` | Output WAV file path |
+| `--text` | str | *(Russian text)* | Text to synthesize |
+| `--language` | str | "russian" | Language of the text to synthesize |
+
+#### By: [Patrick Lumbantobing](https://www.linkedin.com/in/patrick-lumban-tobing)
+
+#### Copyright@[VertoX-AI](https://www.linkedin.com/company/vertoxai/)
+
+### Citation
+
+If you use this system in your research, please cite:
+
+```bibtex
+@misc{vertoxai2026streamingspeechtranslation,
+  title={Qwen3-TTS-Streaming-ONNX — VertoX-AI},
+  author={Tobing, P. L., VertoX-AI},
+  year={2026},
+  publisher={HuggingFace},
+}
+```
+
+## License
+
+This project is licensed under the Apache-2.0, the same license as the original Qwen3-TTS.
+
+```
+Created by: Patrick Lumbantobing, Vertox-AI
+Copyright (c) 2026 Vertox-AI. All rights reserved.
+
+This work is licensed under the Apache License, Version 2.0.
+To view a copy of this license, visit [LICENSE](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md).
+```
+
+---
+
+## Acknowledgements
+
+- [Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) for the original Qwen3-TTS model.
+- [Qwen3-TTS Technical Report](https://arxiv.org/abs/2601.15621) (Hu et al., 2026).
+- [ONNX Runtime](https://onnxruntime.ai/) for high-performance cross-platform inference.

audio_ref/david-attenborough.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:582f8e1e3f3792d7b495e159c29a55bb95c4c46e90725c62807b4b12bf341603
+size 322923

audio_ref/female_shadowheart.flac

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f85324f52b1f0cfb43b12123e8718d2841ed4a6f46f60c533fc89a7a4fb26ec9
+size 1559549

audio_ref/male_old_movie.flac

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e342504ccff3d8808ef194e0aca31e2d1d5e96bbf45a188fec9de9a8b756b7c
+size 303882

audio_ref/male_petergriffin.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0a8b708aee90c7dde4eed747ca0b453456b742650699c26fa6ee4e98c8cee0e
+size 486882

audio_ref/male_stewie.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4eb86807929133d186bd951143121a915726f636101b1860a589d06c7a95ab6
+size 395191

audio_ref/rick-sanchez.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bd9c6ffb765fda23297fae21725bc174a3092d9687c3606f11d00ae0df9fc1e
+size 107943

audio_synth/output_1775946408.2838778.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:756e45aea6414dd5a76917f6507152af65d6d1574dcc6c70806feaec24c88e25
+size 234284

configs/config.json

@@ -0,0 +1,167 @@
+{
+  "architectures": [
+    "Qwen3TTSForConditionalGeneration"
+  ],
+  "assistant_token_id": 77091,
+  "im_end_token_id": 151645,
+  "im_start_token_id": 151644,
+  "tts_bos_token_id": 151672,
+  "tts_eos_token_id": 151673,
+  "tts_pad_token_id": 151671,
+  "model_type": "qwen3_tts",
+  "tokenizer_type": "qwen3_tts_tokenizer_12hz",
+  "tts_model_size": "0b6",
+  "tts_model_type": "base",
+  "speaker_encoder_config": {
+    "enc_dim": 1024,
+    "sample_rate": 24000
+  },
+  "talker_config": {
+    "attention_bias": false,
+    "attention_dropout": 0,
+    "code_predictor_config": {
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_bias": false,
+      "attention_dropout": 0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "head_dim": 128,
+      "hidden_act": "silu",
+      "hidden_size": 1024,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "initializer_range": 0.02,
+      "intermediate_size": 3072,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_types": [
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention"
+      ],
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "max_position_embeddings": 65536,
+      "max_window_layers": 28,
+      "min_length": 0,
+      "model_type": "qwen3_tts_talker_code_predictor",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 16,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_code_groups": 16,
+      "num_hidden_layers": 5,
+      "num_key_value_heads": 8,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "rms_norm_eps": 1e-06,
+      "rope_scaling": null,
+      "rope_theta": 1000000,
+      "sep_token_id": null,
+      "sliding_window": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": false,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "dtype": null,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false,
+      "use_cache": true,
+      "use_sliding_window": false,
+      "vocab_size": 2048
+    },
+    "codec_bos_id": 2149,
+    "codec_eos_token_id": 2150,
+    "codec_think_id": 2154,
+    "codec_language_id": {
+        "chinese": 2055,
+        "english": 2050,
+        "german": 2053,
+        "italian": 2070,
+        "portuguese": 2071,
+        "spanish": 2054,
+        "japanese": 2058,
+        "korean": 2064,
+        "french": 2061,
+        "russian": 2069
+    },
+    "codec_nothink_id": 2155,
+    "codec_pad_id": 2148,
+    "codec_think_bos_id": 2156,
+    "codec_think_eos_id": 2157,
+    "spk_id": {
+    },
+    "spk_is_dialect": {
+    },
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "max_position_embeddings": 32768,
+    "model_type": "qwen3_tts_talker",
+    "num_attention_heads": 16,
+    "num_code_groups": 16,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 8,
+    "position_id_per_seconds": 13,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "interleaved": true,
+      "mrope_section": [
+        24,
+        20,
+        20
+      ],
+      "rope_type": "default",
+      "type": "default"
+    },
+    "rope_theta": 1000000,
+    "sliding_window": null,
+    "text_hidden_size": 2048,
+    "text_vocab_size": 151936,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 3072
+  },
+  "transformers_version": "4.57.3"
+}

configs/preprocessor_config.json

@@ -0,0 +1,6 @@
+{
+  "padding_side": "left",
+  "padding_value": 0.0,
+  "processor_class": "Qwen3TTSProcessor",
+  "return_attention_mask": true
+}

configs/speech_tokenizer_config.json

@@ -0,0 +1,94 @@
+{
+  "architectures": [
+    "Qwen3TTSTokenizerV2Model"
+  ],
+  "model_type": "qwen3_tts_tokenizer_12hz",
+  "encoder_valid_num_quantizers": 16,
+  "input_sample_rate": 24000,
+  "output_sample_rate": 24000,
+  "decode_upsample_rate": 1920,
+  "encode_downsample_rate": 1920,
+  "decoder_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "latent_dim": 1024,
+    "codebook_dim": 512,
+    "codebook_size": 2048,
+    "decoder_dim": 1536,
+    "hidden_act": "silu",
+    "hidden_size": 512,
+    "intermediate_size": 1024,
+    "layer_scale_initial_scale": 0.01,
+    "max_position_embeddings": 8000,
+    "head_dim": 64,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 8,
+    "num_key_value_heads": 16,
+    "num_quantizers": 16,
+    "num_semantic_quantizers": 1,
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 10000,
+    "semantic_codebook_size": 4096,
+    "sliding_window": 72,
+    "upsample_rates": [
+      8,
+      5,
+      4,
+      3
+    ],
+    "upsampling_ratios": [
+      2,
+      2
+    ],
+    "vector_quantization_hidden_dimension": 512
+  },
+  "encoder_config": {
+    "_frame_rate": 12.5,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "audio_channels": 1,
+    "codebook_dim": 256,
+    "codebook_size": 2048,
+    "compress": 2,
+    "dilation_growth_rate": 2,
+    "dtype": "float32",
+    "head_dim": 64,
+    "hidden_act": "gelu",
+    "hidden_size": 512,
+    "initializer_range": 0.02,
+    "intermediate_size": 2048,
+    "kernel_size": 7,
+    "last_kernel_size": 3,
+    "layer_scale_initial_scale": 0.01,
+    "max_position_embeddings": 8000,
+    "norm_eps": 1e-05,
+    "normalize": false,
+    "num_attention_heads": 8,
+    "num_filters": 64,
+    "num_hidden_layers": 8,
+    "num_key_value_heads": 8,
+    "num_quantizers": 32,
+    "num_residual_layers": 1,
+    "num_semantic_quantizers": 1,
+    "pad_mode": "constant",
+    "residual_kernel_size": 3,
+    "rope_theta": 10000.0,
+    "sampling_rate": 24000,
+    "sliding_window": 250,
+    "transformers_version": "4.57.0.dev0",
+    "trim_right_ratio": 1.0,
+    "upsample_groups": 512,
+    "upsampling_ratios": [
+      8,
+      6,
+      5,
+      4
+    ],
+    "use_cache": false,
+    "use_causal_conv": true,
+    "use_conv_shortcut": false,
+    "use_streaming": false,
+    "vector_quantization_hidden_dimension": 256
+  },
+  "transformers_version": "4.57.3"
+}

configs/tokenizer_config.json

@@ -0,0 +1,316 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|audio_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|audio_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<tts_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<tts_text_bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151673": {
+      "content": "<tts_text_eod>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151674": {
+      "content": "<tts_text_bos_single>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151675": {
+      "content": "<|audio_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|audio_start|>",
+    "<|audio_end|>",
+    "<tts_pad>",
+    "<tts_text_bos>",
+    "<tts_text_bos_single>",
+    "<|audio_pad|>"
+  ],
+  "extra_special_tokens": {
+    "image_token": "<|image_pad|>",
+    "audio_token": "<|audio_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>",
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>"
+  },
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "image_token": "<|image_pad|>",
+  "audio_token": "<|audio_pad|>",
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>",
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>"
+}

qwen3-tts_onnx/codec_decoder_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee45ad90b2eb510038fc070127bd91f5b6fc6f0eb46166ecbb5a3d810a5e4527
+size 460939919

qwen3-tts_onnx/speaker_encoder_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f19706d97a2196652ae68dac6930400a8c5e287be1d37aede58fb35484e25506
+size 35628286

qwen3-tts_onnx/talker_codec_embed_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b77e5e356e334771257fa18677ea9e731f02c654dc1d0cc46cd6da60166829b
+size 12583165

qwen3-tts_onnx/talker_local_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38d8d4799ab48759400a55f046a8c9398dfaedf1d1eeabd1dcf38e27a544c368
+size 561644701

qwen3-tts_onnx/talker_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce63eadcd5a9dd103c9b38341ac391bf97e4b870aff3972d6fa53d677589d305
+size 1793942592

qwen3-tts_onnx/text_embed_proj_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8aec6c7a66b85b06405974d3f20d7daa872326e2b3b345770452b13e114e0aca
+size 1269839153

requirements.txt

@@ -0,0 +1,6 @@
+librosa
+numpy
+onnxruntime
+python-box
+soundfile
+transformers==4.57.3

src/core/__init__.py

@@ -0,0 +1,17 @@
+# coding=utf-8
+# Copyright 2026 The Alibaba Qwen team.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from .configuration_qwen3_tts import Qwen3TTSConfig
+from .processing_qwen3_tts import Qwen3TTSProcessor

src/core/configuration_qwen3_tts.py

@@ -0,0 +1,506 @@
+# coding=utf-8
+# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from transformers.configuration_utils import (PretrainedConfig,
+                                              layer_type_validation)
+from transformers.modeling_rope_utils import rope_config_validation
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class Qwen3TTSSpeakerEncoderConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`Qwen3TTSSpeakerEncoder`].
+    It is used to instantiate a Qwen3TTS speaker encoder model according to the specified arguments, defining the model
+    architecture. The architecture is based on the ECAPA-TDNN model.
+
+    Args:
+        mel_dim (`int`, *optional*, defaults to 128):
+            The dimension of the input mel-spectrogram.
+        enc_dim (`int`, *optional*, defaults to 192):
+            The dimension of the final speaker embedding.
+        enc_channels (`list[int]`, *optional*, defaults to `[512, 512, 512, 512, 1536]`):
+            A list of output channels for each TDNN/SERes2Net layer in the encoder. The first channel size is for the initial TDNN layer,
+            the intermediate ones for the `SqueezeExcitationRes2NetBlock` layers, and the last one for the multi-layer feature aggregation.
+        enc_kernel_sizes (`list[int]`, *optional*, defaults to `[5, 3, 3, 3, 1]`):
+            A list of kernel sizes for each layer in the encoder, corresponding to `enc_channels`.
+        enc_dilations (`list[int]`, *optional*, defaults to `[1, 2, 3, 4, 1]`):
+            A list of dilations for each layer in the encoder, corresponding to `enc_channels`.
+        enc_attention_channels (`int`, *optional*, defaults to 128):
+            The number of attention channels in the `AttentiveStatisticsPooling` layer.
+        enc_res2net_scale (`int`, *optional*,defaults to 8):
+            The scale of the `Res2NetBlock` in the encoder.
+        enc_se_channels (`int`, *optional*, defaults to 128):
+            The number of channels in the squeeze part of the `SqueezeExcitationBlock`.
+    """
+
+    def __init__(
+        self,
+        mel_dim=128,
+        enc_dim=1024,
+        enc_channels=[512, 512, 512, 512, 1536],
+        enc_kernel_sizes=[5, 3, 3, 3, 1],
+        enc_dilations=[1, 2, 3, 4, 1],
+        enc_attention_channels=128,
+        enc_res2net_scale=8,
+        enc_se_channels=128,
+        sample_rate=24000,
+    ):
+        self.mel_dim = mel_dim
+        self.enc_dim = enc_dim
+        self.enc_channels = enc_channels
+        self.enc_kernel_sizes = enc_kernel_sizes
+        self.enc_dilations = enc_dilations
+        self.enc_attention_channels = enc_attention_channels
+        self.enc_res2net_scale = enc_res2net_scale
+        self.enc_se_channels = enc_se_channels
+        self.sample_rate = sample_rate
+
+
+class Qwen3TTSTalkerCodePredictorConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`Qwen3TTSTalkerCodePredictorModel`]. It is used to instantiate a
+    Qwen3TTSTalkerCodePredictor model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 151936):
+            Vocabulary size of the Qwen3TTSTalkerCodePredictor model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`Qwen3TTSTalkerCodePredictorModel`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 22016):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 32):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        num_key_value_heads (`int`, *optional*, defaults to 32):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details, check out [this
+            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
+        head_dim (`int`, *optional*, defaults to 128):
+            The attention head dimension.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 32768):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether the model's input and output word embeddings should be tied.
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
+            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
+            accordingly.
+            Expected contents:
+                `rope_type` (`str`):
+                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
+                    'llama3'], with 'default' being the original RoPE implementation.
+                `factor` (`float`, *optional*):
+                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
+                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
+                    original maximum pre-trained length.
+                `original_max_position_embeddings` (`int`, *optional*):
+                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
+                    pretraining.
+                `attention_factor` (`float`, *optional*):
+                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
+                    computation. If unspecified, it defaults to value recommended by the implementation, using the
+                    `factor` field to infer the suggested value.
+                `beta_fast` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 32.
+                `beta_slow` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 1.
+                `short_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `long_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `low_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
+                `high_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        use_sliding_window (`bool`, *optional*, defaults to `False`):
+            Whether to use sliding window attention.
+        sliding_window (`int`, *optional*, defaults to 4096):
+            Sliding window attention (SWA) window size. If not specified, will default to `4096`.
+        max_window_layers (`int`, *optional*, defaults to 28):
+            The number of layers using full attention. The first `max_window_layers` layers will use full attention, while any
+            additional layer afterwards will use SWA (Sliding Window Attention).
+        layer_types (`list`, *optional*):
+            Attention pattern for each layer.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+
+    """
+
+    model_type = "qwen3_tts_talker_code_predictor"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model `Qwen3TTSTalkerCodePredictor`
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=2048,
+        hidden_size=1024,
+        intermediate_size=3072,
+        num_hidden_layers=5,
+        num_attention_heads=16,
+        num_key_value_heads=8,
+        head_dim=128,
+        hidden_act="silu",
+        max_position_embeddings=32768,
+        initializer_range=0.02,
+        rms_norm_eps=0.000001,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=10000,
+        rope_scaling=None,
+        attention_bias=False,
+        use_sliding_window=False,
+        sliding_window=4096,
+        max_window_layers=28,
+        layer_types=None,
+        attention_dropout=0,
+        num_code_groups=32,
+        **kwargs,
+    ):
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.use_sliding_window = use_sliding_window
+        self.sliding_window = sliding_window if self.use_sliding_window else None
+        self.max_window_layers = max_window_layers
+
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.head_dim = head_dim
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+        # Validate the correctness of rotary position embeddings parameters
+        # BC: if there is a 'type' field, move it to 'rope_type'.
+        if self.rope_scaling is not None and "type" in self.rope_scaling:
+            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
+        rope_config_validation(self)
+
+        self.layer_types = layer_types
+        if self.layer_types is None:
+            self.layer_types = [
+                (
+                    "sliding_attention"
+                    if self.sliding_window is not None and i >= self.max_window_layers
+                    else "full_attention"
+                )
+                for i in range(self.num_hidden_layers)
+            ]
+        layer_type_validation(self.layer_types)
+        self.num_code_groups = num_code_groups
+
+
+class Qwen3TTSTalkerConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`Qwen3TTSTalkerModel`]. It is used to instantiate a
+    Qwen3TTSTalker model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 151936):
+            Vocabulary size of the Qwen3TTSTalker model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`Qwen3TTSTalkerModel`]
+        hidden_size (`int`, *optional*, defaults to 2048):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 6144):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 24):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        num_key_value_heads (`int`, *optional*, defaults to 4):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details, check out [this
+            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
+
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 32768):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether the model's input and output word embeddings should be tied.
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
+            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
+            accordingly.
+            Expected contents:
+                `rope_type` (`str`):
+                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
+                    'llama3'], with 'default' being the original RoPE implementation.
+                `factor` (`float`, *optional*):
+                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
+                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
+                    original maximum pre-trained length.
+                `original_max_position_embeddings` (`int`, *optional*):
+                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
+                    pretraining.
+                `attention_factor` (`float`, *optional*):
+                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
+                    computation. If unspecified, it defaults to value recommended by the implementation, using the
+                    `factor` field to infer the suggested value.
+                `beta_fast` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 32.
+                `beta_slow` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 1.
+                `short_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `long_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `low_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
+                `high_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        use_sliding_window (`bool`, *optional*, defaults to `False`):
+            Whether to use sliding window attention.
+        sliding_window (`int`, *optional*, defaults to 4096):
+            Sliding window attention (SWA) window size. If not specified, will default to `4096`.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+    """
+
+    model_type = "qwen3_tts_talker"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model `Qwen3TTSTalker`
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+    sub_configs = {"code_predictor_config": Qwen3TTSTalkerCodePredictorConfig}
+
+    def __init__(
+        self,
+        code_predictor_config=None,
+        vocab_size=3072,
+        hidden_size=1024,
+        intermediate_size=2048,
+        num_hidden_layers=20,
+        num_attention_heads=16,
+        num_key_value_heads=2,
+        hidden_act="silu",
+        max_position_embeddings=32768,
+        initializer_range=0.02,
+        rms_norm_eps=0.000001,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=10000,
+        rope_scaling=None,
+        attention_bias=False,
+        use_sliding_window=False,
+        sliding_window=4096,
+        attention_dropout=0,
+        num_code_groups=32,
+        text_hidden_size=2048,
+        codec_eos_token_id=4198,
+        codec_think_id=4202,
+        codec_nothink_id=4203,
+        codec_think_bos_id=4204,
+        codec_think_eos_id=4205,
+        codec_pad_id=4196,
+        codec_bos_id=4197,
+        spk_id=None,
+        spk_is_dialect=None,
+        codec_language_id=None,
+        **kwargs,
+    ):
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.use_sliding_window = use_sliding_window
+        self.sliding_window = sliding_window if use_sliding_window else None
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+        # Validate the correctness of rotary position embeddings parameters
+        # BC: if there is a 'type' field, move it to 'rope_type'.
+        if self.rope_scaling is not None and "type" in self.rope_scaling:
+            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
+
+        if code_predictor_config is None:
+            code_predictor_config = {}
+            self.code_predictor_config = Qwen3TTSTalkerCodePredictorConfig()
+            logger.info("code_predictor_config is None. Initializing code_predictor model with default values")
+        elif isinstance(code_predictor_config, Qwen3TTSTalkerCodePredictorConfig):
+            self.code_predictor_config = code_predictor_config
+        else:
+            self.code_predictor_config = Qwen3TTSTalkerCodePredictorConfig(**code_predictor_config)
+        self.num_code_groups = num_code_groups
+        self.text_hidden_size = text_hidden_size
+        self.codec_eos_token_id = codec_eos_token_id
+        self.codec_think_id = codec_think_id
+        self.codec_language_id = codec_language_id
+        self.codec_nothink_id = codec_nothink_id
+        self.codec_think_bos_id = codec_think_bos_id
+        self.codec_think_eos_id = codec_think_eos_id
+        self.codec_pad_id = codec_pad_id
+        self.codec_bos_id = codec_bos_id
+        self.spk_id = spk_id
+        self.spk_is_dialect = spk_is_dialect
+
+
+class Qwen3TTSConfig(PretrainedConfig):
+    """
+    This is the configuration class to store the configuration of a [`Qwen3TTSForConditionalGeneration`].
+    """
+
+    model_type = "qwen3_tts"
+    sub_configs = {
+        "talker_config": Qwen3TTSTalkerConfig,
+        "speaker_encoder_config": Qwen3TTSSpeakerEncoderConfig,
+    }
+
+    def __init__(
+        self,
+        talker_config=None,
+        speaker_encoder_config=None,
+        tokenizer_type=None,
+        tts_model_size=None,
+        tts_model_type=None,
+        im_start_token_id=151644,
+        im_end_token_id=151645,
+        tts_pad_token_id=151671,
+        tts_bos_token_id=151672,
+        tts_eos_token_id=151673,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        if talker_config is None:
+            talker_config = {}
+            logger.info("talker_config is None. Initializing talker model with default values")
+        if speaker_encoder_config is None:
+            speaker_encoder_config = {}
+            logger.info("speaker_encoder_config is None. Initializing talker model with default values")
+
+        self.talker_config = Qwen3TTSTalkerConfig(**talker_config)
+        self.speaker_encoder_config = Qwen3TTSSpeakerEncoderConfig(**speaker_encoder_config)
+
+        self.tokenizer_type = tokenizer_type
+        self.tts_model_size = tts_model_size
+        self.tts_model_type = tts_model_type
+
+        self.im_start_token_id = im_start_token_id
+        self.im_end_token_id = im_end_token_id
+        self.tts_pad_token_id = tts_pad_token_id
+        self.tts_bos_token_id = tts_bos_token_id
+        self.tts_eos_token_id = tts_eos_token_id
+
+
+__all__ = ["Qwen3TTSConfig", "Qwen3TTSTalkerConfig", "Qwen3TTSSpeakerEncoderConfig"]

src/core/processing_qwen3_tts.py

@@ -0,0 +1,104 @@
+# coding=utf-8
+# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.processing_utils import ProcessingKwargs, ProcessorMixin
+
+
+class Qwen3TTSProcessorKwargs(ProcessingKwargs, total=False):
+    _defaults = {
+        "text_kwargs": {
+            "padding": False,
+            "padding_side": "left",
+        }
+    }
+
+
+class Qwen3TTSProcessor(ProcessorMixin):
+    r"""
+    Constructs a Qwen3TTS processor.
+
+    Args:
+        tokenizer ([`Qwen2TokenizerFast`], *optional*):
+            The text tokenizer.
+        chat_template (`Optional[str]`, *optional*):
+            The Jinja template to use for formatting the conversation. If not provided, the default chat template is used.
+    """
+
+    attributes = ["tokenizer"]
+    tokenizer_class = ("Qwen2Tokenizer", "Qwen2TokenizerFast")
+
+    def __init__(self, tokenizer=None, chat_template=None):
+        super().__init__(tokenizer, chat_template=chat_template)
+
+    def __call__(self, text=None, **kwargs) -> BatchFeature:
+        """
+        Main method to prepare for the model one or several sequences(s) and audio(s). This method forwards the `text`
+        and `kwargs` arguments to Qwen2TokenizerFast's [`~Qwen2TokenizerFast.__call__`] if `text` is not `None` to encode
+        the text.
+
+        Args:
+            text (`str`, `List[str]`, `List[List[str]]`):
+                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
+                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
+                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
+        """
+
+        if text is None:
+            raise ValueError("You need to specify either a `text` input to process.")
+
+        output_kwargs = self._merge_kwargs(
+            Qwen3TTSProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+        if not isinstance(text, list):
+            text = [text]
+
+        print(f"Qwen3TTSProcessor __call__ text {text}")
+        print(f"Qwen3TTSProcessor __call__ output_kwargs[text_kwargs] {output_kwargs['text_kwargs']}")
+        texts_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
+        print(f"Qwen3TTSProcessor __call__ texts_inputs {texts_inputs}")
+
+        return BatchFeature(
+            data={**texts_inputs},
+            tensor_type=kwargs.get("return_tensors"),
+        )
+
+    def batch_decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to Qwen2TokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please
+        refer to the docstring of this method for more information.
+        """
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to Qwen2TokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to
+        the docstring of this method for more information.
+        """
+        return self.tokenizer.decode(*args, **kwargs)
+
+    def apply_chat_template(self, conversations, chat_template=None, **kwargs):
+        if isinstance(conversations[0], dict):
+            conversations = [conversations]
+        return super().apply_chat_template(conversations, chat_template, **kwargs)
+
+    @property
+    def model_input_names(self):
+        tokenizer_input_names = self.tokenizer.model_input_names
+        return list(dict.fromkeys(tokenizer_input_names))
+
+
+__all__ = ["Qwen3TTSProcessor"]

src/inference/__init__.py

@@ -0,0 +1 @@
+from .qwen3_tts_inferencer_onnx import Qwen3TTSInferencerONNX

src/inference/qwen3_tts_inferencer_onnx.py

@@ -0,0 +1,1112 @@
+# Copyright 2026 Patrick Lumbantobing, Vertox-AI
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+ONNX-based inference engine for Qwen3-TTS-Realtime streaming text-to-speech.
+This module provides a pure NumPy / ONNX Runtime inference pipeline for the
+Qwen3-TTS-Realtime model, enabling streaming TTS without PyTorch dependency
+at runtime. The pipeline follows the same architectural flow as the original
+PyTorch implementation:
+    1. **ECAPA-TDNN speaker encoder** -- encodes a reference waveform into
+       a 1024-dim speaker embedding vector for voice cloning.
+    2. **Talker Backbone LLM** -- autoregressively maps interleaved
+       text + audio tokens embeddings to hidden states and VQ audio semantic
+       token via a causal language model with KV-cache. Sampling is also
+       performed within ONNX graph; sampling parameters can be adjusted.
+    3. **Local Talker Transformer** -- depth-wise decoder that converts each
+       talker hidden state and VQ into a 15-codebook audio frame using its
+       own ephemeral KV-cache (reset per frame). Similarly, sampling is
+       within ONNX graph, and sampling parameters can be adjusted.
+    4. **Codec decoder** -- converts batches of RVQ audio codes (1+15=16 codes
+       per frame) back into 24 kHz waveform samples with streaming KV-cache
+       and convolution cache support.
+Typical usage::
+    inferencer = Qwen3TTSInferencerONNX(
+        talker_llm, talker_local, codec_decoder,
+        speaker_encoder, talker_codec_embed, text_embed_proj,
+        preprocessor_config_dir, model_config, codec_config,
+        audio_ref_path, language,
+    )
+    inferencer.reset_turn(reset_cache=True)
+    for delta in llm_stream:
+        audio_frames = inferencer.push_text(delta)
+        ...
+        for audio_tokens in audio_frames:
+            ...
+            inferencer.push_tokens(audio_tokens)
+            for wav in inferencer.audio_chunks():
+                ...
+                yield wav
+"""
+
+import base64
+import io
+import json
+import logging
+import re
+import urllib.request
+from typing import Iterable, List, Optional, Tuple, Union
+from urllib.parse import urlparse
+
+import librosa
+import numpy as np
+import numpy.typing as npt
+import onnxruntime as ort
+import soundfile as sf
+from box import Box
+from transformers import AutoProcessor
+
+from src.core import Qwen3TTSConfig, Qwen3TTSProcessor
+from src.utils import mel_spectrogram_numpy
+
+log = logging.getLogger(__name__)
+NDArrayInt = npt.NDArray[np.int64]
+"""Typed alias for ``int64`` NumPy arrays used for token sequences."""
+
+NDArrayFloat = npt.NDArray[np.floating]
+"""Typed alias for floating-point NumPy arrays used for audio waveforms."""
+
+AudioLike = Union[
+    str,  # wav path, URL, base64
+    np.ndarray,  # waveform (requires sr)
+    Tuple[np.ndarray, int],  # (waveform, sr)
+]
+
+
+class Qwen3TTSInferencerONNX:
+    """
+    Streaming TTS inference engine backed by six ONNX Runtime sessions.
+    This class orchestrates the full Qwen3-TTS-Realtime pipeline using only
+    NumPy arrays and ONNX Runtime ``InferenceSession`` objects, with no
+    dependency on PyTorch at inference time.
+    Architecture overview::
+        text deltas --> push_text() --> talker LLM (Qwen3)
+                                             |
+                                             v
+                                      local talker transformer
+                                             |
+                                        audio tokens
+                                             |
+                                             v
+                                      codec decoder --> waveform
+    The backbone LLM maintains a growing KV-cache across the entire
+    generation. The local transformer creates a fresh KV-cache per audio
+    frame (15 autoregressive steps for 15 codebooks) and discards it. The
+    codec decoder maintains KV-caches that grow with the decoded audio length
+    with 72 window frames, pre-convolution cache 2 frames,
+    and convolution upsampling cache with 25 window frames.
+    Parameters
+    ----------
+    talker_model_path : str
+        Path to ONNX file for talker model.
+    talker_local_model_path : str
+        Path to ONNX file for local talker model.
+    codec_decoder_model_path : str
+        Path to ONNX file for codec decoder model.
+    speaker_encoder_model_path : str
+        Path to ONNX file for speaker encoder model.
+    talker_codec_embed_model_path : str
+        Path to ONNX file for talker codec embedding.
+    text_embed_proj_model_path : str
+        Path to ONNX file for text embedding and projection.
+    preprocessor_config_dir : str
+        Path to configuration dir that contains preprocessor config files as the original Qwen3-TTS.
+    model_config_path : str
+        Path to model configuration as the original Qwen3-TTS.
+    codec_config_path : str
+        Path to codec configuration as the original Qwen3-TTS.
+    audio_ref_path : str
+        Path to audio reference for voice cloning identity>
+    language : str
+        Language of the synthesized audio.
+    num_threads : int, optional
+        Number of threads used in sess.intra_op_num_threads (default ``4``).
+    chunk_frames : int, optional
+        Number chunk frames in codec decoder forward passes (default ``4`` [0.32 s as it is 12.5 Hz]).
+    temperature : float, optional
+        Sampling temperature for the local transformer (default ``0.85``).
+    top_p : float, optional
+        Nucleus sampling threshold (default ``0.8``).
+    top_k : int, optional
+        Top-k sampling cutoff (default ``50``).
+    repetition_penalty : float, optional
+        Repetition penalty coefficient (default ``1.9``).
+    repetition_window : int, optional
+        Number of recent tokens considered for repetition penalty
+        (default ``50``).
+    """
+
+    _split_pattern = re.compile(
+        r"[。！？!?\.\u2026]\s*"  # sentence boundaries: 。！？ ! ? . …
+        r"|[,，;；:：\u2014\u2013\-]\s*"  # short pauses: , ， ; ； : ： — – -
+        r"|\)\s*|\]\s*"  # closing brackets: ) ]
+        r"|\n"
+    )
+
+    def __init__(
+        self,
+        talker_model_path: str,
+        talker_local_model_path: str,
+        codec_decoder_model_path: str,
+        speaker_encoder_model_path: str,
+        talker_codec_embed_model_path: str,
+        text_embed_proj_model_path: str,
+        preprocessor_config_dir: str,
+        model_config_path: str,
+        codec_config_path: str,
+        audio_ref_path: str,
+        language: str,
+        num_threads: int = 4,
+        chunk_frames: int = 4,
+        temperature=0.725,
+        top_p=0.6,
+        top_k=34,
+        repetition_penalty=1.9,
+        repetition_window=50,
+    ) -> None:
+
+        opts = ort.SessionOptions()
+        opts.intra_op_num_threads = num_threads
+        opts.inter_op_num_threads = 1
+        opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+        opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
+        opts.enable_cpu_mem_arena = True
+        opts.enable_mem_pattern = True
+        providers = ["CPUExecutionProvider"]
+
+        logging.info("Loading ONNX sessions...")
+        logging.info(f"     from {talker_model_path}...")
+        self._talker = ort.InferenceSession(talker_model_path, sess_options=opts, providers=providers)
+        logging.info(f"     from {talker_local_model_path}...")
+        self._talker_local = ort.InferenceSession(talker_local_model_path, sess_options=opts, providers=providers)
+        logging.info(f"     from {codec_decoder_model_path}...")
+        self._codec_decoder = ort.InferenceSession(codec_decoder_model_path, sess_options=opts, providers=providers)
+        logging.info(f"     from {speaker_encoder_model_path}...")
+        self._speaker_encoder = ort.InferenceSession(
+            speaker_encoder_model_path, sess_options=opts, providers=providers
+        )
+        logging.info(f"     from {talker_codec_embed_model_path}...")
+        self._talker_codec_embed = ort.InferenceSession(
+            talker_codec_embed_model_path, sess_options=opts, providers=providers
+        )
+        logging.info(f"     from {text_embed_proj_model_path}...")
+        self._text_embed_proj = ort.InferenceSession(
+            text_embed_proj_model_path, sess_options=opts, providers=providers
+        )
+        logging.info("[OK] All ONNX sessions loaded.")
+
+        AutoProcessor.register(Qwen3TTSConfig, Qwen3TTSProcessor)
+        self._processor = AutoProcessor.from_pretrained(
+            preprocessor_config_dir,
+            fix_mistral_regex=True,
+        )
+
+        self._audio_ref_path = audio_ref_path
+
+        with open(codec_config_path, "r") as f:
+            self._speech_tokenizer_config = Box(json.load(f))
+
+        self._speech_tokenizer_decoder_config = self._speech_tokenizer_config.decoder_config
+        self._speech_tokenizer_latent_dim = self._speech_tokenizer_decoder_config.latent_dim
+        self._speech_tokenizer_codebook_dim = self._speech_tokenizer_decoder_config.codebook_dim
+        self._speech_tokenizer_head_dim = self._speech_tokenizer_decoder_config.head_dim
+        self._speech_tokenizer_num_attention_heads = self._speech_tokenizer_decoder_config.num_attention_heads
+        self._speech_tokenizer_num_hidden_layers = self._speech_tokenizer_decoder_config.num_hidden_layers
+        self._speech_tokenizer_num_key_value_heads = self._speech_tokenizer_decoder_config.num_key_value_heads
+        self._speech_tokenizer_sliding_window = self._speech_tokenizer_decoder_config.sliding_window
+
+        self._speech_tokenizer_decoder_left_context_size = 25
+        self._speech_tokenizer_decoder_total_upsample = 1920
+
+        self.output_sample_rate = self._speech_tokenizer_config.output_sample_rate
+
+        with open(model_config_path, "r") as f:
+            self._config = Box(json.load(f))
+
+        self._talker_config = self._config.talker_config
+        self._code_predictor_config = self._talker_config.code_predictor_config
+        self._speaker_encoder_config = self._config.speaker_encoder_config
+
+        self._speaker_encoder_enc_dim = self._speaker_encoder_config.enc_dim  # 1024
+        self._speaker_encoder_sample_rate = self._speaker_encoder_config.sample_rate  # 24000
+
+        self._head_dim = self._talker_config.head_dim  # 128
+        self._hidden_size = self._talker_config.hidden_size  # 1024
+        self._max_position_embeddings = self._talker_config.max_position_embeddings  # 32768
+        self._num_attention_heads = self._talker_config.num_attention_heads  # 16
+        self._num_code_groups = self._talker_config.num_code_groups  # 16
+        self._num_hidden_layers = self._talker_config.num_hidden_layers  # 28
+        self._num_key_value_heads = self._talker_config.num_key_value_heads  # 8
+        self._text_hidden_size = self._talker_config.text_hidden_size  # 2048
+        self._text_vocab_size = self._talker_config.text_vocab_size  # 151936
+        self._vocab_size = self._talker_config.vocab_size  # 3072
+
+        self._local_head_dim = self._code_predictor_config.head_dim  # 128
+        self._local_hidden_size = self._code_predictor_config.hidden_size  # 1024
+        self._local_max_position_embeddings = self._code_predictor_config.max_position_embeddings  # 65536
+        self._local_num_attention_heads = self._code_predictor_config.num_attention_heads  # 16
+        self._local_num_code_groups = self._code_predictor_config.num_code_groups  # 16
+        self._local_num_hidden_layers = self._code_predictor_config.num_hidden_layers  # 5
+        self._local_num_key_value_heads = self._code_predictor_config.num_key_value_heads  # 8
+        self._local_vocab_size = self._code_predictor_config.vocab_size  # 2048
+
+        self._assistant_token_id = self._config.assistant_token_id  # 77091
+        self._im_end_token_id = self._config.im_end_token_id  # 151645
+        self._im_start_token_id = self._config.im_start_token_id  # 151644
+        self._tts_bos_token_id = self._config.tts_bos_token_id  # 151672
+        self._tts_eos_token_id = self._config.tts_eos_token_id  # 151673
+        self._tts_pad_token_id = self._config.tts_pad_token_id  # 151671
+
+        self._codec_bos_id = self._talker_config.codec_bos_id  # 2149
+        self._codec_eos_token_id = self._talker_config.codec_eos_token_id  # 2150
+        self._codec_think_id = self._talker_config.codec_think_id  # 2154
+
+        # 2048 -> 3072, except 2150 not used in first codebook
+        self._suppress_tokens = [i for i in range(self._local_vocab_size, self._vocab_size)]
+        if self._codec_eos_token_id in self._suppress_tokens:
+            idx_eos = self._suppress_tokens.index(self._codec_eos_token_id)
+            del self._suppress_tokens[idx_eos]
+        self._suppress_tokens = np.array(self._suppress_tokens, dtype=np.int64)
+
+        # "chinese": 2055, "english": 2050, "german": 2053, "italian": 2070, "portuguese": 2071,
+        # "spanish": 2054, "japanese": 2058, "korean": 2064, "french": 2061, "russian": 2069
+        self._codec_language_id = self._talker_config.codec_language_id
+        assert (
+            language in self.get_supported_languages()
+        ), f"language {language} not in {self.get_supported_languages()}"
+        self._language = language
+
+        self._codec_nothink_id = self._talker_config.codec_nothink_id  # 2155
+        self._codec_pad_id = self._talker_config.codec_pad_id  # 2148
+        self._codec_think_bos_id = self._talker_config.codec_think_bos_id  # 2156
+        self._codec_think_eos_id = self._talker_config.codec_think_eos_id  # 2157
+
+        self._temperature = np.array([temperature], dtype=np.float32)
+        self._top_p = np.array([top_p], dtype=np.float32)
+        self._top_k = np.array([top_k], dtype=np.int64)
+        self._repetition_penalty = np.array([repetition_penalty], dtype=np.float32)
+        self._repetition_window = np.array([repetition_window], dtype=np.int64)
+
+        self.text_buffer_size = 32
+        self.min_text_chunk_chars = 8
+
+        self.chunk_frames = chunk_frames
+        self.overlap_frames = 0
+
+        self._max_steps = 2048
+
+        self._prev_tail: Optional[np.array] = None
+        self._buffer: list[np.array] = []
+        self._buffer_len = 0
+
+        self._prefill_key_values_llm = None
+        # For talker kv cache, zero length in time-dim
+        self._past_key_values_llm = []
+        for _ in range(self._num_hidden_layers):
+            self._past_key_values_llm.append(
+                np.zeros((1, self._num_key_value_heads, 0, self._head_dim), dtype=np.float32)
+            )  # key
+            self._past_key_values_llm.append(
+                np.zeros((1, self._num_key_value_heads, 0, self._head_dim), dtype=np.float32)
+            )  # value
+        logging.info(
+            f"inference init self._past_key_values_llm {self._past_key_values_llm} {self._past_key_values_llm[0].shape}"
+        )
+
+        # For speech tokenizer hidden state cache and kv cache, zero length in time-dim
+        self._past_key_values_speech_tokenizer = []
+        for _ in range(self._speech_tokenizer_num_hidden_layers):
+            self._past_key_values_speech_tokenizer.append(
+                np.zeros(
+                    (
+                        1,
+                        self._speech_tokenizer_num_key_value_heads,
+                        0,
+                        self._speech_tokenizer_head_dim,
+                    ),
+                    dtype=np.float32,
+                )
+            )  # key
+            self._past_key_values_speech_tokenizer.append(
+                np.zeros(
+                    (
+                        1,
+                        self._speech_tokenizer_num_key_value_heads,
+                        0,
+                        self._speech_tokenizer_head_dim,
+                    ),
+                    dtype=np.float32,
+                )
+            )  # value
+        logging.info(
+            f"inference init self._past_key_values_speech_tokenizer {self._past_key_values_speech_tokenizer} {self._past_key_values_speech_tokenizer[0].shape}"
+        )
+
+        self._pre_conv_hidden_state_cache_speech_tokenizer = np.zeros(
+            (1, self._speech_tokenizer_codebook_dim, 2), dtype=np.float32
+        )
+        logging.info(
+            f"inference init self._pre_conv_hidden_state_cache_speech_tokenizer {self._pre_conv_hidden_state_cache_speech_tokenizer} {self._pre_conv_hidden_state_cache_speech_tokenizer.shape}"
+        )
+        self._hidden_state_cache_speech_tokenizer = np.zeros(
+            (1, self._speech_tokenizer_latent_dim, 0), dtype=np.float32
+        )
+        logging.info(
+            f"inference init self._hidden_state_cache_speech_tokenizer {self._hidden_state_cache_speech_tokenizer} {self._hidden_state_cache_speech_tokenizer.shape}"
+        )
+
+        # [1, 0, 16]
+        self._generated_tokens = np.zeros((1, 0, self._num_code_groups), dtype=np.int64)
+        logging.info(f"inference init self._generated_tokens {self._generated_tokens} {self._generated_tokens.shape}")
+
+        self._is_stopping = False
+        self._last_audio_tokens = None
+        self._last_first_token = None
+        self._last_first_token_embed = None
+        self._last_local_tokens_embed = None
+        self._last_hidden_states = None
+        self._step_idx = 0
+
+        self._turn_input_ids = None
+        self._turn_idx = 0
+
+        self._text_cache = ""
+        self._pending_tokens: list[int] = []
+        self._prefilled = False
+        self._text_ended = False
+
+    @property
+    def is_finished(self) -> bool:
+        return self._is_stopping or self._step_idx >= self._max_steps
+
+    # ------------------------ HELPERS FOR MAIN FUNCTIONS ------------------------
+
+    def get_supported_languages(self) -> Optional[List[str]]:
+        """
+        List supported language names for the current model.
+
+        This is a convenience wrapper around `model.get_supported_languages()`.
+        If the underlying model does not expose language constraints (returns None),
+        this method also returns None.
+
+        Returns:
+            Optional[List[str]]:
+                - A sorted list of supported language names (lowercased), if available.
+                - None if the model does not provide supported languages.
+        """
+        supported = list(self._config.talker_config.codec_language_id.keys())
+        if supported is None:
+            return None
+        return sorted(set([str(lang).lower() for lang in supported]))
+
+    def _is_url(self, s: str) -> bool:
+        try:
+            u = urlparse(s)
+            return u.scheme in ("http", "https") and bool(u.netloc)
+        except Exception:
+            return False
+
+    def _is_probably_base64(self, s: str) -> bool:
+        if s.startswith("data:audio"):
+            return True
+        if ("/" not in s and "\\" not in s) and len(s) > 256:
+            return True
+        return False
+
+    def _decode_base64_to_wav_bytes(self, b64: str) -> bytes:
+        if "," in b64 and b64.strip().startswith("data:"):
+            b64 = b64.split(",", 1)[1]
+        return base64.b64decode(b64)
+
+    def _load_audio_to_np(self, x: str) -> Tuple[np.ndarray, int]:
+        if self._is_url(x):
+            with urllib.request.urlopen(x) as resp:
+                audio_bytes = resp.read()
+            with io.BytesIO(audio_bytes) as f:
+                audio, sr = sf.read(f, dtype="float32", always_2d=False)
+        elif self._is_probably_base64(x):
+            wav_bytes = self._decode_base64_to_wav_bytes(x)
+            with io.BytesIO(wav_bytes) as f:
+                audio, sr = sf.read(f, dtype="float32", always_2d=False)
+        else:
+            audio, sr = librosa.load(x, sr=None, mono=True)
+
+        if audio.ndim > 1:
+            audio = np.mean(audio, axis=-1)
+
+        return audio.astype(np.float32), int(sr)
+
+    def _normalize_audio_inputs(self, audio: AudioLike) -> Tuple[NDArrayFloat, int]:
+        """
+        Normalize audio inputs into a list of (waveform, sr).
+
+        Supported forms:
+          - str: wav path / URL / base64 audio string
+          - (np.ndarray, sr): waveform + sampling rate
+          - list of the above
+
+        Args:
+            audios:
+                Audio input(s).
+
+        Returns:
+            List[Tuple[np.ndarray, int]]:
+                List of (float32 waveform, original sr).
+
+        Raises:
+            ValueError: If a numpy waveform is provided without sr.
+        """
+        if isinstance(audio, str):
+            audio = self._load_audio_to_np(audio)
+        elif isinstance(audio, tuple) and len(audio) == 2 and isinstance(audio[0], np.ndarray):
+            audio = (audio.astype(np.float32), int(audio[1]))
+        elif isinstance(audio, np.ndarray):
+            raise ValueError("For numpy waveform input, pass a tuple (audio, sr).")
+        else:
+            raise TypeError(f"Unsupported audio input type: {type(audio)}")
+        if audio[0].ndim > 1:
+            audio[0] = np.mean(audio[0], axis=-1).astype(np.float32)
+            audio = (audio[0], audio[1])
+        return audio
+
+    def _build_assistant_text(self) -> str:
+        return "<|im_start|>assistant\n"
+
+    # ------------------------ MAIN FUNCTIONS ------------------------
+
+    def _tokenize_texts(self, text: Union[str, List[str]]) -> List[int]:
+        logging.info(f"_tokenize_texts text {text} {len(text[0]) if isinstance(text, list) else len(text)}")
+        input_ids = self._processor(text=text, return_tensors="np", padding=True)
+        logging.info(f"_tokenize_texts input_ids_dict {input_ids}")
+        input_ids = input_ids["input_ids"]
+        logging.info(f"_tokenize_texts input_ids {input_ids} {input_ids.shape}")
+        input_ids = np.expand_dims(input_ids, axis=0) if input_ids.ndim == 1 else input_ids
+        logging.info(f"_tokenize_texts input_ids_ {input_ids} {input_ids.shape}")
+        return list(input_ids[0])  # [B, T] -> [T]
+
+    def _prefill_embeds(
+        self,
+        audio_ref_path: str,
+        language: Optional[str] = None,
+    ) -> NDArrayFloat:
+        language_id = None
+        if language:
+            if language.lower() != "auto":
+                if language.lower() not in self._codec_language_id:
+                    raise NotImplementedError(f"Language {language} not implemented")
+                else:
+                    language_id = self._codec_language_id[language.lower()]
+        logging.info(f"_prefill_embeds language_id {language_id}")
+        speaker_embed = self.create_voice_clone_spkemb(audio_ref_path)  # [B, 1, 512]
+        logging.info(f"_prefill_embeds speaker_embed {speaker_embed} {speaker_embed.shape}")
+
+        # For prefill
+        if language_id is not None:
+            codec_prefill_list = np.array(
+                [
+                    [
+                        self._codec_think_id,
+                        self._codec_think_bos_id,
+                        language_id,
+                        self._codec_think_eos_id,
+                    ]
+                ],
+                dtype=np.int64,
+            )
+        else:
+            codec_prefill_list = np.array(
+                [
+                    [
+                        self._codec_nothink_id,
+                        self._codec_think_bos_id,
+                        self._codec_think_eos_id,
+                    ]
+                ],
+                dtype=np.int64,
+            )
+        logging.info(f"generate codec_prefill_list {codec_prefill_list}")
+        outputs = self._talker_codec_embed.run(["codec_emb"], {"codec_ids": codec_prefill_list})
+        codec_input_embedding_0 = outputs[0]
+        logging.info(f"generate codec_input_embedding_0 {codec_input_embedding_0} {codec_input_embedding_0.shape}")
+        outputs = self._talker_codec_embed.run(
+            ["codec_emb"], {"codec_ids": np.array([[self._codec_pad_id]], dtype=np.int64)}
+        )
+        codec_input_embedding_1 = outputs[0]
+        # self_codec_bos_id,
+        logging.info(f"generate codec_input_embedding_1 {codec_input_embedding_1} {codec_input_embedding_1.shape}")
+        codec_input_embedding = np.concatenate(
+            [codec_input_embedding_0, speaker_embed, codec_input_embedding_1], axis=1
+        )
+        logging.info(f"generate codec_input_embedding {codec_input_embedding} {codec_input_embedding.shape}")
+
+        # <|im_start|>assistant\n
+        prefix_tokens = np.expand_dims(
+            np.array(self._tokenize_texts([self._build_assistant_text()]), dtype=np.int64), axis=0
+        )
+        outputs = self._text_embed_proj.run(["text_emb_out"], {"text_ids": prefix_tokens})  # 3
+        _talker_input_embed_role = outputs[0]
+        logging.info(f"generate _talker_input_embed_role {_talker_input_embed_role} {_talker_input_embed_role.shape}")
+
+        outputs = self._text_embed_proj.run(
+            ["text_emb_out"],
+            {"text_ids": np.array([[self._tts_bos_token_id, self._tts_pad_token_id]], dtype=np.int64)},
+        )
+        embeds = outputs[0]
+        tts_bos_embed, tts_pad_embed = embeds[:, :1], embeds[:, 1:]  # 2 * [1 1 d]
+        logging.info(f"generate tts_bos_embed {tts_bos_embed} {tts_bos_embed.shape}")
+        logging.info(f"generate tts_pad_embed {tts_pad_embed} {tts_pad_embed.shape}")
+
+        # tts_pad * (4 or 5) + tts_bos; codec_input_embedding_0 (+ speaker_embed) + codec_pad_id --> 5 or 6
+        _talker_input_embed = (
+            np.concatenate(
+                (
+                    np.broadcast_to(
+                        tts_pad_embed,
+                        (tts_pad_embed.shape[0], codec_input_embedding.shape[1] - 1, tts_pad_embed.shape[2]),
+                    ),  # 4 or 5
+                    tts_bos_embed,  # 1
+                ),
+                axis=1,
+            )
+            + codec_input_embedding  # 5 or 6
+        )
+        logging.info(f"generate _talker_input_embed {_talker_input_embed} {_talker_input_embed.shape}")
+
+        talker_input_embed = np.concatenate((_talker_input_embed_role, _talker_input_embed), axis=1)  # 3 + 5/6
+        logging.info(f"generate talker_input_embed {talker_input_embed} {talker_input_embed.shape}")
+
+        return talker_input_embed
+
+    def create_voice_clone_spkemb(
+        self,
+        ref_audio: AudioLike,
+    ) -> NDArrayFloat:
+        normalized = self._normalize_audio_inputs(ref_audio)
+        logging.info(f"create_voice_clone_prompt normalized {normalized} {normalized[0][0].shape} {normalized[0][1]}")
+
+        wav, sr = normalized
+        wav_resample = wav
+        if sr != self._speaker_encoder_sample_rate:
+            wav_resample = librosa.resample(
+                y=wav_resample.astype(np.float32),
+                orig_sr=int(sr),
+                target_sr=self._speaker_encoder_sample_rate,
+            )
+
+        logging.info(
+            f"create_voice_clone_spkemb wav_resample {wav_resample} {wav_resample.shape} {wav_resample.dtype}"
+        )
+        mels = mel_spectrogram_numpy(
+            wav_resample,
+            n_fft=1024,
+            num_mels=128,
+            sampling_rate=24000,
+            hop_size=256,
+            win_size=1024,
+            fmin=0,
+            fmax=12000,
+        )
+        logging.info(f"create_voice_clone_spkemb mels {mels} {mels.shape} {mels.dtype}")
+
+        outputs = self._speaker_encoder.run(["speaker_embedding"], {"mel_spec": mels})
+        spk_emb = outputs[0]
+        logging.info(f"create_voice_clone_prompt spk_emb {spk_emb} {spk_emb.shape}")
+
+        return spk_emb
+
+    def generate_local_transformer(self) -> None:
+        feed = {
+            "past_hidden": self._last_hidden_states,
+            "past_id_hidden": self._last_first_token_embed,
+            "generated_tokens": self._generated_tokens[..., 1:],
+            "temperature": self._temperature,
+            "top_p": self._top_p,
+            "top_k": self._top_k,
+            "repetition_penalty": self._repetition_penalty,
+            "repetition_window": self._repetition_window,
+        }
+        output_names = ["outputs_tokens", "outputs_embeds"]
+        logging.info(
+            f"generate_local_transformer self._last_hidden_states {self._last_hidden_states} {self._last_hidden_states.shape} {self._last_hidden_states.dtype}"
+        )
+        logging.info(
+            f"generate_local_transformer self._last_first_token_embed {self._last_first_token_embed} {self._last_first_token_embed.shape} {self._last_first_token_embed.dtype}"
+        )
+        outputs = self._talker_local.run(output_names, feed)
+        local_tokens, self._last_local_tokens_embed = outputs[0], outputs[1]
+        logging.info(
+            f"generate_local_transformer local_tokens {local_tokens} {local_tokens.shape} {local_tokens.dtype}"
+        )
+        logging.info(
+            f"generate_local_transformer self._last_local_tokens_embed {self._last_local_tokens_embed} {self._last_local_tokens_embed.shape} {self._last_local_tokens_embed.dtype}"
+        )
+        self._last_audio_tokens = np.concatenate(
+            (np.expand_dims(self._last_first_token, axis=-1), local_tokens), axis=1
+        )[None, :, :]
+        logging.info(
+            f"generate_local_transformer self._last_audio_tokens {self._last_audio_tokens} {self._last_audio_tokens.shape} {self._last_audio_tokens.dtype}"
+        )
+        self._generated_tokens = np.concatenate((self._generated_tokens, self._last_audio_tokens), axis=1)
+        logging.info(
+            f"generate_local_transformer self._generated_tokens {self._generated_tokens.shape} {self._generated_tokens.dtype}"
+        )
+        return
+
+    def _set_talker_zero_kv_cache(self, batch_size=1):
+        """Set talker zero KV cache for all layers."""
+        kv = {}
+        for i in range(self._num_hidden_layers):
+            kv[f"past_key_{i}"] = np.zeros(
+                (batch_size, self._num_key_value_heads, 0, self._head_dim), dtype=np.float32
+            )
+            kv[f"past_value_{i}"] = np.zeros(
+                (batch_size, self._num_key_value_heads, 0, self._head_dim), dtype=np.float32
+            )
+        return kv
+
+    def _set_talker_kv_cache(self):
+        """Set talker KV cache for all layers."""
+        kv = {}
+        for i in range(self._num_hidden_layers):
+            kv[f"past_key_{i}"] = self._past_key_values_llm[2 * i]
+            kv[f"past_value_{i}"] = self._past_key_values_llm[2 * i + 1]
+        return kv
+
+    def _set_codec_decoder_kv_cache(self, past_key_values):
+        """Set talker KV cache for all layers."""
+        kv = {}
+        for i in range(self._speech_tokenizer_num_hidden_layers):
+            kv[f"past_key_{i}"] = past_key_values[2 * i]
+            kv[f"past_value_{i}"] = past_key_values[2 * i + 1]
+        return kv
+
+    def prefill(self) -> None:
+        inputs_embeds = self._prefill_embeds(self._audio_ref_path, self._language)
+        logging.info(f"prefill inputs_embeds {inputs_embeds} {inputs_embeds.shape}")
+
+        kv_cache = self._set_talker_zero_kv_cache(batch_size=1)
+        feed = {
+            "inputs_embeds": inputs_embeds,
+            "generated_tokens": self._generated_tokens[..., 0],
+            "temperature": self._temperature,
+            "top_p": self._top_p,
+            "top_k": self._top_k,
+            "repetition_penalty": self._repetition_penalty,
+            "repetition_window": self._repetition_window,
+        }
+        feed.update(kv_cache)
+
+        output_names = ["logits", "token", "token_embed", "hidden_states"]
+        for i in range(self._num_hidden_layers):
+            output_names.extend([f"present_key_{i}", f"present_value_{i}"])
+
+        logging.info(f"prefill inputs_embeds {inputs_embeds} {inputs_embeds.shape} {inputs_embeds.dtype}")
+        logging.info(f"prefill self._prefill_key_values_llm before {self._prefill_key_values_llm}")
+        outputs = self._talker.run(output_names, feed)
+        # logits, self._last_first_token, self._last_first_token_embed, self._last_hidden_states, self._prefill_key_values_llm = outputs[0], outputs[1], outputs[2], outputs[3:]
+        _, _, _, _, self._prefill_key_values_llm = outputs[0], outputs[1], outputs[2], outputs[3], outputs[4:]
+        logging.info(
+            f"prefill self._prefill_key_values_llm after {self._prefill_key_values_llm[0].shape} {len(self._prefill_key_values_llm)}"
+        )
+        self._prefilled = True
+        return
+
+    def step(
+        self,
+        text_token: Optional[int] = None,  # [B, 1]
+    ) -> Union[NDArrayInt, None]:
+        if not self._prefilled:
+            raise ValueError("You must call prefill() before step().")
+        if self.is_finished:
+            return self._last_audio_tokens
+
+        # last codec embeds
+        if self._step_idx > 0:
+            logging.info(
+                f"step-{self._step_idx} self._last_first_token_embed {self._last_first_token_embed.shape} {self._last_first_token_embed.dtype}"
+            )
+            logging.info(
+                f"step-{self._step_idx} self._last_local_tokens_embed {self._last_local_tokens_embed.shape} {self._last_local_tokens_embed.dtype}"
+            )
+            codec_embeds = self._last_first_token_embed + self._last_local_tokens_embed
+            logging.info(f"step-{self._step_idx} codec_embeds {codec_embeds.shape} {codec_embeds.dtype}")
+        else:
+            self._past_key_values_llm = self._prefill_key_values_llm
+            outputs = self._talker_codec_embed.run(
+                ["codec_emb"], {"codec_ids": np.array([[self._codec_bos_id]], dtype=np.int64)}
+            )
+            codec_embeds = outputs[0]
+            logging.info(f"step-{self._step_idx} codec_embeds {codec_embeds.shape} {codec_embeds.dtype}")
+        # tts token step
+        if text_token is not None:
+            text_token = np.array([[text_token]], dtype=np.int64)
+            logging.info(f"step-{self._step_idx} text_token not None {text_token} {text_token.shape}")
+        else:
+            text_token = np.array([[self._tts_pad_token_id]], dtype=np.int64)
+            logging.info(f"step-{self._step_idx} text_token None {text_token} {text_token.shape}")
+        outputs = self._text_embed_proj.run(["text_emb_out"], {"text_ids": text_token})
+        text_embeds = outputs[0]
+        logging.info(f"step-{self._step_idx} text_embeds {text_embeds} {text_embeds.shape} {text_embeds.dtype}")
+        inputs_embeds = text_embeds + codec_embeds
+        logging.info(
+            f"step-{self._step_idx} inputs_embeds {inputs_embeds} {inputs_embeds.shape} {inputs_embeds.dtype}"
+        )
+
+        kv_cache = self._set_talker_kv_cache()
+        feed = {
+            "inputs_embeds": inputs_embeds,
+            "generated_tokens": self._generated_tokens[..., 0],
+            "temperature": self._temperature,
+            "top_p": self._top_p,
+            "top_k": self._top_k,
+            "repetition_penalty": self._repetition_penalty,
+            "repetition_window": self._repetition_window,
+        }
+        feed.update(kv_cache)
+
+        output_names = ["logits", "token", "token_embed", "hidden_states"]
+        for i in range(self._num_hidden_layers):
+            output_names.extend([f"present_key_{i}", f"present_value_{i}"])
+
+        outputs = self._talker.run(output_names, feed)
+        (
+            _,
+            self._last_first_token,
+            self._last_first_token_embed,
+            self._last_hidden_states,
+            self._past_key_values_llm,
+        ) = (outputs[0], outputs[1], outputs[2], outputs[3], outputs[4:])
+        logging.info(
+            f"step-{self._step_idx} self._last_first_token {self._last_first_token} {self._last_first_token.shape} {self._last_first_token.dtype}"
+        )
+        logging.info(
+            f"step-{self._step_idx} self._last_first_token_embed {self._last_first_token_embed.shape} {self._last_first_token_embed.dtype}"
+        )
+        logging.info(
+            f"step-{self._step_idx} self._last_hidden_states {self._last_hidden_states.shape} {self._last_hidden_states.dtype}"
+        )
+        logging.info(
+            f"step-{self._step_idx} self._past_key_values_llm {self._past_key_values_llm[0].shape} {self._past_key_values_llm[0].dtype}"
+        )
+
+        self._is_stopping = self._last_first_token == self._codec_eos_token_id
+        if self.is_finished:
+            return None
+
+        self.generate_local_transformer()
+        logging.info(
+            f"step-{self._step_idx} self._last_audio_tokens {self._last_audio_tokens} {self._last_audio_tokens.shape} {self._last_audio_tokens.dtype}"
+        )
+        logging.info(
+            f"step-{self._step_idx} self._generated_tokens {self._generated_tokens.shape} {self._generated_tokens.dtype}"
+        )
+        self._step_idx += 1
+        return self._last_audio_tokens
+
+    # ------------------------ STREAMING HELPERS ------------------------
+
+    def _drain_pending_tokens(self) -> list[NDArrayInt]:
+        outputs: list[NDArrayInt] = []
+        if not self._prefilled:
+            self.prefill()
+            return outputs
+
+        while self._pending_tokens and not self.is_finished:
+            logging.info(f"pending_tokens before pop {self._pending_tokens}")
+            token = self._pending_tokens.pop(0)
+            logging.info(f"token {token}")
+            logging.info(f"pending_tokens after pop {self._pending_tokens}")
+            output = self.step(token)
+            if output is not None:
+                outputs.append(output)
+            logging.info(f"outputs {outputs} {len(outputs)}")
+
+        return outputs
+
+    def end_text(self) -> list[NDArrayFloat]:
+        self._text_ended = True
+        if self._text_cache:
+            self._pending_tokens.extend(self._tokenize_texts([self._text_cache]))
+            self._text_cache = ""
+        return self._drain_pending_tokens()
+
+    def drain(self, max_steps: Optional[int] = None) -> list[NDArrayFloat]:
+        if not self._prefilled:
+            return []
+        return self.finish(max_steps=max_steps)
+
+    def _extract_text_segments(self, force: bool) -> list[str]:
+        segments = []
+        if force:
+            if self._text_cache:
+                segments.append(self._text_cache)
+                self._text_cache = ""
+            return segments
+
+        while self._text_cache:
+            cut_idx = None
+            if len(self._text_cache) >= self.min_text_chunk_chars:
+                matches = list(self._split_pattern.finditer(self._text_cache))
+                for match in matches:
+                    if match.end() >= self.min_text_chunk_chars:
+                        cut_idx = match.end()
+                        break
+            if cut_idx is None and len(self._text_cache) >= self.text_buffer_size:
+                whitespace_idx = self._text_cache.rfind(" ")
+                if whitespace_idx != -1:
+                    cut_idx = whitespace_idx + 1
+            if cut_idx is None:
+                break
+            segments.append(self._text_cache[:cut_idx])
+            self._text_cache = self._text_cache[cut_idx:]
+        return segments
+
+    def push_text(self, text_fragment: str) -> list[NDArrayFloat]:
+        logging.info(f"text_cache before {self._text_cache}")
+        logging.info(f"text_fragment {text_fragment}")
+        self._text_cache += text_fragment
+        logging.info(f"text_cache after {self._text_cache}")
+        segments = self._extract_text_segments(force=False)
+        logging.info(f"segments {segments}")
+        for segment in segments:
+            logging.info(f"segment {segment}")
+            logging.info(f"pending_tokens before {self._pending_tokens}")
+            self._pending_tokens.extend(self._tokenize_texts([segment]))
+            logging.info(f"pending_tokens after {self._pending_tokens}")
+        return self._drain_pending_tokens()
+
+    def push_tokens(self, audio_tokens: NDArrayInt):
+        if audio_tokens.ndim != 2:
+            raise ValueError(f"Expected [T, C] audio tokens, got {tuple(audio_tokens.shape)}")
+        self._buffer.append(audio_tokens)
+        self._buffer_len += audio_tokens.shape[0]
+        logging.info(
+            f"push_tokens audio_tokens {audio_tokens} {audio_tokens.shape} self.buffer {self._buffer} {self._buffer_len}"
+        )
+
+    def _overlap_samples(self, wav: NDArrayFloat) -> int:
+        if self.chunk_frames <= 0:
+            return 0
+        return int(wav.size * (self.overlap_frames / self.chunk_frames))
+
+    def _apply_crossfade(self, wav: NDArrayFloat, final_chunk: bool = False) -> NDArrayFloat:
+        if self.overlap_frames <= 0:
+            return wav
+
+        overlap = self._overlap_samples(wav)
+        if overlap == 0:
+            return wav
+
+        if self._prev_tail is None:
+            self._prev_tail = wav[-overlap:].copy() if not final_chunk else None
+            return wav
+
+        prev_tail = self._prev_tail
+        if prev_tail.size < overlap:
+            overlap = prev_tail.size
+        if overlap == 0:
+            return wav
+
+        fade_out = np.linspace(1.0, 0.0, overlap, dtype=wav.dtype)
+        fade_in = 1.0 - fade_out
+
+        cross = prev_tail[-overlap:] * fade_out + wav[:overlap] * fade_in
+        merged = np.concatenate([prev_tail[:-overlap], cross, wav[overlap:]], axis=-1)
+
+        self._prev_tail = None if final_chunk else wav[-overlap:].copy()
+        return merged
+
+    def _process_frames_to_audio(self, chunk_frames_length: int) -> NDArrayFloat:
+        chunk_tokens = self._consume_frames(self.chunk_frames)
+        # pad left for pre_conv inside with cache
+        logging.info(f"_process_frames_to_audio chunk_tokens {chunk_tokens} {chunk_tokens.shape}")
+        # past key values with sliding windows
+        len_for_past_key_values = self._speech_tokenizer_sliding_window - chunk_tokens.shape[-1]
+        logging.info(
+            f"_process_frames_to_audio len_for_past_key_values {len_for_past_key_values} {self._past_key_values_speech_tokenizer[0].shape}"
+        )
+        past_key_values = [
+            past_kv[:, :, -len_for_past_key_values:] for past_kv in self._past_key_values_speech_tokenizer
+        ]
+        logging.info(f"_process_frames_to_audio past_key_values {past_key_values[0].shape} {len_for_past_key_values}")
+        # pad hidden_state_cache for input to upsampling conv with left context size
+        hidden_state_cache = self._hidden_state_cache_speech_tokenizer
+        len_hidden_state_cache = self._hidden_state_cache_speech_tokenizer.shape[-1]
+        logging.info(
+            f"_process_frames_to_audio hidden_state_cache {hidden_state_cache.shape} {len_hidden_state_cache}"
+        )
+
+        kv_cache = self._set_codec_decoder_kv_cache(past_key_values)
+        feed = {
+            "codes": chunk_tokens,
+            "hidden_state_cache": hidden_state_cache,
+            "pre_conv_hidden_state_cache": self._pre_conv_hidden_state_cache_speech_tokenizer,
+        }
+        feed.update(kv_cache)
+
+        output_names = ["wav", "current_hidden_state_cache", "current_pre_conv_hidden_state_cache"]
+        for i in range(self._speech_tokenizer_num_hidden_layers):
+            output_names.extend([f"present_key_{i}", f"present_value_{i}"])
+
+        outputs = self._codec_decoder.run(output_names, feed)
+        wav, hidden_state_cache, self._pre_conv_hidden_state_cache_speech_tokenizer, past_key_values = (
+            outputs[0],
+            outputs[1],
+            outputs[2],
+            outputs[3:],
+        )
+
+        self._past_key_values_speech_tokenizer = [
+            past_kv[:, :, -self._speech_tokenizer_sliding_window + 1 :] for past_kv in past_key_values
+        ]
+        self._hidden_state_cache_speech_tokenizer = hidden_state_cache[
+            :, :, -self._speech_tokenizer_decoder_left_context_size :
+        ]
+        logging.info(
+            f"_process_frames_to_audio self._past_key_values_speech_tokenizer {self._past_key_values_speech_tokenizer[0].shape}"
+        )
+        logging.info(
+            f"_process_frames_to_audio self._hidden_state_cache_speech_tokenizer {self._hidden_state_cache_speech_tokenizer.shape}"
+        )
+        logging.info(
+            f"_process_frames_to_audio self._pre_conv_hidden_state_cache_speech_tokenizer {self._pre_conv_hidden_state_cache_speech_tokenizer.shape}"
+        )
+        logging.info(f"_process_frames_to_audio wav before {wav} {wav.shape}")
+        wav = wav[..., len_hidden_state_cache * self._speech_tokenizer_decoder_total_upsample :]
+        logging.info(f"_process_frames_to_audio wav after {wav} {wav.shape}")
+        return wav
+
+    def flush(self) -> Optional[NDArrayFloat]:
+        if self._buffer_len == 0:
+            return None
+        logging.info(f"flush buffer_len {self._buffer_len}")
+        wav = self._process_frames_to_audio(self._buffer_len)
+        return self._apply_crossfade(wav, final_chunk=True)
+
+    def _consume_frames(self, num_frames: int) -> NDArrayFloat:
+        frames = []
+        remaining = num_frames
+        while remaining > 0 and self._buffer:
+            head = self._buffer[0]
+            if head.shape[0] <= remaining:
+                frames.append(head)
+                remaining -= head.shape[0]
+                self._buffer.pop(0)
+            else:
+                frames.append(head[:remaining])
+                self._buffer[0] = head[remaining:]
+                remaining = 0
+        self._buffer_len -= num_frames - remaining
+        return np.expand_dims(np.transpose(np.concatenate(frames, axis=0), (1, 0)), axis=0)
+
+    def audio_chunks(self) -> Iterable[NDArrayFloat]:
+        while self._buffer_len >= self.chunk_frames:
+            logging.info(f"audio_chunks buffer_len chunk_frames {self._buffer_len} {self.chunk_frames}")
+            wav = self._process_frames_to_audio(self.chunk_frames)
+            yield self._apply_crossfade(wav)
+
+    def finish(self, max_steps: Optional[int] = None) -> list[NDArrayInt]:
+        outputs = []
+        steps_left = max_steps if max_steps is not None else self.max_length
+        while steps_left > 0 and not self.is_finished:
+            output = self.step(text_token=None)
+            if output is not None:
+                outputs.append(output)
+            steps_left -= 1
+        return outputs
+
+    # ------------------------ STATE RESET HELPERS ------------------------
+
+    def reset_generation_state(self, keep_prefill_cache: bool = True) -> None:
+        self._past_key_values_llm = []
+        for _ in range(self._num_hidden_layers):
+            self._past_key_values_llm.append(
+                np.zeros((1, self._num_key_value_heads, 0, self._head_dim), dtype=np.float32)
+            )  # key
+            self._past_key_values_llm.append(
+                np.zeros((1, self._num_key_value_heads, 0, self._head_dim), dtype=np.float32)
+            )  # value
+        logging.info(
+            f"reset self._past_key_values_llm {self._past_key_values_llm} {self._past_key_values_llm[0].shape}"
+        )
+
+        self._past_key_values_speech_tokenizer = []
+        for _ in range(self._speech_tokenizer_num_hidden_layers):
+            self._past_key_values_speech_tokenizer.append(
+                np.zeros(
+                    (
+                        1,
+                        self._speech_tokenizer_num_key_value_heads,
+                        0,
+                        self._speech_tokenizer_head_dim,
+                    ),
+                    dtype=np.float32,
+                )
+            )  # key
+            self._past_key_values_speech_tokenizer.append(
+                np.zeros(
+                    (
+                        1,
+                        self._speech_tokenizer_num_key_value_heads,
+                        0,
+                        self._speech_tokenizer_head_dim,
+                    ),
+                    dtype=np.float32,
+                )
+            )  # value
+        logging.info(
+            f"reset self._past_key_values_speech_tokenizer {self._past_key_values_speech_tokenizer} {self._past_key_values_speech_tokenizer[0].shape}"
+        )
+
+        self._pre_conv_hidden_state_cache_speech_tokenizer = np.zeros(
+            (1, self._speech_tokenizer_codebook_dim, 2), dtype=np.float32
+        )
+        logging.info(
+            f"reset self._pre_conv_hidden_state_cache_speech_tokenizer {self._pre_conv_hidden_state_cache_speech_tokenizer} {self._pre_conv_hidden_state_cache_speech_tokenizer.shape}"
+        )
+        self._hidden_state_cache_speech_tokenizer = np.zeros(
+            (1, self._speech_tokenizer_latent_dim, 0), dtype=np.float32
+        )
+        logging.info(
+            f"reset self._hidden_state_cache_speech_tokenizer {self._hidden_state_cache_speech_tokenizer} {self._hidden_state_cache_speech_tokenizer.shape}"
+        )
+
+        # [1, 0, 16]
+        self._generated_tokens = np.zeros((1, 0, self._num_code_groups), dtype=np.int64)
+        logging.info(f"reset self._generated_tokens {self._generated_tokens} {self._generated_tokens.shape}")
+
+        if not keep_prefill_cache:
+            self._prefill_key_values_llm = None
+            self._prefilled = False
+
+        self._is_stopping = None
+        self._last_audio_tokens = None
+        self._last_first_token = None
+        self._last_first_token_embed = None
+        self._last_hidden_states = None
+        self._step_idx = 0
+
+        return
+
+    def reset_turn(self, reset_cache: bool = False) -> None:
+        self._turn_idx += 1
+
+        self._text_cache = ""
+        self._pending_tokens = []
+        self._prefilled = False
+        self._text_ended = False
+
+        self.reset_generation_state(keep_prefill_cache=True)
+
+        return

src/utils/__init__.py

@@ -0,0 +1 @@
+from .audio_utils import mel_spectrogram_numpy

src/utils/audio_utils.py

@@ -0,0 +1,263 @@
+# Copyright 2026 Patrick Lumbantobing, Vertox-AI
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Utilities functions and classes for audio processing.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import numpy.typing as npt
+
+
+def hz_to_mel(freq: npt.NDArray[np.float64]) -> npt.NDArray[np.float64]:
+    """
+    Convert Hz to mel using the HTK formula.
+
+    Args:
+        freq: Frequencies in Hz.
+
+    Returns:
+        Frequencies in mel.
+    """
+    return 2595.0 * np.log10(1.0 + freq / 700.0)
+
+
+def mel_to_hz(mels: npt.NDArray[np.float64]) -> npt.NDArray[np.float64]:
+    """
+    Convert mel to Hz using the HTK formula.
+
+    Args:
+        mels: Values in mel.
+
+    Returns:
+        Frequencies in Hz.
+    """
+    return 700.0 * (10.0 ** (mels / 2595.0) - 1.0)
+
+
+def librosa_style_mel_filterbank(
+    *,
+    sr: int,
+    n_fft: int,
+    n_mels: int,
+    fmin: float,
+    fmax: float | None = None,
+    norm: str | None = "slaney",
+) -> npt.NDArray[np.float32]:
+    """
+    Build a mel filterbank compatible with librosa.filters.mel using Slaney normalization.
+
+    Args:
+        sr: Sample rate.
+        n_fft: FFT size.
+        n_mels: Number of mel bins.
+        fmin: Minimum frequency in Hz.
+        fmax: Maximum frequency in Hz. If None, defaults to sr / 2.
+        norm: If "slaney", apply area normalization.
+
+    Returns:
+        Mel filterbank with shape [n_mels, n_fft // 2 + 1].
+    """
+    if fmax is None:
+        fmax = sr / 2.0
+
+    n_freqs = n_fft // 2 + 1
+    freqs = np.linspace(0.0, sr / 2.0, n_freqs, dtype=np.float64)
+
+    m_min = hz_to_mel(np.array([fmin], dtype=np.float64))[0]
+    m_max = hz_to_mel(np.array([fmax], dtype=np.float64))[0]
+    m_pts = np.linspace(m_min, m_max, n_mels + 2, dtype=np.float64)
+    hz_pts = mel_to_hz(m_pts)
+
+    fb = np.zeros((n_mels, n_freqs), dtype=np.float64)
+
+    for i in range(n_mels):
+        left, center, right = hz_pts[i], hz_pts[i + 1], hz_pts[i + 2]
+
+        left_slope = (freqs - left) / (center - left + 1e-10)
+        right_slope = (right - freqs) / (right - center + 1e-10)
+
+        fb[i] = np.maximum(0.0, np.minimum(left_slope, right_slope))
+
+    if norm == "slaney":
+        # Match Slaney-style area normalization used by librosa/torchaudio.
+        enorm = 2.0 / (hz_pts[2:] - hz_pts[:-2])
+        fb *= enorm[:, None]
+
+    return fb.astype(np.float32)
+
+
+def dynamic_range_compression_np(
+    x: npt.NDArray[np.float32],
+    C: float = 1.0,
+    clip_val: float = 1e-5,
+) -> npt.NDArray[np.float32]:
+    """
+    NumPy equivalent of torch.log(torch.clamp(x, min=clip_val) * C).
+
+    Args:
+        x: Input array.
+        C: Multiplicative constant.
+        clip_val: Minimum allowed value before log.
+
+    Returns:
+        Log-compressed array.
+    """
+    return np.log(np.clip(x * C, a_min=clip_val, a_max=None)).astype(np.float32)
+
+
+def _reflect_pad_1d(x: npt.NDArray[np.float32], pad: int) -> npt.NDArray[np.float32]:
+    """
+    Reflect-pad a [1, T] waveform along the time axis.
+
+    Args:
+        x: Waveform with shape [1, T].
+        pad: Number of samples to pad on each side.
+
+    Returns:
+        Padded waveform with shape [1, T + 2 * pad].
+    """
+    if pad == 0:
+        return x
+    left = x[:, 1 : pad + 1][:, ::-1]
+    right = x[:, -pad - 1 : -1][:, ::-1]
+    return np.concatenate([left, x, right], axis=1)
+
+
+def _stft_magnitude(
+    y: npt.NDArray[np.float32],
+    *,
+    n_fft: int,
+    hop_size: int,
+    win_size: int,
+    center: bool,
+) -> npt.NDArray[np.float32]:
+    """
+    Compute magnitude STFT for a single-channel waveform.
+
+    Args:
+        y: Input waveform of shape [1, T].
+        n_fft: FFT size.
+        hop_size: Hop size between frames.
+        win_size: Window size.
+        center: Whether to pad the input before framing.
+
+    Returns:
+        Magnitude spectrogram with shape [1, frames, n_fft // 2 + 1].
+    """
+    if y.ndim != 2 or y.shape[0] != 1:
+        raise ValueError("Expected waveform shape [1, T].")
+
+    x = y.astype(np.float32, copy=False)
+
+    if center:
+        pad = n_fft // 2
+        x = _reflect_pad_1d(x, pad)
+
+    if x.shape[1] < n_fft:
+        raise ValueError("Input is too short for the requested n_fft.")
+
+    num_frames = 1 + (x.shape[1] - n_fft) // hop_size
+    frame_starts = hop_size * np.arange(num_frames, dtype=np.int64)
+    frame_offsets = np.arange(n_fft, dtype=np.int64)
+
+    frames = x[:, frame_starts[:, None] + frame_offsets[None, :]]  # [1, frames, n_fft]
+
+    window = np.hanning(win_size).astype(np.float32)
+    if n_fft > win_size:
+        pad_left = (n_fft - win_size) // 2
+        pad_right = n_fft - win_size - pad_left
+        window = np.pad(window, (pad_left, pad_right))
+    elif n_fft < win_size:
+        window = window[:n_fft]
+
+    frames = frames * window[None, None, :]
+
+    spec = np.fft.rfft(frames, n=n_fft, axis=-1)
+    mag = np.sqrt(np.real(spec) ** 2 + np.imag(spec) ** 2 + 1e-9).astype(np.float32)
+    return mag
+
+
+def mel_spectrogram_numpy(
+    y: npt.NDArray[np.float32],
+    n_fft: int,
+    num_mels: int,
+    sampling_rate: int,
+    hop_size: int,
+    win_size: int,
+    fmin: int,
+    fmax: int | None = None,
+    center: bool = False,
+    clip_val: float = 1e-5,
+) -> npt.NDArray[np.float32]:
+    """
+    Compute a mel spectrogram in pure NumPy, matching the torch/torchaudio pipeline.
+
+    This mirrors:
+      - librosa.filters.mel(..., norm="slaney")
+      - Hann window STFT
+      - power-magnitude spectrogram
+      - log compression with clipping
+
+    Args:
+        y: Waveform with shape [1, T].
+        n_fft: FFT size.
+        num_mels: Number of mel bins.
+        sampling_rate: Sampling rate in Hz.
+        hop_size: Hop size between frames.
+        win_size: Window size.
+        fmin: Minimum mel frequency in Hz.
+        fmax: Maximum mel frequency in Hz. If None, defaults to sr / 2.
+        center: Whether to pad the signal before framing.
+        clip_val: Minimum value before log compression.
+
+    Returns:
+        Mel spectrogram with shape [1, num_mels, frames].
+    """
+    if y.ndim == 1:
+        y = np.expand_dims(y, axis=0)
+    elif y.ndim == 2 and y.shape[0] != 1:
+        raise ValueError("Expected waveform shape [1, T].")
+    elif y.ndim > 2:
+        raise ValueError("Expected waveform ndim <= 2.")
+
+    if np.min(y) < -1.0:
+        pass
+    if np.max(y) > 1.0:
+        pass
+
+    mel_basis = librosa_style_mel_filterbank(
+        sr=sampling_rate,
+        n_fft=n_fft,
+        n_mels=num_mels,
+        fmin=float(fmin),
+        fmax=float(fmax) if fmax is not None else None,
+        norm="slaney",
+    )  # [num_mels, n_fft//2 + 1]
+
+    spec = _stft_magnitude(
+        y,
+        n_fft=n_fft,
+        hop_size=hop_size,
+        win_size=win_size,
+        center=center,
+    )  # [1, frames, freq]
+
+    mel_spec = np.matmul(mel_basis[None, :, :], np.transpose(spec, (0, 2, 1)))
+    mel_spec = np.transpose(mel_spec, (0, 1, 2))  # [1, num_mels, frames]
+
+    mel_spec = np.log(np.clip(mel_spec, a_min=clip_val, a_max=None)).astype(np.float32)
+    return mel_spec.transpose(0, 2, 1)  # B x T x n_mels

test_qwen3-tts-streaming_onnx.py

@@ -0,0 +1,322 @@
+# Copyright 2026 Patrick Lumbantobing, Vertox-AI
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+End-to-end streaming TTS test script using ONNX Runtime.
+This script demonstrates the full Qwen3-TTS-Realtime ONNX pipeline by:
+1. Loading six ONNX models (talker LLM, local talker transformer, codec decoder,
+   speaker encoder, talker codec embedding, text embedding projection)
+   into ONNX Runtime ``InferenceSession`` instances.
+2. Encoding a reference audio prompt for voice cloning.
+3. Simulating a streaming LLM text source (character-by-character deltas).
+4. Running the streaming TTS pipeline to produce audio chunks.
+5. Writing the concatenated audio to a WAV file.
+Usage:
+    python test_qwen3-tts-streaming_onnx.py \
+        --talker_model_path qwen3-tts_onnx/talker_model.onnx \
+        --talker_local_model_path qwen3-tts_onnx/talker_local_model.onnx \
+        --codec_decoder_model_path qwen3-tts_onnx/codec_decoder_model.onnx \
+        --speaker_encoder_model_path qwen3-tts_onnx/speaker_encoder_model.onnx \
+        --talker_codec_embed_model_path qwen3-tts_onnx/talker_codec_embed_model.onnx \
+        --text_embed_proj_model_path qwen3-tts_onnx/text_embed_proj_model.onnx \
+        --model_config_path configs/config.json \
+        --codec_config_path configs/tokenizer_config.json \
+        --backbone_config_path configs/config_backbone.json \
+        --preprocessor_config_dir configs/ \
+        --temperature 0.85 \
+        --top_p 0.8 \
+        --top_k 50 \
+        --repetition_penalty 1.9 \
+        --repetition_window 50 \
+        --num_threads 4 \
+        --chunk_frames 4 \
+        --prompt_wav audio_ref/speaker.[wav|flac|mp3] \
+        --out_wav output.wav \
+        --text "Text to be synthesized" \
+        --language "english"
+"""
+
+import argparse
+import logging
+import time
+import wave
+from pathlib import Path
+from typing import Iterator
+
+import numpy as np
+
+from src.inference import Qwen3TTSInferencerONNX
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+
+DEFAULT_TALKER_MODEL_PATH = "qwen3-tts_onnx/talker_model.onnx"
+DEFAULT_TALKER_LOCAL_MODEL_PATH = "qwen3-tts_onnx/talker_local_model.onnx"
+DEFAULT_CODEC_DECODER_MODEL_PATH = "qwen3-tts_onnx/codec_decoder_model.onnx"
+DEFAULT_SPEAKER_ENCODER_MODEL_PATH = "qwen3-tts_onnx/speaker_encoder_model.onnx"
+DEFAULT_TALKER_CODEC_EMBED_MODEL_PATH = "qwen3-tts_onnx/talker_codec_embed_model.onnx"
+DEFAULT_TEXT_EMBED_PROJ_MODEL_PATH = "qwen3-tts_onnx/text_embed_proj_model.onnx"
+
+DEFAULT_PREPROCESSOR_CONFIG_DIR = "configs/"
+DEFAULT_MODEL_CONFIG_PATH = "configs/config.json"
+DEFAULT_CODEC_CONFIG_PATH = "configs/speech_tokenizer_config.json"
+
+# DEFAULT_AUDIO_REF_PATH = "audio_ref/male_stewie.mp3"
+# DEFAULT_AUDIO_REF_PATH = "audio_ref/male_petergriffin.wav"
+# DEFAULT_AUDIO_REF_PATH = "audio_ref/male_old_movie.flac"
+DEFAULT_AUDIO_REF_PATH = "audio_ref/female_shadowheart.flac"
+# DEFAULT_AUDIO_REF_PATH = "audio_ref/david-attenborough.mp3"
+# DEFAULT_AUDIO_REF_PATH = "audio_ref/rick-sanchez.mp3"
+
+DEFAULT_OUT_WAV_DIR = "audio_synth/"
+# DEFAULT_OUT_WAV_DIR = "/mnt/d/vertox/Qwen3-TTS/audio_synth/"
+
+# DEFAULT_LANGUAGE = "english"
+DEFAULT_LANGUAGE = "russian"
+
+DEFAULT_TEMPERATURE = 0.85
+DEFAULT_TOP_P = 0.8
+DEFAULT_TOP_K = 50
+DEFAULT_REPETITION_PENALTY = 1.9
+DEFAULT_REPETITION_WINDOW = 50
+
+# DEFAULT_TEXT = "A B"
+# DEFAULT_TEXT = "Один, два"
+# DEFAULT_TEXT = "Test 1 2."
+# DEFAULT_TEXT = "Depending on the time, not only accuracy but also low-latency is important."
+# DEFAULT_TEXT = "Depending on the time, not only accuracy but also low-latency is important. If it is not instant, then the human interaction is lost. We are finally reaching a moment where the technology is fast enough for people to simply communicate, and this is a huge shift for global business."
+# DEFAULT_TEXT="в зависимости от времени не только точность, но и низкая задержка. Если это не мгновенно, то человеческое взаимодействие теряется. Мы наконец-то достигаем момента, когда технология достаточно быстра для того, чтобы люди просто общались, и это является огромным сдвигом для глобального бизнеса."
+DEFAULT_TEXT = "в зависимости от времени не только точнос��ь, но и низкая задержка."
+# DEFAULT_TEXT = "в зависимости от времени не только точность, но и низкая задержка. Если это не мгновенно, то человеческое взаимодействие теряется."
+# DEFAULT_TEXT = "В зависимости от времени, важна не только точность, но и низкая задержка. Если это не происходит мгновенно, человеческое взаимодействие утрачивается. Мы наконец подходим к тому моменту, когда технологии становятся достаточно быстрыми. для того чтобы люди могли просто общаться — и это огромный сдвиг для мирового бизнеса."
+
+
+def fake_llm_text_stream(
+    text: str,
+    chunk_chars: int = 1,
+    delay_s: float = 0.0,
+) -> Iterator[str]:
+    """
+    Simulate streaming text deltas from an LLM.
+    Each iteration yields `chunk_chars` characters with a delay of `delay_s` seconds.
+    In real-world usage, this can be replaced with streaming responses from models such as OpenAI or vLLM.
+    """
+    if not text:
+        return
+    step = max(1, chunk_chars)
+    for idx in range(0, len(text), step):
+        if delay_s > 0 and idx > 0:
+            time.sleep(delay_s)
+        yield text[idx : idx + step]
+
+
+def write_wav(out_path: Path, sample_rate: int, chunks: Iterator[np.ndarray]) -> None:
+    all_chunks: list[np.ndarray] = []
+    for chunk in chunks:
+        all_chunks.append(chunk.astype(np.float32).reshape(-1))
+
+    if not all_chunks:
+        raise RuntimeError("No audio chunks produced.")
+
+    audio = np.concatenate(all_chunks)
+    # float32 → int16 PCM
+    audio = np.clip(audio, -1.0, 1.0)
+    pcm16 = (audio * 32767.0).astype(np.int16)
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    with wave.open(str(out_path), "wb") as wf:
+        wf.setnchannels(1)
+        wf.setsampwidth(2)
+        wf.setframerate(int(sample_rate))
+        wf.writeframes(pcm16.tobytes())
+
+
+def decode_audio_frames(
+    audio_frames: list[np.ndarray],
+    inferencer: Qwen3TTSInferencerONNX,
+) -> Iterator[np.ndarray]:
+    for frame in audio_frames:
+        tokens = frame
+        if tokens.ndim == 3:
+            tokens = tokens[0]
+        if tokens.ndim != 2:
+            raise ValueError(f"Expected [T, C] audio tokens, got {tuple(tokens.shape)}")
+        logging.info(f"tokens {tokens} {tokens.shape}")
+        if tokens.size == 0:
+            continue
+        inferencer.push_tokens(tokens)
+        for wav in inferencer.audio_chunks():
+            if wav.size == 0:
+                continue
+            logging.info(f"decode_audio_frames wav {wav} {wav.shape}")
+            yield wav.reshape(-1)
+
+
+def flush_decoder(inferencer: Qwen3TTSInferencerONNX) -> Iterator[np.ndarray]:
+    final_chunk = inferencer.flush()
+    if final_chunk is not None and final_chunk.size > 0:
+        logging.info(f"final_chunk flush {final_chunk} {final_chunk.shape}")
+        yield final_chunk.reshape(-1)
+
+
+# Core: Streaming generation: text delta → push_text → audio
+def run_streaming_tts(
+    inferencer: Qwen3TTSInferencerONNX,
+    text_deltas: Iterator[str],
+) -> Iterator[np.ndarray]:
+    """
+    Receives streaming text deltas, feeds them into the TTS via `session.push_text()`,
+    and produces playable WAV chunks in real time.
+
+    The pipeline matches the Gradio demo:
+    codec.streaming → push_text → decode_frames → end_text → drain → flush
+
+    Args:
+        session: A streaming session that has been initialized and `reset_turn` has been called.
+        codec: The audio codec (used for streaming context).
+        decoder: An `AudioStreamDecoder` instance.
+        text_deltas: An iterator of text deltas (simulating LLM streaming output).
+    """
+    for delta in text_deltas:
+        logging.info(f"delta {delta}")
+        audio_frames = inferencer.push_text(delta)
+        if len(audio_frames) > 0:
+            logging.info(f"audio_frames {audio_frames} {len(audio_frames)}")
+            for audio_frame in audio_frames:
+                logging.info(f"audio_frame {audio_frame} {audio_frame.shape}")
+        yield from decode_audio_frames(audio_frames, inferencer)
+
+    audio_frames = inferencer.end_text()
+    if len(audio_frames) > 0:
+        logging.info(f"audio_frames end_text {audio_frames} {len(audio_frames)}")
+        for audio_frame in audio_frames:
+            logging.info(f"audio_frame end_text {audio_frame} {audio_frame.shape}")
+    yield from decode_audio_frames(audio_frames, inferencer)
+
+    while True:
+        audio_frames = inferencer.drain(max_steps=1)
+        if len(audio_frames) > 0:
+            logging.info(f"audio_frames drain {audio_frames} {len(audio_frames)}")
+            for audio_frame in audio_frames:
+                logging.info(f"audio_frame drain {audio_frame} {audio_frame.shape}")
+        if not audio_frames:
+            break
+        yield from decode_audio_frames(audio_frames, inferencer)
+        if inferencer.is_finished:
+            break
+
+    yield from flush_decoder(inferencer)
+
+
+def main():
+    p = argparse.ArgumentParser(description="Simulated LLM streaming text → TTS streaming audio。")
+    p.add_argument("--talker_model_path", type=str, default=DEFAULT_TALKER_MODEL_PATH)
+    p.add_argument("--talker_local_model_path", type=str, default=DEFAULT_TALKER_LOCAL_MODEL_PATH)
+    p.add_argument("--codec_decoder_model_path", type=str, default=DEFAULT_CODEC_DECODER_MODEL_PATH)
+    p.add_argument("--speaker_encoder_model_path", type=str, default=DEFAULT_SPEAKER_ENCODER_MODEL_PATH)
+    p.add_argument("--talker_codec_embed_model_path", type=str, default=DEFAULT_TALKER_CODEC_EMBED_MODEL_PATH)
+    p.add_argument("--text_embed_proj_model_path", type=str, default=DEFAULT_TEXT_EMBED_PROJ_MODEL_PATH)
+    p.add_argument("--model_config_path", type=str, default=DEFAULT_MODEL_CONFIG_PATH)
+    p.add_argument("--codec_config_path", type=str, default=DEFAULT_CODEC_CONFIG_PATH)
+    p.add_argument("--preprocessor_config_dir", type=str, default=DEFAULT_PREPROCESSOR_CONFIG_DIR)
+    p.add_argument("--temperature", type=float, default=DEFAULT_TEMPERATURE)
+    p.add_argument("--top_p", type=float, default=DEFAULT_TOP_P)
+    p.add_argument("--top_k", type=int, default=DEFAULT_TOP_K)
+    p.add_argument("--repetition_penalty", type=float, default=DEFAULT_REPETITION_PENALTY)
+    p.add_argument("--repetition_window", type=int, default=DEFAULT_REPETITION_WINDOW)
+    # 模拟 LLM streaming 参数
+    p.add_argument(
+        "--delta_chunk_chars", type=int, default=1, help="Number of characters to output at each delta (1 = verbatim)"
+    )
+    p.add_argument(
+        "--delta_delay_s", type=float, default=0.0, help="Simulated delay in seconds between deltas, let 0 = no delay"
+    )
+    p.add_argument("--num_threads", type=int, default=4, help="Number of threads used for sess.intra_num_op_threads")
+    p.add_argument(
+        "--chunk_frames",
+        type=int,
+        default=4,
+        help="Number of chunk frames for codec decoder forward [default: 4 frames (0.32 s)]",
+    )
+    p.add_argument("--prompt_wav", type=str, default=DEFAULT_AUDIO_REF_PATH)
+    p.add_argument("--out_wav", type=str, default=None)
+    p.add_argument(
+        "--text",
+        type=str,
+        default=DEFAULT_TEXT,
+    )
+    p.add_argument(
+        "--language",
+        type=str,
+        default=DEFAULT_LANGUAGE,
+    )
+
+    args = p.parse_args()
+    inferencer = Qwen3TTSInferencerONNX(
+        talker_model_path=args.talker_model_path,
+        talker_local_model_path=args.talker_local_model_path,
+        codec_decoder_model_path=args.codec_decoder_model_path,
+        speaker_encoder_model_path=args.speaker_encoder_model_path,
+        talker_codec_embed_model_path=args.talker_codec_embed_model_path,
+        text_embed_proj_model_path=args.text_embed_proj_model_path,
+        preprocessor_config_dir=args.preprocessor_config_dir,
+        model_config_path=args.model_config_path,
+        codec_config_path=args.codec_config_path,
+        audio_ref_path=args.prompt_wav,
+        language=args.language,
+        num_threads=args.num_threads,
+        chunk_frames=args.chunk_frames,
+        temperature=args.temperature,
+        top_p=args.top_p,
+        top_k=args.top_k,
+        repetition_penalty=args.repetition_penalty,
+        repetition_window=args.repetition_window,
+    )
+    logging.info("Inferencer loaded.")
+    logging.info(inferencer)
+
+    inferencer.reset_turn(reset_cache=True)
+    logging.info("State initialized.")
+
+    text_deltas = fake_llm_text_stream(
+        args.text,
+        chunk_chars=args.delta_chunk_chars,
+        delay_s=args.delta_delay_s,
+    )
+
+    logging.info("Running streaming tts simulation...")
+    wav_chunks = run_streaming_tts(
+        inferencer=inferencer,
+        text_deltas=text_deltas,
+    )
+    logging.info("Done.")
+
+    if args.out_wav is None:
+        out_wav_dir = Path(DEFAULT_OUT_WAV_DIR).expanduser()
+        out_wav_dir.mkdir(parents=True, exist_ok=True)
+        out_wav_path = out_wav_dir / f"output_{time.time()}.wav"
+    else:
+        out_wav_path = Path(args.out_wav).expanduser()
+        out_wav_path.parent.mkdir(parents=True, exist_ok=True)
+
+    write_wav(out_wav_path, inferencer.output_sample_rate, wav_chunks)
+    # This write_wav should be substituted with streaming playback device
+    logging.info(f"\n[OK] Write complete: {out_wav_path}")
+
+
+if __name__ == "__main__":
+    main()