test

.gitattributes

@@ -33,5 +33,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
-tokenizer.json filter=lfs diff=lfs merge=lfs -text
-assets/moss_tts_family.jpeg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,654 +0,0 @@
----
-license: apache-2.0
-tags:
-- text-to-speech
-language:
-- zh
-- en
-- de
-- es
-- fr
-- ja
-- it
-- he
-- ko
-- ru
-- fa
-- ar
-- pl
-- pt
-- cs
-- da
-- sv
-- hu
-- el
-- tr
----
-# MOSS-TTS Family
-
-<br>
-
-<p align="center">
-  &nbsp;&nbsp;&nbsp;&nbsp;
-  <img src="https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_imgaes_demo/openmoss_x_mosi" height="50" align="middle" />
-</p>
-
-
-
-<div align="center">
-  <a href="https://github.com/OpenMOSS/MOSS-TTS/tree/main"><img src="https://img.shields.io/badge/Project%20Page-GitHub-blue"></a>
-  <a href="https://modelscope.cn/collections/OpenMOSS-Team/MOSS-TTS"><img src="https://img.shields.io/badge/ModelScope-Models-lightgrey?logo=modelscope&amp"></a>
-  <a href="https://mosi.cn/#models"><img src="https://img.shields.io/badge/Blog-View-blue?logo=internet-explorer&amp"></a>
-  <a href="https://github.com/OpenMOSS/MOSS-TTS"><img src="https://img.shields.io/badge/Arxiv-Coming%20soon-red?logo=arxiv&amp"></a>
-
-  <a href="https://studio.mosi.cn"><img src="https://img.shields.io/badge/AIStudio-Try-green?logo=internet-explorer&amp"></a>
-  <a href="https://studio.mosi.cn/docs/moss-tts"><img src="https://img.shields.io/badge/API-Docs-00A3FF?logo=fastapi&amp"></a>
-  <a href="https://x.com/Open_MOSS"><img src="https://img.shields.io/badge/Twitter-Follow-black?logo=x&amp"></a>
-  <a href="https://discord.gg/fvm5TaWjU3"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&amp"></a>
-</div>
-
-## Overview
-MOSS‑TTS Family is an open‑source **speech and sound generation model family** from [MOSI.AI](https://mosi.cn/#hero) and the [OpenMOSS team](https://www.open-moss.com/). It is designed for **high‑fidelity**, **high‑expressiveness**, and **complex real‑world scenarios**, covering stable long‑form speech, multi‑speaker dialogue, voice/character design, environmental sound effects, and real‑time streaming TTS.
-
-
-## Introduction
-
-<p align="center">
-  <img src="https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_imgaes_demo/moss_tts_family_arch.jpeg" width="85%" />
-</p>
-
-
-When a single piece of audio needs to **sound like a real person**, **pronounce every word accurately**, **switch speaking styles across content**, **remain stable over tens of minutes**, and **support dialogue, role‑play, and real‑time interaction**, a single TTS model is often not enough. The **MOSS‑TTS Family** breaks the workflow into five production‑ready models that can be used independently or composed into a complete pipeline.
-
-- **MOSS‑TTS**: MOSS-TTS is the flagship production TTS foundation model, centered on high-fidelity zero-shot voice cloning with controllable long-form synthesis, pronunciation, and multilingual/code-switched speech. It serves as the core engine for scalable narration, dubbing, and voice-driven products.
-- **MOSS‑TTSD**: MOSS-TTSD is a production long-form dialogue model for expressive multi-speaker conversational audio at scale. It supports long-duration continuity, turn-taking control, and zero-shot voice cloning from short references for podcasts, audiobooks, commentary, dubbing, and entertainment dialogue.
-- **MOSS‑VoiceGenerator**: MOSS-VoiceGenerator is an open-source voice design model that creates speaker timbres directly from free-form text, without reference audio. It unifies timbre design, style control, and content synthesis, and can be used standalone or as a voice-design layer for downstream TTS.
-- **MOSS‑SoundEffect**: MOSS-SoundEffect is a high-fidelity text-to-sound model with broad category coverage and controllable duration for real content production. It generates stable audio from prompts across ambience, urban scenes, creatures, human actions, and music-like clips for film, games, interactive media, and data synthesis.
-- **MOSS‑TTS‑Realtime**: MOSS-TTS-Realtime is a context-aware, multi-turn streaming TTS model for real-time voice agents. By conditioning on dialogue history across both text and prior user acoustics, it delivers low-latency synthesis with coherent, consistent voice responses across turns.
-
-
-## Released Models
-
-| Model | Architecture | Size | Model Card | Hugging Face |
-|---|---|---:|---|---|
-| **MOSS-TTS** | MossTTSDelay | 8B | [moss_tts_model_card.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/docs/moss_tts_model_card.md) | 🤗 [Huggingface](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) |
-|  | MossTTSLocal | 1.7B | [moss_tts_model_card.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/docs/moss_tts_model_card.md) | 🤗 [Huggingface](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Local-Transformer) |
-| **MOSS‑TTSD‑V1.0** | MossTTSDelay | 8B | [moss_ttsd_model_card.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/docs/moss_ttsd_model_card.md) | 🤗 [Huggingface](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) |
-| **MOSS‑VoiceGenerator** | MossTTSDelay | 1.7B | [moss_voice_generator_model_card.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/docs/moss_voice_generator_model_card.md) | ���� [Huggingface](https://huggingface.co/OpenMOSS-Team/MOSS-Voice-Generator) |
-| **MOSS‑SoundEffect** | MossTTSDelay | 8B | [moss_sound_effect_model_card.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/docs/moss_sound_effect_model_card.md) | 🤗 [Huggingface](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) |
-| **MOSS‑TTS‑Realtime** | MossTTSRealtime | 1.7B | [moss_tts_realtime_model_card.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/docs/moss_tts_realtime_model_card.md) | 🤗 [Huggingface](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Realtime) |
-
-
-
-# MOSS-TTS
-## 1. Overview
-### 1.1 TTS Family Positioning
-MOSS-TTS is the **flagship base model** in our open-source **TTS Family**. It is designed as a production-ready synthesis backbone that can serve as the primary high-quality engine for scalable voice applications, and as a strong research baseline for controllable TTS and discrete audio token modeling.
-
-**Design goals**
-- **Production readiness**: robust voice cloning with stable, on-brand speaker identity at scale
-- **Controllability**: duration and pronunciation controls that integrate into real workflows
-- **Long-form stability**: consistent identity and delivery for extended narration
-- **Multilingual coverage**: multilingual and code-switched synthesis as first-class capabilities
-
-
-
-### 1.2 Key Capabilities
-
-MOSS-TTS delivers state-of-the-art quality while providing the fine-grained controllability and long-form stability required for production-grade voice applications, from zero-shot cloning and hour-long narration to token- and phoneme-level control across multilingual and code-switched speech.
-
-* **State-of-the-art evaluation performance** — top-tier objective and subjective results across standard TTS benchmarks and in-house human preference testing, validating both fidelity and naturalness.
-* **Zero-shot Voice Cloning (Voice Clone)** — clone a target speaker’s timbre (and part of speaking style) from short reference audio, without speaker-specific fine-tuning.
-* **Ultra-long Speech Generation (up to 1 hour)** — support continuous long-form speech generation for up to one hour in a single run, designed for extended narration and long-session content creation.
-* **Token-level Duration Control** — control pacing, rhythm, pauses, and speaking rate at token resolution for precise alignment and expressive delivery.
-* **Phoneme-level Pronunciation Control** — supports:
-
-  * pure **Pinyin** input
-  * pure **IPA** phoneme input
-  * mixed **Chinese / English / Pinyin / IPA** input in any combination
-* **Multilingual support** — high-quality multilingual synthesis with robust generalization across languages and accents.
-* **Code-switching** — natural mixed-language generation within a single utterance (e.g., Chinese–English), with smooth transitions, consistent speaker identity, and pronunciation-aware rendering on both sides of the switch.
-
-
-
-### 1.3 Model Architecture
-
-MOSS-TTS includes **two complementary architectures**, both trained and released to explore different performance/latency tradeoffs and to support downstream research.
-
-**Architecture A: Delay Pattern (MossTTSDelay)**
-- Single Transformer backbone with **(n_vq + 1) heads**.
-- Uses **delay scheduling** for multi-codebook audio tokens.
-- Strong long-context stability, efficient inference, and production-friendly behavior.
-
-**Architecture B: Global Latent + Local Transformer (MossTTSLocal)**
-- Backbone produces a **global latent** per time step.
-- A lightweight **Local Transformer** emits a token block per step.
-- **Streaming-friendly** with simpler alignment (no delay scheduling).
-
-**Why train both?**
-- **Exploration of architectural potential** and validation across multiple generation paradigms.
-- **Different tradeoffs**: Delay pattern tends to be faster and more stable for long-form synthesis; Local is smaller and excels on objective benchmarks.
-- **Open-source value**: two strong baselines for research, ablation, and downstream innovation.
-
-For full details, see:
-- **[moss_tts_delay/README.md](https://github.com/OpenMOSS/MOSS-TTS/blob/main/moss_tts_delay/README.md)**
-- **[moss_tts_local/README.md](https://github.com/OpenMOSS/MOSS-TTS/tree/main/moss_tts_local)**
-
-
-
-### 1.4 Released Models
-
-| Model | Description |
-|---|---|
-| **MossTTSDelay-8B** | **Recommended for production**. Faster inference, stronger long-context stability, and robust voice cloning quality. Best for large-scale deployment and long-form narration. |
-| **MossTTSLocal-1.7B** | **Recommended for evaluation and research**. Smaller model size with SOTA objective metrics. Great for quick experiments, ablations, and academic studies. |
-
-**Recommended decoding hyperparameters (per model)**
-
-| Model | audio_temperature | audio_top_p | audio_top_k | audio_repetition_penalty |
-|---|---:|---:|---:|---:|
-| **MOSS-TTSDelay-8B** | 1.7 | 0.8 | 25 | 1.0 |
-| **MOSS-TTSLocal-1.7B** | 1.0 | 0.95 | 50 | 1.1 |
-
-> Note: `max_new_tokens` controls duration. At 12.5 tokens per second, **1s ≈ 12.5 tokens**.
-
-
-
-## 2. Quick Start
-
-
-
-### Environment Setup
-
-We recommend a clean, isolated Python environment with **Transformers 5.0.0** to avoid dependency conflicts.
-
-```bash
-conda create -n moss-tts python=3.12 -y
-conda activate moss-tts
-```
-
-Install all required dependencies:
-
-```bash
-git clone https://github.com/OpenMOSS/MOSS-TTS.git
-cd MOSS-TTS
-pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e .
-```
-
-#### (Optional) Install FlashAttention 2
-
-For better speed and lower GPU memory usage, you can install FlashAttention 2 if your hardware supports it.
-
-```bash
-pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]"
-```
-
-If your machine has limited RAM and many CPU cores, you can cap build parallelism:
-
-```bash
-MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]"
-```
-
-Notes:
-- Dependencies are managed in `pyproject.toml`, which currently pins `torch==2.9.1+cu128` and `torchaudio==2.9.1+cu128`.
-- If FlashAttention 2 fails to build on your machine, you can skip it and use the default attention backend.
-- FlashAttention 2 is only available on supported GPUs and is typically used with `torch.float16` or `torch.bfloat16`.
-
-
-### Basic Usage
-
-
-> Tip: For evaluation and research purposes, we recommend using **MOSS-TTSLocal-1.7B**. 
-
-MOSS-TTS provides a convenient `generate` interface for rapid usage. The examples below cover:
-1. Direct generation (Chinese / English / Pinyin / IPA)
-2. Voice cloning
-3. Duration control
-
-```python
-import importlib.util
-from pathlib import Path
-import torch
-import torchaudio
-from transformers import AutoModel, AutoProcessor, GenerationConfig
-# Disable the broken cuDNN SDPA backend
-torch.backends.cuda.enable_cudnn_sdp(False)
-# Keep these enabled as fallbacks
-torch.backends.cuda.enable_flash_sdp(True)
-torch.backends.cuda.enable_mem_efficient_sdp(True)
-torch.backends.cuda.enable_math_sdp(True)
-
-class DelayGenerationConfig(GenerationConfig):
-    def __init__(self, **kwargs):
-        super().__init__(**kwargs)
-        self.layers = kwargs.get("layers", [{} for _ in range(32)])
-        self.do_samples = kwargs.get("do_samples", None)
-        self.n_vq_for_inference = 32
-        
-def initial_config(tokenizer, model_name_or_path):
-    generation_config = DelayGenerationConfig.from_pretrained(model_name_or_path)
-    generation_config.pad_token_id = tokenizer.pad_token_id
-    generation_config.eos_token_id = 151653
-    generation_config.max_new_tokens = 1000000
-    generation_config.temperature = 1.0
-    generation_config.top_p = 0.95
-    generation_config.top_k = 100
-    generation_config.repetition_penalty = 1.1
-    generation_config.use_cache = True
-    generation_config.do_sample = False
-    return generation_config
-
-
-pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS-Local-Transformer"
-device = "cuda" if torch.cuda.is_available() else "cpu"
-dtype = torch.bfloat16 if device == "cuda" else torch.float32
-
-def resolve_attn_implementation() -> str:
-    # Prefer FlashAttention 2 when package + device conditions are met.
-    if (
-        device == "cuda"
-        and importlib.util.find_spec("flash_attn") is not None
-        and dtype in {torch.float16, torch.bfloat16}
-    ):
-        major, _ = torch.cuda.get_device_capability()
-        if major >= 8:
-            return "flash_attention_2"
-
-    # CUDA fallback: use PyTorch SDPA kernels.
-    if device == "cuda":
-        return "sdpa"
-
-    # CPU fallback.
-    return "eager"
-
-
-attn_implementation = resolve_attn_implementation()
-print(f"[INFO] Using attn_implementation={attn_implementation}")
-
-processor = AutoProcessor.from_pretrained(
-    pretrained_model_name_or_path,
-    trust_remote_code=True,
-)
-processor.audio_tokenizer = processor.audio_tokenizer.to(device)
-
-text_1 = """亲爱的你，
-你好呀。
-
-今天，我想用最认真、最温柔的声音，对你说一些重要的话。
-这些话，像一颗小小的星星，希望能在你的心里慢慢发光。
-
-首先，我想祝你——
-每天都能平平安安、快快乐乐。
-
-希望你早上醒来的时候，
-窗外有光，屋子里很安静，
-你的心是轻轻的，没有着急，也没有害怕。
-"""
-text_2 = """We stand on the threshold of the AI era.
-Artificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."""
-text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"
-text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"
-text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"
-text_6 = "/həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/"
-
-ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
-ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"
-
-conversations = [
-    # Direct TTS (no reference)
-    [
-        processor.build_user_message(text=text_1)
-    ],
-    [
-        processor.build_user_message(text=text_2)
-    ],
-    # Pinyin or IPA input
-    [
-        processor.build_user_message(text=text_3)
-    ],
-    [
-        processor.build_user_message(text=text_4)
-    ],
-    [
-        processor.build_user_message(text=text_5)
-    ],
-    [
-        processor.build_user_message(text=text_6)
-    ],
-    # Voice cloning (with reference)
-    [
-        processor.build_user_message(text=text_1, reference=[ref_audio_1])
-    ],
-    [
-        processor.build_user_message(text=text_2, reference=[ref_audio_2])
-    ],
-]
-
-model = AutoModel.from_pretrained(
-    pretrained_model_name_or_path,
-    trust_remote_code=True,
-    attn_implementation=attn_implementation,
-    torch_dtype=dtype,
-).to(device)
-model.eval()
-
-generation_config = initial_config(processor.tokenizer, pretrained_model_name_or_path)
-generation_config.n_vq_for_inference = model.channels - 1
-generation_config.do_samples = [True] * model.channels
-generation_config.layers = [
-    {
-        "repetition_penalty": 1.0, 
-        "temperature": 1.5, 
-        "top_p": 1.0, 
-        "top_k": 50
-    }
-] + [ 
-    {
-        "repetition_penalty": 1.1, 
-        "temperature": 1.0, 
-        "top_p": 0.95,
-        "top_k": 50
-    }
-] * (model.channels - 1) 
-
-batch_size = 1
-
-save_dir = Path(f"inference_root_moss_tts_local_transformer_generation")
-save_dir.mkdir(exist_ok=True, parents=True)
-sample_idx = 0
-with torch.no_grad():
-    for start in range(0, len(conversations), batch_size):
-        batch_conversations = conversations[start : start + batch_size]
-        batch = processor(batch_conversations, mode="generation")
-        input_ids = batch["input_ids"].to(device)
-        attention_mask = batch["attention_mask"].to(device)
-
-        outputs = model.generate(
-            input_ids=input_ids,
-            attention_mask=attention_mask,
-            generation_config=generation_config
-        )
-
-        for message in processor.decode(outputs):
-            audio = message.audio_codes_list[0]
-            out_path = save_dir / f"sample{sample_idx}.wav"
-            sample_idx += 1
-            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
-
-```
-
-### Continuation + Voice Cloning (Prefix Audio + Text)
-
-MOSS-TTS supports continuation-based cloning: provide a prefix audio clip in the assistant message, and make sure the **prefix transcript** is included in the text. The model continues in the same speaker identity and style.
-
-```python
-import importlib.util
-from pathlib import Path
-import torch
-import torchaudio
-from transformers import AutoModel, AutoProcessor, GenerationConfig
-# Disable the broken cuDNN SDPA backend
-torch.backends.cuda.enable_cudnn_sdp(False)
-# Keep these enabled as fallbacks
-torch.backends.cuda.enable_flash_sdp(True)
-torch.backends.cuda.enable_mem_efficient_sdp(True)
-torch.backends.cuda.enable_math_sdp(True)
-
-class DelayGenerationConfig(GenerationConfig):
-    def __init__(self, **kwargs):
-        super().__init__(**kwargs)
-        self.layers = kwargs.get("layers", [{} for _ in range(32)])
-        self.do_samples = kwargs.get("do_samples", None)
-        self.n_vq_for_inference = 32
-        
-def initial_config(tokenizer, model_name_or_path):
-    generation_config = DelayGenerationConfig.from_pretrained(model_name_or_path)
-    generation_config.pad_token_id = tokenizer.pad_token_id
-    generation_config.eos_token_id = 151653
-    generation_config.max_new_tokens = 1000000
-    generation_config.temperature = 1.0
-    generation_config.top_p = 0.95
-    generation_config.top_k = 100
-    generation_config.repetition_penalty = 1.1
-    generation_config.use_cache = True
-    generation_config.do_sample = False
-    return generation_config
-
-
-pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS-Local-Transformer"
-device = "cuda" if torch.cuda.is_available() else "cpu"
-dtype = torch.bfloat16 if device == "cuda" else torch.float32
-
-def resolve_attn_implementation() -> str:
-    # Prefer FlashAttention 2 when package + device conditions are met.
-    if (
-        device == "cuda"
-        and importlib.util.find_spec("flash_attn") is not None
-        and dtype in {torch.float16, torch.bfloat16}
-    ):
-        major, _ = torch.cuda.get_device_capability()
-        if major >= 8:
-            return "flash_attention_2"
-
-    # CUDA fallback: use PyTorch SDPA kernels.
-    if device == "cuda":
-        return "sdpa"
-
-    # CPU fallback.
-    return "eager"
-
-
-attn_implementation = resolve_attn_implementation()
-print(f"[INFO] Using attn_implementation={attn_implementation}")
-
-processor = AutoProcessor.from_pretrained(
-    pretrained_model_name_or_path,
-    trust_remote_code=True,
-)
-processor.audio_tokenizer = processor.audio_tokenizer.to(device)
-
-text_1 = """亲爱的你，
-你好呀。
-
-今天，我想用最认真、最温柔的声音，对你说一些重要的话。
-这些话，像一颗小小的星星，希望能在你的心里慢慢发光。
-
-首先，我想祝你——
-每天都能平平安安、快快乐乐。
-
-希望你早上醒来的时候，
-窗外有光，屋子里很安静，
-你的心是轻轻的，没有着急，也没有害怕。
-"""
-
-ref_text_1 = "太阳系八大行星之一。"
-ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
-
-conversations = [
-    # Continuatoin only
-    [
-        processor.build_user_message(text=ref_text_1 + text_1),
-        processor.build_assistant_message(audio_codes_list=[ref_audio_1])
-    ],
-]
-
-model = AutoModel.from_pretrained(
-    pretrained_model_name_or_path,
-    trust_remote_code=True,
-    attn_implementation=attn_implementation,
-    torch_dtype=dtype,
-).to(device)
-model.eval()
-
-generation_config = initial_config(processor.tokenizer, pretrained_model_name_or_path)
-generation_config.n_vq_for_inference = model.channels - 1
-generation_config.do_samples = [True] * model.channels
-generation_config.layers = [
-    {
-        "repetition_penalty": 1.0, 
-        "temperature": 1.5, 
-        "top_p": 1.0, 
-        "top_k": 50
-    }
-] + [ 
-    {
-        "repetition_penalty": 1.1, 
-        "temperature": 1.0, 
-        "top_p": 0.95,
-        "top_k": 50
-    }
-] * (model.channels - 1) 
-
-batch_size = 1
-
-save_dir = Path("inference_root_moss_tts_local_transformer_continuation")
-save_dir.mkdir(exist_ok=True, parents=True)
-sample_idx = 0
-with torch.no_grad():
-    for start in range(0, len(conversations), batch_size):
-        batch_conversations = conversations[start : start + batch_size]
-        batch = processor(batch_conversations, mode="continuation")
-        input_ids = batch["input_ids"].to(device)
-        attention_mask = batch["attention_mask"].to(device)
-
-        outputs = model.generate(
-            input_ids=input_ids,
-            attention_mask=attention_mask,
-            generation_config=generation_config
-        )
-
-        for message in processor.decode(outputs):
-            audio = message.audio_codes_list[0]
-            out_path = save_dir / f"sample{sample_idx}.wav"
-            sample_idx += 1
-            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
-
-```
-
-
-
-### Input Types
-
-**UserMessage**
-
-| Field | Type | Required | Description |
-|---|---|---:|---|
-| `text` | `str` | Yes | Text to synthesize. Supports Chinese, English, German, French, Spanish, Japanese, Korean, etc. Can mix raw text with Pinyin or IPA for pronunciation control. |
-| `reference` | `List[str]` | No | Reference audio for voice cloning. For current MOSS-TTS, **one audio** is expected in the list. |
-| `tokens` | `int` | No | Expected number of audio tokens. **1s ≈ 12.5 tokens**. |
-
-**AssistantMessage**
-
-| Field | Type | Required | Description |
-|---|---|---:|---|
-| `audio_codes_list` | `List[str]` | Only for continuation | Prefix audio for continuation-based cloning. Use audio file paths or URLs. |
-
-
-
-### Generation Hyperparameters (MOSS-TTS-Local)
-
-MOSS-TTSLocal utilizes `DelayGenerationConfig` to manage hierarchical sampling. Due to the **Progressive Sequence Dropout** training mechanism, the model supports variable bitrate inference by adjusting the RVQ depth.
-
-| Parameter | Type | Recommended (Audio Layers) | Description |
-| :--- | :--- | :---: | :--- |
-| `max_new_tokens` | `int` | — | Controls total generated audio tokens. **1s ≈ 12.5 tokens**. |
-| `n_vq_for_inference` | `int` | 32 | **RVQ Inference Depth**: Controls the number of codebook layers generated. Higher values (max 32) improve audio fidelity but slow down inference; lower values speed up inference but reduce audio quality. |
-| `audio_temperature` | `float` | 1.0 | Temperature for audio token layers (Layer 1+). Lower values ensure more stable and consistent acoustic reconstruction. |
-| `audio_top_p` | `float` | 0.95 | Nucleus sampling cutoff for audio layers. |
-| `audio_top_k` | `int` | 50 | Top-K sampling filter for audio layers. |
-| `audio_repetition_penalty` | `float` | 1.1 | Discourages repeating acoustic patterns. Values > 1.0 help prevent artifacts in long-form synthesis. |
-
-
-
-### Pinyin Input
-
-Use tone-numbered Pinyin such as `ni3 hao3 wo3 men1`. You can convert Chinese text with [pypinyin](https://github.com/mozillazg/python-pinyin), then adjust tones for pronunciation control.
-
-```python
-import re
-from pypinyin import pinyin, Style
-
-CN_PUNCT = r"，。！？；：、（）“”‘’"
-
-
-def fix_punctuation_spacing(s: str) -> str:
-    s = re.sub(rf"\s+([{CN_PUNCT}])", r"\1", s)
-    s = re.sub(rf"([{CN_PUNCT}])\s+", r"\1", s)
-    return s
-
-
-def zh_to_pinyin_tone3(text: str, strict: bool = True) -> str:
-    result = pinyin(
-        text,
-        style=Style.TONE3,
-        heteronym=False,
-        strict=strict,
-        errors="default",
-    )
-
-    s = " ".join(item[0] for item in result)
-    return fix_punctuation_spacing(s)
-
-text = zh_to_pinyin_tone3("您好，请问您来自哪座城市？")
-print(text)
-
-# Expected: nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？
-# Try: nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？
-```
-
-
-
-### IPA Input
-
-Use `/.../` to wrap IPA sequences so they are distinct from normal text. You can use [DeepPhonemizer](https://github.com/spring-media/DeepPhonemizer) to convert English paragraphs or words into IPA sequences.
-
-```python
-from dp.phonemizer import Phonemizer
-
-# Download a phonemizer checkpoint from https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/DeepPhonemizer/en_us_cmudict_ipa_forward.pt
-model_path = "<path-to-phonemizer-checkpoint>"
-phonemizer = Phonemizer.from_checkpoint(model_path)
-
-english_texts = "Hello, may I ask which city you are from?"
-phoneme_outputs = phonemizer(
-    english_texts,
-    lang="en_us",
-    batch_size=8
-)
-model_input_text = f"/{phoneme_outputs}/"
-print(model_input_text)
-
-# Expected: /həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/
-```
-
-
-
-## 3. Evaluation
-MOSS-TTS achieved state-of-the-art results on the open-source zero-shot TTS benchmark Seed-TTS-eval, not only surpassing all open-source models but also rivaling the most powerful closed-source models.
-
-| Model | Params | Open-source | EN WER (%) ↓ | EN SIM (%) ↑ | ZH CER (%) ↓ | ZH SIM (%) ↑ |
-|---|---:|:---:|---:|---:|---:|---:|
-| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 |
-| FishAudio-S1 | 4B | ❌ | 1.72 | 62.57 | 1.22 | 72.1 |
-| Seed-TTS |  | ❌ | 2.25 | 76.2 | 1.12 | 79.6 |
-| MiniMax-Speech |  | ❌ | 1.65 | 69.2 | 0.83 | 78.3 |
-|  |  |  |  |  |  |  |
-| CosyVoice | 0.3B | ✅ | 4.29 | 60.9 | 3.63 | 72.3 |
-| CosyVoice2 | 0.5B | ✅ | 3.09 | 65.9 | 1.38 | 75.7 |
-| CosyVoice3 | 0.5B | ✅ | 2.02 | 71.8 | 1.16 | 78 |
-| CosyVoice3 | 1.5B | ✅ | 2.22 | 72 | 1.12 | 78.1 |
-| F5-TTS | 0.3B | ✅ | 2 | 67 | 1.53 | 76 |
-| SparkTTS | 0.5B | ✅ | 3.14 | 57.3 | 1.54 | 66 |
-| FireRedTTS | 0.5B | ✅ | 3.82 | 46 | 1.51 | 63.5 |
-| FireRedTTS-2 | 1.5B | ✅ | 1.95 | 66.5 | 1.14 | 73.6 |
-| Qwen2.5-Omni | 7B | ✅ | 2.72 | 63.2 | 1.7 | 75.2 |
-| FishAudio-S1-mini | 0.5B | ✅ | 1.94 | 55 | 1.18 | 68.5 |
-| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 |
-| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 |
-| HiggsAudio-v2 | 3B | ✅ | 2.44 | 67.7 | 1.5 | 74 |
-| VoxCPM | 0.5B | ✅ | 1.85 | 72.9 | **0.93** | 77.2 |
-| Qwen3-TTS | 0.6B | ✅ | 1.68 | 70.39 | 1.23 | 76.4 |
-| Qwen3-TTS | 1.7B | ✅ | **1.5** | 71.45 | 1.33 | 76.72 |
-|  |  |  |  |  |  |  |
-| MossTTSDelay | 8B | ✅ | 1.79 | 71.46 | 1.32 | 77.05 |
-| MossTTSLocal | 1.7B | ✅ | 1.85 | **73.42** | 1.2 | **78.82** |

added_tokens.json

@@ -1,28 +0,0 @@
-{
-  "</think>": 151668,
-  "</tool_call>": 151658,
-  "</tool_response>": 151666,
-  "<think>": 151667,
-  "<tool_call>": 151657,
-  "<tool_response>": 151665,
-  "<|audio_end|>": 151653,
-  "<|audio_pad|>": 151654,
-  "<|audio_start|>": 151652,
-  "<|box_end|>": 151649,
-  "<|box_start|>": 151648,
-  "<|endoftext|>": 151643,
-  "<|file_sep|>": 151664,
-  "<|fim_middle|>": 151660,
-  "<|fim_pad|>": 151662,
-  "<|fim_prefix|>": 151659,
-  "<|fim_suffix|>": 151661,
-  "<|im_end|>": 151645,
-  "<|im_start|>": 151644,
-  "<|image_pad|>": 151655,
-  "<|object_ref_end|>": 151647,
-  "<|object_ref_start|>": 151646,
-  "<|quad_end|>": 151651,
-  "<|quad_start|>": 151650,
-  "<|repo_name|>": 151663,
-  "<|video_pad|>": 151656
-}

chat_template.jinja

@@ -1,4 +0,0 @@
-{% for message in messages %}<|im_start|>{{ message['role'] }}
-{% if message['content'] is string %}{{ message['content'] }}{% else %}{% for content in message['content'] %}{% if content.get('type') == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}<|im_end|>
-{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
-{% endif %}

config.json

@@ -1,86 +0,0 @@
-{
-  "model_type": "moss_tts_delay",
-  "architectures": [
-    "MossTTSDelayModel"
-  ],
-  "auto_map": {
-      "AutoConfig": "configuration_moss_tts.MossTTSDelayConfig",
-      "AutoModel": "modeling_moss_tts.MossTTSDelayModel"
-  },
-  "dtype": "bfloat16",
-  "initializer_range": 0.02,
-  "language_config": {
-    "_name_or_path": "Qwen/Qwen3-8B",
-    "architectures": [
-      "Qwen3ForCausalLM"
-    ],
-    "attention_bias": false,
-    "attention_dropout": 0.0,
-    "bos_token_id": 151643,
-    "eos_token_id": 151645,
-    "pad_token_id": 151643,
-    "head_dim": 128,
-    "hidden_act": "silu",
-    "hidden_size": 2048,
-    "initializer_range": 0.02,
-    "intermediate_size": 6144,
-    "layer_types": [
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention",
-      "full_attention"
-    ],
-    "max_position_embeddings": 40960,
-    "max_window_layers": 28,
-    "model_type": "qwen3",
-    "num_attention_heads": 16,
-    "num_hidden_layers": 28,
-    "num_key_value_heads": 8,
-    "rms_norm_eps": 1e-06,
-    "rope_scaling": null,
-    "rope_theta": 1000000,
-    "sliding_window": null,
-    "use_cache": true,
-    "use_sliding_window": false,
-    "vocab_size": 155648
-  },
-  "n_vq": 32,
-  "audio_vocab_size": 1024,
-  "audio_user_slot_token_id": 151654,
-  "audio_assistant_gen_slot_token_id": 151656,
-  "audio_assistant_delay_slot_token_id": 151662,
-  "audio_start_token_id": 151652,
-  "audio_end_token_id": 151653,
-  "audio_pad_code": 1024,
-  "sampling_rate": 24000,
-  "transformers_version": "4.57.1",
-
-  "additional_mlp_ffn_hidden_size": 2048,
-  "local_ffn_hidden_size": 8960,
-  "local_hidden_size": 1536,
-  "local_num_layers": 4
-}

configuration_moss_tts.py

@@ -1,122 +0,0 @@
-# coding=utf-8
-# Copyright 2026 OpenMOSS and the HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-""" MossTTSDelay model configuration """
-
-from typing import Optional, Union
-from transformers.configuration_utils import PretrainedConfig
-from transformers.utils import logging
-from transformers.models.qwen3 import Qwen3Config
-
-logger = logging.get_logger(__name__)
-
-
-class MossTTSDelayConfig(PretrainedConfig):
-    r"""
-    This is the configuration class to store the configuration of a [`MossTTSDelayModel`]. It is used to instantiate an
-    MossTTSDelay model according to the specified arguments, defining the model architecture. Instantiating a configuration
-    with the defaults will yield a similar configuration to that of the MossTTSDelay [MossTTSDelay-8B](https://huggingface.co/OpenMOSS/mosstts-8b) architecture.
-
-    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
-    documentation from [`PretrainedConfig`] for more information.
-
-    Args:
-        language_config (`Union[Qwen3Config, dict]`, *optional*):
-            Configuration for the backbone language model (Qwen3).
-        initializer_range (`float`, *optional*, defaults to 0.02):
-            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
-        n_vq (`int`, *optional*, defaults to 32):
-            Number of additional VQ (Vector Quantization) heads/channels for audio. 
-            Determines the number of codebooks used in the audio representation.
-        audio_vocab_size (`int`, *optional*, defaults to 1024):
-            Vocabulary size for the audio tokens (codebooks 1 to N).
-        audio_user_slot_token_id (`int`, *optional*, defaults to 151654):
-            The specific token ID used as a placeholder/slot for user-side audio inputs in the prompt.
-        audio_assistant_gen_slot_token_id (`int`, *optional*, defaults to 151656):
-            The specific token ID representing the generation slot for the assistant's audio output. 
-            Acting as the trigger for the TTS generation process.
-        audio_assistant_delay_slot_token_id (`int`, *optional*, defaults to 151662):
-            The token ID used in the 'Delay Pattern' paradigm to represent the delayed/offset positions 
-            between different VQ channels.
-        audio_start_token_id (`int`, *optional*, defaults to 151652):
-            Special token ID used to denote the start of an audio sequence in the stream.
-        audio_end_token_id (`int`, *optional*, defaults to 151653):
-            Special token ID used to denote the end of an audio sequence (EOS for audio).
-        audio_pad_code (`int`, *optional*, defaults to 1024):
-            The padding value used within the audio VQ codebooks. Typically equals `audio_vocab_size`.
-    """
-    model_type = "moss_tts_delay"
-    keys_to_ignore_at_inference = ["past_key_values"]
-
-    def __init__(
-        self,
-        language_config: Optional[Union[Qwen3Config, dict]] = None,
-        initializer_range: float = 0.02,
-        n_vq: int = 32,
-        pad_token_id: int = 151643,
-        im_start_token_id: int = 151644,
-        im_end_token_id: int = 151645,
-        audio_vocab_size: int = 1024,
-        audio_user_slot_token_id: int = 151654,
-        audio_assistant_gen_slot_token_id: int = 151656,
-        audio_assistant_delay_slot_token_id: int = 151662,
-        audio_start_token_id: int = 151652,
-        audio_end_token_id: int = 151653,
-        audio_pad_code: int = 1024,
-        sampling_rate: int = 24000,
-        additional_mlp_ffn_hidden_size: int = 2048,
-        local_ffn_hidden_size: int = 8960,
-        local_hidden_size: int = 1536,
-        local_num_layers: int = 4,
-        **kwargs,
-    ):
-        if isinstance(language_config, dict):
-            self.language_config = Qwen3Config(**language_config)
-        elif language_config is None:
-            self.language_config = Qwen3Config()
-        else:
-            self.language_config = language_config
-
-        self.initializer_range = initializer_range
-        self.n_vq = n_vq
-        self.audio_vocab_size = audio_vocab_size
-        self.audio_user_slot_token_id = audio_user_slot_token_id
-        self.audio_assistant_gen_slot_token_id = audio_assistant_gen_slot_token_id
-        self.audio_assistant_delay_slot_token_id = audio_assistant_delay_slot_token_id
-        self.audio_start_token_id = audio_start_token_id
-        self.audio_end_token_id = audio_end_token_id
-        self.audio_pad_code = audio_pad_code
-        self.sampling_rate = sampling_rate
-
-        self.hidden_size = self.language_config.hidden_size
-        self.vocab_size = self.language_config.vocab_size
-        self.im_start_token_id = self.language_config
-        self.pad_token_id = pad_token_id
-        self.im_start_token_id = im_start_token_id
-        self.im_end_token_id = im_end_token_id
-        
-        self.additional_mlp_ffn_hidden_size = additional_mlp_ffn_hidden_size
-        self.local_ffn_hidden_size = local_ffn_hidden_size
-        self.local_hidden_size = local_hidden_size
-        self.local_num_layers = local_num_layers
-        
-        super().__init__(**kwargs)
-
-    def to_dict(self):
-        output = super().to_dict()
-        if hasattr(self.language_config, "to_dict"):
-            output["language_config"] = self.language_config.to_dict()
-        else:
-            output["language_config"] = self.language_config
-        return output

generation_config.json

@@ -1,6 +0,0 @@
-{
-  "_from_model_config": true,
-  "bos_token_id": 151643,
-  "eos_token_id": 151645,
-  "transformers_version": "4.51.3"
-}

inference_utils.py

@@ -1,154 +0,0 @@
-import torch
-import torchaudio
-import torch.nn.functional as F
-from typing import Optional, List, Tuple
-from tqdm import tqdm
-
-
-def apply_top_k(logits, top_k):
-    batch_size, vocab_size = logits.shape
-    top_k = min(top_k, vocab_size)
-    top_k_values, top_k_indices = torch.topk(logits, top_k, dim=-1)
-    filtered_logits = torch.full_like(logits, float("-inf"))
-    batch_indices = torch.arange(batch_size).unsqueeze(-1)
-    filtered_logits[batch_indices, top_k_indices] = top_k_values
-    return filtered_logits
-
-
-def apply_top_p(logits, top_p):
-    probs = F.softmax(logits, dim=-1)
-    sorted_probs, sorted_indices = torch.sort(probs, descending=True, dim=-1)
-    cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
-    sorted_indices_to_remove = cumulative_probs > top_p
-    sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
-    sorted_indices_to_remove[..., 0] = False
-    batch_size = logits.shape[0]
-    filtered_logits = logits.clone()
-    for i in range(batch_size):
-        indices_to_remove = sorted_indices[i][sorted_indices_to_remove[i]]
-        filtered_logits[i, indices_to_remove] = float("-inf")
-    return filtered_logits
-
-
-def apply_top_p_optimized(logits, top_p):
-    probs = F.softmax(logits, dim=-1)
-    sorted_probs, sorted_indices = torch.sort(probs, descending=True, dim=-1)
-
-    cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
-
-    sorted_indices_to_remove = cumulative_probs > top_p
-    sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
-    sorted_indices_to_remove[..., 0] = False
-
-    indices_to_remove = torch.zeros_like(logits, dtype=torch.bool).scatter_(
-        dim=-1, index=sorted_indices, src=sorted_indices_to_remove
-    )
-
-    logits[indices_to_remove] = float("-inf")
-    return logits
-
-
-def apply_repetition_penalty_delay_pattern(
-    logits: torch.Tensor,
-    prev_tokens: torch.LongTensor,
-    penalty: float,
-):
-    """
-    logits: [B, H, V]  or [N, V]
-    prev_tokens: [B, T, H] or [N, T] or [B, H]
-
-    Apply the repetition penalty independently for each H (VQ head).
-    """
-    if penalty == 1.0 or prev_tokens is None:
-        return logits
-
-    vocab_size = logits.size(-1)
-
-    # Case 1: regular [N, V] (text layer)
-    if logits.dim() == 2:
-        prev_tokens_flat = prev_tokens.reshape(-1)
-        unique_tokens = torch.unique(prev_tokens_flat)
-
-        token_logits = logits[:, unique_tokens]
-        pos_mask = token_logits > 0
-        token_logits[pos_mask] /= penalty
-        token_logits[~pos_mask] *= penalty
-        logits[:, unique_tokens] = token_logits
-        return logits
-
-    # Case 2: Delay Pattern audio [B, H, V]
-    assert logits.dim() == 3, "Delay Pattern audio logits must be [B, H, V]"
-    B, H, V = logits.shape
-
-    for h in range(H):
-        # prev_tokens_h: [B, T] or [B]
-        prev_tokens_h = prev_tokens[..., h].reshape(-1)
-        unique_tokens = torch.unique(prev_tokens_h)
-
-        if unique_tokens.numel() == 0:
-            continue
-
-        token_logits = logits[:, h, unique_tokens]
-        pos_mask = token_logits > 0
-        token_logits[pos_mask] /= penalty
-        token_logits[~pos_mask] *= penalty
-        logits[:, h, unique_tokens] = token_logits
-
-    return logits
-
-
-def sample_token(
-    logits,
-    prev_tokens: Optional[torch.LongTensor] = None,
-    repetition_penalty: float = 1.0,
-    top_p=None,
-    top_k=None,
-    do_sample=True,
-):
-    vocab_size = logits.size(-1)
-
-    # ===== Repetition Penalty (before reshaping!) =====
-    if prev_tokens is not None and repetition_penalty != 1.0:
-        logits = apply_repetition_penalty_delay_pattern(
-            logits,
-            prev_tokens,
-            repetition_penalty,
-        )
-
-    if not do_sample:
-        return torch.argmax(logits, dim=-1)
-
-    # ===== Only flatten after this, for top-k / top-p / multinomial =====
-    original_shape = logits.shape
-    reshaped_logits = logits.view(-1, vocab_size)
-
-    if top_k is not None and top_k > 0:
-        reshaped_logits = apply_top_k(reshaped_logits, top_k)
-
-    if top_p is not None and top_p < 1.0:
-        reshaped_logits = apply_top_p_optimized(reshaped_logits, top_p)
-
-    probs = F.softmax(reshaped_logits, dim=-1)
-    next_tokens = torch.multinomial(probs, num_samples=1)
-
-    return next_tokens.view(original_shape[:-1])
-
-
-def find_last_equal_C(tensor, C):
-    """
-    tensor: torch.Tensor of shape [batch_size, seq_len]
-    C: scalar value to match
-    Returns: torch.Tensor of shape [batch_size] with last indices
-    """
-    mask = (tensor == C).int()  # Shape: [batch_size, seq_len], bool tensor
-    flipped_mask = mask.flip(dims=[1])  # Flip along sequence dimension
-    flipped_indices = flipped_mask.argmax(dim=1)  # First True in flipped
-    seq_len = tensor.shape[1]
-    last_indices = (seq_len - 1) - flipped_indices  # Convert to original indices
-
-    # Optional: Handle cases with no C (set to -1), though problem assumes existence
-    actual_values = tensor[torch.arange(tensor.shape[0]), last_indices]
-    no_match = actual_values != C
-    last_indices[no_match] = -1
-
-    return last_indices

model-00001-of-00002.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:60ec23752c0063ecfafb148d1417d99b7b43c067967652dc90bb9bcb9ea3e243
-size 4999026432

model-00002-of-00002.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5045294a3d26c5bbec1dd6a3a6146dc9520cfea35618c5bb80e80a871d8829ad
-size 1122255376

model.safetensors.index.json

@@ -1,563 +0,0 @@
-{
-  "metadata": {
-    "total_size": 6121212928
-  },
-  "weight_map": {
-    "layer_norm_before_lm_heads.0.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.1.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.10.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.11.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.12.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.13.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.14.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.15.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.16.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.17.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.18.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.19.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.2.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.20.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.21.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.22.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.23.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.24.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.25.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.26.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.27.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.28.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.29.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.3.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.30.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.31.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.32.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.4.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.5.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.6.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.7.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.8.weight": "model-00002-of-00002.safetensors",
-    "layer_norm_before_lm_heads.9.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.0.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.1.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.10.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.11.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.12.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.13.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.14.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.15.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.16.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.17.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.18.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.19.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.2.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.20.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.21.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.22.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.23.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.24.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.25.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.26.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.27.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.28.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.29.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.3.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.30.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.31.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.32.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.4.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.5.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.6.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.7.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.8.weight": "model-00002-of-00002.safetensors",
-    "lm_heads.9.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.0.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.0.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.0.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.1.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.1.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.1.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.10.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.10.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.10.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.11.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.11.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.11.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.12.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.12.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.12.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.13.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.13.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.13.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.14.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.14.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.14.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.15.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.15.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.15.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.16.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.16.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.16.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.17.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.17.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.17.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.18.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.18.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.18.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.19.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.19.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.19.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.2.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.2.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.2.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.20.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.20.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.20.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.21.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.21.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.21.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.22.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.22.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.22.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.23.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.23.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.23.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.24.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.24.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.24.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.25.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.25.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.25.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.26.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.26.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.26.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.27.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.27.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.27.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.28.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.28.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.28.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.29.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.29.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.29.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.3.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.3.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.3.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.30.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.30.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.30.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.31.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.31.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.31.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.32.down_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.32.gate_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.32.up_proj.weight": "model-00002-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.4.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.4.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.4.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.5.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.5.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.5.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.6.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.6.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.6.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.7.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.7.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.7.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.8.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.8.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.8.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.9.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.9.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_to_speech_embedding_mlps.9.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "local_transformer.norm.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.0.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.1.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.10.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.11.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.12.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.13.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.14.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.15.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.16.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.17.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.18.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.19.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.2.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.20.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.21.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.22.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.23.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.24.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.25.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.26.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.27.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.28.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.29.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.3.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.30.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.31.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.32.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.4.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.5.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.6.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.7.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.8.weight": "model-00001-of-00002.safetensors",
-    "model.embedding_list.9.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.embed_tokens.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
-    "model.language_model.norm.weight": "model-00001-of-00002.safetensors",
-    "speech_embedding_to_local_mlp.down_proj.weight": "model-00001-of-00002.safetensors",
-    "speech_embedding_to_local_mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
-    "speech_embedding_to_local_mlp.up_proj.weight": "model-00001-of-00002.safetensors"
-  }
-}

modeling_moss_tts.py

@@ -1,743 +0,0 @@
-import os
-import copy
-import torch
-import torch.nn as nn
-import logging
-import sys
-
-from tqdm import tqdm
-from dataclasses import dataclass
-from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS
-from transformers.utils import ModelOutput
-from transformers.cache_utils import Cache
-from typing import Optional, List, Tuple, Union
-from transformers.loss.loss_utils import ForCausalLMLoss
-from transformers import PreTrainedModel, GenerationMixin
-from transformers.generation.streamers import BaseStreamer
-from transformers.models.qwen3.modeling_qwen3 import Qwen3Model, Qwen3Attention, eager_attention_forward
-from transformers.modeling_outputs import BaseModelOutputWithPast
-from transformers.models.qwen3.configuration_qwen3 import Qwen3Config
-from transformers.generation.configuration_utils import GenerationConfig
-from transformers.generation.stopping_criteria import StoppingCriteriaList
-from transformers.generation.logits_process import LogitsProcessorList, RepetitionPenaltyLogitsProcessor, TopKLogitsWarper, TopPLogitsWarper, TemperatureLogitsWarper
-from transformers.masking_utils import create_causal_mask
-
-from .inference_utils import find_last_equal_C
-from .configuration_moss_tts import MossTTSDelayConfig
-
-import math
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-
-
-class MossTTSRMSNorm(nn.Module):
-    def __init__(self, dim: int, eps: float = 1e-6):
-        super().__init__()
-        self.eps = eps
-        self.weight = nn.Parameter(torch.ones(dim))
-
-    def forward(self, x: torch.Tensor) -> torch.Tensor:
-        # x: [..., dim]
-        norm = x.pow(2).mean(dim=-1, keepdim=True)
-        x = x * torch.rsqrt(norm + self.eps)
-        return x * self.weight
-
-
-class MossTTSMLP(nn.Module):
-    """
-    HF-style MLP adapter equivalent to Megatron's SwiGLU FFN:
-      in:  input_size
-      mid: ffn_hidden_size
-      out: output_size
-
-    Computes:
-      y = down( silu(gate(x)) * up(x) )
-
-    Optionally includes a pre-norm on input (common in Megatron blocks).
-    """
-    def __init__(
-        self,
-        input_size: int,
-        ffn_hidden_size: int,
-        output_size: int,
-        bias: bool = False,
-        prenorm: bool = False,
-        norm_eps: float = 1e-6,
-        use_rmsnorm: bool = True,
-    ):
-        super().__init__()
-
-        self.prenorm = prenorm
-        if prenorm:
-            if use_rmsnorm:
-                self.norm = MossTTSRMSNorm(input_size, eps=norm_eps)
-            else:
-                self.norm = nn.LayerNorm(input_size, eps=norm_eps)
-        else:
-            self.norm = None
-
-        # SwiGLU uses two projections to ffn_hidden_size: gate and up
-        self.gate_proj = nn.Linear(input_size, ffn_hidden_size, bias=bias)
-        self.up_proj   = nn.Linear(input_size, ffn_hidden_size, bias=bias)
-
-        # down projection to output_size (note: output can differ from input)
-        self.down_proj = nn.Linear(ffn_hidden_size, output_size, bias=bias)
-
-    def forward(self, x: torch.Tensor) -> torch.Tensor:
-        if self.norm is not None:
-            x = self.norm(x)
-
-        gate = self.gate_proj(x)
-        up   = self.up_proj(x)
-        h = F.silu(gate) * up
-        y = self.down_proj(h)
-        return y
-
-def moss_tts_masked_embedding(embedding: nn.Embedding,
-                     input_ids: torch.LongTensor,
-                     ignore_index: int = -100) -> torch.Tensor:
-    """
-    对 input_ids 中 != ignore_index 的位置做 embedding，ignore_index 的位置输出全 0 向量。
-
-    Args:
-        embedding: 一个 nn.Embedding 层
-        input_ids: 任意形状的 LongTensor，里面允许出现 ignore_index
-        ignore_index: 需要被忽略的位置标记（默认 -100）
-
-    Returns:
-        embeddings: 形状为 (*input_ids.shape, embedding.embedding_dim) 的张量
-    """
-    # mask: True 表示需要正常 embedding，False 表示输出 0
-    mask = (input_ids != ignore_index)  # shape: [...]
-
-    # 为了避免 -100 这种非法 index 传进 embedding，这里先临时替换掉
-    safe_ids = input_ids.clone()
-    safe_ids[~mask] = 0
-
-    # 正常过 embedding
-    out = embedding(safe_ids)  # shape: [..., dim]
-
-    # 把 ignore_index 对应的位置置 0
-    out[~mask] = 0.0
-
-    return out
-
-class MossTTSAttentionWithoutPositionalEmbedding(Qwen3Attention):
-    """Multi-headed attention from 'Attention Is All You Need' paper"""
-
-    def __init__(self, config: MossTTSDelayConfig, layer_idx: int):
-        super().__init__(config, layer_idx)
-
-
-    def forward(
-        self,
-        hidden_states: torch.Tensor,
-        position_embeddings: Tuple[torch.Tensor, torch.Tensor],
-        attention_mask: Optional[torch.Tensor],
-        past_key_value: Optional[Cache] = None,
-        cache_position: Optional[torch.LongTensor] = None,
-        **kwargs,
-    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
-        input_shape = hidden_states.shape[:-1]
-        hidden_shape = (*input_shape, -1, self.head_dim)
-
-        query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
-        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
-        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
-
-        assert past_key_value is None
-
-        attention_interface = eager_attention_forward
-        if self.config._attn_implementation != "eager":
-            if self.config._attn_implementation == "sdpa" and kwargs.get("output_attentions", False):
-                print(
-                    "`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to "
-                    'eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
-                )
-            else:
-                attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
-
-        attn_output, attn_weights = attention_interface(
-            self,
-            query_states,
-            key_states,
-            value_states,
-            is_causal=True,
-            attention_mask=None,
-            dropout=0.0 if not self.training else self.attention_dropout,
-            scaling=self.scaling,
-            sliding_window=self.sliding_window,  # diff with Llama
-            **kwargs,
-        )
-
-        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
-        attn_output = self.o_proj(attn_output)
-        return attn_output, attn_weights
-
-class MossTTSLocalTransformer(Qwen3Model):
-    def __init__(self, config: MossTTSDelayConfig):
-        super().__init__(config)
-        del self.rotary_emb
-        del self.embed_tokens
-        for layer_idx in range(config.num_hidden_layers):
-            self.layers[layer_idx].self_attn = MossTTSAttentionWithoutPositionalEmbedding(config, layer_idx)
-        self.post_init()
-
-    def forward(
-        self,
-        input_ids: Optional[torch.LongTensor] = None,
-        attention_mask: Optional[torch.Tensor] = None,
-        position_ids: Optional[torch.LongTensor] = None,
-        past_key_values: Optional[Cache] = None,
-        inputs_embeds: Optional[torch.FloatTensor] = None,
-        use_cache: Optional[bool] = None,
-        output_attentions: Optional[bool] = None,
-        output_hidden_states: Optional[bool] = None,
-        cache_position: Optional[torch.LongTensor] = None,
-        **flash_attn_kwargs,
-    ) -> BaseModelOutputWithPast:
-        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
-        output_hidden_states = (
-            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
-        )
-        use_cache = use_cache if use_cache is not None else self.config.use_cache
-        use_cache = False
-        assert not use_cache
-
-        if (input_ids is None) ^ (inputs_embeds is not None):
-            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
-
-        if self.gradient_checkpointing and self.training and use_cache:
-            print(
-                "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`."
-            )
-            use_cache = False
-
-        # TODO (joao): remove this exception in v4.56 -- it exists for users that try to pass a legacy cache
-        if not isinstance(past_key_values, (type(None), Cache)):
-            raise ValueError("The `past_key_values` should be either a `Cache` object or `None`.")
-
-        if inputs_embeds is None:
-            inputs_embeds = self.embed_tokens(input_ids)
-
-        if use_cache and past_key_values is None:
-            assert False
-            past_key_values = DynamicCache()
-
-        if cache_position is None:
-            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
-            cache_position = torch.arange(
-                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
-            )
-
-        if position_ids is None:
-            position_ids = cache_position.unsqueeze(0)
-
-        # causal_mask = self._update_causal_mask( # ???
-        #     attention_mask, inputs_embeds, cache_position, past_key_values, output_attentions
-        # )
-        mask_kwargs = {
-            "config": self.config,
-            "input_embeds": inputs_embeds,
-            "attention_mask": attention_mask,
-            "cache_position": cache_position,
-            "past_key_values": past_key_values,
-            "position_ids": position_ids,
-        }
-        causal_mask = create_causal_mask(**mask_kwargs),
-
-
-        hidden_states = inputs_embeds
-
-        # create position embeddings to be shared across the decoder layers
-        # position_embeddings = self.rotary_emb(hidden_states, position_ids)
-
-        # decoder layers
-        all_hidden_states = () if output_hidden_states else None
-        all_self_attns = () if output_attentions else None
-
-        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
-            if output_hidden_states:
-                all_hidden_states += (hidden_states,)
-
-            layer_outputs = decoder_layer(
-                hidden_states,
-                attention_mask=causal_mask,
-                position_ids=None,
-                past_key_value=None,
-                output_attentions=output_attentions,
-                use_cache=use_cache,
-                cache_position=None,
-                position_embeddings=None,
-                **flash_attn_kwargs,
-            )
-
-            hidden_states = layer_outputs
-
-            if output_attentions:
-                all_self_attns += (layer_outputs[1],)
-
-        hidden_states = self.norm(hidden_states)
-
-        # add hidden states from the last decoder layer
-        if output_hidden_states:
-            all_hidden_states += (hidden_states,)
-
-        return BaseModelOutputWithPast(
-            last_hidden_state=hidden_states,
-            past_key_values=past_key_values if use_cache else None,
-            hidden_states=all_hidden_states,
-            attentions=all_self_attns,
-        )
-        
-@dataclass
-class MosiTTSOutputWithPast(ModelOutput):
-    loss: Optional[torch.FloatTensor] = None
-    logits: torch.FloatTensor = None
-    loss_all: Optional[Tuple[torch.FloatTensor]] = None
-    logits_all: Optional[Tuple[torch.FloatTensor]] = None
-    past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
-    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
-    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None
-
-
-@dataclass
-class MossTTSGenerateDecoderOnlyOutput(ModelOutput):
-    sequences: torch.LongTensor = None
-    scores: Optional[Tuple[torch.FloatTensor]] = None
-    logits: Optional[Tuple[torch.FloatTensor]] = None
-    attentions: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
-    hidden_states: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
-    past_key_values: Optional[Tuple[Tuple[Tuple[torch.FloatTensor]]]] = None
-
-
-class CustomMixin(GenerationMixin): # TODO 待检查正确性
-    def _sample(
-        self,
-        input_ids: torch.LongTensor, # (B, T, 1+Nq)
-        logits_processor: LogitsProcessorList,
-        stopping_criteria: StoppingCriteriaList,
-        generation_config: GenerationConfig,
-        synced_gpus: bool,
-        streamer: Optional["BaseStreamer"] = None,
-        **model_kwargs,
-    ) -> Union[MossTTSGenerateDecoderOnlyOutput, torch.LongTensor]:
-        # 提取配置参数
-        # assert False
-        speech_pad_idx = self.config.audio_pad_code
-        device = input_ids.device
-        eos_token_id = generation_config.eos_token_id
-        output_attentions = generation_config.output_attentions
-        output_hidden_states = generation_config.output_hidden_states
-        output_scores = generation_config.output_scores
-        output_logits = generation_config.output_logits
-        return_dict_in_generate = generation_config.return_dict_in_generate
-        max_length = generation_config.max_length
-        has_eos_stopping_criteria = any(hasattr(criteria, "eos_token_id") for criteria in stopping_criteria)
-        do_sample = generation_config.do_sample
-
-        # 初始化输出元组
-        scores = () if (return_dict_in_generate and output_scores) else None
-        raw_logits = () if (return_dict_in_generate and output_logits) else None
-        decoder_attentions = () if (return_dict_in_generate and output_attentions) else None
-        decoder_hidden_states = () if (return_dict_in_generate and output_hidden_states) else None
-
-        # 初始化跟踪变量
-        batch_size, cur_len, channels = input_ids.shape  # channels = 8
-        input_ids_length = cur_len
-        # assert batch_size == 1
-        this_peer_finished = False
-        unfinished_sequences = torch.ones(batch_size, dtype=torch.long, device=input_ids.device) # (B, )
-        base_length = input_ids.shape[1]
-        model_kwargs = self._get_initial_cache_position(cur_len, input_ids.device, model_kwargs)
-        # model_kwargs = self._get_initial_cache_position(input_ids, model_kwargs)
-
-        # 定义logits processor
-        if generation_config.do_samples is not None:
-            do_samples = generation_config.do_samples
-            realprocessor = [LogitsProcessorList() for _ in range(channels)]
-            for i, layer_config in enumerate(generation_config.layers):
-                if not do_samples[i]:
-                    continue
-                if layer_config.get("repetition_penalty") is not None and i != 0: # 文本层不用重复惩罚
-                    realprocessor[i].append(RepetitionPenaltyLogitsProcessor(penalty=layer_config.get("repetition_penalty")))
-                if layer_config.get("temperature") is not None:
-                    realprocessor[i].append(TemperatureLogitsWarper(temperature=layer_config.get("temperature")))
-                if layer_config.get("top_k") is not None:
-                    realprocessor[i].append(TopKLogitsWarper(top_k=layer_config.get("top_k")))
-                if layer_config.get("top_p") is not None:
-                    realprocessor[i].append(TopPLogitsWarper(top_p=layer_config.get("top_p")))
-        else:
-            assert False
-            do_samples = [do_sample for _ in range(channels)]
-            realprocessor = [logits_processor for _ in range(channels)]
-        
-        pbar = tqdm()
-        while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
-             # 准备模型输入
-            pbar.update()
-            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
-            model_inputs.update({"output_attentions": output_attentions} if output_attentions else {})
-            model_inputs.update({"output_hidden_states": output_hidden_states} if output_hidden_states else {})
-            # 前向传递
-            outputs = self(**model_inputs, n_vq_for_inference=generation_config.n_vq_for_inference, return_dict=True, output_hidden_states=True)
-            model_kwargs = self._update_model_kwargs_for_generation(outputs, model_kwargs)
-
-            if synced_gpus and this_peer_finished:
-                continue
-
-            global_trm_output_hidden_states = outputs.hidden_states[-1][:, -1, :] # (B, D)
-            dtype = global_trm_output_hidden_states.dtype
-
-            local_trm_dim = self.local_transformer_config.hidden_size
-            local_transformer_inputs = torch.zeros(batch_size, 0, local_trm_dim).to(device).to(dtype) # (B, 0 <= t <= Nq, D), 维护当前 local trm 的输入
-            current_local_transformer_input = self.speech_embedding_to_local_mlp(global_trm_output_hidden_states) # (B, D) 维护当前 timestamp 的 local trm 的输入，
-
-            next_tokens = [] # 1+Nq * (B, )
-            # n_vq_for_inference = int(os.environ['N_VQ_FOR_INFERENCE'])
-            n_vq_for_inference = generation_config.n_vq_for_inference
-            for layer_index in range(min(channels, 1 + n_vq_for_inference)):
-                local_transformer_inputs = torch.cat([local_transformer_inputs, current_local_transformer_input.unsqueeze(1)], dim=1) # (B, t, D)
-                local_transformer_outputs = self.local_transformer(
-                    input_ids=None,
-                    attention_mask=None,
-                    inputs_embeds=local_transformer_inputs # (B, t=1+Nq, D)
-                )[0] # (B, t=1+Nq, D)
-                local_transformer_outputs = self.layer_norm_before_lm_heads[layer_index](
-                    self.local_to_speech_embedding_mlps[layer_index](local_transformer_outputs) # (B, t=1+Nq, D)
-                ) # (B, t=1+Nq, D)
-
-                next_token_logit = self.lm_heads[layer_index](local_transformer_outputs[:, -1, :]) # (B, V)
-                if layer_index != 0:
-                    next_token_logit[:, speech_pad_idx] = -torch.inf
-                next_token_score = realprocessor[layer_index](input_ids[..., layer_index], next_token_logit) # (B, V)
-
-                if do_samples[layer_index]:
-                    channel_ntk = torch.multinomial(nn.functional.softmax(next_token_score, dim=-1), num_samples=1).squeeze(1) # (B, )
-                else:
-                    channel_ntk = torch.argmax(next_token_score, dim=-1) # (B, )
-
-                next_tokens.append(channel_ntk) # 1+Nq * (B, )
-                current_local_transformer_input = self.model.embedding_list[layer_index](channel_ntk) # (B, D)
-                current_local_transformer_input = self.speech_embedding_to_local_mlp(current_local_transformer_input) # (B, D)
-
-            for layer_index in range(1 + n_vq_for_inference, channels):
-                next_tokens.append(torch.zeros((batch_size, )).to(torch.int).to(device))
-            next_tokens = torch.stack(next_tokens, dim=-1)  # (B, 1+Nq)
-
-            if has_eos_stopping_criteria:
-                for i in range(channels):
-                    pddp = eos_token_id if i == 0 else speech_pad_idx
-                    next_tokens[:, i] = next_tokens[:, i] * unfinished_sequences + pddp * (1 - unfinished_sequences)
-            
-            input_ids = torch.cat([input_ids, next_tokens[:, None, :]], dim=1) # (B, T, 1+Nq)
-            if streamer is not None:
-                streamer.put(next_tokens[:, 0].cpu())
-
-            stopping = stopping_criteria(input_ids[..., 0], scores)
-            # stopping = stopping_criteria(input_ids[..., 0], scores)
-            unfinished_sequences = unfinished_sequences & ~stopping
-            this_peer_finished = unfinished_sequences.max() == 0
-
-            if return_dict_in_generate:
-                if output_scores:
-                    assert False
-                    scores += (next_token_scores,)
-                if output_logits:
-                    assert False
-                    raw_logits += (next_token_logits,)
-                if output_attentions:
-                    decoder_attentions += (outputs.attentions,)
-                if output_hidden_states:
-                    decoder_hidden_states += (outputs.hidden_states,)
-
-            cur_len += 1
-            del outputs
-
-        if streamer is not None:
-            streamer.end()
-
-        if return_dict_in_generate:
-            return MossTTSGenerateDecoderOnlyOutput(
-                sequences=input_ids,
-                scores=scores,
-                logits=raw_logits,
-                attentions=decoder_attentions,
-                hidden_states=decoder_hidden_states,
-                past_key_values=model_kwargs.get("past_key_values"),
-            )
-        else:
-            start_indices = find_last_equal_C(input_ids[..., 0], self.config.audio_start_token_id)
-            start_lengths = input_ids_length - start_indices - 1 # voice clone 下是 0，续写情况下是 prompt 音频的长度，不包括 audio_start_token
-            output = []
-            for start_idx, start_length, cur_generation_ids in zip(start_indices, start_lengths, input_ids):
-                output.append((start_length, cur_generation_ids[start_idx:]))
-            
-            return output
-
-
-class MosiTTSPretrainedModel(PreTrainedModel):
-    config_class = MossTTSDelayConfig
-    base_model_prefix = "model"
-    supports_gradient_checkpointing = True
-    _no_split_modules = ["Qwen2DecoderLayer"]
-    _skip_keys_device_placement = ["past_key_values"]
-    _supports_flash_attn_2 = True
-    _supports_sdpa = True
-    _supports_flex_attn = True
-    _supports_cache_class = True
-    _supports_quantized_cache = True
-    _supports_static_cache = True
-    _supports_attention_backend = True
-
-
-class MosiTTSModel(MosiTTSPretrainedModel):
-    def __init__(self, config: MossTTSDelayConfig):
-        super().__init__(config)
-        self.text_pad_idx = config.pad_token_id
-        self.speech_pad_idx = config.audio_pad_code
-        self.embedding_list = nn.ModuleList([])
-        self.embedding_list.append(nn.Embedding(config.vocab_size, config.hidden_size, self.text_pad_idx))
-        self.channels = 1 + config.n_vq
-        for _ in range(1, self.channels):
-            self.embedding_list.append(nn.Embedding(config.audio_vocab_size + 1, config.hidden_size, self.speech_pad_idx))
-
-        self.language_model = Qwen3Model(config.language_config)
-        self.post_init()
-
-    def get_input_embeddings(self):
-        return self.embedding_list[0]
-
-    def set_input_embeddings(self, value: nn.Embedding):
-        self.embedding_list[0] = value
-
-    def _prepare_multi_modal_inputs(self, input_ids: torch.LongTensor, n_vq_for_inference: int, **kwargs) -> torch.FloatTensor:
-        """
-        Prepares multi-modal embeddings from input_ids of shape (batch_size, channels, sequence_length).
-        For channel 0: text + speech tokens, for channels 1 to channels-1: speech tokens padded with speech_pad_token.
-        """
-        batch_size, seq_length, channels = input_ids.shape
-        if channels != self.channels:
-            raise ValueError(f"Expected {self.config.channels} channels, got {channels}")
-
-        inputs_embeds = torch.zeros(batch_size, seq_length, self.config.hidden_size, device=input_ids.device, dtype=self.embedding_list[0].weight.dtype)
-        for i in range(min(channels, 1 + n_vq_for_inference)):
-            embed_layer = self.embedding_list[i]
-            channel_input = input_ids[...,i]
-            inputs_embeds += embed_layer(channel_input)
-
-        return inputs_embeds # (B, T, D)
-
-    def forward(
-        self,
-        input_ids: torch.LongTensor = None,  # Shape: (batch_size, channels, sequence_length)
-        attention_mask: Optional[torch.Tensor] = None,
-        position_ids: Optional[torch.LongTensor] = None,
-        past_key_values: Optional[List[torch.FloatTensor]] = None,
-        inputs_embeds: Optional[torch.FloatTensor] = None,
-        use_cache: Optional[bool] = None,
-        output_attentions: Optional[bool] = None,
-        output_hidden_states: Optional[bool] = None,
-        return_dict: Optional[bool] = None,
-        cache_position: Optional[torch.LongTensor] = None,
-        **kwargs,
-    ) -> Union[Tuple, BaseModelOutputWithPast]:
-
-        if (input_ids is None) ^ (inputs_embeds is not None):
-            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
-
-        if input_ids is not None:
-            inputs_embeds = self._prepare_multi_modal_inputs(input_ids, **kwargs) # (B, T, D)
-
-        outputs = self.language_model(
-            input_ids=None,
-            attention_mask=attention_mask,
-            position_ids=position_ids,
-            past_key_values=past_key_values,
-            inputs_embeds=inputs_embeds,
-            use_cache=use_cache,
-            output_attentions=output_attentions,
-            output_hidden_states=output_hidden_states,
-            return_dict=return_dict,
-            cache_position=cache_position,
-        )
-        return outputs
-
-
-class MossTTSDelayModel(MosiTTSPretrainedModel, CustomMixin):
-    _tied_weights_keys = []
-    _tp_plan = {"lm_head": "colwise_rep"}
-    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
-
-    def __init__(self, config: MossTTSDelayConfig):
-        super().__init__(config)
-        self.model = MosiTTSModel(config)
-        self.channels = 1 + config.n_vq
-        self.weights = [1 for _ in range(self.channels)]
-        self._tied_weights_keys = [f"lm_heads.{i}.weight" for i in range(self.channels)]
-        self.vocab_size = config.vocab_size
-
-        local_transformer_config = copy.deepcopy(config.language_config)
-        local_transformer_config.num_hidden_layers = config.local_num_layers
-        local_transformer_config.hidden_size = config.local_hidden_size
-        local_transformer_config.intermediate_size = config.local_ffn_hidden_size
-        self.local_transformer_config = local_transformer_config
-        self.local_transformer = MossTTSLocalTransformer(self.local_transformer_config)
-
-        self.speech_embedding_to_local_mlp = MossTTSMLP(
-            input_size=config.hidden_size,
-            ffn_hidden_size=config.additional_mlp_ffn_hidden_size,
-            output_size=config.local_hidden_size
-        )
-        self.local_to_speech_embedding_mlps = nn.ModuleList([
-            MossTTSMLP(
-                input_size=config.local_hidden_size,
-                ffn_hidden_size=config.additional_mlp_ffn_hidden_size,
-                output_size=config.hidden_size
-            )
-            for _ in range(self.channels)
-        ])
-
-        self.layer_norm_before_lm_heads = nn.ModuleList([
-            MossTTSRMSNorm(config.hidden_size)
-            for _ in range(self.channels)
-        ])
-
-        self.lm_heads = nn.ModuleList([])
-        self.lm_heads.append(nn.Linear(config.hidden_size, config.vocab_size, bias=False))
-        for _ in range(1, self.channels):
-            self.lm_heads.append(nn.Linear(config.hidden_size, 1 + config.audio_vocab_size, bias=False))
-        self.post_init()
-
-    def get_input_embeddings(self):
-        return self.model.embedding_list[0]
-
-    def can_generate(self):
-        return True
-
-    # def tie_weights(self):
-    #     ...
-        # for i in range(self.config.channels):
-        #     self._tie_or_clone_weights(self.lm_heads[i], self.model.embedding_list[i])
-
-    def set_input_embeddings(self, value):
-        self.model.embedding_list[0] = value
-
-    def get_output_embeddings(self):
-        return self.lm_heads[0]
-
-    def set_output_embeddings(self, new_embeddings):
-        self.lm_heads[0] = new_embeddings
-
-    def set_decoder(self, decoder):
-        self.model = decoder
-
-    def get_decoder(self):
-        return self.model
-
-    def set_weights(self, weights):
-        self.weights = weights
-
-    def _prepare_shifted_audio_inputs(self, label_ids): # (B, T, 1 + Nq) 可能有 -100
-        text_and_audio_label_embed_list = [] # Nq * (1, T, B, D)
-        for i in range(0, self.local_transformer_config.channels - 1):
-            text_and_audio_label_embed_list.append(
-                moss_tts_masked_embedding(self.model.embedding_list[i], label_ids[:, :, i]).unsqueeze(0).transpose(1, 2) # (B, T) -> (B, T, D) -> (1, B, T, D) -> (1, T, B, D)
-            ) # (1, T, B, D)
-        audio_label_embeds = torch.stack(text_and_audio_label_embed_list, dim=0) # (Nq, 1, T, B, D)
-        audio_label_embeds = audio_label_embeds.contiguous()[:, 0, :, :, :].transpose(1, 2) # (Nq, B, T, D)
-        return audio_label_embeds # (Nq, B, T, D)
-
-    def forward(
-        self,
-        input_ids: torch.LongTensor = None, # (B, T, 1 + Nq)
-        attention_mask: Optional[torch.Tensor] = None,
-        position_ids: Optional[torch.LongTensor] = None,
-        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,
-        inputs_embeds: Optional[torch.FloatTensor] = None,
-        labels: Optional[torch.LongTensor] = None, # (B, T, 1 + Nq), TODO labels 为 input_ids shift 一位的结果
-        use_cache: Optional[bool] = None,
-        output_attentions: Optional[bool] = None,
-        output_hidden_states: Optional[bool] = None,
-        return_dict: Optional[bool] = None,
-        cache_position: Optional[torch.LongTensor] = None,
-        **kwargs,
-    ) -> Union[Tuple, MosiTTSOutputWithPast]:
-        device = input_ids.device if not input_ids is None else inputs_embeds.device
-        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
-        output_hidden_states = output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
-        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-
-        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
-        outputs = self.model(
-            input_ids=input_ids, # (B, T, 1 + Nq)
-            attention_mask=attention_mask,
-            position_ids=position_ids,
-            past_key_values=past_key_values,
-            inputs_embeds=inputs_embeds,
-            use_cache=use_cache,
-            output_attentions=output_attentions,
-            output_hidden_states=output_hidden_states,
-            return_dict=return_dict,
-            cache_position=cache_position,
-            **kwargs,
-        )
-
-        if labels is not None:
-            local_transformer_inputs_from_global = outputs[0].unsqueeze(0) # (1, B, T, D)
-            D_global= local_transformer_inputs_from_global.shape[-1]
-            local_transformer_inputs_from_speech_embeddings = self._prepare_shifted_audio_inputs(labels) # (B, T, 1 + Nq) -> (Nq, B, T, D)
-            local_transformer_input_hidden_states = torch.cat([local_transformer_inputs_from_global, local_transformer_inputs_from_speech_embeddings], dim=0).contiguous() # (1 + Nq, B, T, D)
-            local_transformer_input_hidden_states = self.speech_embedding_to_local_mlp(local_transformer_input_hidden_states) # (1 + Nq, B, T, D)
-            N_channels, B, T, D_local = local_transformer_input_hidden_states.shape
-            local_transformer_input_hidden_states = local_transformer_input_hidden_states.permute(1, 2, 0, 3) # (B, T, 1 + Nq, D)
-            local_transformer_input_hidden_states = local_transformer_input_hidden_states.reshape(B * T, N_channels, D_local) # (batch_size=B * T, time=1+Nq, D)
-            local_transformer_output_hidden_states = self.local_transformer( # TODO 没有开位置编码
-                input_ids=None,
-                attention_mask=None,
-                inputs_embeds=local_transformer_input_hidden_states # (batch_size=B * T, time=1+Nq, D)
-            )[0] # (batch_size=B * T, time=1+Nq, D)
-            after_lm_head_mlp_hidden_states = [] # Nq+1 * (B*T, D) TODO ???
-            for i in range(self.channels):
-                after_lm_head_mlp_hidden_states.append(
-                    self.layer_norm_before_lm_heads[i](
-                        self.local_to_speech_embedding_mlps[i](
-                            local_transformer_output_hidden_states[:, i, :] # (B*T, D)
-                        )
-                    )
-                )  # Nq+1 * (B*T, D)
-
-            after_lm_head_mlp_hidden_states = torch.stack(after_lm_head_mlp_hidden_states, dim=0)  # (1 + Nq, B*T, D)
-            after_lm_head_mlp_hidden_states = after_lm_head_mlp_hidden_states.reshape(N_channels, B, T, D_global) # (1 + Nq, B, T, D)
-            logits_all = [lm_head(h_i) for lm_head, h_i in zip(self.lm_heads, after_lm_head_mlp_hidden_states)] # 1+Nq * (B, T, V)
-
-            loss_all = torch.empty(self.channels, device=device) # (1 + Nq)
-
-            for i in range(self.channels):
-                vocab_size = self.config.vocab_size if i == 0 else self.config.audio_vocab_size
-                loss_all[i] = ForCausalLMLoss(logits_all[i], labels[..., i], vocab_size, shift_labels=labels[..., i]) # (B, T, V), (B, T) => (1, )
-            normalized_weights = [weight_i / sum(self.weights) for weight_i in self.weights] # (1+Nq, )
-
-            total_loss = 0
-            for w, loss in zip(normalized_weights, loss_all):
-                total_loss += w * loss
-        else:
-            total_loss = None
-            loss_all = None,
-            logits_all = [None]
-
-        assert return_dict
-        if not return_dict:
-            output = (logits_all,) + outputs[1:]
-            return (total_loss, loss_all, ) + output if loss is not None else output
-
-        return MosiTTSOutputWithPast(
-            loss=total_loss,
-            logits=logits_all[0],
-            loss_all=loss_all,
-            logits_all=logits_all, # 1+Nq * (B, T, V)
-            past_key_values=outputs.past_key_values,
-            hidden_states=outputs.hidden_states, # L * (B, T, D)
-            attentions=outputs.attentions,
-        )

processing_moss_tts.py

@@ -1,930 +0,0 @@
-# coding=utf-8
-# Copyright 2026 OpenMOSS and the HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-from typing import Any, Dict, List, Optional, Tuple, Type, Union, Literal, Final, cast
-from dataclasses import dataclass
-from pathlib import Path
-import re
-import torchaudio
-
-from transformers import processing_utils
-
-processing_utils.MODALITY_TO_BASE_CLASS_MAPPING["audio_tokenizer"] = "PreTrainedModel"
-
-import torch
-from transformers import (
-    PreTrainedTokenizerBase,
-    BatchFeature,
-    ProcessorMixin,
-    logging,
-    AutoConfig,
-    AutoModel,
-    AutoTokenizer,
-)
-
-from .configuration_moss_tts import MossTTSDelayConfig
-
-
-logger = logging.get_logger(__name__)
-
-
-AUDIO_PLACEHOLDER = "<|audio|>"
-
-
-@dataclass
-class Message:
-    def to_dict(self) -> Dict[str, Any]:
-        raise NotImplementedError
-
-
-@dataclass
-class UserMessage(Message):
-    text: Optional[str] = None
-    reference: Optional[List[Optional[Union[str, torch.Tensor]]]] = None
-    instruction: Optional[str] = None
-    tokens: Optional[int] = None
-    quality: Optional[str] = None
-    sound_event: Optional[str] = None
-    ambient_sound: Optional[str] = None
-    language: Optional[str] = None
-
-    def __post_init__(self):
-        template = """<user_inst>
-- Reference(s):
-{reference}
-- Instruction:
-{instruction}
-- Tokens:
-{tokens}
-- Quality:
-{quality}
-- Sound Event:
-{sound_event}
-- Ambient Sound:
-{ambient_sound}
-- Language:
-{language}
-- Text:
-{text}
-</user_inst>"""
-
-        audio_codes_list = []
-        if self.reference is None:
-            reference = "None"
-        elif isinstance(self.reference, List):
-            reference = []
-            for speaker_idx, speaker_reference in enumerate(self.reference):
-                if speaker_reference is not None:
-                    reference.append(f"[S{speaker_idx+1}]:\n{AUDIO_PLACEHOLDER}")
-            reference = "\n".join(reference)
-            audio_codes_list = [
-                speaker_reference
-                for speaker_reference in self.reference
-                if speaker_reference is not None
-            ]
-        else:
-            raise TypeError("`reference` should be exactly a list when it is not None.")
-
-        content = (
-            template.replace("{reference}", str(reference))
-            .replace("{instruction}", str(self.instruction))
-            .replace("{tokens}", str(self.tokens))
-            .replace("{quality}", str(self.quality))
-            .replace("{sound_event}", str(self.sound_event))
-            .replace("{ambient_sound}", str(self.ambient_sound))
-            .replace("{language}", str(self.language))
-            .replace("{text}", str(self.text))
-        )
-
-        self._content = content
-        self._audio_codes_list = audio_codes_list
-
-    def to_dict(self):
-        return {
-            "role": "user",
-            "content": self._content,
-            "audio_codes_list": self._audio_codes_list,
-        }
-
-
-@dataclass
-class AssistantMessage(Message):
-    audio_codes_list: List[Union[str, torch.Tensor]]
-    content: str = AUDIO_PLACEHOLDER
-
-    def to_dict(self):
-        return {
-            "role": "assistant",
-            "content": self.content,
-            "audio_codes_list": self.audio_codes_list,
-        }
-
-
-USER_MESSAGE_FIELDS = (
-    "text",
-    "reference",
-    "instruction",
-    "tokens",
-    "quality",
-    "sound_event",
-    "ambient_sound",
-    "language",
-)
-
-
-class MossTTSDelayProcessor(ProcessorMixin):
-    tokenizer_class = "AutoTokenizer"
-    audio_tokenizer_class = "AutoModel"
-
-    tokenizer: PreTrainedTokenizerBase
-    audio_tokenizer: Any
-
-    def __init__(
-        self,
-        tokenizer: PreTrainedTokenizerBase,
-        audio_tokenizer: Any = None,
-        model_config: Optional[MossTTSDelayConfig] = None,
-        **kwargs,
-    ):
-        super().__init__(tokenizer=tokenizer, audio_tokenizer=audio_tokenizer, **kwargs)
-
-        # Explicit assignments for type-checkers; ProcessorMixin sets these too.
-        self.tokenizer = tokenizer
-        self.audio_tokenizer = audio_tokenizer
-        if model_config is None:
-            model_config = MossTTSDelayConfig()
-        self.model_config = model_config
-
-        self.imstart_token_id = tokenizer.convert_tokens_to_ids("<|im_start|>")
-        self.imend_token_id = tokenizer.convert_tokens_to_ids("<|im_end|>")
-        self.newline_token_id = 198
-
-        def _id_to_token(token_id: int) -> str:
-            tok = tokenizer.convert_ids_to_tokens(int(token_id))
-            if isinstance(tok, list):
-                return tok[0] if len(tok) > 0 else ""
-            return cast(str, tok)
-
-        self.audio_user_slot_token = _id_to_token(
-            self.model_config.audio_user_slot_token_id
-        )
-        self.audio_assistant_gen_slot_token = _id_to_token(
-            self.model_config.audio_assistant_gen_slot_token_id
-        )
-        self.audio_assistant_delay_slot_token = _id_to_token(
-            self.model_config.audio_assistant_delay_slot_token_id
-        )
-        self.audio_start_token = _id_to_token(self.model_config.audio_start_token_id)
-        self.audio_end_token = _id_to_token(self.model_config.audio_end_token_id)
-
-    @classmethod
-    def from_pretrained(cls, pretrained_model_name_or_path, *args, **kwargs):
-        trust_remote_code = kwargs.pop("trust_remote_code", True)
-        kwargs.pop("_from_auto", None)
-
-        audio_tokenizer_name_or_path = kwargs.pop(
-            "codec_path", "OpenMOSS-Team/MOSS-Audio-Tokenizer"
-        )
-
-        pretrained_model_name_or_path = Path(pretrained_model_name_or_path)
-        model_config = cast(
-            MossTTSDelayConfig,
-            AutoConfig.from_pretrained(
-                pretrained_model_name_or_path,
-                *args,
-                trust_remote_code=trust_remote_code,
-                **kwargs,
-            ),
-        )
-        tokenizer = AutoTokenizer.from_pretrained(
-            pretrained_model_name_or_path,
-            *args,
-            trust_remote_code=trust_remote_code,
-            **kwargs,
-        )
-        audio_tokenizer = AutoModel.from_pretrained(
-            audio_tokenizer_name_or_path,
-            trust_remote_code=trust_remote_code,
-            **kwargs,
-        )
-
-        return cls(
-            tokenizer=tokenizer,
-            audio_tokenizer=audio_tokenizer,
-            model_config=model_config,
-            **kwargs,
-        )
-
-    def __call__(self, *args, **kwargs) -> BatchFeature:
-        conversations = args[0] if len(args) > 0 else kwargs.pop("conversations")
-        mode: str = kwargs.pop("mode", "generation")
-        apply_chat_template: bool = kwargs.pop("apply_chat_template", True)
-        n_vq: Optional[int] = kwargs.pop("n_vq", None)
-
-        # Common ProcessorMixin kwargs that we ignore because we always return torch tensors.
-        kwargs.pop("return_tensors", None)
-        kwargs.pop("padding", None)
-        kwargs.pop("truncation", None)
-
-        """
-        mode only works when a Message is converted to a dict.
-        """
-
-        if mode not in {"generation", "continuation"}:
-            raise RuntimeError
-
-        if isinstance(conversations, (Message, Dict)):
-            conversations = [conversations]
-
-        truncation = False
-        if mode == "continuation":
-            truncation = True
-
-        input_ids_list = []
-        for conversation in conversations:
-            if isinstance(conversation, (Message, Dict)):
-                conversation = [conversation]
-
-            # Normalize early so downstream logic always deals with dict messages.
-            conversation = [self._normalize_message(m) for m in conversation]
-
-            if (mode == "generation") ^ (len(conversation) % 2 != 0):
-                raise ValueError
-
-            if (mode == "generation") ^ (conversation[-1]["role"] == "user"):
-                raise ValueError
-
-            unified_codes = []
-            for message_idx, message in enumerate(conversation):
-                if apply_chat_template:
-                    add_generation_prompt = (
-                        mode == "generation" and message_idx == len(conversation) - 1
-                    )
-                    try:
-                        content = self.tokenizer.apply_chat_template(
-                            [{"role": message["role"], "content": message["content"]}],
-                            add_generation_prompt=add_generation_prompt,
-                            tokenize=False,
-                        )
-                    except TypeError:
-                        try:
-                            content = self.tokenizer.apply_chat_template(
-                                [
-                                    {
-                                        "role": message["role"],
-                                        "content": message["content"],
-                                    }
-                                ],
-                                add_generation_prompt=add_generation_prompt,
-                            )
-                        except Exception:
-                            logger.warning(
-                                "apply_chat_template failed; fallback to raw content."
-                            )
-                            content = message["content"]
-                else:
-                    content = message["content"]
-
-                if not isinstance(content, str):
-                    content = str(content)
-
-                # Batch-encode all path-based references in one call when possible.
-                # This ensures we actually exercise audio_tokenizer.batch_encode for multi-reference prompts,
-                # instead of repeatedly calling it with batch=1.
-                raw_audio_items = message.get("audio_codes_list", [])
-
-                audio_codes_list: List[torch.Tensor] = []
-                if len(raw_audio_items) > 0:
-                    encoded_items: List[Optional[torch.Tensor]] = [None] * len(
-                        raw_audio_items
-                    )
-                    paths: List[str] = []
-                    path_positions: List[int] = []
-
-                    for idx, item in enumerate(raw_audio_items):
-                        if isinstance(item, torch.Tensor):
-                            if n_vq is not None and item.shape[1] != n_vq:
-                                raise RuntimeError(
-                                    "audio_codes's n_vq is not equal to the parameter `n_vq`. Your can set the parameter `n_vq` as None if you have already tokenzied the wavs."
-                                )
-                            encoded_items[idx] = item
-                            continue
-
-                        if isinstance(item, (str, os.PathLike)):
-                            paths.append(str(item))
-                            path_positions.append(idx)
-                            continue
-
-                        raise TypeError(
-                            "Each audio item must be a torch.Tensor of codes or a path-like string."
-                        )
-
-                    if len(paths) > 0:
-                        encoded_from_paths = self.encode_audios_from_path(paths, n_vq)
-                        if len(encoded_from_paths) != len(paths):
-                            raise RuntimeError(
-                                "encode_audios_from_path returned an unexpected number of items."
-                            )
-                        for pos, codes in zip(path_positions, encoded_from_paths):
-                            encoded_items[pos] = codes
-
-                    audio_codes_list = [cast(torch.Tensor, t) for t in encoded_items]
-                unified_codes.append(
-                    self._get_unified_codes(
-                        message["role"], content, audio_codes_list, truncation
-                    )
-                )
-
-            unified_codes = torch.cat(unified_codes) # (T, Nq)
-            if mode == "generation":
-                audio_start_position_tokens = torch.zeros((1, unified_codes.shape[-1])).to(unified_codes.dtype).to(unified_codes.device) # (1, Nq)
-                audio_start_position_tokens[:, 0] = self.tokenizer.encode(self.audio_start_token)[0]
-                audio_start_position_tokens[:, 1:] = self.model_config.audio_pad_code
-                unified_codes = torch.cat([unified_codes, audio_start_position_tokens], dim=0) # (T, Nq)
-            input_ids_list.append(unified_codes)
-
-        return BatchFeature(data=self._pad(input_ids_list))
-
-    @staticmethod
-    def build_user_message(
-        text: Optional[str] = None,
-        reference: Optional[List[Optional[Union[str, torch.Tensor]]]] = None,
-        instruction: Optional[str] = None,
-        tokens: Optional[int] = None,
-        quality: Optional[str] = None,
-        sound_event: Optional[str] = None,
-        ambient_sound: Optional[str] = None,
-        language: Optional[str] = None,
-    ) -> Dict:
-        if reference is not None and not isinstance(reference, list):
-            reference = [reference]
-        return UserMessage(
-            text=text,
-            reference=reference,
-            instruction=instruction,
-            tokens=tokens,
-            quality=quality,
-            sound_event=sound_event,
-            ambient_sound=ambient_sound,
-            language=language,
-        ).to_dict()
-
-    @staticmethod
-    def build_assistant_message(
-        audio_codes_list: List[Union[str, torch.Tensor]],
-        content: str = AUDIO_PLACEHOLDER,
-    ) -> Dict:
-        return AssistantMessage(
-            audio_codes_list=audio_codes_list,
-            content=content,
-        ).to_dict()
-
-    def _normalize_message(self, message: Union[Message, Dict]) -> Dict:
-        if isinstance(message, Message):
-            return message.to_dict()
-        if not isinstance(message, dict):
-            raise TypeError("Each message must be a Message or dict.")
-        if "role" not in message:
-            raise ValueError("Message dict must include a 'role' field.")
-        if "content" in message and "audio_codes_list" in message:
-            return message
-        role = message["role"]
-        if role == "user":
-            kwargs = {key: message.get(key) for key in USER_MESSAGE_FIELDS}
-            return self.build_user_message(**kwargs)
-        if role == "assistant":
-            return self.build_assistant_message(
-                audio_codes_list=message.get("audio_codes_list", []),
-                content=message.get("content", AUDIO_PLACEHOLDER),
-            )
-        raise ValueError(f"Unsupported role: {role}")
-
-    def _pad(self, input_ids_list: List[torch.Tensor]):
-        device = input_ids_list[0].device
-        lengths = torch.tensor([w.shape[0] for w in input_ids_list], device=device)
-        pad_input_ids = torch.nn.utils.rnn.pad_sequence(
-            input_ids_list,
-            batch_first=True,
-            padding_value=self.model_config.audio_pad_code,
-            padding_side="left",
-        )
-        other_channel_mask = (pad_input_ids.shape[1] - lengths).unsqueeze(
-            1
-        ) > torch.arange(pad_input_ids.shape[1], device=device).unsqueeze(0)
-        pad_input_ids[..., 0][other_channel_mask] = self.model_config.pad_token_id
-        attention_mask = torch.zeros(
-            pad_input_ids.shape[0], pad_input_ids.shape[1], device=device
-        )
-        attention_mask[~other_channel_mask] = 1
-        attention_mask = attention_mask.bool()
-        return {
-            "input_ids": pad_input_ids,  # [batch_size, seqlen, n_vq]
-            "attention_mask": attention_mask,
-        }
-
-    @staticmethod
-    def _replace_audio_placeholders(
-        content: str,
-        lengths: List[int],
-        n_vq: int,
-        gen_slot_token: str,
-        delay_slot_token: str,
-        audio_start_token: str,
-        audio_end_token: str,
-    ) -> str:
-        if n_vq < 1:
-            raise ValueError(f"n_vq must be >= 1, got {n_vq}")
-
-        num_placeholders = content.count(AUDIO_PLACEHOLDER)
-        if num_placeholders != len(lengths):
-            raise ValueError(
-                f"Number of {AUDIO_PLACEHOLDER} ({num_placeholders}) "
-                f"does not match lengths ({len(lengths)})"
-            )
-
-        def build_audio_block(length: int) -> str:
-            if length < 0:
-                raise ValueError(f"length must be >= 0, got {length}")
-
-            if length == 0:
-                return f"{audio_start_token}{audio_end_token}"
-
-            step_tokens = gen_slot_token * length
-            return f"{audio_start_token}{step_tokens}{audio_end_token}"
-
-        lengths_iter = iter(lengths)
-
-        def replacer(match: re.Match) -> str:
-            length = next(lengths_iter)
-            return build_audio_block(length)
-
-        result = re.sub(re.escape(AUDIO_PLACEHOLDER), replacer, content)
-
-        return result
-
-    @staticmethod
-    def _merge_consecutive_audio_placeholders(
-        content: str,
-        audio_codes_list: List[torch.Tensor],
-    ) -> Tuple[str, List[torch.Tensor]]:
-        matches = list(re.finditer(re.escape(AUDIO_PLACEHOLDER), content))
-        if len(matches) <= 1:
-            return content, audio_codes_list
-
-        if len(matches) != len(audio_codes_list):
-            raise ValueError(
-                "Audio placeholders do not match the provided audio codes list."
-            )
-
-        new_audio_codes_list = []
-        new_parts = []
-        last_pos = 0
-        i = 0
-        while i < len(matches):
-            j = i
-            while (
-                j + 1 < len(matches)
-                and content[matches[j].end() : matches[j + 1].start()].strip() == ""
-            ):
-                j += 1
-
-            new_parts.append(content[last_pos : matches[i].start()])
-            new_parts.append(AUDIO_PLACEHOLDER)
-            last_pos = matches[j].end()
-
-            if j == i:
-                new_audio_codes_list.append(audio_codes_list[i])
-            else:
-                new_audio_codes_list.append(
-                    torch.cat(audio_codes_list[i : j + 1], dim=0)
-                )
-
-            i = j + 1
-
-        new_parts.append(content[last_pos:])
-        return "".join(new_parts), new_audio_codes_list
-
-    @staticmethod
-    def apply_delay_pattern(codes: torch.Tensor, pad_code: int) -> torch.Tensor:
-        delayed_tokens = torch.full(
-            (codes.shape[0] + codes.shape[1] - 1, codes.shape[1]),
-            pad_code,
-            device=codes.device,
-            dtype=codes.dtype,
-        )
-        for i in range(codes.shape[1]):
-            delayed_tokens[i : i + codes.shape[0], i] = codes[:, i]
-        return delayed_tokens
-
-    @staticmethod
-    def apply_de_delay_pattern(delay_codes: torch.Tensor) -> torch.Tensor:
-        tokens = torch.full(
-            (delay_codes.shape[0] - delay_codes.shape[1] + 1, delay_codes.shape[1]),
-            0,
-            device=delay_codes.device,
-            dtype=delay_codes.dtype,
-        )
-        for i in range(delay_codes.shape[1]):
-            tokens[:, i] = delay_codes[i : i + tokens.shape[0], i]
-        return tokens
-
-    def _get_unified_codes(
-        self,
-        role: str,
-        content: str,
-        audio_codes_list: List[torch.Tensor],
-        truncation: bool,
-    ) -> torch.Tensor:
-        """
-        此时的 content 已经是带上了对话格式
-        """
-        if role == "user":
-            audio_gen_slot_token = audio_delay_slot_token = self.audio_user_slot_token
-            truncation = False
-        else:
-            audio_gen_slot_token = self.audio_assistant_gen_slot_token
-            audio_delay_slot_token = self.audio_assistant_delay_slot_token
-
-        if len(audio_codes_list):
-            n_vq = audio_codes_list[0].shape[1]
-        else:
-            n_vq = self.model_config.n_vq
-
-        if len(audio_codes_list) > 1 and AUDIO_PLACEHOLDER in content:
-            content, audio_codes_list = self._merge_consecutive_audio_placeholders(
-                content, audio_codes_list
-            )
-        content = self._replace_audio_placeholders(
-            content=content,
-            lengths=[len(audio_codes) for audio_codes in audio_codes_list],
-            n_vq=n_vq,
-            gen_slot_token=audio_gen_slot_token,
-            delay_slot_token=audio_delay_slot_token,
-            audio_start_token=self.audio_start_token,
-            audio_end_token=self.audio_end_token,
-        )
-        text_codes = torch.tensor(
-            self.tokenizer.encode(content),
-            device=audio_codes_list[0].device if audio_codes_list else None,
-        )
-
-        audio_start_indices = torch.where(
-            text_codes == self.model_config.audio_start_token_id
-        )[0]
-        audio_end_indices = torch.where(
-            text_codes == self.model_config.audio_end_token_id
-        )[0]
-        if len(audio_start_indices) != len(audio_codes_list) or len(
-            audio_end_indices
-        ) != len(audio_codes_list):
-            raise ValueError(
-                "Audio placeholders do not match the provided audio codes list."
-            )
-
-        delay_audio_codes_list = []
-        assert len(audio_codes_list) <= 1
-        if len(audio_codes_list) == 0:
-            delay_audio_codes_list = torch.full(
-                (len(text_codes), n_vq),
-                self.model_config.audio_pad_code,
-                device=text_codes.device,
-                dtype=text_codes.dtype,
-            )
-        else:
-            prefix_idx = 0
-            for audio_start_idx_t, audio_end_idx_t, audio_codes in zip(
-                audio_start_indices, audio_end_indices, audio_codes_list
-            ):
-                audio_start_idx = int(audio_start_idx_t.item())
-                audio_end_idx = int(audio_end_idx_t.item())
-                delay_audio_codes = audio_codes # not delay 
-                pad_codes = torch.full(
-                    (audio_start_idx - prefix_idx + 1, n_vq),
-                    self.model_config.audio_pad_code,
-                    device=audio_codes.device,
-                    dtype=audio_codes.dtype,
-                )
-                delay_audio_codes_list.extend([pad_codes, delay_audio_codes])
-                prefix_idx = audio_end_idx
-
-            if truncation:
-                raise RuntimeError("Truncation generation is not supported at present")
-            else:
-                last_audio_end_idx = int(audio_end_indices[-1].item())
-                pad_codes = torch.full(
-                    (len(text_codes) - last_audio_end_idx, n_vq),
-                    self.model_config.audio_pad_code,
-                    device=audio_codes_list[0].device,
-                    dtype=audio_codes_list[0].dtype,
-                )
-                delay_audio_codes_list.append(pad_codes)
-
-            delay_audio_codes_list = torch.cat(delay_audio_codes_list)
-
-        if text_codes.shape[0] != delay_audio_codes_list.shape[0]:
-            text_codes = text_codes[: delay_audio_codes_list.shape[0]]
-
-        unified_codes = torch.cat(
-            [text_codes.unsqueeze(1), delay_audio_codes_list], dim=1
-        )
-        return unified_codes
-
-    def _parse_text_codes(self, start_length, text_codes):
-        text = cast(str, self.tokenizer.decode(text_codes))
-        prefix = cast(str, self.tokenizer.decode(text_codes[:start_length]))
-        text = text[len(prefix) :]
-
-        AUDIO_PATTERN = re.compile(
-            rf"(?:{self.audio_start_token})?"
-            rf"(?:{self.audio_assistant_gen_slot_token})*"
-            rf"(?:{self.audio_assistant_delay_slot_token})*"
-            rf"{self.audio_end_token}"
-        )
-
-        def normalize_audio_segments(text: str) -> str:
-            def repl(match: re.Match) -> str:
-                seg = match.group(0)
-                # Replace with <|audio|> if gen_slot is present in the segment;
-                if self.audio_assistant_gen_slot_token in seg:
-                    return AUDIO_PLACEHOLDER
-                # Otherwise, remove it.
-                return ""
-
-            return AUDIO_PATTERN.sub(repl, text)
-
-        return normalize_audio_segments(text)
-
-    def _parse_audio_codes(self, start_length, audio_codes):
-        # De-delay back to [T', n_vq]
-        # Rows that are all pad are separators between real audio segments.
-        is_pad = (audio_codes == self.model_config.audio_pad_code).all(dim=1)
-        non_pad = ~is_pad
-        if not non_pad.any():
-            return []
-
-        idx = torch.nonzero(non_pad).squeeze(1)
-        breaks = torch.where(idx[1:] != idx[:-1] + 1)[0] + 1
-        if breaks.numel() == 0:
-            segments_idx = [idx]
-        else:
-            segments_idx = torch.split(idx, breaks.tolist())
-
-        audio_codes_list = [audio_codes[s] for s in segments_idx]
-
-        # Batch-decode all audio segments together.
-        decoded_audio_list = self.decode_audio_codes(audio_codes_list)
-
-        # Keep codec causal context by decoding the whole first segment first,
-        # then trim at waveform level according to start_length ratio.
-        if (
-            start_length > 0
-            and len(audio_codes_list) > 0
-            and len(decoded_audio_list) > 0
-        ):
-            first_codes_length = audio_codes_list[0].shape[0]
-            if first_codes_length > 0:
-                trim_ratio = max(
-                    0.0, min(float(start_length) / float(first_codes_length), 1.0)
-                )
-                first_audio = decoded_audio_list[0]
-                if trim_ratio >= 1.0:
-                    decoded_audio_list = decoded_audio_list[1:]
-                elif trim_ratio > 0.0:
-                    trim_samples = int(first_audio.shape[-1] * trim_ratio)
-                    decoded_audio_list[0] = first_audio[..., trim_samples:]
-
-        return decoded_audio_list
-
-    def decode(self, output: List[Tuple[int, torch.Tensor]]):
-        """
-        1. 这里不管怎样，都需要一个完整的 assistant generation ids;
-        2. 支持从任意位置进行截断；
-        """
-
-        genearted_messages = []
-        for start_length, generation_ids in output:
-            content = self._parse_text_codes(start_length, generation_ids[:, 0])
-            audio_codes_list = self._parse_audio_codes(
-                start_length, generation_ids[:, 1:]
-            )
-            if content == "":
-                message = None
-            else:
-                message = AssistantMessage(
-                    content=content,
-                    audio_codes_list=cast(
-                        List[Union[str, torch.Tensor]], audio_codes_list
-                    ),
-                )
-            genearted_messages.append(message)
-        return genearted_messages
-
-    @staticmethod
-    def loudness_normalize(
-        wav: torch.Tensor,
-        target_dbfs: float = -20,
-        gain_range: tuple[float, float] = (-3.0, 3.0),
-    ) -> torch.Tensor:
-        wav = wav.to(torch.float32)
-        if wav.numel() == 0:
-            return wav
-        current_dbfs = 10.0 * torch.log10(torch.mean(wav**2) + 1e-9)
-        gain = float(target_dbfs - current_dbfs)
-        gain = max(gain_range[0], min(gain, gain_range[1]))
-        factor = 10.0 ** (gain / 20.0)
-        return wav * factor
-
-    def _get_audio_tokenizer_device(self) -> torch.device:
-        """Best-effort device inference for `self.audio_tokenizer`.
-
-        Notes:
-        - Old TAC wrapper exposed `.device`, but standard `torch.nn.Module` does not.
-        - New MossAudioTokenizerModel is a `PreTrainedModel`; parameters define its device.
-        """
-
-        audio_tokenizer = getattr(self, "audio_tokenizer", None)
-        if audio_tokenizer is None:
-            logger.warning(
-                "audio_tokenizer is not set on processor. Using CPU as default."
-            )
-            return torch.device("cpu")
-
-        device_attr = getattr(audio_tokenizer, "device", None)
-        if isinstance(device_attr, torch.device):
-            return device_attr
-
-        try:
-            return next(audio_tokenizer.parameters()).device
-        except StopIteration:
-            # No parameters (shouldn't happen for real models); default to CPU.
-            logger.warning(
-                "No parameters found on audio_tokenizer. Using CPU as default."
-            )
-            return torch.device("cpu")
-
-    def encode_audios_from_wav(
-        self,
-        wav_list: List[torch.Tensor],
-        sampling_rate: int,
-        n_vq: Optional[int] = None,
-    ):
-        if self.audio_tokenizer is None:
-            raise RuntimeError("audio_tokenizer is not set on processor.")
-        audio_tokenizer = self.audio_tokenizer
-
-        if isinstance(wav_list, torch.Tensor):
-            wav_list = [wav_list]
-        wav_list_ = []
-        resample = False
-        if sampling_rate != self.model_config.sampling_rate:
-            resample = True
-        device = self._get_audio_tokenizer_device()
-        for wav in wav_list:
-            if wav.shape[0] > 1:
-                wav = torch.mean(wav, dim=0, keepdim=True)
-            if resample:
-                wav = torchaudio.functional.resample(
-                    waveform=wav,
-                    orig_freq=sampling_rate,
-                    new_freq=self.model_config.sampling_rate,
-                )
-            wav = wav.to(device)
-            wav_list_.append(self.loudness_normalize(wav.squeeze(0)))
-
-        # New MossAudioTokenizerModel API: prefer batch_encode(list[wav])
-        if hasattr(audio_tokenizer, "batch_encode"):
-            enc = audio_tokenizer.batch_encode(wav_list_, num_quantizers=n_vq)
-            audio_codes = enc.audio_codes  # (NQ, B, T)
-            audio_codes_lengths = enc.audio_codes_lengths  # (B,)
-        else:
-            # Fallback: use encode() with explicit padding.
-            max_len = max(int(wav.shape[-1]) for wav in wav_list_)
-            input_values = torch.zeros(
-                len(wav_list_), 1, max_len, device=device, dtype=torch.float32
-            )
-            padding_mask = torch.zeros(
-                len(wav_list_), max_len, device=device, dtype=torch.bool
-            )
-            for i, wav in enumerate(wav_list_):
-                this_len = int(wav.shape[-1])
-                input_values[i, 0, :this_len] = wav
-                padding_mask[i, :this_len] = True
-            enc = audio_tokenizer.encode(
-                input_values,
-                padding_mask=padding_mask,
-                num_quantizers=n_vq,
-                return_dict=True,
-            )
-            audio_codes = enc.audio_codes
-            audio_codes_lengths = enc.audio_codes_lengths
-
-        if audio_codes is None or audio_codes_lengths is None:
-            raise RuntimeError(
-                "audio_tokenizer.encode() returned empty outputs (audio_codes/audio_codes_lengths)."
-            )
-
-        # Keep processor's historical contract: list[Tensor] with shape (T, NQ)
-        # and on CPU (so downstream text/audio packing remains device-agnostic).
-        codes_list: List[torch.Tensor] = []
-        for i in range(int(audio_codes.shape[1])):
-            length_i = int(audio_codes_lengths[i].item())
-            codes_i = (
-                audio_codes[:, i, :length_i]
-                .transpose(0, 1)
-                .contiguous()
-                .to(torch.long)
-                .cpu()
-            )
-            codes_list.append(codes_i)
-        return codes_list
-
-    def encode_audios_from_path(
-        self, wav_path_list: Union[str, List[str]], n_vq: Optional[int] = None
-    ):
-        if isinstance(wav_path_list, str):
-            wav_path_list = [wav_path_list]
-
-        if len(wav_path_list) == 0:
-            raise ValueError("Empty wav_path_list")
-
-        # Load + (if needed) resample each wav independently, so callers can
-        # pass a heterogeneous batch of files while still benefiting from
-        # audio_tokenizer.batch_encode.
-        target_sr = int(self.model_config.sampling_rate)
-        wav_list: List[torch.Tensor] = []
-        for wav_path in wav_path_list:
-            wav, sr = torchaudio.load(wav_path)
-            if int(sr) != target_sr:
-                wav = torchaudio.functional.resample(
-                    waveform=wav,
-                    orig_freq=int(sr),
-                    new_freq=target_sr,
-                )
-            wav_list.append(wav)
-
-        return self.encode_audios_from_wav(wav_list, target_sr, n_vq)
-
-    def decode_audio_codes(
-        self, audio_tokens_list: Union[torch.Tensor, List[torch.Tensor]]
-    ):
-        if self.audio_tokenizer is None:
-            raise RuntimeError("audio_tokenizer is not set on processor.")
-        audio_tokenizer = self.audio_tokenizer
-
-        if isinstance(audio_tokens_list, torch.Tensor):
-            audio_tokens_list = [audio_tokens_list]
-        if len(audio_tokens_list) == 0:
-            return []
-
-        device = self._get_audio_tokenizer_device()
-
-        # Processor uses (T, NQ); MossAudioTokenizer expects (NQ, T) (or (NQ, B, T)).
-        codes_list = [
-            codes.transpose(0, 1).contiguous().to(device=device, dtype=torch.long)
-            for codes in audio_tokens_list
-        ]
-    
-        # Fallback: pad to (NQ, B, T) + mask, then decode.
-        nq = int(codes_list[0].shape[0])
-        max_t = max(int(c.shape[1]) for c in codes_list)
-        audio_codes = torch.zeros(
-            nq, len(codes_list), max_t, device=device, dtype=torch.long
-        )
-        padding_mask = torch.zeros(
-            len(codes_list), max_t, device=device, dtype=torch.bool
-        )
-        for i, c in enumerate(codes_list):
-            t = int(c.shape[1])
-            audio_codes[:, i, :t] = c
-            padding_mask[i, :t] = True
-        dec = audio_tokenizer.decode(
-            audio_codes, padding_mask=padding_mask, return_dict=True, chunk_duration=8
-        )
-        audio = dec.audio
-        audio_lengths = dec.audio_lengths
-
-        if audio is None or audio_lengths is None:
-            raise RuntimeError(
-                "audio_tokenizer.decode() returned empty outputs (audio/audio_lengths)."
-            )
-
-        # Return historical contract: list of 1D waveforms (T,)
-        wav_list: List[torch.Tensor] = []
-        for i in range(int(audio.shape[0])):
-            length_i = int(audio_lengths[i].item())
-            wav = audio[i, 0, :length_i].contiguous().to(torch.float32).cpu()
-            wav_list.append(wav)
-        return wav_list

processor_config.json

@@ -1,6 +0,0 @@
-{
-    "processor_class": "MossTTSDelayProcessor",
-    "auto_map": {
-        "AutoProcessor": "processing_moss_tts.MossTTSDelayProcessor"
-    }
-}

requirements.txt

@@ -0,0 +1,6 @@
+numpy
+torch==2.7.0
+torchaudio==2.7.0
+soundfile
+huggingface-hub
+transformers

special_tokens_map.json

@@ -1,31 +0,0 @@
-{
-  "additional_special_tokens": [
-    "<|im_start|>",
-    "<|im_end|>",
-    "<|object_ref_start|>",
-    "<|object_ref_end|>",
-    "<|box_start|>",
-    "<|box_end|>",
-    "<|quad_start|>",
-    "<|quad_end|>",
-    "<|audio_start|>",
-    "<|audio_end|>",
-    "<|audio_user_slot|>",
-    "<|image_pad|>",
-    "<|audio_assistant_gen_slot|>"
-  ],
-  "eos_token": {
-    "content": "<|im_end|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": {
-    "content": "<|endoftext|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
-}

tokenizer.json

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:cb3c8fa82993d515469c2800cc455bff4aaa3c4fed9da1f2b0c0668c304f335a
-size 11422691

tokenizer_config.json

@@ -1,240 +0,0 @@
-{
-  "add_bos_token": false,
-  "add_prefix_space": false,
-  "added_tokens_decoder": {
-    "151643": {
-      "content": "<|endoftext|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151644": {
-      "content": "<|im_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151645": {
-      "content": "<|im_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151646": {
-      "content": "<|object_ref_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151647": {
-      "content": "<|object_ref_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151648": {
-      "content": "<|box_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151649": {
-      "content": "<|box_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151650": {
-      "content": "<|quad_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151651": {
-      "content": "<|quad_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151652": {
-      "content": "<|audio_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151653": {
-      "content": "<|audio_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151654": {
-      "content": "<|audio_user_slot|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151655": {
-      "content": "<|image_pad|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151656": {
-      "content": "<|audio_assistant_gen_slot|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151657": {
-      "content": "<tool_call>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151658": {
-      "content": "</tool_call>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151659": {
-      "content": "<|fim_prefix|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151660": {
-      "content": "<|fim_middle|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151661": {
-      "content": "<|fim_suffix|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151662": {
-      "content": "<|audio_assistant_delay_slot|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151663": {
-      "content": "<|repo_name|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151664": {
-      "content": "<|file_sep|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151665": {
-      "content": "<tool_response>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151666": {
-      "content": "</tool_response>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151667": {
-      "content": "<think>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151668": {
-      "content": "</think>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    }
-  },
-  "additional_special_tokens": [
-    "<|im_start|>",
-    "<|im_end|>",
-    "<|object_ref_start|>",
-    "<|object_ref_end|>",
-    "<|box_start|>",
-    "<|box_end|>",
-    "<|quad_start|>",
-    "<|quad_end|>",
-    "<|audio_start|>",
-    "<|audio_end|>",
-    "<|audio_user_slot|>",
-    "<|image_pad|>",
-    "<|audio_assistant_gen_slot|>"
-  ],
-  "bos_token": null,
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "<|im_end|>",
-  "errors": "replace",
-  "extra_special_tokens": {},
-  "model_max_length": 131072,
-  "pad_token": "<|endoftext|>",
-  "processor_class": "AsteroidProcessor",
-  "split_special_tokens": false,
-  "tokenizer_class": "Qwen2Tokenizer",
-  "unk_token": null
-}