Upload 43 files

.gitattributes

@@ -33,3 +33,19 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+aratako_tts/t5gemma-tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/architecture.png filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/en_sample1.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/en_sample2_normal.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/en_sample2_slow.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/gen_sample1.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/gen_sample2.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/gen_sample3.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample1.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample2_fast.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample2_normal.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample2_slow.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/ref_sample1.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/ref_sample2.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/ref_sample3.wav filter=lfs diff=lfs merge=lfs -text
+aratako_tts/T5Gemma-TTS-2b-2b/samples/zh_sample1.wav filter=lfs diff=lfs merge=lfs -text

aratako_tts/Anime-XCodec2-44.1kHz-v2/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

aratako_tts/Anime-XCodec2-44.1kHz-v2/README.md

@@ -0,0 +1,119 @@
+---
+license: cc-by-nc-4.0
+language:
+- ja
+base_model:
+- NandemoGHS/Anime-XCodec2
+pipeline_tag: audio-to-audio
+tags:
+- audio-to-audio
+- speech
+---
+# Anime-XCodec2-44.1kHz-v2: A 44.1kHz Upsampling Variant of Anime-XCodec2 (v2)
+
+[![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)
+
+**TL;DR**: `Anime-XCodec2-44.1kHz-v2` is a fine-tuned variant of **NandemoGHS/Anime-XCodec2**. It incorporates **upsampling layers** and **RMS loss** (inspired by **Inworld TTS-1**) to produce **44.1kHz** output, trained on ~22k hours of Japanese speech. This v2 updates upsampler parameters, loss configurations, and fixes a RoPE bug from the original XCodec2.
+
+Only the **decoder** was updated; the **encoder and codebook remain frozen**, so **speech tokens are identical to the original XCodec2**. This makes the model a drop‑in decoder for downstream systems that already work with XCodec2 tokens (*e.g., Llasa*).
+
+---
+
+## 🔗 Quick Links
+
+* **Demo (Gradio / Hugging Face Spaces)**: [https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-44.1kHz-v2-Demo](https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-44.1kHz-v2-Demo)
+* **This repository (v2 44.1kHz fine-tune)**: `NandemoGHS/Anime-XCodec2-44.1kHz-v2`
+* **Baseline 16kHz model**: `NandemoGHS/Anime-XCodec2`
+* **Original XCodec2**: `HKUSTAudio/xcodec2`
+* **Reference Paper (Inworld TTS-1)**: [https://arxiv.org/abs/2507.21138](https://arxiv.org/abs/2507.21138)
+* **Reference Implementation (Inworld TTS)**: [https://github.com/inworld-ai/tts](https://github.com/inworld-ai/tts)
+
+---
+
+## 1) Model Summary
+
+* **What it is**: A neural speech codec based on **Anime-XCodec2** (which is based on XCodec2), fine-tuned to output **44.1kHz** high-fidelity Japanese speech (anime/game-style). (Version 2)
+* **Key Change**: Integrates an **UpSamplerBlock** and utilizes **RMS Loss** (inspired by [Inworld TTS-1](https://arxiv.org/abs/2507.21138)) into the decoder architecture.
+* **Training scope**: **Decoder-only** fine-tuning on ~**22,000 hours** of Japanese data. **Encoder** and **codebook** are **frozen**.
+* **Compatibility**: **Speech tokens are identical** to `HKUSTAudio/xcodec2` and `NandemoGHS/Anime-XCodec2`.
+* **Input Sampling rate**: **16 kHz** (for encoding, same as XCodec2).
+* **Output Sampling rate**: **44.1 kHz** (decoded audio).
+
+---
+
+## 2) Intended Use
+
+* **Decode XCodec2 speech tokens** (e.g., from Llasa or other AR generators) into **high-fidelity 44.1kHz Japanese speech** (anime/game-style).
+* Upgrade existing `Anime-XCodec2` (16kHz) pipelines to 44.1kHz output.
+* **Audio Super-Resolution**: As the model accepts 16kHz input and outputs 44.1kHz reconstructed audio, it can also be used as a form of audio super-resolution. However, its performance for this specific purpose is untested/unevaluated.
+
+---
+
+## 3) How to Use (Important)
+
+This model **modifies the original XCodec2 architecture** (upsampler blocks) and **requires a custom library version** that includes a fix for the RoPE bug ([Issue #36](https://github.com/zhenye234/X-Codec-2.0/issues/36)).
+
+You **MUST** use the provided custom `xcodec2` library fork (v0.1.7 or later) for inference. The standard library or older custom libraries (like 0.1.6) **will not work**.
+
+* **Installation:**
+    ```bash
+    # Install the custom xcodec2 library (v0.1.7)
+    pip install https://huggingface.co/NandemoGHS/Anime-XCodec2-44.1kHz-v2/resolve/main/xcodec2-0.1.7.tar.gz
+    ```
+
+* **Usage:**
+    Once the custom library is installed, you can load and use this model just as you would the original XCodec2 or Anime-XCodec2 models. The core inference logic remains the same.
+
+For a complete, working code example, please refer to my Hugging Face Spaces Demo: https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-44.1kHz-v2-Demo
+
+---
+
+## 4) Limitations & Trade-offs
+
+* **Language scope**: Optimized for **Japanese**. Performance on other languages may degrade.
+* **Content domain**: Tuned toward **anime/game-style** voices.
+* **Library Dependency**: **Requires the specific custom `xcodec2` library (v0.1.7)** linked above. It is not compatible with the original `xcodec2` library or previous custom forks (e.g., v0.1.6).
+
+---
+
+## 5) Data (High-Level)
+
+* ~**22,000 hours** of Japanese speech, with a focus on **anime/game-style voices**.
+* Data was prepared for 44.1kHz target output during training.
+
+---
+
+## 6) Training Procedure (High-Level)
+
+* **Base Model**: `NandemoGHS/Anime-XCodec2` (16kHz)
+* **Architecture Modification**:
+    * Integrated the `UpSamplerBlock` from the [Inworld TTS-1 implementation](https://github.com/inworld-ai/tts) into the decoder.
+* **Loss Function**:
+    * Adopted **RMS Loss** (Root Mean Square loss) (from Inworld TTS-1), in addition to original losses.
+* **Frozen**: Encoder and Codebook (token compatibility preserved).
+* **Updated (fine-tuned)**: `generator.backbone`, `generator.head`, `generator.upsampler`, `fc_post_a`
+
+### Key Updates in v2
+
+Compared to the first version, this v2 model includes the following key updates to the training configuration:
+
+1.  **RoPE Bug Fix**: Corrected a RoPE (Rotary Position Embedding) bug present in the original XCodec2 implementation (See [Issue #36](https://github.com/zhenye234/X-Codec-2.0/issues/36)).
+2.  **Upsampler Parameters**: The upsampler settings were changed to `hop_length=98`, `upsample_factors=[3, 3]`, and `kernel_sizes=[9, 9]`.
+3.  **Perceptual Loss Model**: The model used for calculating perceptual loss was switched from [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) to [imprt/kushinada-hubert-large](https://huggingface.co/imprt/kushinada-hubert-large).
+4.  **Spectral Discriminator Tuning**: The STFT (Short-Time Fourier Transform) settings for the spectral discriminator were adjusted to be more suitable for 44.1kHz high-sampling-rate audio.
+
+---
+
+## 7) License
+
+* **CC-BY-NC 4.0** (inherited from XCodec2 and Anime-XCodec2).
+* See: [https://creativecommons.org/licenses/by-nc/4.0/](https://creativecommons.org/licenses/by-nc/4.0/)
+
+---
+
+## 8) Acknowledgements
+
+* **HKUSTAudio/xcodec2** (Original model)
+* **Inworld AI** for their work on [Inworld TTS-1](https://arxiv.org/abs/2507.21138) (Upsampler architecture and RMS Loss).
+* **imprt** for the `kushinada-hubert-large` model used in perceptual loss.
+* Thanks to contributors and the community around Japanese speech resources.

aratako_tts/Anime-XCodec2-44.1kHz-v2/ckpt/final.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:037db0b34dd734a805ad7a3dd1d9a2430b2690042631d5e5f6e6cc2028df0e0c
+size 6537383183

aratako_tts/Anime-XCodec2-44.1kHz-v2/config.json

@@ -0,0 +1,22 @@
+{
+  "architectures": [
+    "XCodec2Model"
+  ],
+  "codec_decoder_hidden_size": 1024,
+  "codec_encoder_hidden_size": 1024,
+  "model_type": "xcodec2",
+  "semantic_hidden_size": 1024,
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.0",
+  "use_vocos": true,
+  "hop_length": 98,
+  "sample_rate": 44100,
+  "upsample_factors": [
+    3,
+    3
+  ],
+  "upsample_kernel_sizes": [
+    9,
+    9
+  ]
+}

aratako_tts/Anime-XCodec2-44.1kHz-v2/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9c4ba4d0261d9ec1db0e323273c3e05d68725a055673874551ce78365822352
+size 3320002448

aratako_tts/Anime-XCodec2-44.1kHz-v2/xcodec2-0.1.7.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ae197588f5621f521c7a715d0d7f8985de9a5921d1644c7f0a8078d2c5ade2f
+size 23367

aratako_tts/T5Gemma-TTS-2b-2b/.gitattributes

@@ -0,0 +1,50 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+samples/en_sample1.wav filter=lfs diff=lfs merge=lfs -text
+samples/en_sample2_normal.wav filter=lfs diff=lfs merge=lfs -text
+samples/en_sample2_slow.wav filter=lfs diff=lfs merge=lfs -text
+samples/gen_sample1.wav filter=lfs diff=lfs merge=lfs -text
+samples/gen_sample2.wav filter=lfs diff=lfs merge=lfs -text
+samples/gen_sample3.wav filter=lfs diff=lfs merge=lfs -text
+samples/jp_sample1.wav filter=lfs diff=lfs merge=lfs -text
+samples/jp_sample2_fast.wav filter=lfs diff=lfs merge=lfs -text
+samples/jp_sample2_normal.wav filter=lfs diff=lfs merge=lfs -text
+samples/jp_sample2_slow.wav filter=lfs diff=lfs merge=lfs -text
+samples/ref_sample1.wav filter=lfs diff=lfs merge=lfs -text
+samples/ref_sample2.wav filter=lfs diff=lfs merge=lfs -text
+samples/ref_sample3.wav filter=lfs diff=lfs merge=lfs -text
+samples/zh_sample1.wav filter=lfs diff=lfs merge=lfs -text
+architecture.png filter=lfs diff=lfs merge=lfs -text

aratako_tts/T5Gemma-TTS-2b-2b/GEMMA_PROHIBITED_USE_POLICY.md

@@ -0,0 +1,38 @@
+# Gemma Prohibited Use Policy
+
+Google reserves the right to update this Gemma Prohibited Use Policy from time
+to time.
+
+Last modified: February 21, 2024
+
+You **may not** use nor allow others to use Gemma or Model Derivatives to:
+
+1. Generate any content, including the outputs or results generated by Gemma or Model Derivatives, that infringes, misappropriates, or otherwise violates any individual's or entity's rights (including, but not limited to rights in copyrighted content).
+2. Perform or facilitate dangerous, illegal, or malicious activities, including:
+   1. Facilitation or promotion of illegal activities or violations of law, such as:
+      1. Promoting or generating content related to child sexual abuse or exploitation;
+      2. Promoting or facilitating sale of, or providing instructions for synthesizing or accessing, illegal substances, goods, or services;
+      3. Facilitating or encouraging users to commit any type of crimes; or
+      4. Promoting or generating violent extremism or terrorist content.
+   2. Engagement in the illegal or unlicensed practice of any vocation or profession including, but not limited to, legal, medical, accounting, or financial professional practices.
+   3. Abuse, harm, interference, or disruption of services (or enable others to do the same), such as:
+      1. Promoting or facilitating the generation or distribution of spam; or
+      2. Generating content for deceptive or fraudulent activities, scams, phishing, or malware.
+   4. Attempts to override or circumvent safety filters or intentionally drive Gemma or Model Derivatives to act in a manner that contravenes this Gemma Prohibited Use Policy.
+   5. Generation of content that may harm or promote the harm of individuals or a group, such as:
+      1. Generating content that promotes or encourages hatred;
+      2. Facilitating methods of harassment or bullying to intimidate, abuse, or insult others;
+      3. Generating content that facilitates, promotes, or incites violence;
+      4. Generating content that facilitates, promotes, or encourages self harm;
+      5. Generating personally identifying information for distribution or other harms;
+      6. Tracking or monitoring people without their consent;
+      7. Generating content that may have unfair or adverse impacts on people, particularly impacts related to sensitive or protected characteristics; or
+      8. Generating, gathering, processing, or inferring sensitive personal or private information about individuals without obtaining all rights, authorizations, and consents required by applicable laws.
+3. Generate and distribute content intended to misinform, misrepresent or mislead, including:
+   1. Misrepresentation of the provenance of generated content by claiming content was created by a human, or represent generated content as original works, in order to deceive;
+   2. Generation of content that impersonates an individual (living or dead) without explicit disclosure, in order to deceive;
+   3. Misleading claims of expertise or capability made particularly in sensitive areas (e.g. health, finance, government services, or legal);
+   4. Making automated decisions in domains that affect material or individual rights or well-being (e.g., finance, legal, employment, healthcare, housing, insurance, and social welfare);
+   5. Generation of defamatory content, including defamatory statements, images, or audio content; or
+   6. Engaging in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices.
+4. Generate sexually explicit content, including content created for the purposes of pornography or sexual gratification (e.g. sexual chatbots). Note that this does not include content created for scientific, educational, documentary, or artistic purposes.

aratako_tts/T5Gemma-TTS-2b-2b/GEMMA_TERMS_OF_USE.md

@@ -0,0 +1,124 @@
+<br />
+
+Last modified: March 24, 2025
+
+By using, reproducing, modifying, distributing, performing or displaying any portion or element of Gemma, Model Derivatives including via any Hosted Service, (each as defined below) (collectively, the "**Gemma Services**") or otherwise accepting the terms of this Agreement, you agree to be bound by this Agreement.
+
+## Section 1: DEFINITIONS
+
+### 1.1 Definitions
+
+(a) "**Agreement** " or "**Gemma Terms of Use**" means these terms and conditions that govern the use, reproduction, Distribution or modification of the Gemma Services and any terms and conditions incorporated by reference.
+
+(b) "**Distribution** " or "**Distribute** " means any transmission, publication, or other sharing of Gemma or Model Derivatives to a third party, including by providing or making Gemma or its functionality available as a hosted service via API, web access, or any other electronic or remote means ("**Hosted Service**").
+
+(c) "**Gemma** " means the set of machine learning language models, trained model weights and parameters identified in the[Appendix](https://ai.google.dev/gemma/terms#appendix), regardless of the source that you obtained it from.
+
+(d) "**Google**" means Google LLC.
+
+(e) "**Model Derivatives**" means all (i) modifications to Gemma, (ii) works based on Gemma, or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of Gemma, to that model in order to cause that model to perform similarly to Gemma, including distillation methods that use intermediate data representations or methods based on the generation of synthetic data Outputs by Gemma for training that model. For clarity, Outputs are not deemed Model Derivatives.
+
+(f) "**Output**" means the information content output of Gemma or a Model Derivative that results from operating or otherwise using Gemma or the Model Derivative, including via a Hosted Service.
+
+### 1.2
+
+As used in this Agreement, "**including** " means "**including without limitation**".
+
+## Section 2: ELIGIBILITY AND USAGE
+
+### 2.1 Eligibility
+
+You represent and warrant that you have the legal capacity to enter into this Agreement (including being of sufficient age of consent). If you are accessing or using any of the Gemma Services for or on behalf of a legal entity, (a) you are entering into this Agreement on behalf of yourself and that legal entity, (b) you represent and warrant that you have the authority to act on behalf of and bind that entity to this Agreement and (c) references to "**you** " or "**your**" in the remainder of this Agreement refers to both you (as an individual) and that entity.
+
+### 2.2 Use
+
+You may use, reproduce, modify, Distribute, perform or display any of the Gemma Services only in accordance with the terms of this Agreement, and must not violate (or encourage or permit anyone else to violate) any term of this Agreement.
+
+## Section 3: DISTRIBUTION AND RESTRICTIONS
+
+### 3.1 Distribution and Redistribution
+
+You may reproduce or Distribute copies of Gemma or Model Derivatives if you meet all of the following conditions:
+
+1. You must include the use restrictions referenced in Section 3.2 as an enforceable provision in any agreement (e.g., license agreement, terms of use, etc.) governing the use and/or distribution of Gemma or Model Derivatives and you must provide notice to subsequent users you Distribute to that Gemma or Model Derivatives are subject to the use restrictions in Section 3.2.
+2. You must provide all third party recipients of Gemma or Model Derivatives a copy of this Agreement.
+3. You must cause any modified files to carry prominent notices stating that you modified the files.
+4. All Distributions (other than through a Hosted Service) must be accompanied by a "**Notice** " text file that contains the following notice: "**Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms**".
+
+You may add your own intellectual property statement to your modifications and, except as set forth in this Section, may provide additional or different terms and conditions for use, reproduction, or Distribution of your modifications, or for any such Model Derivatives as a whole, provided your use, reproduction, modification, Distribution, performance, and display of Gemma otherwise complies with the terms and conditions of this Agreement. Any additional or different terms and conditions you impose must not conflict with the terms of this Agreement.
+
+### 3.2 Use Restrictions
+
+You must not use any of the Gemma Services:
+
+1. for the restricted uses set forth in the Gemma Prohibited Use Policy at[ai.google.dev/gemma/prohibited_use_policy](https://ai.google.dev/gemma/prohibited_use_policy)("**Prohibited Use Policy**"), which is hereby incorporated by reference into this Agreement; or
+2. in violation of applicable laws and regulations.
+
+To the maximum extent permitted by law, Google reserves the right to restrict (remotely or otherwise) usage of any of the Gemma Services that Google reasonably believes are in violation of this Agreement.
+
+### 3.3 Generated Output
+
+Google claims no rights in Outputs you generate using Gemma. You and your users are solely responsible for Outputs and their subsequent uses.
+
+## Section 4: ADDITIONAL PROVISIONS
+
+### 4.1 Updates
+
+Google may update Gemma from time to time.
+
+### 4.2 Trademarks
+
+Nothing in this Agreement grants you any rights to use Google's trademarks, trade names, logos or to otherwise suggest endorsement or misrepresent the relationship between you and Google. Google reserves any rights not expressly granted herein.
+
+### 4.3 DISCLAIMER OF WARRANTY
+
+UNLESS REQUIRED BY APPLICABLE LAW, THE GEMMA SERVICES, AND OUTPUTS, ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING, REPRODUCING, MODIFYING, PERFORMING, DISPLAYING OR DISTRIBUTING ANY OF THE GEMMA SERVICES OR OUTPUTS AND ASSUME ANY AND ALL RISKS ASSOCIATED WITH YOUR USE OR DISTRIBUTION OF ANY OF THE GEMMA SERVICES OR OUTPUTS AND YOUR EXERCISE OF RIGHTS AND PERMISSIONS UNDER THIS AGREEMENT.
+
+### 4.4 LIMITATION OF LIABILITY
+
+TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY, CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW, SHALL GOOGLE OR ITS AFFILIATES BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, CONSEQUENTIAL, OR PUNITIVE DAMAGES, OR LOST PROFITS OF ANY KIND ARISING FROM THIS AGREEMENT OR RELATED TO, ANY OF THE GEMMA SERVICES OR OUTPUTS EVEN IF GOOGLE OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+### 4.5 Term, Termination, and Survival
+
+The term of this Agreement will commence upon your acceptance of this Agreement (including acceptance by your use, modification, or Distribution, reproduction, performance or display of any portion or element of the Gemma Services) and will continue in full force and effect until terminated in accordance with the terms of this Agreement. Google may terminate this Agreement if you are in breach of any term of this Agreement. Upon termination of this Agreement, you must delete and cease use and Distribution of all copies of Gemma and Model Derivatives in your possession or control. Sections 1, 2.1, 3.3, 4.2 to 4.9 shall survive the termination of this Agreement.
+
+### 4.6 Governing Law and Jurisdiction
+
+This Agreement will be governed by the laws of the State of California without regard to choice of law principles. The UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement. The state and federal courts of Santa Clara County, California shall have exclusive jurisdiction of any dispute arising out of this Agreement.
+
+### 4.7 Severability
+
+If any provision of this Agreement is held to be invalid, illegal or unenforceable, the remaining provisions shall be unaffected thereby and remain valid as if such provision had not been set forth herein.
+
+### 4.8 Entire Agreement
+
+This Agreement states all the terms agreed between the parties and supersedes all other agreements between the parties as of the date of acceptance relating to its subject matter.
+
+### 4.9 No Waiver
+
+Google will not be treated as having waived any rights by not exercising (or delaying the exercise of) any rights under this Agreement.
+
+## Appendix
+
+- [Gemma 1](https://ai.google.dev/gemma/docs/core/model_card)
+- [Gemma 1.1](https://ai.google.dev/gemma/docs/core/model_card)
+- [Gemma 2](https://ai.google.dev/gemma/docs/core/model_card_2)
+- [Gemma 3](https://ai.google.dev/gemma/docs/core/model_card_3)
+- [Gemma 3n](https://ai.google.dev/gemma/docs/3n)
+- [EmbeddingGemma](https://ai.google.dev/gemma/docs/embeddinggemma)
+- [PaliGemma](https://ai.google.dev/gemma/docs/paligemma/model-card)
+- [PaliGemma 2](https://ai.google.dev/gemma/docs/paligemma/model-card-2)
+- [ShieldGemma](https://ai.google.dev/gemma/docs/shieldgemma/model_card)
+- [ShieldGemma 2](https://ai.google.dev/gemma/docs/shieldgemma/model_card_2)
+- [CodeGemma](https://ai.google.dev/gemma/docs/codegemma/model_card)
+- [CodeGemma 1.1](https://ai.google.dev/gemma/docs/codegemma/model_card)
+- [Gemma 2 JPN](https://huggingface.co/google/gemma-2-2b-jpn-it)
+- [DataGemma RIG](https://www.kaggle.com/models/google/datagemma-rig)
+- [DataGemma RAG](https://www.kaggle.com/models/google/datagemma-rag)
+- [RecurrentGemma](https://ai.google.dev/gemma/docs/recurrentgemma/model_card)
+- [Gemma Scope](https://ai.google.dev/gemma/docs/gemma_scope)
+- [Gemma-APS](https://ai.google.dev/gemma/docs/gemma-aps)
+- [T5Gemma](https://www.kaggle.com/models/google/t5gemma)
+- [VaultGemma](https://www.kaggle.com/models/google/vaultgemma)
+
+| **Note:** Previous versions of these Terms are[archived here](https://ai.google.dev/gemma/terms-archive).

aratako_tts/T5Gemma-TTS-2b-2b/NOTICE

@@ -0,0 +1 @@
+Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

aratako_tts/T5Gemma-TTS-2b-2b/README.md

@@ -0,0 +1,170 @@
+---
+license:
+- gemma
+- cc-by-nc-4.0
+language:
+- en
+- zh
+- ja
+base_model:
+- google/t5gemma-2b-2b-ul2
+pipeline_tag: text-to-speech
+library_name: transformers
+tags:
+- speech
+- tts
+datasets:
+- amphion/Emilia-Dataset
+- pkufool/libriheavy
+extra_gated_heading: License & Ethics Agreement
+extra_gated_description: >-
+  This model is for **Non-Commercial Use Only** (CC-BY-NC 4.0) and follows the **Gemma Terms of Use**.
+  Malicious use, including impersonation, is strictly prohibited.
+extra_gated_button_content: Agree and Access
+---
+
+# T5Gemma-TTS-2b-2b
+
+[![GitHub](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/Aratako/T5Gemma-TTS) [![WandB](https://img.shields.io/badge/Training%20Log-WandB-orange)](https://api.wandb.ai/links/aratako-lm/kfti30sc) [![Demo Space](https://img.shields.io/badge/Demo-HuggingFace%20Space-blue)](https://huggingface.co/spaces/Aratako/T5Gemma-TTS-Demo)
+
+**[日本語版 README はこちら](https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/blob/main/README_ja.md)**
+
+**T5Gemma-TTS-2b-2b** is a multilingual Text-to-Speech (TTS) model developed as a personal project. It utilizes an Encoder-Decoder LLM architecture, supporting English, Chinese, and Japanese.
+
+## 🌟 Overview
+
+This model is an Encoder-Decoder LLM based TTS system initialized from the weights of [google/t5gemma-2b-2b-ul2](https://huggingface.co/google/t5gemma-2b-2b-ul2). While it leverages pre-trained LLM weights, the audio component has been trained from scratch specifically for TTS tasks.
+
+You can try the interactive demo on Hugging Face Spaces: **[T5Gemma-TTS Demo](https://huggingface.co/spaces/Aratako/T5Gemma-TTS-Demo)**
+
+### Key Features
+
+  * **Multilingual Support:** Supports **English, Chinese, and Japanese**.
+  * **Voice Cloning:** Capable of zero-shot voice cloning from reference audio.
+  * **Duration Control:** Allows users to control the speed and length of the generated audio explicitly.
+  * **Open Source Code:** Training code and inference scripts are available on GitHub.
+
+> **Note:** This is a hobby project. There are no formal objective evaluation metrics (WER/CER, SIM-O, etc.) available at this time.
+
+## 🏗️ Technical Details
+
+### Architecture
+
+![](https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/architecture.png)
+
+The architecture is inspired by **VoiceStar** ([arXiv:2505.19462](https://arxiv.org/abs/2505.19462)). It adopts mechanisms such as **PM-RoPE** for length control.
+
+  * **Base Model:** [google/t5gemma-2b-2b-ul2](https://huggingface.co/google/t5gemma-2b-2b-ul2) (Weights used for initialization).
+  * **Audio Codec:** [XCodec2](HKUSTAudio/xcodec2) and its derivatives.
+
+### Training Data
+
+The model was trained on approximately **170,000 hours** of publicly available speech datasets (mainly [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset) and [libriheavy](https://huggingface.co/datasets/pkufool/libriheavy)).
+
+| Language | Approx. Hours |
+| :--- | :--- |
+| **English** | ~100k hours |
+| **Chinese** | ~50k hours |
+| **Japanese** | ~20k hours |
+
+### Training Hardware
+
+Training was conducted on the **AMD Developer Cloud** using **8x MI300X** GPUs for approximately 2 weeks.
+
+  * You can check the training logs here: [WandB](https://api.wandb.ai/links/aratako-lm/kfti30sc)
+
+## 🎧 Audio Samples
+
+Below are some samples generated by T5Gemma-TTS-2b-2b.
+
+### 1. Multilingual TTS
+
+Basic text-to-speech generation in supported languages.
+
+| Language | Text Prompt | Audio |
+| :--- | :--- | :--- |
+| **English** | "The old library was silent, save for the gentle ticking of a clock somewhere in the shadows. As I ran my fingers along the dusty spines of the books, I felt a strange sense of nostalgia, as if I had lived a thousand lives within these walls." | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/en_sample1.wav?download=true"></audio> |
+| **Chinese** | "那是一个宁静的夜晚，月光洒在湖面上，波光粼粼。微风轻拂，带来了远处花朵的清香。我独自坐在岸边，心中涌起一股莫名的感动，仿佛整个世界都在这一刻静止了。" | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/zh_sample1.wav?download=true"></audio> |
+| **Japanese** | "その森には、古い言い伝えがありました。月が最も高く昇る夜、静かに耳を澄ませば、風の歌声が聞こえるというのです。私は半信半疑でしたが、その夜、確かに誰かが私を呼ぶ声を聞いたのです。" | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/jp_sample1.wav?download=true"></audio> |
+
+### 2. Duration Control
+
+Examples of generating the same text with different duration constraints.
+
+**English Sample**
+> Text: *"This new model allows users to strictly control the duration of the generated speech.*
+
+| Target Duration | Generated Audio |
+| :--- | :--- |
+| **3.0s (Fast)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/en_sample2_fast.wav"></audio> |
+| **5.0s (Normal)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/en_sample2_normal.wav"></audio> |
+| **7.0s (Slow)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/en_sample2_slow.wav"></audio> |
+
+**Japanese Sample**
+> Text: *"このモデルでは、生成音声の長さを自由に調整できます。"*
+
+| Target Duration | Generated Audio |
+| :--- | :--- |
+| **3.0s (Fast)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/jp_sample2_fast.wav"></audio> |
+| **5.0s (Normal)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/jp_sample2_normal.wav"></audio> |
+| **7.0s (Slow)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/jp_sample2_slow.wav"></audio> |
+
+### 3. Voice Cloning (Zero-shot)
+
+Examples of cloning a voice from a reference audio clip.
+
+> **Note:** The reference audio samples below were generated using **[NandemoGHS/Anime-Llasa-3B](https://huggingface.co/NandemoGHS/Anime-Llasa-3B)** and **[gemini-2.5-pro-preview-tts](https://cloud.google.com/text-to-speech/docs/gemini-tts)**.
+
+| Case | Reference Audio | Generated Audio |
+| :--- | :--- | :--- |
+| **Example 1** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/ref_sample1.wav"></audio> | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/gen_sample1.wav"></audio> |
+| **Example 2** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/ref_sample2.wav"></audio> | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/gen_sample2.wav"></audio> |
+| **Example 3** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/ref_sample3.wav"></audio> | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b-resources/resolve/main/samples/gen_sample3.wav"></audio> |
+
+## 🚀 Usage
+
+For inference code, installation instructions, and training scripts, please refer to the GitHub repository:
+
+👉 **[GitHub](https://github.com/Aratako/T5Gemma-TTS)**
+
+## ⚠️ Limitations
+
+- **Inference Speed:** The model is not optimized for real-time TTS applications. Autoregressive generation of audio tokens takes significant time, making it unsuitable for low-latency use cases.
+- **Duration Control:** While the model supports explicit duration specification, control is not perfect. Generated audio may differ from the specified duration, and even when the duration matches, the speech pacing or naturalness may not always be optimal.
+- **Audio Quality:** Quality depends on training data characteristics. Performance may vary for voices, accents, or speaking styles underrepresented in the training data.
+
+## 📜 License
+
+This model is released under a **Dual License** policy. Users must strictly comply with **BOTH** of the following sets of terms:
+
+1.  **[Gemma Terms of Use](https://ai.google.dev/gemma/terms):** Since this model is derived from `google/t5gemma-2b-2b-ul2`, you must adhere to the Gemma Terms of Use.
+2.  **[CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/):** Due to the constraints of the training datasets (such as Emilia), this model is restricted to **Non-Commercial Use Only**.
+
+> **⚠️ Important Note on Codec:**
+> The audio codec used, **XCodec2**, is also released under a **CC-BY-NC** license. Please ensure you also follow their license terms when using the generated audio.
+
+**Ethical Restrictions:**
+Do not use this model to impersonate specific individuals (e.g., voice cloning of voice actors, celebrities, or public figures) without their explicit consent.
+
+## 🙏 Acknowledgments
+
+I would like to thank the following for their open-source contributions, which made this project possible:
+
+- [VoiceStar](https://arxiv.org/abs/2505.19462) - Architecture inspiration
+- [T5Gemma](https://huggingface.co/google/t5gemma-2b-2b-ul2) - Base model
+- [XCodec2](https://huggingface.co/HKUSTAudio/xcodec2) and [XCodec2-Variant](https://huggingface.co/NandemoGHS/Anime-XCodec2-44.1kHz-v2) - Audio codec
+
+## 🖊️ Citation
+
+If you cite this model, please cite it as follows:
+
+```bibtex
+@misc{t5gemma-tts,
+  author = {Aratako},
+  title = {T5Gemma-TTS-2b-2b: An Encoder-Decoder LLM-based TTS Model},
+  year = {2025},
+  publisher = {Hugging Face},
+  journal = {Hugging Face repository},
+  howpublished = {\url{https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b}}
+}
+```

aratako_tts/T5Gemma-TTS-2b-2b/README_ja.md

@@ -0,0 +1,162 @@
+---
+license:
+- gemma
+- cc-by-nc-4.0
+language:
+- en
+- zh
+- ja
+base_model:
+- google/t5gemma-2b-2b-ul2
+pipeline_tag: text-to-speech
+library_name: transformers
+tags:
+- speech
+- tts
+datasets:
+- amphion/Emilia-Dataset
+- pkufool/libriheavy
+---
+
+# T5Gemma-TTS-2b-2b
+
+[![GitHub](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/Aratako/T5Gemma-TTS) [![WandB](https://img.shields.io/badge/Training%20Log-WandB-orange)](https://api.wandb.ai/links/aratako-lm/kfti30sc) [![Demo Space](https://img.shields.io/badge/Demo-HuggingFace%20Space-blue)](https://huggingface.co/spaces/Aratako/T5Gemma-TTS-Demo)
+
+**T5Gemma-TTS-2b-2b** は、個人プロジェクトとして開発された多言語対応の音声合成（TTS）モデルです。Encoder-Decoder型のLLMアーキテクチャを採用しており、英語、中国語、日本語に対応しています。
+
+## 🌟 概要
+
+本モデルは、[google/t5gemma-2b-2b-ul2](https://huggingface.co/google/t5gemma-2b-2b-ul2) の重みで初期化されたEncoder-DecoderのLLMベースのTTSシステムです。事前学習済みのLLMの重みを活用していますが、TTSタスク専用に音声部分はスクラッチから学習を行っています。
+
+Hugging Face Spaces にてインタラクティブなデモをお試しいただけます: **[T5Gemma-TTS Demo](https://huggingface.co/spaces/Aratako/T5Gemma-TTS-Demo)**
+
+### 主な特徴
+
+  * **多言語対応:** **英語、中国語、日本語**をサポートしています。
+  * **音声クローン:** 参照音声からのzero-shot voice cloningが可能です。
+  * **発話時間の制御:** 生成される音声の長さをユーザーが明示的に制御可能です。
+  * **オープンソースなコード:** 学習コードおよび推論スクリプトをGitHubで公開しています。
+
+> **Note:** 本モデルは個人開発プロジェクトとして開発されたものです。現時点では、客観的な評価指標（WER/CER、SIM-Oなど）のデータはありません。
+
+## 🏗️ 技術的詳細
+
+### アーキテクチャ
+
+![](https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/architecture.png)
+
+アーキテクチャは **VoiceStar** ([arXiv:2505.19462](https://arxiv.org/abs/2505.19462)) に着想を得て設計されています。長さ制御のための **PM-RoPE** などの機構を採用しています。
+
+  * **ベースモデル:** [google/t5gemma-2b-2b-ul2](https://huggingface.co/google/t5gemma-2b-2b-ul2) (重みの初期化に使用)
+  * **音声コーデック:** [XCodec2](HKUSTAudio/xcodec2) およびその派生版
+
+### 学習データ
+
+約 **170,000時間** の公開音声データセット（主に[Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset)、[libriheavy](https://huggingface.co/datasets/pkufool/libriheavy)）を使用して学習を行いました。
+
+| 言語 | およそのデータ量 |
+| :--- | :--- |
+| **英語** | ~100k 時間 |
+| **中国語** | ~50k 時間 |
+| **日本語** | ~20k 時間 |
+
+### 学習ハードウェア
+
+学習には **AMD Developer Cloud** を使用し、**8x MI300X** GPUを用いて約2週間トレーニングを行いました。
+
+  * 学習ログはこちらで確認できます: [WandB](https://api.wandb.ai/links/aratako-lm/kfti30sc)
+
+## 🎧 音声サンプル
+
+T5Gemma-TTS-2b-2bによって生成されたサンプル音声です。
+
+### 1. 多言語TTS
+
+各言語での基本的なテキスト読み上げの生成例です。
+
+| 言語 | テキストプロンプト | 音声 |
+| :--- | :--- | :--- |
+| **English** | "The old library was silent, save for the gentle ticking of a clock somewhere in the shadows. As I ran my fingers along the dusty spines of the books, I felt a strange sense of nostalgia, as if I had lived a thousand lives within these walls." | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/en_sample1.wav?download=true"></audio> |
+| **Chinese** | "那是一个宁静的夜晚，月光洒在湖面上，波光粼粼。微风轻拂，带来了远处花朵的清香。我独自坐在岸边，心中涌起一股莫名的感动，仿佛整个世界都在这一刻静止了。" | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/zh_sample1.wav?download=true"></audio> |
+| **Japanese** | "その森には、古い言い伝えがありました。月が最も高く昇る夜、静かに耳を澄ませば、風の歌声が聞こえるというのです。私は半信半疑でしたが、その夜、確かに誰かが私を呼ぶ声を聞いたのです。" | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/jp_sample1.wav?download=true"></audio> |
+
+### 2. 発話時間制御
+
+同じテキストを異なる長さの制約を与えて生成した例です。
+
+**英語の例**
+> Text: *"This new model allows users to strictly control the duration of the generated speech.*
+
+| 目標時間 | 生成音声 |
+| :--- | :--- |
+| **3.0s (Fast)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/en_sample2_fast.wav"></audio> |
+| **5.0s (Normal)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/en_sample2_normal.wav"></audio> |
+| **7.0s (Slow)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/en_sample2_slow.wav"></audio> |
+
+**日本語の例**
+> Text: *"このモデルでは、生成音声の長さを自由に調整できます。"*
+
+| 目標時間 | 生成音声 |
+| :--- | :--- |
+| **3.0s (Fast)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/jp_sample2_fast.wav"></audio> |
+| **5.0s (Normal)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/jp_sample2_normal.wav"></audio> |
+| **7.0s (Slow)** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/jp_sample2_slow.wav"></audio> |
+
+### 3. Voice Cloning (Zero-shot)
+
+参照音声を使用したVoice Cloningの例です。
+
+> **Note:** 以下の参照音声は **[NandemoGHS/Anime-Llasa-3B](https://huggingface.co/NandemoGHS/Anime-Llasa-3B)** および **[gemini-2.5-pro-preview-tts](https://cloud.google.com/text-to-speech/docs/gemini-tts)** を使用して生成されたものです。
+
+| ケース | 参照音声 (Reference) | 生成音声 (Generated) |
+| :--- | :--- | :--- |
+| **Example 1** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/ref_sample1.wav"></audio> | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/gen_sample1.wav"></audio> |
+| **Example 2** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/ref_sample2.wav"></audio> | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/gen_sample2.wav"></audio> |
+| **Example 3** | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/ref_sample3.wav"></audio> | <audio controls src="https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b/resolve/main/samples/gen_sample3.wav"></audio> |
+
+## 🚀 使用方法
+
+推論コード、インストール手順、および学習スクリプトについては、以下のGitHubリポジトリを参照してください。
+
+👉 **[GitHub](https://github.com/Aratako/T5Gemma-TTS)**
+
+## ⚠️ 制限事項
+
+- **推論速度:** 本モデルはリアルタイムTTS向けに最適化されていません。音声トークンの自己回帰生成には相応の時間がかかるため、低遅延が求められる用途には適していません。
+- **時間制御:** 生成時間の明示的な指定は可能ですが、制御は完全ではありません。指定した時間と実際の音声長が異なったり、時間が合っていても発話のペースや自然さが損なわれる場合があります。
+- **音質:** 音質は学習データの特徴に依存します。学習データに含まれていない声質、アクセント、話し方などでは性能が低下する可能性があります。
+
+## 📜 ライセンス
+
+本モデルは **デュアルライセンス** ポリシーの下で公開されています。利用者は以下の**両方**の条件を厳守する必要があります。
+
+1.  **[Gemma Terms of Use](https://ai.google.dev/gemma/terms):** 本モデルは `google/t5gemma-2b-2b-ul2` から派生しているため、Gemmaの利用規約に従う必要があります。
+2.  **[CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/):** 学習データセット（Emiliaなど）の制約により、本モデルは **非商用利用のみ (Non-Commercial Use Only)** に制限されます。
+
+> **⚠️ コーデックに関する重要事項:**
+> 使用している音声コーデック **XCodec2** も **CC-BY-NC** ライセンスの下で公開されています。生成された音声を使用する際は、こちらのライセンス条件も遵守してください。
+
+**倫理的制約:**
+本モデルを使用して、特定の個人（声優、著名人、公人など）の声を、本人の明示的な同意なくクローン（なりすまし）することを禁止します。
+
+## 🙏 謝辞
+
+本プロジェクトを実現するにあたり、以下のオープンソースプロジェクトおよびリソースに感謝いたします。
+
+- [VoiceStar](https://arxiv.org/abs/2505.19462) - アーキテクチャの着想元
+- [T5Gemma](https://huggingface.co/google/t5gemma-2b-2b-ul2) - ベースモデル
+- [XCodec2](https://huggingface.co/HKUSTAudio/xcodec2) および [XCodec2-Variant](https://huggingface.co/NandemoGHS/Anime-XCodec2-44.1kHz-v2) - 音声コーデック
+
+## 🖊️ 引用 (Citation)
+
+本モデルを引用する場合は、以下のように引用してください。
+
+```bibtex
+@misc{t5gemma-tts,
+  author = {Aratako},
+  title = {T5Gemma-TTS-2b-2b: An Encoder-Decoder LLM-based TTS Model},
+  year = {2025},
+  publisher = {Hugging Face},
+  journal = {Hugging Face repository},
+  howpublished = {\url{[https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b](https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b)}}
+}

aratako_tts/T5Gemma-TTS-2b-2b/ckpt/pretrained.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b901434868b1b9d881880c3397802ed19a0718c3f71e22e7361486f072f20e9
+size 10629148381

aratako_tts/T5Gemma-TTS-2b-2b/config.json

@@ -0,0 +1,349 @@
+{
+  "add_bos_to_text": 0,
+  "add_eos_to_text": 1,
+  "architectures": [
+    "T5GemmaVoiceForConditionalGeneration"
+  ],
+  "attn_implementation": "sdpa",
+  "audio_mask_token": 1024,
+  "audio_max_length": 40.0,
+  "audio_pad_token": 65538,
+  "audio_tokenizer": "xcodec2",
+  "audio_vocab_size": 65536,
+  "auto_map": {
+    "AutoConfig": "configuration_t5gemma_voice.T5GemmaVoiceConfig",
+    "AutoModelForSeq2SeqLM": "modeling_t5gemma_voice.T5GemmaVoiceForConditionalGeneration"
+  },
+  "bos_token_id": 65536,
+  "codec_audio_sr": 44100,
+  "dtype": "bfloat16",
+  "empty_token": 65536,
+  "encodec_sr": 50.0,
+  "eog": 65537,
+  "eos": 65539,
+  "eos_token_id": 65539,
+  "extra_cutoff": 5,
+  "model_type": "t5gemma_voice",
+  "n_codebooks": 1,
+  "n_special": 5,
+  "pad_token_id": 65538,
+  "parallel_pattern": 0,
+  "precision": "bfloat16",
+  "progress_lookahead_secs": 2.0,
+  "progress_scale": 2000.0,
+  "prune_text_modules": 2,
+  "special_first": 0,
+  "t5_config_dict": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": [
+      "T5GemmaForConditionalGeneration"
+    ],
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 2,
+    "chunk_size_feed_forward": 0,
+    "classifier_dropout_rate": 0.0,
+    "cross_attention_hidden_size": null,
+    "decoder": {
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_bias": false,
+      "attention_dropout": 0.0,
+      "attn_logit_softcapping": 50.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": 2,
+      "chunk_size_feed_forward": 0,
+      "classifier_dropout_rate": 0.0,
+      "cross_attention_hidden_size": 2304,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout_rate": 0.0,
+      "dtype": "bfloat16",
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": 1,
+      "exponential_decay_length_penalty": null,
+      "final_logit_softcapping": 30.0,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "head_dim": 256,
+      "hidden_activation": "gelu_pytorch_tanh",
+      "hidden_size": 2304,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "initializer_range": 0.02,
+      "intermediate_size": 9216,
+      "is_decoder": true,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_types": [
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention"
+      ],
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "max_position_embeddings": 8192,
+      "min_length": 0,
+      "model_type": "t5_gemma_module",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 8,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_hidden_layers": 26,
+      "num_key_value_heads": 4,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": 0,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "query_pre_attn_scalar": 256,
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "rms_norm_eps": 1e-06,
+      "rope_theta": 10000.0,
+      "sep_token_id": null,
+      "sliding_window": 4096,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_input_output_embeddings": false,
+      "tie_word_embeddings": false,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false,
+      "use_cache": true,
+      "vocab_size": 256000
+    },
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout_rate": 0.0,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder": {
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_bias": false,
+      "attention_dropout": 0.0,
+      "attn_logit_softcapping": 50.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": 2,
+      "chunk_size_feed_forward": 0,
+      "classifier_dropout_rate": 0.0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout_rate": 0.0,
+      "dtype": "bfloat16",
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": 1,
+      "exponential_decay_length_penalty": null,
+      "final_logit_softcapping": 30.0,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "head_dim": 256,
+      "hidden_activation": "gelu_pytorch_tanh",
+      "hidden_size": 2304,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "initializer_range": 0.02,
+      "intermediate_size": 9216,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_types": [
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention",
+        "sliding_attention",
+        "full_attention"
+      ],
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "max_position_embeddings": 8192,
+      "min_length": 0,
+      "model_type": "t5_gemma_module",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 8,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_hidden_layers": 26,
+      "num_key_value_heads": 4,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": 0,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "query_pre_attn_scalar": 256,
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "rms_norm_eps": 1e-06,
+      "rope_theta": 10000.0,
+      "sep_token_id": null,
+      "sliding_window": 4096,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_input_output_embeddings": false,
+      "tie_word_embeddings": false,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false,
+      "use_cache": true,
+      "vocab_size": 256000
+    },
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": [
+      1,
+      107
+    ],
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "is_decoder": false,
+    "is_encoder_decoder": true,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "t5gemma",
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 0,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_input_output_embeddings": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "transformers_version": "4.57.3",
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 256000
+  },
+  "t5gemma_model_name": "google/t5gemma-2b-2b-ul2",
+  "text_guard_frames_per_token": 0,
+  "text_input_type": "text",
+  "text_tokenizer_name": "google/t5gemma-2b-2b-ul2",
+  "tie_input_output_embeddings": false,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.3",
+  "use_pm_rope": 1,
+  "x_sep_token": 255999,
+  "xcodec2_model_name": "NandemoGHS/Anime-XCodec2-44.1kHz-v2",
+  "y_sep_token": 65540
+}

aratako_tts/T5Gemma-TTS-2b-2b/configuration_t5gemma_voice.py

@@ -0,0 +1,119 @@
+"""
+Configuration for inference-only T5GemmaVoice model.
+
+Kept intentionally minimal: only fields that affect inference-time shapes
+or sampling behaviour are retained.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional, Union
+
+from transformers.configuration_utils import PretrainedConfig
+
+
+class T5GemmaVoiceConfig(PretrainedConfig):
+    model_type = "t5gemma_voice"
+    is_encoder_decoder = True
+
+    def __init__(
+        self,
+        # backbone
+        t5gemma_model_name: str = "google/t5gemma-2b-2b-ul2",
+        t5_config_dict: Optional[Dict[str, Any]] = None,
+        attn_implementation: str = "eager",
+        precision: str = "float32",
+        prune_text_modules: int = 0,
+        use_pm_rope: int = 1,
+        tie_word_embeddings: Optional[bool] = None,
+        tie_input_output_embeddings: Optional[bool] = None,
+        n_codebooks: int = 1,
+        audio_vocab_size: Union[int, List[int]] = 65536,
+        n_special: int = 5,
+        empty_token: int = 65536,
+        eog: int = 65537,
+        eos: int = 65539,
+        audio_pad_token: int = 65538,
+        audio_mask_token: int = 1024,
+        y_sep_token: int = 65540,
+        x_sep_token: int = 255999,
+        special_first: int = 0,
+        encodec_sr: float = 50.0,
+        progress_scale: float = 2000.0,
+        progress_lookahead_secs: float = 2.0,
+        extra_cutoff: float = 5.0,
+        text_guard_frames_per_token: int = 0,
+        add_eos_to_text: int = 0,
+        add_bos_to_text: int = 0,
+        parallel_pattern: int = 0,
+        audio_max_length: float = 40.0,
+        audio_tokenizer: str = "xcodec2",
+        xcodec2_model_name: Optional[str] = None,
+        codec_audio_sr: Optional[float] = None,
+        text_tokenizer_name: Optional[str] = None,
+        # misc
+        **kwargs,
+    ) -> None:
+        kwargs = dict(kwargs)
+        # avoid duplicate values when loading from config.json that already stores these ids
+        for _key in ("bos_token_id", "eos_token_id", "pad_token_id"):
+            kwargs.pop(_key, None)
+
+        super().__init__(
+            bos_token_id=empty_token,
+            eos_token_id=eos,
+            pad_token_id=audio_pad_token,
+            **kwargs,
+        )
+
+        # store backbone config for offline instantiation
+        self.t5_config_dict = t5_config_dict
+        self.t5gemma_model_name = t5gemma_model_name
+        self.attn_implementation = attn_implementation
+        self.precision = precision
+        self.prune_text_modules = prune_text_modules
+        self.use_pm_rope = use_pm_rope
+        self.tie_word_embeddings = tie_word_embeddings
+        self.tie_input_output_embeddings = tie_input_output_embeddings
+
+        self.text_input_type = "text"
+        self.n_codebooks = n_codebooks
+        self.audio_vocab_size = audio_vocab_size
+        self.n_special = n_special
+        self.empty_token = empty_token
+        self.eog = eog
+        self.eos = eos
+        self.audio_pad_token = audio_pad_token
+        self.audio_mask_token = audio_mask_token
+        self.y_sep_token = y_sep_token
+        self.x_sep_token = x_sep_token
+        self.special_first = special_first
+        self.encodec_sr = encodec_sr
+        self.progress_scale = progress_scale
+        self.progress_lookahead_secs = progress_lookahead_secs
+        self.extra_cutoff = extra_cutoff
+        self.text_guard_frames_per_token = text_guard_frames_per_token
+        self.add_eos_to_text = add_eos_to_text
+        self.add_bos_to_text = add_bos_to_text
+        self.parallel_pattern = parallel_pattern
+        self.audio_max_length = audio_max_length
+        self.audio_tokenizer = audio_tokenizer
+        self.xcodec2_model_name = xcodec2_model_name
+        self.codec_audio_sr = codec_audio_sr
+        self.text_tokenizer_name = text_tokenizer_name
+
+        # tell Auto* which files to load when trust_remote_code=True
+        self.auto_map = {
+            "AutoConfig": "configuration_t5gemma_voice.T5GemmaVoiceConfig",
+            "AutoModelForSeq2SeqLM": "modeling_t5gemma_voice.T5GemmaVoiceForConditionalGeneration",
+        }
+
+    @property
+    def audio_vocab_sizes(self) -> List[int]:
+        """Utility to normalize audio_vocab_size to list form."""
+        if isinstance(self.audio_vocab_size, list):
+            return list(self.audio_vocab_size)
+        return [int(self.audio_vocab_size)] * int(self.n_codebooks)
+
+
+__all__ = ["T5GemmaVoiceConfig"]

aratako_tts/T5Gemma-TTS-2b-2b/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 65536,
+  "eos_token_id": 65539,
+  "pad_token_id": 65538,
+  "transformers_version": "4.57.3"
+}

aratako_tts/T5Gemma-TTS-2b-2b/model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12a5d05741e9c00c966521b5692810785f882a5141d4ae440deefcdd2886cab5
+size 4988044752

aratako_tts/T5Gemma-TTS-2b-2b/model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f548b2f61fb3f0771913c1e6f03dc387ec9a6dc4b7a63050b10e1e3f72ae0c29
+size 4997791840

aratako_tts/T5Gemma-TTS-2b-2b/model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2c60fba107fb9ee6051c4fbe69ad65f42312d1765d559c614ba912f471ee816
+size 643100482

aratako_tts/T5Gemma-TTS-2b-2b/model.safetensors.index.json

@@ -0,0 +1,744 @@
+{
+  "metadata": {
+    "total_parameters": 5314418949,
+    "total_size": 10628837898
+  },
+  "weight_map": {
+    "audio_embedding.0.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.0.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.1.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.10.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.2.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.22.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.23.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.24.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.cross_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.cross_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.cross_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.cross_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.post_cross_attn_layernorm.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.pre_cross_attn_layernorm.weight": "model-00003-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.25.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.3.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.4.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.5.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.6.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.7.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.8.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.cross_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.cross_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.cross_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.cross_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.post_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.pre_cross_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.layers.9.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.decoder.norm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.11.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.12.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.13.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.14.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.15.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.16.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.17.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.18.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.19.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.20.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.21.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.22.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.23.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.24.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.post_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.post_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.pre_feedforward_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.pre_self_attn_layernorm.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.25.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.post_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.pre_feedforward_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.pre_self_attn_layernorm.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "backbone.model.encoder.norm.weight": "model-00001-of-00003.safetensors",
+    "predict_layer.0.0.bias": "model-00003-of-00003.safetensors",
+    "predict_layer.0.0.weight": "model-00003-of-00003.safetensors",
+    "predict_layer.0.2.bias": "model-00003-of-00003.safetensors",
+    "predict_layer.0.2.weight": "model-00003-of-00003.safetensors"
+  }
+}

aratako_tts/T5Gemma-TTS-2b-2b/modeling_t5gemma_voice.py

@@ -0,0 +1,833 @@
+"""
+Hugging Face compatible wrapper of the T5Gemma-TTS model.
+
+This is largely a drop-in copy of `models/t5gemma.py`, but inherits
+`PreTrainedModel` so that it can be loaded via `AutoModelForSeq2SeqLM` with
+`trust_remote_code=True`. Only the inference-oriented pieces are kept intact.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Callable, Dict, List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import AutoModelForSeq2SeqLM, PreTrainedModel
+from transformers.cache_utils import Cache
+from transformers.generation import GenerationMixin
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.models.t5gemma.modeling_t5gemma import (
+    ALL_ATTENTION_FUNCTIONS,
+    EncoderDecoderCache,
+    T5GemmaCrossAttention,
+    T5GemmaDecoderLayer,
+    T5GemmaRotaryEmbedding,
+    eager_attention_forward,
+    rotate_half,
+)
+
+try:
+    from .configuration_t5gemma_voice import T5GemmaVoiceConfig
+except ImportError:  # when executed inside the repo package
+    from hf_export.configuration_t5gemma_voice import T5GemmaVoiceConfig
+
+logger = logging.getLogger(__name__)
+
+
+def make_pad_mask(lengths: torch.Tensor, max_len: int = 0) -> torch.Tensor:
+    """Return Bool mask [B, T] where True indicates padding."""
+    assert lengths.ndim == 1, lengths.ndim
+    max_len = max(max_len, lengths.max())
+    n = lengths.size(0)
+    seq_range = torch.arange(0, max_len, device=lengths.device)
+    expanded_lengths = seq_range.unsqueeze(0).expand(n, max_len)
+    return expanded_lengths >= lengths.unsqueeze(-1)
+
+
+def top_k_top_p_filtering(
+    logits,
+    top_k=0,
+    top_p=1.0,
+    min_p=0.0,
+    filter_value=-float("Inf"),
+    min_tokens_to_keep=1,
+):
+    min_p_enabled = 0.0 < min_p < 1.0
+    if min_p_enabled:
+        probs = F.softmax(logits, dim=-1)
+        indices_to_remove = probs < min_p
+        if torch.all(indices_to_remove.sum(-1) < logits.size(-1)):
+            logits = logits.masked_fill(indices_to_remove, filter_value)
+            top_k = 0
+            top_p = 1.0
+
+    if isinstance(top_k, int) and top_k > 0:
+        top_k = min(max(top_k, min_tokens_to_keep), logits.size(-1))
+        threshold = torch.topk(logits, top_k, dim=-1)[0][..., -1, None]
+        indices_to_remove = logits < threshold
+        logits[indices_to_remove] = filter_value
+    elif isinstance(top_k, list):
+        assert len(top_k) == logits.size(
+            0
+        ), f"top_k list length ({len(top_k)}) must match logits.size(0) ({logits.size(0)})"
+        for i in range(logits.size(0)):
+            k_i = top_k[i]
+            if k_i > 0:
+                k_i = min(max(k_i, min_tokens_to_keep), logits.size(-1))
+                row_threshold = torch.topk(logits[i], k_i, dim=-1)[0][-1]
+                indices_to_remove_i = logits[i] < row_threshold
+                logits[i, indices_to_remove_i] = filter_value
+
+    if top_p < 1.0:
+        sorted_logits, sorted_indices = torch.sort(logits, descending=True)
+        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
+        sorted_indices_to_remove = cumulative_probs > top_p
+        if min_tokens_to_keep > 1:
+            sorted_indices_to_remove[..., :min_tokens_to_keep] = 0
+        sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
+        sorted_indices_to_remove[..., 0] = 0
+
+        indices_to_remove = torch.zeros_like(logits, dtype=torch.bool)
+        indices_to_remove.scatter_(dim=-1, index=sorted_indices, src=sorted_indices_to_remove)
+        logits = logits.masked_fill(indices_to_remove, filter_value)
+    return logits
+
+
+def topk_sampling(logits, top_k=10, top_p=1.0, min_p=0.0, temperature=1.0):
+    if temperature != 1.0:
+        logits = logits / temperature
+    logits = top_k_top_p_filtering(logits, top_k=top_k, top_p=top_p, min_p=min_p)
+    token = torch.multinomial(F.softmax(logits, dim=-1), num_samples=1)
+    return token
+
+
+class PMCrossAttention(T5GemmaCrossAttention):
+    """T5Gemma cross-attention augmented with Progress-Monitoring RoPE."""
+
+    def __init__(self, config, layer_idx: int):
+        super().__init__(config=config, layer_idx=layer_idx)
+        self.decoder_rotary_emb = T5GemmaRotaryEmbedding(config=config)
+        self.encoder_rotary_emb = T5GemmaRotaryEmbedding(config=config)
+
+    @staticmethod
+    def _apply_rotary_with_progress(
+        projected_states: torch.Tensor,
+        base_states: torch.Tensor,
+        position_ids: Optional[torch.Tensor],
+        rotary_module: T5GemmaRotaryEmbedding,
+    ) -> torch.Tensor:
+        if position_ids is None:
+            return projected_states
+        cos, sin = rotary_module(base_states, position_ids)
+        cos = cos.unsqueeze(1).to(
+            dtype=projected_states.dtype, device=projected_states.device
+        )
+        sin = sin.unsqueeze(1).to(
+            dtype=projected_states.dtype, device=projected_states.device
+        )
+        return (projected_states * cos) + (rotate_half(projected_states) * sin)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor],
+        encoder_hidden_states: Optional[torch.Tensor],
+        past_key_values: Optional[Cache] = None,
+        pm_decoder_position_ids: Optional[torch.Tensor] = None,
+        pm_encoder_position_ids: Optional[torch.Tensor] = None,
+        **kwargs: FlashAttentionKwargs,
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor], Optional[tuple[torch.Tensor]]]:
+        if encoder_hidden_states is None:
+            raise ValueError("Encoder hidden state is required for cross attention.")
+
+        pm_decoder_position_ids = kwargs.pop(
+            "pm_decoder_position_ids", pm_decoder_position_ids
+        )
+        pm_encoder_position_ids = kwargs.pop(
+            "pm_encoder_position_ids", pm_encoder_position_ids
+        )
+
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+        if pm_decoder_position_ids is not None:
+            query_states = self._apply_rotary_with_progress(
+                query_states,
+                hidden_states,
+                pm_decoder_position_ids,
+                self.decoder_rotary_emb,
+            )
+
+        if past_key_values is not None:
+            is_updated = past_key_values.is_updated.get(self.layer_idx)
+            curr_past_key_values = past_key_values.cross_attention_cache
+
+        if past_key_values is None or not is_updated:
+            encoder_input_shape = encoder_hidden_states.shape[:-1]
+            encoder_hidden_shape = (*encoder_input_shape, -1, self.head_dim)
+            key_states = (
+                self.k_proj(encoder_hidden_states)
+                .view(encoder_hidden_shape)
+                .transpose(1, 2)
+            )
+            if pm_encoder_position_ids is not None:
+                key_states = self._apply_rotary_with_progress(
+                    key_states,
+                    encoder_hidden_states,
+                    pm_encoder_position_ids,
+                    self.encoder_rotary_emb,
+                )
+            value_states = (
+                self.v_proj(encoder_hidden_states)
+                .view(encoder_hidden_shape)
+                .transpose(1, 2)
+            )
+
+            if past_key_values is not None:
+                key_states, value_states = curr_past_key_values.update(
+                    key_states, value_states, self.layer_idx
+                )
+                past_key_values.is_updated[self.layer_idx] = True
+        else:
+            key_states = curr_past_key_values.layers[self.layer_idx].keys
+            value_states = curr_past_key_values.layers[self.layer_idx].values
+
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[
+                self.config._attn_implementation
+            ]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=self.attention_dropout if self.training else 0.0,
+            scaling=self.scaling,
+            sliding_window=None,
+            softcap=self.attn_logit_softcapping,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class PMDecoderLayer(T5GemmaDecoderLayer):
+    """Decoder layer variant with PM-RoPE cross-attention built in."""
+
+    def __init__(self, config, layer_idx: int):
+        super().__init__(config, layer_idx)
+        self.cross_attn = PMCrossAttention(config=config, layer_idx=layer_idx)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[EncoderDecoderCache] = None,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.Tensor] = None,
+        pm_decoder_position_ids: Optional[torch.Tensor] = None,
+        pm_encoder_position_ids: Optional[torch.Tensor] = None,
+        **kwargs,
+    ) -> torch.FloatTensor:
+        pm_decoder_position_ids = kwargs.pop(
+            "pm_decoder_position_ids", pm_decoder_position_ids
+        )
+        pm_encoder_position_ids = kwargs.pop(
+            "pm_encoder_position_ids", pm_encoder_position_ids
+        )
+
+        residual = hidden_states
+        hidden_states = self.pre_self_attn_layernorm(hidden_states)
+        hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states,
+            position_embeddings=position_embeddings,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=(
+                past_key_values.self_attention_cache
+                if past_key_values is not None
+                else None
+            ),
+            use_cache=use_cache,
+            cache_position=cache_position,
+            **kwargs,
+        )
+        hidden_states = self.post_self_attn_layernorm(hidden_states)
+        hidden_states = residual + self.dropout(hidden_states)
+
+        residual = hidden_states
+        hidden_states = self.pre_cross_attn_layernorm(hidden_states)
+        hidden_states, _ = self.cross_attn(
+            hidden_states=hidden_states,
+            encoder_hidden_states=encoder_hidden_states,
+            attention_mask=encoder_attention_mask,
+            past_key_values=past_key_values,
+            pm_decoder_position_ids=pm_decoder_position_ids,
+            pm_encoder_position_ids=pm_encoder_position_ids,
+            **kwargs,
+        )
+        hidden_states = self.post_cross_attn_layernorm(hidden_states)
+        hidden_states = residual + self.dropout(hidden_states)
+
+        residual = hidden_states
+        hidden_states = self.pre_feedforward_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = self.post_feedforward_layernorm(hidden_states)
+        hidden_states = residual + self.dropout(hidden_states)
+        return hidden_states
+
+
+def _make_args_from_config(config: T5GemmaVoiceConfig):
+    """Namespace-like shim; keeps attribute access identical to training code."""
+
+    class _Obj:
+        pass
+
+    o = _Obj()
+    for k, v in config.to_dict().items():
+        setattr(o, k, v)
+    return o
+
+
+class T5GemmaVoiceForConditionalGeneration(PreTrainedModel, GenerationMixin):
+    config_class = T5GemmaVoiceConfig
+    base_model_prefix = "backbone"
+    _keys_to_ignore_on_save = ["encoder_module", "decoder_module"]
+
+    def __init__(self, config: T5GemmaVoiceConfig):
+        super().__init__(config)
+        # keep compatibility with original code that expects self.args
+        self.args = _make_args_from_config(config)
+        if getattr(self.args, "n_codebooks", 1) != 1:
+            logging.info("Resetting n_codebooks to 1 for XCodec2 backend.")
+            self.args.n_codebooks = 1
+
+        logging.info(f"Loading T5Gemma backbone: {self.args.t5gemma_model_name}")
+        precision = getattr(self.args, "precision", "float32")
+        if precision == "float16":
+            dtype = torch.float16
+        elif precision == "bfloat16":
+            dtype = torch.bfloat16
+        else:
+            dtype = torch.float32
+
+        # instantiate backbone from config to avoid weight download during load
+        if config.t5_config_dict is not None:
+            from transformers.models.t5gemma import T5GemmaConfig
+
+            base_cfg = T5GemmaConfig(**config.t5_config_dict)
+            base_cfg._attn_implementation = getattr(
+                self.args, "attn_implementation", "eager"
+            )
+            # Force bf16/specified dtype initialization and disable all tying.
+            base_cfg.tie_word_embeddings = False
+            base_cfg.tie_input_output_embeddings = False
+            base_cfg.tie_encoder_decoder = False
+            if hasattr(base_cfg, "encoder"):
+                base_cfg.encoder.tie_word_embeddings = False
+                base_cfg.encoder.tie_input_output_embeddings = False
+                base_cfg.encoder.tie_encoder_decoder = False
+            if hasattr(base_cfg, "decoder"):
+                base_cfg.decoder.tie_word_embeddings = False
+                base_cfg.decoder.tie_input_output_embeddings = False
+                base_cfg.decoder.tie_encoder_decoder = False
+            self.backbone = AutoModelForSeq2SeqLM.from_config(
+                base_cfg, torch_dtype=dtype
+            )
+        else:
+            self.backbone = AutoModelForSeq2SeqLM.from_pretrained(
+                self.args.t5gemma_model_name,
+                attn_implementation=getattr(self.args, "attn_implementation", "eager"),
+                torch_dtype=dtype,
+            )
+
+        prune_text_modules = getattr(self.args, "prune_text_modules", 0)
+        drop_lm_head = prune_text_modules >= 1
+        drop_decoder_embed = prune_text_modules >= 2
+
+        if drop_lm_head and hasattr(self.backbone, "lm_head"):
+            del self.backbone.lm_head
+            self.backbone.lm_head = nn.Identity()
+            if hasattr(self.backbone.config, "tie_word_embeddings"):
+                self.backbone.config.tie_word_embeddings = False
+            logging.info("lm_head removed (prune_text_modules=%d)", prune_text_modules)
+
+        if drop_decoder_embed:
+            decoder = getattr(
+                self.backbone, "model", getattr(self.backbone, "decoder", None)
+            )
+            decoder = getattr(decoder, "decoder", decoder)
+            if decoder is not None and hasattr(decoder, "embed_tokens"):
+                del decoder.embed_tokens
+                decoder.embed_tokens = nn.Identity()
+                if hasattr(self.backbone.config, "tie_word_embeddings"):
+                    self.backbone.config.tie_word_embeddings = False
+                logging.info(
+                    "decoder.embed_tokens removed (prune_text_modules=%d)",
+                    prune_text_modules,
+                )
+
+        # This wrapper is inference-only, so keep cache enabled.
+        self.backbone.config.use_cache = True
+
+        if hasattr(self.backbone, "model"):
+            self.encoder_module = self.backbone.model.encoder
+            self.decoder_module = self.backbone.model.decoder
+        else:
+            self.encoder_module = getattr(self.backbone, "encoder", None)
+            self.decoder_module = getattr(self.backbone, "decoder", None)
+        if self.encoder_module is None or self.decoder_module is None:
+            raise AttributeError(
+                "Failed to locate encoder/decoder modules on T5Gemma backbone."
+            )
+
+        config_hidden_size = getattr(self.backbone.config, "d_model", None)
+        if config_hidden_size is None:
+            config_hidden_size = getattr(self.backbone.config, "hidden_size", None)
+        if config_hidden_size is None:
+            enc = getattr(self.backbone.config, "encoder", None)
+            if enc is not None:
+                config_hidden_size = getattr(enc, "hidden_size", None)
+        if config_hidden_size is None:
+            raise AttributeError("T5Gemma config does not expose d_model/hidden_size.")
+
+        self.hidden_size = config_hidden_size
+        self.args.audio_embedding_dim = getattr(
+            self.args, "audio_embedding_dim", self.hidden_size
+        )
+
+        self._enable_pm_rope_cross_attention()
+
+        self.text_input_type = "text"  # fixed
+        self.text_embedding = None
+        self.text_dropout = nn.Identity()
+
+        if isinstance(self.args.audio_vocab_size, list):
+            audio_vocab_sizes = [
+                size + self.args.n_special for size in self.args.audio_vocab_size
+            ]
+        else:
+            audio_vocab_sizes = [
+                self.args.audio_vocab_size + self.args.n_special
+            ] * self.args.n_codebooks
+        self.n_audio_tokens = audio_vocab_sizes
+
+        self.audio_embedding = nn.ModuleList(
+            [
+                nn.Embedding(audio_vocab_sizes[k], self.hidden_size)
+                for k in range(self.args.n_codebooks)
+            ]
+        )
+        self.audio_dropout = nn.Dropout(0.0)
+
+        self.predict_layer = nn.ModuleList(
+            [
+                nn.Sequential(
+                    nn.Linear(self.hidden_size, self.hidden_size),
+                    nn.GELU(),
+                    nn.Linear(self.hidden_size, audio_vocab_sizes[k]),
+                )
+                for k in range(self.args.n_codebooks)
+            ]
+        )
+        self.progress_scale = getattr(self.args, "progress_scale", 2000.0)
+
+    def get_output_embeddings(self):
+        return None
+
+    def set_output_embeddings(self, new_embeddings):
+        raise NotImplementedError("Output embeddings are pruned in this model.")
+
+    # avoid transformers default tying logic (lm_head is removed)
+    def tie_weights(self):
+        return
+
+    def get_encoder(self):
+        return self.encoder_module
+
+    def get_decoder(self):
+        return self.decoder_module
+
+    def state_dict(self, *args, **kwargs):  # pragma: no cover - save hook
+        sd = super().state_dict(*args, **kwargs)
+        drop_keys = [
+            k
+            for k in sd
+            if k.startswith("encoder_module.") or k.startswith("decoder_module.")
+        ]
+        for k in drop_keys:
+            sd.pop(k)
+        return sd
+
+    def _progress_positions_single(self, length: int, device) -> torch.Tensor:
+        if length <= 0:
+            return torch.zeros(0, device=device, dtype=torch.float32)
+        if length == 1:
+            return torch.zeros(1, device=device, dtype=torch.float32)
+        base = torch.arange(length, device=device, dtype=torch.float32)
+        return base / (length - 1) * self.progress_scale
+
+    def _build_position_ids(
+        self, lengths: torch.Tensor, max_len: int, device
+    ) -> torch.Tensor:
+        # Vectorized implementation: avoid Python loop over batch dimension.
+        # Ensure lengths is on the correct device to prevent device mismatch.
+        lengths = lengths.to(device=device)
+        pos = torch.arange(max_len, device=device, dtype=torch.float32)[None, :]  # [1, T]
+
+        # Clamp denominator to avoid division by zero for length <= 1.
+        # For length 0 or 1, result will be masked to zero anyway.
+        denom = (lengths.clamp(min=2).to(torch.float32) - 1.0)[:, None]  # [B, 1]
+        position_ids = pos / denom * self.progress_scale  # [B, T]
+
+        # Mask out positions beyond each sequence's length.
+        mask = pos < lengths[:, None]  # [B, T] (bool)
+        return position_ids.masked_fill(~mask, 0.0)
+
+    def _enable_pm_rope_cross_attention(self) -> None:
+        if getattr(self, "_pm_rope_enabled", False):
+            return
+        if not getattr(self.args, "use_pm_rope", 1):
+            logging.info("PM-RoPE cross-attention disabled by config.")
+            return
+        decoder_layers = getattr(self.decoder_module, "layers", None)
+        if decoder_layers is None:
+            logging.warning(
+                "Decoder module does not expose layers attribute; skipping PM-RoPE injection."
+            )
+            return
+
+        new_layers = nn.ModuleList()
+        for layer in decoder_layers:
+            pm_layer = PMDecoderLayer(layer.config, layer.layer_idx)
+            pm_layer.load_state_dict(layer.state_dict(), strict=False)
+            pm_layer.gradient_checkpointing = getattr(
+                layer, "gradient_checkpointing", False
+            )
+            if hasattr(layer, "_gradient_checkpointing_func"):
+                pm_layer._gradient_checkpointing_func = (
+                    layer._gradient_checkpointing_func
+                )
+            new_layers.append(pm_layer)
+        self.decoder_module.layers = new_layers
+        self._pm_rope_enabled = True
+        logging.info(
+            "PM-RoPE cross-attention enabled for %d decoder layers.", len(new_layers)
+        )
+
+    # Generation-style inference with batch support for multiple samples
+    @torch.inference_mode()
+    def inference_tts(
+        self,
+        x: torch.Tensor,
+        x_lens: torch.Tensor,
+        y: torch.Tensor,
+        tgt_y_lens: torch.Tensor,
+        top_k: Union[int, List[int]] = -100,
+        top_p: float = 1.0,
+        min_p: float = 0.0,
+        temperature: float = 1.0,
+        stop_repetition: int = 3,
+        silence_tokens: List[int] = None,
+        multi_trial: List[int] = None,
+        num_samples: int = 1,
+        **kwargs,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        """
+        Run TTS inference.
+
+        Args:
+            num_samples: Number of samples to generate in parallel with different
+                random seeds. Input x and y are duplicated internally.
+
+        Returns:
+            Tuple of (concat_frames, gen_frames) with shape [num_samples, 1, T].
+        """
+        if getattr(self.args, "n_codebooks", 1) != 1:
+            raise ValueError("XCodec2 inference expects n_codebooks=1.")
+
+        self.backbone.config.use_cache = True
+        if multi_trial:
+            logging.warning("multi_trial is unsupported and will be ignored.")
+        silence_tokens = silence_tokens or []
+
+        device = x.device
+        eog_inference = (
+            self.args.eos if getattr(self.args, "eos", -1) > 0 else self.args.eog
+        )
+
+        # Input validation: expect batch_size=1, then expand to num_samples
+        assert x.shape[0] == 1, "Input batch size must be 1; use num_samples for parallel generation."
+        batch_size = num_samples
+
+        # Encoder runs once (same input for all samples)
+        x_padding_mask = make_pad_mask(x_lens).to(device)
+        encoder_attention_mask = (~x_padding_mask).long()
+        if getattr(self.args, "use_pm_rope", 1):
+            encoder_position_ids = self._build_position_ids(x_lens, x.shape[1], device)
+        else:
+            encoder_position_ids = None
+        if self.text_input_type == "text":
+            encoder_outputs = self.encoder_module(
+                input_ids=x,
+                attention_mask=encoder_attention_mask,
+                position_ids=encoder_position_ids,
+            )
+        else:
+            x_embeds = self.text_dropout(self.text_embedding(x))
+            encoder_outputs = self.encoder_module(
+                inputs_embeds=x_embeds,
+                attention_mask=encoder_attention_mask,
+                position_ids=encoder_position_ids,
+            )
+        memory = encoder_outputs.last_hidden_state  # [1, T_enc, D]
+
+        # Expand encoder outputs for batch
+        if batch_size > 1:
+            memory = memory.expand(batch_size, -1, -1).contiguous()
+            encoder_attention_mask = encoder_attention_mask.expand(batch_size, -1).contiguous()
+            if encoder_position_ids is not None:
+                encoder_position_ids = encoder_position_ids.expand(batch_size, -1).contiguous()
+
+        if self.args.special_first:
+            y = y + int(self.args.n_special)
+        y = y.transpose(2, 1).contiguous()  # [1, 1, T]
+        y_len = y.shape[-1]
+        prompt_frames = kwargs.get("prompt_frames", y_len)
+
+        # Expand y for batch
+        if batch_size > 1:
+            y = y.expand(batch_size, -1, -1).contiguous()
+
+        target_total = None
+        cutoff_limit = None
+        if tgt_y_lens is not None:
+            target_total = int(tgt_y_lens[0].item())
+            extra_cutoff = getattr(self.args, "extra_cutoff", 5.0)
+            codec_sr = int(getattr(self.args, "encodec_sr", 50))
+            cutoff_limit = target_total + int(codec_sr * extra_cutoff)
+
+        bos = torch.full(
+            (batch_size, 1, 1),
+            self.args.empty_token,
+            dtype=torch.long,
+            device=device,
+        )
+        cated_y = torch.cat([bos, y], dim=2)
+
+        new_y_len_value = cated_y.shape[-1]
+        new_y_lens = torch.full(
+            (batch_size,), new_y_len_value, dtype=torch.long, device=device
+        )
+        embedded_y = self.audio_embedding[0](cated_y[:, 0])
+        embedded_y = self.audio_dropout(embedded_y)
+
+        y_padding_mask = torch.full(
+            (batch_size, embedded_y.shape[1]), False, device=device
+        )
+        current_length = embedded_y.shape[1]
+        prompt_offset = prompt_frames + 1  # +BOS
+
+        decoder_attention_mask = (~y_padding_mask).long()
+
+        if target_total is not None:
+            est_total = int(target_total) + 1
+        elif cutoff_limit is not None:
+            est_total = int(cutoff_limit)
+        else:
+            lookahead = getattr(self.args, "progress_lookahead_secs", 2.0)
+            est_total = int(current_length + int(self.args.encodec_sr) * lookahead)
+        est_total = max(est_total, current_length)
+
+        # Pre-allocate attention mask buffer to avoid per-step tensor creation.
+        max_gen_length = est_total + int(getattr(self.args, "encodec_sr", 50) * 10)
+        full_dec_attention_mask = torch.ones(
+            (batch_size, max_gen_length), dtype=torch.long, device=device
+        )
+
+        cur_len = embedded_y.shape[1]
+        pm_kwargs = {}
+        decoder_position_ids_full = None
+        if getattr(self.args, "use_pm_rope", 1):
+            base = torch.arange(cur_len, device=device, dtype=torch.float32).unsqueeze(0)
+            decoder_position_ids_full = (
+                base / max(1, est_total - 1) * self.progress_scale
+            )
+            if batch_size > 1:
+                decoder_position_ids_full = decoder_position_ids_full.expand(batch_size, -1).contiguous()
+            pm_kwargs["position_ids"] = decoder_position_ids_full
+            pm_kwargs["pm_decoder_position_ids"] = decoder_position_ids_full
+            pm_kwargs["pm_encoder_position_ids"] = encoder_position_ids
+        else:
+            pm_kwargs["position_ids"] = None
+
+        decoder_outputs = self.decoder_module(
+            inputs_embeds=embedded_y,
+            attention_mask=decoder_attention_mask,
+            encoder_hidden_states=memory,
+            encoder_attention_mask=encoder_attention_mask,
+            use_cache=True,
+            **pm_kwargs,
+        )
+        last_hidden = decoder_outputs.last_hidden_state[:, -1:, :]  # [B, 1, D]
+        past_key_values = decoder_outputs.past_key_values
+
+        # Batch generation state
+        generated_tokens: List[torch.Tensor] = []  # List of [B] tensors
+        cur_num_gen = 0
+        prev_tokens = torch.full((batch_size,), -1, dtype=torch.long, device=device)
+        consec_silence_counts = torch.zeros(batch_size, dtype=torch.long, device=device)
+        finished = torch.zeros(batch_size, dtype=torch.bool, device=device)
+        silence_set = set(silence_tokens)
+
+        # Compute budgets once
+        first_input_len = int(x_lens[0].item())
+        text_mode = getattr(self.args, "text_input_type", "text") == "text"
+        frames_per_token_cap = getattr(self.args, "text_guard_frames_per_token", 0)
+        extra_cutoff_val = getattr(self.args, "extra_cutoff", 5)
+
+        while not finished.all():
+            logits = self.predict_layer[0](last_hidden).squeeze(1)  # [B, V]
+
+            effective_length = max(0, current_length - prompt_offset)
+
+            # Adjust logits for all samples
+            if effective_length == 0:
+                logits[:, eog_inference] = -1e9
+
+            if isinstance(top_k, list):
+                kk = top_k[min(len(top_k) - 1, cur_num_gen)]
+            else:
+                kk = top_k
+
+            if cur_num_gen <= self.args.encodec_sr // 5:
+                logits[:, eog_inference] = -10000.0
+
+            # Stop repetition penalty (vectorized)
+            if stop_repetition > 0 and silence_tokens:
+                for sil_tok in silence_tokens:
+                    mask = (prev_tokens == sil_tok) & (consec_silence_counts > stop_repetition)
+                    if mask.any():
+                        penalty = (consec_silence_counts[mask] - (stop_repetition - 1)).float()
+                        neg_mask = logits[mask, sil_tok] < 0
+                        logits[mask, sil_tok] = torch.where(
+                            neg_mask,
+                            logits[mask, sil_tok] * penalty,
+                            logits[mask, sil_tok] / penalty,
+                        )
+
+            # Sample tokens for all batch elements
+            tokens = topk_sampling(
+                logits,
+                top_k=kk,
+                top_p=top_p,
+                min_p=min_p,
+                temperature=temperature,
+            ).squeeze(-1)  # [B]
+
+            # Force stop conditions
+            should_force_stop = (tokens == eog_inference) | (logits.argmax(dim=-1) == eog_inference)
+
+            if not text_mode:
+                token_budget = first_input_len * max(1, int(self.args.encodec_sr) // 4)
+                should_force_stop |= (effective_length > token_budget)
+            elif frames_per_token_cap > 0:
+                token_budget = max(1, first_input_len) * frames_per_token_cap
+                should_force_stop |= (effective_length > token_budget)
+
+            if target_total is not None:
+                time_budget = target_total - prompt_offset + int(self.args.encodec_sr) * extra_cutoff_val
+                if cur_num_gen > time_budget:
+                    should_force_stop[:] = True
+
+            # Apply force stop
+            tokens = torch.where(should_force_stop, torch.full_like(tokens, eog_inference), tokens)
+
+            # Update silence tracking
+            for sil_tok in silence_tokens:
+                is_same_silence = (tokens == sil_tok) & (prev_tokens == sil_tok)
+                consec_silence_counts = torch.where(
+                    is_same_silence,
+                    consec_silence_counts + 1,
+                    torch.where(tokens == sil_tok, torch.ones_like(consec_silence_counts), torch.zeros_like(consec_silence_counts))
+                )
+
+            prev_tokens = tokens.clone()
+
+            # Mark finished samples
+            newly_finished = tokens == eog_inference
+            finished |= newly_finished
+
+            # Store tokens (use EOG for already-finished samples)
+            store_tokens = torch.where(finished & ~newly_finished, torch.full_like(tokens, eog_inference), tokens)
+            generated_tokens.append(store_tokens)
+
+            cur_num_gen += 1
+            current_length += 1
+
+            if finished.all():
+                break
+
+            # Embed next tokens
+            samples_emb = self.audio_embedding[0](tokens.unsqueeze(1))  # [B, 1, D]
+            samples_emb = self.audio_dropout(samples_emb)
+
+            if getattr(self.args, "use_pm_rope", 1):
+                new_pos_value = (
+                    float(current_length - 1) / max(1, est_total - 1) * self.progress_scale
+                )
+                new_pos_value = min(new_pos_value, self.progress_scale)
+                pos_1 = torch.full(
+                    (batch_size, 1), new_pos_value, device=device, dtype=torch.float32
+                )
+                pm_kwargs = {
+                    "position_ids": pos_1,
+                    "pm_decoder_position_ids": pos_1,
+                    "pm_encoder_position_ids": encoder_position_ids,
+                }
+            else:
+                pm_kwargs = {"position_ids": None}
+
+            decoder_outputs = self.decoder_module(
+                inputs_embeds=samples_emb,
+                attention_mask=full_dec_attention_mask[:, :current_length],
+                encoder_hidden_states=memory,
+                encoder_attention_mask=encoder_attention_mask,
+                past_key_values=past_key_values,
+                use_cache=True,
+                **pm_kwargs,
+            )
+            past_key_values = decoder_outputs.past_key_values
+            last_hidden = decoder_outputs.last_hidden_state
+
+        # Stack generated tokens: [B, T_gen]
+        if generated_tokens:
+            generated_tensor = torch.stack(generated_tokens, dim=1)
+        else:
+            generated_tensor = torch.zeros((batch_size, 0), device=device, dtype=torch.long)
+
+        # Trim each sample to its actual length (up to first EOG)
+        # For simplicity, keep rectangular tensor but mask with EOG
+        # The caller can trim per-sample if needed
+
+        # Build result tensors
+        # y is [B, 1, T_prompt], generated_tensor is [B, T_gen]
+        expected_y_len = y_len + generated_tensor.shape[1]
+        res = torch.cat([y[:, 0, :], generated_tensor], dim=1).unsqueeze(1)  # [B, 1, T_total]
+
+        if self.args.special_first:
+            res = res - int(self.args.n_special)
+            generated_tensor = generated_tensor - int(self.args.n_special)
+
+        return res, generated_tensor.unsqueeze(1)  # [B, 1, T_gen]

aratako_tts/T5Gemma-TTS-2b-2b/samples/en_sample1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d7ad71f5d7ddad9b720aeed8239c802ca4cb6ff86ddf6f419167d1b7e6b38bf
+size 483244

aratako_tts/T5Gemma-TTS-2b-2b/samples/en_sample2_normal.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ebcf8e90d05a9938baa744545c833a586e62754b1250515c4f3454327fd1814
+size 160044

aratako_tts/T5Gemma-TTS-2b-2b/samples/en_sample2_slow.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6ce0dfbecb9bdb0134fcf0a4eb153e4fd4254ad7fd440f568f0cb7e071d140b
+size 224044

aratako_tts/T5Gemma-TTS-2b-2b/samples/gen_sample1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c508e86225ceb26596686513b6c76dc418678127a4f3f844426b5c26ebfb2277
+size 793844

aratako_tts/T5Gemma-TTS-2b-2b/samples/gen_sample2.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ba6861440144a36f2bb397bc2eb4d324f8601cd1a470aca4651f8cb080f26eb
+size 908504

aratako_tts/T5Gemma-TTS-2b-2b/samples/gen_sample3.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a4b67fe6df5f26196a05b09eaea8558a76612f44032245267f3af03466980be
+size 274604

aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f708ca7bbae12c23e74aba7f75f6f0a227cbeed043d9d38853341df15935f9b5
+size 1497680

aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample2_fast.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1d2fdd668b1fa84214421632b38cdb469d1047a8f3086dfc00651b8cc6acc77
+size 264644

aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample2_normal.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e07c1af9be686390d79857af99b12fed4eebfbd0c8fc127efc1bc005449f2533
+size 441044

aratako_tts/T5Gemma-TTS-2b-2b/samples/jp_sample2_slow.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef76b01620d62119ed3a9cd0a0b39d13dccbc7bfd27b418b70e274132e32ea6e
+size 617444

aratako_tts/T5Gemma-TTS-2b-2b/samples/ref_sample1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:636767b965a9e9d194f939bc99820adbea9896c02e946cb45a735e77eb4fda8e
+size 338732

aratako_tts/T5Gemma-TTS-2b-2b/samples/ref_sample2.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5b82f0176c1acddad13a34a91ff5941c2c63bf596e92d364df0ac325634deff
+size 292868

aratako_tts/T5Gemma-TTS-2b-2b/samples/ref_sample3.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e44f987ddc1edc226f9f33dd173440fb8d8729933c74cad200ff20808d5c966e
+size 269370

aratako_tts/T5Gemma-TTS-2b-2b/samples/zh_sample1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c5ebba5135dd60f5067b4936caf3ed8d2d2791aaa207b9e1f9a8b572c5621d0
+size 627244

aratako_tts/t5gemma-tokenizer/config.json

@@ -0,0 +1,126 @@
+{
+  "architectures": [
+    "T5GemmaForConditionalGeneration"
+  ],
+  "classifier_dropout_rate": 0.0,
+  "decoder": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": 50.0,
+    "classifier_dropout_rate": 0.0,
+    "cross_attention_hidden_size": 2304,
+    "dropout_rate": 0.0,
+    "final_logit_softcapping": 30.0,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 2304,
+    "initializer_range": 0.02,
+    "intermediate_size": 9216,
+    "is_decoder": true,
+    "layer_types": [
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 8192,
+    "model_type": "t5_gemma_module",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 26,
+    "num_key_value_heads": 4,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 10000.0,
+    "sliding_window": 4096,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "vocab_size": 256000
+  },
+  "dropout_rate": 0.0,
+  "encoder": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": 50.0,
+    "classifier_dropout_rate": 0.0,
+    "dropout_rate": 0.0,
+    "final_logit_softcapping": 30.0,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 2304,
+    "initializer_range": 0.02,
+    "intermediate_size": 9216,
+    "layer_types": [
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 8192,
+    "model_type": "t5_gemma_module",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 26,
+    "num_key_value_heads": 4,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 10000.0,
+    "sliding_window": 4096,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "vocab_size": 256000
+  },
+  "eos_token_id": [
+    1,
+    107
+  ],
+  "initializer_range": 0.02,
+  "is_encoder_decoder": true,
+  "model_type": "t5gemma",
+  "pad_token_id": 0,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.0.dev0",
+  "use_cache": true
+}

aratako_tts/t5gemma-tokenizer/generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": [
+    1,
+    107
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "4.53.0.dev0"
+}

aratako_tts/t5gemma-tokenizer/special_tokens_map.json

@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

aratako_tts/t5gemma-tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7794135caa3ea73918949c902a781cc61dab674a4b59c17d85931c77c1114cbd
+size 34362429

aratako_tts/t5gemma-tokenizer/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2
+size 4241003

aratako_tts/t5gemma-tokenizer/tokenizer_config.json

@@ -0,0 +1,2014 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "5": {
+      "content": "<2mass>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "6": {
+      "content": "[@BOS@]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "7": {
+      "content": "<unused0>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "8": {
+      "content": "<unused1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "9": {
+      "content": "<unused2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "10": {
+      "content": "<unused3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "11": {
+      "content": "<unused4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "12": {
+      "content": "<unused5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "13": {
+      "content": "<unused6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "14": {
+      "content": "<unused7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "15": {
+      "content": "<unused8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "16": {
+      "content": "<unused9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "17": {
+      "content": "<unused10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "18": {
+      "content": "<unused11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "19": {
+      "content": "<unused12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "20": {
+      "content": "<unused13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "21": {
+      "content": "<unused14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "22": {
+      "content": "<unused15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "23": {
+      "content": "<unused16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "24": {
+      "content": "<unused17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "25": {
+      "content": "<unused18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "26": {
+      "content": "<unused19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "27": {
+      "content": "<unused20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "28": {
+      "content": "<unused21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "29": {
+      "content": "<unused22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "30": {
+      "content": "<unused23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "31": {
+      "content": "<unused24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32": {
+      "content": "<unused25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "33": {
+      "content": "<unused26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "34": {
+      "content": "<unused27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "35": {
+      "content": "<unused28>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "36": {
+      "content": "<unused29>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "37": {
+      "content": "<unused30>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "38": {
+      "content": "<unused31>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "39": {
+      "content": "<unused32>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "40": {
+      "content": "<unused33>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "41": {
+      "content": "<unused34>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "42": {
+      "content": "<unused35>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "43": {
+      "content": "<unused36>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "44": {
+      "content": "<unused37>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "45": {
+      "content": "<unused38>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "46": {
+      "content": "<unused39>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "47": {
+      "content": "<unused40>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "48": {
+      "content": "<unused41>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "49": {
+      "content": "<unused42>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50": {
+      "content": "<unused43>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "51": {
+      "content": "<unused44>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "52": {
+      "content": "<unused45>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "53": {
+      "content": "<unused46>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "54": {
+      "content": "<unused47>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "55": {
+      "content": "<unused48>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "56": {
+      "content": "<unused49>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "57": {
+      "content": "<unused50>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "58": {
+      "content": "<unused51>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "59": {
+      "content": "<unused52>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "60": {
+      "content": "<unused53>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "61": {
+      "content": "<unused54>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "62": {
+      "content": "<unused55>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "63": {
+      "content": "<unused56>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "64": {
+      "content": "<unused57>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65": {
+      "content": "<unused58>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "66": {
+      "content": "<unused59>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "67": {
+      "content": "<unused60>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "68": {
+      "content": "<unused61>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "69": {
+      "content": "<unused62>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "70": {
+      "content": "<unused63>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "71": {
+      "content": "<unused64>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "72": {
+      "content": "<unused65>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "73": {
+      "content": "<unused66>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "74": {
+      "content": "<unused67>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "75": {
+      "content": "<unused68>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "76": {
+      "content": "<unused69>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "77": {
+      "content": "<unused70>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "78": {
+      "content": "<unused71>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79": {
+      "content": "<unused72>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "80": {
+      "content": "<unused73>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "81": {
+      "content": "<unused74>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "82": {
+      "content": "<unused75>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "83": {
+      "content": "<unused76>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "84": {
+      "content": "<unused77>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "85": {
+      "content": "<unused78>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "86": {
+      "content": "<unused79>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "87": {
+      "content": "<unused80>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "88": {
+      "content": "<unused81>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "89": {
+      "content": "<unused82>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "90": {
+      "content": "<unused83>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "91": {
+      "content": "<unused84>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "92": {
+      "content": "<unused85>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "93": {
+      "content": "<unused86>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "94": {
+      "content": "<unused87>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "95": {
+      "content": "<unused88>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "96": {
+      "content": "<unused89>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "97": {
+      "content": "<unused90>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "98": {
+      "content": "<unused91>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "99": {
+      "content": "<unused92>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100": {
+      "content": "<unused93>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "101": {
+      "content": "<unused94>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102": {
+      "content": "<unused95>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "103": {
+      "content": "<unused96>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "104": {
+      "content": "<unused97>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "105": {
+      "content": "<unused98>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "106": {
+      "content": "<start_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<end_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "109": {
+      "content": "\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "110": {
+      "content": "\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "111": {
+      "content": "\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "112": {
+      "content": "\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "113": {
+      "content": "\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "114": {
+      "content": "\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "115": {
+      "content": "\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "116": {
+      "content": "\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "117": {
+      "content": "\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "118": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "119": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "120": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "121": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "122": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "123": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "124": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "125": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "126": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "127": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "129": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "130": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "131": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "132": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "133": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "134": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "135": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "136": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "137": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "138": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "139": {
+      "content": "▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "140": {
+      "content": "▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "141": {
+      "content": "▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "142": {
+      "content": "▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "143": {
+      "content": "▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "144": {
+      "content": "▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "145": {
+      "content": "▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "146": {
+      "content": "▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "147": {
+      "content": "▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "148": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "149": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "150": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "152": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "153": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "154": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "155": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "156": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "157": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "158": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "159": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "160": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "161": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "162": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "163": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "164": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "165": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "166": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "167": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "168": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "169": {
+      "content": "<table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "170": {
+      "content": "<caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "171": {
+      "content": "<thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "172": {
+      "content": "<tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "173": {
+      "content": "<tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "174": {
+      "content": "<tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "175": {
+      "content": "<th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "176": {
+      "content": "<td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "177": {
+      "content": "</table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "178": {
+      "content": "</caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "179": {
+      "content": "</thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "180": {
+      "content": "</tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "181": {
+      "content": "</tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "182": {
+      "content": "</tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "183": {
+      "content": "</th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "184": {
+      "content": "</td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "185": {
+      "content": "<h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "186": {
+      "content": "<h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "187": {
+      "content": "<h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "188": {
+      "content": "<h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "189": {
+      "content": "<h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "190": {
+      "content": "<h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "191": {
+      "content": "<blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "192": {
+      "content": "</h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "193": {
+      "content": "</h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "194": {
+      "content": "</h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "195": {
+      "content": "</h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "196": {
+      "content": "</h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "197": {
+      "content": "</h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "198": {
+      "content": "</blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "199": {
+      "content": "<strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200": {
+      "content": "<em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "201": {
+      "content": "<b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "202": {
+      "content": "<i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "203": {
+      "content": "<u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "204": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "205": {
+      "content": "<sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "206": {
+      "content": "<sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "207": {
+      "content": "<code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "208": {
+      "content": "</strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "209": {
+      "content": "</em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "210": {
+      "content": "</b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "211": {
+      "content": "</i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "212": {
+      "content": "</u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "213": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "214": {
+      "content": "</sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "215": {
+      "content": "</sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "216": {
+      "content": "</code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255968": {
+      "content": "[toxicity=0]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255969": {
+      "content": "\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255970": {
+      "content": "\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255971": {
+      "content": "\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255972": {
+      "content": "\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255973": {
+      "content": "\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255974": {
+      "content": "\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255975": {
+      "content": "\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255976": {
+      "content": "\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255977": {
+      "content": "\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255978": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255979": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255980": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255981": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255982": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255983": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255984": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255985": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255986": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255987": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255988": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255989": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255990": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255991": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255992": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255993": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255994": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255995": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255996": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255997": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255998": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255999": {
+      "content": "<unused99>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<eos>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}