Upload folder using huggingface_hub

README.md

@@ -0,0 +1,243 @@
+---
+license: apache-2.0
+library_name: transformers
+tags:
+  - audio
+  - audio-tokenizer
+  - neural-codec
+  - moss-tts-family
+  - MOSS Audio Tokenizer Nano
+  - speech-tokenizer
+  - trust-remote-code
+---
+
+# MOSS-Audio-Tokenizer-Nano
+
+This repository contains the Hugging Face remote-code implementation and weights for **MOSS-Audio-Tokenizer-Nano**, the lightweight audio tokenizer used by **MOSS-TTS-Nano**.
+
+MOSS-Audio-Tokenizer-Nano is a compact discrete audio tokenizer based on the **Cat** (**C**ausal **A**udio **T**okenizer with **T**ransformer) architecture from [MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models](https://arxiv.org/abs/2602.10934). The checkpoint in this repository has **21,969,664 parameters** (approximately **22M**), making it much smaller than the full-size MOSS-Audio-Tokenizer while preserving the 48 kHz stereo tokenizer interface used by the MOSS-TTS family.
+
+## Key Features
+
+- **Small model size**: approximately **22M parameters**, including about 10.45M encoder parameters, 10.45M decoder parameters, and 1.07M quantizer parameters.
+- **Native high-resolution audio**: supports **48 kHz** input and output with **2-channel stereo** audio, helping reduce compression loss and improve listening quality.
+- **Low-frame-rate discrete codes**: compresses 48 kHz stereo audio into a **12.5 Hz** token stream with a downsample rate of 7,680 samples.
+- **Variable bitrate reconstruction**: uses a residual quantizer stack with **16 codebooks** and 1,024 entries per codebook. Each codebook contributes about **0.125 kbps**, for an inference range from **0.125 kbps to 2 kbps**.
+- **Transformer-based tokenizer**: uses causal Transformer blocks and supports low-latency streaming encode/decode.
+- **MOSS-TTS family interface**: designed as the audio tokenizer backbone for MOSS-TTS-Nano and compatible MOSS-TTS-family workflows.
+
+**Summary:**
+By combining a compact causal Transformer tokenizer with native 48 kHz stereo modeling, MOSS-Audio-Tokenizer-Nano reduces the deployment cost of the MOSS audio tokenizer interface while keeping high-fidelity reconstruction for speech, general audio, and music. It provides a lightweight, low-frame-rate, and streaming-friendly discrete audio representation for MOSS-TTS-Nano and other real-time speech generation workflows.
+
+This repository contains a lightweight remote-code implementation that mirrors the current Hugging Face Transformers `transformers.models.moss_audio_tokenizer` module. Load it with `trust_remote_code=True` when needed.
+
+## Evaluation Metrics
+
+The table below compares the reconstruction quality of MOSS-Audio-Tokenizer-Nano with open-source audio tokenizers with **no more than 120M parameters** on speech, audio, and music data. MOSS-Audio-Tokenizer-Nano keeps one of the smallest model sizes in the comparison while supporting **48 kHz stereo** reconstruction.
+
+- Speech metrics are evaluated on LibriSpeech test-clean (English) and AISHELL-2 (Chinese), reported as EN/ZH.
+- Audio metrics are evaluated on the AudioSet evaluation subset, while music metrics are evaluated on MUSDB, reported as audio/music.
+- STFT-Dist. denotes the STFT distance.
+- Higher is better for speech metrics, while lower is better for audio/music metrics (Mel-Loss, STFT-Dist.).
+- Ch. denotes the number of input/output channels supported by the audio tokenizer: `ch=1` means mono audio, and `ch=2` means stereo audio.
+- Nvq denotes the number of quantizers.
+
+| Model | Params (M) | Sample rate | Ch. | bps | Nvq | Speech: SIM ↑ (EN/ZH) | Speech: STOI ↑ (EN/ZH) | Speech: PESQ-NB ↑ (EN/ZH) | Speech: PESQ-WB ↑ (EN/ZH) | Audio/Music: Mel-Loss ↓ | Audio/Music: STFT-Dist. ↓ |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Mimi VAE** | 28 | 24k | 1 | -- | -- | 0.75 / 0.54 | 0.91 / 0.83 | 2.92 / 2.20 | 2.30 / 1.73 | 1.35 / 1.31 | 2.70 / 2.59 |
+| **DAC** | 77 | 44.1k | 1 | 861 | 1 | 0.30 / 0.20 | 0.76 / 0.68 | 1.55 / 1.36 | 1.24 / 1.15 | 1.25 / 1.18 | 2.71 / 2.54 |
+| **SpeechTokenizer** | 120 | 16k | 1 | 1000 | 2 | 0.36 / 0.25 | 0.77 / 0.68 | 1.59 / 1.38 | 1.25 / 1.17 | -- / -- | -- / -- |
+| **Mimi** | 96 | 24k | 1 | 1100 | 8 | 0.74 / 0.59 | 0.91 / 0.85 | 2.80 / 2.24 | 2.25 / 1.78 | 1.24 / 1.19 | 2.62 / 2.49 |
+| **MOSS-Audio-Tokenizer-Nano** | 22 | 48k | 2 | 750 | 6 | 0.64 / 0.61 | 0.90 / 0.85 | 2.65 / 2.28 | 2.11 / 1.87 | 1.04 / 1.01 | 2.42 / 2.27 |
+| **MOSS-Audio-Tokenizer-Nano** | 22 | 48k | 2 | 1000 | 8 | **0.75 / 0.69** | **0.92 / 0.87** | **2.92 / 2.48** | **2.36 / 2.04** | **1.00 / 0.97** | **2.37 / 2.22** |
+| **EnCodec** | 19 | 48k | 2 | 1500 | 1 | 0.35 / 0.30 | 0.76 / 0.75 | 1.54 / 1.60 | 1.25 / 1.32 | 1.25 / 1.05 | 2.73 / 2.30 |
+| **SpeechTokenizer** | 120 | 16k | 1 | 1500 | 3 | 0.52 / 0.38 | 0.84 / 0.75 | 2.00 / 1.60 | 1.57 / 1.33 | -- / -- | -- / -- |
+| **Mimi** | 96 | 24k | 1 | 1512.5 | 11 | 0.82 / 0.67 | 0.92 / 0.88 | 3.10 / 2.50 | 2.54 / 2.00 | 1.19 / 1.14 | 2.55 / 2.42 |
+| **DAC** | 77 | 44.1k | 1 | 1723 | 2 | 0.57 / 0.47 | 0.86 / 0.80 | 2.21 / 1.85 | 1.74 / 1.49 | 1.03 / 0.99 | 2.43 / 2.26 |
+| **SpeechTokenizer** | 120 | 16k | 1 | 2000 | 4 | 0.66 / 0.50 | 0.88 / 0.80 | 2.38 / 1.79 | 1.92 / 1.49 | -- / -- | -- / -- |
+| **Mimi** | 96 | 24k | 1 | 2062.5 | 15 | 0.87 / 0.73 | 0.94 / 0.90 | 3.36 / 2.76 | 2.81 / 2.22 | 1.14 / 1.09 | 2.49 / 2.36 |
+| **MOSS-Audio-Tokenizer-Nano** | 22 | 48k | 2 | 1500 | 12 | 0.84 / 0.77 | 0.94 / 0.90 | 3.25 / 2.77 | 2.71 / 2.31 | 0.95 / 0.91 | 2.31 / 2.14 |
+| **MOSS-Audio-Tokenizer-Nano** | 22 | 48k | 2 | 2000 | 16 | **0.88 / 0.81** | **0.95 / 0.91** | **3.40 / 2.93** | **2.89 / 2.47** | **0.93 / 0.89** | **2.28 / 2.11** |
+
+## Usage
+
+### Quickstart
+
+```python
+import torchaudio
+from transformers import AutoModel
+
+repo_id = "OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano"
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
+
+wav, sr = torchaudio.load("demo/demo_gt.wav")
+if sr != model.sampling_rate:
+    wav = torchaudio.functional.resample(wav, sr, model.sampling_rate)
+
+# The public waveform interface expects stereo audio.
+if wav.shape[0] == 1:
+    wav = wav.repeat(model.config.number_channels, 1)
+else:
+    wav = wav[: model.config.number_channels]
+
+wav = wav.unsqueeze(0)
+enc = model.encode(wav, return_dict=True)
+print(f"enc.audio_codes.shape: {enc.audio_codes.shape}")
+
+dec = model.decode(enc.audio_codes, return_dict=True)
+print(f"dec.audio.shape: {dec.audio.shape}")
+
+wav = dec.audio.squeeze(0)
+torchaudio.save("demo/demo_rec.wav", wav, sample_rate=model.sampling_rate)
+
+# Decode with the first 8 codebooks, roughly 1 kbps.
+dec_rvq8 = model.decode(enc.audio_codes[:8], return_dict=True)
+wav_rvq8 = dec_rvq8.audio.squeeze(0)
+torchaudio.save("demo/demo_rec_rvq8.wav", wav_rvq8, sample_rate=model.sampling_rate)
+```
+
+### Attention Backend And Compute Dtype
+
+`config.attention_implementation` controls whether Transformer layers prefer `sdpa` or `flash_attention_2`.
+`config.compute_dtype` controls the non-quantizer autocast dtype and supports `fp32`, `bf16`, and `fp16`.
+
+```python
+model.set_attention_implementation("flash_attention_2")
+model.set_compute_dtype("fp16")
+```
+
+The quantizer always runs in fp32.
+
+### Streaming
+
+`MossAudioTokenizerModel.encode`, `decode`, `batch_encode`, and `batch_decode` all support streaming through a `chunk_duration` argument.
+
+- `chunk_duration` is expressed in seconds.
+- `chunk_duration * MossAudioTokenizerConfig.sampling_rate` must be divisible by `MossAudioTokenizerConfig.downsample_rate`.
+- Streaming batch inference is supported.
+- The public waveform interface expects stereo inputs shaped `(2, T)` or batched stereo inputs shaped `(B, 2, T)`.
+
+```python
+import torch
+from transformers import AutoModel
+
+repo_id = "OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano"
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
+audio = torch.randn(2, 48000 * 6)  # dummy stereo waveform
+
+# 6.0s @ 48kHz = 288000 samples, divisible by downsample_rate=3840
+enc = model.encode(audio.unsqueeze(0), return_dict=True, chunk_duration=0.08)
+dec = model.decode(enc.audio_codes, return_dict=True, chunk_duration=0.08)
+
+batch_enc = model.batch_encode([audio, audio[:, : 48000 * 3]], chunk_duration=0.08)
+codes_list = [
+    batch_enc.audio_codes[:, i, : batch_enc.audio_codes_lengths[i]]
+    for i in range(batch_enc.audio_codes.shape[1])
+]
+batch_dec = model.batch_decode(codes_list, chunk_duration=0.08)
+```
+
+#### Continuous Batch Streaming Decode
+
+For decoder-side continuous batching, prefer `batch_decode(..., streaming=True, ...)`.
+
+- The first streaming call may pass `max_batch_size=...`. If it is omitted, the first batch size reserves the fixed-slot decoder budget for that public stream.
+- Same-size calls continue the existing logical rows in order.
+- If a later call is larger, the new rows are admitted by tail append.
+- `finalize_indices` means "decode these rows one last time, then evict them". The indices are interpreted against the pre-call logical order.
+- After a finalize call returns, the next streaming call may use the smaller survivor batch.
+- `reset_stream=True` discards the hidden public streaming state and starts a fresh stream.
+
+Milestone 1 boundaries:
+
+- decode-only continuous batching
+- one active streaming decode state per model instance
+- fixed-slot decoder reservation from `max_batch_size`
+- no encode-side continuous batching
+- no physical compaction of surviving decode slots
+- no multi-session concurrency on one model instance
+
+```python
+import torch
+from transformers import AutoModel
+
+repo_id = "OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano"
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
+num_quantizers = model.config.quantizer_kwargs["num_quantizers"]
+codebook_size = model.config.quantizer_kwargs["codebook_size"]
+
+codes_a0 = torch.randint(0, codebook_size, (num_quantizers, 2))
+codes_b0 = torch.randint(0, codebook_size, (num_quantizers, 3))
+codes_a1 = torch.randint(0, codebook_size, (num_quantizers, 2))
+codes_b1 = torch.randint(0, codebook_size, (num_quantizers, 2))
+codes_c0 = torch.randint(0, codebook_size, (num_quantizers, 1))
+codes_a2 = torch.randint(0, codebook_size, (num_quantizers, 1))
+codes_b2 = torch.randint(0, codebook_size, (num_quantizers, 2))
+codes_c1 = torch.randint(0, codebook_size, (num_quantizers, 2))
+codes_b3 = torch.randint(0, codebook_size, (num_quantizers, 1))
+codes_c2 = torch.randint(0, codebook_size, (num_quantizers, 1))
+
+# First call reserves 3 fixed decoder slots for A and B.
+out_ab0 = model.batch_decode(
+    [codes_a0, codes_b0],
+    streaming=True,
+    max_batch_size=3,
+    reset_stream=True,
+)
+
+# Same logical rows continue in order; C is a tail append.
+out_abc1 = model.batch_decode(
+    [codes_a1, codes_b1, codes_c0],
+    streaming=True,
+)
+
+# Finalize A against the pre-call logical order. A still decodes in this call,
+# then is evicted immediately afterward.
+out_abc2 = model.batch_decode(
+    [codes_a2, codes_b2, codes_c1],
+    streaming=True,
+    finalize_indices=[0],
+)
+
+# The next call can shrink to the surviving logical rows only.
+out_bc3 = model.batch_decode(
+    [codes_b3, codes_c2],
+    streaming=True,
+)
+```
+
+## Repository Layout
+
+- `configuration_moss_audio_tokenizer.py`
+- `modeling_moss_audio_tokenizer.py`
+- `__init__.py`
+- `config.json`
+- model weights
+
+## Citation
+
+If you use this model or code in your work, please cite:
+
+```bibtex
+@misc{gong2026mossttstechnicalreport,
+  title={MOSS-TTS Technical Report},
+  author={Yitian Gong and Botian Jiang and Yiwei Zhao and Yucheng Yuan and Kuangwei Chen and Yaozhou Jiang and Cheng Chang and Dong Hong and Mingshu Chen and Ruixiao Li and Yiyang Zhang and Yang Gao and Hanfu Chen and Ke Chen and Songlin Wang and Xiaogui Yang and Yuqian Zhang and Kexin Huang and ZhengYuan Lin and Kang Yu and Ziqi Chen and Jin Wang and Zhaoye Fei and Qinyuan Cheng and Shimin Li and Xipeng Qiu},
+  year={2026},
+  eprint={2603.18090},
+  archivePrefix={arXiv},
+  primaryClass={cs.SD},
+  url={https://arxiv.org/abs/2603.18090}
+}
+```
+
+```bibtex
+@misc{gong2026mossaudiotokenizerscalingaudiotokenizers,
+  title={MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models},
+  author={Yitian Gong and Kuangwei Chen and Zhaoye Fei and Xiaogui Yang and Ke Chen and Yang Wang and Kexin Huang and Mingshu Chen and Ruixiao Li and Qingyuan Cheng and Shimin Li and Xipeng Qiu},
+  year={2026},
+  eprint={2602.10934},
+  archivePrefix={arXiv},
+  primaryClass={cs.SD},
+  url={https://arxiv.org/abs/2602.10934}
+}
+```

__init__.py

@@ -0,0 +1 @@
+"""Remote code package for Moss audio tokenizer."""

config.json

@@ -0,0 +1,304 @@
+{
+  "architectures": [
+    "MossAudioTokenizerModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_moss_audio_tokenizer.MossAudioTokenizerConfig",
+    "AutoModel": "modeling_moss_audio_tokenizer.MossAudioTokenizerModel"
+  },
+  "model_type": "moss-audio-tokenizer",
+  "sample_rate": 48000,
+  "sampling_rate": 48000,
+  "downsample_rate": 3840,
+  "causal_transformer_context_duration": 10.0,
+  "number_channels": 2,
+  "enable_channel_interleave": true,
+  "attention_implementation": "sdpa",
+  "compute_dtype": "fp32",
+  "dtype": "float32",
+  "code_dim": 768,
+  "encoder_kwargs": [
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 240
+    },
+    {
+      "causal": true,
+      "context_duration": 4.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 240,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 4,
+      "output_dimension": 384,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 6.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 768,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 2,
+      "output_dimension": 384,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 8.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 768,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 2,
+      "output_dimension": 384,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 10.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 768,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 4,
+      "output_dimension": 192,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 4
+    }
+  ],
+  "decoder_kwargs": [
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 4
+    },
+    {
+      "causal": true,
+      "context_duration": 10.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 192,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 4,
+      "output_dimension": 768,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 8.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 384,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 2,
+      "output_dimension": 768,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 6.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 384,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 2,
+      "output_dimension": 768,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 4.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 384,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 4,
+      "output_dimension": 240,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 240
+    }
+  ],
+  "quantizer_type": "rlfq",
+  "quantizer_kwargs": {
+    "codebook_dim": 8,
+    "codebook_loss_weight": 1.0,
+    "codebook_size": 1024,
+    "commitment_loss_weight": 0.25,
+    "input_dim": 768,
+    "num_quantizers": 16,
+    "output_dim": 768,
+    "quantizer_dropout": 1.0,
+    "quantizer_type": "rlfq",
+    "rvq_dim": 512
+  },
+  "transformers_version": "4.56.0.dev0",
+  "reversed_decoder_kwargs": [
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 240
+    },
+    {
+      "causal": true,
+      "context_duration": 4.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 240,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 4,
+      "output_dimension": 384,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 6.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 768,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 2,
+      "output_dimension": 384,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 8.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 768,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 2,
+      "output_dimension": 384,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 2
+    },
+    {
+      "causal": true,
+      "context_duration": 10.0,
+      "conv_layout": true,
+      "d_model": 256,
+      "dim_feedforward": 1024,
+      "gating": "none",
+      "input_dimension": 768,
+      "layer_scale": 0.01,
+      "max_period": 10000,
+      "module_type": "Transformer",
+      "norm": "layer_norm",
+      "num_heads": 4,
+      "num_layers": 4,
+      "output_dimension": 192,
+      "positional_embedding": "rope"
+    },
+    {
+      "module_type": "PatchedPretransform",
+      "patch_size": 4
+    }
+  ]
+}

configuration_moss_audio_tokenizer.py

@@ -0,0 +1,467 @@
+# coding=utf-8
+# Copyright 2026 OpenMOSS and the HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""MossAudioTokenizer model configuration."""
+
+from typing import Any
+
+try:
+    from transformers.configuration_utils import PreTrainedConfig
+except ImportError:
+    from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+
+class MossAudioTokenizerConfig(PreTrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`MossAudioTokenizerModel`]. It is used to instantiate a
+    MossAudioTokenizer model according to the specified arguments, defining the model architecture.
+
+    Instantiating a configuration with the defaults will yield a similar configuration to that of the
+    [VoiceAgentGroup/moss_audio_tokenizer](https://huggingface.co/VoiceAgentGroup/moss_audio_tokenizer) architecture.
+
+    Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PreTrainedConfig`] for more information.
+
+    Args:
+        sampling_rate (`int`, *optional*, defaults to 48000):
+            The sampling rate at which the audio waveform should be digitalized expressed in hertz (Hz).
+        downsample_rate (`int`, *optional*, defaults to 3840):
+            Total downsampling rate from waveform to tokens.
+        causal_transformer_context_duration (`float`, *optional*, defaults to 10.0):
+            Legacy global fallback context duration in seconds for causal transformer. If an individual transformer
+            entry in `encoder_kwargs` or `decoder_kwargs` provides `context_duration`, that per-module value takes
+            precedence.
+        encoder_kwargs (`list[dict]`, *optional*):
+            List of encoder module configurations. Each dict specifies a module type and its parameters.
+        decoder_kwargs (`list[dict]`, *optional*):
+            List of decoder module configurations in execution order.
+        number_channels (`int`, *optional*, defaults to 2):
+            Number of audio channels exposed by the public waveform interface.
+        enable_channel_interleave (`bool`, *optional*, defaults to `True`):
+            Whether to flatten multi-channel waveforms into a single internal stream before codec inference.
+        attention_implementation (`str`, *optional*, defaults to `"sdpa"`):
+            Attention implementation to prefer for transformer layers. Supported values are `"sdpa"` and
+            `"flash_attention_2"`.
+        compute_dtype (`str`, *optional*, defaults to `"fp32"`):
+            Inference compute dtype for non-quantizer modules. Supported values are `"fp32"`, `"bf16"`, and `"fp16"`.
+        quantizer_type (`str`, *optional*, defaults to `"rlfq"`):
+            Quantizer type. Options include `"rvq"`, `"spec_rvq"`, `"rlfq"`, `"random_prefix_rlfq"`.
+        quantizer_kwargs (`dict`, *optional*):
+            Configuration for the quantizer including `input_dim`, `rvq_dim`, `output_dim`, `num_quantizers`,
+            `codebook_size`, and `codebook_dim`.
+
+    Example:
+
+    ```python
+    >>> from transformers import MossAudioTokenizerModel, MossAudioTokenizerConfig
+
+    >>> # Initializing a MossAudioTokenizer style configuration
+    >>> configuration = MossAudioTokenizerConfig()
+
+    >>> # Initializing a model (with random weights) from the configuration
+    >>> model = MossAudioTokenizerModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```
+    """
+
+    model_type = "moss-audio-tokenizer"
+
+    # Backward-compatible alias used by some checkpoints.
+    attribute_map = {"sample_rate": "sampling_rate"}
+
+    sampling_rate: int
+    downsample_rate: int
+    causal_transformer_context_duration: float
+    encoder_kwargs: list[dict[str, Any]]
+    decoder_kwargs: list[dict[str, Any]]
+    number_channels: int
+    enable_channel_interleave: bool
+    attention_implementation: str
+    compute_dtype: str
+    quantizer_type: str
+    quantizer_kwargs: dict[str, Any]
+
+    def __init__(
+        self,
+        version: str | None = None,
+        sampling_rate: int = 48000,
+        downsample_rate: int = 3840,
+        causal_transformer_context_duration: float = 10.0,
+        encoder_kwargs: list[dict[str, Any]] | None = None,
+        decoder_kwargs: list[dict[str, Any]] | None = None,
+        number_channels: int = 2,
+        enable_channel_interleave: bool = True,
+        attention_implementation: str = "sdpa",
+        compute_dtype: str = "fp32",
+        quantizer_type: str = "rlfq",
+        quantizer_kwargs: dict[str, Any] | None = None,
+        **kwargs,
+    ):
+        # Some checkpoints might include an incorrect/legacy `model_type` (e.g. "speech_tokenizer").
+        # We drop it to avoid overriding the class-level `model_type`.
+        kwargs.pop("model_type", None)
+        if "channels_numbers" in kwargs:
+            number_channels = kwargs.pop("channels_numbers")
+        if "enable_channel_interleave" in kwargs:
+            enable_channel_interleave = kwargs.pop("enable_channel_interleave")
+        if "attention_backend" in kwargs and attention_implementation == "sdpa":
+            attention_implementation = kwargs.pop("attention_backend")
+        if "codec_compute_dtype" in kwargs and compute_dtype == "fp32":
+            compute_dtype = kwargs.pop("codec_compute_dtype")
+        reversed_decoder_kwargs = kwargs.pop("reversed_decoder_kwargs", None)
+
+        # `version` is accepted for compatibility but not used in modeling.
+        self.version = version
+        self.sampling_rate = sampling_rate
+        self.downsample_rate = downsample_rate
+        self.causal_transformer_context_duration = causal_transformer_context_duration
+        self.number_channels = number_channels
+        self.enable_channel_interleave = enable_channel_interleave
+        self.attention_implementation = attention_implementation
+        self.compute_dtype = compute_dtype
+        # Default encoder configuration
+        if encoder_kwargs is None:
+            encoder_kwargs = [
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 240,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 240,
+                    "output_dimension": 384,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 1.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 768,
+                    "output_dimension": 384,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 2.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 768,
+                    "output_dimension": 384,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 4.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 768,
+                    "output_dimension": 384,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 8.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 768,
+                    "output_dimension": 640,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 10.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 1280,
+                    "output_dimension": 768,
+                    "d_model": 1280,
+                    "num_heads": 20,
+                    "num_layers": 32,
+                    "dim_feedforward": 5120,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 10.0,
+                },
+            ]
+        else:
+            encoder_kwargs = [dict(module_kwargs) for module_kwargs in encoder_kwargs]
+        for module_kwargs in encoder_kwargs:
+            if module_kwargs.get("module_type") == "Transformer":
+                module_kwargs.setdefault("context_duration", causal_transformer_context_duration)
+        self.encoder_kwargs = encoder_kwargs
+
+        # Default decoder configuration (execution order)
+        if decoder_kwargs is None and reversed_decoder_kwargs is not None:
+            reversed_decoder_kwargs = [dict(module_kwargs) for module_kwargs in reversed_decoder_kwargs]
+            decoder_kwargs = []
+            for module_kwargs in reversed_decoder_kwargs[::-1]:
+                if module_kwargs.get("module_type") != "Transformer":
+                    decoder_kwargs.append(module_kwargs)
+                    continue
+                module_kwargs = dict(module_kwargs)
+                module_kwargs["input_dimension"], module_kwargs["output_dimension"] = (
+                    module_kwargs["output_dimension"],
+                    module_kwargs["input_dimension"],
+                )
+                decoder_kwargs.append(module_kwargs)
+
+        if decoder_kwargs is None:
+            decoder_kwargs = [
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 768,
+                    "output_dimension": 1280,
+                    "d_model": 1280,
+                    "num_heads": 20,
+                    "num_layers": 32,
+                    "dim_feedforward": 5120,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 10.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 640,
+                    "output_dimension": 768,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 10.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 384,
+                    "output_dimension": 768,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 8.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 384,
+                    "output_dimension": 768,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 4.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 384,
+                    "output_dimension": 768,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 2.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 2,
+                },
+                {
+                    "module_type": "Transformer",
+                    "input_dimension": 384,
+                    "output_dimension": 240,
+                    "d_model": 768,
+                    "num_heads": 12,
+                    "num_layers": 12,
+                    "dim_feedforward": 3072,
+                    "causal": True,
+                    "norm": "layer_norm",
+                    "positional_embedding": "rope",
+                    "max_period": 10000,
+                    "gating": "none",
+                    "layer_scale": 0.01,
+                    "conv_layout": True,
+                    "context_duration": 1.0,
+                },
+                {
+                    "module_type": "PatchedPretransform",
+                    "patch_size": 240,
+                },
+            ]
+        else:
+            decoder_kwargs = [dict(module_kwargs) for module_kwargs in decoder_kwargs]
+        for module_kwargs in decoder_kwargs:
+            if module_kwargs.get("module_type") == "Transformer":
+                module_kwargs.setdefault("context_duration", causal_transformer_context_duration)
+        self.decoder_kwargs = decoder_kwargs
+
+        # Default quantizer configuration
+        if quantizer_kwargs is None:
+            quantizer_kwargs = {
+                "input_dim": 768,
+                "rvq_dim": 512,
+                "output_dim": 768,
+                "num_quantizers": 32,
+                "codebook_size": 1024,
+                "codebook_dim": 8,
+                "quantizer_type": "rlfq",
+            }
+
+        # Handle quantizer_type from kwargs or config
+        kw_qtype = quantizer_kwargs.get("quantizer_type", None)
+        if kw_qtype is not None:
+            self.quantizer_type = kw_qtype
+        else:
+            self.quantizer_type = quantizer_type
+            quantizer_kwargs["quantizer_type"] = quantizer_type
+
+        self.quantizer_kwargs = quantizer_kwargs
+
+        super().__init__(**kwargs)
+
+    @property
+    def num_quantizers(self) -> int:
+        """Return the number of quantizers from quantizer_kwargs."""
+        return self.quantizer_kwargs.get("num_quantizers", 32)
+
+    @property
+    def codebook_size(self) -> int:
+        """Return the codebook size from quantizer_kwargs."""
+        return self.quantizer_kwargs.get("codebook_size", 4096)
+
+    @property
+    def frame_rate(self) -> float:
+        """Return the frame rate (tokens per second)."""
+        return self.sampling_rate / self.downsample_rate
+
+
+__all__ = ["MossAudioTokenizerConfig"]

model-00001-of-00001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34d9880d805eecb21bde975202b1c256dbd0eb98c8680b9d3aeffd2bc6ac2f67
+size 87922568

model.safetensors.index.json

@@ -0,0 +1,382 @@
+{
+  "metadata": {
+    "total_parameters": 21969664,
+    "total_size": 87878656
+  },
+  "weight_map": {
+    "encoder.1.input_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.2.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.transformer.layers.3.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.1.output_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.input_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.3.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.3.output_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.input_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.5.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.5.output_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.input_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.2.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.norm1.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.norm1.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.norm2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.norm2.bias": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.transformer.layers.3.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "encoder.7.output_proj.weight": "model-00001-of-00001.safetensors",
+    "quantizer.input_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.input_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.input_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.output_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.output_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.output_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.0.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.1.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.2.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.3.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.4.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.5.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.6.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.7.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.8.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.9.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.10.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.11.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.12.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.13.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.14.codebook.weight": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.in_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.in_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.in_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.out_proj.bias": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.out_proj.parametrizations.weight.original0": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.out_proj.parametrizations.weight.original1": "model-00001-of-00001.safetensors",
+    "quantizer.quantizers.15.codebook.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.input_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.2.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.transformer.layers.3.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.1.output_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.input_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.3.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.3.output_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.input_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.5.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.5.output_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.input_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.0.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.1.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.2.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.norm1.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.norm1.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.self_attn.in_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.self_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.norm2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.norm2.bias": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.ffn.0.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.ffn.2.weight": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.layer_scale_1.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.transformer.layers.3.layer_scale_2.scale": "model-00001-of-00001.safetensors",
+    "decoder.7.output_proj.weight": "model-00001-of-00001.safetensors"
+  }
+}