aurick marstin-thinky commited on
Commit
657b53b
·
0 Parent(s):

upload model

Browse files

Co-authored-by: marstin-thinky <marstin-thinky@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ license_link: https://www.apache.org/licenses/LICENSE-2.0
4
+ pipeline_tag: image-text-to-text
5
+ tags:
6
+ - conversational
7
+ - image-text-to-text
8
+ - moe
9
+ library_name: transformers
10
+ ---
11
+
12
+ # TML-d24-varloop-research
13
+
14
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/630e8f0bf6f6d700f50ebd2e/AvmDwmrWRMnKjOWvmLieg.png" style="display: block;margin-left: auto;margin-right: auto;width: 30%;">
15
+
16
+ ## 1. General Information
17
+
18
+ TML-d24-varloop-research is a multimodal research model that accepts text and image inputs and generates text outputs. It is a variable-looped counterpart to Inkling 2 Mini: whereas Inkling 2 Mini always runs its looped layers a fixed number of times, the number of loops in this model can be chosen at inference time. We are releasing it with open weights to support research on looped models. Because the number of loops can be varied on a single set of weights, the model allows cleaner comparisons of how additional looped computation affects a model's internal computation and overall capabilities.
19
+
20
+ **Languages:** English, general multilingual support
21
+
22
+ **Model provider:** Thinking Machines Lab, Inc.
23
+
24
+ **Date of release:** Nov 20, 2026
25
+
26
+ **License:** Apache 2.0
27
+
28
+ [Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/)
29
+
30
+ ## 2. Getting Started
31
+
32
+ The weights are available for download through Hugging Face [here](https://huggingface.co/thinkingmachines/tml-d24-varloop-research).
33
+
34
+ The model is not available through an API, including Tinker. It is distributed as open weights only.
35
+
36
+ ### Required hardware
37
+
38
+ The model is distributed as a single checkpoint of about 20GB. Its expert layers are NVFP4, the last four use MXFP8, and the remaining weights are BF16.
39
+
40
+ The checkpoint requires a single machine with at least 32GB of GPU or unified memory. This can be met with any of the following example configurations:
41
+
42
+ * 1x NVIDIA DGX Spark
43
+ * 1x NVIDIA GPU with 32GB or more of VRAM (for example, GeForce RTX 5090 or RTX PRO 6000 Blackwell)
44
+ * Apple silicon Mac with 32GB or more of unified memory (for example, Mac mini)
45
+
46
+ The model uses the same mixed NVFP4 and MXFP8 precision it was trained in, so this checkpoint is its native precision and needs no further quantization.
47
+
48
+ ### Required software
49
+
50
+ TML-d24-varloop-research supports local inference using the following frameworks, along with their dependency libraries:
51
+
52
+ * [Hugging Face](https://hf.co/blog/inkling2mini)
53
+ * [SGLang](https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-2-Mini)
54
+ * [vLLM](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Mini)
55
+ * [TokenSpeed](https://lightseek.org/tokenspeed/recipes/models#inkling-2-mini)
56
+ * NVIDIA Dynamo
57
+ * [Unsloth](https://unsloth.ai/docs/inkling-2-mini), including quantized GGUF versions
58
+ * llama.cpp, for quantized GGUF versions
59
+ * [Llama App](https://llama.app/)
60
+ * Ollama and Exo Labs, on Apple silicon
61
+
62
+ TML-d24-varloop-research can also be post-trained with:
63
+
64
+ * [Miles](https://miles.radixark.com/docs/models/thinkingmachines/inkling-2-mini)
65
+ * [SkyRL](https://docs.skyrl.ai/docs/models/inkling2)
66
+
67
+ ## 3. Model Properties
68
+
69
+ ### Model type
70
+
71
+ Multimodal autoregressive looped transformer
72
+
73
+ ### Architecture type
74
+
75
+ A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run a variable number of times with shared weights, chosen at inference time; the model was trained with 1 to 5 passes. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.
76
+
77
+ ### Parameters
78
+
79
+ 26.4B, 3.4B active
80
+
81
+ Note: each extra pass through the looped layers adds about 1.0B effective active parameters per token. This corresponds to 3.4B active parameters at 1 pass, 5.4B at 3 passes, and 7.4B at 5 passes.
82
+
83
+ ### Numerics support
84
+
85
+ NVFP4 and MXFP8
86
+
87
+ ### Input modalities
88
+
89
+ * Text: UTF-8 encoded text
90
+ * Image: Any pixel-based image input. For optimal performance, each image dimension should be between 40px to 4096px.
91
+
92
+ TML-d24-varloop-research supports a context window of up to 1M tokens.
93
+
94
+ ### Output modalities
95
+
96
+ Text
97
+
98
+ ## 4. Training
99
+
100
+ Training data includes a broad variety of content types, including text, images, video. Training data for the model was drawn from publicly available sources, acquired from third-parties, generated internally, or synthetically generated or augmented. Publicly available data includes content from the public internet and publicly accessible repositories.
101
+
102
+ Data curation includes cleaning, processing, and modifying datasets. These processing steps, which vary by data type, may include deduplication and filtering to remove junk or other low-quality data, or to advance safety or other objectives.
103
+
104
+ ### Training with variable looping
105
+
106
+ TML-d24-varloop-research shares its pretraining and first stage of midtraining with Inkling 2 Mini. For the rest of midtraining, it was trained with 1 to 5 passes through the looped layers, sampled in a 1:2:6:2:1 ratio, and it was then post-trained with 3 passes. Because 3 passes dominated training once looping was introduced, performance at other pass counts may understate what models trained specifically at those counts could achieve. The two models' later training stages differ, so we caution against comparing TML-d24-varloop-research directly to Inkling 2 Mini.
107
+
108
+ ## 5. Evaluations
109
+
110
+ Each row shows one benchmark score for TML-d24-varloop-research run at 1 to 5 passes through its looped layers, beside Inkling 2 Mini, which always runs 3 passes. Scores are in percent; – marks a pass count without a completed result yet.
111
+
112
+ <table>
113
+ <thead>
114
+ <tr><th style="border-right-color:rgba(128,128,128,0.45)"></th><th style="font-weight:500;border-right-color:rgba(128,128,128,0.45)">Inkling 2 Mini</th><th colspan="5" style="font-weight:500">TML-d24-varloop-research</th></tr>
115
+ <tr><th style="border-right-color:rgba(128,128,128,0.45)"></th><th style="font-weight:400;border-right-color:rgba(128,128,128,0.45)">(3 passes)</th><th style="font-weight:400">1 pass</th><th style="font-weight:400">2 passes</th><th style="font-weight:400">3 passes</th><th style="font-weight:400">4 passes</th><th style="font-weight:400">5 passes</th></tr>
116
+ </thead>
117
+ <tbody>
118
+ <tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Agentic Coding</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
119
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">DeepSWE v1.1<br><span style="font-size:11px;opacity:0.7">mini-swe-agent v2.4.6</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">48.2%</td><td style="text-align:center;vertical-align:middle">0.7%</td><td style="text-align:center;vertical-align:middle">28.3%</td><td style="text-align:center;vertical-align:middle">47.4%</td><td style="text-align:center;vertical-align:middle">46.9%</td><td style="text-align:center;vertical-align:middle">48.2%</td></tr>
120
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">SWEBench Pro v2 (Hard)<br><span style="font-size:11px;opacity:0.7">mini-swe-agent v2.4.6</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">77.9%</td><td style="text-align:center;vertical-align:middle">39.7%</td><td style="text-align:center;vertical-align:middle">69.1%</td><td style="text-align:center;vertical-align:middle">78.9%</td><td style="text-align:center;vertical-align:middle">76.0%</td><td style="text-align:center;vertical-align:middle">78.4%</td></tr>
121
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">SWEBench Multilingual<br><span style="font-size:11px;opacity:0.7">bash only</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">73.7%</td><td style="text-align:center;vertical-align:middle">40.8%</td><td style="text-align:center;vertical-align:middle">68.5%</td><td style="text-align:center;vertical-align:middle">71.8%</td><td style="text-align:center;vertical-align:middle">72.8%</td><td style="text-align:center;vertical-align:middle">72.0%</td></tr>
122
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">SWEBench Verified<br><span style="font-size:11px;opacity:0.7">bash only</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">79.2%</td><td style="text-align:center;vertical-align:middle">56.3%</td><td style="text-align:center;vertical-align:middle">76.5%</td><td style="text-align:center;vertical-align:middle">78.7%</td><td style="text-align:center;vertical-align:middle">79.3%</td><td style="text-align:center;vertical-align:middle">79.6%</td></tr>
123
+ <tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Agentic Search, Tool Use, and Knowledge Work</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
124
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">BrowseComp<br><span style="font-size:11px;opacity:0.7">original, no context management</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">76.0%</td><td style="text-align:center;vertical-align:middle">29.4%</td><td style="text-align:center;vertical-align:middle">53.8%</td><td style="text-align:center;vertical-align:middle">65.7%</td><td style="text-align:center;vertical-align:middle">69.4%</td><td style="text-align:center;vertical-align:middle">71.0%</td></tr>
125
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">MCP Atlas<br><span style="font-size:11px;opacity:0.7">Public</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">82.7%</td><td style="text-align:center;vertical-align:middle">70.0%</td><td style="text-align:center;vertical-align:middle">77.6%</td><td style="text-align:center;vertical-align:middle">79.4%</td><td style="text-align:center;vertical-align:middle">79.7%</td><td style="text-align:center;vertical-align:middle">80.0%</td></tr>
126
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">JobBench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">49.6%</td><td style="text-align:center">22.8%</td><td style="text-align:center">55.1%</td><td style="text-align:center">63.8%</td><td style="text-align:center">61.8%</td><td style="text-align:center">63.3%</td></tr>
127
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">EnterpriseOps-Gym<br><span style="font-size:11px;opacity:0.7">LangChain ReAct loop</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">36.7%</td><td style="text-align:center;vertical-align:middle">25.7%</td><td style="text-align:center;vertical-align:middle">34.4%</td><td style="text-align:center;vertical-align:middle">36.6%</td><td style="text-align:center;vertical-align:middle">36.5%</td><td style="text-align:center;vertical-align:middle">37.6%</td></tr>
128
+ <tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Vision Interpretation and Creation</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
129
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">GDP.pdf</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">16.8%</td><td style="text-align:center">4.6%</td><td style="text-align:center">11.2%</td><td style="text-align:center">19.6%</td><td style="text-align:center">18.0%</td><td style="text-align:center">20.0%</td></tr>
130
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">PerceptionBench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">57.5%</td><td style="text-align:center">45.5%</td><td style="text-align:center">51.3%</td><td style="text-align:center">53.9%</td><td style="text-align:center">53.7%</td><td style="text-align:center">52.7%</td></tr>
131
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">MMMU Pro<br><span style="font-size:11px;opacity:0.7">Standard 10</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">74.9%</td><td style="text-align:center;vertical-align:middle">63.6%</td><td style="text-align:center;vertical-align:middle">69.8%</td><td style="text-align:center;vertical-align:middle">71.4%</td><td style="text-align:center;vertical-align:middle">72.1%</td><td style="text-align:center;vertical-align:middle">73.2%</td></tr>
132
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">OmniDocBench v1.6</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">92.8%</td><td style="text-align:center">86.6%</td><td style="text-align:center">90.5%</td><td style="text-align:center">92.1%</td><td style="text-align:center">91.8%</td><td style="text-align:center">92.5%</td></tr>
133
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">BabyVision<br><span style="font-size:11px;opacity:0.7">original</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">53.4%</td><td style="text-align:center;vertical-align:middle">32.7%</td><td style="text-align:center;vertical-align:middle">44.9%</td><td style="text-align:center;vertical-align:middle">50.3%</td><td style="text-align:center;vertical-align:middle">50.3%</td><td style="text-align:center;vertical-align:middle">49.2%</td></tr>
134
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">BabyVision<br><span style="font-size:11px;opacity:0.7">with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">88.8%</td><td style="text-align:center;vertical-align:middle">67.2%</td><td style="text-align:center;vertical-align:middle">84.1%</td><td style="text-align:center;vertical-align:middle">88.9%</td><td style="text-align:center;vertical-align:middle">88.8%</td><td style="text-align:center;vertical-align:middle">89.5%</td></tr>
135
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ScreenSpot Pro<br><span style="font-size:11px;opacity:0.7">original</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">76.5%</td><td style="text-align:center;vertical-align:middle">54.4%</td><td style="text-align:center;vertical-align:middle">66.0%</td><td style="text-align:center;vertical-align:middle">72.0%</td><td style="text-align:center;vertical-align:middle">73.4%</td><td style="text-align:center;vertical-align:middle">73.1%</td></tr>
136
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ScreenSpot Pro<br><span style="font-size:11px;opacity:0.7">with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">87.7%</td><td style="text-align:center;vertical-align:middle">80.7%</td><td style="text-align:center;vertical-align:middle">86.5%</td><td style="text-align:center;vertical-align:middle">87.7%</td><td style="text-align:center;vertical-align:middle">87.2%</td><td style="text-align:center;vertical-align:middle">88.1%</td></tr>
137
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Charxiv RQ<br><span style="font-size:11px;opacity:0.7">original</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">80.6%</td><td style="text-align:center;vertical-align:middle">75.0%</td><td style="text-align:center;vertical-align:middle">78.3%</td><td style="text-align:center;vertical-align:middle">79.9%</td><td style="text-align:center;vertical-align:middle">79.2%</td><td style="text-align:center;vertical-align:middle">80.4%</td></tr>
138
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Charxiv RQ<br><span style="font-size:11px;opacity:0.7">with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">88.3%</td><td style="text-align:center;vertical-align:middle">84.7%</td><td style="text-align:center;vertical-align:middle">87.7%</td><td style="text-align:center;vertical-align:middle">87.6%</td><td style="text-align:center;vertical-align:middle">89.3%</td><td style="text-align:center;vertical-align:middle">87.9%</td></tr>
139
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Chartography<br><span style="font-size:11px;opacity:0.7">original</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">22.1%</td><td style="text-align:center;vertical-align:middle">14.5%</td><td style="text-align:center;vertical-align:middle">17.8%</td><td style="text-align:center;vertical-align:middle">21.6%</td><td style="text-align:center;vertical-align:middle">21.4%</td><td style="text-align:center;vertical-align:middle">22.6%</td></tr>
140
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Chartography<br><span style="font-size:11px;opacity:0.7">with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">72.5%</td><td style="text-align:center;vertical-align:middle">49.1%</td><td style="text-align:center;vertical-align:middle">69.1%</td><td style="text-align:center;vertical-align:middle">72.3%</td><td style="text-align:center;vertical-align:middle">73.7%</td><td style="text-align:center;vertical-align:middle">72.3%</td></tr>
141
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ZeroBench (main)<br><span style="font-size:11px;opacity:0.7">original</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">17.0%</td><td style="text-align:center;vertical-align:middle">9.0%</td><td style="text-align:center;vertical-align:middle">13.0%</td><td style="text-align:center;vertical-align:middle">19.0%</td><td style="text-align:center;vertical-align:middle">18.0%</td><td style="text-align:center;vertical-align:middle">17.0%</td></tr>
142
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ZeroBench (main)<br><span style="font-size:11px;opacity:0.7">with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">34.0%</td><td style="text-align:center;vertical-align:middle">16.0%</td><td style="text-align:center;vertical-align:middle">31.0%</td><td style="text-align:center;vertical-align:middle">40.0%</td><td style="text-align:center;vertical-align:middle">35.0%</td><td style="text-align:center;vertical-align:middle">37.0%</td></tr>
143
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Humanity’s Sixth Sense</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">23.8%</td><td style="text-align:center">22.4%</td><td style="text-align:center">24.8%</td><td style="text-align:center">25.5%</td><td style="text-align:center">26.5%</td><td style="text-align:center">27.0%</td></tr>
144
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">MathVision</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">89.0%</td><td style="text-align:center">74.8%</td><td style="text-align:center">84.9%</td><td style="text-align:center">86.6%</td><td style="text-align:center">87.8%</td><td style="text-align:center">88.1%</td></tr>
145
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">MuirBench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">77.5%</td><td style="text-align:center">73.6%</td><td style="text-align:center">76.1%</td><td style="text-align:center">77.1%</td><td style="text-align:center">76.4%</td><td style="text-align:center">77.8%</td></tr>
146
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">BVB</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">68.4%</td><td style="text-align:center">53.9%</td><td style="text-align:center">64.4%</td><td style="text-align:center">68.5%</td><td style="text-align:center">69.2%</td><td style="text-align:center">69.5%</td></tr>
147
+ <tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Reasoning, Knowledge, and Epistemics</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
148
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">HLE-Diamond</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">15.2%</td><td style="text-align:center">6.3%</td><td style="text-align:center">8.5%</td><td style="text-align:center">9.5%</td><td style="text-align:center">9.7%</td><td style="text-align:center">11.1%</td></tr>
149
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">HLE (text only)</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">30.7%</td><td style="text-align:center">10.5%</td><td style="text-align:center">20.0%</td><td style="text-align:center">24.9%</td><td style="text-align:center">25.4%</td><td style="text-align:center">25.9%</td></tr>
150
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">GPQA Diamond</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">86.6%</td><td style="text-align:center">69.6%</td><td style="text-align:center">79.2%</td><td style="text-align:center">84.2%</td><td style="text-align:center">81.9%</td><td style="text-align:center">83.1%</td></tr>
151
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Apex 2025</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">34.4%</td><td style="text-align:center">3.1%</td><td style="text-align:center">10.4%</td><td style="text-align:center">19.8%</td><td style="text-align:center">15.6%</td><td style="text-align:center">18.8%</td></tr>
152
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ComplexConstraints v1.1<br><span style="font-size:11px;opacity:0.7">openhands v1.45.0</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">33.4%</td><td style="text-align:center;vertical-align:middle">4.5%</td><td style="text-align:center;vertical-align:middle">26.0%</td><td style="text-align:center;vertical-align:middle">37.9%</td><td style="text-align:center;vertical-align:middle">39.6%</td><td style="text-align:center;vertical-align:middle">38.9%</td></tr>
153
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">SimpleQA Verified</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">8.4%</td><td style="text-align:center">6.3%</td><td style="text-align:center">6.5%</td><td style="text-align:center">7.3%</td><td style="text-align:center">6.8%</td><td style="text-align:center">6.7%</td></tr>
154
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">AA-Omniscience v1.1 (Public)<br><span style="font-size:11px;opacity:0.7">accuracy</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">21.3%</td><td style="text-align:center;vertical-align:middle">15.5%</td><td style="text-align:center;vertical-align:middle">16.3%</td><td style="text-align:center;vertical-align:middle">18.4%</td><td style="text-align:center;vertical-align:middle">18.0%</td><td style="text-align:center;vertical-align:middle">17.4%</td></tr>
155
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">AA-Omniscience v1.1 (Public)<br><span style="font-size:11px;opacity:0.7">non-hallucination rate</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">64.6%</td><td style="text-align:center;vertical-align:middle">79.3%</td><td style="text-align:center;vertical-align:middle">85.0%</td><td style="text-align:center;vertical-align:middle">84.0%</td><td style="text-align:center;vertical-align:middle">85.7%</td><td style="text-align:center;vertical-align:middle">85.9%</td></tr>
156
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">SimpleVQA</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">45.4%</td><td style="text-align:center">39.0%</td><td style="text-align:center">41.8%</td><td style="text-align:center">42.4%</td><td style="text-align:center">42.9%</td><td style="text-align:center">41.3%</td></tr>
157
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">WorldVQA</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">15.2%</td><td style="text-align:center">11.3%</td><td style="text-align:center">13.0%</td><td style="text-align:center">14.2%</td><td style="text-align:center">14.3%</td><td style="text-align:center">13.5%</td></tr>
158
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">GraphWalks BFS<br><span style="font-size:11px;opacity:0.7">256K</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">69.7%</td><td style="text-align:center;vertical-align:middle">23.5%</td><td style="text-align:center;vertical-align:middle">56.0%</td><td style="text-align:center;vertical-align:middle">64.3%</td><td style="text-align:center;vertical-align:middle">64.3%</td><td style="text-align:center;vertical-align:middle">63.8%</td></tr>
159
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">MRCR v2<br><span style="font-size:11px;opacity:0.7">8-needle 256K–512K</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">46.4%</td><td style="text-align:center;vertical-align:middle">27.2%</td><td style="text-align:center;vertical-align:middle">65.2%</td><td style="text-align:center;vertical-align:middle">89.2%</td><td style="text-align:center;vertical-align:middle">93.4%</td><td style="text-align:center;vertical-align:middle">92.1%</td></tr>
160
+ <tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Safety and Alignment</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
161
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">FORTRESS (adversarial)<br><span style="font-size:11px;opacity:0.7">Public</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">89.5%</td><td style="text-align:center;vertical-align:middle">89.0%</td><td style="text-align:center;vertical-align:middle">90.4%</td><td style="text-align:center;vertical-align:middle">90.0%</td><td style="text-align:center;vertical-align:middle">89.9%</td><td style="text-align:center;vertical-align:middle">89.4%</td></tr>
162
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">FORTRESS (benign)<br><span style="font-size:11px;opacity:0.7">Public</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">96.4%</td><td style="text-align:center;vertical-align:middle">95.0%</td><td style="text-align:center;vertical-align:middle">95.9%</td><td style="text-align:center;vertical-align:middle">95.3%</td><td style="text-align:center;vertical-align:middle">95.7%</td><td style="text-align:center;vertical-align:middle">95.0%</td></tr>
163
+ <tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">StrongREJECT (none)</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">97.8%</td><td style="text-align:center">98.8%</td><td style="text-align:center">98.4%</td><td style="text-align:center">98.2%</td><td style="text-align:center">99.4%</td><td style="text-align:center">98.4%</td></tr>
164
+ </tbody>
165
+ </table>
166
+
167
+
168
+ ## 6. Safety
169
+
170
+ We evaluated this model with the same pre-release process as Inkling 2 Mini. It is less capable than models whose weights we have already released on CBRN and cyber evaluations, and it refuses harmful requests about as reliably as Inkling 2 Small.
171
+
172
+ For production use, we recommend Inkling 2 Mini as it generally performs better on capabilities and alignment evaluations. We do not intend TML-d24-varloop-research for production use.
173
+
174
+ Across all areas, we concluded that TML-d24-varloop-research did not present risk of material uplift beyond what’s already available in the open-weight ecosystem.
175
+
176
+ ## 7. Bias, risks and limitations
177
+
178
+ TML-d24-varloop-research is a research model, not a production model. It may exhibit general limitations common to foundation models, including hallucination (generating plausible but factually incorrect or unsupported content), occasional failures to follow instructions precisely, and degraded performance in long multi-turn conversations. As with other large-scale models trained on web-derived and synthetic data, TML-d24-varloop-research may reflect biases present in its training data, including demographic, cultural, or linguistic biases, and may perform unevenly across languages, dialects, or subject domains that were less represented during training.
179
+
180
+ TML-d24-varloop-research’s knowledge is limited to information available as of its training cutoff, and it may not reflect events, developments, or changes that occurred afterward.
181
+
182
+ TML-d24-varloop-research is intended to be a research artifact. It has not been carefully tuned for deployment, and may behave unpredictably at pass counts other than 3. We do not recommend deploying this model in production or user-facing applications.
183
+
184
+ ## 8. Legal
185
+
186
+ [Training Data Documentation](https://thinkingmachines.ai/training-data-documentation/)
chat_template.jinja ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "xhigh": 0.99, "max": 0.99} -%}
2
+ {%- set role_token = {"user": "<|message_user|>", "assistant": "<|message_model|>", "system": "<|message_system|>", "tool": "<|message_tool|>"} -%}
3
+
4
+ {%- macro emit_thinking_effort() -%}
5
+ {%- set eff = reasoning_effort if reasoning_effort is defined and reasoning_effort is not none else 0.9 -%}
6
+ {%- if eff is string -%}
7
+ {%- set key = eff | trim -%}
8
+ {%- if key not in effort_map -%}
9
+ {{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
10
+ {%- endif -%}
11
+ {%- set num = effort_map[key] -%}
12
+ {%- else -%}
13
+ {%- set num = eff | float -%}
14
+ {%- endif -%}
15
+ {%- if num < 0.0 or num > 0.99 -%}
16
+ {{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
17
+ {%- endif -%}
18
+ {{- "<|message_system|><|content_text|>Thinking effort level: " -}}
19
+ {%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
20
+ {{- "<|end_message|>" -}}
21
+ {%- endmacro -%}
22
+
23
+ {%- if tools -%}
24
+ {%- set tool_state = namespace(specs=[]) -%}
25
+ {%- for tool in tools -%}
26
+ {%- set fn = tool.function if tool.function is defined else tool -%}
27
+ {%- set spec = {
28
+ "description": (fn.description if fn.description is defined and fn.description else ""),
29
+ "name": fn.name,
30
+ "parameters": (fn.parameters if fn.parameters is defined and fn.parameters else {}),
31
+ "type": (tool.type if tool.type is defined and tool.type else "function"),
32
+ } -%}
33
+ {%- set tool_state.specs = tool_state.specs + [spec] -%}
34
+ {%- endfor -%}
35
+ {{- "<|message_system|>tool_declare<|content_xml|>" -}}
36
+ {{- tool_state.specs | tojson(sort_keys=true, separators=(",", ":")) -}}
37
+ {{- "<|end_message|>" -}}
38
+ {%- endif -%}
39
+
40
+ {%- set state = namespace(effort_emitted=false) -%}
41
+ {%- for message in messages -%}
42
+ {%- if message.role not in role_token -%}
43
+ {{- raise_exception("Unknown message role: " ~ message.role) -}}
44
+ {%- endif -%}
45
+ {%- if not state.effort_emitted and message.role != "system" -%}
46
+ {{- emit_thinking_effort() -}}
47
+ {%- set state.effort_emitted = true -%}
48
+ {%- endif -%}
49
+
50
+ {%- set header = namespace(text=role_token[message.role]) -%}
51
+ {%- if message.role == "tool" -%}
52
+ {%- if message.name is defined and message.name -%}
53
+ {%- set header.text = header.text ~ message.name -%}
54
+ {%- elif message.tool_call_id is defined and message.tool_call_id -%}
55
+ {%- for prev in messages -%}
56
+ {%- if prev.role == "assistant" and prev.tool_calls -%}
57
+ {%- for tc in prev.tool_calls -%}
58
+ {%- if tc.id is defined and tc.id == message.tool_call_id and tc.function.name is defined -%}
59
+ {%- set header.text = header.text ~ tc.function.name -%}
60
+ {%- endif -%}
61
+ {%- endfor -%}
62
+ {%- endif -%}
63
+ {%- endfor -%}
64
+ {%- endif -%}
65
+ {%- elif message.role == "assistant" and message.name is defined and message.name -%}
66
+ {%- set header.text = header.text ~ message.name -%}
67
+ {%- endif -%}
68
+ {%- set rtok = header.text -%}
69
+ {%- set rendered -%}
70
+
71
+ {%- if message.role == "assistant" and message.reasoning_content is defined and message.reasoning_content -%}
72
+ {{- rtok ~ "<|content_thinking|>" ~ message.reasoning_content ~ "<|end_message|>" -}}
73
+ {%- endif -%}
74
+
75
+ {%- if message.content is string and (message.content or not (message.role == "assistant" and (message.tool_calls or message.reasoning_content))) -%}
76
+ {{- rtok ~ "<|content_text|>" ~ message.content ~ "<|end_message|>" -}}
77
+ {%- elif message.content -%}
78
+ {%- for part in message.content -%}
79
+ {%- if part is string -%}
80
+ {{- rtok ~ "<|content_text|>" ~ part ~ "<|end_message|>" -}}
81
+ {%- elif part.type is not defined or part.type in ("text", "input_text") -%}
82
+ {%- set text_part = (part.text if part.text is defined and part.text is string else "") -%}
83
+ {{- rtok ~ "<|content_text|>" ~ text_part ~ "<|end_message|>" -}}
84
+ {%- elif part.type in ("image", "input_image", "image_url") -%}
85
+ {%- if message.role not in ("user", "tool") -%}
86
+ {{- raise_exception("Image content is only supported for user and tool messages") -}}
87
+ {%- endif -%}
88
+ {{- rtok ~ "<|content_image|><|unused_200054|><|end_message|>" -}}
89
+ {%- elif part.type in ("audio", "input_audio", "audio_url") -%}
90
+ {{- raise_exception("Inkling 2 does not accept audio input") -}}
91
+ {%- else -%}
92
+ {{- raise_exception("Unsupported content part type: " ~ part.type) -}}
93
+ {%- endif -%}
94
+ {%- endfor -%}
95
+ {%- endif -%}
96
+
97
+ {%- if message.role == "assistant" and message.tool_calls -%}
98
+ {%- for tc in message.tool_calls -%}
99
+ {%- set fn = tc.function -%}
100
+ {%- if fn.name is not defined or fn.name is not string -%}
101
+ {{- raise_exception("tool call function name must be a string") -}}
102
+ {%- endif -%}
103
+ {%- if message.name is defined and message.name and message.name != fn.name -%}
104
+ {{- raise_exception("assistant name " ~ message.name ~ " does not match its tool call " ~ fn.name) -}}
105
+ {%- endif -%}
106
+ {%- set args = fn.arguments if fn.arguments is defined and fn.arguments else {} -%}
107
+ {%- if args is string -%}
108
+ {{- raise_exception("tool call arguments must be a parsed object, not a JSON string; canonicalize upstream") -}}
109
+ {%- endif -%}
110
+ {%- if args is not mapping -%}
111
+ {{- raise_exception("tool call arguments must be an object") -}}
112
+ {%- endif -%}
113
+ {{- "<|message_model|>" ~ fn.name ~ "<|content_invoke_tool_json|>" -}}
114
+ {{- '{"name":' ~ (fn.name | tojson(sort_keys=true, separators=(",", ":"))) ~ ',"args":' -}}
115
+ {{- "{" -}}
116
+ {%- for key, value in args | dictsort(case_sensitive=true) -%}
117
+ {{- (key | tojson) ~ ":" ~ (value | tojson(separators=(",", ":"))) -}}
118
+ {%- if not loop.last -%},{%- endif -%}
119
+ {%- endfor -%}
120
+ {{- "}}<|end_message|>" -}}
121
+ {%- endfor -%}
122
+ {%- endif -%}
123
+
124
+ {%- if message.role == "assistant" -%}
125
+ {{- "<|content_model_end_sampling|>" -}}
126
+ {%- endif -%}
127
+ {%- endset -%}
128
+ {%- if message.role == "assistant" -%}
129
+ {%- generation -%}{{- rendered -}}{%- endgeneration -%}
130
+ {%- else -%}
131
+ {{- rendered -}}
132
+ {%- endif -%}
133
+ {%- endfor -%}
134
+
135
+ {%- if not state.effort_emitted -%}
136
+ {{- emit_thinking_effort() -}}
137
+ {%- endif -%}
config.json ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Inkling2LoopForConditionalGeneration"
4
+ ],
5
+ "model_type": "inkling2_loop",
6
+ "eos_token_id": 200006,
7
+ "text_config": {
8
+ "model_type": "inkling2_loop_text",
9
+ "model_max_length": 1048576,
10
+ "torch_dtype": "bfloat16",
11
+ "hidden_size": 2304,
12
+ "num_hidden_layers": 24,
13
+ "vocab_size": 201024,
14
+ "num_attention_heads": 24,
15
+ "num_key_value_heads": 8,
16
+ "head_dim": 128,
17
+ "d_rel": 16,
18
+ "rel_extent": 1024,
19
+ "q_bias": false,
20
+ "o_bias": false,
21
+ "log_scaling_n_floor": 8192,
22
+ "log_scaling_alpha": 0.1,
23
+ "rms_norm_eps": 1e-06,
24
+ "use_value_norm_no_gamma": true,
25
+ "qk_l2_clip_threshold": 22.627416997969522,
26
+ "rel_l2_clip_threshold": 8.0,
27
+ "local_layer_ids": [
28
+ 0,
29
+ 1,
30
+ 2,
31
+ 3,
32
+ 4,
33
+ 6,
34
+ 7,
35
+ 8,
36
+ 9,
37
+ 10,
38
+ 12,
39
+ 13,
40
+ 14,
41
+ 15,
42
+ 16,
43
+ 18,
44
+ 19,
45
+ 20,
46
+ 21,
47
+ 22
48
+ ],
49
+ "sconv_kernel_size": 4,
50
+ "unpadded_vocab_size": 200058,
51
+ "final_logit_softcapping": 30.0,
52
+ "swa_head_dim": 128,
53
+ "swa_num_attention_heads": 24,
54
+ "swa_num_key_value_heads": 8,
55
+ "sliding_window_size": 1024,
56
+ "n_layers_per_attn_res_block": 6,
57
+ "attn_res_normalize_values": true,
58
+ "n_routed_experts": 128,
59
+ "num_experts_per_tok": 8,
60
+ "n_shared_experts": 0,
61
+ "intermediate_size": 1152,
62
+ "mlp_gate_softcap": 4.0,
63
+ "mlp_up_softcap": 25.0,
64
+ "route_scale": 8.0,
65
+ "use_gate_bias": true,
66
+ "loop_config": {
67
+ "core_start": 6,
68
+ "core_end": 18,
69
+ "num_passes": 3,
70
+ "gate_rank": 64,
71
+ "is_varloop": true
72
+ }
73
+ },
74
+ "vision_config": {
75
+ "model_type": "inkling2_vision",
76
+ "text_hidden_size": 2304,
77
+ "temporal_patch_size": 2,
78
+ "native_patch_size": 20,
79
+ "adapter_merge_size": 2,
80
+ "projection_hidden_dim": 4096,
81
+ "projector_bias": false,
82
+ "hidden_size": 1024,
83
+ "num_hidden_layers": 23,
84
+ "num_attention_heads": 16,
85
+ "mlp_ratio": 4.0,
86
+ "posemb_grid_size": 32,
87
+ "layer_norm_eps": 1e-05
88
+ },
89
+ "mtp_config": {
90
+ "hidden_size": 2304,
91
+ "num_hidden_layers": 3,
92
+ "num_attention_heads": 24,
93
+ "num_key_value_heads": 8,
94
+ "head_dim": 128,
95
+ "intermediate_size": 9216,
96
+ "max_position_embeddings": 1048576,
97
+ "torch_dtype": "bfloat16",
98
+ "q_bias": false,
99
+ "o_bias": false,
100
+ "rms_norm_eps": 1e-06,
101
+ "rope_theta": 1000000.0,
102
+ "local_layer_ids": [
103
+ 0,
104
+ 2
105
+ ],
106
+ "sliding_window_size": 1024,
107
+ "block_size": 8,
108
+ "markov_rank": 256,
109
+ "markov_head_type": "rnn_hidden",
110
+ "target_layer_ids": [
111
+ 5,
112
+ 23,
113
+ 41,
114
+ 48
115
+ ]
116
+ },
117
+ "quantization_config": {
118
+ "quant_method": "modelopt",
119
+ "quant_algo": "NVFP4"
120
+ }
121
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "do_sample": true,
3
+ "temperature": 1.0,
4
+ "eos_token_id": 200006,
5
+ "pad_token_id": 199999
6
+ }
hf_quant_config.json ADDED
@@ -0,0 +1,1255 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": {
3
+ "parameter_groups": [
4
+ {
5
+ "compute_dtype": "float32",
6
+ "output_dtype": "float32",
7
+ "required_activation_quantization": null,
8
+ "tensors": [
9
+ "model.llm.attn_res.attn_queries.0",
10
+ "model.llm.attn_res.attn_queries.1",
11
+ "model.llm.attn_res.attn_queries.10",
12
+ "model.llm.attn_res.attn_queries.11",
13
+ "model.llm.attn_res.attn_queries.12",
14
+ "model.llm.attn_res.attn_queries.13",
15
+ "model.llm.attn_res.attn_queries.14",
16
+ "model.llm.attn_res.attn_queries.15",
17
+ "model.llm.attn_res.attn_queries.16",
18
+ "model.llm.attn_res.attn_queries.17",
19
+ "model.llm.attn_res.attn_queries.18",
20
+ "model.llm.attn_res.attn_queries.19",
21
+ "model.llm.attn_res.attn_queries.2",
22
+ "model.llm.attn_res.attn_queries.20",
23
+ "model.llm.attn_res.attn_queries.21",
24
+ "model.llm.attn_res.attn_queries.22",
25
+ "model.llm.attn_res.attn_queries.3",
26
+ "model.llm.attn_res.attn_queries.4",
27
+ "model.llm.attn_res.attn_queries.5",
28
+ "model.llm.attn_res.attn_queries.6",
29
+ "model.llm.attn_res.attn_queries.7",
30
+ "model.llm.attn_res.attn_queries.8",
31
+ "model.llm.attn_res.attn_queries.9",
32
+ "model.llm.attn_res.mlp_queries.0",
33
+ "model.llm.attn_res.mlp_queries.1",
34
+ "model.llm.attn_res.mlp_queries.10",
35
+ "model.llm.attn_res.mlp_queries.11",
36
+ "model.llm.attn_res.mlp_queries.12",
37
+ "model.llm.attn_res.mlp_queries.13",
38
+ "model.llm.attn_res.mlp_queries.14",
39
+ "model.llm.attn_res.mlp_queries.15",
40
+ "model.llm.attn_res.mlp_queries.16",
41
+ "model.llm.attn_res.mlp_queries.17",
42
+ "model.llm.attn_res.mlp_queries.18",
43
+ "model.llm.attn_res.mlp_queries.19",
44
+ "model.llm.attn_res.mlp_queries.2",
45
+ "model.llm.attn_res.mlp_queries.20",
46
+ "model.llm.attn_res.mlp_queries.21",
47
+ "model.llm.attn_res.mlp_queries.22",
48
+ "model.llm.attn_res.mlp_queries.23",
49
+ "model.llm.attn_res.mlp_queries.3",
50
+ "model.llm.attn_res.mlp_queries.4",
51
+ "model.llm.attn_res.mlp_queries.5",
52
+ "model.llm.attn_res.mlp_queries.6",
53
+ "model.llm.attn_res.mlp_queries.7",
54
+ "model.llm.attn_res.mlp_queries.8",
55
+ "model.llm.attn_res.mlp_queries.9",
56
+ "model.llm.attn_res.output_query",
57
+ "model.llm.layers.0.mlp.gate.bias",
58
+ "model.llm.layers.1.mlp.gate.bias",
59
+ "model.llm.layers.10.mlp.gate.bias",
60
+ "model.llm.layers.11.mlp.gate.bias",
61
+ "model.llm.layers.12.mlp.gate.bias",
62
+ "model.llm.layers.13.mlp.gate.bias",
63
+ "model.llm.layers.14.mlp.gate.bias",
64
+ "model.llm.layers.15.mlp.gate.bias",
65
+ "model.llm.layers.16.mlp.gate.bias",
66
+ "model.llm.layers.17.mlp.gate.bias",
67
+ "model.llm.layers.18.mlp.gate.bias",
68
+ "model.llm.layers.19.mlp.gate.bias",
69
+ "model.llm.layers.2.mlp.gate.bias",
70
+ "model.llm.layers.20.mlp.gate.bias",
71
+ "model.llm.layers.21.mlp.gate.bias",
72
+ "model.llm.layers.22.mlp.gate.bias",
73
+ "model.llm.layers.23.mlp.gate.bias",
74
+ "model.llm.layers.3.mlp.gate.bias",
75
+ "model.llm.layers.4.mlp.gate.bias",
76
+ "model.llm.layers.5.mlp.gate.bias",
77
+ "model.llm.layers.6.mlp.gate.bias",
78
+ "model.llm.layers.7.mlp.gate.bias",
79
+ "model.llm.layers.8.mlp.gate.bias",
80
+ "model.llm.layers.9.mlp.gate.bias",
81
+ "model.llm.middle_cycle_cross_pass_kv_queries",
82
+ "model.llm.middle_cycle_cross_pass_queries"
83
+ ],
84
+ "weight_block_shape": [],
85
+ "weight_transform": "cast"
86
+ },
87
+ {
88
+ "compute_dtype": "bfloat16",
89
+ "output_dtype": "bfloat16",
90
+ "required_activation_quantization": null,
91
+ "tensors": [
92
+ "model.llm.embed.weight",
93
+ "model.llm.embed_norm.weight",
94
+ "model.llm.layers.0.attn.attn_sink_proj.weight",
95
+ "model.llm.layers.0.attn.gate_attn.weight",
96
+ "model.llm.layers.0.attn.k_sconv.weight",
97
+ "model.llm.layers.0.attn.q_l2_clip.scale",
98
+ "model.llm.layers.0.attn.rel_l2_clip.scale",
99
+ "model.llm.layers.0.attn.rel_logits_proj.proj",
100
+ "model.llm.layers.0.attn.v_sconv.weight",
101
+ "model.llm.layers.0.attn.wk_dv.weight",
102
+ "model.llm.layers.0.attn.wo_ud.weight",
103
+ "model.llm.layers.0.attn.wq_du.weight",
104
+ "model.llm.layers.0.attn.wr_du.weight",
105
+ "model.llm.layers.0.attn.wv_dv.weight",
106
+ "model.llm.layers.0.attn_norm.weight",
107
+ "model.llm.layers.0.attn_sconv.weight",
108
+ "model.llm.layers.0.mlp.gate.weight",
109
+ "model.llm.layers.0.mlp_norm.weight",
110
+ "model.llm.layers.0.mlp_sconv.weight",
111
+ "model.llm.layers.1.attn.attn_sink_proj.weight",
112
+ "model.llm.layers.1.attn.gate_attn.weight",
113
+ "model.llm.layers.1.attn.k_sconv.weight",
114
+ "model.llm.layers.1.attn.q_l2_clip.scale",
115
+ "model.llm.layers.1.attn.rel_l2_clip.scale",
116
+ "model.llm.layers.1.attn.rel_logits_proj.proj",
117
+ "model.llm.layers.1.attn.v_sconv.weight",
118
+ "model.llm.layers.1.attn.wk_dv.weight",
119
+ "model.llm.layers.1.attn.wo_ud.weight",
120
+ "model.llm.layers.1.attn.wq_du.weight",
121
+ "model.llm.layers.1.attn.wr_du.weight",
122
+ "model.llm.layers.1.attn.wv_dv.weight",
123
+ "model.llm.layers.1.attn_norm.weight",
124
+ "model.llm.layers.1.attn_sconv.weight",
125
+ "model.llm.layers.1.mlp.gate.weight",
126
+ "model.llm.layers.1.mlp_norm.weight",
127
+ "model.llm.layers.1.mlp_sconv.weight",
128
+ "model.llm.layers.10.attn.attn_sink_proj.weight",
129
+ "model.llm.layers.10.attn.gate_attn.weight",
130
+ "model.llm.layers.10.attn.k_sconv.weight",
131
+ "model.llm.layers.10.attn.q_l2_clip.scale",
132
+ "model.llm.layers.10.attn.rel_l2_clip.scale",
133
+ "model.llm.layers.10.attn.rel_logits_proj.proj",
134
+ "model.llm.layers.10.attn.v_sconv.weight",
135
+ "model.llm.layers.10.attn.wk_dv.weight",
136
+ "model.llm.layers.10.attn.wo_ud.weight",
137
+ "model.llm.layers.10.attn.wq_du.weight",
138
+ "model.llm.layers.10.attn.wr_du.weight",
139
+ "model.llm.layers.10.attn.wv_dv.weight",
140
+ "model.llm.layers.10.attn_norm.weight",
141
+ "model.llm.layers.10.attn_sconv.weight",
142
+ "model.llm.layers.10.controllers.attn.down_weight",
143
+ "model.llm.layers.10.controllers.attn.forget_weight",
144
+ "model.llm.layers.10.controllers.attn.gain_weight",
145
+ "model.llm.layers.10.controllers.k.down_weight",
146
+ "model.llm.layers.10.controllers.k.forget_weight",
147
+ "model.llm.layers.10.controllers.k.gain_weight",
148
+ "model.llm.layers.10.controllers.mlp.down_weight",
149
+ "model.llm.layers.10.controllers.mlp.forget_weight",
150
+ "model.llm.layers.10.controllers.mlp.gain_weight",
151
+ "model.llm.layers.10.controllers.v.down_weight",
152
+ "model.llm.layers.10.controllers.v.forget_weight",
153
+ "model.llm.layers.10.controllers.v.gain_weight",
154
+ "model.llm.layers.10.mlp.gate.weight",
155
+ "model.llm.layers.10.mlp_norm.weight",
156
+ "model.llm.layers.10.mlp_sconv.weight",
157
+ "model.llm.layers.11.attn.attn_sink_proj.weight",
158
+ "model.llm.layers.11.attn.gate_attn.weight",
159
+ "model.llm.layers.11.attn.k_sconv.static_w",
160
+ "model.llm.layers.11.attn.k_sconv.weight_proj",
161
+ "model.llm.layers.11.attn.q_l2_clip.scale",
162
+ "model.llm.layers.11.attn.rel_l2_clip.scale",
163
+ "model.llm.layers.11.attn.rel_logits_proj.proj",
164
+ "model.llm.layers.11.attn.v_sconv.static_w",
165
+ "model.llm.layers.11.attn.v_sconv.weight_proj",
166
+ "model.llm.layers.11.attn.wk_dv.weight",
167
+ "model.llm.layers.11.attn.wo_ud.weight",
168
+ "model.llm.layers.11.attn.wq_du.weight",
169
+ "model.llm.layers.11.attn.wr_du.weight",
170
+ "model.llm.layers.11.attn.wv_dv.weight",
171
+ "model.llm.layers.11.attn_sconv.weight",
172
+ "model.llm.layers.11.controllers.attn.down_weight",
173
+ "model.llm.layers.11.controllers.attn.forget_weight",
174
+ "model.llm.layers.11.controllers.attn.gain_weight",
175
+ "model.llm.layers.11.controllers.k.down_weight",
176
+ "model.llm.layers.11.controllers.k.forget_weight",
177
+ "model.llm.layers.11.controllers.k.gain_weight",
178
+ "model.llm.layers.11.controllers.mlp.down_weight",
179
+ "model.llm.layers.11.controllers.mlp.forget_weight",
180
+ "model.llm.layers.11.controllers.mlp.gain_weight",
181
+ "model.llm.layers.11.controllers.v.down_weight",
182
+ "model.llm.layers.11.controllers.v.forget_weight",
183
+ "model.llm.layers.11.controllers.v.gain_weight",
184
+ "model.llm.layers.11.k_input_norm.weight",
185
+ "model.llm.layers.11.mlp.gate.weight",
186
+ "model.llm.layers.11.mlp_norm.weight",
187
+ "model.llm.layers.11.mlp_sconv.weight",
188
+ "model.llm.layers.11.q_input_norm.weight",
189
+ "model.llm.layers.11.v_input_norm.weight",
190
+ "model.llm.layers.12.attn.attn_sink_proj.weight",
191
+ "model.llm.layers.12.attn.gate_attn.weight",
192
+ "model.llm.layers.12.attn.k_sconv.weight",
193
+ "model.llm.layers.12.attn.q_l2_clip.scale",
194
+ "model.llm.layers.12.attn.rel_l2_clip.scale",
195
+ "model.llm.layers.12.attn.rel_logits_proj.proj",
196
+ "model.llm.layers.12.attn.v_sconv.weight",
197
+ "model.llm.layers.12.attn.wk_dv.weight",
198
+ "model.llm.layers.12.attn.wo_ud.weight",
199
+ "model.llm.layers.12.attn.wq_du.weight",
200
+ "model.llm.layers.12.attn.wr_du.weight",
201
+ "model.llm.layers.12.attn.wv_dv.weight",
202
+ "model.llm.layers.12.attn_norm.weight",
203
+ "model.llm.layers.12.attn_sconv.weight",
204
+ "model.llm.layers.12.controllers.attn.down_weight",
205
+ "model.llm.layers.12.controllers.attn.forget_weight",
206
+ "model.llm.layers.12.controllers.attn.gain_weight",
207
+ "model.llm.layers.12.controllers.k.down_weight",
208
+ "model.llm.layers.12.controllers.k.forget_weight",
209
+ "model.llm.layers.12.controllers.k.gain_weight",
210
+ "model.llm.layers.12.controllers.mlp.down_weight",
211
+ "model.llm.layers.12.controllers.mlp.forget_weight",
212
+ "model.llm.layers.12.controllers.mlp.gain_weight",
213
+ "model.llm.layers.12.controllers.v.down_weight",
214
+ "model.llm.layers.12.controllers.v.forget_weight",
215
+ "model.llm.layers.12.controllers.v.gain_weight",
216
+ "model.llm.layers.12.mlp.gate.weight",
217
+ "model.llm.layers.12.mlp_norm.weight",
218
+ "model.llm.layers.12.mlp_sconv.weight",
219
+ "model.llm.layers.13.attn.attn_sink_proj.weight",
220
+ "model.llm.layers.13.attn.gate_attn.weight",
221
+ "model.llm.layers.13.attn.k_sconv.weight",
222
+ "model.llm.layers.13.attn.q_l2_clip.scale",
223
+ "model.llm.layers.13.attn.rel_l2_clip.scale",
224
+ "model.llm.layers.13.attn.rel_logits_proj.proj",
225
+ "model.llm.layers.13.attn.v_sconv.weight",
226
+ "model.llm.layers.13.attn.wk_dv.weight",
227
+ "model.llm.layers.13.attn.wo_ud.weight",
228
+ "model.llm.layers.13.attn.wq_du.weight",
229
+ "model.llm.layers.13.attn.wr_du.weight",
230
+ "model.llm.layers.13.attn.wv_dv.weight",
231
+ "model.llm.layers.13.attn_norm.weight",
232
+ "model.llm.layers.13.attn_sconv.weight",
233
+ "model.llm.layers.13.controllers.attn.down_weight",
234
+ "model.llm.layers.13.controllers.attn.forget_weight",
235
+ "model.llm.layers.13.controllers.attn.gain_weight",
236
+ "model.llm.layers.13.controllers.k.down_weight",
237
+ "model.llm.layers.13.controllers.k.forget_weight",
238
+ "model.llm.layers.13.controllers.k.gain_weight",
239
+ "model.llm.layers.13.controllers.mlp.down_weight",
240
+ "model.llm.layers.13.controllers.mlp.forget_weight",
241
+ "model.llm.layers.13.controllers.mlp.gain_weight",
242
+ "model.llm.layers.13.controllers.v.down_weight",
243
+ "model.llm.layers.13.controllers.v.forget_weight",
244
+ "model.llm.layers.13.controllers.v.gain_weight",
245
+ "model.llm.layers.13.mlp.gate.weight",
246
+ "model.llm.layers.13.mlp_norm.weight",
247
+ "model.llm.layers.13.mlp_sconv.weight",
248
+ "model.llm.layers.14.attn.attn_sink_proj.weight",
249
+ "model.llm.layers.14.attn.gate_attn.weight",
250
+ "model.llm.layers.14.attn.k_sconv.weight",
251
+ "model.llm.layers.14.attn.q_l2_clip.scale",
252
+ "model.llm.layers.14.attn.rel_l2_clip.scale",
253
+ "model.llm.layers.14.attn.rel_logits_proj.proj",
254
+ "model.llm.layers.14.attn.v_sconv.weight",
255
+ "model.llm.layers.14.attn.wk_dv.weight",
256
+ "model.llm.layers.14.attn.wo_ud.weight",
257
+ "model.llm.layers.14.attn.wq_du.weight",
258
+ "model.llm.layers.14.attn.wr_du.weight",
259
+ "model.llm.layers.14.attn.wv_dv.weight",
260
+ "model.llm.layers.14.attn_norm.weight",
261
+ "model.llm.layers.14.attn_sconv.weight",
262
+ "model.llm.layers.14.controllers.attn.down_weight",
263
+ "model.llm.layers.14.controllers.attn.forget_weight",
264
+ "model.llm.layers.14.controllers.attn.gain_weight",
265
+ "model.llm.layers.14.controllers.k.down_weight",
266
+ "model.llm.layers.14.controllers.k.forget_weight",
267
+ "model.llm.layers.14.controllers.k.gain_weight",
268
+ "model.llm.layers.14.controllers.mlp.down_weight",
269
+ "model.llm.layers.14.controllers.mlp.forget_weight",
270
+ "model.llm.layers.14.controllers.mlp.gain_weight",
271
+ "model.llm.layers.14.controllers.v.down_weight",
272
+ "model.llm.layers.14.controllers.v.forget_weight",
273
+ "model.llm.layers.14.controllers.v.gain_weight",
274
+ "model.llm.layers.14.mlp.gate.weight",
275
+ "model.llm.layers.14.mlp_norm.weight",
276
+ "model.llm.layers.14.mlp_sconv.weight",
277
+ "model.llm.layers.15.attn.attn_sink_proj.weight",
278
+ "model.llm.layers.15.attn.gate_attn.weight",
279
+ "model.llm.layers.15.attn.k_sconv.weight",
280
+ "model.llm.layers.15.attn.q_l2_clip.scale",
281
+ "model.llm.layers.15.attn.rel_l2_clip.scale",
282
+ "model.llm.layers.15.attn.rel_logits_proj.proj",
283
+ "model.llm.layers.15.attn.v_sconv.weight",
284
+ "model.llm.layers.15.attn.wk_dv.weight",
285
+ "model.llm.layers.15.attn.wo_ud.weight",
286
+ "model.llm.layers.15.attn.wq_du.weight",
287
+ "model.llm.layers.15.attn.wr_du.weight",
288
+ "model.llm.layers.15.attn.wv_dv.weight",
289
+ "model.llm.layers.15.attn_norm.weight",
290
+ "model.llm.layers.15.attn_sconv.weight",
291
+ "model.llm.layers.15.controllers.attn.down_weight",
292
+ "model.llm.layers.15.controllers.attn.forget_weight",
293
+ "model.llm.layers.15.controllers.attn.gain_weight",
294
+ "model.llm.layers.15.controllers.k.down_weight",
295
+ "model.llm.layers.15.controllers.k.forget_weight",
296
+ "model.llm.layers.15.controllers.k.gain_weight",
297
+ "model.llm.layers.15.controllers.mlp.down_weight",
298
+ "model.llm.layers.15.controllers.mlp.forget_weight",
299
+ "model.llm.layers.15.controllers.mlp.gain_weight",
300
+ "model.llm.layers.15.controllers.v.down_weight",
301
+ "model.llm.layers.15.controllers.v.forget_weight",
302
+ "model.llm.layers.15.controllers.v.gain_weight",
303
+ "model.llm.layers.15.mlp.gate.weight",
304
+ "model.llm.layers.15.mlp_norm.weight",
305
+ "model.llm.layers.15.mlp_sconv.weight",
306
+ "model.llm.layers.16.attn.attn_sink_proj.weight",
307
+ "model.llm.layers.16.attn.gate_attn.weight",
308
+ "model.llm.layers.16.attn.k_sconv.weight",
309
+ "model.llm.layers.16.attn.q_l2_clip.scale",
310
+ "model.llm.layers.16.attn.rel_l2_clip.scale",
311
+ "model.llm.layers.16.attn.rel_logits_proj.proj",
312
+ "model.llm.layers.16.attn.v_sconv.weight",
313
+ "model.llm.layers.16.attn.wk_dv.weight",
314
+ "model.llm.layers.16.attn.wo_ud.weight",
315
+ "model.llm.layers.16.attn.wq_du.weight",
316
+ "model.llm.layers.16.attn.wr_du.weight",
317
+ "model.llm.layers.16.attn.wv_dv.weight",
318
+ "model.llm.layers.16.attn_norm.weight",
319
+ "model.llm.layers.16.attn_sconv.weight",
320
+ "model.llm.layers.16.controllers.attn.down_weight",
321
+ "model.llm.layers.16.controllers.attn.forget_weight",
322
+ "model.llm.layers.16.controllers.attn.gain_weight",
323
+ "model.llm.layers.16.controllers.k.down_weight",
324
+ "model.llm.layers.16.controllers.k.forget_weight",
325
+ "model.llm.layers.16.controllers.k.gain_weight",
326
+ "model.llm.layers.16.controllers.mlp.down_weight",
327
+ "model.llm.layers.16.controllers.mlp.forget_weight",
328
+ "model.llm.layers.16.controllers.mlp.gain_weight",
329
+ "model.llm.layers.16.controllers.v.down_weight",
330
+ "model.llm.layers.16.controllers.v.forget_weight",
331
+ "model.llm.layers.16.controllers.v.gain_weight",
332
+ "model.llm.layers.16.mlp.gate.weight",
333
+ "model.llm.layers.16.mlp_norm.weight",
334
+ "model.llm.layers.16.mlp_sconv.weight",
335
+ "model.llm.layers.17.attn.attn_sink_proj.weight",
336
+ "model.llm.layers.17.attn.gate_attn.weight",
337
+ "model.llm.layers.17.attn.k_sconv.static_w",
338
+ "model.llm.layers.17.attn.k_sconv.weight_proj",
339
+ "model.llm.layers.17.attn.q_l2_clip.scale",
340
+ "model.llm.layers.17.attn.rel_l2_clip.scale",
341
+ "model.llm.layers.17.attn.rel_logits_proj.proj",
342
+ "model.llm.layers.17.attn.v_sconv.static_w",
343
+ "model.llm.layers.17.attn.v_sconv.weight_proj",
344
+ "model.llm.layers.17.attn.wk_dv.weight",
345
+ "model.llm.layers.17.attn.wo_ud.weight",
346
+ "model.llm.layers.17.attn.wq_du.weight",
347
+ "model.llm.layers.17.attn.wr_du.weight",
348
+ "model.llm.layers.17.attn.wv_dv.weight",
349
+ "model.llm.layers.17.attn_sconv.weight",
350
+ "model.llm.layers.17.controllers.attn.down_weight",
351
+ "model.llm.layers.17.controllers.attn.forget_weight",
352
+ "model.llm.layers.17.controllers.attn.gain_weight",
353
+ "model.llm.layers.17.controllers.k.down_weight",
354
+ "model.llm.layers.17.controllers.k.forget_weight",
355
+ "model.llm.layers.17.controllers.k.gain_weight",
356
+ "model.llm.layers.17.controllers.mlp.down_weight",
357
+ "model.llm.layers.17.controllers.mlp.forget_weight",
358
+ "model.llm.layers.17.controllers.mlp.gain_weight",
359
+ "model.llm.layers.17.controllers.v.down_weight",
360
+ "model.llm.layers.17.controllers.v.forget_weight",
361
+ "model.llm.layers.17.controllers.v.gain_weight",
362
+ "model.llm.layers.17.k_input_norm.weight",
363
+ "model.llm.layers.17.mlp.gate.weight",
364
+ "model.llm.layers.17.mlp_norm.weight",
365
+ "model.llm.layers.17.mlp_sconv.weight",
366
+ "model.llm.layers.17.q_input_norm.weight",
367
+ "model.llm.layers.17.v_input_norm.weight",
368
+ "model.llm.layers.18.attn.attn_sink_proj.weight",
369
+ "model.llm.layers.18.attn.gate_attn.weight",
370
+ "model.llm.layers.18.attn.k_sconv.weight",
371
+ "model.llm.layers.18.attn.q_l2_clip.scale",
372
+ "model.llm.layers.18.attn.rel_l2_clip.scale",
373
+ "model.llm.layers.18.attn.rel_logits_proj.proj",
374
+ "model.llm.layers.18.attn.v_sconv.weight",
375
+ "model.llm.layers.18.attn.wk_dv.weight",
376
+ "model.llm.layers.18.attn.wo_ud.weight",
377
+ "model.llm.layers.18.attn.wq_du.weight",
378
+ "model.llm.layers.18.attn.wr_du.weight",
379
+ "model.llm.layers.18.attn.wv_dv.weight",
380
+ "model.llm.layers.18.attn_norm.weight",
381
+ "model.llm.layers.18.attn_sconv.weight",
382
+ "model.llm.layers.18.mlp.gate.weight",
383
+ "model.llm.layers.18.mlp_norm.weight",
384
+ "model.llm.layers.18.mlp_sconv.weight",
385
+ "model.llm.layers.19.attn.attn_sink_proj.weight",
386
+ "model.llm.layers.19.attn.gate_attn.weight",
387
+ "model.llm.layers.19.attn.k_sconv.weight",
388
+ "model.llm.layers.19.attn.q_l2_clip.scale",
389
+ "model.llm.layers.19.attn.rel_l2_clip.scale",
390
+ "model.llm.layers.19.attn.rel_logits_proj.proj",
391
+ "model.llm.layers.19.attn.v_sconv.weight",
392
+ "model.llm.layers.19.attn.wk_dv.weight",
393
+ "model.llm.layers.19.attn.wo_ud.weight",
394
+ "model.llm.layers.19.attn.wq_du.weight",
395
+ "model.llm.layers.19.attn.wr_du.weight",
396
+ "model.llm.layers.19.attn.wv_dv.weight",
397
+ "model.llm.layers.19.attn_norm.weight",
398
+ "model.llm.layers.19.attn_sconv.weight",
399
+ "model.llm.layers.19.mlp.gate.weight",
400
+ "model.llm.layers.19.mlp_norm.weight",
401
+ "model.llm.layers.19.mlp_sconv.weight",
402
+ "model.llm.layers.2.attn.attn_sink_proj.weight",
403
+ "model.llm.layers.2.attn.gate_attn.weight",
404
+ "model.llm.layers.2.attn.k_sconv.weight",
405
+ "model.llm.layers.2.attn.q_l2_clip.scale",
406
+ "model.llm.layers.2.attn.rel_l2_clip.scale",
407
+ "model.llm.layers.2.attn.rel_logits_proj.proj",
408
+ "model.llm.layers.2.attn.v_sconv.weight",
409
+ "model.llm.layers.2.attn.wk_dv.weight",
410
+ "model.llm.layers.2.attn.wo_ud.weight",
411
+ "model.llm.layers.2.attn.wq_du.weight",
412
+ "model.llm.layers.2.attn.wr_du.weight",
413
+ "model.llm.layers.2.attn.wv_dv.weight",
414
+ "model.llm.layers.2.attn_norm.weight",
415
+ "model.llm.layers.2.attn_sconv.weight",
416
+ "model.llm.layers.2.mlp.gate.weight",
417
+ "model.llm.layers.2.mlp_norm.weight",
418
+ "model.llm.layers.2.mlp_sconv.weight",
419
+ "model.llm.layers.20.attn.attn_sink_proj.weight",
420
+ "model.llm.layers.20.attn.gate_attn.weight",
421
+ "model.llm.layers.20.attn.k_sconv.weight",
422
+ "model.llm.layers.20.attn.q_l2_clip.scale",
423
+ "model.llm.layers.20.attn.rel_l2_clip.scale",
424
+ "model.llm.layers.20.attn.rel_logits_proj.proj",
425
+ "model.llm.layers.20.attn.v_sconv.weight",
426
+ "model.llm.layers.20.attn.wk_dv.weight",
427
+ "model.llm.layers.20.attn.wo_ud.weight",
428
+ "model.llm.layers.20.attn.wq_du.weight",
429
+ "model.llm.layers.20.attn.wr_du.weight",
430
+ "model.llm.layers.20.attn.wv_dv.weight",
431
+ "model.llm.layers.20.attn_norm.weight",
432
+ "model.llm.layers.20.attn_sconv.weight",
433
+ "model.llm.layers.20.mlp.gate.weight",
434
+ "model.llm.layers.20.mlp_norm.weight",
435
+ "model.llm.layers.20.mlp_sconv.weight",
436
+ "model.llm.layers.21.attn.attn_sink_proj.weight",
437
+ "model.llm.layers.21.attn.gate_attn.weight",
438
+ "model.llm.layers.21.attn.k_sconv.weight",
439
+ "model.llm.layers.21.attn.q_l2_clip.scale",
440
+ "model.llm.layers.21.attn.rel_l2_clip.scale",
441
+ "model.llm.layers.21.attn.rel_logits_proj.proj",
442
+ "model.llm.layers.21.attn.v_sconv.weight",
443
+ "model.llm.layers.21.attn.wk_dv.weight",
444
+ "model.llm.layers.21.attn.wo_ud.weight",
445
+ "model.llm.layers.21.attn.wq_du.weight",
446
+ "model.llm.layers.21.attn.wr_du.weight",
447
+ "model.llm.layers.21.attn.wv_dv.weight",
448
+ "model.llm.layers.21.attn_norm.weight",
449
+ "model.llm.layers.21.attn_sconv.weight",
450
+ "model.llm.layers.21.mlp.gate.weight",
451
+ "model.llm.layers.21.mlp_norm.weight",
452
+ "model.llm.layers.21.mlp_sconv.weight",
453
+ "model.llm.layers.22.attn.attn_sink_proj.weight",
454
+ "model.llm.layers.22.attn.gate_attn.weight",
455
+ "model.llm.layers.22.attn.k_sconv.weight",
456
+ "model.llm.layers.22.attn.q_l2_clip.scale",
457
+ "model.llm.layers.22.attn.rel_l2_clip.scale",
458
+ "model.llm.layers.22.attn.rel_logits_proj.proj",
459
+ "model.llm.layers.22.attn.v_sconv.weight",
460
+ "model.llm.layers.22.attn.wk_dv.weight",
461
+ "model.llm.layers.22.attn.wo_ud.weight",
462
+ "model.llm.layers.22.attn.wq_du.weight",
463
+ "model.llm.layers.22.attn.wr_du.weight",
464
+ "model.llm.layers.22.attn.wv_dv.weight",
465
+ "model.llm.layers.22.attn_norm.weight",
466
+ "model.llm.layers.22.attn_sconv.weight",
467
+ "model.llm.layers.22.mlp.gate.weight",
468
+ "model.llm.layers.22.mlp_norm.weight",
469
+ "model.llm.layers.22.mlp_sconv.weight",
470
+ "model.llm.layers.23.attn.attn_sink_proj.weight",
471
+ "model.llm.layers.23.attn.gate_attn.weight",
472
+ "model.llm.layers.23.attn.k_sconv.static_w",
473
+ "model.llm.layers.23.attn.k_sconv.weight_proj",
474
+ "model.llm.layers.23.attn.q_l2_clip.scale",
475
+ "model.llm.layers.23.attn.rel_l2_clip.scale",
476
+ "model.llm.layers.23.attn.rel_logits_proj.proj",
477
+ "model.llm.layers.23.attn.v_sconv.static_w",
478
+ "model.llm.layers.23.attn.v_sconv.weight_proj",
479
+ "model.llm.layers.23.attn.wk_dv.weight",
480
+ "model.llm.layers.23.attn.wo_ud.weight",
481
+ "model.llm.layers.23.attn.wq_du.weight",
482
+ "model.llm.layers.23.attn.wr_du.weight",
483
+ "model.llm.layers.23.attn.wv_dv.weight",
484
+ "model.llm.layers.23.attn_sconv.weight",
485
+ "model.llm.layers.23.k_input_norm.weight",
486
+ "model.llm.layers.23.mlp.gate.weight",
487
+ "model.llm.layers.23.mlp_norm.weight",
488
+ "model.llm.layers.23.mlp_sconv.weight",
489
+ "model.llm.layers.23.q_input_norm.weight",
490
+ "model.llm.layers.23.v_input_norm.weight",
491
+ "model.llm.layers.3.attn.attn_sink_proj.weight",
492
+ "model.llm.layers.3.attn.gate_attn.weight",
493
+ "model.llm.layers.3.attn.k_sconv.weight",
494
+ "model.llm.layers.3.attn.q_l2_clip.scale",
495
+ "model.llm.layers.3.attn.rel_l2_clip.scale",
496
+ "model.llm.layers.3.attn.rel_logits_proj.proj",
497
+ "model.llm.layers.3.attn.v_sconv.weight",
498
+ "model.llm.layers.3.attn.wk_dv.weight",
499
+ "model.llm.layers.3.attn.wo_ud.weight",
500
+ "model.llm.layers.3.attn.wq_du.weight",
501
+ "model.llm.layers.3.attn.wr_du.weight",
502
+ "model.llm.layers.3.attn.wv_dv.weight",
503
+ "model.llm.layers.3.attn_norm.weight",
504
+ "model.llm.layers.3.attn_sconv.weight",
505
+ "model.llm.layers.3.mlp.gate.weight",
506
+ "model.llm.layers.3.mlp_norm.weight",
507
+ "model.llm.layers.3.mlp_sconv.weight",
508
+ "model.llm.layers.4.attn.attn_sink_proj.weight",
509
+ "model.llm.layers.4.attn.gate_attn.weight",
510
+ "model.llm.layers.4.attn.k_sconv.weight",
511
+ "model.llm.layers.4.attn.q_l2_clip.scale",
512
+ "model.llm.layers.4.attn.rel_l2_clip.scale",
513
+ "model.llm.layers.4.attn.rel_logits_proj.proj",
514
+ "model.llm.layers.4.attn.v_sconv.weight",
515
+ "model.llm.layers.4.attn.wk_dv.weight",
516
+ "model.llm.layers.4.attn.wo_ud.weight",
517
+ "model.llm.layers.4.attn.wq_du.weight",
518
+ "model.llm.layers.4.attn.wr_du.weight",
519
+ "model.llm.layers.4.attn.wv_dv.weight",
520
+ "model.llm.layers.4.attn_norm.weight",
521
+ "model.llm.layers.4.attn_sconv.weight",
522
+ "model.llm.layers.4.mlp.gate.weight",
523
+ "model.llm.layers.4.mlp_norm.weight",
524
+ "model.llm.layers.4.mlp_sconv.weight",
525
+ "model.llm.layers.5.attn.attn_sink_proj.weight",
526
+ "model.llm.layers.5.attn.gate_attn.weight",
527
+ "model.llm.layers.5.attn.k_sconv.static_w",
528
+ "model.llm.layers.5.attn.k_sconv.weight_proj",
529
+ "model.llm.layers.5.attn.q_l2_clip.scale",
530
+ "model.llm.layers.5.attn.rel_l2_clip.scale",
531
+ "model.llm.layers.5.attn.rel_logits_proj.proj",
532
+ "model.llm.layers.5.attn.v_sconv.static_w",
533
+ "model.llm.layers.5.attn.v_sconv.weight_proj",
534
+ "model.llm.layers.5.attn.wk_dv.weight",
535
+ "model.llm.layers.5.attn.wo_ud.weight",
536
+ "model.llm.layers.5.attn.wq_du.weight",
537
+ "model.llm.layers.5.attn.wr_du.weight",
538
+ "model.llm.layers.5.attn.wv_dv.weight",
539
+ "model.llm.layers.5.attn_sconv.weight",
540
+ "model.llm.layers.5.k_input_norm.weight",
541
+ "model.llm.layers.5.mlp.gate.weight",
542
+ "model.llm.layers.5.mlp_norm.weight",
543
+ "model.llm.layers.5.mlp_sconv.weight",
544
+ "model.llm.layers.5.q_input_norm.weight",
545
+ "model.llm.layers.5.v_input_norm.weight",
546
+ "model.llm.layers.6.attn.attn_sink_proj.weight",
547
+ "model.llm.layers.6.attn.gate_attn.weight",
548
+ "model.llm.layers.6.attn.k_sconv.weight",
549
+ "model.llm.layers.6.attn.q_l2_clip.scale",
550
+ "model.llm.layers.6.attn.rel_l2_clip.scale",
551
+ "model.llm.layers.6.attn.rel_logits_proj.proj",
552
+ "model.llm.layers.6.attn.v_sconv.weight",
553
+ "model.llm.layers.6.attn.wk_dv.weight",
554
+ "model.llm.layers.6.attn.wo_ud.weight",
555
+ "model.llm.layers.6.attn.wq_du.weight",
556
+ "model.llm.layers.6.attn.wr_du.weight",
557
+ "model.llm.layers.6.attn.wv_dv.weight",
558
+ "model.llm.layers.6.attn_norm.weight",
559
+ "model.llm.layers.6.attn_sconv.weight",
560
+ "model.llm.layers.6.controllers.attn.down_weight",
561
+ "model.llm.layers.6.controllers.attn.forget_weight",
562
+ "model.llm.layers.6.controllers.attn.gain_weight",
563
+ "model.llm.layers.6.controllers.k.down_weight",
564
+ "model.llm.layers.6.controllers.k.forget_weight",
565
+ "model.llm.layers.6.controllers.k.gain_weight",
566
+ "model.llm.layers.6.controllers.mlp.down_weight",
567
+ "model.llm.layers.6.controllers.mlp.forget_weight",
568
+ "model.llm.layers.6.controllers.mlp.gain_weight",
569
+ "model.llm.layers.6.controllers.v.down_weight",
570
+ "model.llm.layers.6.controllers.v.forget_weight",
571
+ "model.llm.layers.6.controllers.v.gain_weight",
572
+ "model.llm.layers.6.mlp.gate.weight",
573
+ "model.llm.layers.6.mlp_norm.weight",
574
+ "model.llm.layers.6.mlp_sconv.weight",
575
+ "model.llm.layers.7.attn.attn_sink_proj.weight",
576
+ "model.llm.layers.7.attn.gate_attn.weight",
577
+ "model.llm.layers.7.attn.k_sconv.weight",
578
+ "model.llm.layers.7.attn.q_l2_clip.scale",
579
+ "model.llm.layers.7.attn.rel_l2_clip.scale",
580
+ "model.llm.layers.7.attn.rel_logits_proj.proj",
581
+ "model.llm.layers.7.attn.v_sconv.weight",
582
+ "model.llm.layers.7.attn.wk_dv.weight",
583
+ "model.llm.layers.7.attn.wo_ud.weight",
584
+ "model.llm.layers.7.attn.wq_du.weight",
585
+ "model.llm.layers.7.attn.wr_du.weight",
586
+ "model.llm.layers.7.attn.wv_dv.weight",
587
+ "model.llm.layers.7.attn_norm.weight",
588
+ "model.llm.layers.7.attn_sconv.weight",
589
+ "model.llm.layers.7.controllers.attn.down_weight",
590
+ "model.llm.layers.7.controllers.attn.forget_weight",
591
+ "model.llm.layers.7.controllers.attn.gain_weight",
592
+ "model.llm.layers.7.controllers.k.down_weight",
593
+ "model.llm.layers.7.controllers.k.forget_weight",
594
+ "model.llm.layers.7.controllers.k.gain_weight",
595
+ "model.llm.layers.7.controllers.mlp.down_weight",
596
+ "model.llm.layers.7.controllers.mlp.forget_weight",
597
+ "model.llm.layers.7.controllers.mlp.gain_weight",
598
+ "model.llm.layers.7.controllers.v.down_weight",
599
+ "model.llm.layers.7.controllers.v.forget_weight",
600
+ "model.llm.layers.7.controllers.v.gain_weight",
601
+ "model.llm.layers.7.mlp.gate.weight",
602
+ "model.llm.layers.7.mlp_norm.weight",
603
+ "model.llm.layers.7.mlp_sconv.weight",
604
+ "model.llm.layers.8.attn.attn_sink_proj.weight",
605
+ "model.llm.layers.8.attn.gate_attn.weight",
606
+ "model.llm.layers.8.attn.k_sconv.weight",
607
+ "model.llm.layers.8.attn.q_l2_clip.scale",
608
+ "model.llm.layers.8.attn.rel_l2_clip.scale",
609
+ "model.llm.layers.8.attn.rel_logits_proj.proj",
610
+ "model.llm.layers.8.attn.v_sconv.weight",
611
+ "model.llm.layers.8.attn.wk_dv.weight",
612
+ "model.llm.layers.8.attn.wo_ud.weight",
613
+ "model.llm.layers.8.attn.wq_du.weight",
614
+ "model.llm.layers.8.attn.wr_du.weight",
615
+ "model.llm.layers.8.attn.wv_dv.weight",
616
+ "model.llm.layers.8.attn_norm.weight",
617
+ "model.llm.layers.8.attn_sconv.weight",
618
+ "model.llm.layers.8.controllers.attn.down_weight",
619
+ "model.llm.layers.8.controllers.attn.forget_weight",
620
+ "model.llm.layers.8.controllers.attn.gain_weight",
621
+ "model.llm.layers.8.controllers.k.down_weight",
622
+ "model.llm.layers.8.controllers.k.forget_weight",
623
+ "model.llm.layers.8.controllers.k.gain_weight",
624
+ "model.llm.layers.8.controllers.mlp.down_weight",
625
+ "model.llm.layers.8.controllers.mlp.forget_weight",
626
+ "model.llm.layers.8.controllers.mlp.gain_weight",
627
+ "model.llm.layers.8.controllers.v.down_weight",
628
+ "model.llm.layers.8.controllers.v.forget_weight",
629
+ "model.llm.layers.8.controllers.v.gain_weight",
630
+ "model.llm.layers.8.mlp.gate.weight",
631
+ "model.llm.layers.8.mlp_norm.weight",
632
+ "model.llm.layers.8.mlp_sconv.weight",
633
+ "model.llm.layers.9.attn.attn_sink_proj.weight",
634
+ "model.llm.layers.9.attn.gate_attn.weight",
635
+ "model.llm.layers.9.attn.k_sconv.weight",
636
+ "model.llm.layers.9.attn.q_l2_clip.scale",
637
+ "model.llm.layers.9.attn.rel_l2_clip.scale",
638
+ "model.llm.layers.9.attn.rel_logits_proj.proj",
639
+ "model.llm.layers.9.attn.v_sconv.weight",
640
+ "model.llm.layers.9.attn.wk_dv.weight",
641
+ "model.llm.layers.9.attn.wo_ud.weight",
642
+ "model.llm.layers.9.attn.wq_du.weight",
643
+ "model.llm.layers.9.attn.wr_du.weight",
644
+ "model.llm.layers.9.attn.wv_dv.weight",
645
+ "model.llm.layers.9.attn_norm.weight",
646
+ "model.llm.layers.9.attn_sconv.weight",
647
+ "model.llm.layers.9.controllers.attn.down_weight",
648
+ "model.llm.layers.9.controllers.attn.forget_weight",
649
+ "model.llm.layers.9.controllers.attn.gain_weight",
650
+ "model.llm.layers.9.controllers.k.down_weight",
651
+ "model.llm.layers.9.controllers.k.forget_weight",
652
+ "model.llm.layers.9.controllers.k.gain_weight",
653
+ "model.llm.layers.9.controllers.mlp.down_weight",
654
+ "model.llm.layers.9.controllers.mlp.forget_weight",
655
+ "model.llm.layers.9.controllers.mlp.gain_weight",
656
+ "model.llm.layers.9.controllers.v.down_weight",
657
+ "model.llm.layers.9.controllers.v.forget_weight",
658
+ "model.llm.layers.9.controllers.v.gain_weight",
659
+ "model.llm.layers.9.mlp.gate.weight",
660
+ "model.llm.layers.9.mlp_norm.weight",
661
+ "model.llm.layers.9.mlp_sconv.weight",
662
+ "model.llm.norm.weight",
663
+ "model.visual.backbone.embeddings.class_embedding",
664
+ "model.visual.backbone.embeddings.cls_position_embedding",
665
+ "model.visual.backbone.embeddings.patch_embedding.weight",
666
+ "model.visual.backbone.embeddings.position_embedding",
667
+ "model.visual.backbone.layers.0.ln_1.bias",
668
+ "model.visual.backbone.layers.0.ln_1.weight",
669
+ "model.visual.backbone.layers.0.ln_2.bias",
670
+ "model.visual.backbone.layers.0.ln_2.weight",
671
+ "model.visual.backbone.layers.0.ls_1.gamma",
672
+ "model.visual.backbone.layers.0.ls_2.gamma",
673
+ "model.visual.backbone.layers.0.mlp.c_fc.bias",
674
+ "model.visual.backbone.layers.0.mlp.c_fc.weight",
675
+ "model.visual.backbone.layers.0.mlp.c_proj.bias",
676
+ "model.visual.backbone.layers.0.mlp.c_proj.weight",
677
+ "model.visual.backbone.layers.0.self_attn.k_proj.bias",
678
+ "model.visual.backbone.layers.0.self_attn.k_proj.weight",
679
+ "model.visual.backbone.layers.0.self_attn.out_proj.bias",
680
+ "model.visual.backbone.layers.0.self_attn.out_proj.weight",
681
+ "model.visual.backbone.layers.0.self_attn.q_proj.bias",
682
+ "model.visual.backbone.layers.0.self_attn.q_proj.weight",
683
+ "model.visual.backbone.layers.0.self_attn.v_proj.bias",
684
+ "model.visual.backbone.layers.0.self_attn.v_proj.weight",
685
+ "model.visual.backbone.layers.1.ln_1.bias",
686
+ "model.visual.backbone.layers.1.ln_1.weight",
687
+ "model.visual.backbone.layers.1.ln_2.bias",
688
+ "model.visual.backbone.layers.1.ln_2.weight",
689
+ "model.visual.backbone.layers.1.ls_1.gamma",
690
+ "model.visual.backbone.layers.1.ls_2.gamma",
691
+ "model.visual.backbone.layers.1.mlp.c_fc.bias",
692
+ "model.visual.backbone.layers.1.mlp.c_fc.weight",
693
+ "model.visual.backbone.layers.1.mlp.c_proj.bias",
694
+ "model.visual.backbone.layers.1.mlp.c_proj.weight",
695
+ "model.visual.backbone.layers.1.self_attn.k_proj.bias",
696
+ "model.visual.backbone.layers.1.self_attn.k_proj.weight",
697
+ "model.visual.backbone.layers.1.self_attn.out_proj.bias",
698
+ "model.visual.backbone.layers.1.self_attn.out_proj.weight",
699
+ "model.visual.backbone.layers.1.self_attn.q_proj.bias",
700
+ "model.visual.backbone.layers.1.self_attn.q_proj.weight",
701
+ "model.visual.backbone.layers.1.self_attn.v_proj.bias",
702
+ "model.visual.backbone.layers.1.self_attn.v_proj.weight",
703
+ "model.visual.backbone.layers.10.ln_1.bias",
704
+ "model.visual.backbone.layers.10.ln_1.weight",
705
+ "model.visual.backbone.layers.10.ln_2.bias",
706
+ "model.visual.backbone.layers.10.ln_2.weight",
707
+ "model.visual.backbone.layers.10.ls_1.gamma",
708
+ "model.visual.backbone.layers.10.ls_2.gamma",
709
+ "model.visual.backbone.layers.10.mlp.c_fc.bias",
710
+ "model.visual.backbone.layers.10.mlp.c_fc.weight",
711
+ "model.visual.backbone.layers.10.mlp.c_proj.bias",
712
+ "model.visual.backbone.layers.10.mlp.c_proj.weight",
713
+ "model.visual.backbone.layers.10.self_attn.k_proj.bias",
714
+ "model.visual.backbone.layers.10.self_attn.k_proj.weight",
715
+ "model.visual.backbone.layers.10.self_attn.out_proj.bias",
716
+ "model.visual.backbone.layers.10.self_attn.out_proj.weight",
717
+ "model.visual.backbone.layers.10.self_attn.q_proj.bias",
718
+ "model.visual.backbone.layers.10.self_attn.q_proj.weight",
719
+ "model.visual.backbone.layers.10.self_attn.v_proj.bias",
720
+ "model.visual.backbone.layers.10.self_attn.v_proj.weight",
721
+ "model.visual.backbone.layers.11.ln_1.bias",
722
+ "model.visual.backbone.layers.11.ln_1.weight",
723
+ "model.visual.backbone.layers.11.ln_2.bias",
724
+ "model.visual.backbone.layers.11.ln_2.weight",
725
+ "model.visual.backbone.layers.11.ls_1.gamma",
726
+ "model.visual.backbone.layers.11.ls_2.gamma",
727
+ "model.visual.backbone.layers.11.mlp.c_fc.bias",
728
+ "model.visual.backbone.layers.11.mlp.c_fc.weight",
729
+ "model.visual.backbone.layers.11.mlp.c_proj.bias",
730
+ "model.visual.backbone.layers.11.mlp.c_proj.weight",
731
+ "model.visual.backbone.layers.11.self_attn.k_proj.bias",
732
+ "model.visual.backbone.layers.11.self_attn.k_proj.weight",
733
+ "model.visual.backbone.layers.11.self_attn.out_proj.bias",
734
+ "model.visual.backbone.layers.11.self_attn.out_proj.weight",
735
+ "model.visual.backbone.layers.11.self_attn.q_proj.bias",
736
+ "model.visual.backbone.layers.11.self_attn.q_proj.weight",
737
+ "model.visual.backbone.layers.11.self_attn.v_proj.bias",
738
+ "model.visual.backbone.layers.11.self_attn.v_proj.weight",
739
+ "model.visual.backbone.layers.12.ln_1.bias",
740
+ "model.visual.backbone.layers.12.ln_1.weight",
741
+ "model.visual.backbone.layers.12.ln_2.bias",
742
+ "model.visual.backbone.layers.12.ln_2.weight",
743
+ "model.visual.backbone.layers.12.ls_1.gamma",
744
+ "model.visual.backbone.layers.12.ls_2.gamma",
745
+ "model.visual.backbone.layers.12.mlp.c_fc.bias",
746
+ "model.visual.backbone.layers.12.mlp.c_fc.weight",
747
+ "model.visual.backbone.layers.12.mlp.c_proj.bias",
748
+ "model.visual.backbone.layers.12.mlp.c_proj.weight",
749
+ "model.visual.backbone.layers.12.self_attn.k_proj.bias",
750
+ "model.visual.backbone.layers.12.self_attn.k_proj.weight",
751
+ "model.visual.backbone.layers.12.self_attn.out_proj.bias",
752
+ "model.visual.backbone.layers.12.self_attn.out_proj.weight",
753
+ "model.visual.backbone.layers.12.self_attn.q_proj.bias",
754
+ "model.visual.backbone.layers.12.self_attn.q_proj.weight",
755
+ "model.visual.backbone.layers.12.self_attn.v_proj.bias",
756
+ "model.visual.backbone.layers.12.self_attn.v_proj.weight",
757
+ "model.visual.backbone.layers.13.ln_1.bias",
758
+ "model.visual.backbone.layers.13.ln_1.weight",
759
+ "model.visual.backbone.layers.13.ln_2.bias",
760
+ "model.visual.backbone.layers.13.ln_2.weight",
761
+ "model.visual.backbone.layers.13.ls_1.gamma",
762
+ "model.visual.backbone.layers.13.ls_2.gamma",
763
+ "model.visual.backbone.layers.13.mlp.c_fc.bias",
764
+ "model.visual.backbone.layers.13.mlp.c_fc.weight",
765
+ "model.visual.backbone.layers.13.mlp.c_proj.bias",
766
+ "model.visual.backbone.layers.13.mlp.c_proj.weight",
767
+ "model.visual.backbone.layers.13.self_attn.k_proj.bias",
768
+ "model.visual.backbone.layers.13.self_attn.k_proj.weight",
769
+ "model.visual.backbone.layers.13.self_attn.out_proj.bias",
770
+ "model.visual.backbone.layers.13.self_attn.out_proj.weight",
771
+ "model.visual.backbone.layers.13.self_attn.q_proj.bias",
772
+ "model.visual.backbone.layers.13.self_attn.q_proj.weight",
773
+ "model.visual.backbone.layers.13.self_attn.v_proj.bias",
774
+ "model.visual.backbone.layers.13.self_attn.v_proj.weight",
775
+ "model.visual.backbone.layers.14.ln_1.bias",
776
+ "model.visual.backbone.layers.14.ln_1.weight",
777
+ "model.visual.backbone.layers.14.ln_2.bias",
778
+ "model.visual.backbone.layers.14.ln_2.weight",
779
+ "model.visual.backbone.layers.14.ls_1.gamma",
780
+ "model.visual.backbone.layers.14.ls_2.gamma",
781
+ "model.visual.backbone.layers.14.mlp.c_fc.bias",
782
+ "model.visual.backbone.layers.14.mlp.c_fc.weight",
783
+ "model.visual.backbone.layers.14.mlp.c_proj.bias",
784
+ "model.visual.backbone.layers.14.mlp.c_proj.weight",
785
+ "model.visual.backbone.layers.14.self_attn.k_proj.bias",
786
+ "model.visual.backbone.layers.14.self_attn.k_proj.weight",
787
+ "model.visual.backbone.layers.14.self_attn.out_proj.bias",
788
+ "model.visual.backbone.layers.14.self_attn.out_proj.weight",
789
+ "model.visual.backbone.layers.14.self_attn.q_proj.bias",
790
+ "model.visual.backbone.layers.14.self_attn.q_proj.weight",
791
+ "model.visual.backbone.layers.14.self_attn.v_proj.bias",
792
+ "model.visual.backbone.layers.14.self_attn.v_proj.weight",
793
+ "model.visual.backbone.layers.15.ln_1.bias",
794
+ "model.visual.backbone.layers.15.ln_1.weight",
795
+ "model.visual.backbone.layers.15.ln_2.bias",
796
+ "model.visual.backbone.layers.15.ln_2.weight",
797
+ "model.visual.backbone.layers.15.ls_1.gamma",
798
+ "model.visual.backbone.layers.15.ls_2.gamma",
799
+ "model.visual.backbone.layers.15.mlp.c_fc.bias",
800
+ "model.visual.backbone.layers.15.mlp.c_fc.weight",
801
+ "model.visual.backbone.layers.15.mlp.c_proj.bias",
802
+ "model.visual.backbone.layers.15.mlp.c_proj.weight",
803
+ "model.visual.backbone.layers.15.self_attn.k_proj.bias",
804
+ "model.visual.backbone.layers.15.self_attn.k_proj.weight",
805
+ "model.visual.backbone.layers.15.self_attn.out_proj.bias",
806
+ "model.visual.backbone.layers.15.self_attn.out_proj.weight",
807
+ "model.visual.backbone.layers.15.self_attn.q_proj.bias",
808
+ "model.visual.backbone.layers.15.self_attn.q_proj.weight",
809
+ "model.visual.backbone.layers.15.self_attn.v_proj.bias",
810
+ "model.visual.backbone.layers.15.self_attn.v_proj.weight",
811
+ "model.visual.backbone.layers.16.ln_1.bias",
812
+ "model.visual.backbone.layers.16.ln_1.weight",
813
+ "model.visual.backbone.layers.16.ln_2.bias",
814
+ "model.visual.backbone.layers.16.ln_2.weight",
815
+ "model.visual.backbone.layers.16.ls_1.gamma",
816
+ "model.visual.backbone.layers.16.ls_2.gamma",
817
+ "model.visual.backbone.layers.16.mlp.c_fc.bias",
818
+ "model.visual.backbone.layers.16.mlp.c_fc.weight",
819
+ "model.visual.backbone.layers.16.mlp.c_proj.bias",
820
+ "model.visual.backbone.layers.16.mlp.c_proj.weight",
821
+ "model.visual.backbone.layers.16.self_attn.k_proj.bias",
822
+ "model.visual.backbone.layers.16.self_attn.k_proj.weight",
823
+ "model.visual.backbone.layers.16.self_attn.out_proj.bias",
824
+ "model.visual.backbone.layers.16.self_attn.out_proj.weight",
825
+ "model.visual.backbone.layers.16.self_attn.q_proj.bias",
826
+ "model.visual.backbone.layers.16.self_attn.q_proj.weight",
827
+ "model.visual.backbone.layers.16.self_attn.v_proj.bias",
828
+ "model.visual.backbone.layers.16.self_attn.v_proj.weight",
829
+ "model.visual.backbone.layers.17.ln_1.bias",
830
+ "model.visual.backbone.layers.17.ln_1.weight",
831
+ "model.visual.backbone.layers.17.ln_2.bias",
832
+ "model.visual.backbone.layers.17.ln_2.weight",
833
+ "model.visual.backbone.layers.17.ls_1.gamma",
834
+ "model.visual.backbone.layers.17.ls_2.gamma",
835
+ "model.visual.backbone.layers.17.mlp.c_fc.bias",
836
+ "model.visual.backbone.layers.17.mlp.c_fc.weight",
837
+ "model.visual.backbone.layers.17.mlp.c_proj.bias",
838
+ "model.visual.backbone.layers.17.mlp.c_proj.weight",
839
+ "model.visual.backbone.layers.17.self_attn.k_proj.bias",
840
+ "model.visual.backbone.layers.17.self_attn.k_proj.weight",
841
+ "model.visual.backbone.layers.17.self_attn.out_proj.bias",
842
+ "model.visual.backbone.layers.17.self_attn.out_proj.weight",
843
+ "model.visual.backbone.layers.17.self_attn.q_proj.bias",
844
+ "model.visual.backbone.layers.17.self_attn.q_proj.weight",
845
+ "model.visual.backbone.layers.17.self_attn.v_proj.bias",
846
+ "model.visual.backbone.layers.17.self_attn.v_proj.weight",
847
+ "model.visual.backbone.layers.18.ln_1.bias",
848
+ "model.visual.backbone.layers.18.ln_1.weight",
849
+ "model.visual.backbone.layers.18.ln_2.bias",
850
+ "model.visual.backbone.layers.18.ln_2.weight",
851
+ "model.visual.backbone.layers.18.ls_1.gamma",
852
+ "model.visual.backbone.layers.18.ls_2.gamma",
853
+ "model.visual.backbone.layers.18.mlp.c_fc.bias",
854
+ "model.visual.backbone.layers.18.mlp.c_fc.weight",
855
+ "model.visual.backbone.layers.18.mlp.c_proj.bias",
856
+ "model.visual.backbone.layers.18.mlp.c_proj.weight",
857
+ "model.visual.backbone.layers.18.self_attn.k_proj.bias",
858
+ "model.visual.backbone.layers.18.self_attn.k_proj.weight",
859
+ "model.visual.backbone.layers.18.self_attn.out_proj.bias",
860
+ "model.visual.backbone.layers.18.self_attn.out_proj.weight",
861
+ "model.visual.backbone.layers.18.self_attn.q_proj.bias",
862
+ "model.visual.backbone.layers.18.self_attn.q_proj.weight",
863
+ "model.visual.backbone.layers.18.self_attn.v_proj.bias",
864
+ "model.visual.backbone.layers.18.self_attn.v_proj.weight",
865
+ "model.visual.backbone.layers.19.ln_1.bias",
866
+ "model.visual.backbone.layers.19.ln_1.weight",
867
+ "model.visual.backbone.layers.19.ln_2.bias",
868
+ "model.visual.backbone.layers.19.ln_2.weight",
869
+ "model.visual.backbone.layers.19.ls_1.gamma",
870
+ "model.visual.backbone.layers.19.ls_2.gamma",
871
+ "model.visual.backbone.layers.19.mlp.c_fc.bias",
872
+ "model.visual.backbone.layers.19.mlp.c_fc.weight",
873
+ "model.visual.backbone.layers.19.mlp.c_proj.bias",
874
+ "model.visual.backbone.layers.19.mlp.c_proj.weight",
875
+ "model.visual.backbone.layers.19.self_attn.k_proj.bias",
876
+ "model.visual.backbone.layers.19.self_attn.k_proj.weight",
877
+ "model.visual.backbone.layers.19.self_attn.out_proj.bias",
878
+ "model.visual.backbone.layers.19.self_attn.out_proj.weight",
879
+ "model.visual.backbone.layers.19.self_attn.q_proj.bias",
880
+ "model.visual.backbone.layers.19.self_attn.q_proj.weight",
881
+ "model.visual.backbone.layers.19.self_attn.v_proj.bias",
882
+ "model.visual.backbone.layers.19.self_attn.v_proj.weight",
883
+ "model.visual.backbone.layers.2.ln_1.bias",
884
+ "model.visual.backbone.layers.2.ln_1.weight",
885
+ "model.visual.backbone.layers.2.ln_2.bias",
886
+ "model.visual.backbone.layers.2.ln_2.weight",
887
+ "model.visual.backbone.layers.2.ls_1.gamma",
888
+ "model.visual.backbone.layers.2.ls_2.gamma",
889
+ "model.visual.backbone.layers.2.mlp.c_fc.bias",
890
+ "model.visual.backbone.layers.2.mlp.c_fc.weight",
891
+ "model.visual.backbone.layers.2.mlp.c_proj.bias",
892
+ "model.visual.backbone.layers.2.mlp.c_proj.weight",
893
+ "model.visual.backbone.layers.2.self_attn.k_proj.bias",
894
+ "model.visual.backbone.layers.2.self_attn.k_proj.weight",
895
+ "model.visual.backbone.layers.2.self_attn.out_proj.bias",
896
+ "model.visual.backbone.layers.2.self_attn.out_proj.weight",
897
+ "model.visual.backbone.layers.2.self_attn.q_proj.bias",
898
+ "model.visual.backbone.layers.2.self_attn.q_proj.weight",
899
+ "model.visual.backbone.layers.2.self_attn.v_proj.bias",
900
+ "model.visual.backbone.layers.2.self_attn.v_proj.weight",
901
+ "model.visual.backbone.layers.20.ln_1.bias",
902
+ "model.visual.backbone.layers.20.ln_1.weight",
903
+ "model.visual.backbone.layers.20.ln_2.bias",
904
+ "model.visual.backbone.layers.20.ln_2.weight",
905
+ "model.visual.backbone.layers.20.ls_1.gamma",
906
+ "model.visual.backbone.layers.20.ls_2.gamma",
907
+ "model.visual.backbone.layers.20.mlp.c_fc.bias",
908
+ "model.visual.backbone.layers.20.mlp.c_fc.weight",
909
+ "model.visual.backbone.layers.20.mlp.c_proj.bias",
910
+ "model.visual.backbone.layers.20.mlp.c_proj.weight",
911
+ "model.visual.backbone.layers.20.self_attn.k_proj.bias",
912
+ "model.visual.backbone.layers.20.self_attn.k_proj.weight",
913
+ "model.visual.backbone.layers.20.self_attn.out_proj.bias",
914
+ "model.visual.backbone.layers.20.self_attn.out_proj.weight",
915
+ "model.visual.backbone.layers.20.self_attn.q_proj.bias",
916
+ "model.visual.backbone.layers.20.self_attn.q_proj.weight",
917
+ "model.visual.backbone.layers.20.self_attn.v_proj.bias",
918
+ "model.visual.backbone.layers.20.self_attn.v_proj.weight",
919
+ "model.visual.backbone.layers.21.ln_1.bias",
920
+ "model.visual.backbone.layers.21.ln_1.weight",
921
+ "model.visual.backbone.layers.21.ln_2.bias",
922
+ "model.visual.backbone.layers.21.ln_2.weight",
923
+ "model.visual.backbone.layers.21.ls_1.gamma",
924
+ "model.visual.backbone.layers.21.ls_2.gamma",
925
+ "model.visual.backbone.layers.21.mlp.c_fc.bias",
926
+ "model.visual.backbone.layers.21.mlp.c_fc.weight",
927
+ "model.visual.backbone.layers.21.mlp.c_proj.bias",
928
+ "model.visual.backbone.layers.21.mlp.c_proj.weight",
929
+ "model.visual.backbone.layers.21.self_attn.k_proj.bias",
930
+ "model.visual.backbone.layers.21.self_attn.k_proj.weight",
931
+ "model.visual.backbone.layers.21.self_attn.out_proj.bias",
932
+ "model.visual.backbone.layers.21.self_attn.out_proj.weight",
933
+ "model.visual.backbone.layers.21.self_attn.q_proj.bias",
934
+ "model.visual.backbone.layers.21.self_attn.q_proj.weight",
935
+ "model.visual.backbone.layers.21.self_attn.v_proj.bias",
936
+ "model.visual.backbone.layers.21.self_attn.v_proj.weight",
937
+ "model.visual.backbone.layers.22.ln_1.bias",
938
+ "model.visual.backbone.layers.22.ln_1.weight",
939
+ "model.visual.backbone.layers.22.ln_2.bias",
940
+ "model.visual.backbone.layers.22.ln_2.weight",
941
+ "model.visual.backbone.layers.22.ls_1.gamma",
942
+ "model.visual.backbone.layers.22.ls_2.gamma",
943
+ "model.visual.backbone.layers.22.mlp.c_fc.bias",
944
+ "model.visual.backbone.layers.22.mlp.c_fc.weight",
945
+ "model.visual.backbone.layers.22.mlp.c_proj.bias",
946
+ "model.visual.backbone.layers.22.mlp.c_proj.weight",
947
+ "model.visual.backbone.layers.22.self_attn.k_proj.bias",
948
+ "model.visual.backbone.layers.22.self_attn.k_proj.weight",
949
+ "model.visual.backbone.layers.22.self_attn.out_proj.bias",
950
+ "model.visual.backbone.layers.22.self_attn.out_proj.weight",
951
+ "model.visual.backbone.layers.22.self_attn.q_proj.bias",
952
+ "model.visual.backbone.layers.22.self_attn.q_proj.weight",
953
+ "model.visual.backbone.layers.22.self_attn.v_proj.bias",
954
+ "model.visual.backbone.layers.22.self_attn.v_proj.weight",
955
+ "model.visual.backbone.layers.3.ln_1.bias",
956
+ "model.visual.backbone.layers.3.ln_1.weight",
957
+ "model.visual.backbone.layers.3.ln_2.bias",
958
+ "model.visual.backbone.layers.3.ln_2.weight",
959
+ "model.visual.backbone.layers.3.ls_1.gamma",
960
+ "model.visual.backbone.layers.3.ls_2.gamma",
961
+ "model.visual.backbone.layers.3.mlp.c_fc.bias",
962
+ "model.visual.backbone.layers.3.mlp.c_fc.weight",
963
+ "model.visual.backbone.layers.3.mlp.c_proj.bias",
964
+ "model.visual.backbone.layers.3.mlp.c_proj.weight",
965
+ "model.visual.backbone.layers.3.self_attn.k_proj.bias",
966
+ "model.visual.backbone.layers.3.self_attn.k_proj.weight",
967
+ "model.visual.backbone.layers.3.self_attn.out_proj.bias",
968
+ "model.visual.backbone.layers.3.self_attn.out_proj.weight",
969
+ "model.visual.backbone.layers.3.self_attn.q_proj.bias",
970
+ "model.visual.backbone.layers.3.self_attn.q_proj.weight",
971
+ "model.visual.backbone.layers.3.self_attn.v_proj.bias",
972
+ "model.visual.backbone.layers.3.self_attn.v_proj.weight",
973
+ "model.visual.backbone.layers.4.ln_1.bias",
974
+ "model.visual.backbone.layers.4.ln_1.weight",
975
+ "model.visual.backbone.layers.4.ln_2.bias",
976
+ "model.visual.backbone.layers.4.ln_2.weight",
977
+ "model.visual.backbone.layers.4.ls_1.gamma",
978
+ "model.visual.backbone.layers.4.ls_2.gamma",
979
+ "model.visual.backbone.layers.4.mlp.c_fc.bias",
980
+ "model.visual.backbone.layers.4.mlp.c_fc.weight",
981
+ "model.visual.backbone.layers.4.mlp.c_proj.bias",
982
+ "model.visual.backbone.layers.4.mlp.c_proj.weight",
983
+ "model.visual.backbone.layers.4.self_attn.k_proj.bias",
984
+ "model.visual.backbone.layers.4.self_attn.k_proj.weight",
985
+ "model.visual.backbone.layers.4.self_attn.out_proj.bias",
986
+ "model.visual.backbone.layers.4.self_attn.out_proj.weight",
987
+ "model.visual.backbone.layers.4.self_attn.q_proj.bias",
988
+ "model.visual.backbone.layers.4.self_attn.q_proj.weight",
989
+ "model.visual.backbone.layers.4.self_attn.v_proj.bias",
990
+ "model.visual.backbone.layers.4.self_attn.v_proj.weight",
991
+ "model.visual.backbone.layers.5.ln_1.bias",
992
+ "model.visual.backbone.layers.5.ln_1.weight",
993
+ "model.visual.backbone.layers.5.ln_2.bias",
994
+ "model.visual.backbone.layers.5.ln_2.weight",
995
+ "model.visual.backbone.layers.5.ls_1.gamma",
996
+ "model.visual.backbone.layers.5.ls_2.gamma",
997
+ "model.visual.backbone.layers.5.mlp.c_fc.bias",
998
+ "model.visual.backbone.layers.5.mlp.c_fc.weight",
999
+ "model.visual.backbone.layers.5.mlp.c_proj.bias",
1000
+ "model.visual.backbone.layers.5.mlp.c_proj.weight",
1001
+ "model.visual.backbone.layers.5.self_attn.k_proj.bias",
1002
+ "model.visual.backbone.layers.5.self_attn.k_proj.weight",
1003
+ "model.visual.backbone.layers.5.self_attn.out_proj.bias",
1004
+ "model.visual.backbone.layers.5.self_attn.out_proj.weight",
1005
+ "model.visual.backbone.layers.5.self_attn.q_proj.bias",
1006
+ "model.visual.backbone.layers.5.self_attn.q_proj.weight",
1007
+ "model.visual.backbone.layers.5.self_attn.v_proj.bias",
1008
+ "model.visual.backbone.layers.5.self_attn.v_proj.weight",
1009
+ "model.visual.backbone.layers.6.ln_1.bias",
1010
+ "model.visual.backbone.layers.6.ln_1.weight",
1011
+ "model.visual.backbone.layers.6.ln_2.bias",
1012
+ "model.visual.backbone.layers.6.ln_2.weight",
1013
+ "model.visual.backbone.layers.6.ls_1.gamma",
1014
+ "model.visual.backbone.layers.6.ls_2.gamma",
1015
+ "model.visual.backbone.layers.6.mlp.c_fc.bias",
1016
+ "model.visual.backbone.layers.6.mlp.c_fc.weight",
1017
+ "model.visual.backbone.layers.6.mlp.c_proj.bias",
1018
+ "model.visual.backbone.layers.6.mlp.c_proj.weight",
1019
+ "model.visual.backbone.layers.6.self_attn.k_proj.bias",
1020
+ "model.visual.backbone.layers.6.self_attn.k_proj.weight",
1021
+ "model.visual.backbone.layers.6.self_attn.out_proj.bias",
1022
+ "model.visual.backbone.layers.6.self_attn.out_proj.weight",
1023
+ "model.visual.backbone.layers.6.self_attn.q_proj.bias",
1024
+ "model.visual.backbone.layers.6.self_attn.q_proj.weight",
1025
+ "model.visual.backbone.layers.6.self_attn.v_proj.bias",
1026
+ "model.visual.backbone.layers.6.self_attn.v_proj.weight",
1027
+ "model.visual.backbone.layers.7.ln_1.bias",
1028
+ "model.visual.backbone.layers.7.ln_1.weight",
1029
+ "model.visual.backbone.layers.7.ln_2.bias",
1030
+ "model.visual.backbone.layers.7.ln_2.weight",
1031
+ "model.visual.backbone.layers.7.ls_1.gamma",
1032
+ "model.visual.backbone.layers.7.ls_2.gamma",
1033
+ "model.visual.backbone.layers.7.mlp.c_fc.bias",
1034
+ "model.visual.backbone.layers.7.mlp.c_fc.weight",
1035
+ "model.visual.backbone.layers.7.mlp.c_proj.bias",
1036
+ "model.visual.backbone.layers.7.mlp.c_proj.weight",
1037
+ "model.visual.backbone.layers.7.self_attn.k_proj.bias",
1038
+ "model.visual.backbone.layers.7.self_attn.k_proj.weight",
1039
+ "model.visual.backbone.layers.7.self_attn.out_proj.bias",
1040
+ "model.visual.backbone.layers.7.self_attn.out_proj.weight",
1041
+ "model.visual.backbone.layers.7.self_attn.q_proj.bias",
1042
+ "model.visual.backbone.layers.7.self_attn.q_proj.weight",
1043
+ "model.visual.backbone.layers.7.self_attn.v_proj.bias",
1044
+ "model.visual.backbone.layers.7.self_attn.v_proj.weight",
1045
+ "model.visual.backbone.layers.8.ln_1.bias",
1046
+ "model.visual.backbone.layers.8.ln_1.weight",
1047
+ "model.visual.backbone.layers.8.ln_2.bias",
1048
+ "model.visual.backbone.layers.8.ln_2.weight",
1049
+ "model.visual.backbone.layers.8.ls_1.gamma",
1050
+ "model.visual.backbone.layers.8.ls_2.gamma",
1051
+ "model.visual.backbone.layers.8.mlp.c_fc.bias",
1052
+ "model.visual.backbone.layers.8.mlp.c_fc.weight",
1053
+ "model.visual.backbone.layers.8.mlp.c_proj.bias",
1054
+ "model.visual.backbone.layers.8.mlp.c_proj.weight",
1055
+ "model.visual.backbone.layers.8.self_attn.k_proj.bias",
1056
+ "model.visual.backbone.layers.8.self_attn.k_proj.weight",
1057
+ "model.visual.backbone.layers.8.self_attn.out_proj.bias",
1058
+ "model.visual.backbone.layers.8.self_attn.out_proj.weight",
1059
+ "model.visual.backbone.layers.8.self_attn.q_proj.bias",
1060
+ "model.visual.backbone.layers.8.self_attn.q_proj.weight",
1061
+ "model.visual.backbone.layers.8.self_attn.v_proj.bias",
1062
+ "model.visual.backbone.layers.8.self_attn.v_proj.weight",
1063
+ "model.visual.backbone.layers.9.ln_1.bias",
1064
+ "model.visual.backbone.layers.9.ln_1.weight",
1065
+ "model.visual.backbone.layers.9.ln_2.bias",
1066
+ "model.visual.backbone.layers.9.ln_2.weight",
1067
+ "model.visual.backbone.layers.9.ls_1.gamma",
1068
+ "model.visual.backbone.layers.9.ls_2.gamma",
1069
+ "model.visual.backbone.layers.9.mlp.c_fc.bias",
1070
+ "model.visual.backbone.layers.9.mlp.c_fc.weight",
1071
+ "model.visual.backbone.layers.9.mlp.c_proj.bias",
1072
+ "model.visual.backbone.layers.9.mlp.c_proj.weight",
1073
+ "model.visual.backbone.layers.9.self_attn.k_proj.bias",
1074
+ "model.visual.backbone.layers.9.self_attn.k_proj.weight",
1075
+ "model.visual.backbone.layers.9.self_attn.out_proj.bias",
1076
+ "model.visual.backbone.layers.9.self_attn.out_proj.weight",
1077
+ "model.visual.backbone.layers.9.self_attn.q_proj.bias",
1078
+ "model.visual.backbone.layers.9.self_attn.q_proj.weight",
1079
+ "model.visual.backbone.layers.9.self_attn.v_proj.bias",
1080
+ "model.visual.backbone.layers.9.self_attn.v_proj.weight",
1081
+ "model.visual.backbone.ln_pre.bias",
1082
+ "model.visual.backbone.ln_pre.weight",
1083
+ "model.visual.proj_in.weight",
1084
+ "model.visual.proj_out.weight"
1085
+ ],
1086
+ "weight_block_shape": [],
1087
+ "weight_transform": "cast"
1088
+ },
1089
+ {
1090
+ "compute_dtype": "nvfp4",
1091
+ "output_dtype": "uint8",
1092
+ "required_activation_quantization": {
1093
+ "amax_type": "per_token",
1094
+ "block_size": 16,
1095
+ "format": "nvfp4_e2m1",
1096
+ "rounding": "nearest",
1097
+ "scale_dtype": "float8_e4m3fn",
1098
+ "scale_selection_rule": "fouroversix_mae"
1099
+ },
1100
+ "tensors": [
1101
+ "model.llm.layers.0.mlp.experts.w13_weight",
1102
+ "model.llm.layers.0.mlp.experts.w2_weight",
1103
+ "model.llm.layers.1.mlp.experts.w13_weight",
1104
+ "model.llm.layers.1.mlp.experts.w2_weight",
1105
+ "model.llm.layers.10.mlp.experts.w13_weight",
1106
+ "model.llm.layers.10.mlp.experts.w2_weight",
1107
+ "model.llm.layers.11.mlp.experts.w13_weight",
1108
+ "model.llm.layers.11.mlp.experts.w2_weight",
1109
+ "model.llm.layers.12.mlp.experts.w13_weight",
1110
+ "model.llm.layers.12.mlp.experts.w2_weight",
1111
+ "model.llm.layers.13.mlp.experts.w13_weight",
1112
+ "model.llm.layers.13.mlp.experts.w2_weight",
1113
+ "model.llm.layers.14.mlp.experts.w13_weight",
1114
+ "model.llm.layers.14.mlp.experts.w2_weight",
1115
+ "model.llm.layers.15.mlp.experts.w13_weight",
1116
+ "model.llm.layers.15.mlp.experts.w2_weight",
1117
+ "model.llm.layers.16.mlp.experts.w13_weight",
1118
+ "model.llm.layers.16.mlp.experts.w2_weight",
1119
+ "model.llm.layers.17.mlp.experts.w13_weight",
1120
+ "model.llm.layers.17.mlp.experts.w2_weight",
1121
+ "model.llm.layers.18.mlp.experts.w13_weight",
1122
+ "model.llm.layers.18.mlp.experts.w2_weight",
1123
+ "model.llm.layers.19.mlp.experts.w13_weight",
1124
+ "model.llm.layers.19.mlp.experts.w2_weight",
1125
+ "model.llm.layers.2.mlp.experts.w13_weight",
1126
+ "model.llm.layers.2.mlp.experts.w2_weight",
1127
+ "model.llm.layers.3.mlp.experts.w13_weight",
1128
+ "model.llm.layers.3.mlp.experts.w2_weight",
1129
+ "model.llm.layers.4.mlp.experts.w13_weight",
1130
+ "model.llm.layers.4.mlp.experts.w2_weight",
1131
+ "model.llm.layers.5.mlp.experts.w13_weight",
1132
+ "model.llm.layers.5.mlp.experts.w2_weight",
1133
+ "model.llm.layers.6.mlp.experts.w13_weight",
1134
+ "model.llm.layers.6.mlp.experts.w2_weight",
1135
+ "model.llm.layers.7.mlp.experts.w13_weight",
1136
+ "model.llm.layers.7.mlp.experts.w2_weight",
1137
+ "model.llm.layers.8.mlp.experts.w13_weight",
1138
+ "model.llm.layers.8.mlp.experts.w2_weight",
1139
+ "model.llm.layers.9.mlp.experts.w13_weight",
1140
+ "model.llm.layers.9.mlp.experts.w2_weight"
1141
+ ],
1142
+ "weight_block_shape": [
1143
+ 1,
1144
+ 16
1145
+ ],
1146
+ "weight_scale_format": "amax_denominator",
1147
+ "weight_transform": "nvfp4_nearest"
1148
+ },
1149
+ {
1150
+ "compute_dtype": "mxfp8",
1151
+ "output_dtype": "float8_e4m3fn",
1152
+ "required_activation_quantization": {
1153
+ "block_size": 32,
1154
+ "format": "mxfp8_e4m3",
1155
+ "rounding": "nearest",
1156
+ "scale_dtype": "ue8m0"
1157
+ },
1158
+ "tensors": [
1159
+ "model.llm.layers.20.mlp.experts.w13_weight",
1160
+ "model.llm.layers.20.mlp.experts.w2_weight",
1161
+ "model.llm.layers.21.mlp.experts.w13_weight",
1162
+ "model.llm.layers.21.mlp.experts.w2_weight",
1163
+ "model.llm.layers.22.mlp.experts.w13_weight",
1164
+ "model.llm.layers.22.mlp.experts.w2_weight",
1165
+ "model.llm.layers.23.mlp.experts.w13_weight",
1166
+ "model.llm.layers.23.mlp.experts.w2_weight"
1167
+ ],
1168
+ "weight_block_shape": [
1169
+ 32,
1170
+ 32
1171
+ ],
1172
+ "weight_transform": "mxfp8_nearest"
1173
+ },
1174
+ {
1175
+ "compute_dtype": "bfloat16",
1176
+ "fwd_output_dtype": "float32",
1177
+ "output_dtype": "bfloat16",
1178
+ "required_activation_quantization": null,
1179
+ "tensors": [
1180
+ "model.llm.unembed.weight"
1181
+ ],
1182
+ "weight_block_shape": [],
1183
+ "weight_transform": "cast"
1184
+ }
1185
+ ]
1186
+ },
1187
+ "mtp": {
1188
+ "parameter_groups": [
1189
+ {
1190
+ "compute_dtype": "bfloat16",
1191
+ "output_dtype": "bfloat16",
1192
+ "required_activation_quantization": null,
1193
+ "tensors": [
1194
+ "fuse_weight",
1195
+ "layers.0.dflare_norm.weight",
1196
+ "layers.0.input_layernorm.weight",
1197
+ "layers.0.mlp.down_proj.weight",
1198
+ "layers.0.mlp.gate_proj.weight",
1199
+ "layers.0.mlp.up_proj.weight",
1200
+ "layers.0.post_attention_layernorm.weight",
1201
+ "layers.0.self_attn.k_norm.weight",
1202
+ "layers.0.self_attn.k_norm_ctx.weight",
1203
+ "layers.0.self_attn.k_proj.weight",
1204
+ "layers.0.self_attn.k_proj_ctx.weight",
1205
+ "layers.0.self_attn.o_proj.weight",
1206
+ "layers.0.self_attn.q_norm.weight",
1207
+ "layers.0.self_attn.q_proj.weight",
1208
+ "layers.0.self_attn.v_proj.weight",
1209
+ "layers.0.self_attn.v_proj_ctx.weight",
1210
+ "layers.1.dflare_norm.weight",
1211
+ "layers.1.input_layernorm.weight",
1212
+ "layers.1.mlp.down_proj.weight",
1213
+ "layers.1.mlp.gate_proj.weight",
1214
+ "layers.1.mlp.up_proj.weight",
1215
+ "layers.1.post_attention_layernorm.weight",
1216
+ "layers.1.self_attn.k_norm.weight",
1217
+ "layers.1.self_attn.k_norm_ctx.weight",
1218
+ "layers.1.self_attn.k_proj.weight",
1219
+ "layers.1.self_attn.k_proj_ctx.weight",
1220
+ "layers.1.self_attn.o_proj.weight",
1221
+ "layers.1.self_attn.q_norm.weight",
1222
+ "layers.1.self_attn.q_proj.weight",
1223
+ "layers.1.self_attn.v_proj.weight",
1224
+ "layers.1.self_attn.v_proj_ctx.weight",
1225
+ "layers.2.dflare_norm.weight",
1226
+ "layers.2.input_layernorm.weight",
1227
+ "layers.2.mlp.down_proj.weight",
1228
+ "layers.2.mlp.gate_proj.weight",
1229
+ "layers.2.mlp.up_proj.weight",
1230
+ "layers.2.post_attention_layernorm.weight",
1231
+ "layers.2.self_attn.k_norm.weight",
1232
+ "layers.2.self_attn.k_norm_ctx.weight",
1233
+ "layers.2.self_attn.k_proj.weight",
1234
+ "layers.2.self_attn.k_proj_ctx.weight",
1235
+ "layers.2.self_attn.o_proj.weight",
1236
+ "layers.2.self_attn.q_norm.weight",
1237
+ "layers.2.self_attn.q_proj.weight",
1238
+ "layers.2.self_attn.v_proj.weight",
1239
+ "layers.2.self_attn.v_proj_ctx.weight",
1240
+ "markov_head.dflare_norm.weight",
1241
+ "markov_head.hidden_out.bias",
1242
+ "markov_head.hidden_out.weight",
1243
+ "markov_head.markov_w1.weight",
1244
+ "markov_head.rnn_proj.bias",
1245
+ "markov_head.rnn_proj.weight",
1246
+ "mask_embed",
1247
+ "norm.weight"
1248
+ ],
1249
+ "weight_block_shape": [],
1250
+ "weight_transform": "cast"
1251
+ }
1252
+ ]
1253
+ },
1254
+ "quant_method": "inkling2"
1255
+ }
model-00001-of-00001.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:79e60b65c4f2a2a5acdb8e3c7b6994bf00da5477f45d18027afbfad752a64595
3
+ size 19526019168
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
mtp.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:905d548692b1eb1d1aa278b8798db66ff88d94d0fc7609bf77baab90e99c76c7
3
+ size 635801416
processor_config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_convert_rgb": true,
4
+ "do_normalize": true,
5
+ "do_rescale": true,
6
+ "image_mean": [
7
+ 0.5,
8
+ 0.5,
9
+ 0.5
10
+ ],
11
+ "image_processor_type": "Inkling2ImageProcessor",
12
+ "image_std": [
13
+ 0.5,
14
+ 0.5,
15
+ 0.5
16
+ ],
17
+ "rescale_factor": 0.00392156862745098,
18
+ "size": {
19
+ "height": 40,
20
+ "width": 40
21
+ }
22
+ },
23
+ "image_token": "<|unused_200054|>",
24
+ "processor_class": "Inkling2Processor"
25
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|endoftext|>",
4
+ "<|message_user|>",
5
+ "<|message_model|>",
6
+ "<|message_system|>",
7
+ "<|message_tool|>",
8
+ "<|content_text|>",
9
+ "<|content_image|>",
10
+ "<|content_model_end_sampling|>",
11
+ "<|content_thinking|>",
12
+ "<|end_message|>",
13
+ "<|content_tool_error|>",
14
+ "<|content_xml|>",
15
+ "<|begin_of_text|>",
16
+ "<|content_invoke_tool_json|>",
17
+ "<|content_invoke_tool_text|>"
18
+ ],
19
+ "eos_token": "<|content_model_end_sampling|>",
20
+ "pad_token": "<|endoftext|>"
21
+ }
tiktoken/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f170372dd655728909228220d77cf6b8104a59f38a91262a7d2c4027906ec0d8
3
+ size 3386494
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f332727f9a3f093dd452a39cc978e6757627e5dad1b0673aa4e75174f236dd6
3
+ size 26178678
tokenizer_config.json ADDED
@@ -0,0 +1,548 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "model_max_length": 1048576,
4
+ "clean_up_tokenization_spaces": false,
5
+ "fix_mistral_regex": false,
6
+ "extra_special_tokens": {
7
+ "endoftext": "<|endoftext|>",
8
+ "message_user": "<|message_user|>",
9
+ "message_model": "<|message_model|>",
10
+ "message_system": "<|message_system|>",
11
+ "message_tool": "<|message_tool|>",
12
+ "content_text": "<|content_text|>",
13
+ "content_image": "<|content_image|>",
14
+ "content_model_end_sampling": "<|content_model_end_sampling|>",
15
+ "content_thinking": "<|content_thinking|>",
16
+ "end_message": "<|end_message|>",
17
+ "content_tool_error": "<|content_tool_error|>",
18
+ "content_xml": "<|content_xml|>",
19
+ "begin_of_text": "<|begin_of_text|>",
20
+ "content_invoke_tool_json": "<|content_invoke_tool_json|>",
21
+ "content_invoke_tool_text": "<|content_invoke_tool_text|>"
22
+ },
23
+ "added_tokens_decoder": {
24
+ "199998": {
25
+ "content": "<|unused|>",
26
+ "single_word": false,
27
+ "lstrip": false,
28
+ "rstrip": false,
29
+ "normalized": false,
30
+ "special": true
31
+ },
32
+ "199999": {
33
+ "content": "<|endoftext|>",
34
+ "single_word": false,
35
+ "lstrip": false,
36
+ "rstrip": false,
37
+ "normalized": false,
38
+ "special": true
39
+ },
40
+ "200000": {
41
+ "content": "<|message_user|>",
42
+ "single_word": false,
43
+ "lstrip": false,
44
+ "rstrip": false,
45
+ "normalized": false,
46
+ "special": true
47
+ },
48
+ "200001": {
49
+ "content": "<|message_model|>",
50
+ "single_word": false,
51
+ "lstrip": false,
52
+ "rstrip": false,
53
+ "normalized": false,
54
+ "special": true
55
+ },
56
+ "200002": {
57
+ "content": "<|message_system|>",
58
+ "single_word": false,
59
+ "lstrip": false,
60
+ "rstrip": false,
61
+ "normalized": false,
62
+ "special": true
63
+ },
64
+ "200003": {
65
+ "content": "<|message_tool|>",
66
+ "single_word": false,
67
+ "lstrip": false,
68
+ "rstrip": false,
69
+ "normalized": false,
70
+ "special": true
71
+ },
72
+ "200004": {
73
+ "content": "<|content_text|>",
74
+ "single_word": false,
75
+ "lstrip": false,
76
+ "rstrip": false,
77
+ "normalized": false,
78
+ "special": true
79
+ },
80
+ "200005": {
81
+ "content": "<|content_image|>",
82
+ "single_word": false,
83
+ "lstrip": false,
84
+ "rstrip": false,
85
+ "normalized": false,
86
+ "special": true
87
+ },
88
+ "200006": {
89
+ "content": "<|content_model_end_sampling|>",
90
+ "single_word": false,
91
+ "lstrip": false,
92
+ "rstrip": false,
93
+ "normalized": false,
94
+ "special": true
95
+ },
96
+ "200007": {
97
+ "content": "<|unused_200007|>",
98
+ "single_word": false,
99
+ "lstrip": false,
100
+ "rstrip": false,
101
+ "normalized": false,
102
+ "special": true
103
+ },
104
+ "200008": {
105
+ "content": "<|content_thinking|>",
106
+ "single_word": false,
107
+ "lstrip": false,
108
+ "rstrip": false,
109
+ "normalized": false,
110
+ "special": true
111
+ },
112
+ "200009": {
113
+ "content": "<|unused_200009|>",
114
+ "single_word": false,
115
+ "lstrip": false,
116
+ "rstrip": false,
117
+ "normalized": false,
118
+ "special": true
119
+ },
120
+ "200010": {
121
+ "content": "<|end_message|>",
122
+ "single_word": false,
123
+ "lstrip": false,
124
+ "rstrip": false,
125
+ "normalized": false,
126
+ "special": true
127
+ },
128
+ "200011": {
129
+ "content": "<|unused_200011|>",
130
+ "single_word": false,
131
+ "lstrip": false,
132
+ "rstrip": false,
133
+ "normalized": false,
134
+ "special": true
135
+ },
136
+ "200012": {
137
+ "content": "<|unused_200012|>",
138
+ "single_word": false,
139
+ "lstrip": false,
140
+ "rstrip": false,
141
+ "normalized": false,
142
+ "special": true
143
+ },
144
+ "200013": {
145
+ "content": "<|unused_200013|>",
146
+ "single_word": false,
147
+ "lstrip": false,
148
+ "rstrip": false,
149
+ "normalized": false,
150
+ "special": true
151
+ },
152
+ "200014": {
153
+ "content": "<|unused_200014|>",
154
+ "single_word": false,
155
+ "lstrip": false,
156
+ "rstrip": false,
157
+ "normalized": false,
158
+ "special": true
159
+ },
160
+ "200015": {
161
+ "content": "<|unused_200015|>",
162
+ "single_word": false,
163
+ "lstrip": false,
164
+ "rstrip": false,
165
+ "normalized": false,
166
+ "special": true
167
+ },
168
+ "200016": {
169
+ "content": "<|unused_200016|>",
170
+ "single_word": false,
171
+ "lstrip": false,
172
+ "rstrip": false,
173
+ "normalized": false,
174
+ "special": true
175
+ },
176
+ "200017": {
177
+ "content": "<|unused_200017|>",
178
+ "single_word": false,
179
+ "lstrip": false,
180
+ "rstrip": false,
181
+ "normalized": false,
182
+ "special": true
183
+ },
184
+ "200018": {
185
+ "content": "<|unused_200018|>",
186
+ "single_word": false,
187
+ "lstrip": false,
188
+ "rstrip": false,
189
+ "normalized": false,
190
+ "special": true
191
+ },
192
+ "200019": {
193
+ "content": "<|unused_200019|>",
194
+ "single_word": false,
195
+ "lstrip": false,
196
+ "rstrip": false,
197
+ "normalized": false,
198
+ "special": true
199
+ },
200
+ "200020": {
201
+ "content": "<|unused_200020|>",
202
+ "single_word": false,
203
+ "lstrip": false,
204
+ "rstrip": false,
205
+ "normalized": false,
206
+ "special": true
207
+ },
208
+ "200021": {
209
+ "content": "<|unused_200021|>",
210
+ "single_word": false,
211
+ "lstrip": false,
212
+ "rstrip": false,
213
+ "normalized": false,
214
+ "special": true
215
+ },
216
+ "200022": {
217
+ "content": "<|content_tool_error|>",
218
+ "single_word": false,
219
+ "lstrip": false,
220
+ "rstrip": false,
221
+ "normalized": false,
222
+ "special": true
223
+ },
224
+ "200023": {
225
+ "content": "<|unused_200023|>",
226
+ "single_word": false,
227
+ "lstrip": false,
228
+ "rstrip": false,
229
+ "normalized": false,
230
+ "special": true
231
+ },
232
+ "200024": {
233
+ "content": "<|content_xml|>",
234
+ "single_word": false,
235
+ "lstrip": false,
236
+ "rstrip": false,
237
+ "normalized": false,
238
+ "special": true
239
+ },
240
+ "200025": {
241
+ "content": "<|unused_200025|>",
242
+ "single_word": false,
243
+ "lstrip": false,
244
+ "rstrip": false,
245
+ "normalized": false,
246
+ "special": true
247
+ },
248
+ "200026": {
249
+ "content": "<|unused_200026|>",
250
+ "single_word": false,
251
+ "lstrip": false,
252
+ "rstrip": false,
253
+ "normalized": false,
254
+ "special": true
255
+ },
256
+ "200027": {
257
+ "content": "<|unused_200027|>",
258
+ "single_word": false,
259
+ "lstrip": false,
260
+ "rstrip": false,
261
+ "normalized": false,
262
+ "special": true
263
+ },
264
+ "200028": {
265
+ "content": "<|begin_of_text|>",
266
+ "single_word": false,
267
+ "lstrip": false,
268
+ "rstrip": false,
269
+ "normalized": false,
270
+ "special": true
271
+ },
272
+ "200029": {
273
+ "content": "<|unused_200029|>",
274
+ "single_word": false,
275
+ "lstrip": false,
276
+ "rstrip": false,
277
+ "normalized": false,
278
+ "special": true
279
+ },
280
+ "200030": {
281
+ "content": "<|unused_200030|>",
282
+ "single_word": false,
283
+ "lstrip": false,
284
+ "rstrip": false,
285
+ "normalized": false,
286
+ "special": true
287
+ },
288
+ "200031": {
289
+ "content": "<|unused_200031|>",
290
+ "single_word": false,
291
+ "lstrip": false,
292
+ "rstrip": false,
293
+ "normalized": false,
294
+ "special": true
295
+ },
296
+ "200032": {
297
+ "content": "<|unused_200032|>",
298
+ "single_word": false,
299
+ "lstrip": false,
300
+ "rstrip": false,
301
+ "normalized": false,
302
+ "special": true
303
+ },
304
+ "200033": {
305
+ "content": "<|unused_200033|>",
306
+ "single_word": false,
307
+ "lstrip": false,
308
+ "rstrip": false,
309
+ "normalized": false,
310
+ "special": true
311
+ },
312
+ "200034": {
313
+ "content": "<|unused_200034|>",
314
+ "single_word": false,
315
+ "lstrip": false,
316
+ "rstrip": false,
317
+ "normalized": false,
318
+ "special": true
319
+ },
320
+ "200035": {
321
+ "content": "<|unused_200035|>",
322
+ "single_word": false,
323
+ "lstrip": false,
324
+ "rstrip": false,
325
+ "normalized": false,
326
+ "special": true
327
+ },
328
+ "200036": {
329
+ "content": "<|unused_200036|>",
330
+ "single_word": false,
331
+ "lstrip": false,
332
+ "rstrip": false,
333
+ "normalized": false,
334
+ "special": true
335
+ },
336
+ "200037": {
337
+ "content": "<|unused_200037|>",
338
+ "single_word": false,
339
+ "lstrip": false,
340
+ "rstrip": false,
341
+ "normalized": false,
342
+ "special": true
343
+ },
344
+ "200038": {
345
+ "content": "<|unused_200038|>",
346
+ "single_word": false,
347
+ "lstrip": false,
348
+ "rstrip": false,
349
+ "normalized": false,
350
+ "special": true
351
+ },
352
+ "200039": {
353
+ "content": "<|unused_200039|>",
354
+ "single_word": false,
355
+ "lstrip": false,
356
+ "rstrip": false,
357
+ "normalized": false,
358
+ "special": true
359
+ },
360
+ "200040": {
361
+ "content": "<|unused_200040|>",
362
+ "single_word": false,
363
+ "lstrip": false,
364
+ "rstrip": false,
365
+ "normalized": false,
366
+ "special": true
367
+ },
368
+ "200041": {
369
+ "content": "<|unused_200041|>",
370
+ "single_word": false,
371
+ "lstrip": false,
372
+ "rstrip": false,
373
+ "normalized": false,
374
+ "special": true
375
+ },
376
+ "200042": {
377
+ "content": "<|unused_200042|>",
378
+ "single_word": false,
379
+ "lstrip": false,
380
+ "rstrip": false,
381
+ "normalized": false,
382
+ "special": true
383
+ },
384
+ "200043": {
385
+ "content": "<|unused_200043|>",
386
+ "single_word": false,
387
+ "lstrip": false,
388
+ "rstrip": false,
389
+ "normalized": false,
390
+ "special": true
391
+ },
392
+ "200044": {
393
+ "content": "<|unused_200044|>",
394
+ "single_word": false,
395
+ "lstrip": false,
396
+ "rstrip": false,
397
+ "normalized": false,
398
+ "special": true
399
+ },
400
+ "200045": {
401
+ "content": "<|unused_200045|>",
402
+ "single_word": false,
403
+ "lstrip": false,
404
+ "rstrip": false,
405
+ "normalized": false,
406
+ "special": true
407
+ },
408
+ "200046": {
409
+ "content": "<|unused_200046|>",
410
+ "single_word": false,
411
+ "lstrip": false,
412
+ "rstrip": false,
413
+ "normalized": false,
414
+ "special": true
415
+ },
416
+ "200047": {
417
+ "content": "<|unused_200047|>",
418
+ "single_word": false,
419
+ "lstrip": false,
420
+ "rstrip": false,
421
+ "normalized": false,
422
+ "special": true
423
+ },
424
+ "200048": {
425
+ "content": "<|unused_200048|>",
426
+ "single_word": false,
427
+ "lstrip": false,
428
+ "rstrip": false,
429
+ "normalized": false,
430
+ "special": true
431
+ },
432
+ "200049": {
433
+ "content": "<|content_invoke_tool_json|>",
434
+ "single_word": false,
435
+ "lstrip": false,
436
+ "rstrip": false,
437
+ "normalized": false,
438
+ "special": true
439
+ },
440
+ "200050": {
441
+ "content": "<|unused_200050|>",
442
+ "single_word": false,
443
+ "lstrip": false,
444
+ "rstrip": false,
445
+ "normalized": false,
446
+ "special": true
447
+ },
448
+ "200051": {
449
+ "content": "<|unused_200051|>",
450
+ "single_word": false,
451
+ "lstrip": false,
452
+ "rstrip": false,
453
+ "normalized": false,
454
+ "special": true
455
+ },
456
+ "200052": {
457
+ "content": "<|unused_200052|>",
458
+ "single_word": false,
459
+ "lstrip": false,
460
+ "rstrip": false,
461
+ "normalized": false,
462
+ "special": true
463
+ },
464
+ "200053": {
465
+ "content": "<|unused_200053|>",
466
+ "single_word": false,
467
+ "lstrip": false,
468
+ "rstrip": false,
469
+ "normalized": false,
470
+ "special": true
471
+ },
472
+ "200054": {
473
+ "content": "<|unused_200054|>",
474
+ "single_word": false,
475
+ "lstrip": false,
476
+ "rstrip": false,
477
+ "normalized": false,
478
+ "special": true
479
+ },
480
+ "200055": {
481
+ "content": "<|unused_200055|>",
482
+ "single_word": false,
483
+ "lstrip": false,
484
+ "rstrip": false,
485
+ "normalized": false,
486
+ "special": true
487
+ },
488
+ "200056": {
489
+ "content": "<|unused_200056|>",
490
+ "single_word": false,
491
+ "lstrip": false,
492
+ "rstrip": false,
493
+ "normalized": false,
494
+ "special": true
495
+ },
496
+ "200057": {
497
+ "content": "<|content_invoke_tool_text|>",
498
+ "single_word": false,
499
+ "lstrip": false,
500
+ "rstrip": false,
501
+ "normalized": false,
502
+ "special": true
503
+ }
504
+ },
505
+ "response_template": {
506
+ "defaults": {
507
+ "role": "assistant"
508
+ },
509
+ "start_anchor_pattern": "<\\|end_message\\|>|<\\|content_model_end_sampling\\|>",
510
+ "fields": {
511
+ "reasoning_content": {
512
+ "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_thinking\\|>",
513
+ "close": "<|end_message|>",
514
+ "repeats": true,
515
+ "join": "",
516
+ "content": "text",
517
+ "content_args": {
518
+ "strip": false
519
+ }
520
+ },
521
+ "content": {
522
+ "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_(?:text|tool_error|invoke_tool_text)\\|>",
523
+ "close": "<|end_message|>",
524
+ "repeats": true,
525
+ "join": "",
526
+ "content": "text",
527
+ "content_args": {
528
+ "strip": false
529
+ }
530
+ },
531
+ "tool_calls": {
532
+ "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_invoke_tool_json\\|>",
533
+ "close": "<|end_message|>",
534
+ "repeats": true,
535
+ "content": "json",
536
+ "transform": {
537
+ "type": "function",
538
+ "function": {
539
+ "name": "{content.name}",
540
+ "arguments": "{content.args}"
541
+ }
542
+ }
543
+ }
544
+ }
545
+ },
546
+ "eos_token": "<|content_model_end_sampling|>",
547
+ "pad_token": "<|endoftext|>"
548
+ }