Duplicate from zai-org/GLM-5.3-Flash

Co-authored-by: zR <ZHANGYUXUAN-zR@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Z.AI Co., Ltd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,68 @@
+---
+language:
+- en
+- zh
+library_name: transformers
+license: mit
+pipeline_tag: text-generation
+---
+
+# GLM-5.3-Flash
+
+<div align="center">
+<img src=https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/logo.svg width="15%"/>
+</div>
+<p align="center">
+    👋 Join our <a href="https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/wechat.png" target="_blank">WeChat</a> or <a href="https://discord.gg/QR7SARHRxK" target="_blank">Discord</a> community.
+    <br>
+    📖 Check out the GLM-5.3-Flash <a href="https://z.ai/blog/glm-5.3-flash" target="_blank">blog</a> and GLM-5 <a href="https://arxiv.org/abs/2602.15763" target="_blank">Technical report</a>.
+    <br>
+    📍 Use GLM-5.3-Flash API services on <a href="https://docs.z.ai/guides/llm/glm-5.3-flash">Z.ai API Platform. </a>
+</p>
+
+## Introduction
+
+We introduce GLM-5.3-Flash, the first natively multimodal model in the GLM-5 series. With 320B total parameters and just 18B active parameters, it outperforms GLM-5.2 across benchmarks and real-world workloads at one-tenth the price, while approaching Claude Opus 4.8 on coding and agentic benchmarks.
+
+GLM-5.3-Flash starts from a newly trained base model, with its architecture and training recipe redesigned around capability and efficiency. For the first time in the GLM series, we introduce a hybrid architecture combining sparse and linear attention, sharply reducing long-context serving costs while preserving precise long-context capabilities. The model also adopts Manifold-Constrained Hyper-Connections (mHC) to further improve scaling efficiency. Together with our latest 30T-token multimodal pre-training corpus, these changes enable GLM-5.3-Flash to deliver more intelligence with less compute.
+
+![bench_53](https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/bench_53.png)
+
+
+## Serve GLM-5.3-Flash Locally
+
+GLM-5.3-Flash supports deployment with the following frameworks. Feel free to try them out:
+
+- [SGLang](https://github.com/sgl-project/sglang) — see [cookbook](https://cookbook.sglang.io/autoregressive/GLM/GLM-5.3-Flash)
+- [vLLM](https://github.com/vllm-project/vllm) — see [recipes](https://recipes.vllm.ai/zai-org/GLM-5.3-Flash)
+- [TokenSpeed](https://github.com/lightseekorg/tokenspeed) — see [here](https://lightseek.org/tokenspeed/recipes/models#glm-5-3-flash)
+- [KTransformers](https://github.com/kvcache-ai/ktransformers) — see [tutorial](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/GLM-5.3-Flash-Tutorial.md)
+
+
+## Footnotes
+
+* **HLE w/ tools (full set)**: We use sampling parameters of `temperature=1.0` and `top_p=0.95` for evaluation, with a maximum generation length of `163,840` tokens. The evaluation is conducted with a maximum context length of `300,000` tokens, using a context management strategy. We use GPT-5.6-luna (medium) as the judge model.
+* **NL2Repo**: We evaluated NL2Repo with temperature=1.0, top_p=1.0, and max_new_tokens=64k under 1M context. To prevent hacking, we use rule-based and a LLM-based judgement to prevent malicious behaviors (e.g., unauthorized pip or curl operations).
+* **DeepSWE**: We run DeepSWE using the mini-swe-agent harness with `temperature=0.95`, `top_p=1.0`, `timeout=6h` and 400K context.
+* **Terminal-Bench 2.1**: We evaluate in Claude Code 2.1.207 with temperature=1.0, top_p=1, max_new_tokens=65536 with 6h timeout.
+* **Agent’s Last Exam**: 
+* **Toolathlon Verified**: We obtain all results via the official evaluation service and report pass@1 averaged over 3 independent runs.
+* **AutomationBench**: We evaluate on AutomationBench **v1.0.6**, incorporating the fix for the `null`-type handling issue introduced in [PR #13](https://github.com/zapier/AutomationBench/pull/13).
+* **GDPval-AA v2**: Models are evaluated by Artificial Analysis.
+* **BabyVision**: We use temperature=1.0, top_p=0.95, and a maximum context length of 164K tokens. We resize the input images such that their shorter side is at least 1.5K pixels, consistent with other baselines.
+
+## Citation
+
+If you find GLM-5.3-Flash useful in your research, please cite our technical report:
+
+```bibtex
+@misc{glm5team2026glm5vibecodingagentic,
+      title={GLM-5: from Vibe Coding to Agentic Engineering},
+      author={GLM-5-Team and : and Aohan Zeng and Xin Lv and Zhenyu Hou and Zhengxiao Du and Qinkai Zheng and Bin Chen and Da Yin and Chendi Ge and Chenghua Huang and Chengxing Xie and Chenzheng Zhu and Congfeng Yin and Cunxiang Wang and Gengzheng Pan and Hao Zeng and Haoke Zhang and Haoran Wang and Huilong Chen and Jiajie Zhang and Jian Jiao and Jiaqi Guo and Jingsen Wang and Jingzhao Du and Jinzhu Wu and Kedong Wang and Lei Li and Lin Fan and Lucen Zhong and Mingdao Liu and Mingming Zhao and Pengfan Du and Qian Dong and Rui Lu and Shuang-Li and Shulin Cao and Song Liu and Ting Jiang and Xiaodong Chen and Xiaohan Zhang and Xuancheng Huang and Xuezhen Dong and Yabo Xu and Yao Wei and Yifan An and Yilin Niu and Yitong Zhu and Yuanhao Wen and Yukuo Cen and Yushi Bai and Zhongpei Qiao and Zihan Wang and Zikang Wang and Zilin Zhu and Ziqiang Liu and Zixuan Li and Bojie Wang and Bosi Wen and Can Huang and Changpeng Cai and Chao Yu and Chen Li and Chengwei Hu and Chenhui Zhang and Dan Zhang and Daoyan Lin and Dayong Yang and Di Wang and Ding Ai and Erle Zhu and Fangzhou Yi and Feiyu Chen and Guohong Wen and Hailong Sun and Haisha Zhao and Haiyi Hu and Hanchen Zhang and Hanrui Liu and Hanyu Zhang and Hao Peng and Hao Tai and Haobo Zhang and He Liu and Hongwei Wang and Hongxi Yan and Hongyu Ge and Huan Liu and Huanpeng Chu and Jia'ni Zhao and Jiachen Wang and Jiajing Zhao and Jiamin Ren and Jiapeng Wang and Jiaxin Zhang and Jiayi Gui and Jiayue Zhao and Jijie Li and Jing An and Jing Li and Jingwei Yuan and Jinhua Du and Jinxin Liu and Junkai Zhi and Junwen Duan and Kaiyue Zhou and Kangjian Wei and Ke Wang and Keyun Luo and Laiqiang Zhang and Leigang Sha and Liang Xu and Lindong Wu and Lintao Ding and Lu Chen and Minghao Li and Nianyi Lin and Pan Ta and Qiang Zou and Rongjun Song and Ruiqi Yang and Shangqing Tu and Shangtong Yang and Shaoxiang Wu and Shengyan Zhang and Shijie Li and Shuang Li and Shuyi Fan and Wei Qin and Wei Tian and Weining Zhang and Wenbo Yu and Wenjie Liang and Xiang Kuang and Xiangmeng Cheng and Xiangyang Li and Xiaoquan Yan and Xiaowei Hu and Xiaoying Ling and Xing Fan and Xingye Xia and Xinyuan Zhang and Xinze Zhang and Xirui Pan and Xu Zou and Xunkai Zhang and Yadi Liu and Yandong Wu and Yanfu Li and Yidong Wang and Yifan Zhu and Yijun Tan and Yilin Zhou and Yiming Pan and Ying Zhang and Yinpei Su and Yipeng Geng and Yong Yan and Yonglin Tan and Yuean Bi and Yuhan Shen and Yuhao Yang and Yujiang Li and Yunan Liu and Yunqing Wang and Yuntao Li and Yurong Wu and Yutao Zhang and Yuxi Duan and Yuxuan Zhang and Zezhen Liu and Zhengtao Jiang and Zhenhe Yan and Zheyu Zhang and Zhixiang Wei and Zhuo Chen and Zhuoer Feng and Zijun Yao and Ziwei Chai and Ziyuan Wang and Zuzhou Zhang and Bin Xu and Minlie Huang and Hongning Wang and Juanzi Li and Yuxiao Dong and Jie Tang},
+      year={2026},
+      eprint={2602.15763},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2602.15763},
+}
+```

chat_template.jinja

@@ -0,0 +1,210 @@
+[gMASK]<sop>
+{%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}
+{%- if effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
+{%- if tools -%}
+{%- macro tool_to_json(tool) -%}
+    {%- set ns_tool = namespace(first=true) -%}
+    {{ '{' -}}
+    {%- for k, v in tool.items() -%}
+        {%- if k != 'defer_loading' and k != 'strict' -%}
+            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
+            {%- set ns_tool.first = false -%}
+            "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- '}' -}}
+{%- endmacro -%}
+{%- macro tool_references_to_response(refs) -%}
+    {{- '<tool_response><tools>\n' -}}
+    {%- for tr in refs -%}
+        {%- for tool in tools -%}
+            {%- if 'function' in tool -%}
+                {%- set tool = tool['function'] -%}
+            {%- endif -%}
+            {%- if tool.name == tr.name -%}
+                {{- tool_to_json(tool) + '\n' -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endfor -%}
+    {{- '</tools></tool_response>' -}}
+{%- endmacro -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{%- if 'function' in tool -%}
+    {%- set tool = tool['function'] -%}
+{%- endif -%}
+{% if tool.defer_loading is not defined or not tool.defer_loading %}
+{{ tool_to_json(tool) }}
+{% endif %}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
+                {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
+                {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro tool_response(text) -%}
+{{- '<tool_response>' + text + '</tool_response>' -}}
+{%- endmacro -%}
+{%- macro render_tool_response(m) -%}
+{%- if m.content is string -%}
+    {{- tool_response(m.content) -}}
+{%- elif m.content and m.content is not mapping and m.content.0.type == "tool_reference" -%}
+    {{- tool_references_to_response(m.content) -}}
+{%- elif is_list_of_outputs(m) -%}
+    {%- for tr in m.content -%}
+        {%- if tr.output is iterable and tr.output is not string and tr.output is not mapping and tr.output and tr.output.0.type == "tool_reference" -%}
+            {{- tool_references_to_response(tr.output) -}}
+        {%- else -%}
+            {{- tool_response(visible_text(tr.output)) -}}
+        {%- endif -%}
+    {%- endfor -%}
+{%- else -%}
+    {{- tool_response(visible_text(m.content)) -}}
+{%- endif -%}
+{%- endmacro -%}
+{%- macro id_of(obj) -%}
+    {%- if obj.tool_call_id -%}
+        {{- obj.tool_call_id -}}
+    {%- elif obj.id -%}
+        {{- obj.id -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro is_list_of_outputs(m) -%}
+    {%- if m.content and m.content.0.output is defined -%}1{%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) -%}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{%- for m in messages -%}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- elif '</think>' in content %}
+    {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
+    {%- set content = content.split('</think>')[-1] %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
+{{ '<think>' + reasoning_content +  '</think>'}}
+{%- else -%}
+{{ '<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' -}}
+    {%- set block_start = loop.index0 -%}
+    {%- set ns_blk = namespace(end=block_start) -%}
+    {%- for j in range(block_start, messages|length) -%}
+        {%- if messages[j].role == 'tool' -%}
+            {%- set ns_blk.end = j -%}
+        {%- else -%}
+            {%- break -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns_a = namespace(tool_calls=none) -%}
+    {%- if block_start > 0 and messages[block_start - 1].role == 'assistant' and messages[block_start - 1].tool_calls -%}
+        {%- set ns_a.tool_calls = messages[block_start - 1].tool_calls -%}
+    {%- endif -%}
+    {%- set ns_chk = namespace(can_sort=true) -%}
+    {%- if not ns_a.tool_calls -%}
+        {%- set ns_chk.can_sort = false -%}
+    {%- else -%}
+        {%- for k in range(block_start, ns_blk.end + 1) -%}
+            {%- set m = messages[k] -%}
+            {%- if is_list_of_outputs(m) -%}
+                {%- for entry in m.content -%}
+                    {%- set eid = id_of(entry) -%}
+                    {%- if not eid -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- else -%}
+                {%- set tk_id = id_of(m) -%}
+                {%- if not tk_id -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- endif -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- for tc in ns_a.tool_calls -%}
+            {%- set tc_id = id_of(tc) -%}
+            {%- if not tc_id -%}
+                {%- set ns_chk.can_sort = false -%}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endif -%}
+    {%- if ns_chk.can_sort -%}
+        {%- for tc in ns_a.tool_calls -%}
+            {%- set tc_id = id_of(tc) -%}
+            {%- for k in range(block_start, ns_blk.end + 1) -%}
+                {%- set m = messages[k] -%}
+                {%- if is_list_of_outputs(m) -%}
+                    {%- for entry in m.content -%}
+                        {%- set eid = id_of(entry) -%}
+                        {%- if eid == tc_id -%}
+                            {%- if entry.output is iterable and entry.output is not string and entry.output is not mapping and entry.output and entry.output.0.type == "tool_reference" -%}
+                                {{- tool_references_to_response(entry.output) -}}
+                            {%- else -%}
+                                {{- tool_response(visible_text(entry.output)) -}}
+                            {%- endif -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                {%- else -%}
+                    {%- set tk_id = id_of(m) -%}
+                    {%- if tk_id == tc_id -%}
+                        {{- render_tool_response(m) -}}
+                    {%- endif -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- for k in range(block_start, ns_blk.end + 1) -%}
+            {{- render_tool_response(messages[k]) -}}
+        {%- endfor -%}
+    {%- endif -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '<think>' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,1810 @@
+{
+  "architectures": [
+    "Glm5NextForConditionalGeneration"
+  ],
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "eos_token_id": [
+      154820,
+      154827,
+      154829
+    ],
+    "first_k_dense_replace": 3,
+    "hc_eps": 1e-06,
+    "hc_mult": 4,
+    "hc_sinkhorn_iters": 20,
+    "head_dim": 0,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "index_head_dim": 128,
+    "index_kpool": 4,
+    "index_kpool_always_select_tail": true,
+    "index_kpool_compress": true,
+    "index_n_heads": 32,
+    "index_topk": 2048,
+    "index_share_for_mtp_iteration": true,
+    "indexer_rope_interleave": true,
+    "indexer_types": [
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full",
+      "full"
+    ],
+    "initializer_range": 0.02,
+    "intermediate_size": 12288,
+    "kv_lora_rank": 512,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "deepseek_sparse_attention",
+      "linear_attention"
+    ],
+    "linear_attn_config": {
+      "num_heads": 64,
+      "gate_lower_bound": -5.0,
+      "head_dim": 128,
+      "short_conv_kernel_size": 4,
+      "kda_layers": [
+        0,
+        1,
+        2,
+        4,
+        5,
+        6,
+        8,
+        9,
+        10,
+        12,
+        13,
+        14,
+        16,
+        17,
+        18,
+        20,
+        21,
+        22,
+        24,
+        25,
+        26,
+        28,
+        29,
+        30,
+        32,
+        33,
+        34,
+        36,
+        37,
+        38,
+        40,
+        41,
+        42,
+        44
+      ],
+      "full_attn_layers": [
+        3,
+        7,
+        11,
+        15,
+        19,
+        23,
+        27,
+        31,
+        35,
+        39,
+        43
+      ]
+    },
+    "max_position_embeddings": 1048576,
+    "mhc": true,
+    "mla_use_nope": true,
+    "mlp_layer_types": [
+      "dense",
+      "dense",
+      "dense",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse"
+    ],
+    "model_type": "glm5_next_text",
+    "moe_intermediate_size": 2048,
+    "moe_router_dtype": "float32",
+    "n_group": 1,
+    "n_routed_experts": 288,
+    "n_shared_experts": 1,
+    "norm_topk_prob": true,
+    "num_attention_heads": 64,
+    "num_experts_per_tok": 8,
+    "num_hidden_layers": 45,
+    "num_key_value_heads": 64,
+    "num_nextn_predict_layers": 1,
+    "output_router_logits": false,
+    "pad_token_id": 154820,
+    "q_lora_rank": 1536,
+    "qk_head_dim": 256,
+    "qk_nope_head_dim": 256,
+    "qk_rope_head_dim": 0,
+    "rms_norm_eps": 1e-05,
+    "routed_scaling_factor": 2.5,
+    "router_aux_loss_coef": 0.001,
+    "scoring_func": "sigmoid",
+    "swiglu_limit": 10.0,
+    "tie_word_embeddings": false,
+    "topk_group": 1,
+    "topk_method": "noaux_tc",
+    "use_cache": true,
+    "v_head_dim": 256,
+    "vocab_size": 154880
+  },
+  "vision_config": {
+    "attention_bias": true,
+    "attention_dropout": 0.0,
+    "depth": 24,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "image_size": 448,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "model_type": "glm5_next_vision",
+    "num_heads": 16,
+    "out_hidden_size": 4096,
+    "patch_size": 14,
+    "projection_intermediate_size": 10240,
+    "rms_norm_eps": 1e-05,
+    "spatial_merge_size": 2,
+    "swiglu_limit": 10.0,
+    "temporal_patch_size": 2
+  },
+  "image_token_id": 154854,
+  "video_token_id": 154855,
+  "image_start_token_id": 154830,
+  "image_end_token_id": 154831,
+  "video_start_token_id": 154832,
+  "video_end_token_id": 154833,
+  "tie_word_embeddings": false,
+  "model_type": "glm5_next",
+  "transformers_version": "5.16.0",
+  "quantization_config": {
+    "activation_scheme": "dynamic",
+    "fmt": "e4m3",
+    "modules_to_not_convert": [
+      "attn_mha",
+      "attn_mqa",
+      "dt_bias",
+      "hyper_connection",
+      "lm_head",
+      "mapping_proj",
+      "model.embed_tokens",
+      "model.layers.0.hc_attn_base",
+      "model.layers.0.hc_attn_fn",
+      "model.layers.0.hc_attn_scale",
+      "model.layers.0.hc_ffn_base",
+      "model.layers.0.hc_ffn_fn",
+      "model.layers.0.hc_ffn_scale",
+      "model.layers.0.input_layernorm",
+      "model.layers.0.post_attention_layernorm",
+      "model.layers.0.self_attn.A_log",
+      "model.layers.0.self_attn.b_proj",
+      "model.layers.0.self_attn.dt_bias",
+      "model.layers.0.self_attn.f_a_proj",
+      "model.layers.0.self_attn.f_b_proj",
+      "model.layers.0.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.0.self_attn.g_a_proj",
+      "model.layers.0.self_attn.g_b_proj",
+      "model.layers.0.self_attn.k_conv1d",
+      "model.layers.0.self_attn.k_proj",
+      "model.layers.0.self_attn.o_norm",
+      "model.layers.0.self_attn.o_proj",
+      "model.layers.0.self_attn.q_conv1d",
+      "model.layers.0.self_attn.q_proj",
+      "model.layers.0.self_attn.qkv_proj",
+      "model.layers.0.self_attn.v_conv1d",
+      "model.layers.0.self_attn.v_proj",
+      "model.layers.1.hc_attn_base",
+      "model.layers.1.hc_attn_fn",
+      "model.layers.1.hc_attn_scale",
+      "model.layers.1.hc_ffn_base",
+      "model.layers.1.hc_ffn_fn",
+      "model.layers.1.hc_ffn_scale",
+      "model.layers.1.input_layernorm",
+      "model.layers.1.post_attention_layernorm",
+      "model.layers.1.self_attn.A_log",
+      "model.layers.1.self_attn.b_proj",
+      "model.layers.1.self_attn.dt_bias",
+      "model.layers.1.self_attn.f_a_proj",
+      "model.layers.1.self_attn.f_b_proj",
+      "model.layers.1.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.1.self_attn.g_a_proj",
+      "model.layers.1.self_attn.g_b_proj",
+      "model.layers.1.self_attn.k_conv1d",
+      "model.layers.1.self_attn.k_proj",
+      "model.layers.1.self_attn.o_norm",
+      "model.layers.1.self_attn.o_proj",
+      "model.layers.1.self_attn.q_conv1d",
+      "model.layers.1.self_attn.q_proj",
+      "model.layers.1.self_attn.qkv_proj",
+      "model.layers.1.self_attn.v_conv1d",
+      "model.layers.1.self_attn.v_proj",
+      "model.layers.10.hc_attn_base",
+      "model.layers.10.hc_attn_fn",
+      "model.layers.10.hc_attn_scale",
+      "model.layers.10.hc_ffn_base",
+      "model.layers.10.hc_ffn_fn",
+      "model.layers.10.hc_ffn_scale",
+      "model.layers.10.input_layernorm",
+      "model.layers.10.mlp.gate",
+      "model.layers.10.mlp.gate.e_score_correction_bias",
+      "model.layers.10.post_attention_layernorm",
+      "model.layers.10.self_attn.A_log",
+      "model.layers.10.self_attn.b_proj",
+      "model.layers.10.self_attn.dt_bias",
+      "model.layers.10.self_attn.f_a_proj",
+      "model.layers.10.self_attn.f_b_proj",
+      "model.layers.10.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.10.self_attn.g_a_proj",
+      "model.layers.10.self_attn.g_b_proj",
+      "model.layers.10.self_attn.k_conv1d",
+      "model.layers.10.self_attn.k_proj",
+      "model.layers.10.self_attn.o_norm",
+      "model.layers.10.self_attn.o_proj",
+      "model.layers.10.self_attn.q_conv1d",
+      "model.layers.10.self_attn.q_proj",
+      "model.layers.10.self_attn.qkv_proj",
+      "model.layers.10.self_attn.v_conv1d",
+      "model.layers.10.self_attn.v_proj",
+      "model.layers.11.hc_attn_base",
+      "model.layers.11.hc_attn_fn",
+      "model.layers.11.hc_attn_scale",
+      "model.layers.11.hc_ffn_base",
+      "model.layers.11.hc_ffn_fn",
+      "model.layers.11.hc_ffn_scale",
+      "model.layers.11.input_layernorm",
+      "model.layers.11.mlp.gate",
+      "model.layers.11.mlp.gate.e_score_correction_bias",
+      "model.layers.11.post_attention_layernorm",
+      "model.layers.11.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.11.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.11.self_attn.indexer.k_norm",
+      "model.layers.11.self_attn.indexer.k_norm.bias",
+      "model.layers.11.self_attn.indexer.weights_proj",
+      "model.layers.11.self_attn.indexer.wk",
+      "model.layers.11.self_attn.indexer.wq_b",
+      "model.layers.11.self_attn.kv_a_layernorm",
+      "model.layers.11.self_attn.kv_b_proj",
+      "model.layers.11.self_attn.q_a_layernorm",
+      "model.layers.12.hc_attn_base",
+      "model.layers.12.hc_attn_fn",
+      "model.layers.12.hc_attn_scale",
+      "model.layers.12.hc_ffn_base",
+      "model.layers.12.hc_ffn_fn",
+      "model.layers.12.hc_ffn_scale",
+      "model.layers.12.input_layernorm",
+      "model.layers.12.mlp.gate",
+      "model.layers.12.mlp.gate.e_score_correction_bias",
+      "model.layers.12.post_attention_layernorm",
+      "model.layers.12.self_attn.A_log",
+      "model.layers.12.self_attn.b_proj",
+      "model.layers.12.self_attn.dt_bias",
+      "model.layers.12.self_attn.f_a_proj",
+      "model.layers.12.self_attn.f_b_proj",
+      "model.layers.12.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.12.self_attn.g_a_proj",
+      "model.layers.12.self_attn.g_b_proj",
+      "model.layers.12.self_attn.k_conv1d",
+      "model.layers.12.self_attn.k_proj",
+      "model.layers.12.self_attn.o_norm",
+      "model.layers.12.self_attn.o_proj",
+      "model.layers.12.self_attn.q_conv1d",
+      "model.layers.12.self_attn.q_proj",
+      "model.layers.12.self_attn.qkv_proj",
+      "model.layers.12.self_attn.v_conv1d",
+      "model.layers.12.self_attn.v_proj",
+      "model.layers.13.hc_attn_base",
+      "model.layers.13.hc_attn_fn",
+      "model.layers.13.hc_attn_scale",
+      "model.layers.13.hc_ffn_base",
+      "model.layers.13.hc_ffn_fn",
+      "model.layers.13.hc_ffn_scale",
+      "model.layers.13.input_layernorm",
+      "model.layers.13.mlp.gate",
+      "model.layers.13.mlp.gate.e_score_correction_bias",
+      "model.layers.13.post_attention_layernorm",
+      "model.layers.13.self_attn.A_log",
+      "model.layers.13.self_attn.b_proj",
+      "model.layers.13.self_attn.dt_bias",
+      "model.layers.13.self_attn.f_a_proj",
+      "model.layers.13.self_attn.f_b_proj",
+      "model.layers.13.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.13.self_attn.g_a_proj",
+      "model.layers.13.self_attn.g_b_proj",
+      "model.layers.13.self_attn.k_conv1d",
+      "model.layers.13.self_attn.k_proj",
+      "model.layers.13.self_attn.o_norm",
+      "model.layers.13.self_attn.o_proj",
+      "model.layers.13.self_attn.q_conv1d",
+      "model.layers.13.self_attn.q_proj",
+      "model.layers.13.self_attn.qkv_proj",
+      "model.layers.13.self_attn.v_conv1d",
+      "model.layers.13.self_attn.v_proj",
+      "model.layers.14.hc_attn_base",
+      "model.layers.14.hc_attn_fn",
+      "model.layers.14.hc_attn_scale",
+      "model.layers.14.hc_ffn_base",
+      "model.layers.14.hc_ffn_fn",
+      "model.layers.14.hc_ffn_scale",
+      "model.layers.14.input_layernorm",
+      "model.layers.14.mlp.gate",
+      "model.layers.14.mlp.gate.e_score_correction_bias",
+      "model.layers.14.post_attention_layernorm",
+      "model.layers.14.self_attn.A_log",
+      "model.layers.14.self_attn.b_proj",
+      "model.layers.14.self_attn.dt_bias",
+      "model.layers.14.self_attn.f_a_proj",
+      "model.layers.14.self_attn.f_b_proj",
+      "model.layers.14.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.14.self_attn.g_a_proj",
+      "model.layers.14.self_attn.g_b_proj",
+      "model.layers.14.self_attn.k_conv1d",
+      "model.layers.14.self_attn.k_proj",
+      "model.layers.14.self_attn.o_norm",
+      "model.layers.14.self_attn.o_proj",
+      "model.layers.14.self_attn.q_conv1d",
+      "model.layers.14.self_attn.q_proj",
+      "model.layers.14.self_attn.qkv_proj",
+      "model.layers.14.self_attn.v_conv1d",
+      "model.layers.14.self_attn.v_proj",
+      "model.layers.15.hc_attn_base",
+      "model.layers.15.hc_attn_fn",
+      "model.layers.15.hc_attn_scale",
+      "model.layers.15.hc_ffn_base",
+      "model.layers.15.hc_ffn_fn",
+      "model.layers.15.hc_ffn_scale",
+      "model.layers.15.input_layernorm",
+      "model.layers.15.mlp.gate",
+      "model.layers.15.mlp.gate.e_score_correction_bias",
+      "model.layers.15.post_attention_layernorm",
+      "model.layers.15.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.15.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.15.self_attn.indexer.k_norm",
+      "model.layers.15.self_attn.indexer.k_norm.bias",
+      "model.layers.15.self_attn.indexer.weights_proj",
+      "model.layers.15.self_attn.indexer.wk",
+      "model.layers.15.self_attn.indexer.wq_b",
+      "model.layers.15.self_attn.kv_a_layernorm",
+      "model.layers.15.self_attn.kv_b_proj",
+      "model.layers.15.self_attn.q_a_layernorm",
+      "model.layers.16.hc_attn_base",
+      "model.layers.16.hc_attn_fn",
+      "model.layers.16.hc_attn_scale",
+      "model.layers.16.hc_ffn_base",
+      "model.layers.16.hc_ffn_fn",
+      "model.layers.16.hc_ffn_scale",
+      "model.layers.16.input_layernorm",
+      "model.layers.16.mlp.gate",
+      "model.layers.16.mlp.gate.e_score_correction_bias",
+      "model.layers.16.post_attention_layernorm",
+      "model.layers.16.self_attn.A_log",
+      "model.layers.16.self_attn.b_proj",
+      "model.layers.16.self_attn.dt_bias",
+      "model.layers.16.self_attn.f_a_proj",
+      "model.layers.16.self_attn.f_b_proj",
+      "model.layers.16.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.16.self_attn.g_a_proj",
+      "model.layers.16.self_attn.g_b_proj",
+      "model.layers.16.self_attn.k_conv1d",
+      "model.layers.16.self_attn.k_proj",
+      "model.layers.16.self_attn.o_norm",
+      "model.layers.16.self_attn.o_proj",
+      "model.layers.16.self_attn.q_conv1d",
+      "model.layers.16.self_attn.q_proj",
+      "model.layers.16.self_attn.qkv_proj",
+      "model.layers.16.self_attn.v_conv1d",
+      "model.layers.16.self_attn.v_proj",
+      "model.layers.17.hc_attn_base",
+      "model.layers.17.hc_attn_fn",
+      "model.layers.17.hc_attn_scale",
+      "model.layers.17.hc_ffn_base",
+      "model.layers.17.hc_ffn_fn",
+      "model.layers.17.hc_ffn_scale",
+      "model.layers.17.input_layernorm",
+      "model.layers.17.mlp.gate",
+      "model.layers.17.mlp.gate.e_score_correction_bias",
+      "model.layers.17.post_attention_layernorm",
+      "model.layers.17.self_attn.A_log",
+      "model.layers.17.self_attn.b_proj",
+      "model.layers.17.self_attn.dt_bias",
+      "model.layers.17.self_attn.f_a_proj",
+      "model.layers.17.self_attn.f_b_proj",
+      "model.layers.17.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.17.self_attn.g_a_proj",
+      "model.layers.17.self_attn.g_b_proj",
+      "model.layers.17.self_attn.k_conv1d",
+      "model.layers.17.self_attn.k_proj",
+      "model.layers.17.self_attn.o_norm",
+      "model.layers.17.self_attn.o_proj",
+      "model.layers.17.self_attn.q_conv1d",
+      "model.layers.17.self_attn.q_proj",
+      "model.layers.17.self_attn.qkv_proj",
+      "model.layers.17.self_attn.v_conv1d",
+      "model.layers.17.self_attn.v_proj",
+      "model.layers.18.hc_attn_base",
+      "model.layers.18.hc_attn_fn",
+      "model.layers.18.hc_attn_scale",
+      "model.layers.18.hc_ffn_base",
+      "model.layers.18.hc_ffn_fn",
+      "model.layers.18.hc_ffn_scale",
+      "model.layers.18.input_layernorm",
+      "model.layers.18.mlp.gate",
+      "model.layers.18.mlp.gate.e_score_correction_bias",
+      "model.layers.18.post_attention_layernorm",
+      "model.layers.18.self_attn.A_log",
+      "model.layers.18.self_attn.b_proj",
+      "model.layers.18.self_attn.dt_bias",
+      "model.layers.18.self_attn.f_a_proj",
+      "model.layers.18.self_attn.f_b_proj",
+      "model.layers.18.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.18.self_attn.g_a_proj",
+      "model.layers.18.self_attn.g_b_proj",
+      "model.layers.18.self_attn.k_conv1d",
+      "model.layers.18.self_attn.k_proj",
+      "model.layers.18.self_attn.o_norm",
+      "model.layers.18.self_attn.o_proj",
+      "model.layers.18.self_attn.q_conv1d",
+      "model.layers.18.self_attn.q_proj",
+      "model.layers.18.self_attn.qkv_proj",
+      "model.layers.18.self_attn.v_conv1d",
+      "model.layers.18.self_attn.v_proj",
+      "model.layers.19.hc_attn_base",
+      "model.layers.19.hc_attn_fn",
+      "model.layers.19.hc_attn_scale",
+      "model.layers.19.hc_ffn_base",
+      "model.layers.19.hc_ffn_fn",
+      "model.layers.19.hc_ffn_scale",
+      "model.layers.19.input_layernorm",
+      "model.layers.19.mlp.gate",
+      "model.layers.19.mlp.gate.e_score_correction_bias",
+      "model.layers.19.post_attention_layernorm",
+      "model.layers.19.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.19.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.19.self_attn.indexer.k_norm",
+      "model.layers.19.self_attn.indexer.k_norm.bias",
+      "model.layers.19.self_attn.indexer.weights_proj",
+      "model.layers.19.self_attn.indexer.wk",
+      "model.layers.19.self_attn.indexer.wq_b",
+      "model.layers.19.self_attn.kv_a_layernorm",
+      "model.layers.19.self_attn.kv_b_proj",
+      "model.layers.19.self_attn.q_a_layernorm",
+      "model.layers.2.hc_attn_base",
+      "model.layers.2.hc_attn_fn",
+      "model.layers.2.hc_attn_scale",
+      "model.layers.2.hc_ffn_base",
+      "model.layers.2.hc_ffn_fn",
+      "model.layers.2.hc_ffn_scale",
+      "model.layers.2.input_layernorm",
+      "model.layers.2.post_attention_layernorm",
+      "model.layers.2.self_attn.A_log",
+      "model.layers.2.self_attn.b_proj",
+      "model.layers.2.self_attn.dt_bias",
+      "model.layers.2.self_attn.f_a_proj",
+      "model.layers.2.self_attn.f_b_proj",
+      "model.layers.2.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.2.self_attn.g_a_proj",
+      "model.layers.2.self_attn.g_b_proj",
+      "model.layers.2.self_attn.k_conv1d",
+      "model.layers.2.self_attn.k_proj",
+      "model.layers.2.self_attn.o_norm",
+      "model.layers.2.self_attn.o_proj",
+      "model.layers.2.self_attn.q_conv1d",
+      "model.layers.2.self_attn.q_proj",
+      "model.layers.2.self_attn.qkv_proj",
+      "model.layers.2.self_attn.v_conv1d",
+      "model.layers.2.self_attn.v_proj",
+      "model.layers.20.hc_attn_base",
+      "model.layers.20.hc_attn_fn",
+      "model.layers.20.hc_attn_scale",
+      "model.layers.20.hc_ffn_base",
+      "model.layers.20.hc_ffn_fn",
+      "model.layers.20.hc_ffn_scale",
+      "model.layers.20.input_layernorm",
+      "model.layers.20.mlp.gate",
+      "model.layers.20.mlp.gate.e_score_correction_bias",
+      "model.layers.20.post_attention_layernorm",
+      "model.layers.20.self_attn.A_log",
+      "model.layers.20.self_attn.b_proj",
+      "model.layers.20.self_attn.dt_bias",
+      "model.layers.20.self_attn.f_a_proj",
+      "model.layers.20.self_attn.f_b_proj",
+      "model.layers.20.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.20.self_attn.g_a_proj",
+      "model.layers.20.self_attn.g_b_proj",
+      "model.layers.20.self_attn.k_conv1d",
+      "model.layers.20.self_attn.k_proj",
+      "model.layers.20.self_attn.o_norm",
+      "model.layers.20.self_attn.o_proj",
+      "model.layers.20.self_attn.q_conv1d",
+      "model.layers.20.self_attn.q_proj",
+      "model.layers.20.self_attn.qkv_proj",
+      "model.layers.20.self_attn.v_conv1d",
+      "model.layers.20.self_attn.v_proj",
+      "model.layers.21.hc_attn_base",
+      "model.layers.21.hc_attn_fn",
+      "model.layers.21.hc_attn_scale",
+      "model.layers.21.hc_ffn_base",
+      "model.layers.21.hc_ffn_fn",
+      "model.layers.21.hc_ffn_scale",
+      "model.layers.21.input_layernorm",
+      "model.layers.21.mlp.gate",
+      "model.layers.21.mlp.gate.e_score_correction_bias",
+      "model.layers.21.post_attention_layernorm",
+      "model.layers.21.self_attn.A_log",
+      "model.layers.21.self_attn.b_proj",
+      "model.layers.21.self_attn.dt_bias",
+      "model.layers.21.self_attn.f_a_proj",
+      "model.layers.21.self_attn.f_b_proj",
+      "model.layers.21.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.21.self_attn.g_a_proj",
+      "model.layers.21.self_attn.g_b_proj",
+      "model.layers.21.self_attn.k_conv1d",
+      "model.layers.21.self_attn.k_proj",
+      "model.layers.21.self_attn.o_norm",
+      "model.layers.21.self_attn.o_proj",
+      "model.layers.21.self_attn.q_conv1d",
+      "model.layers.21.self_attn.q_proj",
+      "model.layers.21.self_attn.qkv_proj",
+      "model.layers.21.self_attn.v_conv1d",
+      "model.layers.21.self_attn.v_proj",
+      "model.layers.22.hc_attn_base",
+      "model.layers.22.hc_attn_fn",
+      "model.layers.22.hc_attn_scale",
+      "model.layers.22.hc_ffn_base",
+      "model.layers.22.hc_ffn_fn",
+      "model.layers.22.hc_ffn_scale",
+      "model.layers.22.input_layernorm",
+      "model.layers.22.mlp.gate",
+      "model.layers.22.mlp.gate.e_score_correction_bias",
+      "model.layers.22.post_attention_layernorm",
+      "model.layers.22.self_attn.A_log",
+      "model.layers.22.self_attn.b_proj",
+      "model.layers.22.self_attn.dt_bias",
+      "model.layers.22.self_attn.f_a_proj",
+      "model.layers.22.self_attn.f_b_proj",
+      "model.layers.22.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.22.self_attn.g_a_proj",
+      "model.layers.22.self_attn.g_b_proj",
+      "model.layers.22.self_attn.k_conv1d",
+      "model.layers.22.self_attn.k_proj",
+      "model.layers.22.self_attn.o_norm",
+      "model.layers.22.self_attn.o_proj",
+      "model.layers.22.self_attn.q_conv1d",
+      "model.layers.22.self_attn.q_proj",
+      "model.layers.22.self_attn.qkv_proj",
+      "model.layers.22.self_attn.v_conv1d",
+      "model.layers.22.self_attn.v_proj",
+      "model.layers.23.hc_attn_base",
+      "model.layers.23.hc_attn_fn",
+      "model.layers.23.hc_attn_scale",
+      "model.layers.23.hc_ffn_base",
+      "model.layers.23.hc_ffn_fn",
+      "model.layers.23.hc_ffn_scale",
+      "model.layers.23.input_layernorm",
+      "model.layers.23.mlp.gate",
+      "model.layers.23.mlp.gate.e_score_correction_bias",
+      "model.layers.23.post_attention_layernorm",
+      "model.layers.23.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.23.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.23.self_attn.indexer.k_norm",
+      "model.layers.23.self_attn.indexer.k_norm.bias",
+      "model.layers.23.self_attn.indexer.weights_proj",
+      "model.layers.23.self_attn.indexer.wk",
+      "model.layers.23.self_attn.indexer.wq_b",
+      "model.layers.23.self_attn.kv_a_layernorm",
+      "model.layers.23.self_attn.kv_b_proj",
+      "model.layers.23.self_attn.q_a_layernorm",
+      "model.layers.24.hc_attn_base",
+      "model.layers.24.hc_attn_fn",
+      "model.layers.24.hc_attn_scale",
+      "model.layers.24.hc_ffn_base",
+      "model.layers.24.hc_ffn_fn",
+      "model.layers.24.hc_ffn_scale",
+      "model.layers.24.input_layernorm",
+      "model.layers.24.mlp.gate",
+      "model.layers.24.mlp.gate.e_score_correction_bias",
+      "model.layers.24.post_attention_layernorm",
+      "model.layers.24.self_attn.A_log",
+      "model.layers.24.self_attn.b_proj",
+      "model.layers.24.self_attn.dt_bias",
+      "model.layers.24.self_attn.f_a_proj",
+      "model.layers.24.self_attn.f_b_proj",
+      "model.layers.24.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.24.self_attn.g_a_proj",
+      "model.layers.24.self_attn.g_b_proj",
+      "model.layers.24.self_attn.k_conv1d",
+      "model.layers.24.self_attn.k_proj",
+      "model.layers.24.self_attn.o_norm",
+      "model.layers.24.self_attn.o_proj",
+      "model.layers.24.self_attn.q_conv1d",
+      "model.layers.24.self_attn.q_proj",
+      "model.layers.24.self_attn.qkv_proj",
+      "model.layers.24.self_attn.v_conv1d",
+      "model.layers.24.self_attn.v_proj",
+      "model.layers.25.hc_attn_base",
+      "model.layers.25.hc_attn_fn",
+      "model.layers.25.hc_attn_scale",
+      "model.layers.25.hc_ffn_base",
+      "model.layers.25.hc_ffn_fn",
+      "model.layers.25.hc_ffn_scale",
+      "model.layers.25.input_layernorm",
+      "model.layers.25.mlp.gate",
+      "model.layers.25.mlp.gate.e_score_correction_bias",
+      "model.layers.25.post_attention_layernorm",
+      "model.layers.25.self_attn.A_log",
+      "model.layers.25.self_attn.b_proj",
+      "model.layers.25.self_attn.dt_bias",
+      "model.layers.25.self_attn.f_a_proj",
+      "model.layers.25.self_attn.f_b_proj",
+      "model.layers.25.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.25.self_attn.g_a_proj",
+      "model.layers.25.self_attn.g_b_proj",
+      "model.layers.25.self_attn.k_conv1d",
+      "model.layers.25.self_attn.k_proj",
+      "model.layers.25.self_attn.o_norm",
+      "model.layers.25.self_attn.o_proj",
+      "model.layers.25.self_attn.q_conv1d",
+      "model.layers.25.self_attn.q_proj",
+      "model.layers.25.self_attn.qkv_proj",
+      "model.layers.25.self_attn.v_conv1d",
+      "model.layers.25.self_attn.v_proj",
+      "model.layers.26.hc_attn_base",
+      "model.layers.26.hc_attn_fn",
+      "model.layers.26.hc_attn_scale",
+      "model.layers.26.hc_ffn_base",
+      "model.layers.26.hc_ffn_fn",
+      "model.layers.26.hc_ffn_scale",
+      "model.layers.26.input_layernorm",
+      "model.layers.26.mlp.gate",
+      "model.layers.26.mlp.gate.e_score_correction_bias",
+      "model.layers.26.post_attention_layernorm",
+      "model.layers.26.self_attn.A_log",
+      "model.layers.26.self_attn.b_proj",
+      "model.layers.26.self_attn.dt_bias",
+      "model.layers.26.self_attn.f_a_proj",
+      "model.layers.26.self_attn.f_b_proj",
+      "model.layers.26.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.26.self_attn.g_a_proj",
+      "model.layers.26.self_attn.g_b_proj",
+      "model.layers.26.self_attn.k_conv1d",
+      "model.layers.26.self_attn.k_proj",
+      "model.layers.26.self_attn.o_norm",
+      "model.layers.26.self_attn.o_proj",
+      "model.layers.26.self_attn.q_conv1d",
+      "model.layers.26.self_attn.q_proj",
+      "model.layers.26.self_attn.qkv_proj",
+      "model.layers.26.self_attn.v_conv1d",
+      "model.layers.26.self_attn.v_proj",
+      "model.layers.27.hc_attn_base",
+      "model.layers.27.hc_attn_fn",
+      "model.layers.27.hc_attn_scale",
+      "model.layers.27.hc_ffn_base",
+      "model.layers.27.hc_ffn_fn",
+      "model.layers.27.hc_ffn_scale",
+      "model.layers.27.input_layernorm",
+      "model.layers.27.mlp.gate",
+      "model.layers.27.mlp.gate.e_score_correction_bias",
+      "model.layers.27.post_attention_layernorm",
+      "model.layers.27.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.27.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.27.self_attn.indexer.k_norm",
+      "model.layers.27.self_attn.indexer.k_norm.bias",
+      "model.layers.27.self_attn.indexer.weights_proj",
+      "model.layers.27.self_attn.indexer.wk",
+      "model.layers.27.self_attn.indexer.wq_b",
+      "model.layers.27.self_attn.kv_a_layernorm",
+      "model.layers.27.self_attn.kv_b_proj",
+      "model.layers.27.self_attn.q_a_layernorm",
+      "model.layers.28.hc_attn_base",
+      "model.layers.28.hc_attn_fn",
+      "model.layers.28.hc_attn_scale",
+      "model.layers.28.hc_ffn_base",
+      "model.layers.28.hc_ffn_fn",
+      "model.layers.28.hc_ffn_scale",
+      "model.layers.28.input_layernorm",
+      "model.layers.28.mlp.gate",
+      "model.layers.28.mlp.gate.e_score_correction_bias",
+      "model.layers.28.post_attention_layernorm",
+      "model.layers.28.self_attn.A_log",
+      "model.layers.28.self_attn.b_proj",
+      "model.layers.28.self_attn.dt_bias",
+      "model.layers.28.self_attn.f_a_proj",
+      "model.layers.28.self_attn.f_b_proj",
+      "model.layers.28.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.28.self_attn.g_a_proj",
+      "model.layers.28.self_attn.g_b_proj",
+      "model.layers.28.self_attn.k_conv1d",
+      "model.layers.28.self_attn.k_proj",
+      "model.layers.28.self_attn.o_norm",
+      "model.layers.28.self_attn.o_proj",
+      "model.layers.28.self_attn.q_conv1d",
+      "model.layers.28.self_attn.q_proj",
+      "model.layers.28.self_attn.qkv_proj",
+      "model.layers.28.self_attn.v_conv1d",
+      "model.layers.28.self_attn.v_proj",
+      "model.layers.29.hc_attn_base",
+      "model.layers.29.hc_attn_fn",
+      "model.layers.29.hc_attn_scale",
+      "model.layers.29.hc_ffn_base",
+      "model.layers.29.hc_ffn_fn",
+      "model.layers.29.hc_ffn_scale",
+      "model.layers.29.input_layernorm",
+      "model.layers.29.mlp.gate",
+      "model.layers.29.mlp.gate.e_score_correction_bias",
+      "model.layers.29.post_attention_layernorm",
+      "model.layers.29.self_attn.A_log",
+      "model.layers.29.self_attn.b_proj",
+      "model.layers.29.self_attn.dt_bias",
+      "model.layers.29.self_attn.f_a_proj",
+      "model.layers.29.self_attn.f_b_proj",
+      "model.layers.29.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.29.self_attn.g_a_proj",
+      "model.layers.29.self_attn.g_b_proj",
+      "model.layers.29.self_attn.k_conv1d",
+      "model.layers.29.self_attn.k_proj",
+      "model.layers.29.self_attn.o_norm",
+      "model.layers.29.self_attn.o_proj",
+      "model.layers.29.self_attn.q_conv1d",
+      "model.layers.29.self_attn.q_proj",
+      "model.layers.29.self_attn.qkv_proj",
+      "model.layers.29.self_attn.v_conv1d",
+      "model.layers.29.self_attn.v_proj",
+      "model.layers.3.hc_attn_base",
+      "model.layers.3.hc_attn_fn",
+      "model.layers.3.hc_attn_scale",
+      "model.layers.3.hc_ffn_base",
+      "model.layers.3.hc_ffn_fn",
+      "model.layers.3.hc_ffn_scale",
+      "model.layers.3.input_layernorm",
+      "model.layers.3.mlp.gate",
+      "model.layers.3.mlp.gate.e_score_correction_bias",
+      "model.layers.3.post_attention_layernorm",
+      "model.layers.3.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.3.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.3.self_attn.indexer.k_norm",
+      "model.layers.3.self_attn.indexer.k_norm.bias",
+      "model.layers.3.self_attn.indexer.weights_proj",
+      "model.layers.3.self_attn.indexer.wk",
+      "model.layers.3.self_attn.indexer.wq_b",
+      "model.layers.3.self_attn.kv_a_layernorm",
+      "model.layers.3.self_attn.kv_b_proj",
+      "model.layers.3.self_attn.q_a_layernorm",
+      "model.layers.30.hc_attn_base",
+      "model.layers.30.hc_attn_fn",
+      "model.layers.30.hc_attn_scale",
+      "model.layers.30.hc_ffn_base",
+      "model.layers.30.hc_ffn_fn",
+      "model.layers.30.hc_ffn_scale",
+      "model.layers.30.input_layernorm",
+      "model.layers.30.mlp.gate",
+      "model.layers.30.mlp.gate.e_score_correction_bias",
+      "model.layers.30.post_attention_layernorm",
+      "model.layers.30.self_attn.A_log",
+      "model.layers.30.self_attn.b_proj",
+      "model.layers.30.self_attn.dt_bias",
+      "model.layers.30.self_attn.f_a_proj",
+      "model.layers.30.self_attn.f_b_proj",
+      "model.layers.30.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.30.self_attn.g_a_proj",
+      "model.layers.30.self_attn.g_b_proj",
+      "model.layers.30.self_attn.k_conv1d",
+      "model.layers.30.self_attn.k_proj",
+      "model.layers.30.self_attn.o_norm",
+      "model.layers.30.self_attn.o_proj",
+      "model.layers.30.self_attn.q_conv1d",
+      "model.layers.30.self_attn.q_proj",
+      "model.layers.30.self_attn.qkv_proj",
+      "model.layers.30.self_attn.v_conv1d",
+      "model.layers.30.self_attn.v_proj",
+      "model.layers.31.hc_attn_base",
+      "model.layers.31.hc_attn_fn",
+      "model.layers.31.hc_attn_scale",
+      "model.layers.31.hc_ffn_base",
+      "model.layers.31.hc_ffn_fn",
+      "model.layers.31.hc_ffn_scale",
+      "model.layers.31.input_layernorm",
+      "model.layers.31.mlp.gate",
+      "model.layers.31.mlp.gate.e_score_correction_bias",
+      "model.layers.31.post_attention_layernorm",
+      "model.layers.31.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.31.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.31.self_attn.indexer.k_norm",
+      "model.layers.31.self_attn.indexer.k_norm.bias",
+      "model.layers.31.self_attn.indexer.weights_proj",
+      "model.layers.31.self_attn.indexer.wk",
+      "model.layers.31.self_attn.indexer.wq_b",
+      "model.layers.31.self_attn.kv_a_layernorm",
+      "model.layers.31.self_attn.kv_b_proj",
+      "model.layers.31.self_attn.q_a_layernorm",
+      "model.layers.32.hc_attn_base",
+      "model.layers.32.hc_attn_fn",
+      "model.layers.32.hc_attn_scale",
+      "model.layers.32.hc_ffn_base",
+      "model.layers.32.hc_ffn_fn",
+      "model.layers.32.hc_ffn_scale",
+      "model.layers.32.input_layernorm",
+      "model.layers.32.mlp.gate",
+      "model.layers.32.mlp.gate.e_score_correction_bias",
+      "model.layers.32.post_attention_layernorm",
+      "model.layers.32.self_attn.A_log",
+      "model.layers.32.self_attn.b_proj",
+      "model.layers.32.self_attn.dt_bias",
+      "model.layers.32.self_attn.f_a_proj",
+      "model.layers.32.self_attn.f_b_proj",
+      "model.layers.32.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.32.self_attn.g_a_proj",
+      "model.layers.32.self_attn.g_b_proj",
+      "model.layers.32.self_attn.k_conv1d",
+      "model.layers.32.self_attn.k_proj",
+      "model.layers.32.self_attn.o_norm",
+      "model.layers.32.self_attn.o_proj",
+      "model.layers.32.self_attn.q_conv1d",
+      "model.layers.32.self_attn.q_proj",
+      "model.layers.32.self_attn.qkv_proj",
+      "model.layers.32.self_attn.v_conv1d",
+      "model.layers.32.self_attn.v_proj",
+      "model.layers.33.hc_attn_base",
+      "model.layers.33.hc_attn_fn",
+      "model.layers.33.hc_attn_scale",
+      "model.layers.33.hc_ffn_base",
+      "model.layers.33.hc_ffn_fn",
+      "model.layers.33.hc_ffn_scale",
+      "model.layers.33.input_layernorm",
+      "model.layers.33.mlp.gate",
+      "model.layers.33.mlp.gate.e_score_correction_bias",
+      "model.layers.33.post_attention_layernorm",
+      "model.layers.33.self_attn.A_log",
+      "model.layers.33.self_attn.b_proj",
+      "model.layers.33.self_attn.dt_bias",
+      "model.layers.33.self_attn.f_a_proj",
+      "model.layers.33.self_attn.f_b_proj",
+      "model.layers.33.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.33.self_attn.g_a_proj",
+      "model.layers.33.self_attn.g_b_proj",
+      "model.layers.33.self_attn.k_conv1d",
+      "model.layers.33.self_attn.k_proj",
+      "model.layers.33.self_attn.o_norm",
+      "model.layers.33.self_attn.o_proj",
+      "model.layers.33.self_attn.q_conv1d",
+      "model.layers.33.self_attn.q_proj",
+      "model.layers.33.self_attn.qkv_proj",
+      "model.layers.33.self_attn.v_conv1d",
+      "model.layers.33.self_attn.v_proj",
+      "model.layers.34.hc_attn_base",
+      "model.layers.34.hc_attn_fn",
+      "model.layers.34.hc_attn_scale",
+      "model.layers.34.hc_ffn_base",
+      "model.layers.34.hc_ffn_fn",
+      "model.layers.34.hc_ffn_scale",
+      "model.layers.34.input_layernorm",
+      "model.layers.34.mlp.gate",
+      "model.layers.34.mlp.gate.e_score_correction_bias",
+      "model.layers.34.post_attention_layernorm",
+      "model.layers.34.self_attn.A_log",
+      "model.layers.34.self_attn.b_proj",
+      "model.layers.34.self_attn.dt_bias",
+      "model.layers.34.self_attn.f_a_proj",
+      "model.layers.34.self_attn.f_b_proj",
+      "model.layers.34.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.34.self_attn.g_a_proj",
+      "model.layers.34.self_attn.g_b_proj",
+      "model.layers.34.self_attn.k_conv1d",
+      "model.layers.34.self_attn.k_proj",
+      "model.layers.34.self_attn.o_norm",
+      "model.layers.34.self_attn.o_proj",
+      "model.layers.34.self_attn.q_conv1d",
+      "model.layers.34.self_attn.q_proj",
+      "model.layers.34.self_attn.qkv_proj",
+      "model.layers.34.self_attn.v_conv1d",
+      "model.layers.34.self_attn.v_proj",
+      "model.layers.35.hc_attn_base",
+      "model.layers.35.hc_attn_fn",
+      "model.layers.35.hc_attn_scale",
+      "model.layers.35.hc_ffn_base",
+      "model.layers.35.hc_ffn_fn",
+      "model.layers.35.hc_ffn_scale",
+      "model.layers.35.input_layernorm",
+      "model.layers.35.mlp.gate",
+      "model.layers.35.mlp.gate.e_score_correction_bias",
+      "model.layers.35.post_attention_layernorm",
+      "model.layers.35.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.35.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.35.self_attn.indexer.k_norm",
+      "model.layers.35.self_attn.indexer.k_norm.bias",
+      "model.layers.35.self_attn.indexer.weights_proj",
+      "model.layers.35.self_attn.indexer.wk",
+      "model.layers.35.self_attn.indexer.wq_b",
+      "model.layers.35.self_attn.kv_a_layernorm",
+      "model.layers.35.self_attn.kv_b_proj",
+      "model.layers.35.self_attn.q_a_layernorm",
+      "model.layers.36.hc_attn_base",
+      "model.layers.36.hc_attn_fn",
+      "model.layers.36.hc_attn_scale",
+      "model.layers.36.hc_ffn_base",
+      "model.layers.36.hc_ffn_fn",
+      "model.layers.36.hc_ffn_scale",
+      "model.layers.36.input_layernorm",
+      "model.layers.36.mlp.gate",
+      "model.layers.36.mlp.gate.e_score_correction_bias",
+      "model.layers.36.post_attention_layernorm",
+      "model.layers.36.self_attn.A_log",
+      "model.layers.36.self_attn.b_proj",
+      "model.layers.36.self_attn.dt_bias",
+      "model.layers.36.self_attn.f_a_proj",
+      "model.layers.36.self_attn.f_b_proj",
+      "model.layers.36.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.36.self_attn.g_a_proj",
+      "model.layers.36.self_attn.g_b_proj",
+      "model.layers.36.self_attn.k_conv1d",
+      "model.layers.36.self_attn.k_proj",
+      "model.layers.36.self_attn.o_norm",
+      "model.layers.36.self_attn.o_proj",
+      "model.layers.36.self_attn.q_conv1d",
+      "model.layers.36.self_attn.q_proj",
+      "model.layers.36.self_attn.qkv_proj",
+      "model.layers.36.self_attn.v_conv1d",
+      "model.layers.36.self_attn.v_proj",
+      "model.layers.37.hc_attn_base",
+      "model.layers.37.hc_attn_fn",
+      "model.layers.37.hc_attn_scale",
+      "model.layers.37.hc_ffn_base",
+      "model.layers.37.hc_ffn_fn",
+      "model.layers.37.hc_ffn_scale",
+      "model.layers.37.input_layernorm",
+      "model.layers.37.mlp.gate",
+      "model.layers.37.mlp.gate.e_score_correction_bias",
+      "model.layers.37.post_attention_layernorm",
+      "model.layers.37.self_attn.A_log",
+      "model.layers.37.self_attn.b_proj",
+      "model.layers.37.self_attn.dt_bias",
+      "model.layers.37.self_attn.f_a_proj",
+      "model.layers.37.self_attn.f_b_proj",
+      "model.layers.37.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.37.self_attn.g_a_proj",
+      "model.layers.37.self_attn.g_b_proj",
+      "model.layers.37.self_attn.k_conv1d",
+      "model.layers.37.self_attn.k_proj",
+      "model.layers.37.self_attn.o_norm",
+      "model.layers.37.self_attn.o_proj",
+      "model.layers.37.self_attn.q_conv1d",
+      "model.layers.37.self_attn.q_proj",
+      "model.layers.37.self_attn.qkv_proj",
+      "model.layers.37.self_attn.v_conv1d",
+      "model.layers.37.self_attn.v_proj",
+      "model.layers.38.hc_attn_base",
+      "model.layers.38.hc_attn_fn",
+      "model.layers.38.hc_attn_scale",
+      "model.layers.38.hc_ffn_base",
+      "model.layers.38.hc_ffn_fn",
+      "model.layers.38.hc_ffn_scale",
+      "model.layers.38.input_layernorm",
+      "model.layers.38.mlp.gate",
+      "model.layers.38.mlp.gate.e_score_correction_bias",
+      "model.layers.38.post_attention_layernorm",
+      "model.layers.38.self_attn.A_log",
+      "model.layers.38.self_attn.b_proj",
+      "model.layers.38.self_attn.dt_bias",
+      "model.layers.38.self_attn.f_a_proj",
+      "model.layers.38.self_attn.f_b_proj",
+      "model.layers.38.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.38.self_attn.g_a_proj",
+      "model.layers.38.self_attn.g_b_proj",
+      "model.layers.38.self_attn.k_conv1d",
+      "model.layers.38.self_attn.k_proj",
+      "model.layers.38.self_attn.o_norm",
+      "model.layers.38.self_attn.o_proj",
+      "model.layers.38.self_attn.q_conv1d",
+      "model.layers.38.self_attn.q_proj",
+      "model.layers.38.self_attn.qkv_proj",
+      "model.layers.38.self_attn.v_conv1d",
+      "model.layers.38.self_attn.v_proj",
+      "model.layers.39.hc_attn_base",
+      "model.layers.39.hc_attn_fn",
+      "model.layers.39.hc_attn_scale",
+      "model.layers.39.hc_ffn_base",
+      "model.layers.39.hc_ffn_fn",
+      "model.layers.39.hc_ffn_scale",
+      "model.layers.39.input_layernorm",
+      "model.layers.39.mlp.gate",
+      "model.layers.39.mlp.gate.e_score_correction_bias",
+      "model.layers.39.post_attention_layernorm",
+      "model.layers.39.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.39.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.39.self_attn.indexer.k_norm",
+      "model.layers.39.self_attn.indexer.k_norm.bias",
+      "model.layers.39.self_attn.indexer.weights_proj",
+      "model.layers.39.self_attn.indexer.wk",
+      "model.layers.39.self_attn.indexer.wq_b",
+      "model.layers.39.self_attn.kv_a_layernorm",
+      "model.layers.39.self_attn.kv_b_proj",
+      "model.layers.39.self_attn.q_a_layernorm",
+      "model.layers.4.hc_attn_base",
+      "model.layers.4.hc_attn_fn",
+      "model.layers.4.hc_attn_scale",
+      "model.layers.4.hc_ffn_base",
+      "model.layers.4.hc_ffn_fn",
+      "model.layers.4.hc_ffn_scale",
+      "model.layers.4.input_layernorm",
+      "model.layers.4.mlp.gate",
+      "model.layers.4.mlp.gate.e_score_correction_bias",
+      "model.layers.4.post_attention_layernorm",
+      "model.layers.4.self_attn.A_log",
+      "model.layers.4.self_attn.b_proj",
+      "model.layers.4.self_attn.dt_bias",
+      "model.layers.4.self_attn.f_a_proj",
+      "model.layers.4.self_attn.f_b_proj",
+      "model.layers.4.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.4.self_attn.g_a_proj",
+      "model.layers.4.self_attn.g_b_proj",
+      "model.layers.4.self_attn.k_conv1d",
+      "model.layers.4.self_attn.k_proj",
+      "model.layers.4.self_attn.o_norm",
+      "model.layers.4.self_attn.o_proj",
+      "model.layers.4.self_attn.q_conv1d",
+      "model.layers.4.self_attn.q_proj",
+      "model.layers.4.self_attn.qkv_proj",
+      "model.layers.4.self_attn.v_conv1d",
+      "model.layers.4.self_attn.v_proj",
+      "model.layers.40.hc_attn_base",
+      "model.layers.40.hc_attn_fn",
+      "model.layers.40.hc_attn_scale",
+      "model.layers.40.hc_ffn_base",
+      "model.layers.40.hc_ffn_fn",
+      "model.layers.40.hc_ffn_scale",
+      "model.layers.40.input_layernorm",
+      "model.layers.40.mlp.gate",
+      "model.layers.40.mlp.gate.e_score_correction_bias",
+      "model.layers.40.post_attention_layernorm",
+      "model.layers.40.self_attn.A_log",
+      "model.layers.40.self_attn.b_proj",
+      "model.layers.40.self_attn.dt_bias",
+      "model.layers.40.self_attn.f_a_proj",
+      "model.layers.40.self_attn.f_b_proj",
+      "model.layers.40.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.40.self_attn.g_a_proj",
+      "model.layers.40.self_attn.g_b_proj",
+      "model.layers.40.self_attn.k_conv1d",
+      "model.layers.40.self_attn.k_proj",
+      "model.layers.40.self_attn.o_norm",
+      "model.layers.40.self_attn.o_proj",
+      "model.layers.40.self_attn.q_conv1d",
+      "model.layers.40.self_attn.q_proj",
+      "model.layers.40.self_attn.qkv_proj",
+      "model.layers.40.self_attn.v_conv1d",
+      "model.layers.40.self_attn.v_proj",
+      "model.layers.41.hc_attn_base",
+      "model.layers.41.hc_attn_fn",
+      "model.layers.41.hc_attn_scale",
+      "model.layers.41.hc_ffn_base",
+      "model.layers.41.hc_ffn_fn",
+      "model.layers.41.hc_ffn_scale",
+      "model.layers.41.input_layernorm",
+      "model.layers.41.mlp.gate",
+      "model.layers.41.mlp.gate.e_score_correction_bias",
+      "model.layers.41.post_attention_layernorm",
+      "model.layers.41.self_attn.A_log",
+      "model.layers.41.self_attn.b_proj",
+      "model.layers.41.self_attn.dt_bias",
+      "model.layers.41.self_attn.f_a_proj",
+      "model.layers.41.self_attn.f_b_proj",
+      "model.layers.41.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.41.self_attn.g_a_proj",
+      "model.layers.41.self_attn.g_b_proj",
+      "model.layers.41.self_attn.k_conv1d",
+      "model.layers.41.self_attn.k_proj",
+      "model.layers.41.self_attn.o_norm",
+      "model.layers.41.self_attn.o_proj",
+      "model.layers.41.self_attn.q_conv1d",
+      "model.layers.41.self_attn.q_proj",
+      "model.layers.41.self_attn.qkv_proj",
+      "model.layers.41.self_attn.v_conv1d",
+      "model.layers.41.self_attn.v_proj",
+      "model.layers.42.hc_attn_base",
+      "model.layers.42.hc_attn_fn",
+      "model.layers.42.hc_attn_scale",
+      "model.layers.42.hc_ffn_base",
+      "model.layers.42.hc_ffn_fn",
+      "model.layers.42.hc_ffn_scale",
+      "model.layers.42.input_layernorm",
+      "model.layers.42.mlp.gate",
+      "model.layers.42.mlp.gate.e_score_correction_bias",
+      "model.layers.42.post_attention_layernorm",
+      "model.layers.42.self_attn.A_log",
+      "model.layers.42.self_attn.b_proj",
+      "model.layers.42.self_attn.dt_bias",
+      "model.layers.42.self_attn.f_a_proj",
+      "model.layers.42.self_attn.f_b_proj",
+      "model.layers.42.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.42.self_attn.g_a_proj",
+      "model.layers.42.self_attn.g_b_proj",
+      "model.layers.42.self_attn.k_conv1d",
+      "model.layers.42.self_attn.k_proj",
+      "model.layers.42.self_attn.o_norm",
+      "model.layers.42.self_attn.o_proj",
+      "model.layers.42.self_attn.q_conv1d",
+      "model.layers.42.self_attn.q_proj",
+      "model.layers.42.self_attn.qkv_proj",
+      "model.layers.42.self_attn.v_conv1d",
+      "model.layers.42.self_attn.v_proj",
+      "model.layers.43.hc_attn_base",
+      "model.layers.43.hc_attn_fn",
+      "model.layers.43.hc_attn_scale",
+      "model.layers.43.hc_ffn_base",
+      "model.layers.43.hc_ffn_fn",
+      "model.layers.43.hc_ffn_scale",
+      "model.layers.43.input_layernorm",
+      "model.layers.43.mlp.gate",
+      "model.layers.43.mlp.gate.e_score_correction_bias",
+      "model.layers.43.post_attention_layernorm",
+      "model.layers.43.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.43.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.43.self_attn.indexer.k_norm",
+      "model.layers.43.self_attn.indexer.k_norm.bias",
+      "model.layers.43.self_attn.indexer.weights_proj",
+      "model.layers.43.self_attn.indexer.wk",
+      "model.layers.43.self_attn.indexer.wq_b",
+      "model.layers.43.self_attn.kv_a_layernorm",
+      "model.layers.43.self_attn.kv_b_proj",
+      "model.layers.43.self_attn.q_a_layernorm",
+      "model.layers.44.hc_attn_base",
+      "model.layers.44.hc_attn_fn",
+      "model.layers.44.hc_attn_scale",
+      "model.layers.44.hc_ffn_base",
+      "model.layers.44.hc_ffn_fn",
+      "model.layers.44.hc_ffn_scale",
+      "model.layers.44.input_layernorm",
+      "model.layers.44.mlp.gate",
+      "model.layers.44.mlp.gate.e_score_correction_bias",
+      "model.layers.44.post_attention_layernorm",
+      "model.layers.44.self_attn.A_log",
+      "model.layers.44.self_attn.b_proj",
+      "model.layers.44.self_attn.dt_bias",
+      "model.layers.44.self_attn.f_a_proj",
+      "model.layers.44.self_attn.f_b_proj",
+      "model.layers.44.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.44.self_attn.g_a_proj",
+      "model.layers.44.self_attn.g_b_proj",
+      "model.layers.44.self_attn.k_conv1d",
+      "model.layers.44.self_attn.k_proj",
+      "model.layers.44.self_attn.o_norm",
+      "model.layers.44.self_attn.o_proj",
+      "model.layers.44.self_attn.q_conv1d",
+      "model.layers.44.self_attn.q_proj",
+      "model.layers.44.self_attn.qkv_proj",
+      "model.layers.44.self_attn.v_conv1d",
+      "model.layers.44.self_attn.v_proj",
+      "model.layers.45.eh_proj",
+      "model.layers.45.enorm",
+      "model.layers.45.hnorm",
+      "model.layers.45.input_layernorm",
+      "model.layers.45.mlp.gate",
+      "model.layers.45.mlp.gate.e_score_correction_bias",
+      "model.layers.45.post_attention_layernorm",
+      "model.layers.45.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.45.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.45.self_attn.indexer.k_norm",
+      "model.layers.45.self_attn.indexer.k_norm.bias",
+      "model.layers.45.self_attn.indexer.weights_proj",
+      "model.layers.45.self_attn.indexer.wk",
+      "model.layers.45.self_attn.indexer.wq_b",
+      "model.layers.45.self_attn.kv_a_layernorm",
+      "model.layers.45.self_attn.kv_b_proj",
+      "model.layers.45.self_attn.q_a_layernorm",
+      "model.layers.45.shared_head.norm",
+      "model.layers.5.hc_attn_base",
+      "model.layers.5.hc_attn_fn",
+      "model.layers.5.hc_attn_scale",
+      "model.layers.5.hc_ffn_base",
+      "model.layers.5.hc_ffn_fn",
+      "model.layers.5.hc_ffn_scale",
+      "model.layers.5.input_layernorm",
+      "model.layers.5.mlp.gate",
+      "model.layers.5.mlp.gate.e_score_correction_bias",
+      "model.layers.5.post_attention_layernorm",
+      "model.layers.5.self_attn.A_log",
+      "model.layers.5.self_attn.b_proj",
+      "model.layers.5.self_attn.dt_bias",
+      "model.layers.5.self_attn.f_a_proj",
+      "model.layers.5.self_attn.f_b_proj",
+      "model.layers.5.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.5.self_attn.g_a_proj",
+      "model.layers.5.self_attn.g_b_proj",
+      "model.layers.5.self_attn.k_conv1d",
+      "model.layers.5.self_attn.k_proj",
+      "model.layers.5.self_attn.o_norm",
+      "model.layers.5.self_attn.o_proj",
+      "model.layers.5.self_attn.q_conv1d",
+      "model.layers.5.self_attn.q_proj",
+      "model.layers.5.self_attn.qkv_proj",
+      "model.layers.5.self_attn.v_conv1d",
+      "model.layers.5.self_attn.v_proj",
+      "model.layers.6.hc_attn_base",
+      "model.layers.6.hc_attn_fn",
+      "model.layers.6.hc_attn_scale",
+      "model.layers.6.hc_ffn_base",
+      "model.layers.6.hc_ffn_fn",
+      "model.layers.6.hc_ffn_scale",
+      "model.layers.6.input_layernorm",
+      "model.layers.6.mlp.gate",
+      "model.layers.6.mlp.gate.e_score_correction_bias",
+      "model.layers.6.post_attention_layernorm",
+      "model.layers.6.self_attn.A_log",
+      "model.layers.6.self_attn.b_proj",
+      "model.layers.6.self_attn.dt_bias",
+      "model.layers.6.self_attn.f_a_proj",
+      "model.layers.6.self_attn.f_b_proj",
+      "model.layers.6.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.6.self_attn.g_a_proj",
+      "model.layers.6.self_attn.g_b_proj",
+      "model.layers.6.self_attn.k_conv1d",
+      "model.layers.6.self_attn.k_proj",
+      "model.layers.6.self_attn.o_norm",
+      "model.layers.6.self_attn.o_proj",
+      "model.layers.6.self_attn.q_conv1d",
+      "model.layers.6.self_attn.q_proj",
+      "model.layers.6.self_attn.qkv_proj",
+      "model.layers.6.self_attn.v_conv1d",
+      "model.layers.6.self_attn.v_proj",
+      "model.layers.7.hc_attn_base",
+      "model.layers.7.hc_attn_fn",
+      "model.layers.7.hc_attn_scale",
+      "model.layers.7.hc_ffn_base",
+      "model.layers.7.hc_ffn_fn",
+      "model.layers.7.hc_ffn_scale",
+      "model.layers.7.input_layernorm",
+      "model.layers.7.mlp.gate",
+      "model.layers.7.mlp.gate.e_score_correction_bias",
+      "model.layers.7.post_attention_layernorm",
+      "model.layers.7.self_attn.indexer.index_kpool_compress_ape",
+      "model.layers.7.self_attn.indexer.index_kpool_compress_gate",
+      "model.layers.7.self_attn.indexer.k_norm",
+      "model.layers.7.self_attn.indexer.k_norm.bias",
+      "model.layers.7.self_attn.indexer.weights_proj",
+      "model.layers.7.self_attn.indexer.wk",
+      "model.layers.7.self_attn.indexer.wq_b",
+      "model.layers.7.self_attn.kv_a_layernorm",
+      "model.layers.7.self_attn.kv_b_proj",
+      "model.layers.7.self_attn.q_a_layernorm",
+      "model.layers.8.hc_attn_base",
+      "model.layers.8.hc_attn_fn",
+      "model.layers.8.hc_attn_scale",
+      "model.layers.8.hc_ffn_base",
+      "model.layers.8.hc_ffn_fn",
+      "model.layers.8.hc_ffn_scale",
+      "model.layers.8.input_layernorm",
+      "model.layers.8.mlp.gate",
+      "model.layers.8.mlp.gate.e_score_correction_bias",
+      "model.layers.8.post_attention_layernorm",
+      "model.layers.8.self_attn.A_log",
+      "model.layers.8.self_attn.b_proj",
+      "model.layers.8.self_attn.dt_bias",
+      "model.layers.8.self_attn.f_a_proj",
+      "model.layers.8.self_attn.f_b_proj",
+      "model.layers.8.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.8.self_attn.g_a_proj",
+      "model.layers.8.self_attn.g_b_proj",
+      "model.layers.8.self_attn.k_conv1d",
+      "model.layers.8.self_attn.k_proj",
+      "model.layers.8.self_attn.o_norm",
+      "model.layers.8.self_attn.o_proj",
+      "model.layers.8.self_attn.q_conv1d",
+      "model.layers.8.self_attn.q_proj",
+      "model.layers.8.self_attn.qkv_proj",
+      "model.layers.8.self_attn.v_conv1d",
+      "model.layers.8.self_attn.v_proj",
+      "model.layers.9.hc_attn_base",
+      "model.layers.9.hc_attn_fn",
+      "model.layers.9.hc_attn_scale",
+      "model.layers.9.hc_ffn_base",
+      "model.layers.9.hc_ffn_fn",
+      "model.layers.9.hc_ffn_scale",
+      "model.layers.9.input_layernorm",
+      "model.layers.9.mlp.gate",
+      "model.layers.9.mlp.gate.e_score_correction_bias",
+      "model.layers.9.post_attention_layernorm",
+      "model.layers.9.self_attn.A_log",
+      "model.layers.9.self_attn.b_proj",
+      "model.layers.9.self_attn.dt_bias",
+      "model.layers.9.self_attn.f_a_proj",
+      "model.layers.9.self_attn.f_b_proj",
+      "model.layers.9.self_attn.fused_qkvbfg_a_proj",
+      "model.layers.9.self_attn.g_a_proj",
+      "model.layers.9.self_attn.g_b_proj",
+      "model.layers.9.self_attn.k_conv1d",
+      "model.layers.9.self_attn.k_proj",
+      "model.layers.9.self_attn.o_norm",
+      "model.layers.9.self_attn.o_proj",
+      "model.layers.9.self_attn.q_conv1d",
+      "model.layers.9.self_attn.q_proj",
+      "model.layers.9.self_attn.qkv_proj",
+      "model.layers.9.self_attn.v_conv1d",
+      "model.layers.9.self_attn.v_proj",
+      "model.norm",
+      "model.visual",
+      "router",
+      "visual",
+      "visual.blocks.0.attn.k_norm",
+      "visual.blocks.0.attn.proj",
+      "visual.blocks.0.attn.proj.bias",
+      "visual.blocks.0.attn.q_norm",
+      "visual.blocks.0.attn.qkv",
+      "visual.blocks.0.attn.qkv.bias",
+      "visual.blocks.0.mlp.down_proj",
+      "visual.blocks.0.mlp.down_proj.bias",
+      "visual.blocks.0.mlp.gate_proj",
+      "visual.blocks.0.mlp.gate_proj.bias",
+      "visual.blocks.0.mlp.up_proj",
+      "visual.blocks.0.mlp.up_proj.bias",
+      "visual.blocks.0.norm1",
+      "visual.blocks.0.norm2",
+      "visual.blocks.1.attn.k_norm",
+      "visual.blocks.1.attn.proj",
+      "visual.blocks.1.attn.proj.bias",
+      "visual.blocks.1.attn.q_norm",
+      "visual.blocks.1.attn.qkv",
+      "visual.blocks.1.attn.qkv.bias",
+      "visual.blocks.1.mlp.down_proj",
+      "visual.blocks.1.mlp.down_proj.bias",
+      "visual.blocks.1.mlp.gate_proj",
+      "visual.blocks.1.mlp.gate_proj.bias",
+      "visual.blocks.1.mlp.up_proj",
+      "visual.blocks.1.mlp.up_proj.bias",
+      "visual.blocks.1.norm1",
+      "visual.blocks.1.norm2",
+      "visual.blocks.10.attn.k_norm",
+      "visual.blocks.10.attn.proj",
+      "visual.blocks.10.attn.proj.bias",
+      "visual.blocks.10.attn.q_norm",
+      "visual.blocks.10.attn.qkv",
+      "visual.blocks.10.attn.qkv.bias",
+      "visual.blocks.10.mlp.down_proj",
+      "visual.blocks.10.mlp.down_proj.bias",
+      "visual.blocks.10.mlp.gate_proj",
+      "visual.blocks.10.mlp.gate_proj.bias",
+      "visual.blocks.10.mlp.up_proj",
+      "visual.blocks.10.mlp.up_proj.bias",
+      "visual.blocks.10.norm1",
+      "visual.blocks.10.norm2",
+      "visual.blocks.11.attn.k_norm",
+      "visual.blocks.11.attn.proj",
+      "visual.blocks.11.attn.proj.bias",
+      "visual.blocks.11.attn.q_norm",
+      "visual.blocks.11.attn.qkv",
+      "visual.blocks.11.attn.qkv.bias",
+      "visual.blocks.11.mlp.down_proj",
+      "visual.blocks.11.mlp.down_proj.bias",
+      "visual.blocks.11.mlp.gate_proj",
+      "visual.blocks.11.mlp.gate_proj.bias",
+      "visual.blocks.11.mlp.up_proj",
+      "visual.blocks.11.mlp.up_proj.bias",
+      "visual.blocks.11.norm1",
+      "visual.blocks.11.norm2",
+      "visual.blocks.12.attn.k_norm",
+      "visual.blocks.12.attn.proj",
+      "visual.blocks.12.attn.proj.bias",
+      "visual.blocks.12.attn.q_norm",
+      "visual.blocks.12.attn.qkv",
+      "visual.blocks.12.attn.qkv.bias",
+      "visual.blocks.12.mlp.down_proj",
+      "visual.blocks.12.mlp.down_proj.bias",
+      "visual.blocks.12.mlp.gate_proj",
+      "visual.blocks.12.mlp.gate_proj.bias",
+      "visual.blocks.12.mlp.up_proj",
+      "visual.blocks.12.mlp.up_proj.bias",
+      "visual.blocks.12.norm1",
+      "visual.blocks.12.norm2",
+      "visual.blocks.13.attn.k_norm",
+      "visual.blocks.13.attn.proj",
+      "visual.blocks.13.attn.proj.bias",
+      "visual.blocks.13.attn.q_norm",
+      "visual.blocks.13.attn.qkv",
+      "visual.blocks.13.attn.qkv.bias",
+      "visual.blocks.13.mlp.down_proj",
+      "visual.blocks.13.mlp.down_proj.bias",
+      "visual.blocks.13.mlp.gate_proj",
+      "visual.blocks.13.mlp.gate_proj.bias",
+      "visual.blocks.13.mlp.up_proj",
+      "visual.blocks.13.mlp.up_proj.bias",
+      "visual.blocks.13.norm1",
+      "visual.blocks.13.norm2",
+      "visual.blocks.14.attn.k_norm",
+      "visual.blocks.14.attn.proj",
+      "visual.blocks.14.attn.proj.bias",
+      "visual.blocks.14.attn.q_norm",
+      "visual.blocks.14.attn.qkv",
+      "visual.blocks.14.attn.qkv.bias",
+      "visual.blocks.14.mlp.down_proj",
+      "visual.blocks.14.mlp.down_proj.bias",
+      "visual.blocks.14.mlp.gate_proj",
+      "visual.blocks.14.mlp.gate_proj.bias",
+      "visual.blocks.14.mlp.up_proj",
+      "visual.blocks.14.mlp.up_proj.bias",
+      "visual.blocks.14.norm1",
+      "visual.blocks.14.norm2",
+      "visual.blocks.15.attn.k_norm",
+      "visual.blocks.15.attn.proj",
+      "visual.blocks.15.attn.proj.bias",
+      "visual.blocks.15.attn.q_norm",
+      "visual.blocks.15.attn.qkv",
+      "visual.blocks.15.attn.qkv.bias",
+      "visual.blocks.15.mlp.down_proj",
+      "visual.blocks.15.mlp.down_proj.bias",
+      "visual.blocks.15.mlp.gate_proj",
+      "visual.blocks.15.mlp.gate_proj.bias",
+      "visual.blocks.15.mlp.up_proj",
+      "visual.blocks.15.mlp.up_proj.bias",
+      "visual.blocks.15.norm1",
+      "visual.blocks.15.norm2",
+      "visual.blocks.16.attn.k_norm",
+      "visual.blocks.16.attn.proj",
+      "visual.blocks.16.attn.proj.bias",
+      "visual.blocks.16.attn.q_norm",
+      "visual.blocks.16.attn.qkv",
+      "visual.blocks.16.attn.qkv.bias",
+      "visual.blocks.16.mlp.down_proj",
+      "visual.blocks.16.mlp.down_proj.bias",
+      "visual.blocks.16.mlp.gate_proj",
+      "visual.blocks.16.mlp.gate_proj.bias",
+      "visual.blocks.16.mlp.up_proj",
+      "visual.blocks.16.mlp.up_proj.bias",
+      "visual.blocks.16.norm1",
+      "visual.blocks.16.norm2",
+      "visual.blocks.17.attn.k_norm",
+      "visual.blocks.17.attn.proj",
+      "visual.blocks.17.attn.proj.bias",
+      "visual.blocks.17.attn.q_norm",
+      "visual.blocks.17.attn.qkv",
+      "visual.blocks.17.attn.qkv.bias",
+      "visual.blocks.17.mlp.down_proj",
+      "visual.blocks.17.mlp.down_proj.bias",
+      "visual.blocks.17.mlp.gate_proj",
+      "visual.blocks.17.mlp.gate_proj.bias",
+      "visual.blocks.17.mlp.up_proj",
+      "visual.blocks.17.mlp.up_proj.bias",
+      "visual.blocks.17.norm1",
+      "visual.blocks.17.norm2",
+      "visual.blocks.18.attn.k_norm",
+      "visual.blocks.18.attn.proj",
+      "visual.blocks.18.attn.proj.bias",
+      "visual.blocks.18.attn.q_norm",
+      "visual.blocks.18.attn.qkv",
+      "visual.blocks.18.attn.qkv.bias",
+      "visual.blocks.18.mlp.down_proj",
+      "visual.blocks.18.mlp.down_proj.bias",
+      "visual.blocks.18.mlp.gate_proj",
+      "visual.blocks.18.mlp.gate_proj.bias",
+      "visual.blocks.18.mlp.up_proj",
+      "visual.blocks.18.mlp.up_proj.bias",
+      "visual.blocks.18.norm1",
+      "visual.blocks.18.norm2",
+      "visual.blocks.19.attn.k_norm",
+      "visual.blocks.19.attn.proj",
+      "visual.blocks.19.attn.proj.bias",
+      "visual.blocks.19.attn.q_norm",
+      "visual.blocks.19.attn.qkv",
+      "visual.blocks.19.attn.qkv.bias",
+      "visual.blocks.19.mlp.down_proj",
+      "visual.blocks.19.mlp.down_proj.bias",
+      "visual.blocks.19.mlp.gate_proj",
+      "visual.blocks.19.mlp.gate_proj.bias",
+      "visual.blocks.19.mlp.up_proj",
+      "visual.blocks.19.mlp.up_proj.bias",
+      "visual.blocks.19.norm1",
+      "visual.blocks.19.norm2",
+      "visual.blocks.2.attn.k_norm",
+      "visual.blocks.2.attn.proj",
+      "visual.blocks.2.attn.proj.bias",
+      "visual.blocks.2.attn.q_norm",
+      "visual.blocks.2.attn.qkv",
+      "visual.blocks.2.attn.qkv.bias",
+      "visual.blocks.2.mlp.down_proj",
+      "visual.blocks.2.mlp.down_proj.bias",
+      "visual.blocks.2.mlp.gate_proj",
+      "visual.blocks.2.mlp.gate_proj.bias",
+      "visual.blocks.2.mlp.up_proj",
+      "visual.blocks.2.mlp.up_proj.bias",
+      "visual.blocks.2.norm1",
+      "visual.blocks.2.norm2",
+      "visual.blocks.20.attn.k_norm",
+      "visual.blocks.20.attn.proj",
+      "visual.blocks.20.attn.proj.bias",
+      "visual.blocks.20.attn.q_norm",
+      "visual.blocks.20.attn.qkv",
+      "visual.blocks.20.attn.qkv.bias",
+      "visual.blocks.20.mlp.down_proj",
+      "visual.blocks.20.mlp.down_proj.bias",
+      "visual.blocks.20.mlp.gate_proj",
+      "visual.blocks.20.mlp.gate_proj.bias",
+      "visual.blocks.20.mlp.up_proj",
+      "visual.blocks.20.mlp.up_proj.bias",
+      "visual.blocks.20.norm1",
+      "visual.blocks.20.norm2",
+      "visual.blocks.21.attn.k_norm",
+      "visual.blocks.21.attn.proj",
+      "visual.blocks.21.attn.proj.bias",
+      "visual.blocks.21.attn.q_norm",
+      "visual.blocks.21.attn.qkv",
+      "visual.blocks.21.attn.qkv.bias",
+      "visual.blocks.21.mlp.down_proj",
+      "visual.blocks.21.mlp.down_proj.bias",
+      "visual.blocks.21.mlp.gate_proj",
+      "visual.blocks.21.mlp.gate_proj.bias",
+      "visual.blocks.21.mlp.up_proj",
+      "visual.blocks.21.mlp.up_proj.bias",
+      "visual.blocks.21.norm1",
+      "visual.blocks.21.norm2",
+      "visual.blocks.22.attn.k_norm",
+      "visual.blocks.22.attn.proj",
+      "visual.blocks.22.attn.proj.bias",
+      "visual.blocks.22.attn.q_norm",
+      "visual.blocks.22.attn.qkv",
+      "visual.blocks.22.attn.qkv.bias",
+      "visual.blocks.22.mlp.down_proj",
+      "visual.blocks.22.mlp.down_proj.bias",
+      "visual.blocks.22.mlp.gate_proj",
+      "visual.blocks.22.mlp.gate_proj.bias",
+      "visual.blocks.22.mlp.up_proj",
+      "visual.blocks.22.mlp.up_proj.bias",
+      "visual.blocks.22.norm1",
+      "visual.blocks.22.norm2",
+      "visual.blocks.23.attn.k_norm",
+      "visual.blocks.23.attn.proj",
+      "visual.blocks.23.attn.proj.bias",
+      "visual.blocks.23.attn.q_norm",
+      "visual.blocks.23.attn.qkv",
+      "visual.blocks.23.attn.qkv.bias",
+      "visual.blocks.23.mlp.down_proj",
+      "visual.blocks.23.mlp.down_proj.bias",
+      "visual.blocks.23.mlp.gate_proj",
+      "visual.blocks.23.mlp.gate_proj.bias",
+      "visual.blocks.23.mlp.up_proj",
+      "visual.blocks.23.mlp.up_proj.bias",
+      "visual.blocks.23.norm1",
+      "visual.blocks.23.norm2",
+      "visual.blocks.3.attn.k_norm",
+      "visual.blocks.3.attn.proj",
+      "visual.blocks.3.attn.proj.bias",
+      "visual.blocks.3.attn.q_norm",
+      "visual.blocks.3.attn.qkv",
+      "visual.blocks.3.attn.qkv.bias",
+      "visual.blocks.3.mlp.down_proj",
+      "visual.blocks.3.mlp.down_proj.bias",
+      "visual.blocks.3.mlp.gate_proj",
+      "visual.blocks.3.mlp.gate_proj.bias",
+      "visual.blocks.3.mlp.up_proj",
+      "visual.blocks.3.mlp.up_proj.bias",
+      "visual.blocks.3.norm1",
+      "visual.blocks.3.norm2",
+      "visual.blocks.4.attn.k_norm",
+      "visual.blocks.4.attn.proj",
+      "visual.blocks.4.attn.proj.bias",
+      "visual.blocks.4.attn.q_norm",
+      "visual.blocks.4.attn.qkv",
+      "visual.blocks.4.attn.qkv.bias",
+      "visual.blocks.4.mlp.down_proj",
+      "visual.blocks.4.mlp.down_proj.bias",
+      "visual.blocks.4.mlp.gate_proj",
+      "visual.blocks.4.mlp.gate_proj.bias",
+      "visual.blocks.4.mlp.up_proj",
+      "visual.blocks.4.mlp.up_proj.bias",
+      "visual.blocks.4.norm1",
+      "visual.blocks.4.norm2",
+      "visual.blocks.5.attn.k_norm",
+      "visual.blocks.5.attn.proj",
+      "visual.blocks.5.attn.proj.bias",
+      "visual.blocks.5.attn.q_norm",
+      "visual.blocks.5.attn.qkv",
+      "visual.blocks.5.attn.qkv.bias",
+      "visual.blocks.5.mlp.down_proj",
+      "visual.blocks.5.mlp.down_proj.bias",
+      "visual.blocks.5.mlp.gate_proj",
+      "visual.blocks.5.mlp.gate_proj.bias",
+      "visual.blocks.5.mlp.up_proj",
+      "visual.blocks.5.mlp.up_proj.bias",
+      "visual.blocks.5.norm1",
+      "visual.blocks.5.norm2",
+      "visual.blocks.6.attn.k_norm",
+      "visual.blocks.6.attn.proj",
+      "visual.blocks.6.attn.proj.bias",
+      "visual.blocks.6.attn.q_norm",
+      "visual.blocks.6.attn.qkv",
+      "visual.blocks.6.attn.qkv.bias",
+      "visual.blocks.6.mlp.down_proj",
+      "visual.blocks.6.mlp.down_proj.bias",
+      "visual.blocks.6.mlp.gate_proj",
+      "visual.blocks.6.mlp.gate_proj.bias",
+      "visual.blocks.6.mlp.up_proj",
+      "visual.blocks.6.mlp.up_proj.bias",
+      "visual.blocks.6.norm1",
+      "visual.blocks.6.norm2",
+      "visual.blocks.7.attn.k_norm",
+      "visual.blocks.7.attn.proj",
+      "visual.blocks.7.attn.proj.bias",
+      "visual.blocks.7.attn.q_norm",
+      "visual.blocks.7.attn.qkv",
+      "visual.blocks.7.attn.qkv.bias",
+      "visual.blocks.7.mlp.down_proj",
+      "visual.blocks.7.mlp.down_proj.bias",
+      "visual.blocks.7.mlp.gate_proj",
+      "visual.blocks.7.mlp.gate_proj.bias",
+      "visual.blocks.7.mlp.up_proj",
+      "visual.blocks.7.mlp.up_proj.bias",
+      "visual.blocks.7.norm1",
+      "visual.blocks.7.norm2",
+      "visual.blocks.8.attn.k_norm",
+      "visual.blocks.8.attn.proj",
+      "visual.blocks.8.attn.proj.bias",
+      "visual.blocks.8.attn.q_norm",
+      "visual.blocks.8.attn.qkv",
+      "visual.blocks.8.attn.qkv.bias",
+      "visual.blocks.8.mlp.down_proj",
+      "visual.blocks.8.mlp.down_proj.bias",
+      "visual.blocks.8.mlp.gate_proj",
+      "visual.blocks.8.mlp.gate_proj.bias",
+      "visual.blocks.8.mlp.up_proj",
+      "visual.blocks.8.mlp.up_proj.bias",
+      "visual.blocks.8.norm1",
+      "visual.blocks.8.norm2",
+      "visual.blocks.9.attn.k_norm",
+      "visual.blocks.9.attn.proj",
+      "visual.blocks.9.attn.proj.bias",
+      "visual.blocks.9.attn.q_norm",
+      "visual.blocks.9.attn.qkv",
+      "visual.blocks.9.attn.qkv.bias",
+      "visual.blocks.9.mlp.down_proj",
+      "visual.blocks.9.mlp.down_proj.bias",
+      "visual.blocks.9.mlp.gate_proj",
+      "visual.blocks.9.mlp.gate_proj.bias",
+      "visual.blocks.9.mlp.up_proj",
+      "visual.blocks.9.mlp.up_proj.bias",
+      "visual.blocks.9.norm1",
+      "visual.blocks.9.norm2",
+      "visual.downsample",
+      "visual.downsample.bias",
+      "visual.merger.down_proj",
+      "visual.merger.gate_proj",
+      "visual.merger.post_projection_norm",
+      "visual.merger.post_projection_norm.bias",
+      "visual.merger.proj",
+      "visual.merger.up_proj",
+      "visual.patch_embed.proj",
+      "visual.patch_embed.proj.bias",
+      "visual.post_layernorm",
+      "weights_proj"
+    ],
+    "quant_method": "fp8",
+    "weight_block_size": [
+      128,
+      128
+    ]
+  }
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "5.16.0"
+}

model-00001-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ff3c9397e75fc8883f24d8474960f53e993b0c0e4a8a4687e81a1cbec39965b
+size 5365306704

model-00002-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:328225cabdc9b1595bc8bd0062cdc99fb23872eb5bbf7ec5db62f2be8abeaeeb
+size 5320647824

model-00003-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0fc42c270a8f8a2c1a38c91b06c28ca0646721f8d4768fa09ed97faca918300
+size 5363467432

model-00004-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea66d9139b41d1fa83d5da3d29ee66bc4bd594faa143bceaf7e014f584903d24
+size 5364342296

model-00005-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1da1e65d621a46aacb41edb38b6a8efd085fa4cd8b1d8e37dc1452e9814d46f
+size 5363915920

model-00006-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1653c5e3d9af10b57eea0bf5d853ca07090a1210e157bdd22ebc926cf1800021
+size 5361809384

model-00007-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82396e0f55ab6768568fe1292d37c295045363c27676067dcf92d9249af10216
+size 5364342528

model-00008-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91f5cb329f6d3c0648e9ed9e6018f0331e64e33424d6b7aaab2ca100aa820193
+size 5364342304

model-00009-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69cbb58b336f43eb8d2201adce20b8741b128b6450105e0cab7b2fffec8be9c4
+size 5364169856

model-00010-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db5ea14fed53c98c60c43d3326907c93386809690ee5b20c1d0e7052e798d54d
+size 5361982080

model-00011-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:408046268e305f528c6e60211d7fb9c4c5200b8a156751562fc34d15363189c5
+size 5363915912

model-00012-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc61a5684e5fae401ee1efb9be6f8f0673f27f48a02f9455caa7a448970bd3c4
+size 5364342288

model-00013-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfecdc0fd8d936dacc740b660a609e779d6bb71d39a2daaa4bef886544a8aa97
+size 5361809408

model-00014-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bab323a2bb1774e63a8bbf19fc1dd1fa1c34e43d6c024133379c86cdbdfaf141
+size 5364342504

model-00015-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e18e404b26f86a6f29c1db13ca51003ee2b031500f6e275af7019069e2b94bcd
+size 5364342304

model-00016-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cc97434dfac843692ccc3baf4a140d01a4e8dac88001bb0a656205eadf37828
+size 5361809176

model-00017-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:530a4034da64dc4a94aae6ef2bafd40d4b498b324f6143a1fc60d548c356e407
+size 5364084560

model-00018-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:035ee33c3279b45b6eb8894ec224d0edbf5a69f914b92879d85ababf8fc60a83
+size 5364342304

model-00019-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28b540f19710314c95ad625cf9377cdea3b650d52f9c34e4dbf35e27976745d7
+size 5364342272

model-00020-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67a86a8bedaa6d4672f75444fa353fb3435d2a1e3b9083f78d3421926d7ac3f2
+size 5361809528

model-00021-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48c8bbefadcf3d28ca6b27309959cc72e62db128e3e34cc74b6b0e25728e4128
+size 5364342376

model-00022-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b0232707a753fa3d753772ef066998614bcd8ae6234d3648f0dee924ad8fcb6
+size 5363915936

model-00023-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ea8f1bbc7e134c55b6ff91e52ff83acd3e72a199f6c762507872ab5dad7dded
+size 5361809264

model-00024-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cda3a7dcfbfcebb73163f6b45e5b7886276ac3cfec1236045d070bea58afba0
+size 5364342640

model-00025-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7527b7182a3fd498507886d27088c237a3022bc1dbafdbbf9c16a00ca448aac2
+size 5364342304

model-00026-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29aa015bbd8d0d1bea6c14d7c4e8f1a1cf7f17713981fb94ec10764090c68d6b
+size 5364342272

model-00027-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdafd6e7064525704e815090dd550f0928bc169e99df9b164e7c76a9b4e420d8
+size 5361809560

model-00028-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07089bdfe63c5807475f0a1fd379e31679ec9fdecd862d6374793fbdaca12a8b
+size 5363915976

model-00029-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4013ec692173700340046059100aed025d8789f4446e3d959857e608d37f139b
+size 5364342304

model-00030-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a28adbc99919ee9941a37f5de23771f22746331e7254c0822646edb9a151eb73
+size 5361809296

model-00031-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2f31cb37dcf8ca7000e82990f3af061a847f1086f79f207c7223fa090f04502
+size 5364341544

model-00032-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f33b53853ea5576b5aabd259bd9d933e6dbbaf65c511a8cdb328caab4349d8a9
+size 5363915232

model-00033-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a090ee4e5aeeb06b67d46a052b2103864c5f242a3f146f92b7fea06f929772f6
+size 5364342272

model-00034-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:451bb0368e52f65c765507aa7bec1b28575c33fd828d377b18b34f165e23f3f8
+size 5361809552

model-00035-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc98812806e93ca6340e44724e8a7de6ee48b5567727f21c75ba6bb47c6275aa
+size 5363915984

model-00036-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19803f6807229cc40d95afa89110226249f64c3a5cff7d355a58382249f90100
+size 5364342304

model-00037-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e7c51bd69d86a51d23ed25095589e42e1fa804974bae4a17c819f805a366901
+size 5361809288

model-00038-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e903f70ca5e94734fcc9ff036531ae9c2abefb5f5105749659bba372d59ba125
+size 5364342624

model-00039-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eacebc6414c33d9cad3c9c3d784b218437b8a7b0cc73529341361c4ae8de09df
+size 5364342296

model-00040-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f7c05711e1d0bafcd1e7894876a2cf961d4b370243bca17625d2b1a0892ceb6
+size 5363915912

model-00041-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ddfde38b0f0f2062350faecd733762f7abf4b4e6123dcdb7141135c60ad47a4
+size 5361809552

model-00042-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b70ffbe9d8272770a164f950811d5675f18c98d9f35e49a43a897e342ff4c98
+size 5364342368

model-00043-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5fede3a75553705c73f26d1cca6c784bc1a9bbb0735403d1d4885cfdd139760
+size 5364342296

model-00044-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed57a93696b5d999962fba19c3866a66f11ae54989e5af5f377d50043791a1db
+size 5361809312