Upload RivetCoder-9B-A4B-FP8 TorchAO checkpoint

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,4 @@
+__pycache__/
+*.pyc
+*.tmp
+.cache/

LICENSE

@@ -0,0 +1,71 @@
+LFM Open License v1.0 
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by this document.
+
+"Licensor" shall mean Liquid AI, Inc.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work.
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+"Commercial Use" shall mean any use of the Work for direct or indirect commercial advantage or monetary compensation.
+
+"Qualified Non-Profit Organization" shall mean a Legal Entity that is organized and operated exclusively for religious, charitable, scientific, testing for public safety, literary, or educational purposes, and which is exempt from federal income tax under Section 501(c)(3) of the United States Internal Revenue Code of 1986, as amended, or any equivalent non-profit or charitable organization in a foreign jurisdiction.
+
+"Non-Commercial or Research Purposes" shall mean purposes that do not involve any use of the Work or a Derivative Work for Commercial Use.
+
+"Threshold" shall mean annual revenue of 10 million United States dollars ($10,000,000) or more.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, including the Commercial Use limitation set forth in Section 5, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License, including the Commercial Use limitation set forth in Section 5, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+5. Commercial Use Limitation.
+
+(a) The rights granted under this License for Commercial Use are conditioned upon You or Your Legal Entity not exceeding the Threshold.
+
+(b) Any Commercial Use of the Work or a Derivative Work by a Legal Entity that exceeds the Threshold is not licensed under this Agreement.
+
+(c) The Threshold shall not apply to a Qualified Non-Profit Organization's use of the Work or a Derivative Work for Non-Commercial or Research Purposes.
+
+6. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+7. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except for the reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+8. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+9. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+10. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+11. Termination. This License will terminate automatically and immediately if You fail to comply with any of its terms and conditions. Upon termination, You must cease all use of the Work and any Derivative Works and delete all copies in Your possession.
+
+END OF TERMS AND CONDITIONS

NOTICE.md

@@ -0,0 +1,20 @@
+# Attribution notice
+
+RivetCoder-9B-A4B combines the following sources:
+
+- Host model: `LiquidAI/LFM2.5-2.6B`, revision
+  `654f9463ce32b05d0429d76fe1f580b27d4c1ac0`, under the LFM Open License v1.0.
+- Expert donor: `zai-org/GLM-5.3-Flash`, revision
+  `3f1971b7b5f7a528c9c4ef6212c8785298a8c24a`, under the MIT License.
+- Router sequence teacher: `qwen/qwen3.8-27b`, accessed through a local LM Studio
+  endpoint. No Qwen weights are included.
+- Architectural reference: `Akahsizrr/fuse-1-Lite`, revision
+  `430c959e47556ae53fed18a9d97f7cf30876e6ff`.
+
+Modifications and new work include GLM expert selection, a fixed tied
+identity-Hadamard bridge, expert/router weight folding, per-layer Top-4 routing,
+token gating, bounded residual scaling, Qwen-supervised routing-control training,
+and the custom `fuse_glm` Transformers implementation shipped in this repository.
+
+See `provenance/` for the complete local build record with personal filesystem
+paths removed.

README.md

@@ -0,0 +1,150 @@
+---
+license: other
+license_name: lfm-open-license-v1.0
+license_link: LICENSE
+library_name: transformers
+pipeline_tag: text-generation
+base_model: HCHs/RivetCoder-9B-A4B
+base_model_relation: quantized
+language:
+  - en
+  - ko
+  - code
+tags:
+  - custom_code
+  - lfm2
+  - glm
+  - mixture-of-experts
+  - routed-experts
+  - coding
+  - code-generation
+  - fp8
+  - torchao
+  - top-k-routing
+  - trust-remote-code
+---
+
+# RivetCoder-9B-A4B-FP8
+
+RivetCoder-9B-A4B-FP8 is the TorchAO FP8 deployment build of
+[`HCHs/RivetCoder-9B-A4B`](https://huggingface.co/HCHs/RivetCoder-9B-A4B).
+It keeps the same experimental coding-oriented architecture: a frozen
+`LiquidAI/LFM2.5-2.6B` host plus 16 layer-qualified GLM-derived FFN candidates
+at each of 30 layers, with Top-4 routing per token.
+
+This repository contains custom Transformers code and must be loaded with
+`trust_remote_code=True`.
+
+## FP8 format
+
+The checkpoint was quantized with TorchAO
+`Float8DynamicActivationFloat8WeightConfig` using E4M3 FP8 weights and dynamic
+FP8 activations for compatible `nn.Linear` modules.
+
+| Item | Value |
+|---|---:|
+| Source revision | `9a90b1917d9b5438e4d2fe1a4f6aea884db59a60` |
+| Approx. total parameters | 8.74B |
+| Approx. active parameters | 4.21B |
+| FP8 tensor-subclass parameters | 1,636 |
+| FP8-quantized parameter elements | 8,475,574,272 |
+| Stored tensor bytes | 9,000,638,976 |
+| Safetensors shards | 5 |
+| Tested resident CUDA allocation | about 8.4 GiB |
+
+Embeddings, convolution parameters, token gates, correction biases, residual
+scales, and other small or precision-sensitive tensors remain BF16 or FP32.
+Router projection matrices are FP8, while the custom router still computes its
+logits in FP32. “FP8” therefore describes compatible Linear matrices, not every
+scalar in the checkpoint.
+
+## Installation
+
+The exact local stack used to create and validate this build was PyTorch
+`2.12.0+cu130`, Transformers `5.16.1`, Accelerate `1.13.0`, Safetensors `0.8.0`,
+and TorchAO `0.15.0` on an NVIDIA GeForce RTX 5070 Ti (SM 12.0).
+
+```bash
+pip install "torch>=2.12,<2.13" "transformers>=5.16.1,<5.17" "accelerate>=1.13" \
+  "safetensors>=0.8" "torchao==0.15.0"
+```
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "HCHs/RivetCoder-9B-A4B-FP8"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    device_map=0,
+    dtype=torch.bfloat16,
+).eval()
+
+messages = [{
+    "role": "user",
+    "content": "Implement merge_intervals in Python and include concise tests.",
+}]
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+).to("cuda")
+
+# Use no_grad with the tested TorchAO/PyTorch stack.
+with torch.no_grad():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=1024,
+        temperature=0.2,
+        do_sample=True,
+    )
+
+print(tokenizer.decode(
+    output[0, inputs["input_ids"].shape[-1]:],
+    skip_special_tokens=True,
+))
+```
+
+The bundled chat template opens a reasoning segment before the final answer.
+Allocate enough output tokens for both reasoning and code.
+
+## Validation
+
+The saved FP8 tensors were independently reloaded entirely on one RTX 5070 Ti.
+The clean reload took about 338 seconds, found no CPU or `meta` parameters, and
+restored all 1,636 FP8 parameters. A separate six-token forward pass produced
+finite logits with shape `(1, 6, 128000)`.
+
+On the tested Windows stack, `torch.inference_mode()` is incompatible with the
+TorchAO FP8 tensor subclass and raises a version-counter error. Use
+`torch.no_grad()` as shown above.
+
+TorchAO 0.15.0 also reports that its optional C++ extensions are skipped with
+the tested PyTorch 2.12 build. The native CUDA FP8 path used by this checkpoint
+still passed quantization, serialization, clean reload, and forward validation.
+
+## Limitations
+
+- This is an experimental fusion with only 60 routing-control optimizer steps.
+- HumanEval, MBPP, SWE-bench, and broad regression results have not been reported.
+- The fixed GLM-to-LFM bridge is deterministic and was not learned.
+- This TorchAO checkpoint is not a GGUF file and is not directly compatible
+  with llama.cpp, LM Studio, or Ollama.
+- Hardware and software combinations other than the tested stack may need
+  additional compatibility work.
+
+See `provenance/quantization.json` for the local quantization and placement
+report. Architecture, source-model, expert-selection, and router-training
+provenance are retained from the BF16 repository.
+
+## License and attribution
+
+The LFM host remains subject to the included LFM Open License v1.0. GLM-derived
+expert tensors retain the included MIT license and attribution. Review
+`LICENSE`, `NOTICE.md`, and `licenses/` before redistribution or deployment.

chat_template.jinja

@@ -0,0 +1,125 @@
+{{- bos_token -}}
+{%- set preserve_thinking = preserve_thinking | default(false) -%}
+
+{%- macro format_arg_value(arg_value) -%}
+    {%- if arg_value is string -%}
+        {{- "'" + (arg_value | replace("\\", "\\\\") | replace("'", "\\'") | replace("\n", "\\n") | replace("\r", "\\r")) + "'" -}}
+    {%- elif arg_value is mapping or arg_value is iterable -%}
+        {{- arg_value | tojson -}}
+    {%- else -%}
+        {{- arg_value | string -}}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- macro parse_content(content) -%}
+    {%- if content is string -%}
+        {{- content -}}
+    {%- elif content is mapping -%}
+        {{- content | tojson -}}
+    {%- elif content is iterable -%}
+        {%- set _ns = namespace(result="") -%}
+        {%- for item in content -%}
+            {%- if item is string -%}
+                {%- set _ns.result = _ns.result + item -%}
+            {%- elif item is mapping and item.get("type") == "image" -%}
+                {%- set _ns.result = _ns.result + "<image>" -%}
+            {%- elif item is mapping and item.get("type") == "text" -%}
+                {%- set _ns.result = _ns.result + ((item.get("text") or "") | string) -%}
+            {%- else -%}
+                {%- set _ns.result = _ns.result + (item | tojson) -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {{- _ns.result -}}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- macro render_tool_calls(tool_calls) -%}
+    {%- set tool_calls_ns = namespace(tool_calls=[]) -%}
+    {%- for tool_call in tool_calls -%}
+        {%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
+        {%- set func_name = func["name"] -%}
+        {%- set func_args = func.get("arguments") -%}
+        {%- set args_ns = namespace(arg_strings=[]) -%}
+        {%- if func_args is mapping -%}
+            {%- for arg_name, arg_value in func_args.items() -%}
+                {%- set args_ns.arg_strings = args_ns.arg_strings + [arg_name + "=" + format_arg_value(arg_value)] -%}
+            {%- endfor -%}
+        {%- elif func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
+            {{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
+        {%- endif -%}
+        {%- set tool_calls_ns.tool_calls = tool_calls_ns.tool_calls + [func_name + "(" + (args_ns.arg_strings | join(", ")) + ")"] -%}
+    {%- endfor -%}
+    {{- "<|tool_call_start|>[" + (tool_calls_ns.tool_calls | join(", ")) + "]<|tool_call_end|>" -}}
+{%- endmacro -%}
+
+{%- set ns = namespace(system_prompt="", last_user_index=-1) -%}
+{%- if messages and messages[0]["role"] == "system" -%}
+    {%- if messages[0].get("content") -%}
+        {%- set ns.system_prompt = parse_content(messages[0]["content"]) -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "user" -%}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message.role + "\n" -}}
+    {%- if message.role == "assistant" -%}
+        {%- generation -%}
+        {%- set keep_thinking = preserve_thinking or loop.index0 > ns.last_user_index -%}
+        {%- set thinking = message.thinking or message.reasoning or message.reasoning_content -%}
+        {%- set thinking = thinking if thinking is string else "" -%}
+        {%- if thinking and keep_thinking -%}
+            {{- "<think>" + thinking + "</think>" -}}
+        {%- endif -%}
+        {%- set _cfm_tag = "CONTINUE_FINAL_MESSAGE_TAG " -%}
+        {%- set _has_cfm = false -%}
+        {%- set content = "" -%}
+        {%- if message.get("content") -%}
+            {%- set content = parse_content(message.content) -%}
+        {%- endif -%}
+        {%- if not keep_thinking and "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+        {%- if content.endswith(_cfm_tag) -%}
+            {%- set _has_cfm = true -%}
+            {%- set _trunc_len = (content | length) - (_cfm_tag | length) -%}
+            {%- set content = content[:_trunc_len] -%}
+        {%- endif -%}
+        {{- content -}}
+        {%- if message.tool_calls -%}
+            {{- render_tool_calls(message.tool_calls) -}}
+        {%- endif -%}
+        {%- if _has_cfm -%}
+            {{- _cfm_tag -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+        {%- endgeneration -%}
+    {%- else %}
+        {%- if message.get("content") -%}
+            {{- parse_content(message["content"]) -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+    {%- endif %}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n<think>" -}}
+{%- endif -%}

config.json

@@ -0,0 +1,187 @@
+{
+  "architectures": [
+    "FuseGlmForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_fuse_glm.FuseGlmConfig",
+    "AutoModelForCausalLM": "modeling_fuse_glm.FuseGlmForCausalLM"
+  },
+  "block_auto_adjust_ff_dim": false,
+  "block_dim": 2048,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 124894,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 2048,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 124900,
+  "full_attn_idxs": null,
+  "fuse_glm_coding_enabled": true,
+  "fuse_glm_expert_intermediate_size": 2048,
+  "fuse_glm_gate_clamp_max": 10.0,
+  "fuse_glm_hard_token_gate_at_eval": false,
+  "fuse_glm_layer_indices": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29
+  ],
+  "fuse_glm_num_experts": 16,
+  "fuse_glm_output_router_diagnostics": false,
+  "fuse_glm_residual_scale_max": 0.1,
+  "fuse_glm_router_aux_loss_coef": 0.0,
+  "fuse_glm_token_gate_bias": -4.0,
+  "fuse_glm_token_gate_threshold": 0.5,
+  "fuse_glm_top_k": 4,
+  "fuse_glm_up_clamp_max": 10.0,
+  "fuse_glm_up_clamp_min": -10.0,
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 10752,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention"
+  ],
+  "max_position_embeddings": 131072,
+  "model_type": "fuse_glm",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 32,
+  "num_heads": 32,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 8,
+  "pad_token_id": 124893,
+  "quantization_config": {
+    "include_input_output_embeddings": false,
+    "modules_to_not_convert": null,
+    "quant_method": "torchao",
+    "quant_type": {
+      "default": {
+        "_data": {
+          "activation_dtype": {
+            "_data": "float8_e4m3fn",
+            "_type": "torch.dtype"
+          },
+          "activation_value_lb": null,
+          "activation_value_ub": null,
+          "granularity": [
+            {
+              "_data": {},
+              "_type": "PerTensor",
+              "_version": 1
+            },
+            {
+              "_data": {},
+              "_type": "PerTensor",
+              "_version": 1
+            }
+          ],
+          "kernel_preference": {
+            "_data": "AUTO",
+            "_type": "KernelPreference"
+          },
+          "mm_config": {
+            "_data": {
+              "emulate": false,
+              "pad_inner_dim": false,
+              "use_fast_accum": true
+            },
+            "_type": "Float8MMConfig",
+            "_version": 1
+          },
+          "set_inductor_config": true,
+          "weight_dtype": {
+            "_data": "float8_e4m3fn",
+            "_type": "torch.dtype"
+          }
+        },
+        "_type": "Float8DynamicActivationFloat8WeightConfig",
+        "_version": 2
+      }
+    },
+    "untie_embedding_weights": false
+  },
+  "rivet_coder": {
+    "active_parameters_approx": 4210000000,
+    "base_repo_id": "HCHs/RivetCoder-9B-A4B",
+    "donor": "zai-org/GLM-5.3-Flash",
+    "experts_per_layer": 16,
+    "experts_per_token": 4,
+    "host": "LiquidAI/LFM2.5-2.6B",
+    "name": "RivetCoder-9B-A4B-FP8",
+    "quantization": "TorchAO Float8DynamicActivationFloat8WeightConfig",
+    "repo_id": "HCHs/RivetCoder-9B-A4B-FP8",
+    "router_teacher": "qwen/qwen3.8-27b",
+    "total_parameters_approx": 8740000000,
+    "training_steps": 60
+  },
+  "rope_parameters": {
+    "rope_theta": 10000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "use_cache": true,
+  "use_pos_enc": true,
+  "vocab_size": 128000
+}

configuration_fuse_glm.py

@@ -0,0 +1,151 @@
+"""Configuration for the LFM2 + folded GLM expert model.
+
+The defaults describe the intended production architecture: sixteen folded
+GLM experts in every LFM2 layer, with four experts selected per token.
+All dimensions remain configurable so the implementation can be exercised
+with very small, synthetic models without downloading either checkpoint.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+from transformers import Lfm2Config
+
+
+class FuseGlmConfig(Lfm2Config):
+    """Extend :class:`~transformers.Lfm2Config` with sparse GLM sidecars.
+
+    Parameters prefixed with ``fuse_glm_`` only control the added expert
+    branch.  All ordinary LFM2 configuration arguments are forwarded to
+    ``Lfm2Config`` unchanged.
+
+    ``fuse_glm_layer_indices=None`` means that every decoder layer receives a
+    sidecar.  A concrete list can be supplied for ablations or staged builds.
+    The production model uses hidden/intermediate size 2048; tests may select
+    smaller values.
+    """
+
+    model_type = "fuse_glm"
+
+    def __init__(
+        self,
+        *,
+        fuse_glm_num_experts: int = 16,
+        fuse_glm_top_k: int = 4,
+        fuse_glm_expert_intermediate_size: int = 2048,
+        fuse_glm_layer_indices: list[int] | tuple[int, ...] | None = None,
+        fuse_glm_gate_clamp_max: float = 10.0,
+        fuse_glm_up_clamp_min: float = -10.0,
+        fuse_glm_up_clamp_max: float = 10.0,
+        fuse_glm_residual_scale_max: float = 0.1,
+        fuse_glm_router_aux_loss_coef: float = 0.01,
+        fuse_glm_token_gate_bias: float = -4.0,
+        fuse_glm_token_gate_threshold: float = 0.5,
+        fuse_glm_hard_token_gate_at_eval: bool = False,
+        fuse_glm_coding_enabled: bool = True,
+        fuse_glm_output_router_diagnostics: bool = False,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(**kwargs)
+
+        # Transformers 5.16 remaps legacy attention names for configurations
+        # it classifies as custom code.  LFM2's implementation still indexes
+        # masks with the literal key ``conv``, however, so a FuseGlmConfig
+        # must retain the host model's native spelling.  Without this reversal
+        # a real LFM checkpoint fails in Lfm2Model.forward with
+        # ``KeyError: 'linear_attention'`` even though an Lfm2Config loaded
+        # from the same file works correctly.
+        self.layer_types = [
+            "conv" if layer_type == "linear_attention" else layer_type
+            for layer_type in self.layer_types
+        ]
+
+        self.fuse_glm_num_experts = int(fuse_glm_num_experts)
+        self.fuse_glm_top_k = int(fuse_glm_top_k)
+        self.fuse_glm_expert_intermediate_size = int(fuse_glm_expert_intermediate_size)
+        self.fuse_glm_layer_indices = (
+            None if fuse_glm_layer_indices is None else [int(index) for index in fuse_glm_layer_indices]
+        )
+        self.fuse_glm_gate_clamp_max = float(fuse_glm_gate_clamp_max)
+        self.fuse_glm_up_clamp_min = float(fuse_glm_up_clamp_min)
+        self.fuse_glm_up_clamp_max = float(fuse_glm_up_clamp_max)
+        self.fuse_glm_residual_scale_max = float(fuse_glm_residual_scale_max)
+        self.fuse_glm_router_aux_loss_coef = float(fuse_glm_router_aux_loss_coef)
+        self.fuse_glm_token_gate_bias = float(fuse_glm_token_gate_bias)
+        self.fuse_glm_token_gate_threshold = float(fuse_glm_token_gate_threshold)
+        self.fuse_glm_hard_token_gate_at_eval = bool(fuse_glm_hard_token_gate_at_eval)
+        self.fuse_glm_coding_enabled = bool(fuse_glm_coding_enabled)
+        self.fuse_glm_output_router_diagnostics = bool(fuse_glm_output_router_diagnostics)
+
+        self._validate_fuse_glm_fields()
+
+        # Saved checkpoints should resolve to the fused class rather than the
+        # base LFM2 class.  This does not affect loading an ordinary LFM config
+        # through ``from_lfm_config`` below.
+        self.architectures = ["FuseGlmForCausalLM"]
+
+    @property
+    def resolved_fuse_glm_layer_indices(self) -> tuple[int, ...]:
+        """Return the validated decoder layer indices receiving experts."""
+
+        if self.fuse_glm_layer_indices is None:
+            return tuple(range(self.num_hidden_layers))
+        return tuple(self.fuse_glm_layer_indices)
+
+    def _validate_fuse_glm_fields(self) -> None:
+        if self.fuse_glm_num_experts < 1:
+            raise ValueError("fuse_glm_num_experts must be at least 1")
+        if not 1 <= self.fuse_glm_top_k <= self.fuse_glm_num_experts:
+            raise ValueError("fuse_glm_top_k must be between 1 and fuse_glm_num_experts")
+        if self.fuse_glm_expert_intermediate_size < 1:
+            raise ValueError("fuse_glm_expert_intermediate_size must be at least 1")
+        if self.fuse_glm_up_clamp_min >= self.fuse_glm_up_clamp_max:
+            raise ValueError("fuse_glm_up_clamp_min must be smaller than fuse_glm_up_clamp_max")
+        if self.fuse_glm_residual_scale_max <= 0:
+            raise ValueError("fuse_glm_residual_scale_max must be positive")
+        if self.fuse_glm_router_aux_loss_coef < 0:
+            raise ValueError("fuse_glm_router_aux_loss_coef cannot be negative")
+        if not 0.0 <= self.fuse_glm_token_gate_threshold <= 1.0:
+            raise ValueError("fuse_glm_token_gate_threshold must be in [0, 1]")
+
+        indices = self.resolved_fuse_glm_layer_indices
+        if len(set(indices)) != len(indices):
+            raise ValueError("fuse_glm_layer_indices cannot contain duplicates")
+        invalid = [index for index in indices if index < 0 or index >= self.num_hidden_layers]
+        if invalid:
+            raise ValueError(
+                "fuse_glm_layer_indices contains indices outside the LFM2 decoder: "
+                f"{invalid} (num_hidden_layers={self.num_hidden_layers})"
+            )
+
+    @classmethod
+    def from_lfm_config(
+        cls,
+        config: Lfm2Config | Mapping[str, Any],
+        **fuse_overrides: Any,
+    ) -> "FuseGlmConfig":
+        """Create a fused config from an existing LFM2 config.
+
+        This method is purely local and never resolves or downloads a model.
+        ``config`` may be an instantiated ``Lfm2Config`` or its dictionary.
+        """
+
+        if isinstance(config, Lfm2Config):
+            config_dict = config.to_dict()
+        elif isinstance(config, Mapping):
+            config_dict = dict(config)
+        else:
+            raise TypeError("config must be an Lfm2Config or a mapping")
+
+        # These values describe the source class/revision, not constructor
+        # fields of the new local class.
+        for key in ("model_type", "_commit_hash"):
+            config_dict.pop(key, None)
+        config_dict.pop("architectures", None)
+        config_dict.update(fuse_overrides)
+        return cls(**config_dict)
+
+
+__all__ = ["FuseGlmConfig"]

generation_config.json

@@ -0,0 +1,16 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 124894,
+  "do_sample": true,
+  "eos_token_id": [
+    124900
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 124893,
+  "repetition_penalty": 1.1,
+  "temperature": 0.1,
+  "top_k": 50,
+  "transformers_version": "5.16.1",
+  "use_cache": true
+}

licenses/GLM-MIT.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Z.AI Co., Ltd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

licenses/LFM-OPEN-LICENSE-v1.0.txt

@@ -0,0 +1,71 @@
+LFM Open License v1.0 
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by this document.
+
+"Licensor" shall mean Liquid AI, Inc.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work.
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+"Commercial Use" shall mean any use of the Work for direct or indirect commercial advantage or monetary compensation.
+
+"Qualified Non-Profit Organization" shall mean a Legal Entity that is organized and operated exclusively for religious, charitable, scientific, testing for public safety, literary, or educational purposes, and which is exempt from federal income tax under Section 501(c)(3) of the United States Internal Revenue Code of 1986, as amended, or any equivalent non-profit or charitable organization in a foreign jurisdiction.
+
+"Non-Commercial or Research Purposes" shall mean purposes that do not involve any use of the Work or a Derivative Work for Commercial Use.
+
+"Threshold" shall mean annual revenue of 10 million United States dollars ($10,000,000) or more.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, including the Commercial Use limitation set forth in Section 5, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License, including the Commercial Use limitation set forth in Section 5, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+5. Commercial Use Limitation.
+
+(a) The rights granted under this License for Commercial Use are conditioned upon You or Your Legal Entity not exceeding the Threshold.
+
+(b) Any Commercial Use of the Work or a Derivative Work by a Legal Entity that exceeds the Threshold is not licensed under this Agreement.
+
+(c) The Threshold shall not apply to a Qualified Non-Profit Organization's use of the Work or a Derivative Work for Non-Commercial or Research Purposes.
+
+6. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+7. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except for the reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+8. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+9. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+10. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+11. Termination. This License will terminate automatically and immediately if You fail to comply with any of its terms and conditions. Upon termination, You must cease all use of the Work and any Derivative Works and delete all copies in Your possession.
+
+END OF TERMS AND CONDITIONS

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a3c863c63fb2c52b7f881b483ffc7d1f4b352b2de932b718f62b2b2134de1e8
+size 1999384120

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1668117e8a2ad1c7d4ff9a12c1687c9e907a39738dcafd1a48647d38a6efed3
+size 1999530876

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f7266ae364fb4bb0c3f30da7deca8133d7501fb415840861de747716ed483ca
+size 1999519716

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebb55bc7f9cd7139d3e21c0677f2b2b7f99ec25925837729b0c07a6efa2b0074
+size 1999519740

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5aec98e4957c1263db1e79f2014f468620ec1b512831256329d8091f762fc61a
+size 1010472884

modeling_fuse_glm.py

@@ -0,0 +1,757 @@
+"""LFM2 with sparse, folded GLM-5.3-Flash coding experts.
+
+The implementation intentionally depends only on upstream ``transformers``
+LFM2 classes.  It does not import GLM modeling code and it never downloads a
+checkpoint at import or construction time.  Folded expert tensors can be
+copied into the exposed ``gate_proj``, ``up_proj`` and ``down_proj`` modules
+after a separate extraction/folding step.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Any, Iterator, Literal
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+from transformers import Lfm2Config, Lfm2ForCausalLM
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+from .configuration_fuse_glm import FuseGlmConfig
+
+
+class _AddAuxiliaryLoss(torch.autograd.Function):
+    """Attach an auxiliary scalar gradient without changing forward values.
+
+    This is useful here because LFM2 decoder layers return only hidden states.
+    Returning the router loss through that API would require replacing the
+    entire decoder stack and would also interfere with generation caches.  The
+    identity operation keeps the native forward contract and remains valid
+    when gradient checkpointing recomputes a decoder layer.
+    """
+
+    @staticmethod
+    def forward(ctx: Any, hidden_states: torch.Tensor, auxiliary_loss: torch.Tensor) -> torch.Tensor:
+        ctx.auxiliary_dtype = auxiliary_loss.dtype
+        ctx.auxiliary_device = auxiliary_loss.device
+        return hidden_states
+
+    @staticmethod
+    def backward(ctx: Any, grad_output: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+        auxiliary_grad = torch.ones((), dtype=ctx.auxiliary_dtype, device=ctx.auxiliary_device)
+        return grad_output, auxiliary_grad
+
+
+@dataclass(frozen=True)
+class RouterState:
+    """Differentiable routing state from one fused decoder layer."""
+
+    layer_index: int
+    token_indices: torch.Tensor
+    router_logits: torch.Tensor
+    topk_indices: torch.Tensor
+    topk_weights: torch.Tensor
+    token_gate: torch.Tensor
+    auxiliary_loss: torch.Tensor
+
+    def detached(self) -> "RouterState":
+        return RouterState(
+            layer_index=self.layer_index,
+            token_indices=self.token_indices.detach(),
+            router_logits=self.router_logits.detach(),
+            topk_indices=self.topk_indices.detach(),
+            topk_weights=self.topk_weights.detach(),
+            token_gate=self.token_gate.detach(),
+            auxiliary_loss=self.auxiliary_loss.detach(),
+        )
+
+
+@dataclass(frozen=True)
+class RouterDiagnostics:
+    """Compact, detached-by-default statistics for monitoring routing."""
+
+    layer_index: int
+    token_count: int
+    active_token_count: int
+    expert_counts: torch.Tensor
+    mean_selected_weights: torch.Tensor
+    router_entropy: torch.Tensor
+    token_gate_mean: torch.Tensor
+    token_gate_active_fraction: torch.Tensor
+    residual_scale: torch.Tensor
+    auxiliary_loss: torch.Tensor
+    coding_enabled: bool
+
+    def detached(self) -> "RouterDiagnostics":
+        return RouterDiagnostics(
+            layer_index=self.layer_index,
+            token_count=self.token_count,
+            active_token_count=self.active_token_count,
+            expert_counts=self.expert_counts.detach(),
+            mean_selected_weights=self.mean_selected_weights.detach(),
+            router_entropy=self.router_entropy.detach(),
+            token_gate_mean=self.token_gate_mean.detach(),
+            token_gate_active_fraction=self.token_gate_active_fraction.detach(),
+            residual_scale=self.residual_scale.detach(),
+            auxiliary_loss=self.auxiliary_loss.detach(),
+            coding_enabled=self.coding_enabled,
+        )
+
+
+@dataclass
+class FuseGlmCausalLMOutputWithPast(CausalLMOutputWithPast):
+    """Causal LM output augmented with sparse-router training information."""
+
+    router_aux_loss: torch.FloatTensor | None = None
+    router_diagnostics: tuple[RouterDiagnostics, ...] | None = None
+
+
+class FoldedGlmExpert(nn.Module):
+    """A folded GLM-clamped SwiGLU expert in the LFM hidden space.
+
+    For the production configuration all three dimensions are 2048.  The
+    separate ``intermediate_size`` argument exists to enable inexpensive unit
+    tests and later structured compression experiments.
+    """
+
+    def __init__(
+        self,
+        hidden_size: int,
+        intermediate_size: int,
+        *,
+        gate_clamp_max: float = 10.0,
+        up_clamp_min: float = -10.0,
+        up_clamp_max: float = 10.0,
+        initializer_range: float = 0.02,
+    ) -> None:
+        super().__init__()
+        self.hidden_size = int(hidden_size)
+        self.intermediate_size = int(intermediate_size)
+        self.gate_clamp_max = float(gate_clamp_max)
+        self.up_clamp_min = float(up_clamp_min)
+        self.up_clamp_max = float(up_clamp_max)
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.reset_parameters(initializer_range)
+
+    def reset_parameters(self, initializer_range: float) -> None:
+        for projection in (self.gate_proj, self.up_proj, self.down_proj):
+            nn.init.normal_(projection.weight, mean=0.0, std=initializer_range)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        gate = self.gate_proj(hidden_states).clamp(max=self.gate_clamp_max)
+        up = self.up_proj(hidden_states).clamp(min=self.up_clamp_min, max=self.up_clamp_max)
+        return self.down_proj(F.silu(gate) * up)
+
+    @torch.no_grad()
+    def load_folded_weights(
+        self,
+        *,
+        gate_proj: torch.Tensor,
+        up_proj: torch.Tensor,
+        down_proj: torch.Tensor,
+    ) -> None:
+        """Validate and copy three already-folded expert matrices."""
+
+        supplied = {
+            "gate_proj": gate_proj,
+            "up_proj": up_proj,
+            "down_proj": down_proj,
+        }
+        modules = {
+            "gate_proj": self.gate_proj,
+            "up_proj": self.up_proj,
+            "down_proj": self.down_proj,
+        }
+        for name, tensor in supplied.items():
+            expected_shape = tuple(modules[name].weight.shape)
+            if tuple(tensor.shape) != expected_shape:
+                raise ValueError(f"{name} has shape {tuple(tensor.shape)}; expected {expected_shape}")
+            modules[name].weight.copy_(tensor.to(device=modules[name].weight.device, dtype=modules[name].weight.dtype))
+
+
+class TopKFoldedExpertRouter(nn.Module):
+    """GLM-style sigmoid top-k router with post-sigmoid choice correction."""
+
+    def __init__(self, hidden_size: int, num_experts: int, top_k: int, initializer_range: float) -> None:
+        super().__init__()
+        self.num_experts = int(num_experts)
+        self.top_k = int(top_k)
+        self.proj = nn.Linear(hidden_size, num_experts, bias=False)
+        nn.init.normal_(self.proj.weight, mean=0.0, std=initializer_range)
+        # GLM adds this value only while choosing experts.  The routed mixture
+        # weights are gathered from the uncorrected sigmoid scores, so this
+        # must not be represented as a Linear bias.
+        self.register_buffer(
+            "e_score_correction_bias",
+            torch.zeros(self.num_experts, dtype=torch.float32),
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+        # GLM computes router logits in FP32 even when expert weights are FP8
+        # or BF16.  Keeping that behavior also stabilizes small synthetic tests.
+        logits = F.linear(hidden_states.float(), self.proj.weight.float())
+        scores = torch.sigmoid(logits)
+        choice_scores = scores + self.e_score_correction_bias.float()
+        selected_indices = torch.topk(choice_scores, self.top_k, dim=-1).indices
+        selected_scores = scores.gather(-1, selected_indices)
+        selected_weights = selected_scores / selected_scores.sum(dim=-1, keepdim=True).clamp_min(1e-12)
+
+        # Switch-style balancing objective.  The top-k assignment fraction is
+        # normalized by k, making a perfectly balanced value equal to 1.
+        probabilities = torch.softmax(logits, dim=-1)
+        assignment = F.one_hot(selected_indices, num_classes=self.num_experts).float().sum(dim=-2)
+        assignment = assignment / float(self.top_k)
+        probability_fraction = probabilities.mean(dim=0)
+        token_fraction = assignment.mean(dim=0)
+        auxiliary_loss = self.num_experts * torch.sum(probability_fraction * token_fraction)
+        return logits, selected_indices, selected_weights, auxiliary_loss
+
+
+class FuseGlmFeedForward(nn.Module):
+    """Preserve the native LFM FFN and add a sparse expert sidecar in parallel.
+
+    The native projections remain registered directly as ``w1``, ``w2`` and
+    ``w3``.  Consequently their state-dict keys are identical to an ordinary
+    ``Lfm2ForCausalLM`` checkpoint even after this wrapper is installed.
+    """
+
+    def __init__(
+        self,
+        base_ffn: nn.Module,
+        config: FuseGlmConfig,
+        layer_index: int,
+        *,
+        auxiliary_loss_scale: float,
+    ) -> None:
+        super().__init__()
+        for name in ("w1", "w2", "w3"):
+            if not hasattr(base_ffn, name):
+                raise TypeError(f"Unsupported LFM2 feed_forward module: missing {name}")
+
+        # Preserve original checkpoint paths: feed_forward.w1/w2/w3.
+        self.w1 = base_ffn.w1
+        self.w2 = base_ffn.w2
+        self.w3 = base_ffn.w3
+
+        self.layer_index = int(layer_index)
+        self.num_experts = config.fuse_glm_num_experts
+        self.top_k = config.fuse_glm_top_k
+        self.residual_scale_max = config.fuse_glm_residual_scale_max
+        self.token_gate_threshold = config.fuse_glm_token_gate_threshold
+        self.hard_token_gate_at_eval = config.fuse_glm_hard_token_gate_at_eval
+        self.coding_enabled = config.fuse_glm_coding_enabled
+        self.auxiliary_loss_scale = float(auxiliary_loss_scale)
+
+        self.experts = nn.ModuleList(
+            [
+                FoldedGlmExpert(
+                    config.hidden_size,
+                    config.fuse_glm_expert_intermediate_size,
+                    gate_clamp_max=config.fuse_glm_gate_clamp_max,
+                    up_clamp_min=config.fuse_glm_up_clamp_min,
+                    up_clamp_max=config.fuse_glm_up_clamp_max,
+                    initializer_range=config.initializer_range,
+                )
+                for _ in range(self.num_experts)
+            ]
+        )
+        self.router = TopKFoldedExpertRouter(
+            config.hidden_size,
+            self.num_experts,
+            self.top_k,
+            config.initializer_range,
+        )
+        self.token_gate = nn.Linear(config.hidden_size, 1, bias=True)
+        nn.init.zeros_(self.token_gate.weight)
+        nn.init.constant_(self.token_gate.bias, config.fuse_glm_token_gate_bias)
+
+        # tanh(0) is exactly zero, so a newly constructed fused model computes
+        # the same function as its LFM host while retaining nonzero expert
+        # activations from which this scale can learn.
+        self.raw_residual_scale = nn.Parameter(torch.zeros(()))
+        self.last_router_state: RouterState | None = None
+        self.last_router_diagnostics: RouterDiagnostics | None = None
+
+    @property
+    def residual_scale(self) -> torch.Tensor:
+        return self.residual_scale_max * torch.tanh(self.raw_residual_scale)
+
+    def base_forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        return self.w2(F.silu(self.w1(hidden_states)) * self.w3(hidden_states))
+
+    def _disabled_diagnostics(self, hidden_states: torch.Tensor) -> RouterDiagnostics:
+        scalar_zero = hidden_states.new_zeros(())
+        return RouterDiagnostics(
+            layer_index=self.layer_index,
+            token_count=hidden_states.numel() // hidden_states.shape[-1],
+            active_token_count=0,
+            expert_counts=torch.zeros(self.num_experts, dtype=torch.long, device=hidden_states.device),
+            mean_selected_weights=hidden_states.new_zeros(self.num_experts),
+            router_entropy=scalar_zero,
+            token_gate_mean=scalar_zero,
+            token_gate_active_fraction=scalar_zero,
+            residual_scale=self.residual_scale,
+            auxiliary_loss=scalar_zero,
+            coding_enabled=False,
+        )
+
+    def _dispatch(
+        self,
+        hidden_states: torch.Tensor,
+        selected_indices: torch.Tensor,
+        selected_weights: torch.Tensor,
+    ) -> torch.Tensor:
+        output = torch.zeros_like(hidden_states)
+        for expert_index, expert in enumerate(self.experts):
+            token_positions, route_slots = torch.where(selected_indices == expert_index)
+            if token_positions.numel() == 0:
+                continue
+            expert_input = hidden_states.index_select(0, token_positions)
+            expert_output = expert(expert_input)
+            route_weight = selected_weights[token_positions, route_slots].to(expert_output.dtype).unsqueeze(-1)
+            output = output.index_add(0, token_positions, expert_output * route_weight)
+        return output
+
+    def _make_diagnostics(
+        self,
+        state: RouterState,
+        token_count: int,
+        active_mask: torch.Tensor,
+    ) -> RouterDiagnostics:
+        expert_counts = F.one_hot(state.topk_indices, num_classes=self.num_experts).sum(dim=(0, 1))
+        selected_weight_sums = torch.zeros(
+            self.num_experts,
+            device=state.topk_weights.device,
+            dtype=state.topk_weights.dtype,
+        )
+        selected_weight_sums.scatter_add_(0, state.topk_indices.reshape(-1), state.topk_weights.reshape(-1))
+        mean_selected_weights = selected_weight_sums / expert_counts.clamp_min(1).to(selected_weight_sums.dtype)
+
+        if state.router_logits.shape[0] == 0:
+            entropy = state.router_logits.new_zeros(())
+        else:
+            probabilities = torch.softmax(state.router_logits, dim=-1)
+            entropy = -(probabilities * probabilities.clamp_min(1e-12).log()).sum(dim=-1).mean()
+        gate = state.token_gate
+        return RouterDiagnostics(
+            layer_index=self.layer_index,
+            token_count=token_count,
+            active_token_count=int(state.token_indices.numel()),
+            expert_counts=expert_counts,
+            mean_selected_weights=mean_selected_weights,
+            router_entropy=entropy,
+            token_gate_mean=gate.mean(),
+            token_gate_active_fraction=active_mask.float().mean(),
+            residual_scale=self.residual_scale,
+            auxiliary_loss=state.auxiliary_loss,
+            coding_enabled=True,
+        )
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        base_output = self.base_forward(hidden_states)
+        if not self.coding_enabled:
+            self.last_router_state = None
+            self.last_router_diagnostics = self._disabled_diagnostics(hidden_states)
+            return base_output
+
+        original_shape = hidden_states.shape
+        flat_hidden = hidden_states.reshape(-1, original_shape[-1])
+        token_gate = torch.sigmoid(F.linear(flat_hidden.float(), self.token_gate.weight.float(), self.token_gate.bias.float()))
+        token_gate = token_gate.squeeze(-1)
+        active_mask = token_gate >= self.token_gate_threshold
+
+        # Soft gating is used while training so the token gate itself receives
+        # dense gradients.  Optional hard gating at evaluation actually skips
+        # expert computation for low-confidence non-coding tokens.
+        if self.hard_token_gate_at_eval and not self.training:
+            token_indices = torch.where(active_mask)[0]
+        else:
+            token_indices = torch.arange(flat_hidden.shape[0], device=flat_hidden.device)
+
+        if token_indices.numel() == 0:
+            expert_delta = torch.zeros_like(flat_hidden)
+            auxiliary_loss = flat_hidden.sum() * 0.0
+            state = RouterState(
+                layer_index=self.layer_index,
+                token_indices=token_indices,
+                router_logits=flat_hidden.new_empty((0, self.num_experts), dtype=torch.float32),
+                topk_indices=torch.empty((0, self.top_k), dtype=torch.long, device=flat_hidden.device),
+                topk_weights=flat_hidden.new_empty((0, self.top_k), dtype=torch.float32),
+                token_gate=token_gate,
+                auxiliary_loss=auxiliary_loss,
+            )
+        else:
+            active_hidden = flat_hidden.index_select(0, token_indices)
+            logits, selected_indices, selected_weights, auxiliary_loss = self.router(active_hidden)
+            active_delta = self._dispatch(active_hidden, selected_indices, selected_weights)
+            active_gate = token_gate.index_select(0, token_indices).to(active_delta.dtype).unsqueeze(-1)
+            expert_delta = torch.zeros_like(flat_hidden).index_add(
+                0,
+                token_indices,
+                active_delta * active_gate,
+            )
+            state = RouterState(
+                layer_index=self.layer_index,
+                token_indices=token_indices,
+                router_logits=logits,
+                topk_indices=selected_indices,
+                topk_weights=selected_weights,
+                token_gate=token_gate,
+                auxiliary_loss=auxiliary_loss,
+            )
+
+        self.last_router_state = state
+        self.last_router_diagnostics = self._make_diagnostics(state, flat_hidden.shape[0], active_mask)
+
+        output = base_output + self.residual_scale.to(expert_delta.dtype) * expert_delta.reshape(original_shape)
+        if self.training and self.auxiliary_loss_scale > 0.0:
+            output = _AddAuxiliaryLoss.apply(output, auxiliary_loss * self.auxiliary_loss_scale)
+        return output
+
+
+class FuseGlmForCausalLM(Lfm2ForCausalLM):
+    """LFM2 causal LM augmented with folded GLM coding experts."""
+
+    config_class = FuseGlmConfig
+    _no_split_modules = ["Lfm2DecoderLayer", "FuseGlmFeedForward", "FoldedGlmExpert"]
+    _keep_in_fp32_modules_strict = [
+        "router.proj.weight",
+        "e_score_correction_bias",
+        "token_gate.weight",
+        "token_gate.bias",
+        "raw_residual_scale",
+    ]
+
+    def __init__(self, config: FuseGlmConfig) -> None:
+        if not isinstance(config, FuseGlmConfig):
+            if isinstance(config, Lfm2Config):
+                config = FuseGlmConfig.from_lfm_config(config)
+            else:
+                raise TypeError("config must be FuseGlmConfig or Lfm2Config")
+        super().__init__(config)
+        self._install_fusion_wrappers()
+
+    @torch.no_grad()
+    def _initialize_missing_keys(self, is_quantized: bool) -> None:
+        """Initialize fusion-only keys correctly when loading a base LFM.
+
+        ``from_pretrained`` constructs models under an empty/meta-parameter
+        context.  The generic Transformers initializer handles Linear weights
+        but does not know that our scalar scale must be zero, nor that the
+        token gate needs a zero weight and negative bias.  Capture the names
+        that were absent from the checkpoint, let upstream initialize all
+        ordinary parameters, then repair only those absent fusion parameters.
+        Existing values from a fused checkpoint are never overwritten.
+        """
+
+        missing_names = {
+            name
+            for name, parameter in self.named_parameters()
+            if not getattr(parameter, "_is_hf_initialized", False)
+        }
+        super()._initialize_missing_keys(is_quantized)
+
+        parameters = dict(self.named_parameters())
+        for name in missing_names:
+            parameter = parameters.get(name)
+            if parameter is None:
+                continue
+            if name.endswith("feed_forward.raw_residual_scale"):
+                nn.init.zeros_(parameter)
+            elif name.endswith("feed_forward.token_gate.weight"):
+                nn.init.zeros_(parameter)
+            elif name.endswith("feed_forward.token_gate.bias"):
+                nn.init.constant_(parameter, self.config.fuse_glm_token_gate_bias)
+
+    def _install_fusion_wrappers(self) -> None:
+        layer_indices = self.config.resolved_fuse_glm_layer_indices
+        auxiliary_scale = (
+            self.config.fuse_glm_router_aux_loss_coef / len(layer_indices) if layer_indices else 0.0
+        )
+        selected = set(layer_indices)
+        for layer_index, decoder_layer in enumerate(self.model.layers):
+            if layer_index not in selected:
+                continue
+            if isinstance(decoder_layer.feed_forward, FuseGlmFeedForward):
+                continue
+            decoder_layer.feed_forward = FuseGlmFeedForward(
+                decoder_layer.feed_forward,
+                self.config,
+                layer_index,
+                auxiliary_loss_scale=auxiliary_scale,
+            )
+
+    def fusion_layers(self) -> tuple[FuseGlmFeedForward, ...]:
+        return tuple(
+            layer.feed_forward
+            for layer in self.model.layers
+            if isinstance(layer.feed_forward, FuseGlmFeedForward)
+        )
+
+    def iter_folded_experts(self) -> Iterator[tuple[int, int, FoldedGlmExpert]]:
+        for wrapper in self.fusion_layers():
+            for expert_index, expert in enumerate(wrapper.experts):
+                yield wrapper.layer_index, expert_index, expert
+
+    def get_router_states(self, *, detach: bool = False) -> tuple[RouterState, ...]:
+        states = tuple(
+            wrapper.last_router_state
+            for wrapper in self.fusion_layers()
+            if wrapper.last_router_state is not None
+        )
+        if detach:
+            return tuple(state.detached() for state in states)
+        return states
+
+    def get_router_diagnostics(self, *, detach: bool = True) -> tuple[RouterDiagnostics, ...]:
+        diagnostics = tuple(
+            wrapper.last_router_diagnostics
+            for wrapper in self.fusion_layers()
+            if wrapper.last_router_diagnostics is not None
+        )
+        if detach:
+            return tuple(item.detached() for item in diagnostics)
+        return diagnostics
+
+    def get_router_aux_loss(
+        self,
+        reduction: Literal["mean", "sum", "none"] = "mean",
+        *,
+        detach: bool = False,
+    ) -> torch.Tensor:
+        losses = [state.auxiliary_loss for state in self.get_router_states(detach=detach)]
+        if not losses:
+            return next(self.parameters()).new_zeros(())
+        stacked = torch.stack(losses)
+        if reduction == "none":
+            return stacked
+        if reduction == "sum":
+            return stacked.sum()
+        if reduction == "mean":
+            return stacked.mean()
+        raise ValueError("reduction must be 'mean', 'sum', or 'none'")
+
+    def clear_router_state(self) -> None:
+        for wrapper in self.fusion_layers():
+            wrapper.last_router_state = None
+            wrapper.last_router_diagnostics = None
+
+    def set_coding_enabled(self, enabled: bool = True) -> "FuseGlmForCausalLM":
+        """Enable or bypass every coding-expert branch."""
+
+        for wrapper in self.fusion_layers():
+            wrapper.coding_enabled = bool(enabled)
+        return self
+
+    @property
+    def coding_enabled(self) -> bool:
+        layers = self.fusion_layers()
+        return bool(layers) and all(wrapper.coding_enabled for wrapper in layers)
+
+    @contextmanager
+    def coding_experts(self, enabled: bool = True) -> Iterator["FuseGlmForCausalLM"]:
+        """Temporarily enable/disable coding experts for one local operation."""
+
+        wrappers = self.fusion_layers()
+        previous = tuple(wrapper.coding_enabled for wrapper in wrappers)
+        self.set_coding_enabled(enabled)
+        try:
+            yield self
+        finally:
+            for wrapper, old_value in zip(wrappers, previous):
+                wrapper.coding_enabled = old_value
+
+    @torch.no_grad()
+    def load_folded_expert(
+        self,
+        layer_index: int,
+        expert_index: int,
+        *,
+        gate_proj: torch.Tensor,
+        up_proj: torch.Tensor,
+        down_proj: torch.Tensor,
+    ) -> None:
+        """Copy one folded expert into a concrete decoder/expert slot."""
+
+        if layer_index < 0 or layer_index >= len(self.model.layers):
+            raise IndexError(f"layer_index {layer_index} is outside the decoder")
+        wrapper = self.model.layers[layer_index].feed_forward
+        if not isinstance(wrapper, FuseGlmFeedForward):
+            raise ValueError(f"decoder layer {layer_index} has no fused expert branch")
+        if expert_index < 0 or expert_index >= len(wrapper.experts):
+            raise IndexError(f"expert_index {expert_index} is outside layer {layer_index}")
+        wrapper.experts[expert_index].load_folded_weights(
+            gate_proj=gate_proj,
+            up_proj=up_proj,
+            down_proj=down_proj,
+        )
+
+    @torch.no_grad()
+    def load_router_initializer(
+        self,
+        layer_index: int,
+        *,
+        proj_weight: torch.Tensor,
+        e_score_correction_bias: torch.Tensor,
+    ) -> None:
+        """Load one folded GLM router while preserving post-sigmoid bias semantics."""
+
+        wrappers = {wrapper.layer_index: wrapper for wrapper in self.fusion_layers()}
+        if layer_index not in wrappers:
+            raise KeyError(f"decoder layer {layer_index} has no fused expert branch")
+        router = wrappers[layer_index].router
+        expected_weight_shape = tuple(router.proj.weight.shape)
+        expected_bias_shape = tuple(router.e_score_correction_bias.shape)
+        if tuple(proj_weight.shape) != expected_weight_shape:
+            raise ValueError(
+                f"router proj_weight has shape {tuple(proj_weight.shape)}; "
+                f"expected {expected_weight_shape}"
+            )
+        if tuple(e_score_correction_bias.shape) != expected_bias_shape:
+            raise ValueError(
+                "router e_score_correction_bias has shape "
+                f"{tuple(e_score_correction_bias.shape)}; expected {expected_bias_shape}"
+            )
+        if not bool(torch.isfinite(proj_weight).all()):
+            raise ValueError("router proj_weight contains NaN or infinity")
+        if not bool(torch.isfinite(e_score_correction_bias).all()):
+            raise ValueError("router e_score_correction_bias contains NaN or infinity")
+        router.proj.weight.copy_(
+            proj_weight.to(device=router.proj.weight.device, dtype=router.proj.weight.dtype)
+        )
+        router.e_score_correction_bias.copy_(
+            e_score_correction_bias.to(
+                device=router.e_score_correction_bias.device,
+                dtype=router.e_score_correction_bias.dtype,
+            )
+        )
+
+    @classmethod
+    def from_lfm_pretrained(
+        cls,
+        pretrained_model_name_or_path: str,
+        *model_args: Any,
+        config: Lfm2Config | FuseGlmConfig | None = None,
+        fuse_overrides: dict[str, Any] | None = None,
+        **kwargs: Any,
+    ) -> "FuseGlmForCausalLM":
+        """Load native LFM weights and initialize only the fusion branch.
+
+        The method accepts local directories and normal Hugging Face loading
+        arguments.  It does not set ``trust_remote_code`` or force a network
+        lookup.  Use ``local_files_only=True`` when an offline-only guarantee is
+        desired.
+        """
+
+        overrides = dict(fuse_overrides or {})
+        if config is None:
+            config_keys = (
+                "cache_dir",
+                "force_download",
+                "local_files_only",
+                "revision",
+                "subfolder",
+                "token",
+            )
+            config_kwargs = {key: kwargs[key] for key in config_keys if key in kwargs}
+            config = Lfm2Config.from_pretrained(pretrained_model_name_or_path, **config_kwargs)
+        if not isinstance(config, FuseGlmConfig):
+            config = FuseGlmConfig.from_lfm_config(config, **overrides)
+        elif overrides:
+            config = FuseGlmConfig.from_lfm_config(config, **overrides)
+        return cls.from_pretrained(
+            pretrained_model_name_or_path,
+            *model_args,
+            config=config,
+            **kwargs,
+        )
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Any | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        logits_to_keep: int | torch.Tensor = 0,
+        output_router_diagnostics: bool | None = None,
+        coding_enabled: bool | None = None,
+        **kwargs: Any,
+    ) -> FuseGlmCausalLMOutputWithPast | tuple[torch.Tensor, ...]:
+        """Run the native LFM forward plus the configured coding sidecars.
+
+        Passing ``coding_enabled`` provides a convenient per-call override.
+        For concurrent callers, prefer separate model instances because this
+        override temporarily changes local module flags.
+        """
+
+        wrappers = self.fusion_layers()
+        previous = tuple(wrapper.coding_enabled for wrapper in wrappers)
+        if coding_enabled is not None:
+            self.set_coding_enabled(coding_enabled)
+        try:
+            outputs = super().forward(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                inputs_embeds=inputs_embeds,
+                labels=labels,
+                use_cache=use_cache,
+                logits_to_keep=logits_to_keep,
+                **kwargs,
+            )
+        finally:
+            if coding_enabled is not None:
+                for wrapper, old_value in zip(wrappers, previous):
+                    wrapper.coding_enabled = old_value
+
+        # ``return_dict=False`` is retained for compatibility with upstream.
+        # Router information remains accessible through the getter methods.
+        if not isinstance(outputs, CausalLMOutputWithPast):
+            return outputs
+
+        router_aux_loss = self.get_router_aux_loss(reduction="mean")
+        loss = outputs.loss
+        if loss is not None and self.training and self.config.fuse_glm_router_aux_loss_coef > 0:
+            # Gradients are attached inside each wrapper (checkpoint-safe).
+            # This detached term makes the scalar reported to users equal to
+            # CE + coefficient * mean(auxiliary loss) without double-counting.
+            loss = loss + self.config.fuse_glm_router_aux_loss_coef * router_aux_loss.detach()
+
+        include_diagnostics = (
+            self.config.fuse_glm_output_router_diagnostics
+            if output_router_diagnostics is None
+            else bool(output_router_diagnostics)
+        )
+        diagnostics = self.get_router_diagnostics() if include_diagnostics else None
+        return FuseGlmCausalLMOutputWithPast(
+            loss=loss,
+            logits=outputs.logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+            router_aux_loss=router_aux_loss,
+            router_diagnostics=diagnostics,
+        )
+
+
+__all__ = [
+    "FoldedGlmExpert",
+    "FuseGlmCausalLMOutputWithPast",
+    "FuseGlmFeedForward",
+    "FuseGlmForCausalLM",
+    "RouterDiagnostics",
+    "RouterState",
+    "TopKFoldedExpertRouter",
+]

provenance/assembly.json

@@ -0,0 +1,217 @@
+{
+  "created_at": "2026-08-27T13:15:51.908322+00:00",
+  "output": {
+    "config_sha256": "8582a503ded278e710a929f31848356b8ca95ea76a957b0842d6d5871ad16e1a",
+    "configuration_code_sha256": "922dfb636b84d09d2f92ba0c5bc226d00d7b41999c7e9aef1939ccb022192427",
+    "directory": "<workspace>\\RivetCoder-9B-A4B",
+    "index_sha256": "939b9f2bf0c48523d6a1a2ef93d987876a3e20613b33fe3715bf6b78bbc2a1bb",
+    "linked_weight_files": [
+      {
+        "method": "hardlink",
+        "name": "host-model-00001-of-00002.safetensors",
+        "role": "host_weight",
+        "size": 5329406264,
+        "source": "<workspace>\\models\\LiquidAI\\LFM2.5-2.6B\\model-00001-of-00002.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "host-model-00002-of-00002.safetensors",
+        "role": "host_weight",
+        "size": 65021192,
+        "source": "<workspace>\\models\\LiquidAI\\LFM2.5-2.6B\\model-00002-of-00002.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00001-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764144,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00001.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00002-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764160,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00002.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00003-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065632784,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00003.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00004-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764176,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00004.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00005-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764296,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00005.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00006-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065632912,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00006.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00007-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764280,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00007.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00008-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764296,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00008.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00009-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065632912,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00009.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00010-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764288,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00010.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00011-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 1065764296,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00011.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "expert-model-00012-of-00012.safetensors",
+        "role": "expert_weight",
+        "size": 360716032,
+        "source": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\folded-experts-00012.safetensors"
+      },
+      {
+        "method": "hardlink",
+        "name": "routing-controls.safetensors",
+        "role": "routing_controls",
+        "size": 4191720,
+        "source": "<workspace>\\artifacts\\router-training-qwen-all30-fixed16-top4\\checkpoint-00000060\\trainable.safetensors"
+      }
+    ],
+    "max_shard_bytes": 2139130024,
+    "model_card_sha256": "ec7558e18405570c051065b40d0811fe5635fca0b057825b0c9be3490846f676",
+    "modeling_code_sha256": "2fe578e98265fa3704aa6971d1234784c4e1ef6ebf9a6df6661c831e340d3f33",
+    "physical_duplicate_keys": 0,
+    "physical_tensor_count": 1856,
+    "removed_precedence_sources": [
+      "expert-model-00001-of-00012.safetensors",
+      "expert-model-00002-of-00012.safetensors",
+      "expert-model-00003-of-00012.safetensors",
+      "expert-model-00004-of-00012.safetensors",
+      "expert-model-00005-of-00012.safetensors",
+      "expert-model-00006-of-00012.safetensors",
+      "expert-model-00007-of-00012.safetensors",
+      "expert-model-00008-of-00012.safetensors",
+      "expert-model-00009-of-00012.safetensors",
+      "expert-model-00010-of-00012.safetensors",
+      "expert-model-00011-of-00012.safetensors",
+      "expert-model-00012-of-00012.safetensors",
+      "host-model-00001-of-00002.safetensors",
+      "host-model-00002-of-00002.safetensors",
+      "routing-controls.safetensors"
+    ],
+    "shards": [
+      {
+        "name": "model-00001-of-00009.safetensors",
+        "sha256": "65eb65a6001739bee7959aebe2d6ff564065db69658b768e05ee085f2dbceac4",
+        "size": 2139129752,
+        "tensor_count": 261
+      },
+      {
+        "name": "model-00002-of-00009.safetensors",
+        "sha256": "b689619b9816a94218227afd89ce1c29580b6ec27307c971fabf450e27c6586f",
+        "size": 2139129608,
+        "tensor_count": 260
+      },
+      {
+        "name": "model-00003-of-00009.safetensors",
+        "sha256": "8603121c0ae3e5062526deea09a67f6f17964d8d4018772050aeb368e6767e3d",
+        "size": 2139129848,
+        "tensor_count": 260
+      },
+      {
+        "name": "model-00004-of-00009.safetensors",
+        "sha256": "427ed53be708fbc114263b296a1800cde40ae590b47ac5c65a3afbda5f449fe3",
+        "size": 2139130024,
+        "tensor_count": 261
+      },
+      {
+        "name": "model-00005-of-00009.safetensors",
+        "sha256": "e6d934e3c3709e322cf0b9cba7f5962c31aa50f83fab768b49d2677c08c0dacd",
+        "size": 2139129840,
+        "tensor_count": 260
+      },
+      {
+        "name": "model-00006-of-00009.safetensors",
+        "sha256": "3c2bfeae45226c395ee5a80fc5bb40dcdab0564901746e1e204b2ed1f986097c",
+        "size": 2107698648,
+        "tensor_count": 181
+      },
+      {
+        "name": "model-00007-of-00009.safetensors",
+        "sha256": "1fd608240effcf5db7819f0ec066d938b7f6bbd8a13d48f9bdd8639c217d7069",
+        "size": 2126742608,
+        "tensor_count": 111
+      },
+      {
+        "name": "model-00008-of-00009.safetensors",
+        "sha256": "1b15e4727b4b81c5e9d99ba2537d7cf72c6954c6cb235a5821723b48e4fa0701",
+        "size": 2137241792,
+        "tensor_count": 119
+      },
+      {
+        "name": "model-00009-of-00009.safetensors",
+        "sha256": "dbc94cfe4bca880602ca8bc58283d2e2147501e731cb69ae639bb557d3f315a7",
+        "size": 411078784,
+        "tensor_count": 143
+      }
+    ]
+  },
+  "repo_id": "HCHs/RivetCoder-9B-A4B",
+  "schema": "rivetcoder-hf-package",
+  "schema_version": 1,
+  "sources": {
+    "expert_index_sha256": "c156f43793b0cee11290e32619c767d86eaaf4ca1a6ec268ece9eb8d62b6880c",
+    "host_index_sha256": "1be35d0d99dc56f68e6c5b306f7d4d99b6da97d1ab41ee753832bf1495655c4a",
+    "routing_controls_sha256": "740f15e9bcf68efcb4d9c0f4b2bcfebff66ba199aead9cd2666156dffc60b1ad",
+    "training_summary_sha256": "6cef5c9b40ce3f184f8ab32a58cc2a795dedb220b6dc324c6eb5a1c429613c68"
+  },
+  "weights": {
+    "combined_tensor_count": 1856,
+    "combined_total_size": 17478172784,
+    "expert_shards": 12,
+    "final_shards": 9,
+    "folded_total_size": 12083529600,
+    "host_shards": 2,
+    "host_total_size": 5394397184,
+    "physical_duplicate_keys": 0,
+    "repacked": true,
+    "replaced_warmstart_router_bytes": 3932160,
+    "routing_control_files": 1,
+    "trained_control_bytes": 4178160
+  }
+}

provenance/base-bf16-checksums.sha256

@@ -0,0 +1,38 @@
+34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930  .gitattributes
+032227c2d1dd0a5e8b337834a2959d2bd62893b21b92ee9829df7600caf18e0b  .gitignore
+ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef  chat_template.jinja
+8582a503ded278e710a929f31848356b8ca95ea76a957b0842d6d5871ad16e1a  config.json
+922dfb636b84d09d2f92ba0c5bc226d00d7b41999c7e9aef1939ccb022192427  configuration_fuse_glm.py
+7366b93f26f8830e7a94441a3d2b9f344ceb9e1e31a808e0656df40720075874  generation_config.json
+4d28ca14dedc0b3d0fcc2b3339f0e79931faa33874f3d24f522183a8fc70068c  LICENSE
+30b85b6b9659f2e78aa259f8faf5d920a68dee7c9ced3fa6dba1f19f2bc4fca1  licenses/GLM-MIT.txt
+4d28ca14dedc0b3d0fcc2b3339f0e79931faa33874f3d24f522183a8fc70068c  licenses/LFM-OPEN-LICENSE-v1.0.txt
+65eb65a6001739bee7959aebe2d6ff564065db69658b768e05ee085f2dbceac4  model-00001-of-00009.safetensors
+b689619b9816a94218227afd89ce1c29580b6ec27307c971fabf450e27c6586f  model-00002-of-00009.safetensors
+8603121c0ae3e5062526deea09a67f6f17964d8d4018772050aeb368e6767e3d  model-00003-of-00009.safetensors
+427ed53be708fbc114263b296a1800cde40ae590b47ac5c65a3afbda5f449fe3  model-00004-of-00009.safetensors
+e6d934e3c3709e322cf0b9cba7f5962c31aa50f83fab768b49d2677c08c0dacd  model-00005-of-00009.safetensors
+3c2bfeae45226c395ee5a80fc5bb40dcdab0564901746e1e204b2ed1f986097c  model-00006-of-00009.safetensors
+1fd608240effcf5db7819f0ec066d938b7f6bbd8a13d48f9bdd8639c217d7069  model-00007-of-00009.safetensors
+1b15e4727b4b81c5e9d99ba2537d7cf72c6954c6cb235a5821723b48e4fa0701  model-00008-of-00009.safetensors
+dbc94cfe4bca880602ca8bc58283d2e2147501e731cb69ae639bb557d3f315a7  model-00009-of-00009.safetensors
+939b9f2bf0c48523d6a1a2ef93d987876a3e20613b33fe3715bf6b78bbc2a1bb  model.safetensors.index.json
+2fe578e98265fa3704aa6971d1234784c4e1ef6ebf9a6df6661c831e340d3f33  modeling_fuse_glm.py
+a1829a85639489b9b4cef7dbb520e34c914bdc4bc1a726f68de5714083fe59b0  NOTICE.md
+b7d698b9414c814a1d78dce6a414c3340f3cd0cd949fc2c6b192bd2e35688710  provenance/assembly.json
+92eda133f175fa573b88b92fca5995518965117e534f3b732c63e64862488ff0  provenance/bridge.json
+69ae72f0e5add90269c5e949366c10e87d1de64fc815d14d0f330010f0fd1400  provenance/expert-selection.json
+036c4343e5d62e72613b3bb0ad1238d9f8000088d1118439798ac7ade13d7a2f  provenance/folding.json
+2ba3ae47063c1b4a957b8c5983c48d08c0dacb01c5624f856b8414ac7ccfc2c8  provenance/fusion-plan.json
+a0ab71b19384546ea62e09d0c55eee623f59b92fff5140f1baa883d2386ff12f  provenance/pre-repack-index.json
+f6c13f0c8a28f1029132af53138b91352db35791b5da4eb0748cde4a42c59e51  provenance/router-training.json
+b8bf61e3cd2abfe2ad4b9b59e71ee48c7b680485963f3bc4e3b19df7849a3016  provenance/selected-expert-tensors.json
+ac437e75d524dd58e1c6934ad0838c62e11c9d0954d24873cb17862f958f64a5  provenance/source-models.json
+88fd032686145809f3dee3e8082a7a3b1cffe2874661289b59472a56ed5fcc1c  provenance/training-data-manifest.json
+f426ea62b798bb2340b8d9cfdea2f53f309044dbe9caa061c208e358ddc39ea9  provenance/training-summary.json
+19f75f7117ca582e393d9f79452af231f93339c55dfcc6bebd2b953f87ed3152  provenance/validation-metrics.json
+ec7558e18405570c051065b40d0811fe5635fca0b057825b0c9be3490846f676  README.md
+ee1fa8fc1ed52ab1944fc4b65db3e5eae74e98a92bf0ee6f54dbd5f86daeb4b4  requirements.txt
+695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41  tokenizer.json
+11f1de897317b489dd09199284528382eeb17f566398b16e2039bb2266d26b09  tokenizer_config.json
+2adc48a1d558261d69de8f43f35d8b3dc19bf122b604038cc8c56e5eadaf972c  UPLOAD_INSTRUCTIONS.md

provenance/bridge.json

@@ -0,0 +1,48 @@
+{
+  "contract": {
+    "bias": null,
+    "constraint": "P_out @ P_in = I",
+    "forward": "P_in: host hidden -> donor hidden",
+    "reverse": "P_out = P_in.T: donor hidden -> host hidden",
+    "trainable": false
+  },
+  "method": "identity_hadamard",
+  "output": {
+    "path": "<workspace>\\artifacts\\fixed-bridges\\lfm2048-glm4096-identity-hadamard.safetensors",
+    "sha256": "497cae6c91cc46fe16d4c78a95746eb4bac83d140a5daca5658a053b7b0db729",
+    "size": 67109144,
+    "tensors": {
+      "P_in": {
+        "dtype": "torch.float32",
+        "sha256": "477152e3bcca9a407c222443c0f1e832a21cc44ae500ab35d86a4580293a3cbd",
+        "shape": [
+          4096,
+          2048
+        ],
+        "storage_bytes": 33554432
+      },
+      "P_out": {
+        "dtype": "torch.float32",
+        "sha256": "26c5f05e8f90d54f5d5b2e8fa91d7a846aaf3cde92c66eac79da6e351e27b408",
+        "shape": [
+          2048,
+          4096
+        ],
+        "storage_bytes": 33554432
+      }
+    }
+  },
+  "schema": "lfm-glm-fixed-tied-bridge",
+  "schema_version": 1,
+  "seed": 53042016,
+  "validation": {
+    "column_norm_max": 0.9999999403953552,
+    "column_norm_min": 0.9999999403953552,
+    "compute_device": "cuda",
+    "donor_size": 4096,
+    "gram_atol": 1e-05,
+    "gram_max_absolute_error": 5.960464477539063e-08,
+    "host_size": 2048,
+    "transpose_exact": true
+  }
+}

provenance/checksums.sha256

@@ -0,0 +1,35 @@
+34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930  .gitattributes
+032227c2d1dd0a5e8b337834a2959d2bd62893b21b92ee9829df7600caf18e0b  .gitignore
+f3e412383b66fc807c3d0fbe9e6f8a4a58b2638df55d8380e9cab64af30e8675  chat_template.jinja
+1b441cb88813e8c01f8a9ae98452adfe655ba680e1964c787583634ecb8cb473  config.json
+922dfb636b84d09d2f92ba0c5bc226d00d7b41999c7e9aef1939ccb022192427  configuration_fuse_glm.py
+e34e51ccf5a169f0d9cecf8119e9f719132fa0cf5fae7f8b650893821348e62a  generation_config.json
+4d28ca14dedc0b3d0fcc2b3339f0e79931faa33874f3d24f522183a8fc70068c  LICENSE
+30b85b6b9659f2e78aa259f8faf5d920a68dee7c9ced3fa6dba1f19f2bc4fca1  licenses/GLM-MIT.txt
+4d28ca14dedc0b3d0fcc2b3339f0e79931faa33874f3d24f522183a8fc70068c  licenses/LFM-OPEN-LICENSE-v1.0.txt
+7a3c863c63fb2c52b7f881b483ffc7d1f4b352b2de932b718f62b2b2134de1e8  model-00001-of-00005.safetensors
+e1668117e8a2ad1c7d4ff9a12c1687c9e907a39738dcafd1a48647d38a6efed3  model-00002-of-00005.safetensors
+2f7266ae364fb4bb0c3f30da7deca8133d7501fb415840861de747716ed483ca  model-00003-of-00005.safetensors
+ebb55bc7f9cd7139d3e21c0677f2b2b7f99ec25925837729b0c07a6efa2b0074  model-00004-of-00005.safetensors
+5aec98e4957c1263db1e79f2014f468620ec1b512831256329d8091f762fc61a  model-00005-of-00005.safetensors
+95b79151fed3d2716210fb3046282a47bf73e2016c0ae59f413b59e221e50fc5  model.safetensors.index.json
+2fe578e98265fa3704aa6971d1234784c4e1ef6ebf9a6df6661c831e340d3f33  modeling_fuse_glm.py
+a1829a85639489b9b4cef7dbb520e34c914bdc4bc1a726f68de5714083fe59b0  NOTICE.md
+b7d698b9414c814a1d78dce6a414c3340f3cd0cd949fc2c6b192bd2e35688710  provenance/assembly.json
+4dbb9025d648edd432e0fcbd6cc1f6dafa96e4aa0e0222171923feb17965aa25  provenance/base-bf16-checksums.sha256
+92eda133f175fa573b88b92fca5995518965117e534f3b732c63e64862488ff0  provenance/bridge.json
+69ae72f0e5add90269c5e949366c10e87d1de64fc815d14d0f330010f0fd1400  provenance/expert-selection.json
+036c4343e5d62e72613b3bb0ad1238d9f8000088d1118439798ac7ade13d7a2f  provenance/folding.json
+2ba3ae47063c1b4a957b8c5983c48d08c0dacb01c5624f856b8414ac7ccfc2c8  provenance/fusion-plan.json
+a0ab71b19384546ea62e09d0c55eee623f59b92fff5140f1baa883d2386ff12f  provenance/pre-repack-index.json
+15d69ac2fd857a1749578273f84f1385b004cd1e3fb9204e5d7684cc6c534e8c  provenance/quantization.json
+f6c13f0c8a28f1029132af53138b91352db35791b5da4eb0748cde4a42c59e51  provenance/router-training.json
+b8bf61e3cd2abfe2ad4b9b59e71ee48c7b680485963f3bc4e3b19df7849a3016  provenance/selected-expert-tensors.json
+ac437e75d524dd58e1c6934ad0838c62e11c9d0954d24873cb17862f958f64a5  provenance/source-models.json
+88fd032686145809f3dee3e8082a7a3b1cffe2874661289b59472a56ed5fcc1c  provenance/training-data-manifest.json
+f426ea62b798bb2340b8d9cfdea2f53f309044dbe9caa061c208e358ddc39ea9  provenance/training-summary.json
+19f75f7117ca582e393d9f79452af231f93339c55dfcc6bebd2b953f87ed3152  provenance/validation-metrics.json
+3cc74de6c69b37d403f553e376348e1c1addcbae099f90054a4f95640c6a93e3  README.md
+ce31a666514d30d302472da9387348b4958872f7e307a7ee443ae09f8d1b19b5  requirements.txt
+14c60d6814b7f64c69711d5c5d5561d3de9cc3896feebf9613ae8a8523a3497d  tokenizer_config.json
+695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41  tokenizer.json

provenance/fusion-plan.json

@@ -0,0 +1,738 @@
+{
+  "architecture": {
+    "bridge": {
+      "constraint": "P^T P = I",
+      "fold_down": "W_down_folded = P^T @ W_down_glm",
+      "fold_gate_up": "W_folded = W_glm @ P",
+      "fold_router_init": "R_lfm_init = R_glm[selected_experts] @ P",
+      "forward": "2048->4096",
+      "kind": "tied frozen linear projection",
+      "reverse": "4096->2048"
+    },
+    "candidate_expert_count": 480,
+    "donor": "zai-org/GLM-5.3-Flash",
+    "donor_hidden_size": 4096,
+    "expert_intermediate_size": 2048,
+    "experts_per_layer": 16,
+    "folded_expert_parameter_count": 6039797760,
+    "folded_parameter_count_per_expert": 12582912,
+    "frozen": [
+      "LFM host",
+      "GLM experts",
+      "bridge"
+    ],
+    "host": "LiquidAI/LFM2.5-2.6B",
+    "host_hidden_size": 2048,
+    "maximum_active_experts_per_token": 120,
+    "runtime_top_k": 4,
+    "target_layer_count": 30,
+    "trainable": [
+      "per-layer router",
+      "token gate",
+      "bounded residual scale"
+    ]
+  },
+  "canonical_mapping_projection_sha256": "f02ef1f5152697e814f2fb41fedad3f7ea6738ac29eed1587f1d9948ed7360ac",
+  "mapping_status": "provisional",
+  "mappings": [
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        149,
+        171,
+        186,
+        98,
+        286,
+        189,
+        12,
+        10,
+        202,
+        177,
+        66,
+        24,
+        118,
+        107,
+        220,
+        47
+      ],
+      "source_layer": 3,
+      "target_layer": 0
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        241,
+        243,
+        173,
+        268,
+        39,
+        45,
+        216,
+        197,
+        205,
+        223,
+        89,
+        178,
+        248,
+        93,
+        142,
+        74
+      ],
+      "source_layer": 4,
+      "target_layer": 1
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        160,
+        276,
+        104,
+        234,
+        16,
+        83,
+        209,
+        260,
+        270,
+        282,
+        55,
+        174,
+        236,
+        256,
+        137,
+        206
+      ],
+      "source_layer": 6,
+      "target_layer": 2
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        81,
+        57,
+        131,
+        233,
+        68,
+        129,
+        94,
+        145,
+        208,
+        262,
+        184,
+        48,
+        211,
+        51,
+        44,
+        72
+      ],
+      "source_layer": 7,
+      "target_layer": 3
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        129,
+        142,
+        181,
+        82,
+        74,
+        95,
+        177,
+        88,
+        202,
+        150,
+        6,
+        239,
+        98,
+        271,
+        221,
+        219
+      ],
+      "source_layer": 9,
+      "target_layer": 4
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        96,
+        98,
+        285,
+        75,
+        81,
+        284,
+        111,
+        9,
+        177,
+        268,
+        0,
+        159,
+        70,
+        28,
+        65,
+        144
+      ],
+      "source_layer": 10,
+      "target_layer": 5
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        246,
+        103,
+        109,
+        138,
+        128,
+        69,
+        178,
+        219,
+        28,
+        159,
+        220,
+        66,
+        81,
+        91,
+        166,
+        47
+      ],
+      "source_layer": 11,
+      "target_layer": 6
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        178,
+        145,
+        13,
+        25,
+        75,
+        187,
+        70,
+        169,
+        235,
+        20,
+        142,
+        209,
+        156,
+        52,
+        99,
+        177
+      ],
+      "source_layer": 13,
+      "target_layer": 7
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        143,
+        121,
+        277,
+        228,
+        46,
+        72,
+        285,
+        243,
+        70,
+        180,
+        86,
+        23,
+        150,
+        205,
+        28,
+        275
+      ],
+      "source_layer": 14,
+      "target_layer": 8
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        229,
+        219,
+        23,
+        224,
+        64,
+        244,
+        273,
+        90,
+        66,
+        185,
+        77,
+        237,
+        96,
+        136,
+        226,
+        109
+      ],
+      "source_layer": 16,
+      "target_layer": 9
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        279,
+        172,
+        114,
+        217,
+        9,
+        66,
+        157,
+        180,
+        210,
+        102,
+        150,
+        283,
+        175,
+        89,
+        32,
+        186
+      ],
+      "source_layer": 17,
+      "target_layer": 10
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        157,
+        281,
+        111,
+        228,
+        27,
+        5,
+        23,
+        21,
+        282,
+        261,
+        92,
+        81,
+        271,
+        85,
+        80,
+        136
+      ],
+      "source_layer": 19,
+      "target_layer": 11
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        269,
+        92,
+        42,
+        15,
+        173,
+        216,
+        141,
+        282,
+        29,
+        210,
+        35,
+        110,
+        260,
+        64,
+        245,
+        247
+      ],
+      "source_layer": 20,
+      "target_layer": 12
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        123,
+        165,
+        166,
+        178,
+        5,
+        182,
+        33,
+        37,
+        110,
+        211,
+        186,
+        158,
+        63,
+        106,
+        159,
+        267
+      ],
+      "source_layer": 21,
+      "target_layer": 13
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        269,
+        234,
+        202,
+        229,
+        106,
+        90,
+        67,
+        11,
+        133,
+        152,
+        254,
+        70,
+        219,
+        13,
+        57,
+        10
+      ],
+      "source_layer": 23,
+      "target_layer": 14
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        135,
+        49,
+        193,
+        207,
+        272,
+        25,
+        45,
+        101,
+        177,
+        56,
+        29,
+        23,
+        119,
+        198,
+        35,
+        159
+      ],
+      "source_layer": 24,
+      "target_layer": 15
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        227,
+        49,
+        43,
+        214,
+        245,
+        127,
+        248,
+        150,
+        25,
+        223,
+        246,
+        203,
+        145,
+        57,
+        89,
+        208
+      ],
+      "source_layer": 26,
+      "target_layer": 16
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        134,
+        196,
+        89,
+        66,
+        106,
+        205,
+        60,
+        121,
+        209,
+        224,
+        286,
+        114,
+        26,
+        248,
+        188,
+        194
+      ],
+      "source_layer": 27,
+      "target_layer": 17
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        69,
+        214,
+        198,
+        283,
+        272,
+        56,
+        89,
+        107,
+        60,
+        50,
+        245,
+        61,
+        185,
+        67,
+        97,
+        99
+      ],
+      "source_layer": 28,
+      "target_layer": 18
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        93,
+        139,
+        36,
+        279,
+        29,
+        97,
+        111,
+        64,
+        0,
+        195,
+        242,
+        24,
+        140,
+        101,
+        8,
+        136
+      ],
+      "source_layer": 30,
+      "target_layer": 19
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        147,
+        241,
+        11,
+        251,
+        178,
+        268,
+        118,
+        154,
+        103,
+        248,
+        25,
+        157,
+        197,
+        66,
+        210,
+        98
+      ],
+      "source_layer": 31,
+      "target_layer": 20
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        206,
+        72,
+        170,
+        171,
+        237,
+        8,
+        168,
+        61,
+        172,
+        202,
+        130,
+        222,
+        256,
+        128,
+        226,
+        252
+      ],
+      "source_layer": 33,
+      "target_layer": 21
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        52,
+        258,
+        18,
+        59,
+        80,
+        172,
+        137,
+        40,
+        197,
+        89,
+        35,
+        154,
+        84,
+        19,
+        255,
+        38
+      ],
+      "source_layer": 34,
+      "target_layer": 22
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        220,
+        31,
+        280,
+        106,
+        10,
+        178,
+        234,
+        36,
+        7,
+        212,
+        187,
+        199,
+        221,
+        47,
+        186,
+        147
+      ],
+      "source_layer": 36,
+      "target_layer": 23
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        36,
+        197,
+        33,
+        93,
+        181,
+        134,
+        84,
+        244,
+        264,
+        164,
+        207,
+        202,
+        71,
+        59,
+        214,
+        284
+      ],
+      "source_layer": 37,
+      "target_layer": 24
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        54,
+        143,
+        130,
+        15,
+        14,
+        107,
+        76,
+        225,
+        8,
+        195,
+        281,
+        212,
+        161,
+        177,
+        258,
+        259
+      ],
+      "source_layer": 38,
+      "target_layer": 25
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        5,
+        187,
+        78,
+        105,
+        165,
+        9,
+        156,
+        139,
+        235,
+        121,
+        62,
+        256,
+        14,
+        201,
+        180,
+        277
+      ],
+      "source_layer": 40,
+      "target_layer": 26
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        144,
+        154,
+        125,
+        98,
+        92,
+        123,
+        196,
+        80,
+        141,
+        83,
+        249,
+        250,
+        20,
+        34,
+        134,
+        184
+      ],
+      "source_layer": 41,
+      "target_layer": 27
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        249,
+        148,
+        151,
+        221,
+        18,
+        87,
+        239,
+        105,
+        207,
+        35,
+        147,
+        20,
+        84,
+        130,
+        52,
+        17
+      ],
+      "source_layer": 43,
+      "target_layer": 28
+    },
+    {
+      "expert_count": 16,
+      "expert_ids": [
+        44,
+        212,
+        152,
+        127,
+        244,
+        138,
+        156,
+        266,
+        24,
+        135,
+        113,
+        185,
+        147,
+        186,
+        17,
+        184
+      ],
+      "source_layer": 44,
+      "target_layer": 29
+    }
+  ],
+  "schema": "lfm-glm-fixed16-top4-fusion-plan",
+  "schema_version": 1,
+  "source": {
+    "layer_map": "<workspace>\\manifests\\lfm_to_glm_layer_map.json",
+    "layer_map_method": "monotonic normalized depth over GLM main sparse layers 3..44; replace with CKA/CCA mapping after paired activation collection",
+    "layer_map_sha256": "bd7e4efc18148ff1ebd66f58adfc994b1383a4d56d166cc2b576a973b9c4f4d4",
+    "selection_profile": "<workspace>\\artifacts\\glm-code-expert-ranking\\selection_profile_fixed16_hybrid.json",
+    "selection_profile_sha256": "48ee7d8528836935ceffb256050c647b087c31a801a76de3d9169235c8efc58c"
+  }
+}

provenance/quantization.json

@@ -0,0 +1,54 @@
+{
+  "schema": "rivetcoder-fp8-quantization",
+  "schema_version": 1,
+  "created_at": "2026-08-27T23:01:09+09:00",
+  "source": {
+    "repo_id": "HCHs/RivetCoder-9B-A4B",
+    "revision": "9a90b1917d9b5438e4d2fe1a4f6aea884db59a60"
+  },
+  "target": {
+    "repo_id": "HCHs/RivetCoder-9B-A4B-FP8",
+    "local_directory": "RivetCoder-9B-A4B-FP8"
+  },
+  "method": {
+    "framework": "TorchAO",
+    "config": "Float8DynamicActivationFloat8WeightConfig",
+    "weight_dtype": "float8_e4m3fn",
+    "activation_dtype": "float8_e4m3fn",
+    "activation_scheme": "dynamic",
+    "granularity": "per-tensor",
+    "compatible_linear_modules_only": true
+  },
+  "runtime": {
+    "platform": "Windows 11",
+    "python": "3.12.10",
+    "torch": "2.12.0+cu130",
+    "transformers": "5.16.1",
+    "accelerate": "1.13.0",
+    "safetensors": "0.8.0",
+    "torchao": "0.15.0",
+    "gpu": "NVIDIA GeForce RTX 5070 Ti",
+    "compute_capability": [12, 0]
+  },
+  "results": {
+    "parameter_tensors": 1826,
+    "fp8_tensor_subclass_parameters": 1636,
+    "fp8_parameter_elements": 8475574272,
+    "stored_tensor_bytes": 9000638976,
+    "safetensors_shards": 5,
+    "all_parameters_materialized": true,
+    "all_parameters_on_cuda": true,
+    "clean_reload_success": true,
+    "clean_reload_seconds": 338.0,
+    "clean_reload_fp8_parameters": 1636,
+    "clean_reload_non_cuda_parameters": 0,
+    "resident_cuda_allocated_bytes": 9039455232,
+    "forward_finite": true,
+    "forward_logits_shape": [1, 6, 128000]
+  },
+  "compatibility": {
+    "required_context": "torch.no_grad",
+    "known_incompatible_context": "torch.inference_mode",
+    "known_error": "Cannot set version_counter for inference tensor"
+  }
+}

provenance/router-training.json

@@ -0,0 +1,346 @@
+{
+  "created_at": "2026-08-27T12:25:03Z",
+  "completion": {
+    "actual_global_step": 60,
+    "final_checkpoint": "checkpoint-00000060",
+    "note": "The initial run targeted 20 steps and was resumed with identical semantics to step 60; max_steps is excluded from the run fingerprint."
+  },
+  "data": {
+    "assistant_only_coding_loss": true,
+    "balanced_batching": "equal coding and generic microbatch sizes at every accumulation step",
+    "chat_template_sha256": "ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef",
+    "coding_jsonl": "<workspace>\\data\\router_training_qwen\\coding_train.jsonl",
+    "coding_label_scope": "LFM chat-template assistant tokens only",
+    "coding_sha256": "a70575ac0c1e2977054bc3ba7035c63c4432882c983ab9033fac96884e3164ff",
+    "generic_jsonl": "<workspace>\\data\\router_training_qwen\\generic_control.jsonl",
+    "generic_sha256": "e0dca59cf6419f958711ce6dfe6aef06ca00a2526abaf700e3bf1f1f87d8921b",
+    "tokenizer": "same local LFM tokenizer for coding, generic teacher, and generic student",
+    "tokenizer_json_sha256": "695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41",
+    "validation_jsonl": "<workspace>\\data\\router_training_qwen\\coding_validation.jsonl",
+    "validation_sha256": "7cdd0ab2c64eab93c3d61f47b0aa6868b6b7448ae341bb6d5cc916de60d98d44"
+  },
+  "donor": {
+    "raw_glm_logit_kl": false,
+    "reason": "GLM and LFM vocabularies/tokenizers differ",
+    "repo_id": "zai-org/GLM-5.3-Flash",
+    "revision": "3f1971b7b5f7a528c9c4ef6212c8785298a8c24a"
+  },
+  "folded_experts": {
+    "directory": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard",
+    "frozen": true,
+    "hidden_size": 2048,
+    "index": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard\\model.safetensors.index.json",
+    "index_sha256": "c156f43793b0cee11290e32619c767d86eaaf4ca1a6ec268ece9eb8d62b6880c",
+    "intermediate_size": 2048,
+    "layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29
+    ],
+    "loaded_expert_tensor_count": 1440,
+    "loaded_tensor_count": 1500,
+    "num_experts_per_layer": 16,
+    "provenance_sha256": "6dbfc15a45f773b515a1aa0b2fb2f0b3174697a3873d62865a6dfafcc54c586a",
+    "selected_tensor_storage_bytes": 12079595520,
+    "top_k": 4
+  },
+  "host": {
+    "config_sha256": "480f63fa8e1efa534ae8b92774b3b53b8d6812d62a726e9ecfc866933662f273",
+    "frozen": true,
+    "local_path": "<workspace>\\models\\LiquidAI\\LFM2.5-2.6B",
+    "repo_id": "LiquidAI/LFM2.5-2.6B",
+    "revision": "654f9463ce32b05d0429d76fe1f580b27d4c1ac0"
+  },
+  "initial_expert_off_exactness": {
+    "atol": 0.0,
+    "exact_equal": true,
+    "max_abs_diff": 0.0,
+    "within_tolerance": true
+  },
+  "losses": {
+    "coding": "causal CE on fused LFM logits",
+    "coding_gate": "MSE to coding_gate_target",
+    "generic": "teacher||student KL; teacher is the same frozen LFM with coding_enabled=False",
+    "generic_gate": "mean sigmoid token gate (L1-like sparsity)",
+    "router": "valid-token Switch-style top-k load balance"
+  },
+  "router_initialization": {
+    "correction_semantics": "post-sigmoid correction used for top-k choice only; mixture weights use uncorrected sigmoid scores",
+    "correction_trainable": false,
+    "layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29
+    ],
+    "loaded_before_optimizer": true,
+    "loaded_tensor_count": 60,
+    "mode": "folded GLM router warm-start",
+    "storage_bytes": 3934080,
+    "weight_equation": "R_glm[selected expert rows] @ P_in",
+    "weight_trainable": true
+  },
+  "run_fingerprint": "68f9fa90e18e180f7037dea5e057aa296c95b59267aa09f38285363b5e517c2a",
+  "runtime": {
+    "arguments": {
+      "adam_beta1": 0.9,
+      "adam_beta2": 0.95,
+      "adam_epsilon": 1e-08,
+      "allow_vram_oversubscription": false,
+      "assistant_only_coding_loss": true,
+      "attn_implementation": "sdpa",
+      "batch_size": 1,
+      "coding_ce_weight": 1.0,
+      "coding_gate_target": 0.8,
+      "coding_gate_target_weight": 0.25,
+      "coding_jsonl": "<workspace>\\data\\router_training_qwen\\coding_train.jsonl",
+      "deterministic": true,
+      "device": "cuda",
+      "donor_repo_id": "zai-org/GLM-5.3-Flash",
+      "donor_revision": "3f1971b7b5f7a528c9c4ef6212c8785298a8c24a",
+      "exactness_atol": 0.0,
+      "exactness_every": 5,
+      "expert_cpu_offload": true,
+      "expert_cpu_offload_dtype": "bfloat16",
+      "fail_on_exactness_drift": true,
+      "folded_experts": "<workspace>\\artifacts\\folded-glm-experts-fixed16-top4-identity-hadamard",
+      "fusion_layers": "0-29",
+      "generic_gate_sparsity_weight": 0.1,
+      "generic_jsonl": "<workspace>\\data\\router_training_qwen\\generic_control.jsonl",
+      "generic_kl_weight": 1.0,
+      "gradient_accumulation_steps": 1,
+      "host_model": "<workspace>\\models\\LiquidAI\\LFM2.5-2.6B",
+      "host_repo_id": "LiquidAI/LFM2.5-2.6B",
+      "host_revision": "654f9463ce32b05d0429d76fe1f580b27d4c1ac0",
+      "kl_temperature": 1.0,
+      "kl_vocab_chunk_size": 8192,
+      "learning_rate": 0.0005,
+      "log_every": 1,
+      "max_grad_norm": 1.0,
+      "max_length": 768,
+      "max_steps": 20,
+      "num_experts": 16,
+      "output_dir": "<workspace>\\artifacts\\router-training-qwen-all30-fixed16-top4",
+      "precision": "bfloat16",
+      "residual_scale_learning_rate": 0.05,
+      "resume": null,
+      "router_learning_rate": 0.0005,
+      "router_load_balance_weight": 0.005,
+      "save_every": 5,
+      "seed": 1234,
+      "text_field": "text",
+      "token_gate_learning_rate": 0.001,
+      "top_k": 4,
+      "validation_jsonl": "<workspace>\\data\\router_training_qwen\\coding_validation.jsonl",
+      "vram_max_fraction": 0.85,
+      "warmup_steps": 2,
+      "weight_decay": 0.0
+    },
+    "cuda_available": true,
+    "cuda_device": "NVIDIA GeForce RTX 5070 Ti",
+    "platform": "Windows-11-10.0.26200-SP0",
+    "python": "3.12.10",
+    "versions": {
+      "accelerate": "1.13.0",
+      "huggingface_hub": "1.18.0",
+      "numpy": "2.3.3",
+      "safetensors": "0.8.0",
+      "torch": "2.12.0+cu130",
+      "transformers": "5.16.1"
+    },
+    "vram_preflight": {
+      "configured_limit_bytes": 13402321715,
+      "cuda_free_bytes_before_move": 15767437312,
+      "cuda_total_bytes": 17066033152,
+      "device": "cuda",
+      "estimated_resident_parameter_bytes": 5396486264,
+      "expert_cpu_offload": true,
+      "model_parameter_bytes": 17476081784
+    }
+  },
+  "schema": "fuse-glm-router-training",
+  "schema_version": 1,
+  "trainable": {
+    "allowed_modules": [
+      "router",
+      "token_gate",
+      "raw_residual_scale"
+    ],
+    "parameter_count": 1044540,
+    "parameter_names": [
+      "model.layers.0.feed_forward.raw_residual_scale",
+      "model.layers.0.feed_forward.router.proj.weight",
+      "model.layers.0.feed_forward.token_gate.weight",
+      "model.layers.0.feed_forward.token_gate.bias",
+      "model.layers.1.feed_forward.raw_residual_scale",
+      "model.layers.1.feed_forward.router.proj.weight",
+      "model.layers.1.feed_forward.token_gate.weight",
+      "model.layers.1.feed_forward.token_gate.bias",
+      "model.layers.2.feed_forward.raw_residual_scale",
+      "model.layers.2.feed_forward.router.proj.weight",
+      "model.layers.2.feed_forward.token_gate.weight",
+      "model.layers.2.feed_forward.token_gate.bias",
+      "model.layers.3.feed_forward.raw_residual_scale",
+      "model.layers.3.feed_forward.router.proj.weight",
+      "model.layers.3.feed_forward.token_gate.weight",
+      "model.layers.3.feed_forward.token_gate.bias",
+      "model.layers.4.feed_forward.raw_residual_scale",
+      "model.layers.4.feed_forward.router.proj.weight",
+      "model.layers.4.feed_forward.token_gate.weight",
+      "model.layers.4.feed_forward.token_gate.bias",
+      "model.layers.5.feed_forward.raw_residual_scale",
+      "model.layers.5.feed_forward.router.proj.weight",
+      "model.layers.5.feed_forward.token_gate.weight",
+      "model.layers.5.feed_forward.token_gate.bias",
+      "model.layers.6.feed_forward.raw_residual_scale",
+      "model.layers.6.feed_forward.router.proj.weight",
+      "model.layers.6.feed_forward.token_gate.weight",
+      "model.layers.6.feed_forward.token_gate.bias",
+      "model.layers.7.feed_forward.raw_residual_scale",
+      "model.layers.7.feed_forward.router.proj.weight",
+      "model.layers.7.feed_forward.token_gate.weight",
+      "model.layers.7.feed_forward.token_gate.bias",
+      "model.layers.8.feed_forward.raw_residual_scale",
+      "model.layers.8.feed_forward.router.proj.weight",
+      "model.layers.8.feed_forward.token_gate.weight",
+      "model.layers.8.feed_forward.token_gate.bias",
+      "model.layers.9.feed_forward.raw_residual_scale",
+      "model.layers.9.feed_forward.router.proj.weight",
+      "model.layers.9.feed_forward.token_gate.weight",
+      "model.layers.9.feed_forward.token_gate.bias",
+      "model.layers.10.feed_forward.raw_residual_scale",
+      "model.layers.10.feed_forward.router.proj.weight",
+      "model.layers.10.feed_forward.token_gate.weight",
+      "model.layers.10.feed_forward.token_gate.bias",
+      "model.layers.11.feed_forward.raw_residual_scale",
+      "model.layers.11.feed_forward.router.proj.weight",
+      "model.layers.11.feed_forward.token_gate.weight",
+      "model.layers.11.feed_forward.token_gate.bias",
+      "model.layers.12.feed_forward.raw_residual_scale",
+      "model.layers.12.feed_forward.router.proj.weight",
+      "model.layers.12.feed_forward.token_gate.weight",
+      "model.layers.12.feed_forward.token_gate.bias",
+      "model.layers.13.feed_forward.raw_residual_scale",
+      "model.layers.13.feed_forward.router.proj.weight",
+      "model.layers.13.feed_forward.token_gate.weight",
+      "model.layers.13.feed_forward.token_gate.bias",
+      "model.layers.14.feed_forward.raw_residual_scale",
+      "model.layers.14.feed_forward.router.proj.weight",
+      "model.layers.14.feed_forward.token_gate.weight",
+      "model.layers.14.feed_forward.token_gate.bias",
+      "model.layers.15.feed_forward.raw_residual_scale",
+      "model.layers.15.feed_forward.router.proj.weight",
+      "model.layers.15.feed_forward.token_gate.weight",
+      "model.layers.15.feed_forward.token_gate.bias",
+      "model.layers.16.feed_forward.raw_residual_scale",
+      "model.layers.16.feed_forward.router.proj.weight",
+      "model.layers.16.feed_forward.token_gate.weight",
+      "model.layers.16.feed_forward.token_gate.bias",
+      "model.layers.17.feed_forward.raw_residual_scale",
+      "model.layers.17.feed_forward.router.proj.weight",
+      "model.layers.17.feed_forward.token_gate.weight",
+      "model.layers.17.feed_forward.token_gate.bias",
+      "model.layers.18.feed_forward.raw_residual_scale",
+      "model.layers.18.feed_forward.router.proj.weight",
+      "model.layers.18.feed_forward.token_gate.weight",
+      "model.layers.18.feed_forward.token_gate.bias",
+      "model.layers.19.feed_forward.raw_residual_scale",
+      "model.layers.19.feed_forward.router.proj.weight",
+      "model.layers.19.feed_forward.token_gate.weight",
+      "model.layers.19.feed_forward.token_gate.bias",
+      "model.layers.20.feed_forward.raw_residual_scale",
+      "model.layers.20.feed_forward.router.proj.weight",
+      "model.layers.20.feed_forward.token_gate.weight",
+      "model.layers.20.feed_forward.token_gate.bias",
+      "model.layers.21.feed_forward.raw_residual_scale",
+      "model.layers.21.feed_forward.router.proj.weight",
+      "model.layers.21.feed_forward.token_gate.weight",
+      "model.layers.21.feed_forward.token_gate.bias",
+      "model.layers.22.feed_forward.raw_residual_scale",
+      "model.layers.22.feed_forward.router.proj.weight",
+      "model.layers.22.feed_forward.token_gate.weight",
+      "model.layers.22.feed_forward.token_gate.bias",
+      "model.layers.23.feed_forward.raw_residual_scale",
+      "model.layers.23.feed_forward.router.proj.weight",
+      "model.layers.23.feed_forward.token_gate.weight",
+      "model.layers.23.feed_forward.token_gate.bias",
+      "model.layers.24.feed_forward.raw_residual_scale",
+      "model.layers.24.feed_forward.router.proj.weight",
+      "model.layers.24.feed_forward.token_gate.weight",
+      "model.layers.24.feed_forward.token_gate.bias",
+      "model.layers.25.feed_forward.raw_residual_scale",
+      "model.layers.25.feed_forward.router.proj.weight",
+      "model.layers.25.feed_forward.token_gate.weight",
+      "model.layers.25.feed_forward.token_gate.bias",
+      "model.layers.26.feed_forward.raw_residual_scale",
+      "model.layers.26.feed_forward.router.proj.weight",
+      "model.layers.26.feed_forward.token_gate.weight",
+      "model.layers.26.feed_forward.token_gate.bias",
+      "model.layers.27.feed_forward.raw_residual_scale",
+      "model.layers.27.feed_forward.router.proj.weight",
+      "model.layers.27.feed_forward.token_gate.weight",
+      "model.layers.27.feed_forward.token_gate.bias",
+      "model.layers.28.feed_forward.raw_residual_scale",
+      "model.layers.28.feed_forward.router.proj.weight",
+      "model.layers.28.feed_forward.token_gate.weight",
+      "model.layers.28.feed_forward.token_gate.bias",
+      "model.layers.29.feed_forward.raw_residual_scale",
+      "model.layers.29.feed_forward.router.proj.weight",
+      "model.layers.29.feed_forward.token_gate.weight",
+      "model.layers.29.feed_forward.token_gate.bias"
+    ]
+  }
+}

provenance/source-models.json

@@ -0,0 +1,24 @@
+{
+  "donor": {
+    "license": "MIT",
+    "local_path": "models/zai-org/GLM-5.3-Flash",
+    "repo_id": "zai-org/GLM-5.3-Flash",
+    "revision": "3f1971b7b5f7a528c9c4ef6212c8785298a8c24a",
+    "role": "coding-expert donor and profiling teacher",
+    "snapshot_bytes": 328366171529
+  },
+  "host": {
+    "license": "LFM Open License v1.0",
+    "local_path": "models/LiquidAI/LFM2.5-2.6B",
+    "repo_id": "LiquidAI/LFM2.5-2.6B",
+    "revision": "654f9463ce32b05d0429d76fe1f580b27d4c1ac0",
+    "role": "frozen host model and tokenizer",
+    "snapshot_bytes": 5412392560
+  },
+  "reference": {
+    "repo_id": "Akahsizrr/fuse-1-Lite",
+    "revision": "430c959e47556ae53fed18a9d97f7cf30876e6ff",
+    "role": "architecture reference only"
+  },
+  "schema_version": 1
+}

provenance/training-data-manifest.json

@@ -0,0 +1,99 @@
+{
+  "created_at": "2026-08-27T12:12:57.894412+00:00",
+  "generation": {
+    "base_seed": 20261027,
+    "max_tokens": 640,
+    "model": "qwen/qwen3.8-27b",
+    "reasoning_effort": "none",
+    "repaired_ids": [
+      "qwen-coding-001",
+      "qwen-coding-002",
+      "qwen-coding-003",
+      "qwen-coding-004",
+      "qwen-coding-005",
+      "qwen-coding-020"
+    ],
+    "temperature": 0.3,
+    "top_p": 0.9
+  },
+  "output": {
+    "coding_train": {
+      "path": "data\\router_training_qwen\\coding_train.jsonl",
+      "records": 20,
+      "sha256": "a70575ac0c1e2977054bc3ba7035c63c4432882c983ab9033fac96884e3164ff"
+    },
+    "coding_validation": {
+      "path": "data\\router_training_qwen\\coding_validation.jsonl",
+      "records": 4,
+      "sha256": "7cdd0ab2c64eab93c3d61f47b0aa6868b6b7448ae341bb6d5cc916de60d98d44"
+    },
+    "generic_control": {
+      "path": "data\\router_training_qwen\\generic_control.jsonl",
+      "records": 8,
+      "sha256": "e0dca59cf6419f958711ce6dfe6aef06ca00a2526abaf700e3bf1f1f87d8921b"
+    }
+  },
+  "schema": "qwen-router-training-corpus",
+  "schema_version": 1,
+  "source": {
+    "coding": "data\\teacher_qwen3.8_27b\\coding_calibration.jsonl",
+    "coding_sha256": "4b9cb9bdf18a4abe0c8e08fb45b67d94d77dc432c4a56e6e84daeb5e08290ba8",
+    "generic": "data\\teacher_qwen3.8_27b\\generic_control.jsonl",
+    "generic_sha256": "2ebc8f764048e2fe9a8a3b3528c898f31a3bc622a88c0b97388be26c82129f65"
+  },
+  "split": {
+    "generic_ids": [
+      "qwen-generic-000",
+      "qwen-generic-001",
+      "qwen-generic-002",
+      "qwen-generic-003",
+      "qwen-generic-004",
+      "qwen-generic-005",
+      "qwen-generic-006",
+      "qwen-generic-007"
+    ],
+    "training_ids": [
+      "qwen-coding-000",
+      "qwen-coding-001",
+      "qwen-coding-002",
+      "qwen-coding-003",
+      "qwen-coding-004",
+      "qwen-coding-005",
+      "qwen-coding-007",
+      "qwen-coding-008",
+      "qwen-coding-009",
+      "qwen-coding-010",
+      "qwen-coding-011",
+      "qwen-coding-012",
+      "qwen-coding-014",
+      "qwen-coding-015",
+      "qwen-coding-016",
+      "qwen-coding-017",
+      "qwen-coding-019",
+      "qwen-coding-020",
+      "qwen-coding-021",
+      "qwen-coding-022"
+    ],
+    "validation_ids": [
+      "qwen-coding-006",
+      "qwen-coding-013",
+      "qwen-coding-018",
+      "qwen-coding-023"
+    ]
+  },
+  "teacher": {
+    "arch": "qwen3_5",
+    "capabilities": [
+      "tool_use"
+    ],
+    "compatibility_type": "mlx",
+    "id": "qwen/qwen3.8-27b",
+    "loaded_context_length": 42496,
+    "max_context_length": 262144,
+    "object": "model",
+    "publisher": "qwen",
+    "quantization": "4bit",
+    "state": "loaded",
+    "type": "vlm"
+  }
+}

provenance/training-summary.json

@@ -0,0 +1,1737 @@
+{
+  "completed_at": "2026-08-27T13:04:49Z",
+  "final_checkpoint": "<workspace>\\artifacts\\router-training-qwen-all30-fixed16-top4\\checkpoint-00000060",
+  "final_expert_off_exactness": {
+    "atol": 0.0,
+    "exact_equal": true,
+    "max_abs_diff": 0.0,
+    "within_tolerance": true
+  },
+  "final_metrics": {
+    "coding_sampler_epoch": 2,
+    "cuda": {
+      "allocated_bytes": 5474311680,
+      "max_allocated_bytes": 12469938176,
+      "reserved_bytes": 14224982016
+    },
+    "expert_off_exactness": {
+      "atol": 0.0,
+      "exact_equal": true,
+      "max_abs_diff": 0.0,
+      "within_tolerance": true
+    },
+    "generic_sampler_epoch": 7,
+    "global_step": 60,
+    "grad_norm": 0.015457727015018463,
+    "learning_rate": 0.0005,
+    "learning_rates": {
+      "residual_scale": 0.05,
+      "router": 0.0005,
+      "token_gate": 0.001
+    },
+    "loss": {
+      "coding_ce": 0.3947601616382599,
+      "coding_gate_target": 0.40677282214164734,
+      "generic_gate_sparsity": 0.036865234375,
+      "generic_kl": 0.0010666541056707501,
+      "router_load_balance": 0.9907996654510498,
+      "total": 0.5061666369438171
+    },
+    "router_diagnostics": [
+      {
+        "domain": "coding",
+        "expert_counts": [
+          123,
+          219,
+          189,
+          205,
+          175,
+          180,
+          187,
+          164,
+          101,
+          119,
+          234,
+          170,
+          203,
+          192,
+          163,
+          180
+        ],
+        "layer": 0,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 7.45758370612748e-05,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.058837890625,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          140,
+          179,
+          181,
+          170,
+          138,
+          170,
+          192,
+          253,
+          155,
+          144,
+          213,
+          204,
+          200,
+          206,
+          180,
+          79
+        ],
+        "layer": 1,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.07198415696620941,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.04345703125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          167,
+          138,
+          202,
+          209,
+          134,
+          170,
+          192,
+          264,
+          174,
+          148,
+          136,
+          140,
+          167,
+          160,
+          190,
+          213
+        ],
+        "layer": 2,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.0014509361935779452,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0771484375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          251,
+          237,
+          206,
+          136,
+          146,
+          132,
+          170,
+          158,
+          191,
+          161,
+          0,
+          202,
+          157,
+          210,
+          247,
+          200
+        ],
+        "layer": 3,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.0868370309472084,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0615234375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          252,
+          167,
+          150,
+          191,
+          235,
+          182,
+          134,
+          0,
+          198,
+          244,
+          181,
+          201,
+          112,
+          190,
+          175,
+          192
+        ],
+        "layer": 4,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08832801133394241,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.04736328125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          178,
+          129,
+          217,
+          126,
+          143,
+          192,
+          171,
+          220,
+          189,
+          245,
+          133,
+          391,
+          0,
+          149,
+          186,
+          135
+        ],
+        "layer": 5,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.014818619005382061,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.056884765625,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          128,
+          128,
+          238,
+          194,
+          257,
+          141,
+          125,
+          181,
+          218,
+          278,
+          172,
+          238,
+          0,
+          121,
+          225,
+          160
+        ],
+        "layer": 6,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08805593848228455,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.052001953125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          216,
+          145,
+          148,
+          169,
+          173,
+          194,
+          141,
+          157,
+          186,
+          203,
+          22,
+          277,
+          197,
+          264,
+          149,
+          163
+        ],
+        "layer": 7,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.06830453872680664,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.048095703125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          180,
+          194,
+          248,
+          207,
+          180,
+          276,
+          184,
+          113,
+          248,
+          195,
+          124,
+          134,
+          125,
+          20,
+          132,
+          244
+        ],
+        "layer": 8,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09049388021230698,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.046875,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          346,
+          292,
+          124,
+          226,
+          198,
+          262,
+          148,
+          256,
+          90,
+          34,
+          147,
+          189,
+          0,
+          180,
+          218,
+          94
+        ],
+        "layer": 9,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09548121690750122,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.060302734375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          238,
+          178,
+          190,
+          61,
+          218,
+          152,
+          158,
+          143,
+          253,
+          190,
+          0,
+          191,
+          235,
+          154,
+          406,
+          37
+        ],
+        "layer": 10,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09226511418819427,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.047607421875,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          218,
+          158,
+          16,
+          167,
+          229,
+          176,
+          0,
+          93,
+          157,
+          186,
+          182,
+          203,
+          262,
+          210,
+          338,
+          209
+        ],
+        "layer": 11,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08412390202283859,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.044677734375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          238,
+          195,
+          242,
+          203,
+          303,
+          217,
+          73,
+          0,
+          36,
+          193,
+          144,
+          95,
+          222,
+          341,
+          147,
+          155
+        ],
+        "layer": 12,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09192470461130142,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.047607421875,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          242,
+          176,
+          1,
+          203,
+          176,
+          304,
+          216,
+          165,
+          112,
+          145,
+          130,
+          126,
+          139,
+          201,
+          149,
+          319
+        ],
+        "layer": 13,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.0032708614598959684,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.06396484375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          192,
+          161,
+          272,
+          219,
+          1,
+          219,
+          118,
+          279,
+          197,
+          67,
+          197,
+          85,
+          168,
+          160,
+          267,
+          202
+        ],
+        "layer": 14,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08825033158063889,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.05908203125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          180,
+          106,
+          257,
+          221,
+          243,
+          186,
+          296,
+          167,
+          0,
+          191,
+          157,
+          150,
+          201,
+          122,
+          173,
+          154
+        ],
+        "layer": 15,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.08818262815475464,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.059814453125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          133,
+          188,
+          234,
+          223,
+          141,
+          174,
+          152,
+          165,
+          162,
+          159,
+          119,
+          166,
+          186,
+          216,
+          208,
+          178
+        ],
+        "layer": 16,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.0897134467959404,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0693359375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          211,
+          156,
+          168,
+          183,
+          259,
+          163,
+          154,
+          155,
+          130,
+          164,
+          128,
+          352,
+          215,
+          36,
+          169,
+          161
+        ],
+        "layer": 17,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.0917714461684227,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.16015625,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          163,
+          141,
+          173,
+          157,
+          170,
+          142,
+          247,
+          199,
+          144,
+          127,
+          218,
+          147,
+          199,
+          210,
+          114,
+          253
+        ],
+        "layer": 18,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09746404737234116,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0057061342522501945,
+        "token_gate_mean": 0.1787109375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          140,
+          242,
+          192,
+          123,
+          218,
+          101,
+          138,
+          230,
+          179,
+          265,
+          265,
+          137,
+          81,
+          205,
+          148,
+          140
+        ],
+        "layer": 19,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09797238558530807,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.03423680365085602,
+        "token_gate_mean": 0.2021484375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          103,
+          123,
+          141,
+          152,
+          173,
+          135,
+          244,
+          171,
+          176,
+          146,
+          216,
+          249,
+          129,
+          321,
+          177,
+          148
+        ],
+        "layer": 20,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09263704717159271,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.08131241053342819,
+        "token_gate_mean": 0.234375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          124,
+          268,
+          158,
+          85,
+          192,
+          181,
+          220,
+          101,
+          156,
+          254,
+          191,
+          171,
+          92,
+          148,
+          152,
+          311
+        ],
+        "layer": 21,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09646926075220108,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.28388017416000366,
+        "token_gate_mean": 0.396484375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          175,
+          179,
+          222,
+          147,
+          304,
+          237,
+          182,
+          156,
+          212,
+          214,
+          180,
+          72,
+          144,
+          137,
+          63,
+          180
+        ],
+        "layer": 22,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09666198492050171,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.38944366574287415,
+        "token_gate_mean": 0.439453125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          228,
+          196,
+          196,
+          128,
+          168,
+          106,
+          146,
+          271,
+          140,
+          220,
+          153,
+          135,
+          208,
+          99,
+          193,
+          217
+        ],
+        "layer": 23,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08689017593860626,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.5049929022789001,
+        "token_gate_mean": 0.4921875,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          233,
+          183,
+          123,
+          161,
+          133,
+          251,
+          236,
+          81,
+          290,
+          120,
+          213,
+          134,
+          212,
+          155,
+          125,
+          154
+        ],
+        "layer": 24,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09673594683408737,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.48074179887771606,
+        "token_gate_mean": 0.474609375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          284,
+          185,
+          166,
+          123,
+          115,
+          166,
+          175,
+          136,
+          297,
+          140,
+          278,
+          217,
+          90,
+          200,
+          140,
+          92
+        ],
+        "layer": 25,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09465997666120529,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.48644793033599854,
+        "token_gate_mean": 0.462890625,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          294,
+          195,
+          80,
+          127,
+          89,
+          170,
+          172,
+          139,
+          243,
+          279,
+          145,
+          241,
+          169,
+          106,
+          176,
+          179
+        ],
+        "layer": 26,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09768165647983551,
+        "router_entropy": 2.671875,
+        "token_gate_active_fraction": 0.47931528091430664,
+        "token_gate_mean": 0.4609375,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          118,
+          116,
+          190,
+          217,
+          242,
+          166,
+          184,
+          168,
+          100,
+          208,
+          184,
+          213,
+          138,
+          219,
+          143,
+          198
+        ],
+        "layer": 27,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.0952037125825882,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.5606276988983154,
+        "token_gate_mean": 0.53515625,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          147,
+          258,
+          176,
+          108,
+          151,
+          184,
+          228,
+          93,
+          95,
+          101,
+          217,
+          183,
+          146,
+          307,
+          157,
+          253
+        ],
+        "layer": 28,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09579590708017349,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.8388017416000366,
+        "token_gate_mean": 0.73828125,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "coding",
+        "expert_counts": [
+          208,
+          134,
+          183,
+          56,
+          379,
+          296,
+          181,
+          169,
+          212,
+          73,
+          255,
+          172,
+          80,
+          75,
+          199,
+          132
+        ],
+        "layer": 29,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09640846401453018,
+        "router_entropy": 2.625,
+        "token_gate_active_fraction": 0.9358060359954834,
+        "token_gate_mean": 0.796875,
+        "valid_tokens": 701
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          52,
+          76,
+          54,
+          68,
+          59,
+          76,
+          76,
+          59,
+          40,
+          51,
+          65,
+          49,
+          74,
+          56,
+          49,
+          60
+        ],
+        "layer": 0,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 7.45758370612748e-05,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.034912109375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          46,
+          48,
+          71,
+          59,
+          57,
+          65,
+          64,
+          86,
+          51,
+          48,
+          71,
+          79,
+          66,
+          74,
+          65,
+          14
+        ],
+        "layer": 1,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.07198415696620941,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0242919921875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          70,
+          61,
+          53,
+          52,
+          52,
+          60,
+          68,
+          107,
+          75,
+          15,
+          57,
+          65,
+          55,
+          57,
+          56,
+          61
+        ],
+        "layer": 2,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.0014509361935779452,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0260009765625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          60,
+          94,
+          63,
+          90,
+          41,
+          56,
+          65,
+          59,
+          59,
+          52,
+          0,
+          56,
+          65,
+          58,
+          63,
+          83
+        ],
+        "layer": 3,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.0868370309472084,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0224609375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          63,
+          67,
+          60,
+          67,
+          59,
+          65,
+          66,
+          0,
+          62,
+          76,
+          56,
+          51,
+          63,
+          58,
+          76,
+          75
+        ],
+        "layer": 4,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08832801133394241,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.022216796875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          58,
+          62,
+          64,
+          58,
+          62,
+          75,
+          66,
+          50,
+          61,
+          55,
+          60,
+          114,
+          0,
+          61,
+          62,
+          56
+        ],
+        "layer": 5,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.014818619005382061,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0230712890625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          51,
+          66,
+          72,
+          55,
+          81,
+          53,
+          67,
+          59,
+          67,
+          71,
+          53,
+          99,
+          0,
+          59,
+          52,
+          59
+        ],
+        "layer": 6,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08805593848228455,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0238037109375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          61,
+          53,
+          67,
+          56,
+          63,
+          56,
+          44,
+          95,
+          64,
+          57,
+          22,
+          101,
+          60,
+          59,
+          47,
+          59
+        ],
+        "layer": 7,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.06830453872680664,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0230712890625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          59,
+          69,
+          126,
+          62,
+          58,
+          60,
+          54,
+          51,
+          65,
+          76,
+          46,
+          58,
+          59,
+          9,
+          55,
+          57
+        ],
+        "layer": 8,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09049388021230698,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0233154296875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          69,
+          66,
+          66,
+          72,
+          56,
+          75,
+          73,
+          63,
+          66,
+          6,
+          77,
+          67,
+          1,
+          67,
+          77,
+          63
+        ],
+        "layer": 9,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09548121690750122,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.024658203125,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          60,
+          72,
+          63,
+          38,
+          60,
+          61,
+          57,
+          105,
+          65,
+          56,
+          0,
+          66,
+          66,
+          65,
+          88,
+          42
+        ],
+        "layer": 10,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09226511418819427,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.02197265625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          73,
+          65,
+          51,
+          49,
+          68,
+          68,
+          0,
+          22,
+          50,
+          110,
+          64,
+          75,
+          61,
+          68,
+          74,
+          66
+        ],
+        "layer": 11,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08412390202283859,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0213623046875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          63,
+          55,
+          75,
+          67,
+          66,
+          73,
+          21,
+          0,
+          49,
+          61,
+          63,
+          86,
+          69,
+          86,
+          65,
+          65
+        ],
+        "layer": 12,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09192470461130142,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.023681640625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          76,
+          34,
+          5,
+          44,
+          58,
+          103,
+          64,
+          59,
+          47,
+          50,
+          56,
+          91,
+          64,
+          77,
+          71,
+          65
+        ],
+        "layer": 13,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.0032708614598959684,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0242919921875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          77,
+          50,
+          61,
+          71,
+          31,
+          42,
+          62,
+          57,
+          62,
+          54,
+          58,
+          59,
+          69,
+          67,
+          76,
+          68
+        ],
+        "layer": 14,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08825033158063889,
+        "router_entropy": 2.75,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.022705078125,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          61,
+          65,
+          93,
+          58,
+          58,
+          67,
+          65,
+          60,
+          0,
+          63,
+          38,
+          59,
+          60,
+          65,
+          60,
+          92
+        ],
+        "layer": 15,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.08818262815475464,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0240478515625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          57,
+          94,
+          66,
+          48,
+          51,
+          65,
+          47,
+          66,
+          47,
+          103,
+          54,
+          50,
+          51,
+          59,
+          57,
+          49
+        ],
+        "layer": 16,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.0897134467959404,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0264892578125,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          58,
+          74,
+          41,
+          64,
+          60,
+          64,
+          49,
+          65,
+          48,
+          60,
+          78,
+          65,
+          65,
+          31,
+          72,
+          70
+        ],
+        "layer": 17,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.0917714461684227,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0380859375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          65,
+          51,
+          76,
+          77,
+          61,
+          48,
+          59,
+          61,
+          46,
+          47,
+          55,
+          120,
+          60,
+          34,
+          55,
+          49
+        ],
+        "layer": 18,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09746404737234116,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.035888671875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          49,
+          58,
+          64,
+          38,
+          61,
+          67,
+          79,
+          65,
+          71,
+          52,
+          56,
+          51,
+          53,
+          85,
+          81,
+          34
+        ],
+        "layer": 19,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09797238558530807,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.03857421875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          61,
+          87,
+          68,
+          63,
+          56,
+          61,
+          61,
+          59,
+          58,
+          61,
+          33,
+          75,
+          51,
+          67,
+          44,
+          59
+        ],
+        "layer": 20,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09263704717159271,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.043212890625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          43,
+          92,
+          66,
+          61,
+          53,
+          49,
+          62,
+          61,
+          53,
+          69,
+          73,
+          56,
+          38,
+          61,
+          51,
+          76
+        ],
+        "layer": 21,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09646926075220108,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.04833984375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          64,
+          59,
+          72,
+          63,
+          55,
+          60,
+          61,
+          55,
+          74,
+          61,
+          56,
+          25,
+          62,
+          86,
+          41,
+          70
+        ],
+        "layer": 22,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09666198492050171,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.047607421875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          64,
+          64,
+          54,
+          68,
+          48,
+          62,
+          65,
+          57,
+          63,
+          70,
+          70,
+          41,
+          62,
+          68,
+          50,
+          58
+        ],
+        "layer": 23,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.08689017593860626,
+        "router_entropy": 2.703125,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.04833984375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          53,
+          67,
+          62,
+          65,
+          58,
+          57,
+          59,
+          66,
+          61,
+          52,
+          55,
+          62,
+          57,
+          59,
+          76,
+          55
+        ],
+        "layer": 24,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09673594683408737,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.0322265625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          56,
+          54,
+          63,
+          57,
+          63,
+          58,
+          43,
+          78,
+          54,
+          63,
+          61,
+          56,
+          58,
+          75,
+          53,
+          72
+        ],
+        "layer": 25,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09465997666120529,
+        "router_entropy": 2.71875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.031494140625,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          69,
+          58,
+          63,
+          70,
+          61,
+          51,
+          60,
+          61,
+          62,
+          49,
+          73,
+          59,
+          57,
+          56,
+          52,
+          63
+        ],
+        "layer": 26,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09768165647983551,
+        "router_entropy": 2.671875,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.03125,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          56,
+          60,
+          51,
+          71,
+          72,
+          75,
+          59,
+          46,
+          43,
+          65,
+          60,
+          70,
+          75,
+          50,
+          43,
+          68
+        ],
+        "layer": 27,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.0952037125825882,
+        "router_entropy": 2.734375,
+        "token_gate_active_fraction": 0.0,
+        "token_gate_mean": 0.05029296875,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          49,
+          76,
+          43,
+          71,
+          76,
+          78,
+          56,
+          63,
+          58,
+          51,
+          55,
+          61,
+          75,
+          52,
+          39,
+          61
+        ],
+        "layer": 28,
+        "mean_selected_weight": 0.25,
+        "residual_scale": -0.09579590708017349,
+        "router_entropy": 2.6875,
+        "token_gate_active_fraction": 0.012448133900761604,
+        "token_gate_mean": 0.10693359375,
+        "valid_tokens": 241
+      },
+      {
+        "domain": "generic",
+        "expert_counts": [
+          83,
+          35,
+          60,
+          52,
+          76,
+          74,
+          83,
+          57,
+          73,
+          54,
+          44,
+          63,
+          73,
+          49,
+          39,
+          49
+        ],
+        "layer": 29,
+        "mean_selected_weight": 0.25,
+        "residual_scale": 0.09640846401453018,
+        "router_entropy": 2.640625,
+        "token_gate_active_fraction": 0.0456431545317173,
+        "token_gate_mean": 0.1435546875,
+        "valid_tokens": 241
+      }
+    ],
+    "timestamp": "2026-08-27T13:02:49Z"
+  },
+  "global_step": 60,
+  "run_fingerprint": "68f9fa90e18e180f7037dea5e057aa296c95b59267aa09f38285363b5e517c2a",
+  "validation": {
+    "ce_improvement": 0.0043891161448869065,
+    "finite": true,
+    "fused_ce": 0.6014361901441425,
+    "host_expert_off_ce": 0.6058253062890294,
+    "mean_absolute_residual_scale": 0.07969798704336123,
+    "mean_residual_scale": -0.002938792503376438,
+    "mean_token_gate": 0.19380086263020832,
+    "records": 4,
+    "supervised_tokens": 1417
+  }
+}

provenance/validation-metrics.json

@@ -0,0 +1,11 @@
+{
+  "ce_improvement": 0.0043891161448869065,
+  "finite": true,
+  "fused_ce": 0.6014361901441425,
+  "host_expert_off_ce": 0.6058253062890294,
+  "mean_absolute_residual_scale": 0.07969798704336123,
+  "mean_residual_scale": -0.002938792503376438,
+  "mean_token_gate": 0.19380086263020832,
+  "records": 4,
+  "supervised_tokens": 1417
+}

requirements.txt

@@ -0,0 +1,5 @@
+torch>=2.12.0,<2.13
+transformers>=5.16.1,<5.17
+accelerate>=1.13.0
+safetensors>=0.8.0
+torchao==0.15.0

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41
+size 17905598

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false
+}