Add files using upload-large-folder tool

README.md

@@ -12,7 +12,6 @@ tags:
   - gemma-4
   - vllm
   - fp8
-  - fp8-dynamic
   - compressed-tensors
   - quantization
   - h200
@@ -20,7 +19,6 @@ tags:
   - mixture-of-experts
   - moe
   - inference
-  - production-ready
   - largitdata
 quantized_by: largitdata-inc
 base_model:
@@ -28,107 +26,108 @@ base_model:
 model_type: gemma4
 ---
 
-# Gemma 4 26B-A4B IT FP8 Dynamic Norouter
+# Gemma 4 26B-A4B IT FP8
 
-**Production-ready offline FP8 checkpoint for vLLM — 47% less VRAM, 80% more concurrency vs BF16.**
+Packed-expert offline FP8 checkpoint for [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it), built for vLLM serving on H200-class GPUs.
 
-We searched for a usable offline FP8 checkpoint of Gemma 4 26B-A4B-it but couldn't find one that worked cleanly with vLLM. So we vibe-coded our own and are sharing it with the community.
+This artifact is the final production checkpoint we derived after patching both:
 
-This repository hosts an offline FP8 checkpoint derived from [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) for vLLM serving. No on-the-fly quantization needed at startup.
+- `llmcompressor model_free_ptq`, so packed MoE experts are actually quantized to `FP8`
+- `vLLM Gemma4` loader, so expert `weight_scale` tensors can be loaded correctly
 
-Published by [**Largitdata Inc.**](https://www.largitdata.com/)
+Published by [Largitdata Inc.](https://www.largitdata.com/).
 
-> **Note:** This is a derived operational checkpoint, not an official Google release. The original model's license terms, safety guidance, and documentation remain authoritative.
+> This is a derived operational checkpoint, not an official Google release. The upstream model card, license terms, and safety guidance remain authoritative.
 
-📖 [中文說明 / Chinese Version](#中文說明)
+## Model Details
 
----
+- Base model: [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it)
+- Format: offline `FP8` checkpoint for vLLM
+- Quantization tool: [`llmcompressor`](https://github.com/vllm-project/llm-compressor)
+- Quantization method: `FP8_DYNAMIC`
+- Packed expert quantization: enabled
+- Excluded weights:
+  - `norm`-class 1D tensors
+  - `router.proj`
+- Final checkpoint size: about `26 GB`
+- Weight shards:
+  - `model-00001-of-00002.safetensors`: about `25 GB`
+  - `model-00002-of-00002.safetensors`: about `817 MB`
 
-## Model Details
+## Important Compatibility Note
 
-- **Base model:** [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it)
-- **Derived format:** offline FP8 checkpoint for vLLM
-- **Quantization tool:** [`llmcompressor`](https://github.com/vllm-project/llm-compressor)
-- **Quantization method:** `FP8_DYNAMIC`
-- **Calibration data:** None required (dynamic quantization)
-- **Excluded weights:**
-  - `norm`-class 1D tensors — excluded to avoid `expected 2D linear weight` validation errors during quantization
-  - `re:.*router\.proj$` — MoE router weights excluded to maintain compatibility with the Gemma4 vLLM loading path
-- **Output directory name:** `gemma-4-26B-A4B-it-FP8-DYNAMIC-NOROUTER`
-- **Primary serving target:** `vllm/vllm-openai:gemma4`
-- **Organization:** [Largitdata Inc.](https://www.largitdata.com/)
+This checkpoint requires a patched Gemma4 loader in vLLM to load packed-expert `weight_scale` tensors correctly.
 
-## Test Environment
+The production image we used is:
 
-- **GPU:** `NVIDIA H200 NVL` (`143 GB VRAM`)
-- **Runtime:** `vllm/vllm-openai:gemma4`
-- **KV cache dtype:** `fp8`
-- **`max_model_len`:** `32768`
-- **`gpu_memory_utilization`:** `0.55`
+- `vllm-gemma4:packed-expert-loader-v1`
 
-Observed vLLM startup characteristics:
+If you use an unpatched upstream vLLM image, loading may fail with errors similar to:
+
+```text
+KeyError: 'layers.0.moe.experts.0.down_proj.weight_scale'
+```
 
-- model weight loading: `15.76 s`
-- model loading total: `16.88 s`
-- `torch.compile`: `56.98 s`
-- engine init: `102.17 s`
-- total time to `/v1/models` ready: about `153 s`
+## Tested Environment
 
-Observed runtime capacity:
+- GPU: `NVIDIA H200 NVL` (`143 GB VRAM`)
+- Runtime image: `vllm-gemma4:packed-expert-loader-v1`
+- KV cache dtype: `fp8`
+- Final production serving path: `/models/gemma-4-26B-A4B-it-FP8`
 
+### Production Configuration
+
+- `gpu_memory_utilization = 0.55`
+- `max_model_len = 32768`
 - `max_num_batched_tokens = 8192`
+
+Observed startup and capacity:
+
+- model loading memory: `25.75 GiB`
 - available KV cache memory: `46.37 GiB`
 - GPU KV cache size: `405,184 tokens`
 - maximum concurrency at `32,768` tokens/request: `38.87x`
 
-## Serving Capacity Comparison
+Observed warm single-request benchmark:
 
-| Metric | FP8 Dynamic Norouter | BF16 Baseline |
-|---|---|---|
-| Model loading memory | **25.75 GiB** | 48.5 GiB |
-| GPU KV cache size | **405,184 tokens** | 225,376 tokens |
-| Max concurrency @ 32K tokens/req | **38.87x** | 21.62x |
-| VRAM savings | **47% less** | — |
-| KV cache gain | **80% more** | — |
+- `~1k` prompt: `156.50 tok/s`
+- `~8k` prompt: `136.57 tok/s`
 
-## Basic Benchmark
+### Apples-to-Apples Comparison at `gpu_memory_utilization = 0.75`
 
-Single-request warm benchmark against the OpenAI-compatible vLLM endpoint:
+Same H200, same `max_model_len = 32768`, same benchmark method, same single-request setting:
 
-- prompt tokens: `38`
-- completion tokens: `256`
-- temperature: `0`
-
-| Metric | FP8 Dynamic Norouter | BF16 Baseline |
-|---|---|---|
-| Avg end-to-end latency | 1.629 s | **1.536 s** |
-| Avg completion throughput | 157.19 tok/s | **166.62 tok/s** |
-| Avg total throughput | 180.53 tok/s | **191.36 tok/s** |
-
-These numbers are single-request warm-path measurements, not multi-client throughput tests. In production multi-client scenarios, the FP8 variant's larger KV cache is expected to provide superior aggregate throughput.
-
-**BF16 is ~6% faster on single-request latency, but the FP8 variant uses 47% less VRAM and provides 80% more KV cache capacity.** For production environments serving multiple concurrent users, the FP8 variant offers a better trade-off.
+| Metric | FP8 | BF16 |
+| --- | ---: | ---: |
+| Model loading memory | **25.75 GiB** | 48.5 GiB |
+| Available KV cache memory | **74.33 GiB** | 51.59 GiB |
+| GPU KV cache size | **649,504 tokens** | 225,376 tokens |
+| Max concurrency @ `32k` | **62.31x** | 21.62x |
+| `~1k` prompt decode throughput | 156.28 tok/s | **161.07 tok/s** |
+| `~8k` prompt decode throughput | 136.32 tok/s | **138.01 tok/s** |
 
-### Accuracy Evaluation
+Takeaway:
 
-Formal accuracy benchmarks (MMLU, MT-Bench, etc.) have not yet been conducted on this FP8 checkpoint. Based on prior community findings with FP8 dynamic quantization on similar architectures, accuracy degradation is typically negligible (<0.5% on MMLU). Community contributions with benchmark results are welcome — please open a discussion or PR.
+- `BF16` is still slightly faster on single-request decode speed
+- `FP8` cuts model memory sharply and converts most of that headroom into KV cache
+- for concurrency-oriented serving, the FP8 checkpoint is the better trade-off
 
 ## Usage
 
-Example vLLM launch:
+Example launch command:
 
 ```bash
 docker run -d \
-  --name vllm-gemma4-26b-fp8-norouter \
+  --name vllm-gemma4-26b-fp8 \
   --restart unless-stopped \
   --ipc=host \
   --shm-size 16G \
   --gpus all \
-  -v /models \
+  -v /models:/models \
   -p 8001:8000 \
   -e NVIDIA_VISIBLE_DEVICES=0 \
-  vllm/vllm-openai:gemma4 \
-  --model /models/gemma-4-26B-A4B-it-FP8-DYNAMIC-NOROUTER \
+  vllm-gemma4:packed-expert-loader-v1 \
+  --model /models/gemma-4-26B-A4B-it-FP8 \
   --trust-remote-code \
   --kv-cache-dtype fp8 \
   --gpu-memory-utilization 0.55 \
@@ -141,21 +140,10 @@ docker run -d \
 
 ## Known Limitations
 
-- Single-request latency is ~6% higher than BF16 due to FP8 dequantization overhead.
-- No formal accuracy benchmarks (MMLU, MT-Bench, etc.) have been run yet. Community contributions are welcome.
-- Only tested on NVIDIA H200 NVL. Other GPUs (A100, H100) may require adjusting `gpu-memory-utilization`.
-- MoE router weights (`router.proj`) and `norm`-class 1D tensors are excluded from quantization for vLLM compatibility. No routing degradation has been observed, but systematic evaluation has not been performed.
-
-## Intended Use
-
-This artifact is intended for:
-
-- Operational vLLM deployment on H200-class hardware
-- Reproducible offline FP8 serving experiments
-- Environments where startup-time on-the-fly quantization is undesirable
-- Production inference with higher concurrency requirements
-
-This artifact is not intended to replace the original base model documentation, safety guidance, or license terms.
+- Requires patched vLLM loader support for packed-expert `weight_scale`
+- `BF16` remains slightly faster for single-request decode throughput
+- No formal benchmark suite such as MMLU or MT-Bench has been run yet
+- Tested on `NVIDIA H200 NVL`; other GPUs may need different settings
 
 ## License
 
@@ -163,15 +151,13 @@ This repository contains a derived checkpoint based on [`google/gemma-4-26B-A4B-
 
 ## Citation
 
-If you use this artifact, please cite both the derived checkpoint and the upstream base model.
-
 ```bibtex
-@misc{largitdata_gemma4_26b_a4b_it_fp8_dynamic_norouter_2026,
-  title        = {Gemma 4 26B-A4B IT FP8 Dynamic Norouter},
+@misc{largitdata_gemma4_26b_a4b_it_fp8_2026,
+  title        = {Gemma 4 26B-A4B IT FP8},
   author       = {David Chiu},
   year         = {2026},
-  howpublished = {\url{https://huggingface.co/largitdata-inc/gemma-4-26b-a4b-it-fp8-dynamic-norouter}},
-  note         = {Derived offline FP8 checkpoint from google/gemma-4-26B-A4B-it for vLLM serving, published by Largitdata Inc. \url{https://www.largitdata.com/}}
+  howpublished = {\url{https://huggingface.co/LargitData/gemma-4-26b-a4b-it-fp8}},
+  note         = {Derived offline FP8 packed-expert checkpoint from google/gemma-4-26B-A4B-it for patched vLLM serving}
 }
 
 @misc{google_gemma4_26b_a4b_it,
@@ -182,139 +168,27 @@ If you use this artifact, please cite both the derived checkpoint and the upstre
 }
 ```
 
-## Disclaimer
-
-Users are responsible for verifying license compatibility, downstream serving behavior, numerical quality, and safety characteristics for their own environment.
-
 ---
 
 ## 中文說明
 
-**給 vLLM 用的離線 FP8 checkpoint — 比 BF16 省 47% VRAM，平行處理能力多 80%。**
-
-我們在網路上找了一輪，沒有找到堪用的 Gemma 4 26B 離線 FP8 版本，索性自己 vibe coding 做了一版，貢獻給社群。
-
-這個 Repo 提供從 [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) 衍生出的離線 `FP8` checkpoint，讓 `vLLM` 可以直接載入服務，不需要在啟動時執行 on-the-fly 量化。
-
-由 [**Largitdata Inc.**](https://www.largitdata.com/) 發佈。
-
-> **注意：** 這是衍生的操作用 checkpoint，並非 Google 官方發佈。原始模型的授權條款、安全指引與文件仍以官方為準。
-
-### 模型細節
-
-- **基底模型：** [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it)
-- **格式：** 離線 FP8 checkpoint，供 vLLM 使用
-- **量化工具：** [`llmcompressor`](https://github.com/vllm-project/llm-compressor)
-- **量化方式：** `FP8_DYNAMIC`
-- **校準資料：** 不需要（動態量化）
-- **排除的權重：**
-  - `norm` 類一維 tensor — 避免量化驗證時產生 `expected 2D linear weight` 類錯誤
-  - `re:.*router\.proj$` — MoE router 權重，維持與 Gemma4 vLLM 載入路徑的相容性
-- **主要部署目標：** `vllm/vllm-openai:gemma4`
-
-### 測試環境
-
-- **GPU：** `NVIDIA H200 NVL`（`143 GB VRAM`）
-- **Runtime：** `vllm/vllm-openai:gemma4`
-- **KV cache dtype：** `fp8`
-- **`max_model_len`：** `32768`
-- **`gpu_memory_utilization`：** `0.55`
-
-啟動實測數據：
-
-- 模型權重載入：`15.76 s`
-- 模型載入總計：`16.88 s`
-- `torch.compile`：`56.98 s`
-- 引擎初始化：`102.17 s`
-- `/v1/models` 就緒總時間：約 `153 s`
-
-執行期容量：
-
-- `max_num_batched_tokens = 8192`
-- 可用 KV cache 記憶體：`46.37 GiB`
-- GPU KV cache 大小：`405,184 tokens`
-- 最大平行處理量（`32,768` tokens/request）：`38.87x`
-
-### 服務容量比較
-
-| 指標 | FP8 Dynamic Norouter | BF16 原版 |
-|---|---|---|
-| 模型載入記憶體 | **25.75 GiB** | 48.5 GiB |
-| GPU KV cache 大小 | **405,184 tokens** | 225,376 tokens |
-| 最大平行處理量 @ 32K tokens/req | **38.87x** | 21.62x |
-| VRAM 節省 | **47%** | — |
-| KV cache 增加 | **80%** | — |
-
-### 基礎效能測試
-
-單請求暖機測試（OpenAI 相容 vLLM endpoint）：
-
-- prompt tokens：`38`
-- completion tokens：`256`
-- temperature：`0`
-
-| 指標 | FP8 Dynamic Norouter | BF16 原版 |
-|---|---|---|
-| 平均端到端延遲 | 1.629 s | **1.536 s** |
-| 平均 completion 吞吐量 | 157.19 tok/s | **166.62 tok/s** |
-| 平均總吞吐量 | 180.53 tok/s | **191.36 tok/s** |
-
-以上為單請求暖機路徑測量值，非多用戶吞吐量測試。在生產環境多用戶場景下，FP8 版本更大的 KV cache 預期能提供更好的整體吞吐量。
-
-**結論**：`BF16` 單請求略快（約 6%），但 FP8 版本 VRAM 用量減少 47%，可用 KV cache 增加 80%。需要同時服務多用戶的生產環境，FP8 版本更具優勢。
-
-### 精度評估
-
-尚未對此 FP8 checkpoint 進行正式精度 benchmark（MMLU、MT-Bench 等）。根據社群先前在類似架構上使用 FP8 動態量化的經驗，精度下降通常可忽略（MMLU < 0.5%）。歡迎社群貢獻 benchmark 結果，請開 discussion 或提交 PR。
-
-### 使用方式
-
-vLLM 啟動範例：
-
-```bash
-docker run -d \
-  --name vllm-gemma4-26b-fp8-norouter \
-  --restart unless-stopped \
-  --ipc=host \
-  --shm-size 16G \
-  --gpus all \
-  -v /models \
-  -p 8001:8000 \
-  -e NVIDIA_VISIBLE_DEVICES=0 \
-  vllm/vllm-openai:gemma4 \
-  --model /models/gemma-4-26B-A4B-it-FP8-DYNAMIC-NOROUTER \
-  --trust-remote-code \
-  --kv-cache-dtype fp8 \
-  --gpu-memory-utilization 0.55 \
-  --max-model-len 32768 \
-  --enable-auto-tool-choice \
-  --tool-call-parser gemma4 \
-  --host 0.0.0.0 \
-  --port 8000
-```
-
-### 已知限制
-
-- 單請求延遲比 BF16 高約 6%，主因為 FP8 dequantization 的額外開銷
-- 尚未進行 MMLU / MT-Bench 等精度 benchmark（歡迎社群補充）
-- 僅在 H200 NVL 上實測，其他 GPU（如 A100、H100）可能需要調整 `gpu-memory-utilization`
-- MoE router 權重（`router.proj`）與 `norm` 類一維 tensor 被排除在量化範圍外以維持 vLLM 相容性，目前未觀察到分流品質下降，但尚無系統性評估
-
-### 使用場景
-
-此 checkpoint 適用於：
-
-- 在 H200 等級硬體上以 vLLM 進行生產部署
-- 可重現的離線 FP8 服務實驗
-- 不希望在啟動時執行 on-the-fly 量化的環境
-- 需要更高平行處理能力的生產推論場景
+這個 Repo 提供從 [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) 衍生出的最終版離線 `FP8` checkpoint。這一版不是早期的 `Dynamic Norouter` 中間產物，而是：
 
-此 checkpoint 不取��原始基底模型的文件、安全指引或授權條款。
+- `llmcompressor` 已補成可量化 packed MoE experts
+- `vLLM Gemma4` loader 已補成可讀取 expert `weight_scale`
 
-### 授權
+### 重點
 
-此 Repo 包含基於 [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) 的衍生 checkpoint，使用須遵守 [Gemma 使用條款](https://ai.google.dev/gemma/terms)。
+- 最終 checkpoint 大小：約 `26 GB`
+- 模型載入顯存：約 `25.75 GiB`
+- `gpu_memory_utilization=0.55` 時：
+  - KV cache：`46.37 GiB`
+  - GPU KV cache：`405,184 tokens`
+  - `32k` concurrency：`38.87x`
+- `gpu_memory_utilization=0.75` 時，和 BF16 同條件相比：
+  - `FP8` 單請求速度略慢一點
+  - 但 KV cache 明顯更大，`32k` concurrency 由 `21.62x` 提升到 `62.31x`
 
-### 免責聲明
+### 使用限制
 
-使用者需自行驗證授權相容性、下游服務行為、數值品質與安全特性。
+這份 checkpoint 需要 patched `vLLM` loader。若直接使用未修補的 upstream `vLLM`，可能會在載入時遇到 expert `weight_scale` 相關錯誤。

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb3c810035ce85853a1be7dd348f2f73d1417959d623a882e51522de7b1fdf1
+size 26305626460

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5efcdd24651d5061d49178ba88b86dac518729d25e84f1879d15570852806c6
+size 856521512

model.safetensors.index.json

@@ -1,12 +1,14 @@
 {
   "metadata": {
-    "total_size": 49967520412
+    "total_size": 27161975452
   },
   "weight_map": {
     "model.embed_vision.embedding_projection.weight": "model-00001-of-00002.safetensors",
     "model.language_model.embed_tokens.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.0.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.0.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.0.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.0.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.0.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -35,7 +37,9 @@
     "model.language_model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.0.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.1.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.1.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.1.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.1.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.1.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -64,7 +68,9 @@
     "model.language_model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.1.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.10.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.10.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.10.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.10.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.10.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -93,7 +99,9 @@
     "model.language_model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.10.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.11.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.11.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.11.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.11.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.11.input_layernorm.weight": "model-00002-of-00002.safetensors",
     "model.language_model.layers.11.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.11.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
@@ -120,7 +128,9 @@
     "model.language_model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.11.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.12.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.12.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.12.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.12.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.12.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -149,7 +159,9 @@
     "model.language_model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.12.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.13.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.13.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.13.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.13.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.13.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -178,7 +190,9 @@
     "model.language_model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.13.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.14.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.14.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.14.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.14.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.14.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -207,7 +221,9 @@
     "model.language_model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.14.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.15.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.15.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.15.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.15.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.15.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -236,7 +252,9 @@
     "model.language_model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.15.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.16.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.16.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.16.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.16.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.16.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -265,7 +283,9 @@
     "model.language_model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.16.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.17.experts.down_proj": "model-00002-of-00002.safetensors",
+    "model.language_model.layers.17.experts.down_proj.weight_scale": "model-00002-of-00002.safetensors",
     "model.language_model.layers.17.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.17.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.17.input_layernorm.weight": "model-00002-of-00002.safetensors",
     "model.language_model.layers.17.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.17.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
@@ -292,7 +312,9 @@
     "model.language_model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.17.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.18.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.18.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.18.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.18.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.18.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -321,7 +343,9 @@
     "model.language_model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.18.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.19.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.19.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.19.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.19.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.19.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -350,7 +374,9 @@
     "model.language_model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.19.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.2.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.2.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.2.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.2.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.2.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -379,7 +405,9 @@
     "model.language_model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.2.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.20.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.20.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.20.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.20.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.20.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -408,7 +436,9 @@
     "model.language_model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.20.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.21.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.21.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.21.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.21.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.21.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -437,7 +467,9 @@
     "model.language_model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.21.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.22.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.22.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.22.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.22.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.22.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -466,7 +498,9 @@
     "model.language_model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.22.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.23.experts.down_proj": "model-00002-of-00002.safetensors",
+    "model.language_model.layers.23.experts.down_proj.weight_scale": "model-00002-of-00002.safetensors",
     "model.language_model.layers.23.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.23.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
     "model.language_model.layers.23.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
@@ -493,7 +527,9 @@
     "model.language_model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.23.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.24.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.24.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.24.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.24.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.24.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -522,7 +558,9 @@
     "model.language_model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.24.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.25.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.25.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.25.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.25.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.25.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -551,7 +589,9 @@
     "model.language_model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.25.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.26.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.26.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.26.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.26.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.26.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -580,7 +620,9 @@
     "model.language_model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.26.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.27.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.27.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.27.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.27.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.27.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -609,7 +651,9 @@
     "model.language_model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.27.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.28.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.28.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.28.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.28.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.28.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.28.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -638,7 +682,9 @@
     "model.language_model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.28.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.29.experts.down_proj": "model-00002-of-00002.safetensors",
+    "model.language_model.layers.29.experts.down_proj.weight_scale": "model-00002-of-00002.safetensors",
     "model.language_model.layers.29.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.29.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
     "model.language_model.layers.29.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.29.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
@@ -665,7 +711,9 @@
     "model.language_model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.29.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.3.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.3.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.3.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.3.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.3.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -694,7 +742,9 @@
     "model.language_model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.3.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.4.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.4.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.4.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.4.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.4.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -723,7 +773,9 @@
     "model.language_model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.4.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.5.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.5.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.5.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.5.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.5.input_layernorm.weight": "model-00002-of-00002.safetensors",
     "model.language_model.layers.5.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.5.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
@@ -750,7 +802,9 @@
     "model.language_model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.5.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.6.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.6.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.6.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.6.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.6.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -779,7 +833,9 @@
     "model.language_model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.6.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.7.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.7.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.7.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.7.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.7.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -808,7 +864,9 @@
     "model.language_model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.7.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.8.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.8.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.8.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.8.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.8.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
@@ -837,7 +895,9 @@
     "model.language_model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.8.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.9.experts.down_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.9.experts.down_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.9.experts.gate_up_proj": "model-00001-of-00002.safetensors",
+    "model.language_model.layers.9.experts.gate_up_proj.weight_scale": "model-00001-of-00002.safetensors",
     "model.language_model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
     "model.language_model.layers.9.layer_scalar": "model-00001-of-00002.safetensors",
     "model.language_model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",