Hugging Face's logo

sam33339999
/
Qwen3.8-27B-Coder390-dflash2

Safetensors
qwen3
exl3
dflash2
speculative-decoding
qwen3.8
exllamav3
5-bit
Model card Files
xet
 Community
sam33339999 commited on
Commit
6ca42f8
·
verified ·
1 Parent(s): 5126eb7

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. README.md +67 -0
  2. config.json +114 -0
  3. manifest.json +20 -0
  4. model.safetensors +3 -0
  5. quantization_config.json +1693 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2
4
+ base_model_relation: quantized
5
+ tags:
6
+ - exl3
7
+ - dflash2
8
+ - speculative-decoding
9
+ - qwen3.8
10
+ - exllamav3
11
+ ---
12
+
13
+ # Qwen3.8-27B Coder390 — DFlash2 EXL3 5.0 bpw
14
+
15
+ 這不是一個可以單獨聊天的模型。它是 [nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2](https://huggingface.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2) 附帶的 DFlash2 草稿，轉成 EXL3 之後的版本，用來配合同一次轉換的 `EXL3-3.5bpw` 目標模型。
16
+
17
+ 原始 repo 把 FP8 草稿放在 `BF16/DFlash2-FP8/`。原始卡上的速度（大約 180 tok/s、接受長度大約 5）是在 SGLang 上量那個 FP8 草稿，不是這份 EXL3。
18
+
19
+ ## 量化
20
+
21
+ 2026-10-09，exllamav3 1.4.2。載入時先把 FP8 E4M3、128×128 block scale 還原，再壓成 EXL3。DFlash2 走的是不使用校正 Hessian 的量化路徑，36 個有量化標記的線性層都是 5.0 bpw。目錄大約 1.37 GiB。
22
+
23
+ 和原來的 FP8 草稿不同：原來的 q/k/v 和 `fc` 留在 BF16，這份 EXL3 把它們也壓了。下面這些維持未量化：
24
+
25
+ - 各層 `attention_conv` / `mlp_conv` 的 `base_kernel` 和 `kernel_projection`
26
+ - `candidate_selector` 的 codebook 和 `hidden_projection`
27
+ - norm
28
+
29
+ 架構是 `DFlash2DraftModel`，5 層 sliding attention，hidden 5120。`dflash_config.block_size` 是 8，這個引擎預設一次草稿 7 個 token（block size 減 1）。抽取目標模型第 5、19、33、47、61 層的輸出。
30
+
31
+ ## 怎麼載入
32
+
33
+ 草稿和目標一次只用一種加速。開了這個目錄就不要再加 MTP。
34
+
35
+ exllamav3：
36
+
37
+ ```bash
38
+ python -u tools/serve_openai.py \
39
+ -m /path/to/EXL3-3.5bpw \
40
+ -dm /path/to/DFlash2-EXL3-5.0bpw \
41
+ -cs 262144 \
42
+ --host 127.0.0.1 \
43
+ --port 5000
44
+ ```
45
+
46
+ TabbyAPI：
47
+
48
+ ```yaml
49
+ model:
50
+ model_dir: /path/to/parent
51
+ model_name: EXL3-3.5bpw
52
+ backend: exllamav3
53
+ max_seq_len: 262144
54
+ cache_size: 262144
55
+
56
+ draft_model:
57
+ draft_mode: model
58
+ draft_model_dir: /path/to/parent
59
+ draft_model_name: DFlash2-EXL3-5.0bpw
60
+ draft_num_tokens: 7
61
+ ```
62
+
63
+ DFlash2 是這個 exllamav3 fork 的架構。TabbyAPI 若裝的是不含 `DFlash2DraftModel` 的上游套件，會載入失敗。
64
+
65
+ ## 授權
66
+
67
+ Apache-2.0，沿用[原始模型](https://huggingface.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2)。
config.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DFlash2DraftModel"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": null,
8
+ "is_causal": false,
9
+ "dflash_config": {
10
+ "block_size": 8,
11
+ "conv_group_size": 16,
12
+ "conv_kernel_size": 2,
13
+ "mask_token_id": 248070,
14
+ "selector_rank": 256,
15
+ "selector_top_k": 16,
16
+ "target_layer_ids": [
17
+ 5,
18
+ 19,
19
+ 33,
20
+ 47,
21
+ 61
22
+ ]
23
+ },
24
+ "dtype": "bfloat16",
25
+ "eos_token_id": 248044,
26
+ "head_dim": 128,
27
+ "hidden_act": "silu",
28
+ "hidden_size": 5120,
29
+ "initializer_range": 0.02,
30
+ "intermediate_size": 17408,
31
+ "layer_types": [
32
+ "sliding_attention",
33
+ "sliding_attention",
34
+ "sliding_attention",
35
+ "sliding_attention",
36
+ "sliding_attention"
37
+ ],
38
+ "max_position_embeddings": 262144,
39
+ "max_window_layers": 5,
40
+ "model_type": "qwen3",
41
+ "num_attention_heads": 32,
42
+ "num_hidden_layers": 5,
43
+ "num_key_value_heads": 8,
44
+ "num_target_layers": 64,
45
+ "pad_token_id": 248044,
46
+ "rms_norm_eps": 1e-06,
47
+ "rope_parameters": {
48
+ "rope_theta": 10000000,
49
+ "rope_type": "default"
50
+ },
51
+ "sliding_window": 2048,
52
+ "tie_word_embeddings": false,
53
+ "transformers_version": "5.15.0",
54
+ "use_cache": true,
55
+ "use_sliding_window": true,
56
+ "vocab_size": 248320,
57
+ "quantization_config": {
58
+ "quant_method": "exl3",
59
+ "version": "1.4.2",
60
+ "bits": 5.0,
61
+ "head_bits": 6,
62
+ "calibration": {
63
+ "rows": 250,
64
+ "cols": 2048
65
+ },
66
+ "out_scales": "always",
67
+ "codebook": "mul1",
68
+ "original_quantization_config": {
69
+ "activation_scheme": "dynamic",
70
+ "dequantize": false,
71
+ "fmt": "e4m3",
72
+ "quant_method": "fp8",
73
+ "scale_fmt": "float",
74
+ "weight_block_size": [
75
+ 128,
76
+ 128
77
+ ],
78
+ "modules_to_not_convert": [
79
+ "q_proj",
80
+ "k_proj",
81
+ "v_proj",
82
+ "qkv_proj",
83
+ "fc",
84
+ "lm_head",
85
+ "layers.0.self_attn.q_proj",
86
+ "layers.0.self_attn.k_proj",
87
+ "layers.0.self_attn.v_proj",
88
+ "layers.0.attention_conv.kernel_projection",
89
+ "layers.0.mlp_conv.kernel_projection",
90
+ "layers.1.self_attn.q_proj",
91
+ "layers.1.self_attn.k_proj",
92
+ "layers.1.self_attn.v_proj",
93
+ "layers.1.attention_conv.kernel_projection",
94
+ "layers.1.mlp_conv.kernel_projection",
95
+ "layers.2.self_attn.q_proj",
96
+ "layers.2.self_attn.k_proj",
97
+ "layers.2.self_attn.v_proj",
98
+ "layers.2.attention_conv.kernel_projection",
99
+ "layers.2.mlp_conv.kernel_projection",
100
+ "layers.3.self_attn.q_proj",
101
+ "layers.3.self_attn.k_proj",
102
+ "layers.3.self_attn.v_proj",
103
+ "layers.3.attention_conv.kernel_projection",
104
+ "layers.3.mlp_conv.kernel_projection",
105
+ "layers.4.self_attn.q_proj",
106
+ "layers.4.self_attn.k_proj",
107
+ "layers.4.self_attn.v_proj",
108
+ "layers.4.attention_conv.kernel_projection",
109
+ "layers.4.mlp_conv.kernel_projection",
110
+ "candidate_selector.hidden_projection"
111
+ ]
112
+ }
113
+ }
114
+ }
manifest.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "artifact_role": "DFlash2 speculative draft model",
4
+ "artifact_format": "exllamav3 EXL3 safetensors",
5
+ "precision": "EXL3 5.0 bpw on quantizable linears; conv kernels, selector, and norms left unquantized",
6
+ "source": {
7
+ "model": "nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2",
8
+ "path": "BF16/DFlash2-FP8",
9
+ "note": "FP8 E4M3 128x128 block scales were dequantized on load, then requantized to EXL3. The FP8 manifest that used to sit in this directory described that source, not these weights."
10
+ },
11
+ "runtime_contract": {
12
+ "engine": "exllamav3",
13
+ "architecture": "DFlash2DraftModel",
14
+ "algorithm": "DFLASH",
15
+ "draft_tokens": 7,
16
+ "block_size": 8,
17
+ "target_layer_ids": [5, 19, 33, 47, 61]
18
+ },
19
+ "paired_target": "EXL3-3.5bpw from the same BF16 checkpoint"
20
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0375708821d585cd0d0f89f1f1e07cd38a9c052b0075abcb17937ee169914494
3
+ size 1470916078
quantization_config.json ADDED
@@ -0,0 +1,1693 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "quant_method": "exl3",
3
+ "version": "1.4.2",
4
+ "bits": 5.0,
5
+ "head_bits": 6,
6
+ "calibration": {
7
+ "rows": 250,
8
+ "cols": 2048
9
+ },
10
+ "out_scales": "always",
11
+ "codebook": "mul1",
12
+ "original_quantization_config": {
13
+ "activation_scheme": "dynamic",
14
+ "dequantize": false,
15
+ "fmt": "e4m3",
16
+ "quant_method": "fp8",
17
+ "scale_fmt": "float",
18
+ "weight_block_size": [
19
+ 128,
20
+ 128
21
+ ],
22
+ "modules_to_not_convert": [
23
+ "q_proj",
24
+ "k_proj",
25
+ "v_proj",
26
+ "qkv_proj",
27
+ "fc",
28
+ "lm_head",
29
+ "layers.0.self_attn.q_proj",
30
+ "layers.0.self_attn.k_proj",
31
+ "layers.0.self_attn.v_proj",
32
+ "layers.0.attention_conv.kernel_projection",
33
+ "layers.0.mlp_conv.kernel_projection",
34
+ "layers.1.self_attn.q_proj",
35
+ "layers.1.self_attn.k_proj",
36
+ "layers.1.self_attn.v_proj",
37
+ "layers.1.attention_conv.kernel_projection",
38
+ "layers.1.mlp_conv.kernel_projection",
39
+ "layers.2.self_attn.q_proj",
40
+ "layers.2.self_attn.k_proj",
41
+ "layers.2.self_attn.v_proj",
42
+ "layers.2.attention_conv.kernel_projection",
43
+ "layers.2.mlp_conv.kernel_projection",
44
+ "layers.3.self_attn.q_proj",
45
+ "layers.3.self_attn.k_proj",
46
+ "layers.3.self_attn.v_proj",
47
+ "layers.3.attention_conv.kernel_projection",
48
+ "layers.3.mlp_conv.kernel_projection",
49
+ "layers.4.self_attn.q_proj",
50
+ "layers.4.self_attn.k_proj",
51
+ "layers.4.self_attn.v_proj",
52
+ "layers.4.attention_conv.kernel_projection",
53
+ "layers.4.mlp_conv.kernel_projection",
54
+ "candidate_selector.hidden_projection"
55
+ ]
56
+ },
57
+ "tensor_storage": {
58
+ "fc": {
59
+ "stored_tensors": {
60
+ "fc.suh": {
61
+ "shape": [
62
+ 25600
63
+ ],
64
+ "n_bytes": 51200,
65
+ "dtype": "torch.float16"
66
+ },
67
+ "fc.svh": {
68
+ "shape": [
69
+ 5120
70
+ ],
71
+ "n_bytes": 10240,
72
+ "dtype": "torch.float16"
73
+ },
74
+ "fc.mul1": {
75
+ "shape": [],
76
+ "n_bytes": 4,
77
+ "dtype": "torch.int32"
78
+ },
79
+ "fc.trellis": {
80
+ "shape": [
81
+ 1600,
82
+ 320,
83
+ 80
84
+ ],
85
+ "n_bytes": 81920000,
86
+ "dtype": "torch.int16"
87
+ }
88
+ },
89
+ "quant_format": "exl3",
90
+ "bits_per_weight": 5,
91
+ "mul1_multiplier": 2212286765
92
+ },
93
+ "hidden_norm": {
94
+ "stored_tensors": {
95
+ "hidden_norm.weight": {
96
+ "shape": [
97
+ 5120
98
+ ],
99
+ "n_bytes": 10240,
100
+ "dtype": "torch.bfloat16"
101
+ }
102
+ }
103
+ },
104
+ "layers.0.self_attn.q_proj": {
105
+ "stored_tensors": {
106
+ "layers.0.self_attn.q_proj.suh": {
107
+ "shape": [
108
+ 5120
109
+ ],
110
+ "n_bytes": 10240,
111
+ "dtype": "torch.float16"
112
+ },
113
+ "layers.0.self_attn.q_proj.svh": {
114
+ "shape": [
115
+ 4096
116
+ ],
117
+ "n_bytes": 8192,
118
+ "dtype": "torch.float16"
119
+ },
120
+ "layers.0.self_attn.q_proj.mul1": {
121
+ "shape": [],
122
+ "n_bytes": 4,
123
+ "dtype": "torch.int32"
124
+ },
125
+ "layers.0.self_attn.q_proj.trellis": {
126
+ "shape": [
127
+ 320,
128
+ 256,
129
+ 80
130
+ ],
131
+ "n_bytes": 13107200,
132
+ "dtype": "torch.int16"
133
+ }
134
+ },
135
+ "quant_format": "exl3",
136
+ "bits_per_weight": 5,
137
+ "mul1_multiplier": 2212286765
138
+ },
139
+ "layers.0.self_attn.k_proj": {
140
+ "stored_tensors": {
141
+ "layers.0.self_attn.k_proj.suh": {
142
+ "shape": [
143
+ 5120
144
+ ],
145
+ "n_bytes": 10240,
146
+ "dtype": "torch.float16"
147
+ },
148
+ "layers.0.self_attn.k_proj.svh": {
149
+ "shape": [
150
+ 1024
151
+ ],
152
+ "n_bytes": 2048,
153
+ "dtype": "torch.float16"
154
+ },
155
+ "layers.0.self_attn.k_proj.mul1": {
156
+ "shape": [],
157
+ "n_bytes": 4,
158
+ "dtype": "torch.int32"
159
+ },
160
+ "layers.0.self_attn.k_proj.trellis": {
161
+ "shape": [
162
+ 320,
163
+ 64,
164
+ 80
165
+ ],
166
+ "n_bytes": 3276800,
167
+ "dtype": "torch.int16"
168
+ }
169
+ },
170
+ "quant_format": "exl3",
171
+ "bits_per_weight": 5,
172
+ "mul1_multiplier": 2212286765
173
+ },
174
+ "layers.0.self_attn.v_proj": {
175
+ "stored_tensors": {
176
+ "layers.0.self_attn.v_proj.suh": {
177
+ "shape": [
178
+ 5120
179
+ ],
180
+ "n_bytes": 10240,
181
+ "dtype": "torch.float16"
182
+ },
183
+ "layers.0.self_attn.v_proj.svh": {
184
+ "shape": [
185
+ 1024
186
+ ],
187
+ "n_bytes": 2048,
188
+ "dtype": "torch.float16"
189
+ },
190
+ "layers.0.self_attn.v_proj.mul1": {
191
+ "shape": [],
192
+ "n_bytes": 4,
193
+ "dtype": "torch.int32"
194
+ },
195
+ "layers.0.self_attn.v_proj.trellis": {
196
+ "shape": [
197
+ 320,
198
+ 64,
199
+ 80
200
+ ],
201
+ "n_bytes": 3276800,
202
+ "dtype": "torch.int16"
203
+ }
204
+ },
205
+ "quant_format": "exl3",
206
+ "bits_per_weight": 5,
207
+ "mul1_multiplier": 2212286765
208
+ },
209
+ "layers.0.self_attn.o_proj": {
210
+ "stored_tensors": {
211
+ "layers.0.self_attn.o_proj.suh": {
212
+ "shape": [
213
+ 4096
214
+ ],
215
+ "n_bytes": 8192,
216
+ "dtype": "torch.float16"
217
+ },
218
+ "layers.0.self_attn.o_proj.svh": {
219
+ "shape": [
220
+ 5120
221
+ ],
222
+ "n_bytes": 10240,
223
+ "dtype": "torch.float16"
224
+ },
225
+ "layers.0.self_attn.o_proj.mul1": {
226
+ "shape": [],
227
+ "n_bytes": 4,
228
+ "dtype": "torch.int32"
229
+ },
230
+ "layers.0.self_attn.o_proj.trellis": {
231
+ "shape": [
232
+ 256,
233
+ 320,
234
+ 80
235
+ ],
236
+ "n_bytes": 13107200,
237
+ "dtype": "torch.int16"
238
+ }
239
+ },
240
+ "quant_format": "exl3",
241
+ "bits_per_weight": 5,
242
+ "mul1_multiplier": 2212286765
243
+ },
244
+ "layers.0.self_attn.q_norm": {
245
+ "stored_tensors": {
246
+ "layers.0.self_attn.q_norm.weight": {
247
+ "shape": [
248
+ 128
249
+ ],
250
+ "n_bytes": 256,
251
+ "dtype": "torch.bfloat16"
252
+ }
253
+ }
254
+ },
255
+ "layers.0.self_attn.k_norm": {
256
+ "stored_tensors": {
257
+ "layers.0.self_attn.k_norm.weight": {
258
+ "shape": [
259
+ 128
260
+ ],
261
+ "n_bytes": 256,
262
+ "dtype": "torch.bfloat16"
263
+ }
264
+ }
265
+ },
266
+ "layers.0.mlp.up_proj": {
267
+ "stored_tensors": {
268
+ "layers.0.mlp.up_proj.suh": {
269
+ "shape": [
270
+ 5120
271
+ ],
272
+ "n_bytes": 10240,
273
+ "dtype": "torch.float16"
274
+ },
275
+ "layers.0.mlp.up_proj.svh": {
276
+ "shape": [
277
+ 17408
278
+ ],
279
+ "n_bytes": 34816,
280
+ "dtype": "torch.float16"
281
+ },
282
+ "layers.0.mlp.up_proj.mul1": {
283
+ "shape": [],
284
+ "n_bytes": 4,
285
+ "dtype": "torch.int32"
286
+ },
287
+ "layers.0.mlp.up_proj.trellis": {
288
+ "shape": [
289
+ 320,
290
+ 1088,
291
+ 80
292
+ ],
293
+ "n_bytes": 55705600,
294
+ "dtype": "torch.int16"
295
+ }
296
+ },
297
+ "quant_format": "exl3",
298
+ "bits_per_weight": 5,
299
+ "mul1_multiplier": 2212286765
300
+ },
301
+ "layers.0.mlp.gate_proj": {
302
+ "stored_tensors": {
303
+ "layers.0.mlp.gate_proj.suh": {
304
+ "shape": [
305
+ 5120
306
+ ],
307
+ "n_bytes": 10240,
308
+ "dtype": "torch.float16"
309
+ },
310
+ "layers.0.mlp.gate_proj.svh": {
311
+ "shape": [
312
+ 17408
313
+ ],
314
+ "n_bytes": 34816,
315
+ "dtype": "torch.float16"
316
+ },
317
+ "layers.0.mlp.gate_proj.mul1": {
318
+ "shape": [],
319
+ "n_bytes": 4,
320
+ "dtype": "torch.int32"
321
+ },
322
+ "layers.0.mlp.gate_proj.trellis": {
323
+ "shape": [
324
+ 320,
325
+ 1088,
326
+ 80
327
+ ],
328
+ "n_bytes": 55705600,
329
+ "dtype": "torch.int16"
330
+ }
331
+ },
332
+ "quant_format": "exl3",
333
+ "bits_per_weight": 5,
334
+ "mul1_multiplier": 2212286765
335
+ },
336
+ "layers.0.mlp.down_proj": {
337
+ "stored_tensors": {
338
+ "layers.0.mlp.down_proj.suh": {
339
+ "shape": [
340
+ 17408
341
+ ],
342
+ "n_bytes": 34816,
343
+ "dtype": "torch.float16"
344
+ },
345
+ "layers.0.mlp.down_proj.svh": {
346
+ "shape": [
347
+ 5120
348
+ ],
349
+ "n_bytes": 10240,
350
+ "dtype": "torch.float16"
351
+ },
352
+ "layers.0.mlp.down_proj.mul1": {
353
+ "shape": [],
354
+ "n_bytes": 4,
355
+ "dtype": "torch.int32"
356
+ },
357
+ "layers.0.mlp.down_proj.trellis": {
358
+ "shape": [
359
+ 1088,
360
+ 320,
361
+ 80
362
+ ],
363
+ "n_bytes": 55705600,
364
+ "dtype": "torch.int16"
365
+ }
366
+ },
367
+ "quant_format": "exl3",
368
+ "bits_per_weight": 5,
369
+ "mul1_multiplier": 2212286765
370
+ },
371
+ "layers.0.input_layernorm": {
372
+ "stored_tensors": {
373
+ "layers.0.input_layernorm.weight": {
374
+ "shape": [
375
+ 5120
376
+ ],
377
+ "n_bytes": 10240,
378
+ "dtype": "torch.bfloat16"
379
+ }
380
+ }
381
+ },
382
+ "layers.0.post_attention_layernorm": {
383
+ "stored_tensors": {
384
+ "layers.0.post_attention_layernorm.weight": {
385
+ "shape": [
386
+ 5120
387
+ ],
388
+ "n_bytes": 10240,
389
+ "dtype": "torch.bfloat16"
390
+ }
391
+ }
392
+ },
393
+ "layers.0.attention_conv.kernel_projection": {
394
+ "stored_tensors": {
395
+ "layers.0.attention_conv.kernel_projection.weight": {
396
+ "shape": [
397
+ 1280,
398
+ 5120
399
+ ],
400
+ "n_bytes": 13107200,
401
+ "dtype": "torch.float16"
402
+ }
403
+ }
404
+ },
405
+ "layers.0.mlp_conv.kernel_projection": {
406
+ "stored_tensors": {
407
+ "layers.0.mlp_conv.kernel_projection.weight": {
408
+ "shape": [
409
+ 1280,
410
+ 5120
411
+ ],
412
+ "n_bytes": 13107200,
413
+ "dtype": "torch.float16"
414
+ }
415
+ }
416
+ },
417
+ "layers.1.self_attn.q_proj": {
418
+ "stored_tensors": {
419
+ "layers.1.self_attn.q_proj.suh": {
420
+ "shape": [
421
+ 5120
422
+ ],
423
+ "n_bytes": 10240,
424
+ "dtype": "torch.float16"
425
+ },
426
+ "layers.1.self_attn.q_proj.svh": {
427
+ "shape": [
428
+ 4096
429
+ ],
430
+ "n_bytes": 8192,
431
+ "dtype": "torch.float16"
432
+ },
433
+ "layers.1.self_attn.q_proj.mul1": {
434
+ "shape": [],
435
+ "n_bytes": 4,
436
+ "dtype": "torch.int32"
437
+ },
438
+ "layers.1.self_attn.q_proj.trellis": {
439
+ "shape": [
440
+ 320,
441
+ 256,
442
+ 80
443
+ ],
444
+ "n_bytes": 13107200,
445
+ "dtype": "torch.int16"
446
+ }
447
+ },
448
+ "quant_format": "exl3",
449
+ "bits_per_weight": 5,
450
+ "mul1_multiplier": 2212286765
451
+ },
452
+ "layers.1.self_attn.k_proj": {
453
+ "stored_tensors": {
454
+ "layers.1.self_attn.k_proj.suh": {
455
+ "shape": [
456
+ 5120
457
+ ],
458
+ "n_bytes": 10240,
459
+ "dtype": "torch.float16"
460
+ },
461
+ "layers.1.self_attn.k_proj.svh": {
462
+ "shape": [
463
+ 1024
464
+ ],
465
+ "n_bytes": 2048,
466
+ "dtype": "torch.float16"
467
+ },
468
+ "layers.1.self_attn.k_proj.mul1": {
469
+ "shape": [],
470
+ "n_bytes": 4,
471
+ "dtype": "torch.int32"
472
+ },
473
+ "layers.1.self_attn.k_proj.trellis": {
474
+ "shape": [
475
+ 320,
476
+ 64,
477
+ 80
478
+ ],
479
+ "n_bytes": 3276800,
480
+ "dtype": "torch.int16"
481
+ }
482
+ },
483
+ "quant_format": "exl3",
484
+ "bits_per_weight": 5,
485
+ "mul1_multiplier": 2212286765
486
+ },
487
+ "layers.1.self_attn.v_proj": {
488
+ "stored_tensors": {
489
+ "layers.1.self_attn.v_proj.suh": {
490
+ "shape": [
491
+ 5120
492
+ ],
493
+ "n_bytes": 10240,
494
+ "dtype": "torch.float16"
495
+ },
496
+ "layers.1.self_attn.v_proj.svh": {
497
+ "shape": [
498
+ 1024
499
+ ],
500
+ "n_bytes": 2048,
501
+ "dtype": "torch.float16"
502
+ },
503
+ "layers.1.self_attn.v_proj.mul1": {
504
+ "shape": [],
505
+ "n_bytes": 4,
506
+ "dtype": "torch.int32"
507
+ },
508
+ "layers.1.self_attn.v_proj.trellis": {
509
+ "shape": [
510
+ 320,
511
+ 64,
512
+ 80
513
+ ],
514
+ "n_bytes": 3276800,
515
+ "dtype": "torch.int16"
516
+ }
517
+ },
518
+ "quant_format": "exl3",
519
+ "bits_per_weight": 5,
520
+ "mul1_multiplier": 2212286765
521
+ },
522
+ "layers.1.self_attn.o_proj": {
523
+ "stored_tensors": {
524
+ "layers.1.self_attn.o_proj.suh": {
525
+ "shape": [
526
+ 4096
527
+ ],
528
+ "n_bytes": 8192,
529
+ "dtype": "torch.float16"
530
+ },
531
+ "layers.1.self_attn.o_proj.svh": {
532
+ "shape": [
533
+ 5120
534
+ ],
535
+ "n_bytes": 10240,
536
+ "dtype": "torch.float16"
537
+ },
538
+ "layers.1.self_attn.o_proj.mul1": {
539
+ "shape": [],
540
+ "n_bytes": 4,
541
+ "dtype": "torch.int32"
542
+ },
543
+ "layers.1.self_attn.o_proj.trellis": {
544
+ "shape": [
545
+ 256,
546
+ 320,
547
+ 80
548
+ ],
549
+ "n_bytes": 13107200,
550
+ "dtype": "torch.int16"
551
+ }
552
+ },
553
+ "quant_format": "exl3",
554
+ "bits_per_weight": 5,
555
+ "mul1_multiplier": 2212286765
556
+ },
557
+ "layers.1.self_attn.q_norm": {
558
+ "stored_tensors": {
559
+ "layers.1.self_attn.q_norm.weight": {
560
+ "shape": [
561
+ 128
562
+ ],
563
+ "n_bytes": 256,
564
+ "dtype": "torch.bfloat16"
565
+ }
566
+ }
567
+ },
568
+ "layers.1.self_attn.k_norm": {
569
+ "stored_tensors": {
570
+ "layers.1.self_attn.k_norm.weight": {
571
+ "shape": [
572
+ 128
573
+ ],
574
+ "n_bytes": 256,
575
+ "dtype": "torch.bfloat16"
576
+ }
577
+ }
578
+ },
579
+ "layers.1.mlp.up_proj": {
580
+ "stored_tensors": {
581
+ "layers.1.mlp.up_proj.suh": {
582
+ "shape": [
583
+ 5120
584
+ ],
585
+ "n_bytes": 10240,
586
+ "dtype": "torch.float16"
587
+ },
588
+ "layers.1.mlp.up_proj.svh": {
589
+ "shape": [
590
+ 17408
591
+ ],
592
+ "n_bytes": 34816,
593
+ "dtype": "torch.float16"
594
+ },
595
+ "layers.1.mlp.up_proj.mul1": {
596
+ "shape": [],
597
+ "n_bytes": 4,
598
+ "dtype": "torch.int32"
599
+ },
600
+ "layers.1.mlp.up_proj.trellis": {
601
+ "shape": [
602
+ 320,
603
+ 1088,
604
+ 80
605
+ ],
606
+ "n_bytes": 55705600,
607
+ "dtype": "torch.int16"
608
+ }
609
+ },
610
+ "quant_format": "exl3",
611
+ "bits_per_weight": 5,
612
+ "mul1_multiplier": 2212286765
613
+ },
614
+ "layers.1.mlp.gate_proj": {
615
+ "stored_tensors": {
616
+ "layers.1.mlp.gate_proj.suh": {
617
+ "shape": [
618
+ 5120
619
+ ],
620
+ "n_bytes": 10240,
621
+ "dtype": "torch.float16"
622
+ },
623
+ "layers.1.mlp.gate_proj.svh": {
624
+ "shape": [
625
+ 17408
626
+ ],
627
+ "n_bytes": 34816,
628
+ "dtype": "torch.float16"
629
+ },
630
+ "layers.1.mlp.gate_proj.mul1": {
631
+ "shape": [],
632
+ "n_bytes": 4,
633
+ "dtype": "torch.int32"
634
+ },
635
+ "layers.1.mlp.gate_proj.trellis": {
636
+ "shape": [
637
+ 320,
638
+ 1088,
639
+ 80
640
+ ],
641
+ "n_bytes": 55705600,
642
+ "dtype": "torch.int16"
643
+ }
644
+ },
645
+ "quant_format": "exl3",
646
+ "bits_per_weight": 5,
647
+ "mul1_multiplier": 2212286765
648
+ },
649
+ "layers.1.mlp.down_proj": {
650
+ "stored_tensors": {
651
+ "layers.1.mlp.down_proj.suh": {
652
+ "shape": [
653
+ 17408
654
+ ],
655
+ "n_bytes": 34816,
656
+ "dtype": "torch.float16"
657
+ },
658
+ "layers.1.mlp.down_proj.svh": {
659
+ "shape": [
660
+ 5120
661
+ ],
662
+ "n_bytes": 10240,
663
+ "dtype": "torch.float16"
664
+ },
665
+ "layers.1.mlp.down_proj.mul1": {
666
+ "shape": [],
667
+ "n_bytes": 4,
668
+ "dtype": "torch.int32"
669
+ },
670
+ "layers.1.mlp.down_proj.trellis": {
671
+ "shape": [
672
+ 1088,
673
+ 320,
674
+ 80
675
+ ],
676
+ "n_bytes": 55705600,
677
+ "dtype": "torch.int16"
678
+ }
679
+ },
680
+ "quant_format": "exl3",
681
+ "bits_per_weight": 5,
682
+ "mul1_multiplier": 2212286765
683
+ },
684
+ "layers.1.input_layernorm": {
685
+ "stored_tensors": {
686
+ "layers.1.input_layernorm.weight": {
687
+ "shape": [
688
+ 5120
689
+ ],
690
+ "n_bytes": 10240,
691
+ "dtype": "torch.bfloat16"
692
+ }
693
+ }
694
+ },
695
+ "layers.1.post_attention_layernorm": {
696
+ "stored_tensors": {
697
+ "layers.1.post_attention_layernorm.weight": {
698
+ "shape": [
699
+ 5120
700
+ ],
701
+ "n_bytes": 10240,
702
+ "dtype": "torch.bfloat16"
703
+ }
704
+ }
705
+ },
706
+ "layers.1.attention_conv.kernel_projection": {
707
+ "stored_tensors": {
708
+ "layers.1.attention_conv.kernel_projection.weight": {
709
+ "shape": [
710
+ 1280,
711
+ 5120
712
+ ],
713
+ "n_bytes": 13107200,
714
+ "dtype": "torch.float16"
715
+ }
716
+ }
717
+ },
718
+ "layers.1.mlp_conv.kernel_projection": {
719
+ "stored_tensors": {
720
+ "layers.1.mlp_conv.kernel_projection.weight": {
721
+ "shape": [
722
+ 1280,
723
+ 5120
724
+ ],
725
+ "n_bytes": 13107200,
726
+ "dtype": "torch.float16"
727
+ }
728
+ }
729
+ },
730
+ "layers.2.self_attn.q_proj": {
731
+ "stored_tensors": {
732
+ "layers.2.self_attn.q_proj.suh": {
733
+ "shape": [
734
+ 5120
735
+ ],
736
+ "n_bytes": 10240,
737
+ "dtype": "torch.float16"
738
+ },
739
+ "layers.2.self_attn.q_proj.svh": {
740
+ "shape": [
741
+ 4096
742
+ ],
743
+ "n_bytes": 8192,
744
+ "dtype": "torch.float16"
745
+ },
746
+ "layers.2.self_attn.q_proj.mul1": {
747
+ "shape": [],
748
+ "n_bytes": 4,
749
+ "dtype": "torch.int32"
750
+ },
751
+ "layers.2.self_attn.q_proj.trellis": {
752
+ "shape": [
753
+ 320,
754
+ 256,
755
+ 80
756
+ ],
757
+ "n_bytes": 13107200,
758
+ "dtype": "torch.int16"
759
+ }
760
+ },
761
+ "quant_format": "exl3",
762
+ "bits_per_weight": 5,
763
+ "mul1_multiplier": 2212286765
764
+ },
765
+ "layers.2.self_attn.k_proj": {
766
+ "stored_tensors": {
767
+ "layers.2.self_attn.k_proj.suh": {
768
+ "shape": [
769
+ 5120
770
+ ],
771
+ "n_bytes": 10240,
772
+ "dtype": "torch.float16"
773
+ },
774
+ "layers.2.self_attn.k_proj.svh": {
775
+ "shape": [
776
+ 1024
777
+ ],
778
+ "n_bytes": 2048,
779
+ "dtype": "torch.float16"
780
+ },
781
+ "layers.2.self_attn.k_proj.mul1": {
782
+ "shape": [],
783
+ "n_bytes": 4,
784
+ "dtype": "torch.int32"
785
+ },
786
+ "layers.2.self_attn.k_proj.trellis": {
787
+ "shape": [
788
+ 320,
789
+ 64,
790
+ 80
791
+ ],
792
+ "n_bytes": 3276800,
793
+ "dtype": "torch.int16"
794
+ }
795
+ },
796
+ "quant_format": "exl3",
797
+ "bits_per_weight": 5,
798
+ "mul1_multiplier": 2212286765
799
+ },
800
+ "layers.2.self_attn.v_proj": {
801
+ "stored_tensors": {
802
+ "layers.2.self_attn.v_proj.suh": {
803
+ "shape": [
804
+ 5120
805
+ ],
806
+ "n_bytes": 10240,
807
+ "dtype": "torch.float16"
808
+ },
809
+ "layers.2.self_attn.v_proj.svh": {
810
+ "shape": [
811
+ 1024
812
+ ],
813
+ "n_bytes": 2048,
814
+ "dtype": "torch.float16"
815
+ },
816
+ "layers.2.self_attn.v_proj.mul1": {
817
+ "shape": [],
818
+ "n_bytes": 4,
819
+ "dtype": "torch.int32"
820
+ },
821
+ "layers.2.self_attn.v_proj.trellis": {
822
+ "shape": [
823
+ 320,
824
+ 64,
825
+ 80
826
+ ],
827
+ "n_bytes": 3276800,
828
+ "dtype": "torch.int16"
829
+ }
830
+ },
831
+ "quant_format": "exl3",
832
+ "bits_per_weight": 5,
833
+ "mul1_multiplier": 2212286765
834
+ },
835
+ "layers.2.self_attn.o_proj": {
836
+ "stored_tensors": {
837
+ "layers.2.self_attn.o_proj.suh": {
838
+ "shape": [
839
+ 4096
840
+ ],
841
+ "n_bytes": 8192,
842
+ "dtype": "torch.float16"
843
+ },
844
+ "layers.2.self_attn.o_proj.svh": {
845
+ "shape": [
846
+ 5120
847
+ ],
848
+ "n_bytes": 10240,
849
+ "dtype": "torch.float16"
850
+ },
851
+ "layers.2.self_attn.o_proj.mul1": {
852
+ "shape": [],
853
+ "n_bytes": 4,
854
+ "dtype": "torch.int32"
855
+ },
856
+ "layers.2.self_attn.o_proj.trellis": {
857
+ "shape": [
858
+ 256,
859
+ 320,
860
+ 80
861
+ ],
862
+ "n_bytes": 13107200,
863
+ "dtype": "torch.int16"
864
+ }
865
+ },
866
+ "quant_format": "exl3",
867
+ "bits_per_weight": 5,
868
+ "mul1_multiplier": 2212286765
869
+ },
870
+ "layers.2.self_attn.q_norm": {
871
+ "stored_tensors": {
872
+ "layers.2.self_attn.q_norm.weight": {
873
+ "shape": [
874
+ 128
875
+ ],
876
+ "n_bytes": 256,
877
+ "dtype": "torch.bfloat16"
878
+ }
879
+ }
880
+ },
881
+ "layers.2.self_attn.k_norm": {
882
+ "stored_tensors": {
883
+ "layers.2.self_attn.k_norm.weight": {
884
+ "shape": [
885
+ 128
886
+ ],
887
+ "n_bytes": 256,
888
+ "dtype": "torch.bfloat16"
889
+ }
890
+ }
891
+ },
892
+ "layers.2.mlp.up_proj": {
893
+ "stored_tensors": {
894
+ "layers.2.mlp.up_proj.suh": {
895
+ "shape": [
896
+ 5120
897
+ ],
898
+ "n_bytes": 10240,
899
+ "dtype": "torch.float16"
900
+ },
901
+ "layers.2.mlp.up_proj.svh": {
902
+ "shape": [
903
+ 17408
904
+ ],
905
+ "n_bytes": 34816,
906
+ "dtype": "torch.float16"
907
+ },
908
+ "layers.2.mlp.up_proj.mul1": {
909
+ "shape": [],
910
+ "n_bytes": 4,
911
+ "dtype": "torch.int32"
912
+ },
913
+ "layers.2.mlp.up_proj.trellis": {
914
+ "shape": [
915
+ 320,
916
+ 1088,
917
+ 80
918
+ ],
919
+ "n_bytes": 55705600,
920
+ "dtype": "torch.int16"
921
+ }
922
+ },
923
+ "quant_format": "exl3",
924
+ "bits_per_weight": 5,
925
+ "mul1_multiplier": 2212286765
926
+ },
927
+ "layers.2.mlp.gate_proj": {
928
+ "stored_tensors": {
929
+ "layers.2.mlp.gate_proj.suh": {
930
+ "shape": [
931
+ 5120
932
+ ],
933
+ "n_bytes": 10240,
934
+ "dtype": "torch.float16"
935
+ },
936
+ "layers.2.mlp.gate_proj.svh": {
937
+ "shape": [
938
+ 17408
939
+ ],
940
+ "n_bytes": 34816,
941
+ "dtype": "torch.float16"
942
+ },
943
+ "layers.2.mlp.gate_proj.mul1": {
944
+ "shape": [],
945
+ "n_bytes": 4,
946
+ "dtype": "torch.int32"
947
+ },
948
+ "layers.2.mlp.gate_proj.trellis": {
949
+ "shape": [
950
+ 320,
951
+ 1088,
952
+ 80
953
+ ],
954
+ "n_bytes": 55705600,
955
+ "dtype": "torch.int16"
956
+ }
957
+ },
958
+ "quant_format": "exl3",
959
+ "bits_per_weight": 5,
960
+ "mul1_multiplier": 2212286765
961
+ },
962
+ "layers.2.mlp.down_proj": {
963
+ "stored_tensors": {
964
+ "layers.2.mlp.down_proj.suh": {
965
+ "shape": [
966
+ 17408
967
+ ],
968
+ "n_bytes": 34816,
969
+ "dtype": "torch.float16"
970
+ },
971
+ "layers.2.mlp.down_proj.svh": {
972
+ "shape": [
973
+ 5120
974
+ ],
975
+ "n_bytes": 10240,
976
+ "dtype": "torch.float16"
977
+ },
978
+ "layers.2.mlp.down_proj.mul1": {
979
+ "shape": [],
980
+ "n_bytes": 4,
981
+ "dtype": "torch.int32"
982
+ },
983
+ "layers.2.mlp.down_proj.trellis": {
984
+ "shape": [
985
+ 1088,
986
+ 320,
987
+ 80
988
+ ],
989
+ "n_bytes": 55705600,
990
+ "dtype": "torch.int16"
991
+ }
992
+ },
993
+ "quant_format": "exl3",
994
+ "bits_per_weight": 5,
995
+ "mul1_multiplier": 2212286765
996
+ },
997
+ "layers.2.input_layernorm": {
998
+ "stored_tensors": {
999
+ "layers.2.input_layernorm.weight": {
1000
+ "shape": [
1001
+ 5120
1002
+ ],
1003
+ "n_bytes": 10240,
1004
+ "dtype": "torch.bfloat16"
1005
+ }
1006
+ }
1007
+ },
1008
+ "layers.2.post_attention_layernorm": {
1009
+ "stored_tensors": {
1010
+ "layers.2.post_attention_layernorm.weight": {
1011
+ "shape": [
1012
+ 5120
1013
+ ],
1014
+ "n_bytes": 10240,
1015
+ "dtype": "torch.bfloat16"
1016
+ }
1017
+ }
1018
+ },
1019
+ "layers.2.attention_conv.kernel_projection": {
1020
+ "stored_tensors": {
1021
+ "layers.2.attention_conv.kernel_projection.weight": {
1022
+ "shape": [
1023
+ 1280,
1024
+ 5120
1025
+ ],
1026
+ "n_bytes": 13107200,
1027
+ "dtype": "torch.float16"
1028
+ }
1029
+ }
1030
+ },
1031
+ "layers.2.mlp_conv.kernel_projection": {
1032
+ "stored_tensors": {
1033
+ "layers.2.mlp_conv.kernel_projection.weight": {
1034
+ "shape": [
1035
+ 1280,
1036
+ 5120
1037
+ ],
1038
+ "n_bytes": 13107200,
1039
+ "dtype": "torch.float16"
1040
+ }
1041
+ }
1042
+ },
1043
+ "layers.3.self_attn.q_proj": {
1044
+ "stored_tensors": {
1045
+ "layers.3.self_attn.q_proj.suh": {
1046
+ "shape": [
1047
+ 5120
1048
+ ],
1049
+ "n_bytes": 10240,
1050
+ "dtype": "torch.float16"
1051
+ },
1052
+ "layers.3.self_attn.q_proj.svh": {
1053
+ "shape": [
1054
+ 4096
1055
+ ],
1056
+ "n_bytes": 8192,
1057
+ "dtype": "torch.float16"
1058
+ },
1059
+ "layers.3.self_attn.q_proj.mul1": {
1060
+ "shape": [],
1061
+ "n_bytes": 4,
1062
+ "dtype": "torch.int32"
1063
+ },
1064
+ "layers.3.self_attn.q_proj.trellis": {
1065
+ "shape": [
1066
+ 320,
1067
+ 256,
1068
+ 80
1069
+ ],
1070
+ "n_bytes": 13107200,
1071
+ "dtype": "torch.int16"
1072
+ }
1073
+ },
1074
+ "quant_format": "exl3",
1075
+ "bits_per_weight": 5,
1076
+ "mul1_multiplier": 2212286765
1077
+ },
1078
+ "layers.3.self_attn.k_proj": {
1079
+ "stored_tensors": {
1080
+ "layers.3.self_attn.k_proj.suh": {
1081
+ "shape": [
1082
+ 5120
1083
+ ],
1084
+ "n_bytes": 10240,
1085
+ "dtype": "torch.float16"
1086
+ },
1087
+ "layers.3.self_attn.k_proj.svh": {
1088
+ "shape": [
1089
+ 1024
1090
+ ],
1091
+ "n_bytes": 2048,
1092
+ "dtype": "torch.float16"
1093
+ },
1094
+ "layers.3.self_attn.k_proj.mul1": {
1095
+ "shape": [],
1096
+ "n_bytes": 4,
1097
+ "dtype": "torch.int32"
1098
+ },
1099
+ "layers.3.self_attn.k_proj.trellis": {
1100
+ "shape": [
1101
+ 320,
1102
+ 64,
1103
+ 80
1104
+ ],
1105
+ "n_bytes": 3276800,
1106
+ "dtype": "torch.int16"
1107
+ }
1108
+ },
1109
+ "quant_format": "exl3",
1110
+ "bits_per_weight": 5,
1111
+ "mul1_multiplier": 2212286765
1112
+ },
1113
+ "layers.3.self_attn.v_proj": {
1114
+ "stored_tensors": {
1115
+ "layers.3.self_attn.v_proj.suh": {
1116
+ "shape": [
1117
+ 5120
1118
+ ],
1119
+ "n_bytes": 10240,
1120
+ "dtype": "torch.float16"
1121
+ },
1122
+ "layers.3.self_attn.v_proj.svh": {
1123
+ "shape": [
1124
+ 1024
1125
+ ],
1126
+ "n_bytes": 2048,
1127
+ "dtype": "torch.float16"
1128
+ },
1129
+ "layers.3.self_attn.v_proj.mul1": {
1130
+ "shape": [],
1131
+ "n_bytes": 4,
1132
+ "dtype": "torch.int32"
1133
+ },
1134
+ "layers.3.self_attn.v_proj.trellis": {
1135
+ "shape": [
1136
+ 320,
1137
+ 64,
1138
+ 80
1139
+ ],
1140
+ "n_bytes": 3276800,
1141
+ "dtype": "torch.int16"
1142
+ }
1143
+ },
1144
+ "quant_format": "exl3",
1145
+ "bits_per_weight": 5,
1146
+ "mul1_multiplier": 2212286765
1147
+ },
1148
+ "layers.3.self_attn.o_proj": {
1149
+ "stored_tensors": {
1150
+ "layers.3.self_attn.o_proj.suh": {
1151
+ "shape": [
1152
+ 4096
1153
+ ],
1154
+ "n_bytes": 8192,
1155
+ "dtype": "torch.float16"
1156
+ },
1157
+ "layers.3.self_attn.o_proj.svh": {
1158
+ "shape": [
1159
+ 5120
1160
+ ],
1161
+ "n_bytes": 10240,
1162
+ "dtype": "torch.float16"
1163
+ },
1164
+ "layers.3.self_attn.o_proj.mul1": {
1165
+ "shape": [],
1166
+ "n_bytes": 4,
1167
+ "dtype": "torch.int32"
1168
+ },
1169
+ "layers.3.self_attn.o_proj.trellis": {
1170
+ "shape": [
1171
+ 256,
1172
+ 320,
1173
+ 80
1174
+ ],
1175
+ "n_bytes": 13107200,
1176
+ "dtype": "torch.int16"
1177
+ }
1178
+ },
1179
+ "quant_format": "exl3",
1180
+ "bits_per_weight": 5,
1181
+ "mul1_multiplier": 2212286765
1182
+ },
1183
+ "layers.3.self_attn.q_norm": {
1184
+ "stored_tensors": {
1185
+ "layers.3.self_attn.q_norm.weight": {
1186
+ "shape": [
1187
+ 128
1188
+ ],
1189
+ "n_bytes": 256,
1190
+ "dtype": "torch.bfloat16"
1191
+ }
1192
+ }
1193
+ },
1194
+ "layers.3.self_attn.k_norm": {
1195
+ "stored_tensors": {
1196
+ "layers.3.self_attn.k_norm.weight": {
1197
+ "shape": [
1198
+ 128
1199
+ ],
1200
+ "n_bytes": 256,
1201
+ "dtype": "torch.bfloat16"
1202
+ }
1203
+ }
1204
+ },
1205
+ "layers.3.mlp.up_proj": {
1206
+ "stored_tensors": {
1207
+ "layers.3.mlp.up_proj.suh": {
1208
+ "shape": [
1209
+ 5120
1210
+ ],
1211
+ "n_bytes": 10240,
1212
+ "dtype": "torch.float16"
1213
+ },
1214
+ "layers.3.mlp.up_proj.svh": {
1215
+ "shape": [
1216
+ 17408
1217
+ ],
1218
+ "n_bytes": 34816,
1219
+ "dtype": "torch.float16"
1220
+ },
1221
+ "layers.3.mlp.up_proj.mul1": {
1222
+ "shape": [],
1223
+ "n_bytes": 4,
1224
+ "dtype": "torch.int32"
1225
+ },
1226
+ "layers.3.mlp.up_proj.trellis": {
1227
+ "shape": [
1228
+ 320,
1229
+ 1088,
1230
+ 80
1231
+ ],
1232
+ "n_bytes": 55705600,
1233
+ "dtype": "torch.int16"
1234
+ }
1235
+ },
1236
+ "quant_format": "exl3",
1237
+ "bits_per_weight": 5,
1238
+ "mul1_multiplier": 2212286765
1239
+ },
1240
+ "layers.3.mlp.gate_proj": {
1241
+ "stored_tensors": {
1242
+ "layers.3.mlp.gate_proj.suh": {
1243
+ "shape": [
1244
+ 5120
1245
+ ],
1246
+ "n_bytes": 10240,
1247
+ "dtype": "torch.float16"
1248
+ },
1249
+ "layers.3.mlp.gate_proj.svh": {
1250
+ "shape": [
1251
+ 17408
1252
+ ],
1253
+ "n_bytes": 34816,
1254
+ "dtype": "torch.float16"
1255
+ },
1256
+ "layers.3.mlp.gate_proj.mul1": {
1257
+ "shape": [],
1258
+ "n_bytes": 4,
1259
+ "dtype": "torch.int32"
1260
+ },
1261
+ "layers.3.mlp.gate_proj.trellis": {
1262
+ "shape": [
1263
+ 320,
1264
+ 1088,
1265
+ 80
1266
+ ],
1267
+ "n_bytes": 55705600,
1268
+ "dtype": "torch.int16"
1269
+ }
1270
+ },
1271
+ "quant_format": "exl3",
1272
+ "bits_per_weight": 5,
1273
+ "mul1_multiplier": 2212286765
1274
+ },
1275
+ "layers.3.mlp.down_proj": {
1276
+ "stored_tensors": {
1277
+ "layers.3.mlp.down_proj.suh": {
1278
+ "shape": [
1279
+ 17408
1280
+ ],
1281
+ "n_bytes": 34816,
1282
+ "dtype": "torch.float16"
1283
+ },
1284
+ "layers.3.mlp.down_proj.svh": {
1285
+ "shape": [
1286
+ 5120
1287
+ ],
1288
+ "n_bytes": 10240,
1289
+ "dtype": "torch.float16"
1290
+ },
1291
+ "layers.3.mlp.down_proj.mul1": {
1292
+ "shape": [],
1293
+ "n_bytes": 4,
1294
+ "dtype": "torch.int32"
1295
+ },
1296
+ "layers.3.mlp.down_proj.trellis": {
1297
+ "shape": [
1298
+ 1088,
1299
+ 320,
1300
+ 80
1301
+ ],
1302
+ "n_bytes": 55705600,
1303
+ "dtype": "torch.int16"
1304
+ }
1305
+ },
1306
+ "quant_format": "exl3",
1307
+ "bits_per_weight": 5,
1308
+ "mul1_multiplier": 2212286765
1309
+ },
1310
+ "layers.3.input_layernorm": {
1311
+ "stored_tensors": {
1312
+ "layers.3.input_layernorm.weight": {
1313
+ "shape": [
1314
+ 5120
1315
+ ],
1316
+ "n_bytes": 10240,
1317
+ "dtype": "torch.bfloat16"
1318
+ }
1319
+ }
1320
+ },
1321
+ "layers.3.post_attention_layernorm": {
1322
+ "stored_tensors": {
1323
+ "layers.3.post_attention_layernorm.weight": {
1324
+ "shape": [
1325
+ 5120
1326
+ ],
1327
+ "n_bytes": 10240,
1328
+ "dtype": "torch.bfloat16"
1329
+ }
1330
+ }
1331
+ },
1332
+ "layers.3.attention_conv.kernel_projection": {
1333
+ "stored_tensors": {
1334
+ "layers.3.attention_conv.kernel_projection.weight": {
1335
+ "shape": [
1336
+ 1280,
1337
+ 5120
1338
+ ],
1339
+ "n_bytes": 13107200,
1340
+ "dtype": "torch.float16"
1341
+ }
1342
+ }
1343
+ },
1344
+ "layers.3.mlp_conv.kernel_projection": {
1345
+ "stored_tensors": {
1346
+ "layers.3.mlp_conv.kernel_projection.weight": {
1347
+ "shape": [
1348
+ 1280,
1349
+ 5120
1350
+ ],
1351
+ "n_bytes": 13107200,
1352
+ "dtype": "torch.float16"
1353
+ }
1354
+ }
1355
+ },
1356
+ "layers.4.self_attn.q_proj": {
1357
+ "stored_tensors": {
1358
+ "layers.4.self_attn.q_proj.suh": {
1359
+ "shape": [
1360
+ 5120
1361
+ ],
1362
+ "n_bytes": 10240,
1363
+ "dtype": "torch.float16"
1364
+ },
1365
+ "layers.4.self_attn.q_proj.svh": {
1366
+ "shape": [
1367
+ 4096
1368
+ ],
1369
+ "n_bytes": 8192,
1370
+ "dtype": "torch.float16"
1371
+ },
1372
+ "layers.4.self_attn.q_proj.mul1": {
1373
+ "shape": [],
1374
+ "n_bytes": 4,
1375
+ "dtype": "torch.int32"
1376
+ },
1377
+ "layers.4.self_attn.q_proj.trellis": {
1378
+ "shape": [
1379
+ 320,
1380
+ 256,
1381
+ 80
1382
+ ],
1383
+ "n_bytes": 13107200,
1384
+ "dtype": "torch.int16"
1385
+ }
1386
+ },
1387
+ "quant_format": "exl3",
1388
+ "bits_per_weight": 5,
1389
+ "mul1_multiplier": 2212286765
1390
+ },
1391
+ "layers.4.self_attn.k_proj": {
1392
+ "stored_tensors": {
1393
+ "layers.4.self_attn.k_proj.suh": {
1394
+ "shape": [
1395
+ 5120
1396
+ ],
1397
+ "n_bytes": 10240,
1398
+ "dtype": "torch.float16"
1399
+ },
1400
+ "layers.4.self_attn.k_proj.svh": {
1401
+ "shape": [
1402
+ 1024
1403
+ ],
1404
+ "n_bytes": 2048,
1405
+ "dtype": "torch.float16"
1406
+ },
1407
+ "layers.4.self_attn.k_proj.mul1": {
1408
+ "shape": [],
1409
+ "n_bytes": 4,
1410
+ "dtype": "torch.int32"
1411
+ },
1412
+ "layers.4.self_attn.k_proj.trellis": {
1413
+ "shape": [
1414
+ 320,
1415
+ 64,
1416
+ 80
1417
+ ],
1418
+ "n_bytes": 3276800,
1419
+ "dtype": "torch.int16"
1420
+ }
1421
+ },
1422
+ "quant_format": "exl3",
1423
+ "bits_per_weight": 5,
1424
+ "mul1_multiplier": 2212286765
1425
+ },
1426
+ "layers.4.self_attn.v_proj": {
1427
+ "stored_tensors": {
1428
+ "layers.4.self_attn.v_proj.suh": {
1429
+ "shape": [
1430
+ 5120
1431
+ ],
1432
+ "n_bytes": 10240,
1433
+ "dtype": "torch.float16"
1434
+ },
1435
+ "layers.4.self_attn.v_proj.svh": {
1436
+ "shape": [
1437
+ 1024
1438
+ ],
1439
+ "n_bytes": 2048,
1440
+ "dtype": "torch.float16"
1441
+ },
1442
+ "layers.4.self_attn.v_proj.mul1": {
1443
+ "shape": [],
1444
+ "n_bytes": 4,
1445
+ "dtype": "torch.int32"
1446
+ },
1447
+ "layers.4.self_attn.v_proj.trellis": {
1448
+ "shape": [
1449
+ 320,
1450
+ 64,
1451
+ 80
1452
+ ],
1453
+ "n_bytes": 3276800,
1454
+ "dtype": "torch.int16"
1455
+ }
1456
+ },
1457
+ "quant_format": "exl3",
1458
+ "bits_per_weight": 5,
1459
+ "mul1_multiplier": 2212286765
1460
+ },
1461
+ "layers.4.self_attn.o_proj": {
1462
+ "stored_tensors": {
1463
+ "layers.4.self_attn.o_proj.suh": {
1464
+ "shape": [
1465
+ 4096
1466
+ ],
1467
+ "n_bytes": 8192,
1468
+ "dtype": "torch.float16"
1469
+ },
1470
+ "layers.4.self_attn.o_proj.svh": {
1471
+ "shape": [
1472
+ 5120
1473
+ ],
1474
+ "n_bytes": 10240,
1475
+ "dtype": "torch.float16"
1476
+ },
1477
+ "layers.4.self_attn.o_proj.mul1": {
1478
+ "shape": [],
1479
+ "n_bytes": 4,
1480
+ "dtype": "torch.int32"
1481
+ },
1482
+ "layers.4.self_attn.o_proj.trellis": {
1483
+ "shape": [
1484
+ 256,
1485
+ 320,
1486
+ 80
1487
+ ],
1488
+ "n_bytes": 13107200,
1489
+ "dtype": "torch.int16"
1490
+ }
1491
+ },
1492
+ "quant_format": "exl3",
1493
+ "bits_per_weight": 5,
1494
+ "mul1_multiplier": 2212286765
1495
+ },
1496
+ "layers.4.self_attn.q_norm": {
1497
+ "stored_tensors": {
1498
+ "layers.4.self_attn.q_norm.weight": {
1499
+ "shape": [
1500
+ 128
1501
+ ],
1502
+ "n_bytes": 256,
1503
+ "dtype": "torch.bfloat16"
1504
+ }
1505
+ }
1506
+ },
1507
+ "layers.4.self_attn.k_norm": {
1508
+ "stored_tensors": {
1509
+ "layers.4.self_attn.k_norm.weight": {
1510
+ "shape": [
1511
+ 128
1512
+ ],
1513
+ "n_bytes": 256,
1514
+ "dtype": "torch.bfloat16"
1515
+ }
1516
+ }
1517
+ },
1518
+ "layers.4.mlp.up_proj": {
1519
+ "stored_tensors": {
1520
+ "layers.4.mlp.up_proj.suh": {
1521
+ "shape": [
1522
+ 5120
1523
+ ],
1524
+ "n_bytes": 10240,
1525
+ "dtype": "torch.float16"
1526
+ },
1527
+ "layers.4.mlp.up_proj.svh": {
1528
+ "shape": [
1529
+ 17408
1530
+ ],
1531
+ "n_bytes": 34816,
1532
+ "dtype": "torch.float16"
1533
+ },
1534
+ "layers.4.mlp.up_proj.mul1": {
1535
+ "shape": [],
1536
+ "n_bytes": 4,
1537
+ "dtype": "torch.int32"
1538
+ },
1539
+ "layers.4.mlp.up_proj.trellis": {
1540
+ "shape": [
1541
+ 320,
1542
+ 1088,
1543
+ 80
1544
+ ],
1545
+ "n_bytes": 55705600,
1546
+ "dtype": "torch.int16"
1547
+ }
1548
+ },
1549
+ "quant_format": "exl3",
1550
+ "bits_per_weight": 5,
1551
+ "mul1_multiplier": 2212286765
1552
+ },
1553
+ "layers.4.mlp.gate_proj": {
1554
+ "stored_tensors": {
1555
+ "layers.4.mlp.gate_proj.suh": {
1556
+ "shape": [
1557
+ 5120
1558
+ ],
1559
+ "n_bytes": 10240,
1560
+ "dtype": "torch.float16"
1561
+ },
1562
+ "layers.4.mlp.gate_proj.svh": {
1563
+ "shape": [
1564
+ 17408
1565
+ ],
1566
+ "n_bytes": 34816,
1567
+ "dtype": "torch.float16"
1568
+ },
1569
+ "layers.4.mlp.gate_proj.mul1": {
1570
+ "shape": [],
1571
+ "n_bytes": 4,
1572
+ "dtype": "torch.int32"
1573
+ },
1574
+ "layers.4.mlp.gate_proj.trellis": {
1575
+ "shape": [
1576
+ 320,
1577
+ 1088,
1578
+ 80
1579
+ ],
1580
+ "n_bytes": 55705600,
1581
+ "dtype": "torch.int16"
1582
+ }
1583
+ },
1584
+ "quant_format": "exl3",
1585
+ "bits_per_weight": 5,
1586
+ "mul1_multiplier": 2212286765
1587
+ },
1588
+ "layers.4.mlp.down_proj": {
1589
+ "stored_tensors": {
1590
+ "layers.4.mlp.down_proj.suh": {
1591
+ "shape": [
1592
+ 17408
1593
+ ],
1594
+ "n_bytes": 34816,
1595
+ "dtype": "torch.float16"
1596
+ },
1597
+ "layers.4.mlp.down_proj.svh": {
1598
+ "shape": [
1599
+ 5120
1600
+ ],
1601
+ "n_bytes": 10240,
1602
+ "dtype": "torch.float16"
1603
+ },
1604
+ "layers.4.mlp.down_proj.mul1": {
1605
+ "shape": [],
1606
+ "n_bytes": 4,
1607
+ "dtype": "torch.int32"
1608
+ },
1609
+ "layers.4.mlp.down_proj.trellis": {
1610
+ "shape": [
1611
+ 1088,
1612
+ 320,
1613
+ 80
1614
+ ],
1615
+ "n_bytes": 55705600,
1616
+ "dtype": "torch.int16"
1617
+ }
1618
+ },
1619
+ "quant_format": "exl3",
1620
+ "bits_per_weight": 5,
1621
+ "mul1_multiplier": 2212286765
1622
+ },
1623
+ "layers.4.input_layernorm": {
1624
+ "stored_tensors": {
1625
+ "layers.4.input_layernorm.weight": {
1626
+ "shape": [
1627
+ 5120
1628
+ ],
1629
+ "n_bytes": 10240,
1630
+ "dtype": "torch.bfloat16"
1631
+ }
1632
+ }
1633
+ },
1634
+ "layers.4.post_attention_layernorm": {
1635
+ "stored_tensors": {
1636
+ "layers.4.post_attention_layernorm.weight": {
1637
+ "shape": [
1638
+ 5120
1639
+ ],
1640
+ "n_bytes": 10240,
1641
+ "dtype": "torch.bfloat16"
1642
+ }
1643
+ }
1644
+ },
1645
+ "layers.4.attention_conv.kernel_projection": {
1646
+ "stored_tensors": {
1647
+ "layers.4.attention_conv.kernel_projection.weight": {
1648
+ "shape": [
1649
+ 1280,
1650
+ 5120
1651
+ ],
1652
+ "n_bytes": 13107200,
1653
+ "dtype": "torch.float16"
1654
+ }
1655
+ }
1656
+ },
1657
+ "layers.4.mlp_conv.kernel_projection": {
1658
+ "stored_tensors": {
1659
+ "layers.4.mlp_conv.kernel_projection.weight": {
1660
+ "shape": [
1661
+ 1280,
1662
+ 5120
1663
+ ],
1664
+ "n_bytes": 13107200,
1665
+ "dtype": "torch.float16"
1666
+ }
1667
+ }
1668
+ },
1669
+ "norm": {
1670
+ "stored_tensors": {
1671
+ "norm.weight": {
1672
+ "shape": [
1673
+ 5120
1674
+ ],
1675
+ "n_bytes": 10240,
1676
+ "dtype": "torch.bfloat16"
1677
+ }
1678
+ }
1679
+ },
1680
+ "candidate_selector.hidden_projection": {
1681
+ "stored_tensors": {
1682
+ "candidate_selector.hidden_projection.weight": {
1683
+ "shape": [
1684
+ 256,
1685
+ 5120
1686
+ ],
1687
+ "n_bytes": 2621440,
1688
+ "dtype": "torch.float16"
1689
+ }
1690
+ }
1691
+ }
1692
+ }
1693
+ }