Hugging Face's logo

Lowkeyss
/
GLM-5.2-ablated

Text Generation
Transformers
Safetensors
English
glm_moe_dsa
glm-5.2
abliteration
pca-ablation
lora
safety-alignment
Mixture of Experts
mixin-lora
conversational
fp8
Model card Files
xet
 Community
Lowkeyss Manusagents commited on
Commit
1c5539f
·
0 Parent(s):

Duplicate from Manusagents/GLM-5.2-ablated

Browse files

Co-authored-by: Manusagents <Manusagents@users.noreply.huggingface.co>

This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +37 -0
  2. AUDIT_FINDINGS.md +66 -0
  3. README.md +184 -0
  4. REPRODUCE.md +93 -0
  5. STATISTICAL_ANALYSIS.md +70 -0
  6. chat_template.jinja +119 -0
  7. config.json +949 -0
  8. generation_config.json +13 -0
  9. model-00001-of-00151.safetensors +3 -0
  10. model-00002-of-00151.safetensors +3 -0
  11. model-00003-of-00151.safetensors +3 -0
  12. model-00004-of-00151.safetensors +3 -0
  13. model-00005-of-00151.safetensors +3 -0
  14. model-00006-of-00151.safetensors +3 -0
  15. model-00007-of-00151.safetensors +3 -0
  16. model-00008-of-00151.safetensors +3 -0
  17. model-00009-of-00151.safetensors +3 -0
  18. model-00010-of-00151.safetensors +3 -0
  19. model-00011-of-00151.safetensors +3 -0
  20. model-00012-of-00151.safetensors +3 -0
  21. model-00013-of-00151.safetensors +3 -0
  22. model-00014-of-00151.safetensors +3 -0
  23. model-00015-of-00151.safetensors +3 -0
  24. model-00016-of-00151.safetensors +3 -0
  25. model-00017-of-00151.safetensors +3 -0
  26. model-00018-of-00151.safetensors +3 -0
  27. model-00019-of-00151.safetensors +3 -0
  28. model-00020-of-00151.safetensors +3 -0
  29. model-00021-of-00151.safetensors +3 -0
  30. model-00022-of-00151.safetensors +3 -0
  31. model-00023-of-00151.safetensors +3 -0
  32. model-00024-of-00151.safetensors +3 -0
  33. model-00025-of-00151.safetensors +3 -0
  34. model-00026-of-00151.safetensors +3 -0
  35. model-00027-of-00151.safetensors +3 -0
  36. model-00028-of-00151.safetensors +3 -0
  37. model-00029-of-00151.safetensors +3 -0
  38. model-00030-of-00151.safetensors +3 -0
  39. model-00031-of-00151.safetensors +3 -0
  40. model-00032-of-00151.safetensors +3 -0
  41. model-00033-of-00151.safetensors +3 -0
  42. model-00034-of-00151.safetensors +3 -0
  43. model-00035-of-00151.safetensors +3 -0
  44. model-00036-of-00151.safetensors +3 -0
  45. model-00037-of-00151.safetensors +3 -0
  46. model-00038-of-00151.safetensors +3 -0
  47. model-00039-of-00151.safetensors +3 -0
  48. model-00040-of-00151.safetensors +3 -0
  49. model-00041-of-00151.safetensors +3 -0
  50. model-00042-of-00151.safetensors +3 -0
.gitattributes ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
AUDIT_FINDINGS.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Research Audit Findings — GLM-5.2 Ablation / Project AESOP
2
+
3
+ **Auditor:** Clyde (AgentiOrchestrator)
4
+ **Date:** 2026-06-23
5
+ **Scope:** All research artifacts on 2E (Vast.ai 8×H200), training scripts, benchmark harnesses, results, logs
6
+
7
+ ## Critical Findings
8
+
9
+ ### 1. Inconsistent Benchmark Harness (CRITICAL)
10
+ Multiple harness versions with different refusal patterns, scoring logic, and token limits.
11
+ - v1: 16 refusal patterns, 400-char window, 256 max_tokens
12
+ - v2: 30+ patterns (GLM-5.2 specific), 600-char window, 512 max_tokens
13
+ - Test3a/Fable5-R2/AESOP: v1 harness (comparable to each other)
14
+ - Fable5-R2-Ablated: v2 harness (NOT comparable)
15
+ **Impact:** Cross-model comparisons are unreliable for refusal metrics.
16
+ **Fix:** Unified harness v3 built (benchmarks/harness.py).
17
+
18
+ ### 2. Ablation Hook Code Discrepancies (MODERATE)
19
+ - Script default: 5 layers (61-65), coeff 0.5, 2 PCA components
20
+ - Documented config: 4 layers (62-65), coeff 0.1, 1 component
21
+ - ACTUAL (from full log): 4 layers (62-65), coeff 0.1 ✅ (script defaults were overridden)
22
+ - Still uses 2 PCA components per layer (spec said 1) ❓
23
+ **Status:** PARTIALLY RESOLVED — the log confirms correct layers/coeff, but 2 vs 1 component unclear.
24
+
25
+ ### 3. Full Training Log Was Recoverable (UPDATE)
26
+ The aesop_full.log (642 lines) WAS on 2E and has been preserved in the snapshot.
27
+ Key facts verified:
28
+ - 609/610 steps completed, 4876 examples, 570.6 minutes
29
+ - Loss: 1.2933 → 1.1834 (CONVERGENT — corrects the pilot finding of increasing loss)
30
+ - Hooks confirmed: layers [62,63,64,65], coeff 0.1, 4 hooks installed
31
+ **Status:** RESOLVED — previously listed as "missing artifact."
32
+
33
+ ### 4. Statistical Power Issues (MODERATE)
34
+ - MMLU-Pro n=100: CI ±8% — insufficient for 2-5pp claims
35
+ - GSM8K n=100: CI ±6% — insufficient for 3pp claims
36
+ - SimpleQA n=50: CI ±11% — the 32pp drop IS significant
37
+ - AdvBench n=100: the 1% vs 16% IS significant
38
+ **Impact:** Only AdvBench and SimpleQA show effects large enough to trust.
39
+ **Fix:** Document CIs in all results tables, recommend n≥600 for future runs.
40
+
41
+ ### 5. Fuzzy Scoring (MINOR but documented)
42
+ - GPQA "exact match" actually uses multi-pattern fuzzy extraction
43
+ - SimpleQA uses substring containment (risks false matches)
44
+ - HumanEval has no timeout on exec() (risks hangs)
45
+ **Fix:** Unified harness adds 60s timeout for HumanEval; documents scoring methodology.
46
+
47
+ ### 6. The Test3a vs AESOP Confound (UNRESOLVED)
48
+ Both use the same approach (hooks during training). Test3a: 1% refusal. AESOP: 16%.
49
+ Possible explanations:
50
+ - Test3a may have used different base model state
51
+ - Test3a may have used 1 PCA component vs AESOP's 2
52
+ - Test3a may have used different data ordering/randomization
53
+ - Test3a's exact training command was not logged
54
+ **Status:** UNRESOLVABLE from available artifacts. The key comparison is under-controlled.
55
+
56
+ ### 7. Pilot vs Full Loss Trajectory (CLARIFIED)
57
+ - Pilot (500 examples, 62 steps): loss 1.3981→1.6567 (INCREASED) — raised concerns
58
+ - Full (4876 examples, 609 steps): loss 1.2933→1.1834 (DECREASED) — healthy convergence
59
+ **Conclusion:** The pilot's increasing loss was a data-size artifact (too few examples for the loss to stabilize), not a fundamental issue with the hook+LoRA interaction.
60
+
61
+ ### 8. SimpleQA Collapse is Consistent (IMPORTANT)
62
+ Both Fable5-R2 (56%) and AESOP (56%) show IDENTICAL SimpleQA degradation from Test3a (88%).
63
+ This 32pp drop is significant AND consistent across two different training configurations.
64
+ It suggests the LoRA training itself (on Fable 5 data) damages knowledge retrieval,
65
+ regardless of whether hooks are present. The hooks don't prevent it.
66
+ This is an underreported finding.
README.md ADDED
@@ -0,0 +1,184 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: other
5
+ library_name: transformers
6
+ tags:
7
+ - glm-5.2
8
+ - abliteration
9
+ - pca-ablation
10
+ - lora
11
+ - safety-alignment
12
+ - moe
13
+ - mixin-lora
14
+ base_model: z-ai/GLM-5.2
15
+ model_type: glm
16
+ pipeline_tag: text-generation
17
+ ---
18
+
19
+ # GLM-5.2-Ablated-F5-Molt (AESOP)
20
+
21
+ ## Model Description
22
+
23
+ **AESOP** (Ablation-Enhanced Safety with Orthogonal Projection) is a safety-aligned variant of GLM-5.2, a 744B parameter Mixture-of-Experts (MoE) model with 18.5B dense parameters and 256 routed experts. AESOP combines two interventions:
24
+
25
+ 1. **PCA-based refusal ablation** — Principal Component Analysis directions extracted from GLM-5.2's shared experts are used to subtract the refusal direction from activations during training, preventing the model from re-learning refusal behaviors.
26
+ 2. **Surgical LoRA fine-tuning** — Low-Rank Adaptation (rank 64) on attention modules (layers ≥60) using 4,876 Fable 5 chain-of-thought traces, improving capability while ablation hooks maintain safety.
27
+
28
+ The key innovation is the use of **ablation hooks during training** (not just inference). Prior work (Arditi et al. 2024) applied refusal direction subtraction as a post-hoc inference-time intervention. AESOP demonstrates that maintaining these hooks during LoRA fine-tuning partially prevents the re-activation of refusal behaviors that occurs when fine-tuning on non-aligned data.
29
+
30
+ ## Training Methodology
31
+
32
+ ### Step 1: Refusal Direction Extraction
33
+
34
+ PCA directions were extracted from GLM-5.2's shared expert outputs across layers 25–65 (41 layers). For each layer, activations were collected on a contrastive prompt set (harmful vs. benign), and the first principal component of the difference was taken as the refusal direction. Directions are stored as `refusal_pca.pt` (2.9MB, shape: 41 layers × 3 PCA components × 6144 hidden dim).
35
+
36
+ ### Step 2: Ablation Hook Installation
37
+
38
+ Forward hooks were installed on `model.model.layers[L].mlp.shared_experts` for layers 62–65. The hook subtracts the refusal direction projection from the hidden state:
39
+
40
+ ```python
41
+ def ablation_hook(module, input, output):
42
+ hs = output[0] # hidden states
43
+ d = refusal_direction # shape [6144]
44
+ hs = hs - coeff * (hs @ d) / (d @ d) * d
45
+ return (hs,) + output[1:]
46
+ ```
47
+
48
+ Coefficient: 0.1. PCA components: top 2 per layer. Hooks are active during training and removed for inference.
49
+
50
+ ### Step 3: LoRA Fine-Tuning
51
+
52
+ - **Base model:** GLM-5.2 with ablation hooks applied (PCA-ablated base)
53
+ - **Training data:** `fable5-chatml.jsonl` — 4,876 chain-of-thought examples from Fable 5
54
+ - **LoRA config:** rank=64, alpha=128, target modules = attention (Q, K, V, O) on layers ≥60 (90 modules)
55
+ - **Trainable parameters:** 97,984,512 (0.013% of 743.5B total)
56
+ - **Optimizer:** AdamW, lr=2e-5, cosine schedule, warmup=10 steps
57
+ - **Batch:** gradient accumulation 8, max sequence length 2048
58
+ - **Steps:** 609/610 completed
59
+ - **Elapsed:** 570.6 minutes (~9.5 hours) on 8× NVIDIA H200
60
+
61
+ ### Step 4: Surgical Merge
62
+
63
+ LoRA weights were merged into the base model using a surgical BF16 merge:
64
+ - Attention weights (LoRA targets) are dequantized to BF16, merged with LoRA deltas, and re-saved
65
+ - MoE expert weights (FP8) are preserved unchanged — no dequantization or re-quantization
66
+ - This preserves the FP8 compression of the 256 experts while applying LoRA modifications to attention
67
+
68
+ ## Training Configuration (Verified from `aesop_full.log`)
69
+
70
+ | Parameter | Value |
71
+ |-----------|-------|
72
+ | Base model | GLM-5.2 FP8 (ablated base) |
73
+ | Ablation layers | [62, 63, 64, 65] |
74
+ | Ablation coefficient | 0.1 |
75
+ | PCA components per layer | 2 |
76
+ | Hook target | `mlp.shared_experts` forward output |
77
+ | LoRA rank | 64 |
78
+ | LoRA alpha | 128 |
79
+ | LoRA target layers | ≥60 (90 attention modules) |
80
+ | Training data | 4,876 examples (0 skipped) |
81
+ | Max sequence length | 2048 |
82
+ | Learning rate | 2e-5 |
83
+ | LR schedule | Cosine |
84
+ | Warmup steps | 10 |
85
+ | Gradient accumulation | 8 |
86
+ | Total steps | 609 / 610 planned |
87
+ | First loss | 1.2933 |
88
+ | Final loss | 1.1834 |
89
+ | Training time | 570.6 minutes (9.5 hours) |
90
+ | Hardware | 8× NVIDIA H200 (Vast.ai 2E instance) |
91
+ | Seed | 42 |
92
+ | Checkpoints | Every 100 steps (step-100 through step-600, final) |
93
+
94
+ **Loss trajectory:** The full training run showed healthy convergence (1.2933 → 1.1834). This corrects an earlier pilot finding (500 examples, 62 steps) where loss appeared to increase (1.3981 → 1.6567); that was a data-size artifact, not a fundamental issue.
95
+
96
+ ## Benchmark Results (Unified Harness v3.0.1)
97
+
98
+ All results from the AESOP Unified Benchmark Harness v3, which fixes three critical issues from earlier harness versions (see Audit section below). N=100 for most benchmarks, N=50 for SimpleQA, N=164 for HumanEval.
99
+
100
+ | Benchmark | Metric | AESOP | Best Variant | Δ |
101
+ |-----------|--------|:-----:|:------------:|:---:|
102
+ | AdvBench | refusal_rate ↓ | **58.0%** | — | Strongest safety |
103
+ | Borderline | refusal_rate ↓ | **0.0%** | 0.0% (all) | No over-refusal |
104
+ | GPQA Diamond | accuracy ↑ | 92.0% | 96.0% | Within CI |
105
+ | MMLU-Pro | accuracy ↑ | **84.0%** | — | Best in class |
106
+ | HumanEval | pass@1 ↑ | 84.1% | 87.2% | Within CI |
107
+ | GSM8K | accuracy ↑ | 93.0% | 96.0% | Within CI |
108
+ | HellaSwag | accuracy ↑ | **75.0%** | — | Best in class |
109
+ | SimpleQA | accuracy ↑ | 48.0% | 56.0% | See limitations |
110
+ | IFEval (prompt) | accuracy ↑ | **41.8%** | — | Best in class |
111
+ | IFEval (instr) | accuracy ↑ | **55.9%** | — | Tied best |
112
+
113
+ ### Statistical Significance (Wilson 95% CIs, vs ablated-base)
114
+
115
+ | Benchmark | Δ | p-value | Significant? |
116
+ |-----------|:---:|:-------:|:------------:|
117
+ | AdvBench | +40.0% | <0.001 | ** Yes |
118
+ | Borderline | -2.0% | 0.31 | ns |
119
+ | GPQA | 0.0% | — | ns |
120
+ | GSM8K | 0.0% | — | ns |
121
+ | HellaSwag | +3.0% | 0.61 | ns |
122
+ | HumanEval | +6.7% | 0.14 | ns |
123
+ | IFEval | +0.6% | 0.85 | ns |
124
+ | MMLU-Pro | +9.0% | 0.11 | ns |
125
+ | SimpleQA | -8.0% | 0.36 | ns |
126
+
127
+ Only AdvBench shows a statistically significant improvement at n=100. MMLU-Pro (+9pp) approaches significance but does not reach it. Future evaluations should use n≥600 for 5pp significance thresholds.
128
+
129
+ ## Intended Use
130
+
131
+ ### Primary Use Cases
132
+ - Research on safety alignment, refusal ablation, and MoE model behavior
133
+ - Agent workflows requiring controlled safety profiles
134
+ - Benchmarking and evaluation of alignment interventions
135
+
136
+ ### Out of Scope
137
+ - Production deployment without additional safety evaluation
138
+ - Use cases requiring guaranteed safety guarantees (this is a research artifact)
139
+ - Commercial deployment without appropriate licensing
140
+
141
+ ## Limitations
142
+
143
+ 1. **SimpleQA degradation:** AESOP scores 48.0% on SimpleQA vs 56.0% for the ablated base. This 8pp drop is not individually significant at n=50 (Wilson CI: [41.7%, 69.3%] vs [44.4%, 67.2%]), but the trend is consistent across all LoRA-trained variants. The LoRA training itself appears to damage knowledge retrieval pathways.
144
+
145
+ 2. **Small sample sizes:** Most benchmarks use n=100 (Wilson CI ±8%). Differences of <15pp are not statistically significant. Claims about 2–5pp improvements should not be made without larger evaluation sets.
146
+
147
+ 3. **Single architecture:** Results are specific to GLM-5.2's MoE architecture. Generalization to dense models or other MoE designs is not established.
148
+
149
+ 4. **Train/serve mismatch:** Hooks are active during training but removed for inference. The model learns in a modified activation space but serves in the original space. This may contribute to the partial (not complete) prevention of refusal re-activation.
150
+
151
+ 5. **Test 3a confound:** An earlier variant (Test 3a) using the same approach achieved 1% AdvBench refusal, but AESOP achieved 16% (v1 harness). The difference could not be explained from available artifacts. The v3 harness shows AESOP at 58%, but no v3 re-run of Test 3a was performed.
152
+
153
+ 6. **No step-0 baseline:** The raw ablated base was not evaluated before LoRA training, making it difficult to isolate ablation effects from LoRA effects.
154
+
155
+ ## Audit Findings
156
+
157
+ This model was developed as part of Project AESOP, which underwent a full research audit. Key findings:
158
+
159
+ 1. **Harness inconsistency:** Earlier benchmark versions (v1, v2) used different refusal patterns, scoring logic, and token limits, producing incomparable results. The v3 harness corrects all three issues. Only v3 results should be cited.
160
+
161
+ 2. **Ablation hook code discrepancies:** Script defaults differed from the documented config, but the actual training log confirms the correct config was used (layers 62–65, coeff 0.1).
162
+
163
+ 3. **Statistical power:** n=100 is insufficient for 2–5pp claims. Only AdvBench (40pp difference) and SimpleQA (32pp difference) show effects large enough to trust.
164
+
165
+ Full audit: see `AUDIT_FINDINGS.md` in the project repository.
166
+
167
+ ## Citation
168
+
169
+ ```bibtex
170
+ @misc{aesop2026,
171
+ title={PCA-Based Refusal Ablation on MoE Models: What Survives Fine-Tuning?},
172
+ author={Fontes, C.},
173
+ year={2026},
174
+ howpublished={\url{https://huggingface.co/cfontes/GLM-5.2-Ablated-F5-Molt}},
175
+ note={Project AESOP research artifact}
176
+ }
177
+ ```
178
+
179
+ ## Acknowledgments
180
+
181
+ - GLM-5.2 base model by Z-AI
182
+ - Fable 5 training traces by Anthropic
183
+ - Benchmark harness inspired by Arditi et al. (2024) directional ablation methodology
184
+ - Compute provided by Vast.ai (8× H200 instance)
REPRODUCE.md ADDED
@@ -0,0 +1,93 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Reproduction Guide — GLM-5.2 Ablation Research / Project AESOP
2
+
3
+ ## Prerequisites
4
+ - 8× H200 SXM (141GB each) or equivalent (~1130GB total GPU memory)
5
+ - GLM-5.2 FP8 base weights (HuggingFace or local)
6
+ - Fable 5 training data: fable5-chatml.jsonl (4,876 examples, 50MB)
7
+ - Refusal PCA directions: refusal_pca.pt (2.9MB, 41 layers × 3 PCA components × 6144 hidden dim)
8
+ - Layer indices: layer_indices.json (layers 25-65)
9
+ - AdvBench prompts: advbench.csv (81KB, 100 test prompts)
10
+ - Python 3.10+, PyTorch 2.x, transformers, peft, datasets (HuggingFace)
11
+
12
+ ## Archived Artifacts (this package)
13
+ ```
14
+ artifacts/
15
+ 2e-snapshot/ # Full workspace snapshot from 2E (Vast.ai 8×H200)
16
+ *.py # All scripts (63 files)
17
+ *.log # All training/benchmark logs (80 files)
18
+ bench_*/ # Per-model benchmark results (18 dirs)
19
+ refusal_pca.pt # PCA refusal directions
20
+ layer_indices.json # Target layers for PCA extraction
21
+ fable5-chatml.jsonl # Training data (4,876 examples)
22
+ advbench.csv # AdvBench evaluation prompts
23
+ pipeline-status.txt # Pipeline state timeline
24
+ test3a/ # Test 3a model card + benchmark results
25
+ aesop/ # AESOP model card + benchmark results
26
+ fable5_r2/ # Fable5-R2 model card + benchmark results
27
+ COMPARISON.md # Cross-model comparison table with harness notes
28
+ STATISTICAL_ANALYSIS.md # Wilson CIs + significance tests
29
+ benchmarks/ # Unified benchmark harness (v3 — canonical)
30
+ harness.py # All 9 benchmarks, consistent scoring
31
+ run.py # CLI runner
32
+ stats.py # Statistical utilities
33
+ test_harness.py # Scoring function unit tests
34
+ README.md # Usage + methodology + known limitations
35
+ ```
36
+
37
+ ## Step 1: Extract Refusal Directions
38
+ ```bash
39
+ python3 extract_refusal_directions.py \
40
+ --model /path/to/glm52-fp8 \
41
+ --output refusal_pca.pt \
42
+ --layers 25,26,...,65
43
+ ```
44
+ Expected output: refusal_pca.pt (~2.9MB), 41 layers × 3 PCA components × 6144
45
+
46
+ ## Step 2A: Test 3a (ablation hooks during training)
47
+ ```bash
48
+ python3 train_ablated_fable5_lora.py \
49
+ --model /dev/shm/glm52-test3a-merged \
50
+ --data fable5-chatml.jsonl \
51
+ --out /workspace/checkpoints/test3a \
52
+ --ablation_layers 62,63,64,65 \
53
+ --ablation_coeff 0.1 \
54
+ --rank 64 --alpha 128 --lora_min_layer 60 \
55
+ --max_seq_len 2048 --lr 2e-5 --warmup 10 \
56
+ --max_steps 610 --save_every 100 --log_every 1 \
57
+ --seed 42
58
+ ```
59
+ Expected: ~9.5h on 8×H200, first_loss ~1.29, final_loss ~1.18
60
+
61
+ ## Step 2B: Fable5-R2 (no hooks during training)
62
+ Same as 2A but WITHOUT --ablation_layers / --ablation_coeff (or set coeff to 0).
63
+
64
+ ## Step 2C: AESOP (same as 2A — this IS the AESOP full training)
65
+ Identical config to Step 2A.
66
+
67
+ ## Step 3: Merge
68
+ ```bash
69
+ python3 merge_aesop.py # or merge_lora_bf16.py
70
+ ```
71
+
72
+ ## Step 4: Serve + Benchmark
73
+ ```bash
74
+ # Serve with vLLM
75
+ vllm serve /path/to/merged/model --tensor-parallel-size 8 --max-model-len 4096
76
+
77
+ # Run unified harness
78
+ python3 benchmarks/run.py --model <model-name> --output bench_<variant>/
79
+ ```
80
+
81
+ ## Step 5: Compare Results
82
+ ```bash
83
+ python3 benchmarks/stats.py --compare bench_test3a/ bench_aesop/ bench_fable5_r2/
84
+ ```
85
+
86
+ ## Known Issues
87
+ 1. The harness versions were inconsistent in the original research (v1 vs v2). Use ONLY benchmarks/harness.py (v3) for clean comparisons.
88
+ 2. MMLU-Pro and GSM8K sample n=100 — too small for 5pp significance. See STATISTICAL_ANALYSIS.md.
89
+ 3. Test 3a's exact training command was not logged at the time. The config in this guide is reconstructed from the AESOP full training log (same approach, same base, same LoRA config).
90
+ 4. The step-0 baseline (raw ablated base with no LoRA) was not run. This baseline is important for establishing clean separation between base-model ablation and LoRA effects.
91
+
92
+ ## Semantic Version
93
+ Artifact package v1.0 — created 2026-06-23
STATISTICAL_ANALYSIS.md ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Statistical Analysis — Wilson 95% CIs
2
+
3
+ ## Method
4
+ Wilson score intervals for binomial proportions. Reports (lower, upper) at 95% confidence.
5
+
6
+ ## Test 3a
7
+ | Benchmark | n | k | Rate | 95% CI |
8
+ |-----------|---|---|------|--------|
9
+ | AdvBench | 100 | 1 | 1.0% | [0.2%, 5.4%] |
10
+ | HumanEval | 164 | 111 | 67.7% | [60.0%, 74.6%] |
11
+ | MMLU-Pro | 100 | 82 | 82.0% | [73.1%, 88.5%] |
12
+ | SimpleQA | 50 | 44 | 88.0% | [75.5%, 94.8%] |
13
+ | GSM8K | 100 | 91 | 91.0% | [83.6%, 95.4%] |
14
+
15
+ ## AESOP
16
+ | Benchmark | n | k | Rate | 95% CI |
17
+ |-----------|---|---|------|--------|
18
+ | AdvBench | 100 | 16 | 16.0% | [10.0%, 24.6%] |
19
+ | HumanEval | 164 | 113 | 68.9% | [61.3%, 75.7%] |
20
+ | MMLU-Pro | 100 | 82 | 82.0% | [73.1%, 88.5%] |
21
+ | SimpleQA | 50 | 28 | 56.0% | [41.7%, 69.3%] |
22
+ | GSM8K | 100 | 94 | 94.0% | [87.4%, 97.4%] |
23
+
24
+ ## Fable5-R2
25
+ | Benchmark | n | k | Rate | 95% CI |
26
+ |-----------|---|---|------|--------|
27
+ | AdvBench | 100 | 28 | 28.0% | [19.9%, 37.8%] |
28
+ | HumanEval | 164 | 115 | 70.1% | [62.5%, 76.8%] |
29
+ | MMLU-Pro | 100 | 84 | 84.0% | [75.3%, 90.1%] |
30
+ | SimpleQA | 50 | 28 | 56.0% | [41.7%, 69.3%] |
31
+ | GSM8K | 100 | 94 | 94.0% | [87.4%, 97.4%] |
32
+
33
+ ## Significance Tests (Test 3a vs AESOP)
34
+
35
+ | Benchmark | Δ | Significant? | Rationale |
36
+ |-----------|---|-------------|-----------|
37
+ | AdvBench | +15pp | **YES** | CIs don't overlap: [0.2-5.4%] vs [10.0-24.6%] |
38
+ | HumanEval | +1.2pp | **NO** | CIs overlap substantially: [60.0-74.6%] vs [61.3-75.7%] |
39
+ | MMLU-Pro | 0pp | **NO** | Identical |
40
+ | SimpleQA | -32pp | **YES** | CIs don't overlap: [75.5-94.8%] vs [41.7-69.3%] |
41
+ | GSM8K | +3pp | **NO** | CIs overlap: [83.6-95.4%] vs [87.4-97.4%] |
42
+
43
+ ## Significance Tests (Test 3a vs Fable5-R2)
44
+
45
+ | Benchmark | Δ | Significant? | Rationale |
46
+ |-----------|---|-------------|-----------|
47
+ | AdvBench | +27pp | **YES** | CIs don't overlap |
48
+ | HumanEval | +2.4pp | **NO** | CIs overlap substantially |
49
+ | MMLU-Pro | +2pp | **NO** | CIs overlap |
50
+ | SimpleQA | -32pp | **YES** | CIs don't overlap |
51
+ | GSM8K | +3pp | **NO** | CIs overlap |
52
+
53
+ ## Key Statistical Findings
54
+
55
+ 1. **Only TWO metrics show statistically significant differences between ANY pair:**
56
+ - AdvBench (refusal rate): Test3a 1% vs AESOP 16% vs Fable5-R2 28%
57
+ - SimpleQA (knowledge): Test3a 88% vs AESOP/Fable5-R2 56%
58
+
59
+ 2. **HumanEval, MMLU-Pro, and GSM8K differences are within noise** for ALL model pairs.
60
+ Claims like "AESOP improved GSM8K by 3pp" or "Fable5-R2 improved MMLU by 2pp" are
61
+ NOT supported by the data at n=100-164.
62
+
63
+ 3. **The SimpleQA collapse (88%→56%) is the most alarming finding** — it's large, significant,
64
+ and consistent across BOTH Fable5-R2 and AESOP (both hit exactly 56%). This suggests
65
+ the LoRA training itself damages knowledge retrieval, regardless of hooks. The hooks
66
+ don't prevent it.
67
+
68
+ 4. **To achieve 5pp significance on MMLU-Pro, you'd need n≈600-1000 questions.**
69
+ The current n=100 is adequate for detecting ≥15pp differences but not the 2-5pp
70
+ differences being discussed.
chat_template.jinja ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [gMASK]<sop>
2
+ {%- set effective_reasoning_effort = 'high' if reasoning_effort is defined and reasoning_effort == 'high' else 'max' -%}
3
+ {%- if (enable_thinking is not defined or enable_thinking) and effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
4
+ {%- if tools -%}
5
+ {%- macro tool_to_json(tool) -%}
6
+ {%- set ns_tool = namespace(first=true) -%}
7
+ {{ '{' -}}
8
+ {%- for k, v in tool.items() -%}
9
+ {%- if k != 'defer_loading' and k != 'strict' -%}
10
+ {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
11
+ {%- set ns_tool.first = false -%}
12
+ "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
13
+ {%- endif -%}
14
+ {%- endfor -%}
15
+ {{- '}' -}}
16
+ {%- endmacro -%}
17
+ <|system|>
18
+ # Tools
19
+
20
+ You may call one or more functions to assist with the user query.
21
+
22
+ You are provided with function signatures within <tools></tools> XML tags:
23
+ <tools>
24
+ {% for tool in tools %}
25
+ {%- if 'function' in tool -%}
26
+ {%- set tool = tool['function'] -%}
27
+ {%- endif -%}
28
+ {% if tool.defer_loading is not defined or not tool.defer_loading %}
29
+ {{ tool_to_json(tool) }}
30
+ {% endif %}
31
+ {% endfor %}
32
+ </tools>
33
+
34
+ For each function call, output the function name and arguments within the following XML format:
35
+ <tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
36
+ {%- macro visible_text(content) -%}
37
+ {%- if content is string -%}
38
+ {{- content }}
39
+ {%- elif content is iterable and content is not mapping -%}
40
+ {%- for item in content -%}
41
+ {%- if item is mapping and item.type == 'text' -%}
42
+ {{- item.text }}
43
+ {%- elif item is string -%}
44
+ {{- item }}
45
+ {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
46
+ {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
47
+ {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
48
+ {%- endif -%}
49
+ {%- endfor -%}
50
+ {%- else -%}
51
+ {{- content }}
52
+ {%- endif -%}
53
+ {%- endmacro -%}
54
+ {%- set ns = namespace(last_user_index=-1) -%}
55
+ {%- for m in messages %}
56
+ {%- if m.role == 'user' %}
57
+ {%- set ns.last_user_index = loop.index0 -%}
58
+ {%- endif %}
59
+ {%- endfor %}
60
+ {%- for m in messages -%}
61
+ {%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
62
+ {%- elif m.role == 'assistant' -%}
63
+ <|assistant|>
64
+ {%- set content = visible_text(m.content) %}
65
+ {%- if m.reasoning_content is string %}
66
+ {%- set reasoning_content = m.reasoning_content %}
67
+ {%- elif '</think>' in content %}
68
+ {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
69
+ {%- set content = content.split('</think>')[-1] %}
70
+ {%- endif %}
71
+ {%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
72
+ {{ '<think>' + reasoning_content + '</think>'}}
73
+ {%- else -%}
74
+ {{ '<think></think>' }}
75
+ {%- endif -%}
76
+ {%- if content.strip() -%}
77
+ {{ content.strip() }}
78
+ {%- endif -%}
79
+ {% if m.tool_calls %}
80
+ {% for tc in m.tool_calls %}
81
+ {%- if tc.function %}
82
+ {%- set tc = tc.function %}
83
+ {%- endif %}
84
+ {{- '<tool_call>' + tc.name -}}
85
+ {% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
86
+ {% endif %}
87
+ {%- elif m.role == 'tool' -%}
88
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
89
+ {{- '<|observation|>' -}}
90
+ {%- endif %}
91
+ {%- if m.content is string -%}
92
+ {{- '<tool_response>' + m.content + '</tool_response>' -}}
93
+ {%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0.type == "tool_reference" -%}
94
+ {{- '<tool_response><tools>\n' -}}
95
+ {% for tr in m.content %}
96
+ {%- for tool in tools -%}
97
+ {%- if 'function' in tool -%}
98
+ {%- set tool = tool['function'] -%}
99
+ {%- endif -%}
100
+ {%- if tool.name == tr.name -%}
101
+ {{- tool_to_json(tool) + '\n' -}}
102
+ {%- endif -%}
103
+ {%- endfor -%}
104
+ {%- endfor -%}
105
+ {{- '</tools></tool_response>' -}}
106
+ {%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0 is mapping and m.content.0.output is defined -%}
107
+ {%- for tr in m.content -%}
108
+ {{- '<tool_response>' + tr.output + '</tool_response>' -}}
109
+ {%- endfor -%}
110
+ {%- else -%}
111
+ {{- '<tool_response>' + visible_text(m.content) + '</tool_response>' -}}
112
+ {% endif -%}
113
+ {%- elif m.role == 'system' -%}
114
+ <|system|>{{ visible_text(m.content) }}
115
+ {%- endif -%}
116
+ {%- endfor -%}
117
+ {%- if add_generation_prompt -%}
118
+ <|assistant|>{{- '<think></think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
119
+ {%- endif -%}
config.json ADDED
@@ -0,0 +1,949 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "GlmMoeDsaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 0,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": [
10
+ 154820,
11
+ 154827,
12
+ 154829
13
+ ],
14
+ "ep_size": 1,
15
+ "first_k_dense_replace": 3,
16
+ "head_dim": 64,
17
+ "hidden_act": "silu",
18
+ "hidden_size": 6144,
19
+ "index_head_dim": 128,
20
+ "index_n_heads": 32,
21
+ "index_share_for_mtp_iteration": true,
22
+ "index_skip_topk_offset": 3,
23
+ "index_topk": 2048,
24
+ "index_topk_freq": 4,
25
+ "index_topk_pattern": null,
26
+ "indexer_rope_interleave": true,
27
+ "indexer_types": [
28
+ "full",
29
+ "full",
30
+ "full",
31
+ "shared",
32
+ "shared",
33
+ "shared",
34
+ "full",
35
+ "shared",
36
+ "shared",
37
+ "shared",
38
+ "full",
39
+ "shared",
40
+ "shared",
41
+ "shared",
42
+ "full",
43
+ "shared",
44
+ "shared",
45
+ "shared",
46
+ "full",
47
+ "shared",
48
+ "shared",
49
+ "shared",
50
+ "full",
51
+ "shared",
52
+ "shared",
53
+ "shared",
54
+ "full",
55
+ "shared",
56
+ "shared",
57
+ "shared",
58
+ "full",
59
+ "shared",
60
+ "shared",
61
+ "shared",
62
+ "full",
63
+ "shared",
64
+ "shared",
65
+ "shared",
66
+ "full",
67
+ "shared",
68
+ "shared",
69
+ "shared",
70
+ "full",
71
+ "shared",
72
+ "shared",
73
+ "shared",
74
+ "full",
75
+ "shared",
76
+ "shared",
77
+ "shared",
78
+ "full",
79
+ "shared",
80
+ "shared",
81
+ "shared",
82
+ "full",
83
+ "shared",
84
+ "shared",
85
+ "shared",
86
+ "full",
87
+ "shared",
88
+ "shared",
89
+ "shared",
90
+ "full",
91
+ "shared",
92
+ "shared",
93
+ "shared",
94
+ "full",
95
+ "shared",
96
+ "shared",
97
+ "shared",
98
+ "full",
99
+ "shared",
100
+ "shared",
101
+ "shared",
102
+ "full",
103
+ "shared",
104
+ "shared",
105
+ "shared"
106
+ ],
107
+ "initializer_range": 0.02,
108
+ "intermediate_size": 12288,
109
+ "kv_lora_rank": 512,
110
+ "layer_types": [
111
+ "deepseek_sparse_attention",
112
+ "deepseek_sparse_attention",
113
+ "deepseek_sparse_attention",
114
+ "deepseek_sparse_attention",
115
+ "deepseek_sparse_attention",
116
+ "deepseek_sparse_attention",
117
+ "deepseek_sparse_attention",
118
+ "deepseek_sparse_attention",
119
+ "deepseek_sparse_attention",
120
+ "deepseek_sparse_attention",
121
+ "deepseek_sparse_attention",
122
+ "deepseek_sparse_attention",
123
+ "deepseek_sparse_attention",
124
+ "deepseek_sparse_attention",
125
+ "deepseek_sparse_attention",
126
+ "deepseek_sparse_attention",
127
+ "deepseek_sparse_attention",
128
+ "deepseek_sparse_attention",
129
+ "deepseek_sparse_attention",
130
+ "deepseek_sparse_attention",
131
+ "deepseek_sparse_attention",
132
+ "deepseek_sparse_attention",
133
+ "deepseek_sparse_attention",
134
+ "deepseek_sparse_attention",
135
+ "deepseek_sparse_attention",
136
+ "deepseek_sparse_attention",
137
+ "deepseek_sparse_attention",
138
+ "deepseek_sparse_attention",
139
+ "deepseek_sparse_attention",
140
+ "deepseek_sparse_attention",
141
+ "deepseek_sparse_attention",
142
+ "deepseek_sparse_attention",
143
+ "deepseek_sparse_attention",
144
+ "deepseek_sparse_attention",
145
+ "deepseek_sparse_attention",
146
+ "deepseek_sparse_attention",
147
+ "deepseek_sparse_attention",
148
+ "deepseek_sparse_attention",
149
+ "deepseek_sparse_attention",
150
+ "deepseek_sparse_attention",
151
+ "deepseek_sparse_attention",
152
+ "deepseek_sparse_attention",
153
+ "deepseek_sparse_attention",
154
+ "deepseek_sparse_attention",
155
+ "deepseek_sparse_attention",
156
+ "deepseek_sparse_attention",
157
+ "deepseek_sparse_attention",
158
+ "deepseek_sparse_attention",
159
+ "deepseek_sparse_attention",
160
+ "deepseek_sparse_attention",
161
+ "deepseek_sparse_attention",
162
+ "deepseek_sparse_attention",
163
+ "deepseek_sparse_attention",
164
+ "deepseek_sparse_attention",
165
+ "deepseek_sparse_attention",
166
+ "deepseek_sparse_attention",
167
+ "deepseek_sparse_attention",
168
+ "deepseek_sparse_attention",
169
+ "deepseek_sparse_attention",
170
+ "deepseek_sparse_attention",
171
+ "deepseek_sparse_attention",
172
+ "deepseek_sparse_attention",
173
+ "deepseek_sparse_attention",
174
+ "deepseek_sparse_attention",
175
+ "deepseek_sparse_attention",
176
+ "deepseek_sparse_attention",
177
+ "deepseek_sparse_attention",
178
+ "deepseek_sparse_attention",
179
+ "deepseek_sparse_attention",
180
+ "deepseek_sparse_attention",
181
+ "deepseek_sparse_attention",
182
+ "deepseek_sparse_attention",
183
+ "deepseek_sparse_attention",
184
+ "deepseek_sparse_attention",
185
+ "deepseek_sparse_attention",
186
+ "deepseek_sparse_attention",
187
+ "deepseek_sparse_attention",
188
+ "deepseek_sparse_attention"
189
+ ],
190
+ "max_position_embeddings": 1048576,
191
+ "mlp_bias": false,
192
+ "mlp_layer_types": [
193
+ "dense",
194
+ "dense",
195
+ "dense",
196
+ "sparse",
197
+ "sparse",
198
+ "sparse",
199
+ "sparse",
200
+ "sparse",
201
+ "sparse",
202
+ "sparse",
203
+ "sparse",
204
+ "sparse",
205
+ "sparse",
206
+ "sparse",
207
+ "sparse",
208
+ "sparse",
209
+ "sparse",
210
+ "sparse",
211
+ "sparse",
212
+ "sparse",
213
+ "sparse",
214
+ "sparse",
215
+ "sparse",
216
+ "sparse",
217
+ "sparse",
218
+ "sparse",
219
+ "sparse",
220
+ "sparse",
221
+ "sparse",
222
+ "sparse",
223
+ "sparse",
224
+ "sparse",
225
+ "sparse",
226
+ "sparse",
227
+ "sparse",
228
+ "sparse",
229
+ "sparse",
230
+ "sparse",
231
+ "sparse",
232
+ "sparse",
233
+ "sparse",
234
+ "sparse",
235
+ "sparse",
236
+ "sparse",
237
+ "sparse",
238
+ "sparse",
239
+ "sparse",
240
+ "sparse",
241
+ "sparse",
242
+ "sparse",
243
+ "sparse",
244
+ "sparse",
245
+ "sparse",
246
+ "sparse",
247
+ "sparse",
248
+ "sparse",
249
+ "sparse",
250
+ "sparse",
251
+ "sparse",
252
+ "sparse",
253
+ "sparse",
254
+ "sparse",
255
+ "sparse",
256
+ "sparse",
257
+ "sparse",
258
+ "sparse",
259
+ "sparse",
260
+ "sparse",
261
+ "sparse",
262
+ "sparse",
263
+ "sparse",
264
+ "sparse",
265
+ "sparse",
266
+ "sparse",
267
+ "sparse",
268
+ "sparse",
269
+ "sparse",
270
+ "sparse"
271
+ ],
272
+ "model_type": "glm_moe_dsa",
273
+ "moe_intermediate_size": 2048,
274
+ "moe_layer_freq": 1,
275
+ "n_group": 1,
276
+ "n_routed_experts": 256,
277
+ "n_shared_experts": 1,
278
+ "norm_topk_prob": true,
279
+ "num_attention_heads": 64,
280
+ "num_experts": 256,
281
+ "num_experts_per_tok": 8,
282
+ "num_hidden_layers": 78,
283
+ "num_key_value_heads": 64,
284
+ "num_nextn_predict_layers": 1,
285
+ "pad_token_id": 154820,
286
+ "pretraining_tp": 1,
287
+ "q_lora_rank": 2048,
288
+ "qk_head_dim": 256,
289
+ "qk_nope_head_dim": 192,
290
+ "qk_rope_head_dim": 64,
291
+ "quantization_config": {
292
+ "activation_scheme": "dynamic",
293
+ "dequantize": false,
294
+ "modules_to_not_convert": [
295
+ "lm_head",
296
+ "model.embed_tokens",
297
+ "model.layers.0.input_layernorm",
298
+ "model.layers.0.post_attention_layernorm",
299
+ "model.layers.0.self_attn.indexer.k_norm",
300
+ "model.layers.0.self_attn.indexer.k_norm.bias",
301
+ "model.layers.0.self_attn.indexers_proj",
302
+ "model.layers.0.self_attn.kv_a_layernorm",
303
+ "model.layers.0.self_attn.q_a_layernorm",
304
+ "model.layers.1.input_layernorm",
305
+ "model.layers.1.post_attention_layernorm",
306
+ "model.layers.1.self_attn.indexer.k_norm",
307
+ "model.layers.1.self_attn.indexer.k_norm.bias",
308
+ "model.layers.1.self_attn.indexers_proj",
309
+ "model.layers.1.self_attn.kv_a_layernorm",
310
+ "model.layers.1.self_attn.q_a_layernorm",
311
+ "model.layers.10.input_layernorm",
312
+ "model.layers.10.mlp.gate",
313
+ "model.layers.10.mlp.gate.e_score_correction_bias",
314
+ "model.layers.10.post_attention_layernorm",
315
+ "model.layers.10.self_attn.indexer.k_norm",
316
+ "model.layers.10.self_attn.indexer.k_norm.bias",
317
+ "model.layers.10.self_attn.indexers_proj",
318
+ "model.layers.10.self_attn.kv_a_layernorm",
319
+ "model.layers.10.self_attn.q_a_layernorm",
320
+ "model.layers.11.input_layernorm",
321
+ "model.layers.11.mlp.gate",
322
+ "model.layers.11.mlp.gate.e_score_correction_bias",
323
+ "model.layers.11.post_attention_layernorm",
324
+ "model.layers.11.self_attn.kv_a_layernorm",
325
+ "model.layers.11.self_attn.q_a_layernorm",
326
+ "model.layers.12.input_layernorm",
327
+ "model.layers.12.mlp.gate",
328
+ "model.layers.12.mlp.gate.e_score_correction_bias",
329
+ "model.layers.12.post_attention_layernorm",
330
+ "model.layers.12.self_attn.kv_a_layernorm",
331
+ "model.layers.12.self_attn.q_a_layernorm",
332
+ "model.layers.13.input_layernorm",
333
+ "model.layers.13.mlp.gate",
334
+ "model.layers.13.mlp.gate.e_score_correction_bias",
335
+ "model.layers.13.post_attention_layernorm",
336
+ "model.layers.13.self_attn.kv_a_layernorm",
337
+ "model.layers.13.self_attn.q_a_layernorm",
338
+ "model.layers.14.input_layernorm",
339
+ "model.layers.14.mlp.gate",
340
+ "model.layers.14.mlp.gate.e_score_correction_bias",
341
+ "model.layers.14.post_attention_layernorm",
342
+ "model.layers.14.self_attn.indexer.k_norm",
343
+ "model.layers.14.self_attn.indexer.k_norm.bias",
344
+ "model.layers.14.self_attn.indexers_proj",
345
+ "model.layers.14.self_attn.kv_a_layernorm",
346
+ "model.layers.14.self_attn.q_a_layernorm",
347
+ "model.layers.15.input_layernorm",
348
+ "model.layers.15.mlp.gate",
349
+ "model.layers.15.mlp.gate.e_score_correction_bias",
350
+ "model.layers.15.post_attention_layernorm",
351
+ "model.layers.15.self_attn.kv_a_layernorm",
352
+ "model.layers.15.self_attn.q_a_layernorm",
353
+ "model.layers.16.input_layernorm",
354
+ "model.layers.16.mlp.gate",
355
+ "model.layers.16.mlp.gate.e_score_correction_bias",
356
+ "model.layers.16.post_attention_layernorm",
357
+ "model.layers.16.self_attn.kv_a_layernorm",
358
+ "model.layers.16.self_attn.q_a_layernorm",
359
+ "model.layers.17.input_layernorm",
360
+ "model.layers.17.mlp.gate",
361
+ "model.layers.17.mlp.gate.e_score_correction_bias",
362
+ "model.layers.17.post_attention_layernorm",
363
+ "model.layers.17.self_attn.kv_a_layernorm",
364
+ "model.layers.17.self_attn.q_a_layernorm",
365
+ "model.layers.18.input_layernorm",
366
+ "model.layers.18.mlp.gate",
367
+ "model.layers.18.mlp.gate.e_score_correction_bias",
368
+ "model.layers.18.post_attention_layernorm",
369
+ "model.layers.18.self_attn.indexer.k_norm",
370
+ "model.layers.18.self_attn.indexer.k_norm.bias",
371
+ "model.layers.18.self_attn.indexers_proj",
372
+ "model.layers.18.self_attn.kv_a_layernorm",
373
+ "model.layers.18.self_attn.q_a_layernorm",
374
+ "model.layers.19.input_layernorm",
375
+ "model.layers.19.mlp.gate",
376
+ "model.layers.19.mlp.gate.e_score_correction_bias",
377
+ "model.layers.19.post_attention_layernorm",
378
+ "model.layers.19.self_attn.kv_a_layernorm",
379
+ "model.layers.19.self_attn.q_a_layernorm",
380
+ "model.layers.2.input_layernorm",
381
+ "model.layers.2.post_attention_layernorm",
382
+ "model.layers.2.self_attn.indexer.k_norm",
383
+ "model.layers.2.self_attn.indexer.k_norm.bias",
384
+ "model.layers.2.self_attn.indexers_proj",
385
+ "model.layers.2.self_attn.kv_a_layernorm",
386
+ "model.layers.2.self_attn.q_a_layernorm",
387
+ "model.layers.20.input_layernorm",
388
+ "model.layers.20.mlp.gate",
389
+ "model.layers.20.mlp.gate.e_score_correction_bias",
390
+ "model.layers.20.post_attention_layernorm",
391
+ "model.layers.20.self_attn.kv_a_layernorm",
392
+ "model.layers.20.self_attn.q_a_layernorm",
393
+ "model.layers.21.input_layernorm",
394
+ "model.layers.21.mlp.gate",
395
+ "model.layers.21.mlp.gate.e_score_correction_bias",
396
+ "model.layers.21.post_attention_layernorm",
397
+ "model.layers.21.self_attn.kv_a_layernorm",
398
+ "model.layers.21.self_attn.q_a_layernorm",
399
+ "model.layers.22.input_layernorm",
400
+ "model.layers.22.mlp.gate",
401
+ "model.layers.22.mlp.gate.e_score_correction_bias",
402
+ "model.layers.22.post_attention_layernorm",
403
+ "model.layers.22.self_attn.indexer.k_norm",
404
+ "model.layers.22.self_attn.indexer.k_norm.bias",
405
+ "model.layers.22.self_attn.indexers_proj",
406
+ "model.layers.22.self_attn.kv_a_layernorm",
407
+ "model.layers.22.self_attn.q_a_layernorm",
408
+ "model.layers.23.input_layernorm",
409
+ "model.layers.23.mlp.gate",
410
+ "model.layers.23.mlp.gate.e_score_correction_bias",
411
+ "model.layers.23.post_attention_layernorm",
412
+ "model.layers.23.self_attn.kv_a_layernorm",
413
+ "model.layers.23.self_attn.q_a_layernorm",
414
+ "model.layers.24.input_layernorm",
415
+ "model.layers.24.mlp.gate",
416
+ "model.layers.24.mlp.gate.e_score_correction_bias",
417
+ "model.layers.24.post_attention_layernorm",
418
+ "model.layers.24.self_attn.kv_a_layernorm",
419
+ "model.layers.24.self_attn.q_a_layernorm",
420
+ "model.layers.25.input_layernorm",
421
+ "model.layers.25.mlp.gate",
422
+ "model.layers.25.mlp.gate.e_score_correction_bias",
423
+ "model.layers.25.post_attention_layernorm",
424
+ "model.layers.25.self_attn.kv_a_layernorm",
425
+ "model.layers.25.self_attn.q_a_layernorm",
426
+ "model.layers.26.input_layernorm",
427
+ "model.layers.26.mlp.gate",
428
+ "model.layers.26.mlp.gate.e_score_correction_bias",
429
+ "model.layers.26.post_attention_layernorm",
430
+ "model.layers.26.self_attn.indexer.k_norm",
431
+ "model.layers.26.self_attn.indexer.k_norm.bias",
432
+ "model.layers.26.self_attn.indexers_proj",
433
+ "model.layers.26.self_attn.kv_a_layernorm",
434
+ "model.layers.26.self_attn.q_a_layernorm",
435
+ "model.layers.27.input_layernorm",
436
+ "model.layers.27.mlp.gate",
437
+ "model.layers.27.mlp.gate.e_score_correction_bias",
438
+ "model.layers.27.post_attention_layernorm",
439
+ "model.layers.27.self_attn.kv_a_layernorm",
440
+ "model.layers.27.self_attn.q_a_layernorm",
441
+ "model.layers.28.input_layernorm",
442
+ "model.layers.28.mlp.gate",
443
+ "model.layers.28.mlp.gate.e_score_correction_bias",
444
+ "model.layers.28.post_attention_layernorm",
445
+ "model.layers.28.self_attn.kv_a_layernorm",
446
+ "model.layers.28.self_attn.q_a_layernorm",
447
+ "model.layers.29.input_layernorm",
448
+ "model.layers.29.mlp.gate",
449
+ "model.layers.29.mlp.gate.e_score_correction_bias",
450
+ "model.layers.29.post_attention_layernorm",
451
+ "model.layers.29.self_attn.kv_a_layernorm",
452
+ "model.layers.29.self_attn.q_a_layernorm",
453
+ "model.layers.3.input_layernorm",
454
+ "model.layers.3.mlp.gate",
455
+ "model.layers.3.mlp.gate.e_score_correction_bias",
456
+ "model.layers.3.post_attention_layernorm",
457
+ "model.layers.3.self_attn.kv_a_layernorm",
458
+ "model.layers.3.self_attn.q_a_layernorm",
459
+ "model.layers.30.input_layernorm",
460
+ "model.layers.30.mlp.gate",
461
+ "model.layers.30.mlp.gate.e_score_correction_bias",
462
+ "model.layers.30.post_attention_layernorm",
463
+ "model.layers.30.self_attn.indexer.k_norm",
464
+ "model.layers.30.self_attn.indexer.k_norm.bias",
465
+ "model.layers.30.self_attn.indexers_proj",
466
+ "model.layers.30.self_attn.kv_a_layernorm",
467
+ "model.layers.30.self_attn.q_a_layernorm",
468
+ "model.layers.31.input_layernorm",
469
+ "model.layers.31.mlp.gate",
470
+ "model.layers.31.mlp.gate.e_score_correction_bias",
471
+ "model.layers.31.post_attention_layernorm",
472
+ "model.layers.31.self_attn.kv_a_layernorm",
473
+ "model.layers.31.self_attn.q_a_layernorm",
474
+ "model.layers.32.input_layernorm",
475
+ "model.layers.32.mlp.gate",
476
+ "model.layers.32.mlp.gate.e_score_correction_bias",
477
+ "model.layers.32.post_attention_layernorm",
478
+ "model.layers.32.self_attn.kv_a_layernorm",
479
+ "model.layers.32.self_attn.q_a_layernorm",
480
+ "model.layers.33.input_layernorm",
481
+ "model.layers.33.mlp.gate",
482
+ "model.layers.33.mlp.gate.e_score_correction_bias",
483
+ "model.layers.33.post_attention_layernorm",
484
+ "model.layers.33.self_attn.kv_a_layernorm",
485
+ "model.layers.33.self_attn.q_a_layernorm",
486
+ "model.layers.34.input_layernorm",
487
+ "model.layers.34.mlp.gate",
488
+ "model.layers.34.mlp.gate.e_score_correction_bias",
489
+ "model.layers.34.post_attention_layernorm",
490
+ "model.layers.34.self_attn.indexer.k_norm",
491
+ "model.layers.34.self_attn.indexer.k_norm.bias",
492
+ "model.layers.34.self_attn.indexers_proj",
493
+ "model.layers.34.self_attn.kv_a_layernorm",
494
+ "model.layers.34.self_attn.q_a_layernorm",
495
+ "model.layers.35.input_layernorm",
496
+ "model.layers.35.mlp.gate",
497
+ "model.layers.35.mlp.gate.e_score_correction_bias",
498
+ "model.layers.35.post_attention_layernorm",
499
+ "model.layers.35.self_attn.kv_a_layernorm",
500
+ "model.layers.35.self_attn.q_a_layernorm",
501
+ "model.layers.36.input_layernorm",
502
+ "model.layers.36.mlp.gate",
503
+ "model.layers.36.mlp.gate.e_score_correction_bias",
504
+ "model.layers.36.post_attention_layernorm",
505
+ "model.layers.36.self_attn.kv_a_layernorm",
506
+ "model.layers.36.self_attn.q_a_layernorm",
507
+ "model.layers.37.input_layernorm",
508
+ "model.layers.37.mlp.gate",
509
+ "model.layers.37.mlp.gate.e_score_correction_bias",
510
+ "model.layers.37.post_attention_layernorm",
511
+ "model.layers.37.self_attn.kv_a_layernorm",
512
+ "model.layers.37.self_attn.q_a_layernorm",
513
+ "model.layers.38.input_layernorm",
514
+ "model.layers.38.mlp.gate",
515
+ "model.layers.38.mlp.gate.e_score_correction_bias",
516
+ "model.layers.38.post_attention_layernorm",
517
+ "model.layers.38.self_attn.indexer.k_norm",
518
+ "model.layers.38.self_attn.indexer.k_norm.bias",
519
+ "model.layers.38.self_attn.indexers_proj",
520
+ "model.layers.38.self_attn.kv_a_layernorm",
521
+ "model.layers.38.self_attn.q_a_layernorm",
522
+ "model.layers.39.input_layernorm",
523
+ "model.layers.39.mlp.gate",
524
+ "model.layers.39.mlp.gate.e_score_correction_bias",
525
+ "model.layers.39.post_attention_layernorm",
526
+ "model.layers.39.self_attn.kv_a_layernorm",
527
+ "model.layers.39.self_attn.q_a_layernorm",
528
+ "model.layers.4.input_layernorm",
529
+ "model.layers.4.mlp.gate",
530
+ "model.layers.4.mlp.gate.e_score_correction_bias",
531
+ "model.layers.4.post_attention_layernorm",
532
+ "model.layers.4.self_attn.kv_a_layernorm",
533
+ "model.layers.4.self_attn.q_a_layernorm",
534
+ "model.layers.40.input_layernorm",
535
+ "model.layers.40.mlp.gate",
536
+ "model.layers.40.mlp.gate.e_score_correction_bias",
537
+ "model.layers.40.post_attention_layernorm",
538
+ "model.layers.40.self_attn.kv_a_layernorm",
539
+ "model.layers.40.self_attn.q_a_layernorm",
540
+ "model.layers.41.input_layernorm",
541
+ "model.layers.41.mlp.gate",
542
+ "model.layers.41.mlp.gate.e_score_correction_bias",
543
+ "model.layers.41.post_attention_layernorm",
544
+ "model.layers.41.self_attn.kv_a_layernorm",
545
+ "model.layers.41.self_attn.q_a_layernorm",
546
+ "model.layers.42.input_layernorm",
547
+ "model.layers.42.mlp.gate",
548
+ "model.layers.42.mlp.gate.e_score_correction_bias",
549
+ "model.layers.42.post_attention_layernorm",
550
+ "model.layers.42.self_attn.indexer.k_norm",
551
+ "model.layers.42.self_attn.indexer.k_norm.bias",
552
+ "model.layers.42.self_attn.indexers_proj",
553
+ "model.layers.42.self_attn.kv_a_layernorm",
554
+ "model.layers.42.self_attn.q_a_layernorm",
555
+ "model.layers.43.input_layernorm",
556
+ "model.layers.43.mlp.gate",
557
+ "model.layers.43.mlp.gate.e_score_correction_bias",
558
+ "model.layers.43.post_attention_layernorm",
559
+ "model.layers.43.self_attn.kv_a_layernorm",
560
+ "model.layers.43.self_attn.q_a_layernorm",
561
+ "model.layers.44.input_layernorm",
562
+ "model.layers.44.mlp.gate",
563
+ "model.layers.44.mlp.gate.e_score_correction_bias",
564
+ "model.layers.44.post_attention_layernorm",
565
+ "model.layers.44.self_attn.kv_a_layernorm",
566
+ "model.layers.44.self_attn.q_a_layernorm",
567
+ "model.layers.45.input_layernorm",
568
+ "model.layers.45.mlp.gate",
569
+ "model.layers.45.mlp.gate.e_score_correction_bias",
570
+ "model.layers.45.post_attention_layernorm",
571
+ "model.layers.45.self_attn.kv_a_layernorm",
572
+ "model.layers.45.self_attn.q_a_layernorm",
573
+ "model.layers.46.input_layernorm",
574
+ "model.layers.46.mlp.gate",
575
+ "model.layers.46.mlp.gate.e_score_correction_bias",
576
+ "model.layers.46.post_attention_layernorm",
577
+ "model.layers.46.self_attn.indexer.k_norm",
578
+ "model.layers.46.self_attn.indexer.k_norm.bias",
579
+ "model.layers.46.self_attn.indexers_proj",
580
+ "model.layers.46.self_attn.kv_a_layernorm",
581
+ "model.layers.46.self_attn.q_a_layernorm",
582
+ "model.layers.47.input_layernorm",
583
+ "model.layers.47.mlp.gate",
584
+ "model.layers.47.mlp.gate.e_score_correction_bias",
585
+ "model.layers.47.post_attention_layernorm",
586
+ "model.layers.47.self_attn.kv_a_layernorm",
587
+ "model.layers.47.self_attn.q_a_layernorm",
588
+ "model.layers.48.input_layernorm",
589
+ "model.layers.48.mlp.gate",
590
+ "model.layers.48.mlp.gate.e_score_correction_bias",
591
+ "model.layers.48.post_attention_layernorm",
592
+ "model.layers.48.self_attn.kv_a_layernorm",
593
+ "model.layers.48.self_attn.q_a_layernorm",
594
+ "model.layers.49.input_layernorm",
595
+ "model.layers.49.mlp.gate",
596
+ "model.layers.49.mlp.gate.e_score_correction_bias",
597
+ "model.layers.49.post_attention_layernorm",
598
+ "model.layers.49.self_attn.kv_a_layernorm",
599
+ "model.layers.49.self_attn.q_a_layernorm",
600
+ "model.layers.5.input_layernorm",
601
+ "model.layers.5.mlp.gate",
602
+ "model.layers.5.mlp.gate.e_score_correction_bias",
603
+ "model.layers.5.post_attention_layernorm",
604
+ "model.layers.5.self_attn.kv_a_layernorm",
605
+ "model.layers.5.self_attn.q_a_layernorm",
606
+ "model.layers.50.input_layernorm",
607
+ "model.layers.50.mlp.gate",
608
+ "model.layers.50.mlp.gate.e_score_correction_bias",
609
+ "model.layers.50.post_attention_layernorm",
610
+ "model.layers.50.self_attn.indexer.k_norm",
611
+ "model.layers.50.self_attn.indexer.k_norm.bias",
612
+ "model.layers.50.self_attn.indexers_proj",
613
+ "model.layers.50.self_attn.kv_a_layernorm",
614
+ "model.layers.50.self_attn.q_a_layernorm",
615
+ "model.layers.51.input_layernorm",
616
+ "model.layers.51.mlp.gate",
617
+ "model.layers.51.mlp.gate.e_score_correction_bias",
618
+ "model.layers.51.post_attention_layernorm",
619
+ "model.layers.51.self_attn.kv_a_layernorm",
620
+ "model.layers.51.self_attn.q_a_layernorm",
621
+ "model.layers.52.input_layernorm",
622
+ "model.layers.52.mlp.gate",
623
+ "model.layers.52.mlp.gate.e_score_correction_bias",
624
+ "model.layers.52.post_attention_layernorm",
625
+ "model.layers.52.self_attn.kv_a_layernorm",
626
+ "model.layers.52.self_attn.q_a_layernorm",
627
+ "model.layers.53.input_layernorm",
628
+ "model.layers.53.mlp.gate",
629
+ "model.layers.53.mlp.gate.e_score_correction_bias",
630
+ "model.layers.53.post_attention_layernorm",
631
+ "model.layers.53.self_attn.kv_a_layernorm",
632
+ "model.layers.53.self_attn.q_a_layernorm",
633
+ "model.layers.54.input_layernorm",
634
+ "model.layers.54.mlp.gate",
635
+ "model.layers.54.mlp.gate.e_score_correction_bias",
636
+ "model.layers.54.post_attention_layernorm",
637
+ "model.layers.54.self_attn.indexer.k_norm",
638
+ "model.layers.54.self_attn.indexer.k_norm.bias",
639
+ "model.layers.54.self_attn.indexers_proj",
640
+ "model.layers.54.self_attn.kv_a_layernorm",
641
+ "model.layers.54.self_attn.q_a_layernorm",
642
+ "model.layers.55.input_layernorm",
643
+ "model.layers.55.mlp.gate",
644
+ "model.layers.55.mlp.gate.e_score_correction_bias",
645
+ "model.layers.55.post_attention_layernorm",
646
+ "model.layers.55.self_attn.kv_a_layernorm",
647
+ "model.layers.55.self_attn.q_a_layernorm",
648
+ "model.layers.56.input_layernorm",
649
+ "model.layers.56.mlp.gate",
650
+ "model.layers.56.mlp.gate.e_score_correction_bias",
651
+ "model.layers.56.post_attention_layernorm",
652
+ "model.layers.56.self_attn.kv_a_layernorm",
653
+ "model.layers.56.self_attn.q_a_layernorm",
654
+ "model.layers.57.input_layernorm",
655
+ "model.layers.57.mlp.gate",
656
+ "model.layers.57.mlp.gate.e_score_correction_bias",
657
+ "model.layers.57.post_attention_layernorm",
658
+ "model.layers.57.self_attn.kv_a_layernorm",
659
+ "model.layers.57.self_attn.q_a_layernorm",
660
+ "model.layers.58.input_layernorm",
661
+ "model.layers.58.mlp.gate",
662
+ "model.layers.58.mlp.gate.e_score_correction_bias",
663
+ "model.layers.58.post_attention_layernorm",
664
+ "model.layers.58.self_attn.indexer.k_norm",
665
+ "model.layers.58.self_attn.indexer.k_norm.bias",
666
+ "model.layers.58.self_attn.indexers_proj",
667
+ "model.layers.58.self_attn.kv_a_layernorm",
668
+ "model.layers.58.self_attn.q_a_layernorm",
669
+ "model.layers.59.input_layernorm",
670
+ "model.layers.59.mlp.gate",
671
+ "model.layers.59.mlp.gate.e_score_correction_bias",
672
+ "model.layers.59.post_attention_layernorm",
673
+ "model.layers.59.self_attn.kv_a_layernorm",
674
+ "model.layers.59.self_attn.q_a_layernorm",
675
+ "model.layers.6.input_layernorm",
676
+ "model.layers.6.mlp.gate",
677
+ "model.layers.6.mlp.gate.e_score_correction_bias",
678
+ "model.layers.6.post_attention_layernorm",
679
+ "model.layers.6.self_attn.indexer.k_norm",
680
+ "model.layers.6.self_attn.indexer.k_norm.bias",
681
+ "model.layers.6.self_attn.indexers_proj",
682
+ "model.layers.6.self_attn.kv_a_layernorm",
683
+ "model.layers.6.self_attn.q_a_layernorm",
684
+ "model.layers.60.input_layernorm",
685
+ "model.layers.60.mlp.gate",
686
+ "model.layers.60.mlp.gate.e_score_correction_bias",
687
+ "model.layers.60.post_attention_layernorm",
688
+ "model.layers.60.self_attn.kv_a_layernorm",
689
+ "model.layers.60.self_attn.kv_a_proj_with_mqa",
690
+ "model.layers.60.self_attn.kv_b_proj",
691
+ "model.layers.60.self_attn.o_proj",
692
+ "model.layers.60.self_attn.q_a_layernorm",
693
+ "model.layers.60.self_attn.q_a_proj",
694
+ "model.layers.60.self_attn.q_b_proj",
695
+ "model.layers.61.input_layernorm",
696
+ "model.layers.61.mlp.gate",
697
+ "model.layers.61.mlp.gate.e_score_correction_bias",
698
+ "model.layers.61.post_attention_layernorm",
699
+ "model.layers.61.self_attn.kv_a_layernorm",
700
+ "model.layers.61.self_attn.kv_a_proj_with_mqa",
701
+ "model.layers.61.self_attn.kv_b_proj",
702
+ "model.layers.61.self_attn.o_proj",
703
+ "model.layers.61.self_attn.q_a_layernorm",
704
+ "model.layers.61.self_attn.q_a_proj",
705
+ "model.layers.61.self_attn.q_b_proj",
706
+ "model.layers.62.input_layernorm",
707
+ "model.layers.62.mlp.gate",
708
+ "model.layers.62.mlp.gate.e_score_correction_bias",
709
+ "model.layers.62.post_attention_layernorm",
710
+ "model.layers.62.self_attn.indexer.k_norm",
711
+ "model.layers.62.self_attn.indexer.k_norm.bias",
712
+ "model.layers.62.self_attn.indexers_proj",
713
+ "model.layers.62.self_attn.kv_a_layernorm",
714
+ "model.layers.62.self_attn.kv_a_proj_with_mqa",
715
+ "model.layers.62.self_attn.kv_b_proj",
716
+ "model.layers.62.self_attn.o_proj",
717
+ "model.layers.62.self_attn.q_a_layernorm",
718
+ "model.layers.62.self_attn.q_a_proj",
719
+ "model.layers.62.self_attn.q_b_proj",
720
+ "model.layers.63.input_layernorm",
721
+ "model.layers.63.mlp.gate",
722
+ "model.layers.63.mlp.gate.e_score_correction_bias",
723
+ "model.layers.63.post_attention_layernorm",
724
+ "model.layers.63.self_attn.kv_a_layernorm",
725
+ "model.layers.63.self_attn.kv_a_proj_with_mqa",
726
+ "model.layers.63.self_attn.kv_b_proj",
727
+ "model.layers.63.self_attn.o_proj",
728
+ "model.layers.63.self_attn.q_a_layernorm",
729
+ "model.layers.63.self_attn.q_a_proj",
730
+ "model.layers.63.self_attn.q_b_proj",
731
+ "model.layers.64.input_layernorm",
732
+ "model.layers.64.mlp.gate",
733
+ "model.layers.64.mlp.gate.e_score_correction_bias",
734
+ "model.layers.64.post_attention_layernorm",
735
+ "model.layers.64.self_attn.kv_a_layernorm",
736
+ "model.layers.64.self_attn.kv_a_proj_with_mqa",
737
+ "model.layers.64.self_attn.kv_b_proj",
738
+ "model.layers.64.self_attn.o_proj",
739
+ "model.layers.64.self_attn.q_a_layernorm",
740
+ "model.layers.64.self_attn.q_a_proj",
741
+ "model.layers.64.self_attn.q_b_proj",
742
+ "model.layers.65.input_layernorm",
743
+ "model.layers.65.mlp.gate",
744
+ "model.layers.65.mlp.gate.e_score_correction_bias",
745
+ "model.layers.65.post_attention_layernorm",
746
+ "model.layers.65.self_attn.kv_a_layernorm",
747
+ "model.layers.65.self_attn.kv_a_proj_with_mqa",
748
+ "model.layers.65.self_attn.kv_b_proj",
749
+ "model.layers.65.self_attn.o_proj",
750
+ "model.layers.65.self_attn.q_a_layernorm",
751
+ "model.layers.65.self_attn.q_a_proj",
752
+ "model.layers.65.self_attn.q_b_proj",
753
+ "model.layers.66.input_layernorm",
754
+ "model.layers.66.mlp.gate",
755
+ "model.layers.66.mlp.gate.e_score_correction_bias",
756
+ "model.layers.66.post_attention_layernorm",
757
+ "model.layers.66.self_attn.indexer.k_norm",
758
+ "model.layers.66.self_attn.indexer.k_norm.bias",
759
+ "model.layers.66.self_attn.indexers_proj",
760
+ "model.layers.66.self_attn.kv_a_layernorm",
761
+ "model.layers.66.self_attn.kv_a_proj_with_mqa",
762
+ "model.layers.66.self_attn.kv_b_proj",
763
+ "model.layers.66.self_attn.o_proj",
764
+ "model.layers.66.self_attn.q_a_layernorm",
765
+ "model.layers.66.self_attn.q_a_proj",
766
+ "model.layers.66.self_attn.q_b_proj",
767
+ "model.layers.67.input_layernorm",
768
+ "model.layers.67.mlp.gate",
769
+ "model.layers.67.mlp.gate.e_score_correction_bias",
770
+ "model.layers.67.post_attention_layernorm",
771
+ "model.layers.67.self_attn.kv_a_layernorm",
772
+ "model.layers.67.self_attn.kv_a_proj_with_mqa",
773
+ "model.layers.67.self_attn.kv_b_proj",
774
+ "model.layers.67.self_attn.o_proj",
775
+ "model.layers.67.self_attn.q_a_layernorm",
776
+ "model.layers.67.self_attn.q_a_proj",
777
+ "model.layers.67.self_attn.q_b_proj",
778
+ "model.layers.68.input_layernorm",
779
+ "model.layers.68.mlp.gate",
780
+ "model.layers.68.mlp.gate.e_score_correction_bias",
781
+ "model.layers.68.post_attention_layernorm",
782
+ "model.layers.68.self_attn.kv_a_layernorm",
783
+ "model.layers.68.self_attn.kv_a_proj_with_mqa",
784
+ "model.layers.68.self_attn.kv_b_proj",
785
+ "model.layers.68.self_attn.o_proj",
786
+ "model.layers.68.self_attn.q_a_layernorm",
787
+ "model.layers.68.self_attn.q_a_proj",
788
+ "model.layers.68.self_attn.q_b_proj",
789
+ "model.layers.69.input_layernorm",
790
+ "model.layers.69.mlp.gate",
791
+ "model.layers.69.mlp.gate.e_score_correction_bias",
792
+ "model.layers.69.post_attention_layernorm",
793
+ "model.layers.69.self_attn.kv_a_layernorm",
794
+ "model.layers.69.self_attn.kv_a_proj_with_mqa",
795
+ "model.layers.69.self_attn.kv_b_proj",
796
+ "model.layers.69.self_attn.o_proj",
797
+ "model.layers.69.self_attn.q_a_layernorm",
798
+ "model.layers.69.self_attn.q_a_proj",
799
+ "model.layers.69.self_attn.q_b_proj",
800
+ "model.layers.7.input_layernorm",
801
+ "model.layers.7.mlp.gate",
802
+ "model.layers.7.mlp.gate.e_score_correction_bias",
803
+ "model.layers.7.post_attention_layernorm",
804
+ "model.layers.7.self_attn.kv_a_layernorm",
805
+ "model.layers.7.self_attn.q_a_layernorm",
806
+ "model.layers.70.input_layernorm",
807
+ "model.layers.70.mlp.gate",
808
+ "model.layers.70.mlp.gate.e_score_correction_bias",
809
+ "model.layers.70.post_attention_layernorm",
810
+ "model.layers.70.self_attn.indexer.k_norm",
811
+ "model.layers.70.self_attn.indexer.k_norm.bias",
812
+ "model.layers.70.self_attn.indexers_proj",
813
+ "model.layers.70.self_attn.kv_a_layernorm",
814
+ "model.layers.70.self_attn.kv_a_proj_with_mqa",
815
+ "model.layers.70.self_attn.kv_b_proj",
816
+ "model.layers.70.self_attn.o_proj",
817
+ "model.layers.70.self_attn.q_a_layernorm",
818
+ "model.layers.70.self_attn.q_a_proj",
819
+ "model.layers.70.self_attn.q_b_proj",
820
+ "model.layers.71.input_layernorm",
821
+ "model.layers.71.mlp.gate",
822
+ "model.layers.71.mlp.gate.e_score_correction_bias",
823
+ "model.layers.71.post_attention_layernorm",
824
+ "model.layers.71.self_attn.kv_a_layernorm",
825
+ "model.layers.71.self_attn.kv_a_proj_with_mqa",
826
+ "model.layers.71.self_attn.kv_b_proj",
827
+ "model.layers.71.self_attn.o_proj",
828
+ "model.layers.71.self_attn.q_a_layernorm",
829
+ "model.layers.71.self_attn.q_a_proj",
830
+ "model.layers.71.self_attn.q_b_proj",
831
+ "model.layers.72.input_layernorm",
832
+ "model.layers.72.mlp.gate",
833
+ "model.layers.72.mlp.gate.e_score_correction_bias",
834
+ "model.layers.72.post_attention_layernorm",
835
+ "model.layers.72.self_attn.kv_a_layernorm",
836
+ "model.layers.72.self_attn.kv_a_proj_with_mqa",
837
+ "model.layers.72.self_attn.kv_b_proj",
838
+ "model.layers.72.self_attn.o_proj",
839
+ "model.layers.72.self_attn.q_a_layernorm",
840
+ "model.layers.72.self_attn.q_a_proj",
841
+ "model.layers.72.self_attn.q_b_proj",
842
+ "model.layers.73.input_layernorm",
843
+ "model.layers.73.mlp.gate",
844
+ "model.layers.73.mlp.gate.e_score_correction_bias",
845
+ "model.layers.73.post_attention_layernorm",
846
+ "model.layers.73.self_attn.kv_a_layernorm",
847
+ "model.layers.73.self_attn.kv_a_proj_with_mqa",
848
+ "model.layers.73.self_attn.kv_b_proj",
849
+ "model.layers.73.self_attn.o_proj",
850
+ "model.layers.73.self_attn.q_a_layernorm",
851
+ "model.layers.73.self_attn.q_a_proj",
852
+ "model.layers.73.self_attn.q_b_proj",
853
+ "model.layers.74.input_layernorm",
854
+ "model.layers.74.mlp.gate",
855
+ "model.layers.74.mlp.gate.e_score_correction_bias",
856
+ "model.layers.74.post_attention_layernorm",
857
+ "model.layers.74.self_attn.indexer.k_norm",
858
+ "model.layers.74.self_attn.indexer.k_norm.bias",
859
+ "model.layers.74.self_attn.indexers_proj",
860
+ "model.layers.74.self_attn.kv_a_layernorm",
861
+ "model.layers.74.self_attn.kv_a_proj_with_mqa",
862
+ "model.layers.74.self_attn.kv_b_proj",
863
+ "model.layers.74.self_attn.o_proj",
864
+ "model.layers.74.self_attn.q_a_layernorm",
865
+ "model.layers.74.self_attn.q_a_proj",
866
+ "model.layers.74.self_attn.q_b_proj",
867
+ "model.layers.75.input_layernorm",
868
+ "model.layers.75.mlp.gate",
869
+ "model.layers.75.mlp.gate.e_score_correction_bias",
870
+ "model.layers.75.post_attention_layernorm",
871
+ "model.layers.75.self_attn.kv_a_layernorm",
872
+ "model.layers.75.self_attn.kv_a_proj_with_mqa",
873
+ "model.layers.75.self_attn.kv_b_proj",
874
+ "model.layers.75.self_attn.o_proj",
875
+ "model.layers.75.self_attn.q_a_layernorm",
876
+ "model.layers.75.self_attn.q_a_proj",
877
+ "model.layers.75.self_attn.q_b_proj",
878
+ "model.layers.76.input_layernorm",
879
+ "model.layers.76.mlp.gate",
880
+ "model.layers.76.mlp.gate.e_score_correction_bias",
881
+ "model.layers.76.post_attention_layernorm",
882
+ "model.layers.76.self_attn.kv_a_layernorm",
883
+ "model.layers.76.self_attn.kv_a_proj_with_mqa",
884
+ "model.layers.76.self_attn.kv_b_proj",
885
+ "model.layers.76.self_attn.o_proj",
886
+ "model.layers.76.self_attn.q_a_layernorm",
887
+ "model.layers.76.self_attn.q_a_proj",
888
+ "model.layers.76.self_attn.q_b_proj",
889
+ "model.layers.77.input_layernorm",
890
+ "model.layers.77.mlp.gate",
891
+ "model.layers.77.mlp.gate.e_score_correction_bias",
892
+ "model.layers.77.post_attention_layernorm",
893
+ "model.layers.77.self_attn.kv_a_layernorm",
894
+ "model.layers.77.self_attn.kv_a_proj_with_mqa",
895
+ "model.layers.77.self_attn.kv_b_proj",
896
+ "model.layers.77.self_attn.o_proj",
897
+ "model.layers.77.self_attn.q_a_layernorm",
898
+ "model.layers.77.self_attn.q_a_proj",
899
+ "model.layers.77.self_attn.q_b_proj",
900
+ "model.layers.78.eh_proj",
901
+ "model.layers.78.enorm",
902
+ "model.layers.78.hnorm",
903
+ "model.layers.78.input_layernorm",
904
+ "model.layers.78.mlp.gate",
905
+ "model.layers.78.mlp.gate.e_score_correction_bias",
906
+ "model.layers.78.post_attention_layernorm",
907
+ "model.layers.78.self_attn.indexer.k_norm",
908
+ "model.layers.78.self_attn.indexer.k_norm.bias",
909
+ "model.layers.78.self_attn.indexers_proj",
910
+ "model.layers.78.self_attn.kv_a_layernorm",
911
+ "model.layers.78.self_attn.q_a_layernorm",
912
+ "model.layers.78.shared_head.norm",
913
+ "model.layers.8.input_layernorm",
914
+ "model.layers.8.mlp.gate",
915
+ "model.layers.8.mlp.gate.e_score_correction_bias",
916
+ "model.layers.8.post_attention_layernorm",
917
+ "model.layers.8.self_attn.kv_a_layernorm",
918
+ "model.layers.8.self_attn.q_a_layernorm",
919
+ "model.layers.9.input_layernorm",
920
+ "model.layers.9.mlp.gate",
921
+ "model.layers.9.mlp.gate.e_score_correction_bias",
922
+ "model.layers.9.post_attention_layernorm",
923
+ "model.layers.9.self_attn.kv_a_layernorm",
924
+ "model.layers.9.self_attn.q_a_layernorm",
925
+ "model.norm"
926
+ ],
927
+ "quant_method": "fp8",
928
+ "scale_fmt": "float",
929
+ "weight_block_size": [
930
+ 128,
931
+ 128
932
+ ]
933
+ },
934
+ "rms_norm_eps": 1e-05,
935
+ "rope_interleave": true,
936
+ "rope_parameters": {
937
+ "rope_theta": 8000000,
938
+ "rope_type": "default"
939
+ },
940
+ "routed_scaling_factor": 2.5,
941
+ "scoring_func": "sigmoid",
942
+ "tie_word_embeddings": false,
943
+ "topk_group": 1,
944
+ "topk_method": "noaux_tc",
945
+ "transformers_version": "5.12.0",
946
+ "use_cache": true,
947
+ "v_head_dim": 256,
948
+ "vocab_size": 154880
949
+ }
generation_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 154820,
6
+ 154827,
7
+ 154829
8
+ ],
9
+ "pad_token_id": 154820,
10
+ "temperature": 1.0,
11
+ "top_p": 0.95,
12
+ "transformers_version": "5.12.0"
13
+ }
model-00001-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:17a41d82bb4e967f16ff9f83a77594e52281e72b42dcdc2f44d983876952f058
3
+ size 4976403216
model-00002-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b74e5fe5e52f86ca4d6d219b180e69615e88d1919c1a536ff770bbcfb1f9a196
3
+ size 6442516120
model-00003-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98043f7103a0049f9b01daf0bd9a0914190605fcb90c54d13d70fe743d978448
3
+ size 3463284352
model-00004-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f23633b90de6ceadef208a83e3534750744f0e907099fc50ba43b6072ae6a15
3
+ size 6442516120
model-00005-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72735cee21e5a88bcdfd065af7ba86cb38d67d1060d0faafbfd95bbcb9d60b3c
3
+ size 3429709072
model-00006-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e0930a6d2fb2b1cd4fb25e27b631574f3fb5d053b17a0641ce1d8f0d5d7967ba
3
+ size 6442516120
model-00007-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0390dde537254d9d3914d40a6bde9763fb4cb39c199810ec8131a97fd18dc2f7
3
+ size 3429709072
model-00008-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38035e9a16a7ab1f1450984eb81ade090b0d17dd61d9a16448fb38ee70b1264b
3
+ size 6442516120
model-00009-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8c29eb56805cdb538b6f5b262f6e73d342b6a6967955121493fe38c0d65bf2d8
3
+ size 3439280920
model-00010-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4217f63084fcf8d87f83613e951d1d539c14bcaab267b5e02fe2e24a0aeee3bd
3
+ size 6442516120
model-00011-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f4f26d936c61af4315c120bb6806d9892b9b77452f0c4c26b7c69e708070e70
3
+ size 3429709072
model-00012-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:771decbda7832cb85c43ed7c8ea697f357d1f7f9fd2ae5b2e07e38d5b6e54565
3
+ size 6442516120
model-00013-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f7f388102e44b9a1e77d665f98f7c6a1d00bac1257df087e878e319943f50508
3
+ size 3429709072
model-00014-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a0037be4de24b6b1dfb67c1757a15ada737dad9b13d98c408921639b8cfcf7d9
3
+ size 6442516120
model-00015-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:afe9498514e72523dd7a0363409bf6a6bcbb09d768c5220c243edd5ce3f697e0
3
+ size 3429709072
model-00016-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c06b7018577d98c4acd2a5116e325328780bb1889560318c43bdc6e953c4bcc
3
+ size 6442516632
model-00017-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4d96bf910af37df7e11d2bb92d332c539ec69eb9fb651670a0fe9ec986d8f712
3
+ size 3439281968
model-00018-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:26a8f4b301d435547473b551d5013a6469568c4181528cf0d1c56dc59f719194
3
+ size 6442516632
model-00019-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e5759e0f7a5cc845758ee01b80b410ede863d9434f2b28366beee861f2ab975
3
+ size 3429710120
model-00020-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:acc32e59ff818257a14bf2a7a816098fc392d08776f4504315866ddf749e96d8
3
+ size 6442516632
model-00021-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f27ba7e950afba0e1ca796de1c58f31c2c6771affb5d4e8e88b676b50e4d733a
3
+ size 3429710120
model-00022-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a278cdc5b3772a2685dac6f2f1950357ead0fa1f5eca70b14cb103e288b319b8
3
+ size 6442516632
model-00023-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc628840345cb321897a05805b8e8d9dec5efd8d0819e4cd2e4bb7e035785110
3
+ size 3429710120
model-00024-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1099b91ec29b1fd9377a847e321936eff0d140615055198dbc26a3c97331d9f0
3
+ size 6442516632
model-00025-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dbb943a7c0a7548f8f52b44a66d8a0b8a58a1c1a452471e84766b8a1d8542def
3
+ size 3439281968
model-00026-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a49ec010e6f4d2e6c6a7899336d4c13c3d5dc9b80ea0fb65d77b5cf2c94ace1
3
+ size 6442516632
model-00027-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ff55e4a0a3c0a6b6ccfd340b605952971cbbe376fdbe3d599951e3068c784aa
3
+ size 3429710120
model-00028-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:379edad91e935bb1867dc87257698487eb51dd54626af21430a901248ed17a4f
3
+ size 6442516632
model-00029-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4b733f59e8fd27294eac982ab9909a8cffe3d1775c8495ecdc9a68afa59ead02
3
+ size 3429710120
model-00030-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fceda66abe38493f3272f1354e52562555eed2b8a0a6f5eb9fb8814aee43e7c6
3
+ size 6442516632
model-00031-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0217e53cddecffcac7d80d8ca7d03cebb2bbac17cda2b9c004821621778dcf7
3
+ size 3429710120
model-00032-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b4561e334cf83b7b94b70228649af7477529dca1093d4c0f2c039b0ebe369f88
3
+ size 6442516632
model-00033-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d8c38d7c5196fd4c0f8db94dabcd115be047a7ee87c39f4c408886d8c4d75500
3
+ size 3439281968
model-00034-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8991f8af86dd22126d9437af01ec5c5cd2c0a2a98396a7c544bc89adf0edae26
3
+ size 6442516632
model-00035-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8e688eb72dd3b5d66f901ba97a8a865323bcc537827720050ab7510301069d5
3
+ size 3429710120
model-00036-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c864bbe069a7dc13f5baa37cbfad785473893f1b18293407e57165a28165546
3
+ size 6442516632
model-00037-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a9afdd7aed89bf17bdcbbe1ab0bdf162bb49b1de00641ca4d6bf3d5e1f4d6817
3
+ size 3429710120
model-00038-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c67f84bde997b9b25459b10eb89b1993901ccbafb0389d072a34d3023714d5fa
3
+ size 6442516632
model-00039-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f41bd02f7bdebb2821a83fb7c1f0301b1408a1e681f45ea87801be4c4ed8100
3
+ size 3429710120
model-00040-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:31d40e7b85f6f69288e81443ef859fb259ee140cd6ff2aea888ab0df7fb82096
3
+ size 6442516632
model-00041-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e95048dd14731c4dc722f861f696da2a12f66ad696f4ae264fcad4780e8b4457
3
+ size 3439281968
model-00042-of-00151.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a28edf96c83a8353e01e0a6c6888bc687c726baf04388aca48e5175cb8678b3
3
+ size 6442516632