strata-engine commited on
Commit
f0597df
·
verified ·
1 Parent(s): aa5d010

Remove fine-tuning guide and section from model card

Browse files
Files changed (2) hide show
  1. FINETUNING_WITH_STRATA.md +0 -103
  2. README.md +1 -11
FINETUNING_WITH_STRATA.md DELETED
@@ -1,103 +0,0 @@
1
- # Fine-Tuning Hy4-Flash for `.strata` v1.0
2
-
3
- This guide covers best practices for fine-tuning [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) (LoRA, QLoRA, SFT, or DPO) and packaging your checkpoint into native **`.strata` v1.0** packs (`SQ-64`, `SQ-96`, `SQ-128`) that run at full speed in [Strata](https://github.com/Niko1221/Strata).
4
-
5
- ---
6
-
7
- ## 1. What to Train vs. What to Freeze (Save 68.5 GB of Downloads)
8
-
9
- `.strata` v1.0 splits `Hy4-Flash` (`220B-A12B`) into **three decoupled files**:
10
- 1. **Your model pack (`*-SQ-{64,96,128}.strata`):** Section 0 manifest/tokenizer, Section 1 GPU trunk + 5-layer `DFly2` drafter, Section 2 `64 MiB` hot-Engram seed, Section 3 expert rank directory, Section 4 `4 KiB`-aligned `[1536, 1536]` LatentMoE experts, and Section 5 golden self-check vectors.
11
- 2. **Shared Engram lookup (`Hy4-Flash-Engram-Q8.strata`, `67.58 GB`):** Four `4 KiB`-paged `Q8_INT8_SCALE_F32` lookup tables (Layers `1, 11, 21, 31` — `63.36B` parameters).
12
- 3. **Shared Vision reader (`Hy4-Flash-Vision.strata`, `0.93 GB`):** `SigLIP2-400M` ViT + projector.
13
-
14
- ### Recommended parameter policy
15
- - **Freeze the 4 Engram tables (`model.layers.{1,11,21,31}.engram.*`):**
16
- Every `.strata` pack stores the SHA-256 of Section 4 of `Hy4-Flash-Engram-Q8.strata` (`176944953bf2b601645d206171964f7fa51b77522ef5dc14ae61b0eb1c803d74`) in its header frame. Keeping Engram frozen means **every Strata user who already downloaded `Hy4-Flash-Engram-Q8.strata` can run your fine-tune immediately without downloading another 67.58 GB**, and you do not need to re-upload the Engram shard.
17
- - **Freeze the Vision tower (`vision_model.*`, `perceive.*`)** unless you are explicitly training vision-language alignment — your fine-tune will then reuse the standard `Hy4-Flash-Vision.strata` (`0.93 GB`) shard seamlessly.
18
- - **Where to apply LoRA / SFT:**
19
- - **Fastest & highest-leverage targets (Trunk + Shared Projections):** MLA attention projections (`q_a_proj`, `q_b_proj`, `kv_a_proj_with_mqa`, `kv_b_proj`, `o_proj`), shared experts (`shared_experts.gate_proj`, `up_proj`, `down_proj`), and LatentMoE shared projections (`routed_expert_down_proj`, `routed_expert_up_proj`).
20
- - **Domain-heavy SFT / Expert LoRA:** You may also adapt the `20,992` square `[1536, 1536]` routed experts (`experts.*.{gate,up,down}_proj`) and router gates (`gate.weight`), then **merge all adapters back into a clean BF16 `safetensors` checkpoint** before quantization.
21
-
22
- > **Important:** Always merge LoRA adapters into BF16 weights before packing. `.strata` v1.0 folds per-channel activation smoothing and a 64-wide Block-Hadamard transform ($H_{64}$, `MACaQ s_xform=2`) directly into `routed_expert_down_proj` so there is zero runtime transform cost.
23
-
24
- ---
25
-
26
- ## 2. Preserve Tokenizer Special Tokens & Chat Template Contract
27
-
28
- Strata exports the tokenizer and `chat_template.jinja` directly from Section 0 of your `.strata` file (`strata-hy4 --export-tokenizer`). To keep Strata's native streaming tool calls, thinking budget, and stop detection working out of the box:
29
-
30
- 1. **Do not renumber or delete Hy4-Flash's special/tagged tokens:**
31
- - Stop tokens: `<|hy_end:opensource|>` (`120025`) and `<|hy_EOT|>` (`120008`)
32
- - Thinking tags: `<think:opensource>` and `</think:opensource>`
33
- - Tool-call tags: `<tool_calls:opensource>`, `<tool_call:opensource>`, `<arg_key:opensource>`, `<arg_value:opensource>`
34
- - Image pad token: `120120`
35
- 2. **Keep `reasoning_effort` modes compatible (`no_think`, `high`, `max`):**
36
- - Train both reasoned traces (`<think:opensource>...</think:opensource>`) and direct `no_think` turns (`<think:opensource></think:opensource>`) if you want your fine-tune to support switching thinking **Off / High / Max** in the Strata app.
37
-
38
- ---
39
-
40
- ## 3. Domain-Matched Calibration & Rank-Ordered Expert Placement
41
-
42
- Unlike uniform GGUF quantization, `.strata` v1.0 sorts all `512` experts per layer **hot-to-cold by router-weighted token traffic (`sq.rank_map`)** in Sections 3 & 4:
43
- - **Hot band** (`ranks 0..79` on `SQ-64`, `0..127` on `SQ-96`, `0..159` on `SQ-128`) gets higher bit-width `S_K` codes **and** is prioritized into GPU VRAM first at runtime.
44
- - **Calibration mix (`64K–96K` tokens recommended):** Mix **~30% representative prompts from your fine-tune domain** with **~70% diverse general prompts** (code, math/reasoning, multi-turn chat, and tool calls), 13-gram decontaminated against your eval split (`tools/squant/datasets.py`). This ensures experts specialized by your fine-tune are promoted into the high-precision VRAM band without degrading general capabilities.
45
-
46
- ### Standard `.strata` v1.0 Pack Recipes (`tools/squant/pipeline.py`)
47
-
48
- | Pack | Target PC RAM | Trunk (`MLA` / `Layer 0`) | Hot Experts | Warm Experts | Cold Experts | Routed Experts Cap |
49
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
50
- | **`SQ-64`** | **64 GiB** | `S5_K` / `S6_K` | `0..79`: `S4_K + S4_K` | `80..255`: `S3_K + S3_K` | `256..511`: `S2_K + S3_K` | `≤ 56.30 GB` |
51
- | **`SQ-96`** | **96 GiB** | `S6_K` / `S6_K` | `0..127`: `S5_K + S6_K` | `128..351`: `S4_K + S5_K` | `352..511`: `S3_K + S4_K` | `≤ 83.20 GB` |
52
- | **`SQ-128`** | **128 GiB** | `S8_K` / `S8_K` | `0..159`: `S6_K + S8_K` | `160..383`: `S5_K + S6_K` | `384..511`: `S4_K + S5_K` | `≤ 106.00 GB` |
53
-
54
- ---
55
-
56
- ## 4. Re-Distilling the Embedded 5-Layer `DFly2` Drafter (Recommended)
57
-
58
- Every `.strata` file embeds a `0.45 GB` (`S4_K`) 5-layer **`DFly2`** speculative drafter in Section 1 that reuses the main model's Sparse MLA KV cache (`0 B` extra KV memory) for **`1.7×–2.6×` lossless Tree-8 decoding**.
59
-
60
- - **Light LoRA / style tune:** The base `DFly2` drafter remains 100% lossless (speculative verification always matches the target distribution), though acceptance may dip slightly (`~2.5 -> ~2.0` tokens/step) if vocabulary style shifted.
61
- - **Heavy SFT / new domain:** Run proximal KL distillation of the 5 `model.dfly_layers.*` blocks against your quantized target (`~50K–100K` tokens from your domain, ~10 minutes on one GPU) and splice Section 1 losslessly. Zero target weights are modified, and full `3+` tokens/step Tree-8 speed is restored.
62
-
63
- ---
64
-
65
- ## 5. Quantizing & Verifying Your Pack (`tools/squant`)
66
-
67
- From a checkout of [Strata](https://github.com/Niko1221/Strata):
68
-
69
- ```bash
70
- # 1. Stream-encode BF16 fine-tune checkpoint -> HBM Block-LDL (GPTQ) + MACaQ 64-flip + MACaQ-R compensation
71
- python tools/squant/pipeline.py stream-encode \
72
- --model-dir /path/to/your-merged-bf16-finetune \
73
- --engram /path/to/Hy4-Flash-Engram-Q8.strata \
74
- --work-dir /path/to/squant-work \
75
- --builds SQ-64,SQ-96,SQ-128
76
-
77
- # 2. End-to-end Scale-QAT + pack 4 KiB-aligned .strata v1.0 files
78
- python tools/squant/pipeline.py scale-qat-pack \
79
- --work-dir /path/to/squant-work \
80
- --out-dir /path/to/dist \
81
- --builds SQ-64,SQ-96,SQ-128
82
- ```
83
-
84
- ### Mandatory Pre-Release Quality & Integrity Checklist
85
-
86
- Before publishing a `.strata` fine-tune on Hugging Face, verify all four gates:
87
-
88
- 1. **Container & SHA-256 Self-Check (`< 5 s`):**
89
- ```bash
90
- strata-hy4-verify MyFineTune-SQ-64.strata Hy4-Flash-Engram-Q8.strata
91
- ```
92
- Verifies the `STRATA\x00\x01` header, all 6 section SHA-256 digests, `4 KiB` alignment of all `20,992` experts, `engram_shard_sha256` pairing, and Section 5 `sq.selfcheck` golden dot-product vectors.
93
- 2. **Speculative Tree-8 Bitwise Parity:**
94
- ```bash
95
- strata-hy4 --model MyFineTune-SQ-64.strata --engram Hy4-Flash-Engram-Q8.strata \
96
- --draft dfly2 --spec-tree 1 --spec-fixed --spec-check --gen 128
97
- ```
98
- Must report **`0 differ`** against plain greedy decoding.
99
- 3. **KL Divergence vs. Your Merged BF16 Fine-Tune (`tools/squant/scale_qat_eval.py`):**
100
- Target held-out mean KL divergence relative to your BF16 fine-tune: **`< 0.10`** (`SQ-128`), **`< 0.16`** (`SQ-96`), and **`< 0.25`** (`SQ-64`).
101
- 4. **Publish Cleanly on Hugging Face:**
102
- - Upload your `*-SQ-{64,96,128}.strata` file(s) and `SHA256SUMS.txt` (with Xet enabled on Hugging Face, single `.strata` files over `50 GB` upload directly without splitting).
103
- - Note in your model card that users can reuse the shared **`Hy4-Flash-Engram-Q8.strata`** (`67.58 GB`) and **`Hy4-Flash-Vision.strata`** (`0.93 GB`) from [`strata-engine/Hy4-Flash-Strata`](https://huggingface.co/strata-engine/Hy4-Flash-Strata), and declare [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) as `base_model` under the [Tencent Hy License](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -29,7 +29,7 @@ tags:
29
  </p>
30
 
31
  <p align="center">
32
- <a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="#fine-tuning-with-strata">Fine-tuning</a> · <a href="#about-strata">Support Strata</a>
33
  </p>
34
 
35
  <p>
@@ -157,16 +157,6 @@ ef09bceff8f01b8d226d2dc56aef09cec719b21b45783ee9b5d64e7241b72a09 Hy4-Flash-Visi
157
 
158
  </details>
159
 
160
- ## Fine-tuning with `.strata`
161
-
162
- Want to train your own LoRA or SFT fine-tune of `tencent/Hy4-flash` and ship it as a fast `.strata` pack?
163
-
164
- - **Freeze the 4 Engram lookup tables (`layers.{1,11,21,31}.engram.*`) and Vision tower** during training — your fine-tune will automatically reuse the shared `Hy4-Flash-Engram-Q8.strata` (`67.58 GB`) and `Hy4-Flash-Vision.strata` (`0.93 GB`) files users already have on disk, saving 68.5 GB of downloads.
165
- - **Merge adapters to BF16 before packing** with `tools/squant/pipeline.py` so `MACaQ` Hadamard smoothing, HBM Block-LDL (`GPTQ`) + 64-flip Hessian refinement, and rank-ordered expert placement fold cleanly with zero runtime cost.
166
- - **Keep special tokens intact** (`<think:opensource>`, `<tool_calls:opensource>`, stop tokens) and verify every build with `strata-hy4-verify` + `--spec-check` before publishing.
167
-
168
- Read the full step-by-step guide: **[Fine-Tuning Hy4-Flash for `.strata` v1.0 →](./FINETUNING_WITH_STRATA.md)**
169
-
170
  ## About Strata
171
 
172
  <p>
 
29
  </p>
30
 
31
  <p align="center">
32
+ <a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="#about-strata">Support Strata</a>
33
  </p>
34
 
35
  <p>
 
157
 
158
  </details>
159
 
 
 
 
 
 
 
 
 
 
 
160
  ## About Strata
161
 
162
  <p>