strata-engine commited on
Commit
88fa29f
·
verified ·
1 Parent(s): 378a1c2

Upgrade About Strata conversion card/CTAs and add FINETUNING_WITH_STRATA.md guide

Browse files
FINETUNING_WITH_STRATA.md ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Fine-Tuning Hy4-Flash for `.strata` v1.0
2
+
3
+ This guide covers best practices for fine-tuning [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) (LoRA, QLoRA, SFT, or DPO) and packaging your checkpoint into native **`.strata` v1.0** packs (`SQ-64`, `SQ-96`, `SQ-128`) that run at full speed in [Strata](https://github.com/Niko1221/Strata).
4
+
5
+ ---
6
+
7
+ ## 1. What to Train vs. What to Freeze (Save 68.5 GB of Downloads)
8
+
9
+ `.strata` v1.0 splits `Hy4-Flash` (`220B-A12B`) into **three decoupled files**:
10
+ 1. **Your model pack (`*-SQ-{64,96,128}.strata`):** Section 0 manifest/tokenizer, Section 1 GPU trunk + 5-layer `DFly2` drafter, Section 2 `64 MiB` hot-Engram seed, Section 3 expert rank directory, Section 4 `4 KiB`-aligned `[1536, 1536]` LatentMoE experts, and Section 5 golden self-check vectors.
11
+ 2. **Shared Engram lookup (`Hy4-Flash-Engram-Q8.strata`, `67.58 GB`):** Four `4 KiB`-paged `Q8_INT8_SCALE_F32` lookup tables (Layers `1, 11, 21, 31` — `63.36B` parameters).
12
+ 3. **Shared Vision reader (`Hy4-Flash-Vision.strata`, `0.93 GB`):** `SigLIP2-400M` ViT + projector.
13
+
14
+ ### Recommended parameter policy
15
+ - **Freeze the 4 Engram tables (`model.layers.{1,11,21,31}.engram.*`):**
16
+ Every `.strata` pack stores the SHA-256 of Section 4 of `Hy4-Flash-Engram-Q8.strata` (`176944953bf2b601645d206171964f7fa51b77522ef5dc14ae61b0eb1c803d74`) in its header frame. Keeping Engram frozen means **every Strata user who already downloaded `Hy4-Flash-Engram-Q8.strata` can run your fine-tune immediately without downloading another 67.58 GB**, and you do not need to re-upload the Engram shard.
17
+ - **Freeze the Vision tower (`vision_model.*`, `perceive.*`)** unless you are explicitly training vision-language alignment — your fine-tune will then reuse the standard `Hy4-Flash-Vision.strata` (`0.93 GB`) shard seamlessly.
18
+ - **Where to apply LoRA / SFT:**
19
+ - **Fastest & highest-leverage targets (Trunk + Shared Projections):** MLA attention projections (`q_a_proj`, `q_b_proj`, `kv_a_proj_with_mqa`, `kv_b_proj`, `o_proj`), shared experts (`shared_experts.gate_proj`, `up_proj`, `down_proj`), and LatentMoE shared projections (`routed_expert_down_proj`, `routed_expert_up_proj`).
20
+ - **Domain-heavy SFT / Expert LoRA:** You may also adapt the `20,992` square `[1536, 1536]` routed experts (`experts.*.{gate,up,down}_proj`) and router gates (`gate.weight`), then **merge all adapters back into a clean BF16 `safetensors` checkpoint** before quantization.
21
+
22
+ > **Important:** Always merge LoRA adapters into BF16 weights before packing. `.strata` v1.0 folds per-channel activation smoothing and a 64-wide Block-Hadamard transform ($H_{64}$, `MACaQ s_xform=2`) directly into `routed_expert_down_proj` so there is zero runtime transform cost.
23
+
24
+ ---
25
+
26
+ ## 2. Preserve Tokenizer Special Tokens & Chat Template Contract
27
+
28
+ Strata exports the tokenizer and `chat_template.jinja` directly from Section 0 of your `.strata` file (`strata-hy4 --export-tokenizer`). To keep Strata's native streaming tool calls, thinking budget, and stop detection working out of the box:
29
+
30
+ 1. **Do not renumber or delete Hy4-Flash's special/tagged tokens:**
31
+ - Stop tokens: `<|hy_end:opensource|>` (`120025`) and `<|hy_EOT|>` (`120008`)
32
+ - Thinking tags: `<think:opensource>` and `</think:opensource>`
33
+ - Tool-call tags: `<tool_calls:opensource>`, `<tool_call:opensource>`, `<arg_key:opensource>`, `<arg_value:opensource>`
34
+ - Image pad token: `120120`
35
+ 2. **Keep `reasoning_effort` modes compatible (`no_think`, `high`, `max`):**
36
+ - Train both reasoned traces (`<think:opensource>...</think:opensource>`) and direct `no_think` turns (`<think:opensource></think:opensource>`) if you want your fine-tune to support switching thinking **Off / High / Max** in the Strata app.
37
+
38
+ ---
39
+
40
+ ## 3. Domain-Matched Calibration & Rank-Ordered Expert Placement
41
+
42
+ Unlike uniform GGUF quantization, `.strata` v1.0 sorts all `512` experts per layer **hot-to-cold by router-weighted token traffic (`sq.rank_map`)** in Sections 3 & 4:
43
+ - **Hot band** (`ranks 0..79` on `SQ-64`, `0..127` on `SQ-96`, `0..159` on `SQ-128`) gets higher bit-width `S_K` codes **and** is prioritized into GPU VRAM first at runtime.
44
+ - **Calibration mix (`64K–96K` tokens recommended):** Mix **~30% representative prompts from your fine-tune domain** with **~70% diverse general prompts** (code, math/reasoning, multi-turn chat, and tool calls), 13-gram decontaminated against your eval split (`tools/squant/datasets.py`). This ensures experts specialized by your fine-tune are promoted into the high-precision VRAM band without degrading general capabilities.
45
+
46
+ ### Standard `.strata` v1.0 Pack Recipes (`tools/squant/pipeline.py`)
47
+
48
+ | Pack | Target PC RAM | Trunk (`MLA` / `Layer 0`) | Hot Experts | Warm Experts | Cold Experts | Routed Experts Cap |
49
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
50
+ | **`SQ-64`** | **64 GiB** | `S5_K` / `S6_K` | `0..79`: `S4_K + S4_K` | `80..255`: `S3_K + S3_K` | `256..511`: `S2_K + S3_K` | `≤ 56.30 GB` |
51
+ | **`SQ-96`** | **96 GiB** | `S6_K` / `S6_K` | `0..127`: `S5_K + S6_K` | `128..351`: `S4_K + S5_K` | `352..511`: `S3_K + S4_K` | `≤ 83.20 GB` |
52
+ | **`SQ-128`** | **128 GiB** | `S8_K` / `S8_K` | `0..159`: `S6_K + S8_K` | `160..383`: `S5_K + S6_K` | `384..511`: `S4_K + S5_K` | `≤ 106.00 GB` |
53
+
54
+ ---
55
+
56
+ ## 4. Re-Distilling the Embedded 5-Layer `DFly2` Drafter (Recommended)
57
+
58
+ Every `.strata` file embeds a `0.45 GB` (`S4_K`) 5-layer **`DFly2`** speculative drafter in Section 1 that reuses the main model's Sparse MLA KV cache (`0 B` extra KV memory) for **`1.7×–2.6×` lossless Tree-8 decoding**.
59
+
60
+ - **Light LoRA / style tune:** The base `DFly2` drafter remains 100% lossless (speculative verification always matches the target distribution), though acceptance may dip slightly (`~2.5 -> ~2.0` tokens/step) if vocabulary style shifted.
61
+ - **Heavy SFT / new domain:** Run proximal KL distillation of the 5 `model.dfly_layers.*` blocks against your quantized target (`~50K–100K` tokens from your domain, ~10 minutes on one GPU) and splice Section 1 losslessly. Zero target weights are modified, and full `3+` tokens/step Tree-8 speed is restored.
62
+
63
+ ---
64
+
65
+ ## 5. Quantizing & Verifying Your Pack (`tools/squant`)
66
+
67
+ From a checkout of [Strata](https://github.com/Niko1221/Strata):
68
+
69
+ ```bash
70
+ # 1. Stream-encode BF16 fine-tune checkpoint -> HBM Block-LDL (GPTQ) + MACaQ 64-flip + MACaQ-R compensation
71
+ python tools/squant/pipeline.py stream-encode \
72
+ --model-dir /path/to/your-merged-bf16-finetune \
73
+ --engram /path/to/Hy4-Flash-Engram-Q8.strata \
74
+ --work-dir /path/to/squant-work \
75
+ --builds SQ-64,SQ-96,SQ-128
76
+
77
+ # 2. End-to-end Scale-QAT + pack 4 KiB-aligned .strata v1.0 files
78
+ python tools/squant/pipeline.py scale-qat-pack \
79
+ --work-dir /path/to/squant-work \
80
+ --out-dir /path/to/dist \
81
+ --builds SQ-64,SQ-96,SQ-128
82
+ ```
83
+
84
+ ### Mandatory Pre-Release Quality & Integrity Checklist
85
+
86
+ Before publishing a `.strata` fine-tune on Hugging Face, verify all four gates:
87
+
88
+ 1. **Container & SHA-256 Self-Check (`< 5 s`):**
89
+ ```bash
90
+ strata-hy4-verify MyFineTune-SQ-64.strata Hy4-Flash-Engram-Q8.strata
91
+ ```
92
+ Verifies the `STRATA\x00\x01` header, all 6 section SHA-256 digests, `4 KiB` alignment of all `20,992` experts, `engram_shard_sha256` pairing, and Section 5 `sq.selfcheck` golden dot-product vectors.
93
+ 2. **Speculative Tree-8 Bitwise Parity:**
94
+ ```bash
95
+ strata-hy4 --model MyFineTune-SQ-64.strata --engram Hy4-Flash-Engram-Q8.strata \
96
+ --draft dfly2 --spec-tree 1 --spec-fixed --spec-check --gen 128
97
+ ```
98
+ Must report **`0 differ`** against plain greedy decoding.
99
+ 3. **KL Divergence vs. Your Merged BF16 Fine-Tune (`tools/squant/scale_qat_eval.py`):**
100
+ Target held-out mean KL divergence relative to your BF16 fine-tune: **`< 0.10`** (`SQ-128`), **`< 0.16`** (`SQ-96`), and **`< 0.25`** (`SQ-64`).
101
+ 4. **Publish Cleanly on Hugging Face:**
102
+ - Upload your `*-SQ-{64,96,128}.strata` file(s) and `SHA256SUMS.txt` (with Xet enabled on Hugging Face, single `.strata` files over `50 GB` upload directly without splitting).
103
+ - Note in your model card that users can reuse the shared **`Hy4-Flash-Engram-Q8.strata`** (`67.58 GB`) and **`Hy4-Flash-Vision.strata`** (`0.93 GB`) from [`strata-engine/Hy4-Flash-Strata`](https://huggingface.co/strata-engine/Hy4-Flash-Strata), and declare [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) as `base_model` under the [Tencent Hy License](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE).
README.md CHANGED
@@ -29,7 +29,7 @@ tags:
29
  </p>
30
 
31
  <p align="center">
32
- <a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="https://github.com/Niko1221/Strata">Strata on GitHub</a>
33
  </p>
34
 
35
  <p>
@@ -157,18 +157,39 @@ ef09bceff8f01b8d226d2dc56aef09cec719b21b45783ee9b5d64e7241b72a09 Hy4-Flash-Visi
157
 
158
  </details>
159
 
 
 
 
 
 
 
 
 
 
 
160
  ## About Strata
161
 
162
- **Run powerful AI on your own computer.** Chat, code and work with images, locally. Strata is free and open source.
 
 
 
163
 
164
  <p align="center">
165
- <a href="https://github.com/Niko1221/Strata#models">Models</a> · <a href="https://github.com/Niko1221/Strata#install"><b>Install</b></a> · <a href="https://github.com/Niko1221/Strata#hardware">Hardware</a> · <a href="https://github.com/Niko1221/Strata#tools">Tools</a> · <a href="https://github.com/Niko1221/Strata#help">Help</a> · <a href="https://github.com/Niko1221/Strata#support-strata">Support</a>
166
  </p>
 
 
 
 
 
 
 
 
 
 
 
167
  <p align="center">
168
- <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="./assets/strata-download.svg" alt="Download Strata" width="108" height="20"></a>
169
- <a href="https://github.com/Niko1221/Strata/blob/main/LICENSE"><img src="./assets/strata-engine-license.svg" alt="Strata engine license: MIT" width="120" height="20"></a>
170
- <a href="https://github.com/sponsors/Niko1221"><img src="./assets/strata-sponsor.svg" alt="Sponsor Strata on GitHub" width="73" height="20"></a>
171
- <a href="https://buymeacoffee.com/strataengine"><img src="./assets/strata-coffee.svg" alt="Buy me a coffee" width="117" height="20"></a>
172
  </p>
173
  <p align="center"><sub><a href="https://github.com/Niko1221/Strata#readme">English</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.zh-CN.md">简体中文</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.ja.md">日本語</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.de.md">Deutsch</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.fr.md">Français</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.es.md">Español</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pt-BR.md">Português</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pl.md">Polski</a></sub></p>
174
 
 
29
  </p>
30
 
31
  <p align="center">
32
+ <a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="#fine-tuning-with-strata">Fine-tuning</a> · <a href="#about-strata">Support Strata</a>
33
  </p>
34
 
35
  <p>
 
157
 
158
  </details>
159
 
160
+ ## Fine-tuning with `.strata`
161
+
162
+ Want to train your own LoRA or SFT fine-tune of `tencent/Hy4-flash` and ship it as a fast `.strata` pack?
163
+
164
+ - **Freeze the 4 Engram lookup tables (`layers.{1,11,21,31}.engram.*`) and Vision tower** during training — your fine-tune will automatically reuse the shared `Hy4-Flash-Engram-Q8.strata` (`67.58 GB`) and `Hy4-Flash-Vision.strata` (`0.93 GB`) files users already have on disk, saving 68.5 GB of downloads.
165
+ - **Merge adapters to BF16 before packing** with `tools/squant/pipeline.py` so `MACaQ` Hadamard smoothing, HBM Block-LDL (`GPTQ`) + 64-flip Hessian refinement, and rank-ordered expert placement fold cleanly with zero runtime cost.
166
+ - **Keep special tokens intact** (`<think:opensource>`, `<tool_calls:opensource>`, `<|hy_end:opensource|>`) and verify every build with `strata-hy4-verify` + `--spec-check` before publishing.
167
+
168
+ Read the full step-by-step guide: **[Fine-Tuning Hy4-Flash for `.strata` v1.0 →](./FINETUNING_WITH_STRATA.md)**
169
+
170
  ## About Strata
171
 
172
+ <p>
173
+ <img class="dark:hidden" src="./assets/about-light.svg" width="100%" alt="Strata Engine: Run frontier AI on hardware you own. Desktop and terminal app on Windows, Linux and macOS; works with Claude Code, Codex and OpenCode; 100% private and MIT open source.">
174
+ <img class="hidden dark:block" src="./assets/about-dark.svg" width="100%" alt="Strata Engine: Run frontier AI on hardware you own. Desktop and terminal app on Windows, Linux and macOS; works with Claude Code, Codex and OpenCode; 100% private and MIT open source.">
175
+ </p>
176
 
177
  <p align="center">
178
+ <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="./assets/btn-download.svg" alt="Download Strata for Windows, Linux or Mac" height="44"></a>
179
  </p>
180
+
181
+ <p align="center">
182
+ <b>Help keep local AI fast, independent, and open to everyone.</b><br>
183
+ <sub>Strata is built independently and released free under the MIT license. Sponsoring on GitHub or buying a coffee directly funds new GPU kernels, model quantizations, and hardware testing.</sub>
184
+ </p>
185
+
186
+ <p align="center">
187
+ <a href="https://github.com/sponsors/Niko1221"><img src="./assets/btn-sponsor.svg" alt="Sponsor Strata on GitHub" height="44"></a>&nbsp;&nbsp;
188
+ <a href="https://buymeacoffee.com/strataengine"><img src="./assets/btn-coffee.svg" alt="Buy Strata a coffee" height="44"></a>
189
+ </p>
190
+
191
  <p align="center">
192
+ <a href="https://github.com/Niko1221/Strata#models">Models</a> · <a href="https://github.com/Niko1221/Strata#install"><b>Install</b></a> · <a href="https://github.com/Niko1221/Strata#hardware">Hardware</a> · <a href="https://github.com/Niko1221/Strata#tools">Tools</a> · <a href="https://github.com/Niko1221/Strata#help">Help</a> · <a href="https://github.com/Niko1221/Strata/blob/main/LICENSE">MIT License</a>
 
 
 
193
  </p>
194
  <p align="center"><sub><a href="https://github.com/Niko1221/Strata#readme">English</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.zh-CN.md">简体中文</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.ja.md">日本語</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.de.md">Deutsch</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.fr.md">Français</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.es.md">Español</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pt-BR.md">Português</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pl.md">Polski</a></sub></p>
195
 
assets/about-dark.svg ADDED
assets/about-light.svg ADDED
assets/btn-coffee.svg ADDED
assets/btn-download.svg ADDED
assets/btn-sponsor.svg ADDED
assets/closeness-dark.svg CHANGED
assets/closeness-light.svg CHANGED
assets/hero-dark.svg CHANGED
assets/hero-light.svg CHANGED
assets/masthead-dark.svg CHANGED
assets/masthead-light.svg CHANGED
assets/packs-dark.svg CHANGED
assets/packs-light.svg CHANGED
assets/speed-dark.svg CHANGED
assets/speed-light.svg CHANGED