Upgrade About Strata conversion card/CTAs and add FINETUNING_WITH_STRATA.md guide
Browse files- FINETUNING_WITH_STRATA.md +103 -0
- README.md +28 -7
- assets/about-dark.svg +9 -0
- assets/about-light.svg +9 -0
- assets/btn-coffee.svg +10 -0
- assets/btn-download.svg +10 -0
- assets/btn-sponsor.svg +6 -0
- assets/closeness-dark.svg +1 -1
- assets/closeness-light.svg +1 -1
- assets/hero-dark.svg +1 -1
- assets/hero-light.svg +1 -1
- assets/masthead-dark.svg +4 -4
- assets/masthead-light.svg +4 -4
- assets/packs-dark.svg +1 -1
- assets/packs-light.svg +1 -1
- assets/speed-dark.svg +1 -1
- assets/speed-light.svg +1 -1
FINETUNING_WITH_STRATA.md
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Fine-Tuning Hy4-Flash for `.strata` v1.0
|
| 2 |
+
|
| 3 |
+
This guide covers best practices for fine-tuning [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) (LoRA, QLoRA, SFT, or DPO) and packaging your checkpoint into native **`.strata` v1.0** packs (`SQ-64`, `SQ-96`, `SQ-128`) that run at full speed in [Strata](https://github.com/Niko1221/Strata).
|
| 4 |
+
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
## 1. What to Train vs. What to Freeze (Save 68.5 GB of Downloads)
|
| 8 |
+
|
| 9 |
+
`.strata` v1.0 splits `Hy4-Flash` (`220B-A12B`) into **three decoupled files**:
|
| 10 |
+
1. **Your model pack (`*-SQ-{64,96,128}.strata`):** Section 0 manifest/tokenizer, Section 1 GPU trunk + 5-layer `DFly2` drafter, Section 2 `64 MiB` hot-Engram seed, Section 3 expert rank directory, Section 4 `4 KiB`-aligned `[1536, 1536]` LatentMoE experts, and Section 5 golden self-check vectors.
|
| 11 |
+
2. **Shared Engram lookup (`Hy4-Flash-Engram-Q8.strata`, `67.58 GB`):** Four `4 KiB`-paged `Q8_INT8_SCALE_F32` lookup tables (Layers `1, 11, 21, 31` — `63.36B` parameters).
|
| 12 |
+
3. **Shared Vision reader (`Hy4-Flash-Vision.strata`, `0.93 GB`):** `SigLIP2-400M` ViT + projector.
|
| 13 |
+
|
| 14 |
+
### Recommended parameter policy
|
| 15 |
+
- **Freeze the 4 Engram tables (`model.layers.{1,11,21,31}.engram.*`):**
|
| 16 |
+
Every `.strata` pack stores the SHA-256 of Section 4 of `Hy4-Flash-Engram-Q8.strata` (`176944953bf2b601645d206171964f7fa51b77522ef5dc14ae61b0eb1c803d74`) in its header frame. Keeping Engram frozen means **every Strata user who already downloaded `Hy4-Flash-Engram-Q8.strata` can run your fine-tune immediately without downloading another 67.58 GB**, and you do not need to re-upload the Engram shard.
|
| 17 |
+
- **Freeze the Vision tower (`vision_model.*`, `perceive.*`)** unless you are explicitly training vision-language alignment — your fine-tune will then reuse the standard `Hy4-Flash-Vision.strata` (`0.93 GB`) shard seamlessly.
|
| 18 |
+
- **Where to apply LoRA / SFT:**
|
| 19 |
+
- **Fastest & highest-leverage targets (Trunk + Shared Projections):** MLA attention projections (`q_a_proj`, `q_b_proj`, `kv_a_proj_with_mqa`, `kv_b_proj`, `o_proj`), shared experts (`shared_experts.gate_proj`, `up_proj`, `down_proj`), and LatentMoE shared projections (`routed_expert_down_proj`, `routed_expert_up_proj`).
|
| 20 |
+
- **Domain-heavy SFT / Expert LoRA:** You may also adapt the `20,992` square `[1536, 1536]` routed experts (`experts.*.{gate,up,down}_proj`) and router gates (`gate.weight`), then **merge all adapters back into a clean BF16 `safetensors` checkpoint** before quantization.
|
| 21 |
+
|
| 22 |
+
> **Important:** Always merge LoRA adapters into BF16 weights before packing. `.strata` v1.0 folds per-channel activation smoothing and a 64-wide Block-Hadamard transform ($H_{64}$, `MACaQ s_xform=2`) directly into `routed_expert_down_proj` so there is zero runtime transform cost.
|
| 23 |
+
|
| 24 |
+
---
|
| 25 |
+
|
| 26 |
+
## 2. Preserve Tokenizer Special Tokens & Chat Template Contract
|
| 27 |
+
|
| 28 |
+
Strata exports the tokenizer and `chat_template.jinja` directly from Section 0 of your `.strata` file (`strata-hy4 --export-tokenizer`). To keep Strata's native streaming tool calls, thinking budget, and stop detection working out of the box:
|
| 29 |
+
|
| 30 |
+
1. **Do not renumber or delete Hy4-Flash's special/tagged tokens:**
|
| 31 |
+
- Stop tokens: `<|hy_end:opensource|>` (`120025`) and `<|hy_EOT|>` (`120008`)
|
| 32 |
+
- Thinking tags: `<think:opensource>` and `</think:opensource>`
|
| 33 |
+
- Tool-call tags: `<tool_calls:opensource>`, `<tool_call:opensource>`, `<arg_key:opensource>`, `<arg_value:opensource>`
|
| 34 |
+
- Image pad token: `120120`
|
| 35 |
+
2. **Keep `reasoning_effort` modes compatible (`no_think`, `high`, `max`):**
|
| 36 |
+
- Train both reasoned traces (`<think:opensource>...</think:opensource>`) and direct `no_think` turns (`<think:opensource></think:opensource>`) if you want your fine-tune to support switching thinking **Off / High / Max** in the Strata app.
|
| 37 |
+
|
| 38 |
+
---
|
| 39 |
+
|
| 40 |
+
## 3. Domain-Matched Calibration & Rank-Ordered Expert Placement
|
| 41 |
+
|
| 42 |
+
Unlike uniform GGUF quantization, `.strata` v1.0 sorts all `512` experts per layer **hot-to-cold by router-weighted token traffic (`sq.rank_map`)** in Sections 3 & 4:
|
| 43 |
+
- **Hot band** (`ranks 0..79` on `SQ-64`, `0..127` on `SQ-96`, `0..159` on `SQ-128`) gets higher bit-width `S_K` codes **and** is prioritized into GPU VRAM first at runtime.
|
| 44 |
+
- **Calibration mix (`64K–96K` tokens recommended):** Mix **~30% representative prompts from your fine-tune domain** with **~70% diverse general prompts** (code, math/reasoning, multi-turn chat, and tool calls), 13-gram decontaminated against your eval split (`tools/squant/datasets.py`). This ensures experts specialized by your fine-tune are promoted into the high-precision VRAM band without degrading general capabilities.
|
| 45 |
+
|
| 46 |
+
### Standard `.strata` v1.0 Pack Recipes (`tools/squant/pipeline.py`)
|
| 47 |
+
|
| 48 |
+
| Pack | Target PC RAM | Trunk (`MLA` / `Layer 0`) | Hot Experts | Warm Experts | Cold Experts | Routed Experts Cap |
|
| 49 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 50 |
+
| **`SQ-64`** | **64 GiB** | `S5_K` / `S6_K` | `0..79`: `S4_K + S4_K` | `80..255`: `S3_K + S3_K` | `256..511`: `S2_K + S3_K` | `≤ 56.30 GB` |
|
| 51 |
+
| **`SQ-96`** | **96 GiB** | `S6_K` / `S6_K` | `0..127`: `S5_K + S6_K` | `128..351`: `S4_K + S5_K` | `352..511`: `S3_K + S4_K` | `≤ 83.20 GB` |
|
| 52 |
+
| **`SQ-128`** | **128 GiB** | `S8_K` / `S8_K` | `0..159`: `S6_K + S8_K` | `160..383`: `S5_K + S6_K` | `384..511`: `S4_K + S5_K` | `≤ 106.00 GB` |
|
| 53 |
+
|
| 54 |
+
---
|
| 55 |
+
|
| 56 |
+
## 4. Re-Distilling the Embedded 5-Layer `DFly2` Drafter (Recommended)
|
| 57 |
+
|
| 58 |
+
Every `.strata` file embeds a `0.45 GB` (`S4_K`) 5-layer **`DFly2`** speculative drafter in Section 1 that reuses the main model's Sparse MLA KV cache (`0 B` extra KV memory) for **`1.7×–2.6×` lossless Tree-8 decoding**.
|
| 59 |
+
|
| 60 |
+
- **Light LoRA / style tune:** The base `DFly2` drafter remains 100% lossless (speculative verification always matches the target distribution), though acceptance may dip slightly (`~2.5 -> ~2.0` tokens/step) if vocabulary style shifted.
|
| 61 |
+
- **Heavy SFT / new domain:** Run proximal KL distillation of the 5 `model.dfly_layers.*` blocks against your quantized target (`~50K–100K` tokens from your domain, ~10 minutes on one GPU) and splice Section 1 losslessly. Zero target weights are modified, and full `3+` tokens/step Tree-8 speed is restored.
|
| 62 |
+
|
| 63 |
+
---
|
| 64 |
+
|
| 65 |
+
## 5. Quantizing & Verifying Your Pack (`tools/squant`)
|
| 66 |
+
|
| 67 |
+
From a checkout of [Strata](https://github.com/Niko1221/Strata):
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
# 1. Stream-encode BF16 fine-tune checkpoint -> HBM Block-LDL (GPTQ) + MACaQ 64-flip + MACaQ-R compensation
|
| 71 |
+
python tools/squant/pipeline.py stream-encode \
|
| 72 |
+
--model-dir /path/to/your-merged-bf16-finetune \
|
| 73 |
+
--engram /path/to/Hy4-Flash-Engram-Q8.strata \
|
| 74 |
+
--work-dir /path/to/squant-work \
|
| 75 |
+
--builds SQ-64,SQ-96,SQ-128
|
| 76 |
+
|
| 77 |
+
# 2. End-to-end Scale-QAT + pack 4 KiB-aligned .strata v1.0 files
|
| 78 |
+
python tools/squant/pipeline.py scale-qat-pack \
|
| 79 |
+
--work-dir /path/to/squant-work \
|
| 80 |
+
--out-dir /path/to/dist \
|
| 81 |
+
--builds SQ-64,SQ-96,SQ-128
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
### Mandatory Pre-Release Quality & Integrity Checklist
|
| 85 |
+
|
| 86 |
+
Before publishing a `.strata` fine-tune on Hugging Face, verify all four gates:
|
| 87 |
+
|
| 88 |
+
1. **Container & SHA-256 Self-Check (`< 5 s`):**
|
| 89 |
+
```bash
|
| 90 |
+
strata-hy4-verify MyFineTune-SQ-64.strata Hy4-Flash-Engram-Q8.strata
|
| 91 |
+
```
|
| 92 |
+
Verifies the `STRATA\x00\x01` header, all 6 section SHA-256 digests, `4 KiB` alignment of all `20,992` experts, `engram_shard_sha256` pairing, and Section 5 `sq.selfcheck` golden dot-product vectors.
|
| 93 |
+
2. **Speculative Tree-8 Bitwise Parity:**
|
| 94 |
+
```bash
|
| 95 |
+
strata-hy4 --model MyFineTune-SQ-64.strata --engram Hy4-Flash-Engram-Q8.strata \
|
| 96 |
+
--draft dfly2 --spec-tree 1 --spec-fixed --spec-check --gen 128
|
| 97 |
+
```
|
| 98 |
+
Must report **`0 differ`** against plain greedy decoding.
|
| 99 |
+
3. **KL Divergence vs. Your Merged BF16 Fine-Tune (`tools/squant/scale_qat_eval.py`):**
|
| 100 |
+
Target held-out mean KL divergence relative to your BF16 fine-tune: **`< 0.10`** (`SQ-128`), **`< 0.16`** (`SQ-96`), and **`< 0.25`** (`SQ-64`).
|
| 101 |
+
4. **Publish Cleanly on Hugging Face:**
|
| 102 |
+
- Upload your `*-SQ-{64,96,128}.strata` file(s) and `SHA256SUMS.txt` (with Xet enabled on Hugging Face, single `.strata` files over `50 GB` upload directly without splitting).
|
| 103 |
+
- Note in your model card that users can reuse the shared **`Hy4-Flash-Engram-Q8.strata`** (`67.58 GB`) and **`Hy4-Flash-Vision.strata`** (`0.93 GB`) from [`strata-engine/Hy4-Flash-Strata`](https://huggingface.co/strata-engine/Hy4-Flash-Strata), and declare [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) as `base_model` under the [Tencent Hy License](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE).
|
README.md
CHANGED
|
@@ -29,7 +29,7 @@ tags:
|
|
| 29 |
</p>
|
| 30 |
|
| 31 |
<p align="center">
|
| 32 |
-
<a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="
|
| 33 |
</p>
|
| 34 |
|
| 35 |
<p>
|
|
@@ -157,18 +157,39 @@ ef09bceff8f01b8d226d2dc56aef09cec719b21b45783ee9b5d64e7241b72a09 Hy4-Flash-Visi
|
|
| 157 |
|
| 158 |
</details>
|
| 159 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
## About Strata
|
| 161 |
|
| 162 |
-
|
|
|
|
|
|
|
|
|
|
| 163 |
|
| 164 |
<p align="center">
|
| 165 |
-
<a href="https://github.com/Niko1221/Strata
|
| 166 |
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 167 |
<p align="center">
|
| 168 |
-
<a href="https://github.com/Niko1221/Strata/
|
| 169 |
-
<a href="https://github.com/Niko1221/Strata/blob/main/LICENSE"><img src="./assets/strata-engine-license.svg" alt="Strata engine license: MIT" width="120" height="20"></a>
|
| 170 |
-
<a href="https://github.com/sponsors/Niko1221"><img src="./assets/strata-sponsor.svg" alt="Sponsor Strata on GitHub" width="73" height="20"></a>
|
| 171 |
-
<a href="https://buymeacoffee.com/strataengine"><img src="./assets/strata-coffee.svg" alt="Buy me a coffee" width="117" height="20"></a>
|
| 172 |
</p>
|
| 173 |
<p align="center"><sub><a href="https://github.com/Niko1221/Strata#readme">English</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.zh-CN.md">简体中文</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.ja.md">日本語</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.de.md">Deutsch</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.fr.md">Français</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.es.md">Español</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pt-BR.md">Português</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pl.md">Polski</a></sub></p>
|
| 174 |
|
|
|
|
| 29 |
</p>
|
| 30 |
|
| 31 |
<p align="center">
|
| 32 |
+
<a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="#fine-tuning-with-strata">Fine-tuning</a> · <a href="#about-strata">Support Strata</a>
|
| 33 |
</p>
|
| 34 |
|
| 35 |
<p>
|
|
|
|
| 157 |
|
| 158 |
</details>
|
| 159 |
|
| 160 |
+
## Fine-tuning with `.strata`
|
| 161 |
+
|
| 162 |
+
Want to train your own LoRA or SFT fine-tune of `tencent/Hy4-flash` and ship it as a fast `.strata` pack?
|
| 163 |
+
|
| 164 |
+
- **Freeze the 4 Engram lookup tables (`layers.{1,11,21,31}.engram.*`) and Vision tower** during training — your fine-tune will automatically reuse the shared `Hy4-Flash-Engram-Q8.strata` (`67.58 GB`) and `Hy4-Flash-Vision.strata` (`0.93 GB`) files users already have on disk, saving 68.5 GB of downloads.
|
| 165 |
+
- **Merge adapters to BF16 before packing** with `tools/squant/pipeline.py` so `MACaQ` Hadamard smoothing, HBM Block-LDL (`GPTQ`) + 64-flip Hessian refinement, and rank-ordered expert placement fold cleanly with zero runtime cost.
|
| 166 |
+
- **Keep special tokens intact** (`<think:opensource>`, `<tool_calls:opensource>`, `<|hy_end:opensource|>`) and verify every build with `strata-hy4-verify` + `--spec-check` before publishing.
|
| 167 |
+
|
| 168 |
+
Read the full step-by-step guide: **[Fine-Tuning Hy4-Flash for `.strata` v1.0 →](./FINETUNING_WITH_STRATA.md)**
|
| 169 |
+
|
| 170 |
## About Strata
|
| 171 |
|
| 172 |
+
<p>
|
| 173 |
+
<img class="dark:hidden" src="./assets/about-light.svg" width="100%" alt="Strata Engine: Run frontier AI on hardware you own. Desktop and terminal app on Windows, Linux and macOS; works with Claude Code, Codex and OpenCode; 100% private and MIT open source.">
|
| 174 |
+
<img class="hidden dark:block" src="./assets/about-dark.svg" width="100%" alt="Strata Engine: Run frontier AI on hardware you own. Desktop and terminal app on Windows, Linux and macOS; works with Claude Code, Codex and OpenCode; 100% private and MIT open source.">
|
| 175 |
+
</p>
|
| 176 |
|
| 177 |
<p align="center">
|
| 178 |
+
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="./assets/btn-download.svg" alt="Download Strata for Windows, Linux or Mac" height="44"></a>
|
| 179 |
</p>
|
| 180 |
+
|
| 181 |
+
<p align="center">
|
| 182 |
+
<b>Help keep local AI fast, independent, and open to everyone.</b><br>
|
| 183 |
+
<sub>Strata is built independently and released free under the MIT license. Sponsoring on GitHub or buying a coffee directly funds new GPU kernels, model quantizations, and hardware testing.</sub>
|
| 184 |
+
</p>
|
| 185 |
+
|
| 186 |
+
<p align="center">
|
| 187 |
+
<a href="https://github.com/sponsors/Niko1221"><img src="./assets/btn-sponsor.svg" alt="Sponsor Strata on GitHub" height="44"></a>
|
| 188 |
+
<a href="https://buymeacoffee.com/strataengine"><img src="./assets/btn-coffee.svg" alt="Buy Strata a coffee" height="44"></a>
|
| 189 |
+
</p>
|
| 190 |
+
|
| 191 |
<p align="center">
|
| 192 |
+
<a href="https://github.com/Niko1221/Strata#models">Models</a> · <a href="https://github.com/Niko1221/Strata#install"><b>Install</b></a> · <a href="https://github.com/Niko1221/Strata#hardware">Hardware</a> · <a href="https://github.com/Niko1221/Strata#tools">Tools</a> · <a href="https://github.com/Niko1221/Strata#help">Help</a> · <a href="https://github.com/Niko1221/Strata/blob/main/LICENSE">MIT License</a>
|
|
|
|
|
|
|
|
|
|
| 193 |
</p>
|
| 194 |
<p align="center"><sub><a href="https://github.com/Niko1221/Strata#readme">English</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.zh-CN.md">简体中文</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.ja.md">日本語</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.de.md">Deutsch</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.fr.md">Français</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.es.md">Español</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pt-BR.md">Português</a> · <a href="https://github.com/Niko1221/Strata/blob/main/docs/lang/README.pl.md">Polski</a></sub></p>
|
| 195 |
|
assets/about-dark.svg
ADDED
|
|
assets/about-light.svg
ADDED
|
|
assets/btn-coffee.svg
ADDED
|
|
assets/btn-download.svg
ADDED
|
|
assets/btn-sponsor.svg
ADDED
|
|
assets/closeness-dark.svg
CHANGED
|
|
|
|
assets/closeness-light.svg
CHANGED
|
|
|
|
assets/hero-dark.svg
CHANGED
|
|
|
|
assets/hero-light.svg
CHANGED
|
|
|
|
assets/masthead-dark.svg
CHANGED
|
|
|
|
assets/masthead-light.svg
CHANGED
|
|
|
|
assets/packs-dark.svg
CHANGED
|
|
|
|
assets/packs-light.svg
CHANGED
|
|
|
|
assets/speed-dark.svg
CHANGED
|
|
|
|
assets/speed-light.svg
CHANGED
|
|
|
|