Remove fine-tuning guide and section from model card
Browse files- FINETUNING_WITH_STRATA.md +0 -103
- README.md +1 -11
FINETUNING_WITH_STRATA.md
DELETED
|
@@ -1,103 +0,0 @@
|
|
| 1 |
-
# Fine-Tuning Hy4-Flash for `.strata` v1.0
|
| 2 |
-
|
| 3 |
-
This guide covers best practices for fine-tuning [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) (LoRA, QLoRA, SFT, or DPO) and packaging your checkpoint into native **`.strata` v1.0** packs (`SQ-64`, `SQ-96`, `SQ-128`) that run at full speed in [Strata](https://github.com/Niko1221/Strata).
|
| 4 |
-
|
| 5 |
-
---
|
| 6 |
-
|
| 7 |
-
## 1. What to Train vs. What to Freeze (Save 68.5 GB of Downloads)
|
| 8 |
-
|
| 9 |
-
`.strata` v1.0 splits `Hy4-Flash` (`220B-A12B`) into **three decoupled files**:
|
| 10 |
-
1. **Your model pack (`*-SQ-{64,96,128}.strata`):** Section 0 manifest/tokenizer, Section 1 GPU trunk + 5-layer `DFly2` drafter, Section 2 `64 MiB` hot-Engram seed, Section 3 expert rank directory, Section 4 `4 KiB`-aligned `[1536, 1536]` LatentMoE experts, and Section 5 golden self-check vectors.
|
| 11 |
-
2. **Shared Engram lookup (`Hy4-Flash-Engram-Q8.strata`, `67.58 GB`):** Four `4 KiB`-paged `Q8_INT8_SCALE_F32` lookup tables (Layers `1, 11, 21, 31` — `63.36B` parameters).
|
| 12 |
-
3. **Shared Vision reader (`Hy4-Flash-Vision.strata`, `0.93 GB`):** `SigLIP2-400M` ViT + projector.
|
| 13 |
-
|
| 14 |
-
### Recommended parameter policy
|
| 15 |
-
- **Freeze the 4 Engram tables (`model.layers.{1,11,21,31}.engram.*`):**
|
| 16 |
-
Every `.strata` pack stores the SHA-256 of Section 4 of `Hy4-Flash-Engram-Q8.strata` (`176944953bf2b601645d206171964f7fa51b77522ef5dc14ae61b0eb1c803d74`) in its header frame. Keeping Engram frozen means **every Strata user who already downloaded `Hy4-Flash-Engram-Q8.strata` can run your fine-tune immediately without downloading another 67.58 GB**, and you do not need to re-upload the Engram shard.
|
| 17 |
-
- **Freeze the Vision tower (`vision_model.*`, `perceive.*`)** unless you are explicitly training vision-language alignment — your fine-tune will then reuse the standard `Hy4-Flash-Vision.strata` (`0.93 GB`) shard seamlessly.
|
| 18 |
-
- **Where to apply LoRA / SFT:**
|
| 19 |
-
- **Fastest & highest-leverage targets (Trunk + Shared Projections):** MLA attention projections (`q_a_proj`, `q_b_proj`, `kv_a_proj_with_mqa`, `kv_b_proj`, `o_proj`), shared experts (`shared_experts.gate_proj`, `up_proj`, `down_proj`), and LatentMoE shared projections (`routed_expert_down_proj`, `routed_expert_up_proj`).
|
| 20 |
-
- **Domain-heavy SFT / Expert LoRA:** You may also adapt the `20,992` square `[1536, 1536]` routed experts (`experts.*.{gate,up,down}_proj`) and router gates (`gate.weight`), then **merge all adapters back into a clean BF16 `safetensors` checkpoint** before quantization.
|
| 21 |
-
|
| 22 |
-
> **Important:** Always merge LoRA adapters into BF16 weights before packing. `.strata` v1.0 folds per-channel activation smoothing and a 64-wide Block-Hadamard transform ($H_{64}$, `MACaQ s_xform=2`) directly into `routed_expert_down_proj` so there is zero runtime transform cost.
|
| 23 |
-
|
| 24 |
-
---
|
| 25 |
-
|
| 26 |
-
## 2. Preserve Tokenizer Special Tokens & Chat Template Contract
|
| 27 |
-
|
| 28 |
-
Strata exports the tokenizer and `chat_template.jinja` directly from Section 0 of your `.strata` file (`strata-hy4 --export-tokenizer`). To keep Strata's native streaming tool calls, thinking budget, and stop detection working out of the box:
|
| 29 |
-
|
| 30 |
-
1. **Do not renumber or delete Hy4-Flash's special/tagged tokens:**
|
| 31 |
-
- Stop tokens: `<|hy_end:opensource|>` (`120025`) and `<|hy_EOT|>` (`120008`)
|
| 32 |
-
- Thinking tags: `<think:opensource>` and `</think:opensource>`
|
| 33 |
-
- Tool-call tags: `<tool_calls:opensource>`, `<tool_call:opensource>`, `<arg_key:opensource>`, `<arg_value:opensource>`
|
| 34 |
-
- Image pad token: `120120`
|
| 35 |
-
2. **Keep `reasoning_effort` modes compatible (`no_think`, `high`, `max`):**
|
| 36 |
-
- Train both reasoned traces (`<think:opensource>...</think:opensource>`) and direct `no_think` turns (`<think:opensource></think:opensource>`) if you want your fine-tune to support switching thinking **Off / High / Max** in the Strata app.
|
| 37 |
-
|
| 38 |
-
---
|
| 39 |
-
|
| 40 |
-
## 3. Domain-Matched Calibration & Rank-Ordered Expert Placement
|
| 41 |
-
|
| 42 |
-
Unlike uniform GGUF quantization, `.strata` v1.0 sorts all `512` experts per layer **hot-to-cold by router-weighted token traffic (`sq.rank_map`)** in Sections 3 & 4:
|
| 43 |
-
- **Hot band** (`ranks 0..79` on `SQ-64`, `0..127` on `SQ-96`, `0..159` on `SQ-128`) gets higher bit-width `S_K` codes **and** is prioritized into GPU VRAM first at runtime.
|
| 44 |
-
- **Calibration mix (`64K–96K` tokens recommended):** Mix **~30% representative prompts from your fine-tune domain** with **~70% diverse general prompts** (code, math/reasoning, multi-turn chat, and tool calls), 13-gram decontaminated against your eval split (`tools/squant/datasets.py`). This ensures experts specialized by your fine-tune are promoted into the high-precision VRAM band without degrading general capabilities.
|
| 45 |
-
|
| 46 |
-
### Standard `.strata` v1.0 Pack Recipes (`tools/squant/pipeline.py`)
|
| 47 |
-
|
| 48 |
-
| Pack | Target PC RAM | Trunk (`MLA` / `Layer 0`) | Hot Experts | Warm Experts | Cold Experts | Routed Experts Cap |
|
| 49 |
-
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 50 |
-
| **`SQ-64`** | **64 GiB** | `S5_K` / `S6_K` | `0..79`: `S4_K + S4_K` | `80..255`: `S3_K + S3_K` | `256..511`: `S2_K + S3_K` | `≤ 56.30 GB` |
|
| 51 |
-
| **`SQ-96`** | **96 GiB** | `S6_K` / `S6_K` | `0..127`: `S5_K + S6_K` | `128..351`: `S4_K + S5_K` | `352..511`: `S3_K + S4_K` | `≤ 83.20 GB` |
|
| 52 |
-
| **`SQ-128`** | **128 GiB** | `S8_K` / `S8_K` | `0..159`: `S6_K + S8_K` | `160..383`: `S5_K + S6_K` | `384..511`: `S4_K + S5_K` | `≤ 106.00 GB` |
|
| 53 |
-
|
| 54 |
-
---
|
| 55 |
-
|
| 56 |
-
## 4. Re-Distilling the Embedded 5-Layer `DFly2` Drafter (Recommended)
|
| 57 |
-
|
| 58 |
-
Every `.strata` file embeds a `0.45 GB` (`S4_K`) 5-layer **`DFly2`** speculative drafter in Section 1 that reuses the main model's Sparse MLA KV cache (`0 B` extra KV memory) for **`1.7×–2.6×` lossless Tree-8 decoding**.
|
| 59 |
-
|
| 60 |
-
- **Light LoRA / style tune:** The base `DFly2` drafter remains 100% lossless (speculative verification always matches the target distribution), though acceptance may dip slightly (`~2.5 -> ~2.0` tokens/step) if vocabulary style shifted.
|
| 61 |
-
- **Heavy SFT / new domain:** Run proximal KL distillation of the 5 `model.dfly_layers.*` blocks against your quantized target (`~50K–100K` tokens from your domain, ~10 minutes on one GPU) and splice Section 1 losslessly. Zero target weights are modified, and full `3+` tokens/step Tree-8 speed is restored.
|
| 62 |
-
|
| 63 |
-
---
|
| 64 |
-
|
| 65 |
-
## 5. Quantizing & Verifying Your Pack (`tools/squant`)
|
| 66 |
-
|
| 67 |
-
From a checkout of [Strata](https://github.com/Niko1221/Strata):
|
| 68 |
-
|
| 69 |
-
```bash
|
| 70 |
-
# 1. Stream-encode BF16 fine-tune checkpoint -> HBM Block-LDL (GPTQ) + MACaQ 64-flip + MACaQ-R compensation
|
| 71 |
-
python tools/squant/pipeline.py stream-encode \
|
| 72 |
-
--model-dir /path/to/your-merged-bf16-finetune \
|
| 73 |
-
--engram /path/to/Hy4-Flash-Engram-Q8.strata \
|
| 74 |
-
--work-dir /path/to/squant-work \
|
| 75 |
-
--builds SQ-64,SQ-96,SQ-128
|
| 76 |
-
|
| 77 |
-
# 2. End-to-end Scale-QAT + pack 4 KiB-aligned .strata v1.0 files
|
| 78 |
-
python tools/squant/pipeline.py scale-qat-pack \
|
| 79 |
-
--work-dir /path/to/squant-work \
|
| 80 |
-
--out-dir /path/to/dist \
|
| 81 |
-
--builds SQ-64,SQ-96,SQ-128
|
| 82 |
-
```
|
| 83 |
-
|
| 84 |
-
### Mandatory Pre-Release Quality & Integrity Checklist
|
| 85 |
-
|
| 86 |
-
Before publishing a `.strata` fine-tune on Hugging Face, verify all four gates:
|
| 87 |
-
|
| 88 |
-
1. **Container & SHA-256 Self-Check (`< 5 s`):**
|
| 89 |
-
```bash
|
| 90 |
-
strata-hy4-verify MyFineTune-SQ-64.strata Hy4-Flash-Engram-Q8.strata
|
| 91 |
-
```
|
| 92 |
-
Verifies the `STRATA\x00\x01` header, all 6 section SHA-256 digests, `4 KiB` alignment of all `20,992` experts, `engram_shard_sha256` pairing, and Section 5 `sq.selfcheck` golden dot-product vectors.
|
| 93 |
-
2. **Speculative Tree-8 Bitwise Parity:**
|
| 94 |
-
```bash
|
| 95 |
-
strata-hy4 --model MyFineTune-SQ-64.strata --engram Hy4-Flash-Engram-Q8.strata \
|
| 96 |
-
--draft dfly2 --spec-tree 1 --spec-fixed --spec-check --gen 128
|
| 97 |
-
```
|
| 98 |
-
Must report **`0 differ`** against plain greedy decoding.
|
| 99 |
-
3. **KL Divergence vs. Your Merged BF16 Fine-Tune (`tools/squant/scale_qat_eval.py`):**
|
| 100 |
-
Target held-out mean KL divergence relative to your BF16 fine-tune: **`< 0.10`** (`SQ-128`), **`< 0.16`** (`SQ-96`), and **`< 0.25`** (`SQ-64`).
|
| 101 |
-
4. **Publish Cleanly on Hugging Face:**
|
| 102 |
-
- Upload your `*-SQ-{64,96,128}.strata` file(s) and `SHA256SUMS.txt` (with Xet enabled on Hugging Face, single `.strata` files over `50 GB` upload directly without splitting).
|
| 103 |
-
- Note in your model card that users can reuse the shared **`Hy4-Flash-Engram-Q8.strata`** (`67.58 GB`) and **`Hy4-Flash-Vision.strata`** (`0.93 GB`) from [`strata-engine/Hy4-Flash-Strata`](https://huggingface.co/strata-engine/Hy4-Flash-Strata), and declare [`tencent/Hy4-flash`](https://huggingface.co/tencent/Hy4-flash) as `base_model` under the [Tencent Hy License](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README.md
CHANGED
|
@@ -29,7 +29,7 @@ tags:
|
|
| 29 |
</p>
|
| 30 |
|
| 31 |
<p align="center">
|
| 32 |
-
<a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="#
|
| 33 |
</p>
|
| 34 |
|
| 35 |
<p>
|
|
@@ -157,16 +157,6 @@ ef09bceff8f01b8d226d2dc56aef09cec719b21b45783ee9b5d64e7241b72a09 Hy4-Flash-Visi
|
|
| 157 |
|
| 158 |
</details>
|
| 159 |
|
| 160 |
-
## Fine-tuning with `.strata`
|
| 161 |
-
|
| 162 |
-
Want to train your own LoRA or SFT fine-tune of `tencent/Hy4-flash` and ship it as a fast `.strata` pack?
|
| 163 |
-
|
| 164 |
-
- **Freeze the 4 Engram lookup tables (`layers.{1,11,21,31}.engram.*`) and Vision tower** during training — your fine-tune will automatically reuse the shared `Hy4-Flash-Engram-Q8.strata` (`67.58 GB`) and `Hy4-Flash-Vision.strata` (`0.93 GB`) files users already have on disk, saving 68.5 GB of downloads.
|
| 165 |
-
- **Merge adapters to BF16 before packing** with `tools/squant/pipeline.py` so `MACaQ` Hadamard smoothing, HBM Block-LDL (`GPTQ`) + 64-flip Hessian refinement, and rank-ordered expert placement fold cleanly with zero runtime cost.
|
| 166 |
-
- **Keep special tokens intact** (`<think:opensource>`, `<tool_calls:opensource>`, stop tokens) and verify every build with `strata-hy4-verify` + `--spec-check` before publishing.
|
| 167 |
-
|
| 168 |
-
Read the full step-by-step guide: **[Fine-Tuning Hy4-Flash for `.strata` v1.0 →](./FINETUNING_WITH_STRATA.md)**
|
| 169 |
-
|
| 170 |
## About Strata
|
| 171 |
|
| 172 |
<p>
|
|
|
|
| 29 |
</p>
|
| 30 |
|
| 31 |
<p align="center">
|
| 32 |
+
<a href="#pick-your-pack"><b>Pick your pack</b></a> · <a href="#get-started">Get started</a> · <a href="#how-close-is-it-to-the-original">Quality</a> · <a href="#performance-with-strata">Speed</a> · <a href="#about-strata">Support Strata</a>
|
| 33 |
</p>
|
| 34 |
|
| 35 |
<p>
|
|
|
|
| 157 |
|
| 158 |
</details>
|
| 159 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
## About Strata
|
| 161 |
|
| 162 |
<p>
|