Add GGUF exports for analysis (F16, Q8_0, Q5_K_M, Q4_K_M)
#4
by pylotlight - opened
- .gitattributes +4 -0
- gguf/README.md +46 -0
- gguf/THX-02-F16.gguf +3 -0
- gguf/THX-02-Q4_K_M.gguf +3 -0
- gguf/THX-02-Q5_K_M.gguf +3 -0
- gguf/THX-02-Q8_0.gguf +3 -0
.gitattributes
CHANGED
|
@@ -46,3 +46,7 @@ assets/plays_snake.gif filter=lfs diff=lfs merge=lfs -text
|
|
| 46 |
assets/training_distillation.png filter=lfs diff=lfs merge=lfs -text
|
| 47 |
assets/progress.png filter=lfs diff=lfs merge=lfs -text
|
| 48 |
assets/training_thinking.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
assets/training_distillation.png filter=lfs diff=lfs merge=lfs -text
|
| 47 |
assets/progress.png filter=lfs diff=lfs merge=lfs -text
|
| 48 |
assets/training_thinking.png filter=lfs diff=lfs merge=lfs -text
|
| 49 |
+
gguf/THX-02-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 50 |
+
gguf/THX-02-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 51 |
+
gguf/THX-02-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 52 |
+
gguf/THX-02-F16.gguf filter=lfs diff=lfs merge=lfs -text
|
gguf/README.md
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# THX-02 GGUF (backbone, for analysis)
|
| 2 |
+
|
| 3 |
+
Community-contributed GGUF exports of the **THX-02 backbone** for analysis/inspection, as requested in [discussion #2](https://huggingface.co/HAL-X-AI/THX-02/discussions/2). These are **not** a drop-in replacement for the THX-02 decision runtime.
|
| 4 |
+
|
| 5 |
+
## Files
|
| 6 |
+
|
| 7 |
+
| file | size | notes |
|
| 8 |
+
|---|---:|---|
|
| 9 |
+
| `THX-02-F16.gguf` | 7.85 GB | f16 base (from bf16 weights) |
|
| 10 |
+
| `THX-02-Q8_0.gguf` | 4.17 GB | 8-bit quant |
|
| 11 |
+
| `THX-02-Q5_K_M.gguf` | 2.87 GB | k-quant, 5-bit mix (medium) |
|
| 12 |
+
| `THX-02-Q4_K_M.gguf` | 2.54 GB | k-quant, 4-bit mix (medium) |
|
| 13 |
+
|
| 14 |
+
All are `qwen35`-architecture GGUFs (378 tensors, hybrid Gated-Delta/linear-attention + full-attention layers preserved).
|
| 15 |
+
|
| 16 |
+
## What this is (and isn’t)
|
| 17 |
+
|
| 18 |
+
- **Included:** converted backbone weights (`model.safetensors` → `language_model.*`/`visual.*` mapped to standard Qwen3.5 layout) as a single GGUF per quant. Useful for weight analysis, size/quant comparisons, and llama.cpp/gguf tooling experiments.
|
| 19 |
+
- **Not included / not exercised:** `readout.safetensors` (decision readout head), `thinker/` (adapter/MTP), `decision_config.json`/`joint_config.json` marker-readout protocol, and THX-02’s calibrated decision API (`Decider.decide()`). **Standard llama.cpp inference will not reproduce THX-02 decision behavior.**
|
| 20 |
+
- **Vision/multimodal:** THX-02 is multimodal; conversion targeted the backbone in a Qwen3.5-compatible form for analysis. End-to-end multimodal/vision runtime support in llama.cpp for this export is not validated here.
|
| 21 |
+
|
| 22 |
+
## Source and conversion
|
| 23 |
+
|
| 24 |
+
- **Source:** `HAL-X-AI/THX-02` (THX-02 1.0.0), backbone `model.safetensors` (Qwen3.5-4B hybrid, 32 layers) from local snapshot.
|
| 25 |
+
- **Tooling:** `llama.cpp` @ `3ba1ce0` (`convert_hf_to_gguf.py`, `llama-quantize`), `gguf` Python 0.19.0, torch 2.15.0, transformers 5.22.2. Built with Metal on macOS.
|
| 26 |
+
|
| 27 |
+
THX-02 ships a custom wrapper config (`model_type: thx02`, `THXDecisionModel`) and non-standard tensor key prefixes, so direct conversion failed. Conversion was done from a prepared copy:
|
| 28 |
+
1. `config.json` = `backbone_config` with `architectures: ["Qwen3_5ForCausalLM"]` (kept nested `text_config`/`vision_config`).
|
| 29 |
+
2. Tensor keys renamed in `model.safetensors`: `language_model.*` → `model.language_model.*`, `visual.*` → `model.visual.*`.
|
| 30 |
+
3. Convert and quantize:
|
| 31 |
+
```bash
|
| 32 |
+
PYTHONPATH=<llama.cpp>/gguf-py python <llama.cpp>/convert_hf_to_gguf.py \
|
| 33 |
+
--outfile THX-02-F16.gguf --outtype f16 <prepared-src>
|
| 34 |
+
llama-quantize THX-02-F16.gguf THX-02-Q8_0.gguf Q8_0
|
| 35 |
+
llama-quantize THX-02-F16.gguf THX-02-Q5_K_M.gguf Q5_K_M
|
| 36 |
+
llama-quantize THX-02-F16.gguf THX-02-Q4_K_M.gguf Q4_K_M
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
## Quant type notes (Q4_K_M vs Q8_0)
|
| 40 |
+
- **Q8_0:** simple/legacy 8-bit block quantization (near-lossless, ~8.5 BPW here), uniform scheme.
|
| 41 |
+
- **K-quants (`Q4_K_M`, `Q5_K_M`, etc.):** “k-quant” family that mixes bit-widths/precision across tensors/groups (importance-weighted) for better quality-per-size at low bit widths. The suffix indicates the size/quality tradeoff: **S**mall, **M**edium, **L**arge (e.g. `_K_M` is the balanced default; `_K_S` smaller/faster, `_K_L` higher quality/larger). Q8_0 has no S/M/L suffix because it isn’t a k-quant mix.
|
| 42 |
+
|
| 43 |
+
## Caveats
|
| 44 |
+
- Backbone-only export for analysis; **not** a functional THX-02 decision model in llama.cpp.
|
| 45 |
+
- Validated by successful conversion/quantization and GGUF metadata inspection (arch `qwen35`, SSM tensors present). Full llama.cpp inference/vision behavior not exhaustively validated.
|
| 46 |
+
- If you need exact THX-02 decisions, use the official `thx02` runtime with the original weights.
|
gguf/THX-02-F16.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f01135793fecfcb7e176b385fcbfc7c9c88f35a57bafb438bc5d6f8d824ce60a
|
| 3 |
+
size 8424390528
|
gguf/THX-02-Q4_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6f955fb91d545d1dd892422011b5cb462e8de00969010c5aa172df29f4a8633b
|
| 3 |
+
size 2741241728
|
gguf/THX-02-Q5_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:af099f07645cc408a28736cf2bd3a5d50f0e4fc5dd8961d2eb6adccef2823fee
|
| 3 |
+
size 3091695488
|
gguf/THX-02-Q8_0.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a40874a903d3d49250e354f644ff7aa72624af516a95cae47370c10b74e37887
|
| 3 |
+
size 4482400128
|