Add GGUF exports for analysis (F16, Q8_0, Q5_K_M, Q4_K_M)

#4
.gitattributes CHANGED
@@ -46,3 +46,7 @@ assets/plays_snake.gif filter=lfs diff=lfs merge=lfs -text
46
  assets/training_distillation.png filter=lfs diff=lfs merge=lfs -text
47
  assets/progress.png filter=lfs diff=lfs merge=lfs -text
48
  assets/training_thinking.png filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
46
  assets/training_distillation.png filter=lfs diff=lfs merge=lfs -text
47
  assets/progress.png filter=lfs diff=lfs merge=lfs -text
48
  assets/training_thinking.png filter=lfs diff=lfs merge=lfs -text
49
+ gguf/THX-02-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
50
+ gguf/THX-02-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
51
+ gguf/THX-02-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
52
+ gguf/THX-02-F16.gguf filter=lfs diff=lfs merge=lfs -text
gguf/README.md ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # THX-02 GGUF (backbone, for analysis)
2
+
3
+ Community-contributed GGUF exports of the **THX-02 backbone** for analysis/inspection, as requested in [discussion #2](https://huggingface.co/HAL-X-AI/THX-02/discussions/2). These are **not** a drop-in replacement for the THX-02 decision runtime.
4
+
5
+ ## Files
6
+
7
+ | file | size | notes |
8
+ |---|---:|---|
9
+ | `THX-02-F16.gguf` | 7.85 GB | f16 base (from bf16 weights) |
10
+ | `THX-02-Q8_0.gguf` | 4.17 GB | 8-bit quant |
11
+ | `THX-02-Q5_K_M.gguf` | 2.87 GB | k-quant, 5-bit mix (medium) |
12
+ | `THX-02-Q4_K_M.gguf` | 2.54 GB | k-quant, 4-bit mix (medium) |
13
+
14
+ All are `qwen35`-architecture GGUFs (378 tensors, hybrid Gated-Delta/linear-attention + full-attention layers preserved).
15
+
16
+ ## What this is (and isn’t)
17
+
18
+ - **Included:** converted backbone weights (`model.safetensors` → `language_model.*`/`visual.*` mapped to standard Qwen3.5 layout) as a single GGUF per quant. Useful for weight analysis, size/quant comparisons, and llama.cpp/gguf tooling experiments.
19
+ - **Not included / not exercised:** `readout.safetensors` (decision readout head), `thinker/` (adapter/MTP), `decision_config.json`/`joint_config.json` marker-readout protocol, and THX-02’s calibrated decision API (`Decider.decide()`). **Standard llama.cpp inference will not reproduce THX-02 decision behavior.**
20
+ - **Vision/multimodal:** THX-02 is multimodal; conversion targeted the backbone in a Qwen3.5-compatible form for analysis. End-to-end multimodal/vision runtime support in llama.cpp for this export is not validated here.
21
+
22
+ ## Source and conversion
23
+
24
+ - **Source:** `HAL-X-AI/THX-02` (THX-02 1.0.0), backbone `model.safetensors` (Qwen3.5-4B hybrid, 32 layers) from local snapshot.
25
+ - **Tooling:** `llama.cpp` @ `3ba1ce0` (`convert_hf_to_gguf.py`, `llama-quantize`), `gguf` Python 0.19.0, torch 2.15.0, transformers 5.22.2. Built with Metal on macOS.
26
+
27
+ THX-02 ships a custom wrapper config (`model_type: thx02`, `THXDecisionModel`) and non-standard tensor key prefixes, so direct conversion failed. Conversion was done from a prepared copy:
28
+ 1. `config.json` = `backbone_config` with `architectures: ["Qwen3_5ForCausalLM"]` (kept nested `text_config`/`vision_config`).
29
+ 2. Tensor keys renamed in `model.safetensors`: `language_model.*` → `model.language_model.*`, `visual.*` → `model.visual.*`.
30
+ 3. Convert and quantize:
31
+ ```bash
32
+ PYTHONPATH=<llama.cpp>/gguf-py python <llama.cpp>/convert_hf_to_gguf.py \
33
+ --outfile THX-02-F16.gguf --outtype f16 <prepared-src>
34
+ llama-quantize THX-02-F16.gguf THX-02-Q8_0.gguf Q8_0
35
+ llama-quantize THX-02-F16.gguf THX-02-Q5_K_M.gguf Q5_K_M
36
+ llama-quantize THX-02-F16.gguf THX-02-Q4_K_M.gguf Q4_K_M
37
+ ```
38
+
39
+ ## Quant type notes (Q4_K_M vs Q8_0)
40
+ - **Q8_0:** simple/legacy 8-bit block quantization (near-lossless, ~8.5 BPW here), uniform scheme.
41
+ - **K-quants (`Q4_K_M`, `Q5_K_M`, etc.):** “k-quant” family that mixes bit-widths/precision across tensors/groups (importance-weighted) for better quality-per-size at low bit widths. The suffix indicates the size/quality tradeoff: **S**mall, **M**edium, **L**arge (e.g. `_K_M` is the balanced default; `_K_S` smaller/faster, `_K_L` higher quality/larger). Q8_0 has no S/M/L suffix because it isn’t a k-quant mix.
42
+
43
+ ## Caveats
44
+ - Backbone-only export for analysis; **not** a functional THX-02 decision model in llama.cpp.
45
+ - Validated by successful conversion/quantization and GGUF metadata inspection (arch `qwen35`, SSM tensors present). Full llama.cpp inference/vision behavior not exhaustively validated.
46
+ - If you need exact THX-02 decisions, use the official `thx02` runtime with the original weights.
gguf/THX-02-F16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f01135793fecfcb7e176b385fcbfc7c9c88f35a57bafb438bc5d6f8d824ce60a
3
+ size 8424390528
gguf/THX-02-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f955fb91d545d1dd892422011b5cb462e8de00969010c5aa172df29f4a8633b
3
+ size 2741241728
gguf/THX-02-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af099f07645cc408a28736cf2bd3a5d50f0e4fc5dd8961d2eb6adccef2823fee
3
+ size 3091695488
gguf/THX-02-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a40874a903d3d49250e354f644ff7aa72624af516a95cae47370c10b74e37887
3
+ size 4482400128