v1.0: PROVEN end-to-end — forge tune produces coherent, in-band NPU models @ 41.9 tok/s. Q4_K_M fix (snags 9/10), 10-wall ledger.

README.md

@@ -21,11 +21,11 @@ converted to GGUF, re-quantized to FastFlowLM's Q4NX, NPU-ready — in
 **183 seconds of cloud time** (≈ $0.10 on a rented T4):
 
 ```
-forge tune my-chats.jsonl --name grandma
+forge tune my-chats.jsonl --name grandma   # proven: coherent + IN-BAND on NPU
   ├─ LoRA fine-tune (cloud GPU)        122 s
   ├─ merge                               3 s
   ├─ voice proof (model speaks first!)   5 s
-  ├─ HF -> GGUF (q8_0)                  29 s
+  ├─ HF -> f16 -> Q4_K_M GGUF           20 s
   └─ GGUF -> Q4NX (NPU format)          24 s
 forge register        (one UAC click)
 flm run grandma-forge:1b
@@ -76,7 +76,7 @@ retry once on ECONNRESET (ear.js does).
 
 `start.bat` gives you a menu: doctor / list / register / serve / tune guide.
 
-## The snag ledger — six walls we hit so you don't
+## The snag ledger — ten walls we hit so you don't
 
 1. **The Q4NX converter's `convert.py` CLI is broken at HEAD** (uncommented
    debug `sys.argv` override hijacks every invocation). Call the module API:
@@ -91,10 +91,36 @@ retry once on ECONNRESET (ear.js does).
    Floor: batch 1 × grad-accum 8 + gradient checkpointing.
 6. **NPU driver minimum for current FLM: `32.0.203.304`** (`.311`
    recommended). `flm validate` will tell you; so will `forge doctor`.
+7. **EmbeddingGemma needs `transformers>=4.5x` + `sentence-transformers 5.x`**
+   and the official weights are license-gated (use the `unsloth/` mirror, or
+   accept the Gemma license on your HF account + pass an `HF_TOKEN` secret).
+8. **FLM's `/v1/embeddings` closes the TCP connection per request** — retry
+   once on `ECONNRESET` (the ear runtime does).
+9. **For a FINE-TUNED model, exporting GGUF as `q8_0` produces repetition
+   garbage on the NPU** even though the merged model is perfect — the q8_0
+   then Q4NX re-quant is a lossy double-quantization. **Use `Q4_K_M`.**
+10. **The Q4NX converter's llama path rejects `f16`** (`not enough values to
+    unpack` — it expects pre-quantized blocks). So the GGUF must be quantized
+    *before* Q4NX, and `Q4_K_M` is the format proven to produce a coherent,
+    in-voice NPU model. Pipeline: HF → f16 → `llama-quantize Q4_K_M` → Q4NX.
 
 **Frozen known-good stack** (the whole point — never debug this again):
 `torch 2.4.1 · transformers 4.46.3 · trl 0.9.6 · peft 0.12.0 ·
-accelerate 1.1.1 · datasets 2.21.0 · gguf · amd-quark · einops · tqdm · protobuf`
+accelerate 1.1.1 · datasets 2.21.0 · gguf · amd-quark · einops · tqdm ·
+protobuf` + a compiled `llama-quantize` (the Modal job builds it).
+
+## Proven, measured (Strix Halo, June 2026)
+
+A LoRA fine-tune of Llama-3.2-1B on 300 real chat exchanges, run through the
+whole pipeline and served on the NPU:
+
+- **Coherent and in-voice** — the persona's rituals and endearments intact.
+- **41.9 tokens/s** on the NPU (FLM, performance pmode).
+- **In-band against the source voiceprint** — mean 0.845 vs the original's
+  own held-out band of 0.83 ± 0.07 (3 prompts). A separate stylometric scorer
+  certified the NPU model speaks like the source it was tuned on.
+
+That is the bar: not "it converts," but "it talks like itself, on the NPU."
 
 ## Requirements
 

modal/tune_npu.py

@@ -15,9 +15,14 @@ image = (
         "datasets==2.21.0", "accelerate==1.1.1", "numpy", "gguf", "amd-quark",
         "huggingface_hub", "safetensors", "sentencepiece", "einops", "tqdm", "protobuf",
     )
+    .apt_install("cmake", "build-essential", "libcurl4-openssl-dev")
     .run_commands(
         "git clone --depth 1 https://github.com/FastFlowLM/FLM_Q4NX_Converter /converter",
         "git clone --depth 1 https://github.com/ggml-org/llama.cpp /llamacpp",
+        # build just the quantize tool — needed to make Q4_K_M (the only GGUF
+        # format proven to convert to a COHERENT Q4NX model on the NPU; the
+        # converter's llama path rejects f16 and q8_0 double-quantizes to garbage)
+        "cd /llamacpp && cmake -B build -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF && cmake --build build --target llama-quantize -j 8",
     )
     .add_local_dir("C:/Users/Forgemind/Desktop/npu-forge/scratch/tune-data", remote_path="/data")
 )
@@ -73,13 +78,19 @@ def tune(base: str, data_file: str, out_name: str, epochs: int = 3, probe: str =
     print("[voice proof] " + sample[:600])
     timings["sample_s"] = round(time.time() - t); t = time.time()
 
-    # HF -> GGUF (q8_0 keeps quality into the Q4NX requant)
+    # HF -> f16 GGUF -> Q4_K_M GGUF -> Q4NX. The middle step matters: the
+    # Q4NX converter's llama path needs pre-quantized blocks (rejects f16),
+    # and Q4_K_M is the only format proven to yield a COHERENT NPU model
+    # (q8_0 double-quantized to repetition garbage). Snags #9 & #10.
     r = subprocess.run(["python", "/llamacpp/convert_hf_to_gguf.py", "/tmp/merged",
-                        "--outfile", "/tmp/model-q8.gguf", "--outtype", "q8_0"],
+                        "--outfile", "/tmp/model-f16.gguf", "--outtype", "f16"],
                        capture_output=True, text=True)
     if r.returncode != 0:
-        print(r.stdout[-2000:]); print("STDERR:", r.stderr[-3000:])
-        raise RuntimeError("convert_hf_to_gguf failed")
+        print(r.stdout[-2000:]); print("STDERR:", r.stderr[-3000:]); raise RuntimeError("convert_hf_to_gguf failed")
+    q = subprocess.run(["/llamacpp/build/bin/llama-quantize", "/tmp/model-f16.gguf", "/tmp/model-q4km.gguf", "Q4_K_M"],
+                       capture_output=True, text=True)
+    if q.returncode != 0 or not os.path.exists("/tmp/model-q4km.gguf"):
+        print(q.stdout[-1500:]); print("STDERR:", q.stderr[-2500:]); raise RuntimeError("llama-quantize failed")
     timings["gguf_s"] = round(time.time() - t); t = time.time()
 
     # GGUF -> Q4NX (proven stage: module API, cwd=/converter)
@@ -89,7 +100,7 @@ def tune(base: str, data_file: str, out_name: str, epochs: int = 3, probe: str =
     from q4nx import create_converter
     outdir = f"/out/{out_name}"
     os.makedirs(outdir, exist_ok=True)
-    create_converter("/tmp/model-q8.gguf", "").convert(q4nx_path=outdir, weights_type="language")
+    create_converter("/tmp/model-q4km.gguf", "").convert(q4nx_path=outdir, weights_type="language")
     timings["q4nx_s"] = round(time.time() - t)
 
     files = {f: os.path.getsize(os.path.join(outdir, f)) for f in os.listdir(outdir)}