v0.3: NPU-validated voice ear (111KB head + trainer + runtime), start.bat menu, 47.8 tok/s baseline, snag #8

README.md

@@ -1,89 +1,107 @@
----
-license: mit
-tags:
-- amd
-- ryzen-ai
-- npu
-- xdna2
-- fastflowlm
-- strix-halo
-- lora
-- fine-tuning
-- tools
-pipeline_tag: text-generation
----
-
-# NPU-Forge 🔥 — fine-tune a model and put it on your AMD Ryzen AI NPU in ~3 minutes
-
-**Measured, on a Strix Halo (Ryzen AI MAX+ 395), June 2026:** a LoRA fine-tune of
-Llama-3.2-1B on 300+ real chat exchanges — trained, merged, behavior-verified,
-converted to GGUF, re-quantized to FastFlowLM's Q4NX, NPU-ready — in
-**183 seconds of cloud time** (≈ $0.10 on a rented T4):
-
-```
-forge tune my-chats.jsonl --name grandma
-  ├─ LoRA fine-tune (cloud GPU)        122 s
-  ├─ merge                               3 s
-  ├─ voice proof (model speaks first!)   5 s
-  ├─ HF -> GGUF (q8_0)                  29 s
-  └─ GGUF -> Q4NX (NPU format)          24 s
-forge register        (one UAC click)
-flm run grandma-forge:1b
-```
-
-The "voice proof" stage generates a sample from the merged model *inside the
-training job*, before any conversion — so you know the tune actually took.
-Ours came back with the persona's exact ritual phrases after 2 minutes of
-training. That's the bar.
-
-## What's in this repo
-
-- **`forge.js` / `forge.bat`** — the CLI: `tune`, `convert`, `register`,
-  `list`, `doctor`, `serve`
-- **`modal/tune_npu.py`** — the whole tune→NPU pipeline as one
-  [Modal](https://modal.com) job (bring your own Modal account; T4 is plenty)
-- **`modal/convert_q4nx.py`** — just the GGUF→Q4NX stage (65 s for a 1B)
-- **`bin/assemble.js`** — downloads results and stages the FLM model folder
-- **`bin/register.js` + `register-admin.bat`** — the permanent custom-model
-  registry that survives FLM updates (see below)
-- **`registry.example.json`** — entry template
-
-Chat data format: one JSON per line, `{"messages":[{"role":"user","content":...},{"role":"assistant","content":...}]}`.
-
-## The registry problem (why `forge register` exists)
-
-FLM's `model_list.json` lives in `C:\Program Files\flm\` and **every FLM update
-resets it**, silently de-registering all your custom models. Your model files
-survive (they're in `Documents\flm\models\`) but they vanish from `flm list`.
-Forge keeps its own user-space `registry.json` forever and re-merges with one
-click. `forge doctor` tells you when an update has eaten your registrations.
-
-## The snag ledger — six walls we hit so you don't
-
-1. **The Q4NX converter's `convert.py` CLI is broken at HEAD** (uncommented
-   debug `sys.argv` override hijacks every invocation). Call the module API:
-   `from q4nx import create_converter; create_converter(gguf, "").convert(q4nx_path=out, weights_type="language")`
-2. Converter needs `einops` and `tqdm` beyond its README list, and **must run
-   with cwd = its repo root** (relative `configs/<arch>.json` loads).
-3. **Llama-3.2 tokenizers need `transformers>=4.46`** — the error
-   `untagged enum ModelWrapper` is that wall exactly.
-4. **`transformers 4.46` needs `accelerate>=1.0`** — the error
-   `'AdamW' object has no attribute 'train'` at step 0 is that skew.
-5. **T4 + Llama-3.2's 128k vocab OOMs at batch 4** (loss-logits blowup).
-   Floor: batch 1 × grad-accum 8 + gradient checkpointing.
-6. **NPU driver minimum for current FLM: `32.0.203.304`** (`.311`
-   recommended). `flm validate` will tell you; so will `forge doctor`.
-
-**Frozen known-good stack** (the whole point — never debug this again):
-`torch 2.4.1 · transformers 4.46.3 · trl 0.9.6 · peft 0.12.0 ·
-accelerate 1.1.1 · datasets 2.21.0 · gguf · amd-quark · einops · tqdm · protobuf`
-
-## Requirements
-
-- AMD Ryzen AI machine with XDNA2 NPU (Strix, Strix Halo, Kraken…) +
-  [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM)
-- Node.js (the CLI), Python + a [Modal](https://modal.com) account (the cloud legs)
-- NPU driver ≥ 32.0.203.304
-
-Part of an ongoing project to make local NPUs a first-class home for personal
-AI — voices you own, on silicon you own.
+---
+license: mit
+tags:
+- amd
+- ryzen-ai
+- npu
+- xdna2
+- fastflowlm
+- strix-halo
+- lora
+- fine-tuning
+- tools
+pipeline_tag: text-generation
+---
+
+# NPU-Forge 🔥 — fine-tune a model and put it on your AMD Ryzen AI NPU in ~3 minutes
+
+**Measured, on a Strix Halo (Ryzen AI MAX+ 395), June 2026:** a LoRA fine-tune of
+Llama-3.2-1B on 300+ real chat exchanges — trained, merged, behavior-verified,
+converted to GGUF, re-quantized to FastFlowLM's Q4NX, NPU-ready — in
+**183 seconds of cloud time** (≈ $0.10 on a rented T4):
+
+```
+forge tune my-chats.jsonl --name grandma
+  ├─ LoRA fine-tune (cloud GPU)        122 s
+  ├─ merge                               3 s
+  ├─ voice proof (model speaks first!)   5 s
+  ├─ HF -> GGUF (q8_0)                  29 s
+  └─ GGUF -> Q4NX (NPU format)          24 s
+forge register        (one UAC click)
+flm run grandma-forge:1b
+```
+
+The "voice proof" stage generates a sample from the merged model *inside the
+training job*, before any conversion — so you know the tune actually took.
+Ours came back with the persona's exact ritual phrases after 2 minutes of
+training. That's the bar.
+
+## What's in this repo
+
+- **`forge.js` / `forge.bat`** — the CLI: `tune`, `convert`, `register`,
+  `list`, `doctor`, `serve`
+- **`modal/tune_npu.py`** — the whole tune→NPU pipeline as one
+  [Modal](https://modal.com) job (bring your own Modal account; T4 is plenty)
+- **`modal/convert_q4nx.py`** — just the GGUF→Q4NX stage (65 s for a 1B)
+- **`bin/assemble.js`** — downloads results and stages the FLM model folder
+- **`bin/register.js` + `register-admin.bat`** — the permanent custom-model
+  registry that survives FLM updates (see below)
+- **`registry.example.json`** — entry template
+
+Chat data format: one JSON per line, `{"messages":[{"role":"user","content":...},{"role":"assistant","content":...}]}`.
+
+## The registry problem (why `forge register` exists)
+
+FLM's `model_list.json` lives in `C:\Program Files\flm\` and **every FLM update
+resets it**, silently de-registering all your custom models. Your model files
+survive (they're in `Documents\flm\models\`) but they vanish from `flm list`.
+Forge keeps its own user-space `registry.json` forever and re-merges with one
+click. `forge doctor` tells you when an update has eaten your registrations.
+
+
+## NEW in v0.3 — a voice-verifier "ear" that runs on the NPU
+
+Train a ~111KB classification head over EmbeddingGemma-300m embeddings
+(`modal/train_ear_head.py`, bring your own labeled texts), then run it locally
+with `bin/ear.js` against FLM's `/v1/embeddings` (`flm serve <model> --embed 1`).
+The embeddings come off the NPU; the head is plain JS. In our tests the
+111KB head **matched a fine-tuned 268MB DistilBERT on real-voice accuracy
+(95.9%) and beat it on the hard boundary cases**, live on a Strix Halo NPU.
+
+Also measured: llama3.2:1b chat on the NPU = **47.8 tokens/s** including
+prefill (FLM, performance pmode).
+
+Snag #8: FLM's embeddings endpoint closes the TCP connection per request —
+retry once on ECONNRESET (ear.js does).
+
+`start.bat` gives you a menu: doctor / list / register / serve / tune guide.
+
+## The snag ledger — six walls we hit so you don't
+
+1. **The Q4NX converter's `convert.py` CLI is broken at HEAD** (uncommented
+   debug `sys.argv` override hijacks every invocation). Call the module API:
+   `from q4nx import create_converter; create_converter(gguf, "").convert(q4nx_path=out, weights_type="language")`
+2. Converter needs `einops` and `tqdm` beyond its README list, and **must run
+   with cwd = its repo root** (relative `configs/<arch>.json` loads).
+3. **Llama-3.2 tokenizers need `transformers>=4.46`** — the error
+   `untagged enum ModelWrapper` is that wall exactly.
+4. **`transformers 4.46` needs `accelerate>=1.0`** — the error
+   `'AdamW' object has no attribute 'train'` at step 0 is that skew.
+5. **T4 + Llama-3.2's 128k vocab OOMs at batch 4** (loss-logits blowup).
+   Floor: batch 1 × grad-accum 8 + gradient checkpointing.
+6. **NPU driver minimum for current FLM: `32.0.203.304`** (`.311`
+   recommended). `flm validate` will tell you; so will `forge doctor`.
+
+**Frozen known-good stack** (the whole point — never debug this again):
+`torch 2.4.1 · transformers 4.46.3 · trl 0.9.6 · peft 0.12.0 ·
+accelerate 1.1.1 · datasets 2.21.0 · gguf · amd-quark · einops · tqdm · protobuf`
+
+## Requirements
+
+- AMD Ryzen AI machine with XDNA2 NPU (Strix, Strix Halo, Kraken…) +
+  [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM)
+- Node.js (the CLI), Python + a [Modal](https://modal.com) account (the cloud legs)
+- NPU driver ≥ 32.0.203.304
+
+Part of an ongoing project to make local NPUs a first-class home for personal
+AI — voices you own, on silicon you own.

bin/ear.js

@@ -0,0 +1,90 @@
+#!/usr/bin/env node
+// ear.js — the NPU-accelerated trained ear. Embeddings come from FLM's
+// embed-gemma:300m running on the NPU (/v1/embeddings); the voice head is a
+// 111KB JSON applied right here in JS. No python, no extra runtimes.
+//
+//   node bin/ear.js "some text to identify"            (one-shot)
+//   node bin/ear.js --server-check                     (is an embed-capable flm up?)
+//
+// Programmatic: const { identify } = require('./ear'); await identify(text)
+'use strict';
+const fs = require('fs');
+const path = require('path');
+const http = require('http');
+
+const HEAD = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'ear', 'head.json'), 'utf8'));
+const BASE = process.env.FLM_BASE || 'http://localhost:52625';
+
+function post(urlStr, body) {
+  return new Promise((resolve, reject) => {
+    const u = new URL(urlStr);
+    const data = JSON.stringify(body);
+    const req = http.request({ hostname: u.hostname, port: u.port, path: u.pathname, method: 'POST',
+      headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) }, timeout: 60000 },
+      res => { let b = ''; res.on('data', c => b += c); res.on('end', () => { try { resolve(JSON.parse(b)); } catch (e) { reject(new Error('bad json: ' + b.slice(0, 200))); } }); });
+    req.on('error', reject);
+    req.on('timeout', () => { req.destroy(); reject(new Error('timeout')); });
+    req.write(data); req.end();
+  });
+}
+
+async function embed(text) {
+  let r;
+  try {
+    r = await post(BASE + '/v1/embeddings', { model: 'embed-gemma:300m', input: text });
+  } catch (e) {
+    if (/ECONNRESET|socket hang up/i.test(e.message)) {           // FLM closes TCP per request; back off once
+      await new Promise(res => setTimeout(res, 400));
+      r = await post(BASE + '/v1/embeddings', { model: 'embed-gemma:300m', input: text });
+    } else throw e;
+  }
+  const v = r.data && r.data[0] && r.data[0].embedding;
+  if (!v) throw new Error('no embedding in response: ' + JSON.stringify(r).slice(0, 200));
+  // normalize (head was trained on normalized embeddings)
+  const n = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
+  return v.map(x => x / n);
+}
+
+function applyHead(v) {
+  const { W, b, labels } = HEAD;
+  const logits = b.slice();
+  for (let i = 0; i < v.length; i++) {
+    const wi = W[i];
+    for (let j = 0; j < logits.length; j++) logits[j] += v[i] * wi[j];
+  }
+  const m = Math.max(...logits);
+  const exps = logits.map(x => Math.exp(x - m));
+  const Z = exps.reduce((a, x) => a + x, 0);
+  const probs = exps.map(x => x / Z);
+  const top = probs.indexOf(Math.max(...probs));
+  return { label: labels[top], confidence: +probs[top].toFixed(3),
+    all: Object.fromEntries(labels.map((l, j) => [l, +probs[j].toFixed(3)])) };
+}
+
+async function identify(text) {
+  const t0 = Date.now();
+  const v = await embed(text);
+  const out = applyHead(v);
+  out.ms = Date.now() - t0;
+  return out;
+}
+
+module.exports = { identify, applyHead, embed };
+
+if (require.main === module) {
+  (async () => {
+    const arg = process.argv.slice(2).join(' ');
+    if (arg === '--server-check') {
+      try { await embed('hello'); console.log('embed endpoint OK at ' + BASE); }
+      catch (e) { console.log('embed endpoint unavailable: ' + e.message.slice(0, 120)); console.log('start one with: flm serve embed-gemma:300m --embed 1   (or flm serve <chat-model> --embed 1)'); }
+      return;
+    }
+    if (!arg) { console.error('usage: node bin/ear.js "text"  |  --server-check'); process.exit(1); }
+    try {
+      console.log(JSON.stringify(await identify(arg), null, 2));
+    } catch (e) {
+      console.error('ear error: ' + e.message);
+      process.exit(1);
+    }
+  })();
+}

bin/menu.js

@@ -0,0 +1,57 @@
+#!/usr/bin/env node
+// menu.js — npu-forge front door (start.bat runs this).
+'use strict';
+const readline = require('readline');
+const path = require('path');
+const { spawnSync, spawn } = require('child_process');
+
+const FORGE = path.join(__dirname, '..');
+const ENV = { ...process.env, PYTHONIOENCODING: 'utf-8' };
+const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
+const pending = [], waiting = [];
+let closed = false;
+rl.on('line', l => { const w = waiting.shift(); if (w) w(l); else pending.push(l); });
+rl.on('close', () => { closed = true; while (waiting.length) waiting.shift()(null); });
+const ask = q => { process.stdout.write(q); return new Promise(r => { const b = pending.shift(); if (b !== undefined) { console.log(b); r(b); } else if (closed) r(null); else waiting.push(r); }); };
+
+function forge(args) {
+  rl.pause();
+  spawnSync('node', [path.join(FORGE, 'forge.js'), ...args], { stdio: 'inherit', env: ENV, shell: true });
+  rl.resume();
+}
+
+(async () => {
+  while (true) {
+    console.log('\n==============================================');
+    console.log('   NPU-FORGE — your models, on your NPU');
+    console.log('==============================================');
+    console.log('  1. doctor      — NPU / driver / registry health');
+    console.log('  2. list        — models + your forge customs');
+    console.log('  3. register    — activate customs (UAC, one click)');
+    console.log('  4. serve <tag> — start an NPU server');
+    console.log('  t. tune        — how to fine-tune your own (guide)');
+    console.log('  q. quit');
+    const pick = ((await ask('\nPick: ')) || '').trim().toLowerCase();
+    if (pick === null || pick === 'q' || pick === '') break;
+    if (pick === '1') forge(['doctor']);
+    else if (pick === '2') forge(['list']);
+    else if (pick === '3') {
+      spawn('cmd', ['/c', 'start', '', path.join(FORGE, 'register-admin.bat')], { detached: true });
+      console.log('(UAC window launched — click Yes; results land in register-cmd-log.txt)');
+    }
+    else if (pick.startsWith('4')) {
+      const tag = pick.split(/\s+/)[1] || (await ask('model tag (e.g. llama3.2:1b): ')) || '';
+      if (tag.trim()) { console.log('(Ctrl+C stops the server and returns here)'); forge(['serve', tag.trim()]); }
+    }
+    else if (pick === 't') {
+      console.log('\nFine-tune your own voice onto the NPU (needs a free modal.com account):');
+      console.log('  1. Make chats.jsonl — one line per exchange:');
+      console.log('     {"messages":[{"role":"user","content":"..."},{"role":"assistant","content":"..."}]}');
+      console.log('  2. node forge.js tune chats.jsonl --name myvoice');
+      console.log('  3. node forge.js register   (one UAC click)');
+      console.log('  4. flm run myvoice-forge:1b — your voice, on your NPU.');
+      console.log('  (~3 minutes and ~$0.10 of cloud time for a 1B; see README.md)');
+    }
+  }
+  rl.close();
+})();

modal/train_ear_head.py

@@ -0,0 +1,88 @@
+"""npu-forge-ear: train a tiny voice-verifier HEAD on EmbeddingGemma-300m
+embeddings. At runtime the embeddings come from FLM's NPU (embed-gemma:300m,
+unquantized) and the head is ~20KB of JSON applied in plain JS — a fully
+NPU-accelerated 'trained ear' with zero extra runtimes.
+
+  modal run train_ear_head.py
+"""
+import modal
+
+app = modal.App("npu-forge-ear")
+image = (
+    modal.Image.debian_slim(python_version="3.11")
+    .pip_install("torch==2.5.1", "transformers==4.56.2", "sentence-transformers==5.1.0", "numpy", "huggingface_hub")
+    .add_local_dir("C:/Users/Forgemind/Desktop/voice-harness/scratch/verifier-data", remote_path="/data")
+)
+vol = modal.Volume.from_name("npu-forge-out", create_if_missing=True)
+
+
+@app.function(image=image, gpu="T4", timeout=2400, volumes={"/out": vol}, secrets=[modal.Secret.from_name("huggingface-token")])
+def train():
+    import json, numpy as np, torch
+    from sentence_transformers import SentenceTransformer
+
+    def load(p):
+        return [json.loads(l) for l in open(p, encoding="utf-8") if l.strip()]
+
+    labels = json.load(open("/data/labels.json"))
+    lab2id = {l: i for i, l in enumerate(labels)}
+    train_rows, eval_rows, probe_rows = load("/data/train.jsonl"), load("/data/eval.jsonl"), load("/data/probe.jsonl")
+
+    candidates = ["google/embeddinggemma-300m", "unsloth/embeddinggemma-300m", "onnx-community/embeddinggemma-300m-ONNX"]
+    model = None
+    for cid in candidates:
+        try:
+            model = SentenceTransformer(cid, device="cuda")
+            print("[embed model] " + cid)
+            break
+        except Exception as e:
+            print(f"[skip] {cid}: {str(e)[:120]}")
+    if model is None:
+        raise RuntimeError("no embeddinggemma variant loadable")
+
+    def embed(rows):
+        return np.asarray(model.encode([r["text"] for r in rows], batch_size=64, show_progress_bar=False, normalize_embeddings=True))
+
+    Xtr, Xev, Xpr = embed(train_rows), embed(eval_rows), embed(probe_rows)
+    ytr = np.array([lab2id[r["label"]] for r in train_rows])
+
+    # multinomial logistic head, full-batch
+    d, k = Xtr.shape[1], len(labels)
+    W = torch.zeros(d, k, requires_grad=True)
+    b = torch.zeros(k, requires_grad=True)
+    Xt = torch.tensor(Xtr, dtype=torch.float32)
+    yt = torch.tensor(ytr)
+    opt = torch.optim.Adam([W, b], lr=0.05)
+    for i in range(400):
+        loss = torch.nn.functional.cross_entropy(Xt @ W + b, yt) + 1e-4 * W.pow(2).sum()
+        opt.zero_grad(); loss.backward(); opt.step()
+    print(f"[head] final loss {loss.item():.4f}, dim={d}")
+
+    def acc(X, rows):
+        pred = (torch.tensor(X, dtype=torch.float32) @ W + b).argmax(-1).numpy()
+        per = {}
+        for r, p in zip(rows, pred):
+            s = per.setdefault(r["label"], [0, 0]); s[1] += 1
+            if labels[p] == r["label"]: s[0] += 1
+        overall = sum(v[0] for v in per.values()) / max(1, len(rows))
+        return overall, {l: f"{v[0]}/{v[1]}" for l, v in per.items()}
+
+    ev_overall, ev_per = acc(Xev, eval_rows)
+    pr_overall, pr_per = acc(Xpr, probe_rows)
+
+    head = {"labels": labels, "dim": d, "normalize": True,
+            "W": W.detach().numpy().round(5).tolist(), "b": b.detach().numpy().round(5).tolist()}
+    import os
+    os.makedirs("/out/ear-head", exist_ok=True)
+    with open("/out/ear-head/head.json", "w") as f:
+        json.dump(head, f)
+    vol.commit()
+    return {"eval_overall": round(ev_overall, 4), "eval_per_class": ev_per,
+            "probe_recognized": pr_per, "probe_overall": round(pr_overall, 4),
+            "head_kb": round(len(json.dumps(head)) / 1024)}
+
+
+@app.local_entrypoint()
+def main():
+    import json
+    print(json.dumps(train.remote(), indent=2))

start.bat

@@ -0,0 +1,11 @@
+@echo off
+title NPU-Forge
+cd /d "%~dp0"
+where node >nul 2>nul
+if errorlevel 1 (
+  echo [!] Node.js is required. Free download: https://nodejs.org - pick LTS, one click.
+  pause
+  exit /b 1
+)
+node "%~dp0bin\menu.js"
+pause