Add RESULTS.md

RESULTS.md

@@ -1,57 +1,52 @@
 # Lesson 07 RESULTS: tiny transformer decider (pankh-decider-v0)
 
-Run: 2026-10-11, Apple M4 Max, PyTorch 2.14.1, device `mps` for training, CPU for latency. Executed end to end with papermill (`lesson_07.executed.ipynb`, kept locally). Whole notebook: **35.5 s** (papermill wall clock 37 s). Seed 42 (plus 43, 44 for the seed check).
+Run (after grading fixes): 2026-10-11, Apple M4 Max, PyTorch 2.14.1, device `mps` for training (GPU shared with other jobs, so training was slower than the first run), CPU for latency. Executed end to end with papermill (`lesson_07.executed.ipynb`, kept locally; re-executed, not hand-patched). Whole notebook: **172.4 s**. Seed 42 (plus 43, 44). Colab T4 time: estimated, not timed. This run supersedes the first run (whose numbers were 82.3% / 46.5% at 94.52%); every number below is from this run.
 
 ## Data
-- pankhllm `evals/pharma-kpi/gen.py` (MIT), run with `python -I`: `gen.py 20000` (training templates, gen.py seed 7) and `gen.py 2000 --heldout` (held-out templates, seed 99). 7 operations + UNSUPPORTED, 250 test questions per class.
-- Templates: 71 training, 35 held out (gen.py holds out every third template per operation).
-- Shape abstraction (dates, codes, numbers, quoted text -> `<date>`, `<code>`, `<num>`, `<text>`, lower-cased), after pankhllm `learn.rs`.
-- Training questions split by shape hash (pankhllm miner recipe): fit 15,205 questions (5,892 shapes), calibration 4,795 questions (1,499 shapes). Both models train on fit only; calibration is used only to choose the fixed threshold.
-- Share of held-out-template words never seen in training: 13.9%.
-
-## Leakage check (done in the notebook)
-- Identical question text in train and test: **0**.
-- Shapes shared by train and test: **0**.
-- Every test question matches one of its own operation's held-out templates (2,000 / 2,000).
-- 271 / 2,000 test questions also fit the pattern of a training template of the same operation (loose patterns such as "{m} for {g} {c}" match text produced by other templates). They are generated from held-out templates; they are not copies. Reported, not removed.
+- pankhllm `evals/pharma-kpi/gen.py` (MIT), run with `python -I`: `gen.py 20000` (training templates, seed 7) and `gen.py 2000 --heldout` (held-out templates, seed 99). 7 operations + UNSUPPORTED, 250 test questions per class.
+- Templates: 71 training, 35 held out. 2,000 test questions = **1,150 distinct shapes**; UNSUPPORTED = **9 shapes**, the most repeated being "what's our company holiday schedule?" **x 46**.
+- Shape abstraction (dates, codes, numbers, quoted text), after pankhllm `learn.rs` (its capitalised-word-run -> `<text>` rule is omitted).
+- Shape-hash split (pankhllm miner style): fit 15,205 questions (5,892 shapes), calibration 4,795 (1,499 shapes). Calibration shapes share templates with training, so they are easier than held-out templates. 13.9% of held-out words never seen in training.
 
-## Models
-- Logistic regression, pankhllm recipe: hashed word + bigram features, 2^16 buckets (pankhllm `native.rs` uses 2^16), distinct shapes only (5,892), class-balanced SGD, L2 1e-3, 30 epochs. Trained in 1.9 s (NumPy).
-- Tiny GPT decider: Lesson 01 GPT class, 4 layers, 4 heads, 128 wide, block 64, dropout 0.1; **word-level tokenizer** over abstracted shapes (210 tokens incl. `<pad>`, `<unk>`, `<sep>` and the 8 operation tokens). 828,416 parameters. Masked loss on the decision token only; word dropout 0.15. 800 steps x batch 64, AdamW, lr 1e-3, warm-up 50 + cosine. Trained in 11.8 s on MPS. Final loss 0.0154.
+## Leakage check
+- Identical questions train/test: **0**. Shared shapes: **0**. 2,000/2,000 test questions match their own held-out templates; 271/2,000 also fit a loose training-template pattern (generated from held-out templates; not copies).
 
-## Numbers (held-out templates, 2,000 questions)
-Fixed threshold = lowest threshold at which the calibration shapes hold >= 98% precision (LR 0.9666, GPT 0.9961). Oracle = best threshold swept on the test set itself (upper bound, not deployable).
+## Models
+- Logistic regression (pankhllm recipe, NumPy): hashed word + bigram, 2^16 buckets, 5,892 distinct shapes, class-balanced SGD, L2 1e-3, 30 epochs; 2.5 s.
+- Tiny GPT (Lesson 01 class): 4 layers, 4 heads, 128 wide, block 64, dropout 0.1, word-level tokenizer (210 tokens), 828,416 params, masked loss on the decision token, word dropout 0.15, 800 steps x 64; 62.3 s on shared MPS; final loss 0.0146.
 
-| Model | Accuracy | Decided @ fixed | Precision @ fixed | Wrong @ fixed | Decided @ oracle 98% | Precision there |
-|---|---|---|---|---|---|---|
-| Logistic regression | 66.2% | 7.8% | 97.45% | 4 | 6.6% | 98.48% |
-| Tiny GPT decider | 82.3% | 46.5% | 94.52% | 51 | 36.4% | 99.45% |
+## Thresholds
+Fixed = lowest threshold at which the calibration shapes hold >= 98% (LR 0.9666, GPT 0.9956). pankhllm's miner searches a 0.5-0.95 grid; under that rule neither model would get a threshold on this calibration set. Oracle = swept on the test set (not deployable).
 
-**CHECK (Lesson 07):** at the threshold fixed on calibration, the transformer decided **46.5%** of held-out-template questions at **94.52%** precision (51 wrong of 930); the logistic regression decided **7.8%** at **97.45%** (4 wrong of 157). Neither held 98% at its fixed threshold. With the threshold swept on the test set, the transformer could decide 36.4% at 99.45%; the regression 6.6% at 98.48%.
+## Known-words gate (new, pankhllm `decisions.min_known_words` = 0.7)
+An operation is not acted on if < 70% of the shape's words were seen in non-UNSUPPORTED fit questions; UNSUPPORTED unaffected. Applied at the same fixed thresholds. It stops 336 of 2,000 held-out questions (for any operation choice).
 
-Calibration shapes: LR accuracy 89.4%, decided 75.6% at 100%; GPT accuracy 95.5%, decided 88.5% at 100%.
+## Numbers (held-out templates, 2,000 questions)
+| Model | Accuracy | Decided @ fixed | Precision | Wrong | Oracle 98% |
+|---|---|---|---|---|---|
+| Logistic regression | 66.2% | 7.8% | 97.45% | 4 | 6.6% |
+| Logistic regression + gate | 66.2% | 7.8% | 97.45% | 4 | 33.9% |
+| Tiny GPT | 81.5% | 49.6% | 94.46% | 55 | 42.2% |
+| Tiny GPT + gate | 81.5% | 41.2% | **98.91%** | 9 | 51.5% |
 
-Seed check (test accuracy / oracle-98 coverage): seed 42 82.35% / 36.35% (kept); seed 43 89.15% / 32.8%; seed 44 81.25% / 26.9%. Spread is large; MPS training is not bit-for-bit repeatable either.
+**CHECK (Lesson 07):** without pankhllm's known-words gate neither model holds 98% at its fixed threshold (transformer 49.6% at 94.46%, regression 7.8% at 97.45%). With the gate the transformer does, deciding **41.2% at 98.91%** (9 wrong); the regression is unchanged at 7.8% at 97.45%.
 
-Most common transformer mistakes: coverage_owner -> kpi_value 64; UNSUPPORTED -> kpi_value 46; kpi_value -> prescriber_list 44; kpi_value -> UNSUPPORTED 41; UNSUPPORTED -> kpi_vs_benchmark 41. Confident wrong answers above the fixed threshold were dominated by "what's our company holiday schedule?" (said kpi_value, 0.9970) and "give me <metric> for T-xxx MTD" (said prescriber_list, ~0.997).
+The transformer's 55 wrong decisions (no gate) are **10 distinct shapes**, 46 of them one question ("what's our company holiday schedule?", said kpi_value): the 94.46% figure is effectively decided by one template. Other confident errors: "give me <metric> for <code> MTD/QTD/YTD" -> prescriber_list (~0.996-0.998).
+Calibration: LR accuracy 89.4%, GPT 95.5% (decided 87.2% at 100%).
 
-## 14 decisions benchmark questions (pankhllm `evals/decisions/questions.jsonl`)
-Different, three-operation catalog; mapped metric_by_period -> kpi_value, top_entities -> kpi_rank, entity_owner -> coverage_owner. "revenue" is not in the pharma vocabulary.
-- Logistic regression: 10/14 correct; decided 5/14 at the fixed threshold, 0 wrong.
-- Tiny GPT decider: 13/14 correct (missed "Why did calls fall in T-112 in 2026-08?", said kpi_value 0.894); decided 7/14 at the fixed threshold, 0 wrong.
-- pankhllm's published 14/14 (BENCHMARK-DECISIONS.md) is a model trained on 120 questions of that three-operation catalog: different test, quoted only as published.
+Seed check (accuracy; decided at its own fixed threshold, precision; oracle-98):
+- seed 42: 81.55%; 49.6% at 94.46%; 42.25% (kept)
+- seed 43: 88.3%; 53.7% at 95.25%; 34.75%
+- seed 44: 81.4%; 56.85% at 93.93%; 26.7%
 
-## Decision latency (CPU, batch 1, 2,000 held-out questions, shape -> decision)
-- Logistic regression (Python + NumPy): p50 0.012 ms, p95 0.017 ms.
-- Tiny GPT decider (PyTorch, 4 threads): p50 0.47 ms, p95 0.526 ms.
-- Model file: pankh-decider-v0/model.pt 3.3 MB (fp32).
+## 14 decisions benchmark questions (mapped onto the pharma catalog; "revenue" not in it)
+LR 10/14 correct, decided 5/14 at its fixed threshold, 0 wrong. GPT 13/14 correct (missed "Why did calls fall in T-112 in 2026-08?" -> kpi_value 0.870), decided 7/14, 0 wrong. pankhllm's published 14/14 is a different model on a three-operation catalog.
 
-## Published pankhllm numbers (from its docs, NOT measured here)
-BENCHMARK-DECISIONS.md, pharma KPI catalog, release build, CPU: template-only training -> templates 52.5% decided at 100% (0 wrong of 1,049), 0.2 ms; + 183 teacher labels -> templates 63.3% at 100%, teacher-written A 67.1% at 100%, teacher-written B held-out half 61.4% at 96.5%. These include pankhllm's structural (slot-fit) check and disagreement check, which this lesson's models do not have.
+## Latency (CPU, batch 1, 2,000 questions, shape -> decision)
+LR p50 0.016 ms, p95 0.023 ms; GPT p50 0.5 ms, p95 0.545 ms (4 threads). model.pt 3.3 MB fp32.
 
-## What went wrong / caveats
-- First draft of the notebook omitted the regression cell (NameError); fixed and re-run. The numbers above are from the successful run.
-- Prototype runs before the notebook (not quoted) showed the same spread across runs (test accuracy 0.68 to 0.80 depending on word dropout and run).
-- Our pure-NumPy LR is not pankhllm's Rust binary; its latency and its features (no capitalised-word or product-vocabulary text slots, no known-words guard) differ.
+## Published pankhllm numbers (BENCHMARK-DECISIONS.md, NOT measured here)
+Template-only: templates 52.5% at 100% (0 wrong of 1,049); + 183 teacher labels: templates 63.3% at 100%, teacher-written A 67.1% at 100%, B 61.4% at 96.5%. These include pankhllm's structural and disagreement checks.
 
-- After execution, two non-computational edits were applied to both the clean and the executed notebook without re-running (a re-run on MPS would change the numbers): one self-check question was reworded, and the threshold widget legend was moved out of the chart (it overlapped the curves). No code that produces a number changed.
+## History / caveats
+- First draft omitted the regression cell (NameError), fixed. A first full run gave different numbers (MPS training is not bit-exact); after grading, the notebook gained the gate, distinct-shape counts, per-seed fixed-threshold numbers, a broken precision line at zero coverage in the widget and figure alt text, and was re-executed. Lesson 08 was re-executed from this run's v0.