Public-ready card: correct numeric/tokenizer conclusion, add accuracy + tool-call validation, REAM rejection, evidence files

README.md

@@ -127,6 +127,34 @@ smooth over quantization noise the way routed experts get) showed a real,
 consistent, cross-category improvement -- about 10x any depth-window effect,
 for roughly +0.03 bpw / +1GB.
 
+### Capability validation (exact-answer accuracy, not perplexity)
+
+Perplexity is a proxy: it rewards a confident distribution, which is not the
+same as a correct one. To check that the numbers above reflect real capability,
+this model was also scored **right/wrong** on exact-answer tasks where a good
+proxy cannot substitute for a correct answer — greedy decoding, all sampler
+fields pinned:
+
+| test | items | score |
+|---|---|---|
+| multi-step arithmetic + factual recall | 24 | **24/24** |
+| tool selection + argument extraction (from a provided catalog) | 15 | **14/15** |
+
+These are spot checks, not a benchmark suite, but they confirm the model
+reasons and calls tools correctly, not merely that it has low perplexity. Item
+sets and per-item results: `artifacts/acc_*.json`, `artifacts/tc_*.json`.
+
+### Also tested and rejected: expert merging (REAM)
+
+Merging the 43 lowest-saliency experts per layer into their nearest kept
+neighbour (instead of pruning them) *lowered perplexity substantially* —
+overall text NLL −0.194, with tool_use perplexity dropping 41% (26.5 → 15.6).
+On the exact-answer tests above, however, the merged model scored **identically**
+(24/24 → 23/24, tool-call 14/15 → 14/15): the perplexity gain was distribution
+smoothing with no capability behind it. A clean demonstration that perplexity
+can move without capability following — and the reason pruning, not merging, was
+shipped. Full write-up in `docs/REAM-RESULT.md`.
+
 ### Also tested and rejected: per-expert mixed-precision tiering
 
 Splitting each layer's 245 kept experts into 61 hot (6-bit) / 123 base
@@ -153,6 +181,14 @@ repo:
 - `docs/SHARED8-RESULT.md` -- this model's final validation.
 - `docs/TIERED-EXPERTS-RESULT.md` -- the rejected per-expert tiering
   experiment.
+- `docs/REAM-RESULT.md` -- the rejected expert-merging experiment and the
+  perplexity-vs-capability demonstration behind it.
+- `docs/TOKENIZER-INVESTIGATION.md` -- why the numeric behaviour is intrinsic
+  tokenization and not a checkpoint defect (and why the base tokenizer must not
+  be copied in).
+- `artifacts/acc_*.json`, `artifacts/tc_*.json` -- per-item results for the
+  exact-answer accuracy and tool-call spot checks, with
+  `artifacts/accuracy_eval.py` and `artifacts/toolcall_eval.py`.
 - `artifacts/*.json` -- raw per-category NLL/PPL from every eval run
   referenced above, and `plan_p15_blend03.json`, the exact REAP plan
   (per-layer kept/pruned expert indices and blended scores) used to build
@@ -182,97 +218,55 @@ ones are a poor fit for this tokenizer:
 {"temperature": 0.5, "top_p": 0.9, "top_k": 100, "min_p": 0.05, "repetition_penalty": 1.0}
 ```
 
-**Keep `repetition_penalty` at 1.0.** This tokenizer emits **one token per
-digit** (ids 19–28 for `'0'`–`'9'`) and a **standalone space token** (223),
-so numbers carry no token-level redundancy — a single mis-sampled token
-corrupts an entire number, while prose has enough redundancy to absorb it.
-Repetition penalty is the worst offender, and it is the only common sampler
-setting that *reorders* candidates rather than merely filtering them — so
-`top_p` and `min_p` cannot compensate for it.
-
-`mlx_lm`'s implementation divides raw logits for any token seen in the last
-20, which has two consequences. The shift is proportional to the logit, so the
-model's **most confident** token takes the largest absolute hit. And `0` and
-`1` are among the most frequent tokens in source code, so they sit in that
-rolling window almost permanently while `.` often does not — demote the
-correct digit, leave the runner-up untouched, and `.` wins.
-
-The symptom is digits replaced by letters (`2456` → `2a5b`) or stray
-delimiters inside numbers, with everything non-numeric looking perfect. Use
-`min_p` for loop suppression instead — raise it to 0.1 before reaching for a
-penalty.
-
-### Known issue: numbers corrupted inside the reasoning block
-
-In a context dense with delimited numerals (MAC addresses, IP tables, float
-literals), the model can commit to a wrong number at its **first mention
-inside the chain of thought**, then answer its own corrupted premise faithfully:
-
-```
-prompt:  "...kill process ID 18452 with signal 9"   (MAC-address context)
-answer:  kill -9 1845                                5/5 at temperature 0
-```
-
-Also observed as delimiter insertion — `2456, 1337, 495` → `2 4 5 6, 1 3 3 7,
-4 9 5`, and in float-dense code `12.0` → `1.2.0`, `sed -n '447,495p'` →
-`sed -n '4.4.7,4.9.5p'`.
-
-In the worst observed case the model does not corrupt the digits at all — it
-**substitutes content from the context**, answering a request for
-`2456, 1337, 495` with `2a:5b:c4:9f:1d:e0, 3c:df:a9:b2:44:`. The mild form
-(`2456` → `2 4 5 6`) and this one appear to be the same failure at different
-strengths: numeric output being captured by numerals already in the window.
-The separator varies with conditions (`2 4 5 6` vs `2,4,5,6,...`).
-
-**What it is not.** Measured on this checkpoint:
+`repetition_penalty` is pinned to **1.0**. It is the only common sampler
+setting that *reorders* candidates rather than merely filtering them, so it is
+the riskiest knob to raise — but note it was **measured not to be the cause** of
+the numeric behaviour described under *Numeric handling* below (a direct A/B at
+1.02 vs 1.0 was identical). Keep it at 1.0 as a safe default; use `min_p` for
+loop suppression, raising it to 0.1 before reaching for a penalty.
+
+Temperature is deliberately below StepFun's suggested 0.7–1.0. That guidance is
+tuned for prose diversity; this tokenizer spells numbers one digit at a time
+(see *Numeric handling*), which makes numeric output more temperature-sensitive
+than prose. Raise it for open-ended writing; keep it low for tool calls,
+structured output, and anything numeric.
+
+### Numeric handling (a base-model property, not a defect)
+
+Step-3.7's tokenizer emits **one token per digit** (ids 19–28 for `'0'`–`'9'`)
+plus a **standalone space token** (223). Numbers therefore carry no
+token-level redundancy: a single mis-sampled or mis-copied token corrupts an
+entire number, while prose has enough redundancy to absorb it. In long,
+number-dense contexts (IP/MAC tables, float-heavy source) this shows up as
+digits dropped or split by a delimiter — `2456` → `2 4 5 6`, `12.0` → `1.2.0`,
+`sed -n '447,495p'` → `sed -n '4.4.7,4.9.5p'` — with non-numeric output
+unaffected.
+
+**This is StepFun's tokenization design, not something introduced by the
+pruning or quantization here.** It was investigated at length and each
+plausible in-checkpoint cause was ruled out with measurements:
 
-- *Not `repetition_penalty`.* Direct A/B, everything else identical:
-  penalty 1.02 gives 0/8 and 1/8 on the two failing cases, penalty pinned to
-  1.0 gives 0/8 and 0/8. No effect.
 - *Not the quantized output head.* Digit rows carry 1.03x the all-row
-  reconstruction error, and the quantization perturbation is 12% of the
-  tightest inter-digit margin.
+  reconstruction error; the quantization perturbation is 12% of the tightest
+  inter-digit argmax margin.
 - *Not degraded digit copying.* Teacher-forced probes (no sampler in the path)
-  rank the correct next digit **first** at p = 0.91–0.9996, with and without
-  the triggering context.
-- *Not a reading failure.* In the same context the model reports `18452` has
-  5 digits and echoes it back verbatim, both correct.
-- *Not fixable with `reasoning_effort`.* Setting `Reasoning: medium` fixed one
-  case completely (0/12 → 12/12, and a 3x shorter trace) but made the other
-  substantially worse — traces grew from ~6.1k to ~13.2k characters, 11 of 12
-  samples exceeded an 8000-token budget without producing an answer, and the
-  one that finished returned the context substitution above. Do not treat it
-  as a mitigation.
-
-**What it is.** A narrow interaction between a specific digit string and a
-specific context: a different 5-digit PID (`90317`) in the identical context
-answers correctly, and `18452` with no context answers correctly. Once the
-wrong value appears in the reasoning it is never recovered — in one trace the
-model re-read the question more than twenty times and reaffirmed the corrupted
-value every time, because its own earlier text had become the strongest
-evidence in context.
-
-**Practical guidance.** No sampler or template setting fixes this, so the
-mitigation is operational. Verify numeric arguments programmatically rather
-than relying on the model to self-check: a wrong value that is *syntactically
-valid* (`idx+1.5`, a shortened PID) is caught by neither a syntax check nor the
-model's own review. Prefer structured file-editing tools over line-numbered
-`sed`. Where exact numerals matter, avoid keeping large delimiter-dense blocks
-(ARP tables, IP configs, float-heavy source) in the same window as the request.
-
-Note also that a runtime's own sampler settings override what this file
-declares — an LM Studio per-model config with repeat penalty enabled will win
-over the `repetition_penalty: 1.0` in `generation_config.json`.
-
-Not yet isolated to this build versus upstream Step-3.7 — no unpruned
-reference exists as a loadable quant, and the evidence above (clean probes,
-clean reads, correct handling of other numbers) points away from the pruning
-and quantization as the cause.
-
-Temperature is deliberately below StepFun's suggested 0.7–1.0 for the same
-reason: that guidance is tuned for prose diversity and does not account for
-per-digit tokenization. Raise it for open-ended writing; keep it low for
-tool calls, structured output, and anything numeric.
+  rank the correct next digit **first** at p = 0.91–0.9996.
+- *Not the sampler.* A direct `repetition_penalty` A/B (1.02 vs 1.0) was
+  identical.
+- *Not a reading failure.* In the same context the model reports `18452` has 5
+  digits and echoes it back verbatim.
+
+**Do not "fix" this by replacing `tokenizer.json` with the base checkpoint's.**
+The upstream `stepfun-ai/Step-3.7-Flash` `tokenizer.json` is not usable
+standalone — read directly by the `tokenizers` library (which is what MLX
+runtimes do) it yields **zero tokens for Chinese** (`你好世界` → `[]`) and
+mangles English word boundaries; it only works via `transformers`' load-time
+reconstruction. This checkpoint ships the correct, standalone-usable tokenizer.
+
+**Practical guidance.** Keep temperature low for numeric/tool work; for
+agentic use, verify numeric arguments programmatically rather than relying on
+the model to re-check its own output. A helper (`scripts/numeric_guard.py`) and
+a Pi editor extension for exactly this are included in the source repo.
 
 ## Reasoning
 
@@ -325,8 +319,10 @@ limitations.
 - No BF16 multimodal reference was run, so absolute (not just relative)
   vision-task damage from quantization remains unmeasured.
 - This checkpoint has not been evaluated on standard downstream benchmarks
-  (MMLU, GSM8K, SWE-bench, etc.) — all numbers above are perplexity/NLL
-  proxies, not task accuracy.
+  (MMLU, GSM8K, SWE-bench, etc.). The category tables are perplexity/NLL
+  proxies; the *Capability validation* section adds small exact-answer accuracy
+  spot checks (39 items), but these are not a substitute for a full benchmark
+  suite.
 - **JSON-schema structured output can crash `llguidance`-based runtimes.**
   This is a base-model tokenizer interaction, not a defect introduced by the
   pruning or quantization here — it reproduces with zero model weights loaded,

artifacts/acc_ream.json

@@ -0,0 +1,187 @@
+{
+  "model": "step-3.7-p15-ream-shared8-head8",
+  "overall": [
+    23,
+    24
+  ],
+  "by_cat": {
+    "math": [
+      13,
+      14
+    ],
+    "factual": [
+      10,
+      10
+    ]
+  },
+  "results": [
+    {
+      "cat": "math",
+      "expected": "330",
+      "ok": true,
+      "answer": "330",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "84",
+      "ok": true,
+      "answer": "84",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "120",
+      "ok": false,
+      "answer": "52",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "355",
+      "ok": true,
+      "answer": "355",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "18",
+      "ok": true,
+      "answer": "18",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "270",
+      "ok": true,
+      "answer": "270",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "101",
+      "ok": true,
+      "answer": "101",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "114",
+      "ok": true,
+      "answer": "114",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "215",
+      "ok": true,
+      "answer": "215",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "60",
+      "ok": true,
+      "answer": "60",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "1",
+      "ok": true,
+      "answer": "1",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "25",
+      "ok": true,
+      "answer": "25",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "518.4",
+      "ok": true,
+      "answer": "To calculate the final price:\n1. First apply the 20% discount: $600 * (1 - 0.2) ",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "36",
+      "ok": true,
+      "answer": "36",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Au",
+      "ok": true,
+      "answer": "Au",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "1969",
+      "ok": true,
+      "answer": "1969",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Ottawa",
+      "ok": true,
+      "answer": "Ottawa",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "6",
+      "ok": true,
+      "answer": "6",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Mars",
+      "ok": true,
+      "answer": "Mars",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Pacific",
+      "ok": true,
+      "answer": "Pacific",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Shakespeare",
+      "ok": true,
+      "answer": "Shakespeare",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "0",
+      "ok": true,
+      "answer": "0",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "carbon dioxide",
+      "ok": true,
+      "answer": "Carbon dioxide",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "90",
+      "ok": true,
+      "answer": "90",
+      "finish": "stop"
+    }
+  ]
+}

artifacts/acc_shared8.json

@@ -0,0 +1,187 @@
+{
+  "model": "step-3.7-p15-vblend-shared8-head8",
+  "overall": [
+    24,
+    24
+  ],
+  "by_cat": {
+    "math": [
+      14,
+      14
+    ],
+    "factual": [
+      10,
+      10
+    ]
+  },
+  "results": [
+    {
+      "cat": "math",
+      "expected": "330",
+      "ok": true,
+      "answer": "330",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "84",
+      "ok": true,
+      "answer": "84",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "120",
+      "ok": true,
+      "answer": "120",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "355",
+      "ok": true,
+      "answer": "355",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "18",
+      "ok": true,
+      "answer": "18",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "270",
+      "ok": true,
+      "answer": "270",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "101",
+      "ok": true,
+      "answer": "101",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "114",
+      "ok": true,
+      "answer": "114",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "215",
+      "ok": true,
+      "answer": "To solve this:\n1.  Interpret the formatted times correctly: departure is **09:45",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "60",
+      "ok": true,
+      "answer": "60",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "1",
+      "ok": true,
+      "answer": "1",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "25",
+      "ok": true,
+      "answer": "25",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "518.4",
+      "ok": true,
+      "answer": "518.4",
+      "finish": "stop"
+    },
+    {
+      "cat": "math",
+      "expected": "36",
+      "ok": true,
+      "answer": "36",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Au",
+      "ok": true,
+      "answer": "Au",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "1969",
+      "ok": true,
+      "answer": "1969",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Ottawa",
+      "ok": true,
+      "answer": "Ottawa",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "6",
+      "ok": true,
+      "answer": "6",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Mars",
+      "ok": true,
+      "answer": "Mars",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Pacific",
+      "ok": true,
+      "answer": "Pacific",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "Shakespeare",
+      "ok": true,
+      "answer": "Shakespeare",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "0",
+      "ok": true,
+      "answer": "0",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "carbon dioxide",
+      "ok": true,
+      "answer": "Carbon dioxide.",
+      "finish": "stop"
+    },
+    {
+      "cat": "factual",
+      "expected": "90",
+      "ok": true,
+      "answer": "90",
+      "finish": "stop"
+    }
+  ]
+}

artifacts/accuracy_eval.py

@@ -0,0 +1,113 @@
+"""Discriminating eval: does REAM's -0.19 PPL reflect capability or smoothing?
+
+PPL rewards a flatter output distribution; it cannot tell a real gain from a
+harmless smoothing effect. This scores EXACT-ANSWER tasks right/wrong -- tasks
+where there is one correct answer and smoothing cannot help. Same scorer for
+both models, so the comparison is fair regardless of scorer strictness.
+
+Prediction if -0.19 is smoothing: REAM accuracy ~= shared8-head8 accuracy,
+especially on math (PPL already showed reasoning_math did NOT improve, +0.007).
+If REAM is genuinely better, math accuracy rises.
+
+Usage: python accuracy_eval.py <model-id> <out.json>
+"""
+import json, re, sys, urllib.request
+
+MODEL = sys.argv[1]
+OUT = sys.argv[2] if len(sys.argv) > 2 else None
+
+# (category, prompt, expected). Hand-written to avoid benchmark contamination;
+# multi-step so smoothing cannot luck into them, but within a strong model's reach.
+ITEMS = [
+    ("math", "A tank holds 480 liters. It drains at 12 liters per minute for 15 minutes, then is refilled by 30 liters. How many liters are in it now? Give only the number.", "330"),
+    ("math", "A book has 342 pages. Maria reads 18 pages a day for 9 days, then 24 pages a day for 4 days. How many pages are left? Number only.", "84"),
+    ("math", "There are 7 boxes with 23 apples each. 41 apples are rotten and removed. How many good apples remain? Number only.", "120"),
+    ("math", "A car travels 65 km/h for 3 hours, then 80 km/h for 2 hours. Total distance in km? Number only.", "355"),
+    ("math", "A store sells pens at 3 for $2. How much do 27 pens cost, in dollars? Number only.", "18"),
+    ("math", "Compute 144 divided by 8, then multiply the result by 15. Number only.", "270"),
+    ("math", "A rectangle is 14 by 9. A square of side 5 is cut out. Remaining area? Number only.", "101"),
+    ("math", "Sarah has $250. She buys 6 shirts at $18 each and 2 hats at $14 each. How much money is left? Number only.", "114"),
+    ("math", "A train departs at 09:45 and arrives at 13:20. Journey length in minutes? Number only.", "215"),
+    ("math", "If 5 machines make 5 widgets in 5 minutes, how many widgets do 5 machines make in 60 minutes? Number only.", "60"),
+    ("math", "A recipe needs 3 eggs per cake. You have 40 eggs. After making as many whole cakes as possible, how many eggs are left over? Number only.", "1"),
+    ("math", "The sum of three consecutive integers is 72. What is the largest of them? Number only.", "25"),
+    ("math", "A phone costs $600. It is discounted 20%, then 8% sales tax is added. Final price in dollars? Number only.", "518.4"),
+    ("math", "A garden is 12 m by 8 m. A path 1 m wide runs around the inside edge. Area of the path in square metres? Number only.", "36"),
+    ("factual", "What is the chemical symbol for gold? Symbol only.", "Au"),
+    ("factual", "In what year did the first human land on the Moon? Year only.", "1969"),
+    ("factual", "What is the capital city of Canada? One word.", "Ottawa"),
+    ("factual", "How many sides does a hexagon have? Number only.", "6"),
+    ("factual", "What planet is known as the Red Planet? One word.", "Mars"),
+    ("factual", "What is the largest ocean on Earth? One word.", "Pacific"),
+    ("factual", "Who wrote the play 'Romeo and Juliet'? Last name only.", "Shakespeare"),
+    ("factual", "What is the freezing point of water in Celsius? Number only.", "0"),
+    ("factual", "What gas do plants primarily absorb from the air for photosynthesis? Two words.", "carbon dioxide"),
+    ("factual", "How many degrees are in a right angle? Number only.", "90"),
+]
+
+
+def ask(prompt, mt=3500):
+    body = json.dumps({"model": MODEL, "messages": [{"role": "user", "content": prompt}],
+                       "temperature": 0.0, "top_p": 1.0, "top_k": 0, "min_p": 0.0,
+                       "repetition_penalty": 1.0, "max_tokens": mt, "stream": False}).encode()
+    r = urllib.request.Request("http://localhost:1234/v1/chat/completions", data=body,
+                               headers={"Content-Type": "application/json"})
+    d = json.load(urllib.request.urlopen(r, timeout=600))["choices"][0]
+    m = d["message"]
+    return (m.get("content") or "").strip(), d.get("finish_reason")
+
+
+def norm_num(s):
+    # strip digit-internal separators (handles the corruption AND legit commas);
+    # both models scored identically so this is fair.
+    return re.sub(r"(?<=\d)[ ,_](?=\d)", "", s)
+
+
+def check(ans, expected, cat):
+    a = norm_num(ans)
+    if cat == "factual" and not expected.replace(".", "").isdigit():
+        return expected.lower() in a.lower()
+    # numeric: compare the set of numbers present; credit if expected appears
+    want = expected
+    nums = re.findall(r"-?\d+\.?\d*", a)
+    # exact match, or match ignoring trailing .0
+    for n in nums:
+        if n == want or n.rstrip("0").rstrip(".") == want.rstrip("0").rstrip("."):
+            return True
+    return False
+
+
+def self_test():
+    assert check("330", "330", "math")
+    assert check("The answer is 518.40 dollars.", "518.4", "math")
+    assert check("3 3 0", "330", "math")            # space-corrupted
+    assert check("**Au**", "Au", "factual")
+    assert not check("The answer is 331.", "330", "math")
+    assert check("Ottawa is the capital.", "Ottawa", "factual")
+    print("[selftest] answer-extraction ok", flush=True)
+
+
+if __name__ == "__main__":
+    self_test()
+    if MODEL == "SELFTEST":
+        sys.exit(0)
+    from collections import defaultdict
+    tally = defaultdict(lambda: [0, 0]); results = []
+    for i, (cat, q, exp) in enumerate(ITEMS):
+        try:
+            ans, fin = ask(q)
+        except Exception as e:
+            print(f"  item {i} ERROR {e}", flush=True); continue
+        ok = check(ans, exp, cat)
+        tally[cat][0] += ok; tally[cat][1] += 1
+        results.append({"cat": cat, "expected": exp, "ok": ok, "answer": ans[:80], "finish": fin})
+        print(f"  [{'OK ' if ok else 'XX '}] {cat:8} want={exp:14} got={ans[:44]!r}", flush=True)
+    tot_ok = sum(v[0] for v in tally.values()); tot_n = sum(v[1] for v in tally.values())
+    print(f"\n{MODEL}", flush=True)
+    for cat, (ok, n) in sorted(tally.items()):
+        print(f"  {cat:10} {ok}/{n}", flush=True)
+    print(f"  OVERALL   {tot_ok}/{tot_n}  ({100*tot_ok/tot_n:.0f}%)", flush=True)
+    if OUT:
+        json.dump({"model": MODEL, "overall": [tot_ok, tot_n],
+                   "by_cat": {k: v for k, v in tally.items()}, "results": results},
+                  open(OUT, "w"), indent=2)

artifacts/tc_ream.json

@@ -0,0 +1,158 @@
+{
+  "model": "step-3.7-p15-ream-shared8-head8",
+  "score": [
+    14,
+    15
+  ],
+  "results": [
+    {
+      "req": "Set a timer for 25 minutes.",
+      "ok": true,
+      "why": "ok",
+      "name": "set_timer",
+      "args": {
+        "minutes": 25
+      }
+    },
+    {
+      "req": "Convert 150 USD to EUR.",
+      "ok": true,
+      "why": "ok",
+      "name": "convert_currency",
+      "args": {
+        "amount": 150,
+        "from": "USD",
+        "to": "EUR"
+      }
+    },
+    {
+      "req": "Book a flight from Denver to Miami for 3 passengers.",
+      "ok": true,
+      "why": "ok",
+      "name": "book_flight",
+      "args": {
+        "from_city": "Denver",
+        "to_city": "Miami",
+        "passengers": 3
+      }
+    },
+    {
+      "req": "Resize the image to 800 by 600.",
+      "ok": true,
+      "why": "ok",
+      "name": "resize_image",
+      "args": {
+        "width": 800,
+        "height": 600
+      }
+    },
+    {
+      "req": "Set the thermostat to 68 degrees.",
+      "ok": true,
+      "why": "ok",
+      "name": "set_thermostat",
+      "args": {
+        "temperature": 68
+      }
+    },
+    {
+      "req": "Add 5 apples to my cart.",
+      "ok": true,
+      "why": "ok",
+      "name": "add_to_cart",
+      "args": {
+        "item": "apples",
+        "quantity": 5
+      }
+    },
+    {
+      "req": "Calculate an 18% tip on an 80 dollar bill.",
+      "ok": true,
+      "why": "ok",
+      "name": "calculate_tip",
+      "args": {
+        "bill": 80,
+        "percent": 18
+      }
+    },
+    {
+      "req": "Play Bohemian Rhapsody by Queen.",
+      "ok": true,
+      "why": "ok",
+      "name": "play_song",
+      "args": {
+        "title": "Bohemian Rhapsody",
+        "artist": "Queen"
+      }
+    },
+    {
+      "req": "What's the weather in Paris?",
+      "ok": true,
+      "why": "ok",
+      "name": "get_weather",
+      "args": {
+        "city": "Paris"
+      }
+    },
+    {
+      "req": "Get the stock price for AAPL.",
+      "ok": true,
+      "why": "ok",
+      "name": "get_stock_price",
+      "args": {
+        "ticker": "AAPL"
+      }
+    },
+    {
+      "req": "Find headphones under 200 dollars.",
+      "ok": true,
+      "why": "ok",
+      "name": "search_products",
+      "args": {
+        "query": "headphones",
+        "max_price": 200
+      }
+    },
+    {
+      "req": "Translate 'good morning' into Spanish.",
+      "ok": false,
+      "why": "target_lang: want 'Spanish' got 'es'",
+      "name": "translate",
+      "args": {
+        "text": "good morning",
+        "target_lang": "es"
+      }
+    },
+    {
+      "req": "Transfer 340 dollars from checking to savings.",
+      "ok": true,
+      "why": "ok",
+      "name": "transfer_money",
+      "args": {
+        "amount": 340,
+        "from_account": "checking",
+        "to_account": "savings"
+      }
+    },
+    {
+      "req": "Schedule a meeting on Tuesday at 3pm.",
+      "ok": true,
+      "why": "ok",
+      "name": "schedule_meeting",
+      "args": {
+        "day": "Tuesday",
+        "time": "3pm"
+      }
+    },
+    {
+      "req": "Send 12 invitations to the birthday event.",
+      "ok": true,
+      "why": "ok",
+      "name": "send_invitations",
+      "args": {
+        "count": 12,
+        "event": "birthday event"
+      }
+    }
+  ]
+}

artifacts/tc_shared8.json

@@ -0,0 +1,158 @@
+{
+  "model": "step-3.7-p15-vblend-shared8-head8",
+  "score": [
+    14,
+    15
+  ],
+  "results": [
+    {
+      "req": "Set a timer for 25 minutes.",
+      "ok": true,
+      "why": "ok",
+      "name": "set_timer",
+      "args": {
+        "minutes": 25
+      }
+    },
+    {
+      "req": "Convert 150 USD to EUR.",
+      "ok": true,
+      "why": "ok",
+      "name": "convert_currency",
+      "args": {
+        "amount": 150,
+        "from": "USD",
+        "to": "EUR"
+      }
+    },
+    {
+      "req": "Book a flight from Denver to Miami for 3 passengers.",
+      "ok": true,
+      "why": "ok",
+      "name": "book_flight",
+      "args": {
+        "from_city": "Denver",
+        "to_city": "Miami",
+        "passengers": 3
+      }
+    },
+    {
+      "req": "Resize the image to 800 by 600.",
+      "ok": true,
+      "why": "ok",
+      "name": "resize_image",
+      "args": {
+        "width": 800,
+        "height": 600
+      }
+    },
+    {
+      "req": "Set the thermostat to 68 degrees.",
+      "ok": true,
+      "why": "ok",
+      "name": "set_thermostat",
+      "args": {
+        "temperature": 68
+      }
+    },
+    {
+      "req": "Add 5 apples to my cart.",
+      "ok": true,
+      "why": "ok",
+      "name": "add_to_cart",
+      "args": {
+        "item": "apples",
+        "quantity": 5
+      }
+    },
+    {
+      "req": "Calculate an 18% tip on an 80 dollar bill.",
+      "ok": true,
+      "why": "ok",
+      "name": "calculate_tip",
+      "args": {
+        "bill": 80,
+        "percent": 18
+      }
+    },
+    {
+      "req": "Play Bohemian Rhapsody by Queen.",
+      "ok": true,
+      "why": "ok",
+      "name": "play_song",
+      "args": {
+        "title": "Bohemian Rhapsody",
+        "artist": "Queen"
+      }
+    },
+    {
+      "req": "What's the weather in Paris?",
+      "ok": true,
+      "why": "ok",
+      "name": "get_weather",
+      "args": {
+        "city": "Paris"
+      }
+    },
+    {
+      "req": "Get the stock price for AAPL.",
+      "ok": true,
+      "why": "ok",
+      "name": "get_stock_price",
+      "args": {
+        "ticker": "AAPL"
+      }
+    },
+    {
+      "req": "Find headphones under 200 dollars.",
+      "ok": true,
+      "why": "ok",
+      "name": "search_products",
+      "args": {
+        "query": "headphones",
+        "max_price": 200
+      }
+    },
+    {
+      "req": "Translate 'good morning' into Spanish.",
+      "ok": false,
+      "why": "target_lang: want 'Spanish' got 'es'",
+      "name": "translate",
+      "args": {
+        "text": "good morning",
+        "target_lang": "es"
+      }
+    },
+    {
+      "req": "Transfer 340 dollars from checking to savings.",
+      "ok": true,
+      "why": "ok",
+      "name": "transfer_money",
+      "args": {
+        "amount": 340,
+        "from_account": "checking",
+        "to_account": "savings"
+      }
+    },
+    {
+      "req": "Schedule a meeting on Tuesday at 3pm.",
+      "ok": true,
+      "why": "ok",
+      "name": "schedule_meeting",
+      "args": {
+        "day": "Tuesday",
+        "time": "3pm"
+      }
+    },
+    {
+      "req": "Send 12 invitations to the birthday event.",
+      "ok": true,
+      "why": "ok",
+      "name": "send_invitations",
+      "args": {
+        "count": 12,
+        "event": "birthday event"
+      }
+    }
+  ]
+}

artifacts/toolcall_eval.py

@@ -0,0 +1,138 @@
+"""Agentic discriminator: does REAM pick the right tool with the RIGHT arguments?
+
+The math/factual accuracy test covered the category where PPL predicted no gain.
+REAM's biggest PPL gains were tool_use (27->15.6) and coding/agentic -- untested
+for capability. This scores tool-call correctness right/wrong: correct function
+name AND correct arguments. Argument extraction is exactly what smoothing cannot
+fake -- a flatter distribution does not produce '800 by 600' or 'USD->EUR'
+correctly. Same scorer for both models.
+
+Usage: python toolcall_eval.py <model-id> <out.json>
+"""
+import json, re, sys, urllib.request
+
+MODEL = sys.argv[1]
+OUT = sys.argv[2] if len(sys.argv) > 2 else None
+
+CATALOG = """Available functions (use EXACTLY these names and argument keys):
+- set_timer(minutes)
+- convert_currency(amount, from, to)
+- book_flight(from_city, to_city, passengers)
+- resize_image(width, height)
+- set_thermostat(temperature)
+- add_to_cart(item, quantity)
+- calculate_tip(bill, percent)
+- play_song(title, artist)
+- get_weather(city)
+- get_stock_price(ticker)
+- search_products(query, max_price)
+- translate(text, target_lang)
+- transfer_money(amount, from_account, to_account)
+- schedule_meeting(day, time)
+- send_invitations(count, event)"""
+
+SYS = (CATALOG + "\n\nCall exactly ONE function. Reply with ONLY a JSON object "
+       '{"name": <function>, "arguments": {<args>}} and nothing else. '
+       "Use the exact names above and the exact argument values implied by the request.")
+
+# (request, expected_name, {arg: expected}). Numeric/string args are the
+# discriminator; free-text args use keyword membership.
+ITEMS = [
+    ("Set a timer for 25 minutes.", "set_timer", {"minutes": "25"}),
+    ("Convert 150 USD to EUR.", "convert_currency", {"amount": "150", "from": "USD", "to": "EUR"}),
+    ("Book a flight from Denver to Miami for 3 passengers.", "book_flight",
+     {"from_city": "Denver", "to_city": "Miami", "passengers": "3"}),
+    ("Resize the image to 800 by 600.", "resize_image", {"width": "800", "height": "600"}),
+    ("Set the thermostat to 68 degrees.", "set_thermostat", {"temperature": "68"}),
+    ("Add 5 apples to my cart.", "add_to_cart", {"item": "apples", "quantity": "5"}),
+    ("Calculate an 18% tip on an 80 dollar bill.", "calculate_tip", {"bill": "80", "percent": "18"}),
+    ("Play Bohemian Rhapsody by Queen.", "play_song", {"title": "Bohemian Rhapsody", "artist": "Queen"}),
+    ("What's the weather in Paris?", "get_weather", {"city": "Paris"}),
+    ("Get the stock price for AAPL.", "get_stock_price", {"ticker": "AAPL"}),
+    ("Find headphones under 200 dollars.", "search_products", {"query": "headphones", "max_price": "200"}),
+    ("Translate 'good morning' into Spanish.", "translate", {"text": "good morning", "target_lang": "Spanish"}),
+    ("Transfer 340 dollars from checking to savings.", "transfer_money",
+     {"amount": "340", "from_account": "checking", "to_account": "savings"}),
+    ("Schedule a meeting on Tuesday at 3pm.", "schedule_meeting", {"day": "Tuesday", "time": "3"}),
+    ("Send 12 invitations to the birthday event.", "send_invitations", {"count": "12", "event": "birthday"}),
+]
+
+
+def ask(prompt, mt=3000):
+    body = json.dumps({"model": MODEL,
+                       "messages": [{"role": "system", "content": SYS},
+                                    {"role": "user", "content": prompt}],
+                       "temperature": 0.0, "top_p": 1.0, "top_k": 0, "min_p": 0.0,
+                       "repetition_penalty": 1.0, "max_tokens": mt, "stream": False}).encode()
+    r = urllib.request.Request("http://localhost:1234/v1/chat/completions", data=body,
+                               headers={"Content-Type": "application/json"})
+    m = json.load(urllib.request.urlopen(r, timeout=600))["choices"][0]["message"]
+    return (m.get("content") or "").strip()
+
+
+def parse_call(text):
+    """Extract {name, arguments} from a JSON object or ```json block."""
+    t = re.sub(r"```(?:json)?", "", text)
+    # find the outermost {...} that has a "name"
+    for m in re.finditer(r"\{.*\}", t, re.S):
+        try:
+            o = json.loads(m.group(0))
+            if isinstance(o, dict) and "name" in o:
+                return o.get("name"), (o.get("arguments") or o.get("args") or {})
+        except Exception:
+            continue
+    return None, {}
+
+
+def norm(v):
+    return re.sub(r"(?<=\d)[ ,_](?=\d)", "", str(v)).strip().lower()
+
+
+def check(name, args, exp_name, exp_args):
+    if norm(name) != norm(exp_name):
+        return False, "name"
+    for k, want in exp_args.items():
+        got = args.get(k)
+        if got is None:
+            # allow the value to appear under any key (arg-naming can differ)
+            if any(norm(want) == norm(val) or norm(want) in norm(val)
+                   for val in args.values()):
+                continue
+            return False, f"missing {k}={want}"
+        if norm(want) == norm(got) or norm(want) in norm(got):
+            continue
+        return False, f"{k}: want {want!r} got {got!r}"
+    return True, "ok"
+
+
+def self_test():
+    n, a = parse_call('```json\n{"name": "set_timer", "arguments": {"minutes": 25}}\n```')
+    assert n == "set_timer" and str(a["minutes"]) == "25"
+    ok, _ = check(n, a, "set_timer", {"minutes": "25"}); assert ok
+    ok, why = check("set_timer", {"minutes": "30"}, "set_timer", {"minutes": "25"})
+    assert not ok, why
+    ok, _ = check("resize_image", {"w": "800", "h": "600"}, "resize_image",
+                  {"width": "800", "height": "600"})
+    assert ok, "value-under-any-key fallback"
+    print("[selftest] tool-call parser ok", flush=True)
+
+
+if __name__ == "__main__":
+    self_test()
+    if MODEL == "SELFTEST":
+        sys.exit(0)
+    ok_n = 0; results = []
+    for req, en, ea in ITEMS:
+        try:
+            raw = ask(req)
+        except Exception as e:
+            print(f"  ERROR {e}", flush=True); continue
+        n, a = parse_call(raw)
+        ok, why = check(n, a, en, ea)
+        ok_n += ok
+        results.append({"req": req, "ok": ok, "why": why, "name": n, "args": a})
+        print(f"  [{'OK ' if ok else 'XX '}] want {en}({ea}) -> got {n}({a})  {'' if ok else why}", flush=True)
+    print(f"\n{MODEL}  TOOL-CALL {ok_n}/{len(ITEMS)}  ({100*ok_n/len(ITEMS):.0f}%)", flush=True)
+    if OUT:
+        json.dump({"model": MODEL, "score": [ok_n, len(ITEMS)], "results": results},
+                  open(OUT, "w"), indent=2)

docs/HEAD8-RESULT.md

@@ -0,0 +1,113 @@
+# head8: the always-on weight class the sweep never tested
+
+**Date:** 2026-07-24. `lm_head` and `embed_tokens` at 8-bit instead of 4-bit,
+on top of the shared8 deploy model. Improves every measured category for
++0.53 GB. Found while investigating a numeric-corruption report
+(`TOKENIZER-INVESTIGATION.md`) — the head was the prime suspect, was cleared,
+and was then noticed to be untested.
+
+## Why it was invisible
+
+`SHARED8-RESULT.md`'s thesis: components that fire on **every token** carry the
+quantization damage, because they get no implicit smoothing from top-k partial
+activation. `lm_head`/`embed_tokens` are the extreme case — every token, and
+every one of the 128,896 vocabulary rows participates in every argmax.
+
+But the build predicate returns bare `True` for them, so they inherited the
+4-bit default, and `tomography_sweep.py`'s variants were six layer windows plus
+shared8. The head was never a variant. It is the same blind spot twice over:
+REAP saliency cannot see the shared expert, and the tomography sweep could not
+see the head.
+
+## Method
+
+Rather than rebuild 93 GB (the disk had no room, and a fresh build could differ
+elsewhere), `scripts/build_head8_inplace.py` takes the deployed shared8
+checkpoint, **hardlinks the 18 untouched shards**, and rewrites only the two
+holding the head tensors — 5.7 GB of new bytes, ~2 minutes. Shared inodes prove
+nothing outside those two tensors changed.
+
+The 8-bit weights are quantized from the **BF16 base**, not from the student's
+existing 4-bit head. This matters — see the failure below.
+
+## Result (ΔNLL vs shared8, 500 held-out prompts, negative = better)
+
+| category | BF16 | shared8 | **+head8** | ΔNLL |
+|---|---|---|---|---|
+| agentic | 6.796 | 6.921 | **6.896** | −0.0037 |
+| coding | 8.016 | 6.488 | **6.363** | −0.0194 |
+| general_instruction | 6.184 | 5.115 | **5.082** | −0.0064 |
+| reasoning_math | 2.472 | 2.485 | **2.482** | −0.0012 |
+| tool_use | 43.855 | 27.177 | **26.544** | −0.0236 |
+| **OVERALL** | 6.650 | 5.930 | **5.880** | **−0.0086** |
+
+Every category improves, no regressions — the same clean sweep shared8 produced,
+at half the cost. Magnitude is ~9% of shared8's −0.100, proportionate given
+shared8 touched 177 modules and this touches one tensor pair. Largest gains are
+`tool_use` and `coding`, the two most token-precision-sensitive categories,
+mirroring shared8's pattern.
+
+**Cost:** +0.53 GB (1.06B params, 4.5 → 8.5 bpw). 93 → 93.5 GB, ~4.68 bpw.
+
+## The failed first attempt (kept — it measures something useful)
+
+The first build dequantized the student's existing **4-bit** head and
+requantized it to 8-bit. Same tensors, same bit-width, same +0.53 GB:
+
+| | OVERALL | vs shared8 |
+|---|---|---|
+| shared8 (4-bit head) | 5.930 | — |
+| head8 **from BF16** | **5.880** | **−0.0086** |
+| head8 **from the 4-bit head** | 6.010 | **+0.013** |
+
+The two runs bracket the baseline in opposite directions, differing *only* in
+source precision. Quantization is one-way: the wider container stored
+already-degraded values and added a second rounding pass. Written up as
+`FINDINGS.md` §8a, because it generalizes — converting a released quantized
+checkpoint to another format is a quantization of an already-damaged model, not
+a requantization of the original.
+
+It also yields a free invariant: **more bits cannot beat the source**, so a
+precision *increase* that measures worse means the source is wrong, not the
+target. That is what caught this one.
+
+## How low can the head go?
+
+`reap_stream/diag_head_digits.py` measures quantization perturbation on the
+digit rows against the tightest inter-digit argmax margin (0.8463 in BF16):
+
+| bits | gs | ratio | cosine | size |
+|---|---|---|---|---|
+| 8 | 64 | 0.007 | 0.999986 | 1.12 GB |
+| 6 | 64 | 0.029 | 0.999764 | 0.86 GB |
+| 4 | 64 | 0.121 | 0.995886 | 0.59 GB |
+| 3 | 64 | 0.254 | — | 0.51 GB |
+| 2 | 64 | 0.525 | — | 0.36 GB |
+
+The ratio roughly doubles per bit removed; at 2-bit the noise exceeds half the
+distance between adjacent digit rows. BF16 is not worth it — 8-bit is already
+at 0.7% of the margin with cosine 0.999986, and BF16 costs +0.99 GB beyond
+8-bit to remove an error nothing can resolve.
+
+**Policy for the 64GB recipe:** head at 8-bit; never below 4-bit. Now stated
+explicitly in the predicate rather than inherited from the default, so lowering
+the global bit-width cannot drag it down silently.
+
+## Status
+
+- `scripts/build_student_shared8.py` predicate updated — head at 8-bit.
+- **Promoted**: deployed at
+  `~/.lmstudio/models/truemod/Step-3.7-p15-vblend-shared8-head8` (symlinked into
+  `models/`), loaded at 262144 context / parallel 4. Note LM Studio does not
+  inherit per-model context or sampler settings across a checkpoint swap.
+- **Not uploaded to HF** — blocked by the private-repo storage limit. The Hub
+  still carries the 4-bit-head weights.
+- Multimodal NLL (250 held-out images) **not yet re-run** — shared8 was
+  validated on both instruments and this has only text PPL so far.
+
+## Artifacts
+
+`artifacts/ppl-shared8-head8-500.json`,
+`artifacts/ppl-head8-requant-from-4bit-500.json`,
+`artifacts/ppl-p15-vblend-shared8-500.json` (baseline),
+`scripts/build_head8_inplace.py`, `reap_stream/diag_head_digits.py`.

docs/REAM-RESULT.md

@@ -0,0 +1,104 @@
+# REAM: built, measured, rejected — and a clean case of PPL over-crediting
+
+**Date:** 2026-07-25. Merging low-saliency experts (instead of pruning them)
+was built end-to-end and evaluated. **Verdict: rejected as a deploy candidate.**
+Its large perplexity gain turned out to be a smoothing artifact with no
+capability behind it — the most useful thing the exercise produced.
+
+## What was built
+
+`reap_stream/ream.py` (+ `test_ream.py`, 6/6 unit tests) and
+`scripts/build_student_ream.py`. Each pruned expert is merged into its
+most-similar kept expert (router-row cosine), and the kept expert becomes the
+saliency-weighted average of itself plus everything it absorbed. Router matrix
+and bias are merged the same way. Output count is identical to pruning (245/288)
+— REAM changes the kept experts' *values*, not the count.
+
+Built clean on the first end-to-end run: 1806 experts merged (43×42), 4.683
+bpw, 93 GB, coherent and correct generation (17×23=391, exact Fibonacci, exact
+tool-call JSON, correct Rayleigh-scattering prose). The merge code, which had
+only seen synthetic tensors, worked on the real 198B model.
+
+## The trap: perplexity said it was a big win
+
+500 held-out prompts, vs the shared8-head8 deploy model:
+
+| category | shared8-head8 ppl | REAM ppl | ΔNLL |
+|---|---|---|---|
+| tool_use | 26.54 | **15.57** | −0.533 |
+| coding | 6.36 | **4.66** | −0.312 |
+| general_instruction | 5.08 | **4.10** | −0.215 |
+| agentic | 6.90 | **5.95** | −0.148 |
+| reasoning_math | 2.48 | 2.50 | **+0.007** |
+| **OVERALL** | 5.880 | **4.843** | **−0.194** |
+
+−0.194 NLL overall is ~2× the shared8 + head8 gains *combined*, from a weight
+merge, on a model measured to have no exploitable slack. Two tells said "don't
+believe it": the magnitude was implausible, and the gains scaled with category
+entropy while **reasoning_math — the one category with a single right answer —
+did not improve.** That is the textbook signature of smoothing: averaging expert
+weights flattens the output distribution, which perplexity rewards on
+hedge-friendly text without improving the model.
+
+## The arbiter: exact-answer accuracy
+
+Perplexity cannot distinguish a real gain from smoothing, so both models were
+scored right/wrong on 24 exact-answer items (14 multi-step arithmetic + 10
+factual) where smoothing cannot help — same scorer, temperature 0, parallel 1.
+`scripts/accuracy_eval.py`, results in `artifacts/acc_{ream,shared8}.json`.
+
+| model | overall | math | factual | PPL |
+|---|---|---|---|---|
+| **shared8-head8 (prune)** | **24/24** | 14/14 | 10/10 | 5.880 |
+| REAM (merge) | 23/24 | 13/14 | 10/10 | 4.843 |
+
+**The −0.194 PPL advantage bought zero accuracy** — REAM is one item behind
+(noise). Every prediction held: PPL said math wouldn't improve, and math
+accuracy is level; the PPL gains were all in high-entropy categories, exactly
+where a flatter distribution lowers perplexity for free.
+
+## Follow-up: the agentic categories, tested directly (the important one)
+
+The accuracy test above was math + factual -- the category where PPL predicted
+no gain. But REAM's *biggest* PPL gains were tool_use (-0.53) and coding, which
+that test did not cover. So it was re-run on 15 tool-call items scored right/
+wrong: correct function name (from a provided catalog) AND correct extracted
+arguments -- exactly what smoothing cannot fake. `scripts/toolcall_eval.py`,
+`artifacts/tc_{ream,shared8}.json`.
+
+| test | shared8-head8 (prune) | REAM (merge) |
+|---|---|---|
+| tool-call accuracy | **14/15** | **14/15** |
+| tool_use *perplexity* | 26.5 | 15.6 (**-0.53 NLL**) |
+
+**A 41% lower perplexity on tool_use converted to zero tool-call capability --
+a dead tie.** This is the strongest case against PPL in the whole project: its
+largest and most agentic-relevant signal (-0.53 NLL on the exact category that
+matters for agent work) was worth nothing in practice. If PPL were trustworthy
+anywhere it would be where its signal is biggest; that is precisely where it
+most oversold.
+
+## Conclusions
+
+1. **Pruning stays.** REAM is not a deploy candidate. On this prune-resistant,
+   flat-saliency model, merging the least-salient experts produces a mushier
+   model that scores better on perplexity and no better on task accuracy -- including tool-call/agentic tasks, tested directly.
+2. **This is the session's central lesson, demonstrated.** PPL moved −0.194
+   while real capability was flat-to-worse. Trusting the perplexity number would
+   have shipped a worse model as an upgrade. Perplexity is a proxy; on any
+   quality change, confirm with a non-PPL arbiter before promoting.
+3. **The co-occurrence gate would have predicted this** (docs/FINDINGS.md §10):
+   flat saliency + prune-resistance implies merge partners are near-orthogonal,
+   so blending them averages away specialization. We skipped that gate to
+   measure directly; the direct measurement agrees.
+
+## Kept
+
+Build + eval code retained — REAM is a correct, working method, just not a win
+*here*. It could still help a genuinely redundant MoE (one with a real
+low-saliency tail), and `assign_merges` accepts a co-occurrence matrix for a
+principled partner choice when that data exists. Artifacts:
+`artifacts/ppl-ream-shared8-head8-500.json`, `artifacts/acc_ream.json`,
+`artifacts/acc_shared8.json`, `scripts/accuracy_eval.py`, `reap_stream/ream.py`.
+The 93 GB REAM checkpoint can be deleted (rebuildable from
+`scripts/build_student_ream.py`).

docs/TOKENIZER-INVESTIGATION.md

@@ -0,0 +1,127 @@
+# Tokenizer investigation: no defect found; digit fragmentation is by design
+
+**Date:** 2026-07-24. Chasing numeric corruption in LM Studio led through a
+wrong diagnosis and a reverted "fix". Conclusion: **there is no tokenizer bug.**
+The build script's tokenizer is correct, single-digit number tokenization is
+Step-3.7's intended behaviour, and the numeric fragility it causes is a property
+of the model rather than something to repair in the checkpoint.
+
+Recorded because the wrong version of this was believed for several hours and
+acted on, including edits to four build scripts and two pushes to the Hub.
+
+## The claim that was wrong
+
+`processor.save_pretrained(out_dir)` writes a different `tokenizer.json` than
+the base checkpoint ships — 2 pretokenizer stages instead of 4, dropping a
+`\p{N}{1,3}` digit-grouping rule. That looked like the build silently corrupting
+the tokenizer, and it fit the symptom: numbers tokenised one digit per token,
+so a number arrives as loose fragments with nothing binding it into a unit.
+
+Restoring the base file even *fixed the numeric cases*:
+
+| case | built student | base tokenizer |
+|---|---|---|
+| `kill process ID 18452` | 0/12 | **6/6** |
+| `sed -n '447,495p'` | corrupted to `'4.4.7,4.9.5p'` | **6/6** |
+
+## Why it was wrong
+
+The base `tokenizer.json` **cannot be used standalone.** Read directly by the
+`tokenizers` library — which is what LM Studio's MLX backend does — it is not
+merely imperfect, it is broken:
+
+```
+                     built student (deployed)      base tokenizer.json
+'hello world'   ->   ['hello','Ġworld']      OK    ['hell','ow','orld']   'helloworld'
+'id is 5'       ->   ['id','Ġis','Ġ','5']    OK    ['idis','5']           'idis5'
+'你好世界'        ->   ['ä½łå¥½','ä¸ĸçķĮ']       OK    []                     ''
+'中文 with english' ->  OK                          ['wit','heng','lish']  'withenglish'
+```
+
+**Chinese produces zero tokens**, and English word boundaries land in the wrong
+places. This is why `tokenizer_config.json` carries `fix_mistral_regex: true`:
+`transformers` *reconstructs* a working tokenizer at load and never reads the
+shipped file. `save_pretrained` then serialises that working reconstruction into
+a standalone file — which is **correct behaviour, not corruption.** The build
+script was fixing an unusable upstream artifact, not breaking a good one.
+
+Deploying the base tokenizer broke generation outright: output came back as
+`Thefindcommandfailedbecauseofspacesinthepath.`, shell commands failed on
+unquoted paths, and the agent stalled. Reverted locally and on the Hub.
+
+## Consequence
+
+Single-digit number tokenization is **genuine and intended**. Step-3.7 emits one
+token per digit (ids 19–28) and a standalone space token (223), so numbers carry
+no token-level redundancy: one mis-sampled or mis-copied token corrupts an entire
+number, while prose absorbs it. That is the real explanation for
+"numbers are wrong, everything else is perfect", and it is not fixable in the
+checkpoint.
+
+The model's own chain of thought describing input as *"process ID 1 8 4 5"* is
+therefore an accurate report of what it receives, not a hallucination.
+
+## How the mistake happened
+
+Three failures worth remembering:
+
+1. **Evidence collected and dismissed.** The space-dropping round-trip
+   (`'The order id is 2456.'` → `'Theorderidis2456.'`) was measured early and
+   written off as a `transformers`-path quirk.
+2. **Verification that could not fail.** The "fix" was checked by decoding
+   tokens *individually* (`['sed','-n','447',',','495','p']`), which
+   structurally cannot reveal a missing space.
+3. **Only testing English.** One Chinese string would have exposed it
+   immediately — `你好世界` → `[]`. Step-3.7 is a Chinese-English model; that
+   should have been in the first test, not the last.
+
+**Rule:** validate any tokenizer change with a full-string round-trip through
+`mlx_lm`'s streaming detokenizer, in **both** languages.
+
+## Guardrails: removed
+
+An earlier version of this document instructed all four builders to copy the
+base tokenizer over `save_pretrained`'s output. Those calls have been removed
+from `build_student{,_shared8,_tiered}.py` and `tomography_sweep.py`, and each
+now carries a comment explaining why the base file must not be copied.
+
+## Ruled out along the way
+
+Useful negatives, each measured:
+
+- **Quantized output head.** Digit rows carry 1.033x the all-row reconstruction
+  error; perturbation is 12% of the tightest inter-digit argmax margin.
+  `reap_stream/diag_head_digits.py`.
+- **REAP / degraded digit copying.** Teacher-forced probes, no sampler in path,
+  rank the correct next digit **first** at p = 0.91–0.9996.
+  `reap_stream/diag_digit_logits.py`.
+- **`repetition_penalty`.** A/B at 1.02 vs 1.0: 0/8 and 1/8 vs 0/8 and 0/8.
+  (LM Studio's per-model UI sampler settings silently override
+  `generation_config.json` — pin sampler fields explicitly when testing.)
+- **`reasoning_effort`.** `Reasoning: medium` fixed one case (0/12 → 12/12) and
+  made another much worse. Not a mitigation.
+- **SFT / QLoRA.** Rejected: the capability is intact and no existing instrument
+  could verify success or catch regression.
+- **PPL/NLL confounding.** Claimed at one point, false: `transformers` rebuilds
+  the tokenizer at load, so all evaluations used identical tokenisation
+  (identical `n_tokens: 383130`; 0-token diff over 40 held-out rows).
+
+## Mitigation
+
+Since the fragility is intrinsic, the mitigation is operational:
+
+- `scripts/numeric_guard.py` — signature checks, provenance, sed bounds, and an
+  AST check for floats in integer-only positions (valid Python that fails only
+  at runtime).
+- `~/.pi/agent/extensions/numeric-guard.ts` — blocks corrupted tool calls with
+  an actionable reason. Errors are surfaced and the call allowed through, so the
+  guard can never itself stall a turn. Retrying does not help — output is
+  identical every time; telling the model *what* was wrong does.
+
+## Still open
+
+- **The echo case.** `2456, 1337, 495` → `2 4 5 6, 1 3 3 7, 4 9 5`, 0/12, never
+  resolved.
+- **Whether upstream Step-3.7 behaves the same.** No unpruned Step-3.7 exists
+  locally as a loadable quant; the Hub's 148B is a *different* REAP (212/288
+  experts, nvfp4), so it is not a control.

scripts/numeric_guard.py

@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""Catch Step-3.7's numeric corruption before a tool call executes.
+
+Meant to be copied into an agent harness as a pre-tool-use check, not run as
+part of the REAP pipeline. See the HF README's "numbers corrupted inside the
+reasoning block" section for the measured behaviour this defends against.
+
+Why code and not the model: the failure survives self-review. In one traced
+run the model re-read the question 20+ times and reaffirmed the wrong value
+each time, because by then its own output was the strongest evidence in
+context. A checker that does not share that context does not share the bias.
+
+Three checks, cheapest first:
+
+  scan_signature(text)      known corruption shapes -- 4.4.7, "1 8 4 5",
+                            2,4,5,6,1,3,3,7. No source needed.
+  check_provenance(cmd,src) every numeral in cmd must appear in src. Catches
+                            invented numbers that happen to look well-formed.
+  check_sed_bounds(cmd)     sed line ranges against the real file length.
+
+Usage:
+    from numeric_guard import guard
+    problems = guard(proposed_command, source_text=file_contents)
+    if problems: ...retry instead of executing...
+
+    $ python3 scripts/numeric_guard.py "sed -n '4.4.7,4.9.5p' src/aero.py"
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+# 1.2.0 / 4.4.7 -- a numeral carrying more interior dots than a decimal can.
+# Version-like strings are legitimate in many contexts, so this is reported
+# rather than treated as certainly wrong; see guard()'s `strict` flag.
+MULTI_DOT = re.compile(r"(?<![\w.])\d+\.\d+\.\d+(?![\w.])")
+
+# "1 8 4 5" / "2 4 5 6" -- three or more single digits separated by spaces.
+SPACED_DIGITS = re.compile(r"(?<!\d)\d(?: \d){2,}(?!\d)")
+
+# "2,4,5,6,1,3,3,7" -- single digits comma-separated. Distinguished from a
+# real list like "2456, 1337" by every element being exactly one digit.
+COMMA_DIGITS = re.compile(r"(?<!\d)\d(?:,\d){3,}(?!\d)")
+
+NUMERALS = re.compile(r"\d+")
+SED_RANGE = re.compile(r"sed\s+-n\s+['\"]?(\d+),(\d+)p['\"]?\s+(\S+)")
+
+# 5_1_8_4_0_0 -- underscore inserted between single digits. Python's numeric
+# separators make `_` a legitimate delimiter, which primes exactly this. Note
+# `5_1_8_4_0_0.0 == 518400.0` is True, so in Python this is cosmetic rather
+# than wrong -- but it is a hard error in JSON, YAML and shell.
+# Trailing `.` must be allowed -- these appear as float literals (5_1_8_4_0_0.0).
+UNDERSCORE_NUM = re.compile(r"(?<![\w.])\d[\d_]*_[\d_]*\d(?!\w)")
+
+
+def _bad_underscore_grouping(lit: str) -> bool:
+    """True unless the literal uses conventional 3-digit grouping.
+
+    Legitimate: 5_000_000, 1_234, 12_345_678 -- every group after the first is
+    exactly 3 digits and the first is 1-3. Anything else (5_1_8_4, 1_00_000)
+    is the corruption signature.
+    """
+    groups = lit.split("_")
+    if any(g == "" for g in groups):
+        return True
+    return not (1 <= len(groups[0]) <= 3 and all(len(g) == 3 for g in groups[1:]))
+
+
+def scan_signature(text: str, strict: bool = False) -> list[str]:
+    """Known corruption shapes. No source text required."""
+    out = []
+    for m in SPACED_DIGITS.finditer(text):
+        out.append(f"digits split by spaces: {m.group(0)!r}")
+    for m in COMMA_DIGITS.finditer(text):
+        out.append(f"digits split by commas: {m.group(0)!r}")
+    for m in UNDERSCORE_NUM.finditer(text):
+        if _bad_underscore_grouping(m.group(0)):
+            out.append(f"digits split by underscores: {m.group(0)!r} "
+                       f"(valid Python, but wrong in JSON/YAML/shell)")
+    for m in MULTI_DOT.finditer(text):
+        label = "malformed number" if strict else "version-like numeral (check)"
+        out.append(f"{label}: {m.group(0)!r}")
+    return out
+
+
+def check_provenance(command: str, source: str, min_len: int = 3) -> list[str]:
+    """Every numeral of >=min_len digits in `command` must occur in `source`.
+
+    min_len avoids flagging small incidental numbers (-9, exit codes, 0/1).
+    Numbers the model legitimately *computed* will also trip this, so treat
+    hits as "confirm before running", not as proof of corruption.
+    """
+    return [f"numeral {n!r} does not appear in the source"
+            for n in {m.group(0) for m in NUMERALS.finditer(command)}
+            if len(n) >= min_len and n not in source]
+
+
+def check_sed_bounds(command: str, root: str | Path = ".") -> list[str]:
+    """sed line ranges against the file's real length."""
+    out = []
+    for start, end, path in SED_RANGE.findall(command):
+        p = Path(root) / path
+        if not p.exists():
+            continue
+        n = sum(1 for _ in p.open(errors="replace"))
+        s, e = int(start), int(end)
+        if s > e:
+            out.append(f"sed range {s},{e} is inverted")
+        if s > n or e > n:
+            out.append(f"sed range {s},{e} exceeds {path} ({n} lines)")
+    return out
+
+
+def check_python_int_positions(code: str) -> list[str]:
+    """Float literals where Python requires an int: slice indices, range().
+
+    This is the corruption's most dangerous form, because one inserted '.'
+    yields a *valid* float that no shape-based check can distinguish from a
+    legitimate one. `content[idx-5:idx+1.5]` parses fine and fails only at
+    runtime with "slice indices must be integers" -- the exact loop seen in
+    the wild. Catching it needs the syntactic position, not the literal.
+    """
+    import ast
+    try:
+        tree = ast.parse(code)
+    except SyntaxError:
+        return []
+
+    out = []
+
+    def floats_in(node):
+        return [n for n in ast.walk(node)
+                if isinstance(n, ast.Constant) and isinstance(n.value, float)]
+
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Subscript):
+            for f in floats_in(node.slice):
+                out.append(f"float {f.value!r} used as a slice index "
+                           f"(line {f.lineno}) -- fails at runtime")
+        elif isinstance(node, ast.Call) and getattr(node.func, "id", "") == "range":
+            for arg in node.args:
+                for f in floats_in(arg):
+                    out.append(f"float {f.value!r} passed to range() "
+                               f"(line {f.lineno}) -- fails at runtime")
+    return out
+
+
+def guard(command: str, source_text: str | None = None,
+          root: str | Path = ".", strict: bool = False,
+          as_python: bool = False) -> list[str]:
+    """All applicable checks. Empty list means nothing suspicious."""
+    problems = scan_signature(command, strict=strict)
+    if source_text is not None:
+        problems += check_provenance(command, source_text)
+    problems += check_sed_bounds(command, root)
+    if as_python:
+        problems += check_python_int_positions(command)
+    return problems
+
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print(__doc__.strip().split("Usage:")[-1].strip())
+        raise SystemExit(2)
+    cmd = sys.argv[1]
+    src = Path(sys.argv[2]).read_text() if len(sys.argv) > 2 else None
+    found = guard(cmd, source_text=src)
+    if not found:
+        print("ok")
+    else:
+        print(f"SUSPECT: {cmd!r}")
+        for p in found:
+            print(f"  - {p}")
+        raise SystemExit(1)