Clarify metric: verifier@96 self-consistency (single answer), not pass@96

README.md

@@ -20,9 +20,10 @@ grade-school math word problems **without any LLM at inference time**.
 
 Instead of generating text, SPROG abstracts the numbers in a question to slots
 (`[N0]`, `[N1]`, …) and predicts a **postfix program** over them, which is then executed
-symbolically. It draws 96 temperature samples and selects the answer with a **free
-symbolic verifier** (0 trainable parameters). The whole thing runs on a CPU/Apple-Silicon
-GPU via MLX.
+symbolically. It draws 96 temperature samples and selects **a single answer** with a **free
+symbolic verifier** (0 trainable parameters) that never sees the ground truth — i.e.
+self-consistency selection, **not** a pass@k oracle. The whole thing runs on a
+CPU/Apple-Silicon GPU via MLX.
 
 Trained on [`codelion/gsm8k-synth`](https://huggingface.co/datasets/codelion/gsm8k-synth).
 
@@ -50,17 +51,29 @@ print(solve(model, stoi, "Tom has 15 apples. He buys 27 more, then gives away 12
 
 ## Results
 
-Evaluated on the **full GSM8K test set** (1,319 problems), 3 training seeds:
-
-| metric | GSM8K test |
-|---|---|
-| **accuracy (symbolic verifier)** | **11.8%** mean — 12.6% best seed |
-| accuracy (plurality vote, no verifier) | ≈9.3% |
-| trainable parameters | **9.37M** |
-| LLM used at inference | **none** |
-
-The free symbolic verifier adds **+2.5 points** over plain majority voting. The model is
-stable across seeds on the test set (range 11.1–12.6%).
+Evaluated on the **full GSM8K test set** (1,319 problems), averaged over 3 training seeds.
+
+**The model commits to one answer per question.** It draws 96 temperature samples, then a
+0-parameter symbolic verifier picks a **single** answer **without ever seeing the gold
+answer**. This is a single-answer accuracy (the self-consistency / maj@k family) — **not a
+pass@k oracle**.
+
+| metric | GSM8K test | what it measures |
+|---|---|---|
+| **verifier @ 96** (headline) | **11.8%** (best seed 12.6%) | verifier commits to one answer; gold never used |
+| plurality @ 96 | ≈9.3% | most-voted answer; gold never used |
+| pass@96 (oracle) | ≈39% | gold is *somewhere* in the 96 samples — an upper bound that **uses the gold to check** |
+| trainable parameters | 9.37M | — |
+| LLM used at inference | none | — |
+
+So **11.8% is a committed single answer, not pass@96** (that would be the ≈39% oracle). The
+free symbolic verifier adds ≈+2.5 points over majority voting, at 96× the inference cost of a
+single decode. Results are stable across seeds (range 11.1–12.6%).
+
+**Why 96 samples?** Recall rises with sample count (≈39% gold-in-pool at 96 → ≈50% at 288),
+but the verifier's *conversion* peaks around 64–96 then declines — extra samples add
+plausible-but-wrong distractors that hurt selection (measured: 192 → 8.5%, 288 → 8.3%). 96 is
+the sweet spot between recall and selectability.
 
 ## How it was built