Buckets:

Mercity
/

Desicion-Model-Custom

Files

xet

Mercity/Desicion-Model-Custom / code /FINAL_REPORT.md

Pranav2748

7 days ago

preview code

download

raw

14.1 kB

# Reasoning decision model on Qwen3.5-4B: final report

Oct 30 - Nov 1, 2026 · one A100 80GB · code, data, logs and checkpoints in the public bucket `Mercity/Desicion-Model-Custom`
(no keys) · notebook `results/NOTEBOOK.md`, every change to the plan in `results/DEVIATIONS.md` (31 entries) · dashboard
https://claude.ai/artifact/XehYsr1iD4hq2zA3ni241r

**V2 + RL is the model to use.** Its fast readout answers 72.5% of our 5,442 eval questions (base Qwen3.5-4B: 62.7%);
routing the least certain 30% to thinking mode lifts that to 81.8% at 729 thinking tokens per question. On the external
S1MB benchmark, training took the model's Task Avg from 32.8 to 47.9. Report doc: https://claude.ai/artifact/RUdY6E2PZgStqNURLGT8g8

## What was built

A Jev-style typed decision model: given evidence, a criterion and options, it answers in two ways.

- **Fast readout (System One):** one forward pass, a probability per option from the letter logits, ~40 ms.
- **Thinking mode:** reasons, then the same letter readout after the chain.
- **Routing:** the readout answers by default; the lowest-margin share of questions goes to thinking.

Base model Qwen3.5-4B (hybrid Gated DeltaNet + attention), LoRA r32 / α64 on all linear layers. Three model generations
were trained, plus a ~2B model:

| Model | What changed |
|---|---|
| v1 | readout on 3K questions (gold mixed 50/50 with the model's own vote shares) + 2.7K of its own correct chains |
| V2 | 14x the readout data (42K questions, every pool question, half in the eval's option order), 11K chains (4B + 662 questions from Qwen3.5-35B-A3B via API, $6.60), stricter chain filters, lr 2e-4 from a 3-rate sweep, 1.5x faster training setup |
| V2 + RL | GRPO on V2's thinking mode: 60 steps (40 at 1e-5, 20 at 2.5e-5), 8 chains per question, +1 / -1 reward (annotator share for soft labels), -1.5 for no answer by 8,192 tokens, 240 questions (60% never trained on, mostly reasoning-heavy tasksource sources) |
| 2B | MiniCPM5-2B picked over Qwen3.5-2B by a pre-registered probe rule; V2 recipe, and a readout-only variant |

## Headline results (our eval: 5,442 questions; thinking on a 1,498-question subset)

| Model | Fast readout | Thinking | Looped chains | Routed 30% to thinking | Thinking tokens per question at 30% |
|---|---|---|---|---|---|
| Base Qwen3.5-4B | 62.7% | 84.0% | 0.9% | 73.0% | 1,069 |
| v1 | 69.5% | 84.4% | 1.4% | 78.0% | 1,166 |
| V2 | **72.7%** | 81.2% | 5.1% | 81.8% | 1,749 |
| **V2 + RL (step 60)** | 72.5% | 83.1% | **0.2%** | **81.8%** | **729** |
| V2 + RL (step 40, 1e-5 only) | 72.7% | **83.6%** | 2.0% | 82.0% | 1,079 |

(Routing columns use the 1,498-question subset: fast readout there is 61.8 / 70.3 / 76.0 / 75.9 / 76.0%.)

- **The fast readout gained 10 points over the base model** (62.7 -> 72.5%), most of it from data: V2's 42K questions
  took it from v1's 69.5% to 72.7% and its held-out dev score from 65% (1,500 questions) to 81.3%.
- **V2's chain training brought loops into thinking mode** (5.1% of chains ran to the 32K cap with no answer), costing
  3.2 points; **RL removed them** (0.2%) and recovered most of the loss, at half the thinking tokens.
- **The routed system is the deployable unit:** V2 + RL reaches 81.8% at 30% routed with 729 thinking tokens per
  question, vs v1's 78.0% at 1,166 and the base model's 73.0%. It recovers 79% of the gap between fast and thinking mode
  (the plan's target was 80%; v1 recovered 52%).
- Calibration stays good: fast-readout ECE 0.030 (15 bins, after per-type temperatures fitted on calibration items);
  after-chain readout ECE 0.057 for V2 + RL vs 0.043 for V2, a difference within noise (+0.014, 95% CI -0.006 to +0.028).

By eval set (fast readout): procedural 46.8 -> 89.5, tasksource 60.4 -> 72.3, OpenDecisions 66.5 -> 73.9, LocalLLaMA
63.0 -> 66.3 (base -> V2 + RL). The two sets least like the training data gained least.

## What the ablations and side experiments showed

- **Votes vs gold labels (V2 ablations, the same 12K questions):** mixing the model's own vote shares into the readout
  target lowered accuracy (70.5% vs 71.7% gold-only; -1.2 points, 95% CI -2.1 to -0.3) and made calibration worse on the eval sets (ECE 0.047 vs
  0.031), although it looked better on in-distribution dev (0.017 vs 0.034). The votes help on familiar question types
  and hurt on new ones. v1's gold-only readout remains the best calibrated (0.019).
- **RL learning rate:** the 1e-5 phase gave the accuracy (+2.4 points over V2); the 2.5e-5 phase mostly removed the
  remaining loops (2.0% -> 0.2%) and 30% of the tokens. Training-time accuracy at equal difficulty was flat; the clear
  training-time effect was decisiveness ("Wait" per finished chain 10 -> 5).
- **Speed:** keeping activations for micro-batches up to 8K tokens (no recompute) gave 1.48x training throughput at
  equal numerics (user's call over a 72 GB guard; peak 71.5 GiB of 80, no OOM). Liger kernels failed the fp32 equivalence
  check and were not used.
- **Data hygiene:** dev and eval ids are disjoint from training; 61 of 999 dev items and 50 of 526 procedural eval items
  repeat a training question exactly (procedural generators); removing them changes procedural accuracy by <= 0.7 points.

## The 2B model

| 2B variant | Fast readout | Thinking | Looped | Routed 30% | Routed 50% |
|---|---|---|---|---|---|
| MiniCPM5-2B, untrained | 50.2% | **79.4%** | 0.1% | | |
| V2 recipe (with chains) | 65.1% | 71.7% | 7.4% | 71.9% | 72.0% |
| Readout-only (no chains) | **66.0%** | 70.8% | 2.1% | 71.0% | 71.0% |
| **Readout-only fast + untrained thinking** | 66.0% | 79.4% | 0.1% | **75.6%** (253 tok) | **78.0%** (411 tok) |

(Routing in this table: lowest readout margin to thinking, the thinking answer is the chain's parsed letter; the same
method gives 81.9% / 742 tokens for V2 + RL at 30%.)

- MiniCPM5-2B thinks almost as well as our 4B untrained (79.4% vs 83-84%, at ~40% of the tokens), so it was picked.
- **Every fine-tune damaged its thinking** (-8 to -9 points): the V2 recipe by copying the 4B's and the teacher's longer,
  loop-prone chains; readout-only training by teaching it to answer at once (median chain 441 -> 127 tokens).
- **The working 2B is two states of one model:** the trained LoRA for the fast readout, the adapter off for thinking.
  That reaches 75.6% at 30% routed (253 thinking tokens per question) and 78.0% at 50%, within 4-6 points of the 4B.

## External benchmarks

### Jevals (jevals.com, suite 0.1.0, 3 boards x 300; Decision Score = Brier / RPS skill vs the label prior)

| Board | Gemini 3.8 Flash | Jev | Ours, V2 + RL fast | Ours, V2 + RL thinking | Ours, 2B readout-only fast |
|---|---|---|---|---|---|
| PubMedQA (yes/no, unseen in training) | 73.0 | 69.0 | 58.6 (5th of 8) | **63.5** (3rd) | 44.6 |
| Banking77 (77 intents) | 74.1 | 67.8 | 61.3 (6th) | 59.3 | 53.9 |
| HelpSteer2 (5-level score) | 4.6 | 9.2 | 2.4 | -20.1 | -0.4 |

Fast mode sits mid-pack, level with mid-tier LLMs and 6-10 points behind Jev; its probabilities are among the best
calibrated on the board (Banking77 ECE 2.8 points vs Jev 9.8). Thinking raises accuracy where the task is reading and
reasoning (PubMedQA 86 -> 90%) but its post-chain probabilities are overconfident, which Brier punishes. Training lifted
the base 4B's PubMedQA score from 48.1 (raw; 36.4 with our temperatures) to 58.6 on a dataset it never saw.

### S1MB (System One Mosaic Benchmark, English v1: 137 benchmarks, 26,269 judgments; 211 complete published models)

Training took the 4B's Task Avg from 32.8 to 47.9 and the 2B's from 18.1 to 37.3; 74 of 211 published models rank
above V2 + RL in the leaderboard's Borda order. All runs use the fast readout.

| Model | Task Avg | Yes/no | Choice | Score | Published models above (Borda) |
|---|---|---|---|---|---|
| **V2 + RL (4B)** | **47.9** | 50.7 | 57.2 | 35.8 | 74 |
| v1 (4B) | 43.5 | 50.1 | 52.3 | 28.0 | 100 |
| 2B readout-only | 37.3 | 42.3 | 45.7 | 24.0 | 123 |
| Qwen3.5-4B, untrained | 32.8 | 33.8 | 45.7 | 18.9 | 134 |
| MiniCPM5-2B, untrained | 18.1 | 22.9 | 29.5 | 2.0 | 162 |
| superion_s1 (best Task Avg) | 74.4 | | | | |
| KnowLine-4B-Gen5 (best published 4B) | 67.0 | 65.7 | 76.5 | 58.8 | 19 |
| KnowLine-2B-Gen1 (best published 2B) | 44.8 | 48.4 | 57.4 | 28.7 | 88 |
| Median published model | 41.2 | | | | |
- **Most of the gain came with v1** (+10.7 of +15.1). V2's data and RL then added choice (+4.9) and score (+7.8) skill
  but almost no yes/no skill (50.1 -> 50.7). Benchmarks: 105 of 137 up, 16 down (by more than 0.01).
- **Training fixed a "no" bias and overshot on balanced sets.** The untrained 4B said yes on 18% of yes/no judgments
  where the target is yes 28% of the time; V2 + RL says yes 32% (true recall 44% -> 70%, false recall 91% -> 83%). It
  now over-says yes on SciTail, Qasper, CanTTalk and Open-Jev customer-control, 4 of the 5 benchmarks that fell by more
  than 0.1.
- **The largest gains are Open-Jev control and workflow tasks:** drone control +1.0, entity alignment +0.83, invoice
  processing +0.65, VizDoom +0.64 (adjusted skill, 0-1).
- **The 2B gained more than the 4B:** readout-only training added 19.2 points; among the ~2B published models only
  KnowLine-2B-Gen1 (44.8) and unee-2b (44.4) score higher.
- **Method.** `s1mb_eval.py` reimplements the published SCORING.md (running their evaluator was refused by the
  permission check as third-party code) and is checked on 3 published models: 48 benchmark scores match to 6e-16.
  Per-task temperatures fitted on our own calibration items (never S1MB) change only the score task: V2 + RL is 46.9
  without them. Prompts are our native format; the 904 judgments with more than 26 options use a tournament readout.
  V2 and the V2-recipe 2B were not run (V2's fast readout equals V2 + RL's; the V2-recipe 2B is dominated by the
  readout-only 2B).

### S1MB with routing (V2 + RL, least certain 30% of judgments to thinking)

| Task | Fast readout | 30% routed | Difference [95% CI] |
|---|---|---|---|
| Yes/no | 50.7 | 58.3 | +7.6 [+4.5, +11.6] |
| Choice | 57.2 | 67.6 | +10.4 [+6.2, +14.9] |
| Score | 35.8 | 39.8 | +4.0 [-1.1, +9.3] |
| **Task Avg** | **47.9** | **55.2** | **+7.3 [+4.9, +9.9]** |

Published models above it in Borda order: 74 → 50 (Task Avg level with KnowLine-4B-Gen2, 55.3). 92 benchmarks up, 20 down:
thinking wins on computation and multi-step reasoning (Snake +0.86, logical entailment +0.62, temporal NLI +0.53,
tic-tac-toe +0.51, AQuA-RAT +0.45) and loses on judgment and graded relevance (ETHICS -0.33, search-relevance score tasks
-0.09 to -0.13). 943 thinking tokens per judgment; 158 of 7,881 chains (2.0%) gave no answer and kept the fast answer.
Not a like-for-like leaderboard entry: S1MB measures single-pass judgments. The 2B hybrid's routed run was skipped.

## Robustness and error analysis (added Nov 1)

- **Bootstrap intervals** (`results/final_stats.md`): readout gains over the base (+9.8 [8.4, 11.2]) and v1 (+3.0 [1.9, 4.2])
  hold; RL's thinking gain over V2 holds (+1.9 [0.3, 3.5]); routed V2 + RL ties V2 (+0.1 [-1.3, 1.1]) at 1,020 fewer
  thinking tokens per question. Corrected: vote-mixed targets *lowered* readout accuracy (-1.2 [-2.1, -0.3]); the
  post-chain ECE rise after RL is within noise. Hypotheses: H2 confirmed, H3 met by V2 + RL only, H1 not confirmed,
  H4 undecided (79.5% [64%, 97%]).
- **S1MB overlap check:** gains are larger on benchmarks from datasets never trained on (+17.4 vs +8.4, difference +9.0
  [3.4, 14.5]); only 104 of ~18,700 cases share most of their text with training data.
- **Chains read by hand** (`results/examples_analysis.md`): thinking fixes by computing (OpenDecisions 77.0 → 95.8%) and
  breaks even on judgment sets, where margin routing spends most of its budget; routing by task type could add up to
  2.3 points. Chains that guess at dataset conventions are right ~55% vs ~90%. Only 49% of RL questions gave a learning
  signal (the fast readout's p(gold) did not predict thinking-mode difficulty).
- **Yes/no bias:** V2 + RL says yes on 62% of our yes/no questions (gold 50%); one calibration-fitted offset lifts their
  balanced accuracy 78.4 → 83.6%, but does not help S1MB.

The full write-up (23 pages, 23 figures): `results/report/decision_model_report.pdf` (bucket: `report/`).

## What I would do next

1. **Fix the chain recipe before more data:** filter re-verification loops ("Wait, I'll consider ...") out of training
   chains, and only train chains a model cannot already produce better itself (MiniCPM's own were better than ours).
2. **Calibrate the after-thinking readout** (soft post-chain targets or a calibration term in the RL reward): it is
   what costs thinking mode on Brier-scored benchmarks.
3. **More RL, set up better:** questions screened to split 30-70%, 12-16 chains per question, a 6K-token cap, no spatial
   puzzles, and the speed work first (batched learning steps, the fast conv kernel) to afford hundreds of steps.
4. **Ship the 2B as fast-readout LoRA + untouched thinking**, and test RL on its readout-only variant to recover
   thinking without losing the readout.
5. **Out-of-distribution gains are the weak spot** (LocalLLaMA +3 points over the base, OpenDecisions +7): more diverse
   training sources beat more questions from the same sources.

## Where things are

- Bucket: `run1/` (v1), `run2_v2/` (V2, ablations, RL with every sampled chain, merged `v2_joint`, `rl60`), `run3_2b/`
  (the 2B, incl. merged `v2b_ro_merged`), `jevals/`, `s1mb/` (items, our raw readouts, score table), `code/`, `dashboard/`,
  `README.md`, `FINAL_REPORT.md`.
- Deployable models: `run2_v2/checkpoints/rl60_merged` (4B, bf16, text-only Qwen3_5ForCausalLM); the 2B readout LoRA in
  `run3_2b/results/adapters/v2b_ro` on top of `openbmb/MiniCPM5-2B`.
- Known caveats: single seeds; the RL run was short (60 steps); Jevals and S1MB were scored with our own reimplementation
  of their rules (checked against their published numbers), our prompts are our native format, and our training pool
  contains train-split items of a few of their source datasets (Banking77, HelpSteer2; never the evaluated items).

Xet Storage Details

Size:: 14.1 kB
Xet hash:: 2df5f3eb4cf484864decac333c15c8d4711eb8fbffb3acacc6b47113d3d36f64

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Reasoning decision model on Qwen3.5-4B: final report

	Oct 30 - Nov 1, 2026 · one A100 80GB · code, data, logs and checkpoints in the public bucket `Mercity/Desicion-Model-Custom`
	(no keys) · notebook `results/NOTEBOOK.md`, every change to the plan in `results/DEVIATIONS.md` (31 entries) · dashboard
	https://claude.ai/artifact/XehYsr1iD4hq2zA3ni241r

	V2 + RL is the model to use. Its fast readout answers 72.5% of our 5,442 eval questions (base Qwen3.5-4B: 62.7%);
	routing the least certain 30% to thinking mode lifts that to 81.8% at 729 thinking tokens per question. On the external
	S1MB benchmark, training took the model's Task Avg from 32.8 to 47.9. Report doc: https://claude.ai/artifact/RUdY6E2PZgStqNURLGT8g8

	## What was built

	A Jev-style typed decision model: given evidence, a criterion and options, it answers in two ways.

	- Fast readout (System One): one forward pass, a probability per option from the letter logits, ~40 ms.
	- Thinking mode: reasons, then the same letter readout after the chain.
	- Routing: the readout answers by default; the lowest-margin share of questions goes to thinking.

	Base model Qwen3.5-4B (hybrid Gated DeltaNet + attention), LoRA r32 / α64 on all linear layers. Three model generations
	were trained, plus a ~2B model:

	\| Model \| What changed \|
	\|---\|---\|
	\| v1 \| readout on 3K questions (gold mixed 50/50 with the model's own vote shares) + 2.7K of its own correct chains \|
	\| V2 \| 14x the readout data (42K questions, every pool question, half in the eval's option order), 11K chains (4B + 662 questions from Qwen3.5-35B-A3B via API, $6.60), stricter chain filters, lr 2e-4 from a 3-rate sweep, 1.5x faster training setup \|
	\| V2 + RL \| GRPO on V2's thinking mode: 60 steps (40 at 1e-5, 20 at 2.5e-5), 8 chains per question, +1 / -1 reward (annotator share for soft labels), -1.5 for no answer by 8,192 tokens, 240 questions (60% never trained on, mostly reasoning-heavy tasksource sources) \|
	\| 2B \| MiniCPM5-2B picked over Qwen3.5-2B by a pre-registered probe rule; V2 recipe, and a readout-only variant \|

	## Headline results (our eval: 5,442 questions; thinking on a 1,498-question subset)

	\| Model \| Fast readout \| Thinking \| Looped chains \| Routed 30% to thinking \| Thinking tokens per question at 30% \|
	\|---\|---\|---\|---\|---\|---\|
	\| Base Qwen3.5-4B \| 62.7% \| 84.0% \| 0.9% \| 73.0% \| 1,069 \|
	\| v1 \| 69.5% \| 84.4% \| 1.4% \| 78.0% \| 1,166 \|
	\| V2 \| 72.7% \| 81.2% \| 5.1% \| 81.8% \| 1,749 \|
	\| V2 + RL (step 60) \| 72.5% \| 83.1% \| 0.2% \| 81.8% \| 729 \|
	\| V2 + RL (step 40, 1e-5 only) \| 72.7% \| 83.6% \| 2.0% \| 82.0% \| 1,079 \|

	(Routing columns use the 1,498-question subset: fast readout there is 61.8 / 70.3 / 76.0 / 75.9 / 76.0%.)

	- The fast readout gained 10 points over the base model (62.7 -> 72.5%), most of it from data: V2's 42K questions
	took it from v1's 69.5% to 72.7% and its held-out dev score from 65% (1,500 questions) to 81.3%.
	- V2's chain training brought loops into thinking mode (5.1% of chains ran to the 32K cap with no answer), costing
	3.2 points; RL removed them (0.2%) and recovered most of the loss, at half the thinking tokens.
	- The routed system is the deployable unit: V2 + RL reaches 81.8% at 30% routed with 729 thinking tokens per
	question, vs v1's 78.0% at 1,166 and the base model's 73.0%. It recovers 79% of the gap between fast and thinking mode
	(the plan's target was 80%; v1 recovered 52%).
	- Calibration stays good: fast-readout ECE 0.030 (15 bins, after per-type temperatures fitted on calibration items);
	after-chain readout ECE 0.057 for V2 + RL vs 0.043 for V2, a difference within noise (+0.014, 95% CI -0.006 to +0.028).

	By eval set (fast readout): procedural 46.8 -> 89.5, tasksource 60.4 -> 72.3, OpenDecisions 66.5 -> 73.9, LocalLLaMA
	63.0 -> 66.3 (base -> V2 + RL). The two sets least like the training data gained least.

	## What the ablations and side experiments showed

	- Votes vs gold labels (V2 ablations, the same 12K questions): mixing the model's own vote shares into the readout
	target lowered accuracy (70.5% vs 71.7% gold-only; -1.2 points, 95% CI -2.1 to -0.3) and made calibration worse on the eval sets (ECE 0.047 vs
	0.031), although it looked better on in-distribution dev (0.017 vs 0.034). The votes help on familiar question types
	and hurt on new ones. v1's gold-only readout remains the best calibrated (0.019).
	- RL learning rate: the 1e-5 phase gave the accuracy (+2.4 points over V2); the 2.5e-5 phase mostly removed the
	remaining loops (2.0% -> 0.2%) and 30% of the tokens. Training-time accuracy at equal difficulty was flat; the clear
	training-time effect was decisiveness ("Wait" per finished chain 10 -> 5).
	- Speed: keeping activations for micro-batches up to 8K tokens (no recompute) gave 1.48x training throughput at
	equal numerics (user's call over a 72 GB guard; peak 71.5 GiB of 80, no OOM). Liger kernels failed the fp32 equivalence
	check and were not used.
	- Data hygiene: dev and eval ids are disjoint from training; 61 of 999 dev items and 50 of 526 procedural eval items
	repeat a training question exactly (procedural generators); removing them changes procedural accuracy by <= 0.7 points.

	## The 2B model

	\| 2B variant \| Fast readout \| Thinking \| Looped \| Routed 30% \| Routed 50% \|
	\|---\|---\|---\|---\|---\|---\|
	\| MiniCPM5-2B, untrained \| 50.2% \| 79.4% \| 0.1% \| \| \|
	\| V2 recipe (with chains) \| 65.1% \| 71.7% \| 7.4% \| 71.9% \| 72.0% \|
	\| Readout-only (no chains) \| 66.0% \| 70.8% \| 2.1% \| 71.0% \| 71.0% \|
	\| Readout-only fast + untrained thinking \| 66.0% \| 79.4% \| 0.1% \| 75.6% (253 tok) \| 78.0% (411 tok) \|

	(Routing in this table: lowest readout margin to thinking, the thinking answer is the chain's parsed letter; the same
	method gives 81.9% / 742 tokens for V2 + RL at 30%.)

	- MiniCPM5-2B thinks almost as well as our 4B untrained (79.4% vs 83-84%, at ~40% of the tokens), so it was picked.
	- Every fine-tune damaged its thinking (-8 to -9 points): the V2 recipe by copying the 4B's and the teacher's longer,
	loop-prone chains; readout-only training by teaching it to answer at once (median chain 441 -> 127 tokens).
	- The working 2B is two states of one model: the trained LoRA for the fast readout, the adapter off for thinking.
	That reaches 75.6% at 30% routed (253 thinking tokens per question) and 78.0% at 50%, within 4-6 points of the 4B.

	## External benchmarks

	### Jevals (jevals.com, suite 0.1.0, 3 boards x 300; Decision Score = Brier / RPS skill vs the label prior)

	\| Board \| Gemini 3.8 Flash \| Jev \| Ours, V2 + RL fast \| Ours, V2 + RL thinking \| Ours, 2B readout-only fast \|
	\|---\|---\|---\|---\|---\|---\|
	\| PubMedQA (yes/no, unseen in training) \| 73.0 \| 69.0 \| 58.6 (5th of 8) \| 63.5 (3rd) \| 44.6 \|
	\| Banking77 (77 intents) \| 74.1 \| 67.8 \| 61.3 (6th) \| 59.3 \| 53.9 \|
	\| HelpSteer2 (5-level score) \| 4.6 \| 9.2 \| 2.4 \| -20.1 \| -0.4 \|

	Fast mode sits mid-pack, level with mid-tier LLMs and 6-10 points behind Jev; its probabilities are among the best
	calibrated on the board (Banking77 ECE 2.8 points vs Jev 9.8). Thinking raises accuracy where the task is reading and
	reasoning (PubMedQA 86 -> 90%) but its post-chain probabilities are overconfident, which Brier punishes. Training lifted
	the base 4B's PubMedQA score from 48.1 (raw; 36.4 with our temperatures) to 58.6 on a dataset it never saw.

	### S1MB (System One Mosaic Benchmark, English v1: 137 benchmarks, 26,269 judgments; 211 complete published models)

	Training took the 4B's Task Avg from 32.8 to 47.9 and the 2B's from 18.1 to 37.3; 74 of 211 published models rank
	above V2 + RL in the leaderboard's Borda order. All runs use the fast readout.

	\| Model \| Task Avg \| Yes/no \| Choice \| Score \| Published models above (Borda) \|
	\|---\|---\|---\|---\|---\|---\|
	\| V2 + RL (4B) \| 47.9 \| 50.7 \| 57.2 \| 35.8 \| 74 \|
	\| v1 (4B) \| 43.5 \| 50.1 \| 52.3 \| 28.0 \| 100 \|
	\| 2B readout-only \| 37.3 \| 42.3 \| 45.7 \| 24.0 \| 123 \|
	\| Qwen3.5-4B, untrained \| 32.8 \| 33.8 \| 45.7 \| 18.9 \| 134 \|
	\| MiniCPM5-2B, untrained \| 18.1 \| 22.9 \| 29.5 \| 2.0 \| 162 \|
	\| superion_s1 (best Task Avg) \| 74.4 \| \| \| \| \|
	\| KnowLine-4B-Gen5 (best published 4B) \| 67.0 \| 65.7 \| 76.5 \| 58.8 \| 19 \|
	\| KnowLine-2B-Gen1 (best published 2B) \| 44.8 \| 48.4 \| 57.4 \| 28.7 \| 88 \|
	\| Median published model \| 41.2 \| \| \| \| \|

	- Most of the gain came with v1 (+10.7 of +15.1). V2's data and RL then added choice (+4.9) and score (+7.8) skill
	but almost no yes/no skill (50.1 -> 50.7). Benchmarks: 105 of 137 up, 16 down (by more than 0.01).
	- Training fixed a "no" bias and overshot on balanced sets. The untrained 4B said yes on 18% of yes/no judgments
	where the target is yes 28% of the time; V2 + RL says yes 32% (true recall 44% -> 70%, false recall 91% -> 83%). It
	now over-says yes on SciTail, Qasper, CanTTalk and Open-Jev customer-control, 4 of the 5 benchmarks that fell by more
	than 0.1.
	- The largest gains are Open-Jev control and workflow tasks: drone control +1.0, entity alignment +0.83, invoice
	processing +0.65, VizDoom +0.64 (adjusted skill, 0-1).
	- The 2B gained more than the 4B: readout-only training added 19.2 points; among the ~2B published models only
	KnowLine-2B-Gen1 (44.8) and unee-2b (44.4) score higher.
	- Method. `s1mb_eval.py` reimplements the published SCORING.md (running their evaluator was refused by the
	permission check as third-party code) and is checked on 3 published models: 48 benchmark scores match to 6e-16.
	Per-task temperatures fitted on our own calibration items (never S1MB) change only the score task: V2 + RL is 46.9
	without them. Prompts are our native format; the 904 judgments with more than 26 options use a tournament readout.
	V2 and the V2-recipe 2B were not run (V2's fast readout equals V2 + RL's; the V2-recipe 2B is dominated by the
	readout-only 2B).

	### S1MB with routing (V2 + RL, least certain 30% of judgments to thinking)

	\| Task \| Fast readout \| 30% routed \| Difference [95% CI] \|
	\|---\|---\|---\|---\|
	\| Yes/no \| 50.7 \| 58.3 \| +7.6 [+4.5, +11.6] \|
	\| Choice \| 57.2 \| 67.6 \| +10.4 [+6.2, +14.9] \|
	\| Score \| 35.8 \| 39.8 \| +4.0 [-1.1, +9.3] \|
	\| Task Avg \| 47.9 \| 55.2 \| +7.3 [+4.9, +9.9] \|

	Published models above it in Borda order: 74 → 50 (Task Avg level with KnowLine-4B-Gen2, 55.3). 92 benchmarks up, 20 down:
	thinking wins on computation and multi-step reasoning (Snake +0.86, logical entailment +0.62, temporal NLI +0.53,
	tic-tac-toe +0.51, AQuA-RAT +0.45) and loses on judgment and graded relevance (ETHICS -0.33, search-relevance score tasks
	-0.09 to -0.13). 943 thinking tokens per judgment; 158 of 7,881 chains (2.0%) gave no answer and kept the fast answer.
	Not a like-for-like leaderboard entry: S1MB measures single-pass judgments. The 2B hybrid's routed run was skipped.

	## Robustness and error analysis (added Nov 1)

	- Bootstrap intervals (`results/final_stats.md`): readout gains over the base (+9.8 [8.4, 11.2]) and v1 (+3.0 [1.9, 4.2])
	hold; RL's thinking gain over V2 holds (+1.9 [0.3, 3.5]); routed V2 + RL ties V2 (+0.1 [-1.3, 1.1]) at 1,020 fewer
	thinking tokens per question. Corrected: vote-mixed targets lowered readout accuracy (-1.2 [-2.1, -0.3]); the
	post-chain ECE rise after RL is within noise. Hypotheses: H2 confirmed, H3 met by V2 + RL only, H1 not confirmed,
	H4 undecided (79.5% [64%, 97%]).
	- S1MB overlap check: gains are larger on benchmarks from datasets never trained on (+17.4 vs +8.4, difference +9.0
	[3.4, 14.5]); only 104 of ~18,700 cases share most of their text with training data.
	- Chains read by hand (`results/examples_analysis.md`): thinking fixes by computing (OpenDecisions 77.0 → 95.8%) and
	breaks even on judgment sets, where margin routing spends most of its budget; routing by task type could add up to
	2.3 points. Chains that guess at dataset conventions are right ~55% vs ~90%. Only 49% of RL questions gave a learning
	signal (the fast readout's p(gold) did not predict thinking-mode difficulty).
	- Yes/no bias: V2 + RL says yes on 62% of our yes/no questions (gold 50%); one calibration-fitted offset lifts their
	balanced accuracy 78.4 → 83.6%, but does not help S1MB.

	The full write-up (23 pages, 23 figures): `results/report/decision_model_report.pdf` (bucket: `report/`).

	## What I would do next

	1. Fix the chain recipe before more data: filter re-verification loops ("Wait, I'll consider ...") out of training
	chains, and only train chains a model cannot already produce better itself (MiniCPM's own were better than ours).
	2. Calibrate the after-thinking readout (soft post-chain targets or a calibration term in the RL reward): it is
	what costs thinking mode on Brier-scored benchmarks.
	3. More RL, set up better: questions screened to split 30-70%, 12-16 chains per question, a 6K-token cap, no spatial
	puzzles, and the speed work first (batched learning steps, the fast conv kernel) to afford hundreds of steps.
	4. Ship the 2B as fast-readout LoRA + untouched thinking, and test RL on its readout-only variant to recover
	thinking without losing the readout.
	5. Out-of-distribution gains are the weak spot (LocalLLaMA +3 points over the base, OpenDecisions +7): more diverse
	training sources beat more questions from the same sources.

	## Where things are

	- Bucket: `run1/` (v1), `run2_v2/` (V2, ablations, RL with every sampled chain, merged `v2_joint`, `rl60`), `run3_2b/`
	(the 2B, incl. merged `v2b_ro_merged`), `jevals/`, `s1mb/` (items, our raw readouts, score table), `code/`, `dashboard/`,
	`README.md`, `FINAL_REPORT.md`.
	- Deployable models: `run2_v2/checkpoints/rl60_merged` (4B, bf16, text-only Qwen3_5ForCausalLM); the 2B readout LoRA in
	`run3_2b/results/adapters/v2b_ro` on top of `openbmb/MiniCPM5-2B`.
	- Known caveats: single seeds; the RL run was short (60 steps); Jevals and S1MB were scored with our own reimplementation
	of their rules (checked against their published numbers), our prompts are our native format, and our training pool
	contains train-split items of a few of their source datasets (Banking77, HelpSteer2; never the evaluated items).