code/FINAL_REPORT.md

Reasoning decision model on Qwen3.5-4B: final report

Oct 30 - Nov 1, 2026 · one A100 80GB · code, data, logs and checkpoints in the public bucket Mercity/Desicion-Model-Custom (no keys) · notebook results/NOTEBOOK.md, every change to the plan in results/DEVIATIONS.md (31 entries) · dashboard https://claude.ai/artifact/XehYsr1iD4hq2zA3ni241r

V2 + RL is the model to use. Its fast readout answers 72.5% of our 5,442 eval questions (base Qwen3.5-4B: 62.7%); routing the least certain 30% to thinking mode lifts that to 81.8% at 729 thinking tokens per question. On the external S1MB benchmark, training took the model's Task Avg from 32.8 to 47.9. Report doc: https://claude.ai/artifact/RUdY6E2PZgStqNURLGT8g8

What was built

A Jev-style typed decision model: given evidence, a criterion and options, it answers in two ways.

• Fast readout (System One): one forward pass, a probability per option from the letter logits, ~40 ms.
• Thinking mode: reasons, then the same letter readout after the chain.
• Routing: the readout answers by default; the lowest-margin share of questions goes to thinking.

Base model Qwen3.5-4B (hybrid Gated DeltaNet + attention), LoRA r32 / α64 on all linear layers. Three model generations were trained, plus a ~2B model:

Model	What changed
v1	readout on 3K questions (gold mixed 50/50 with the model's own vote shares) + 2.7K of its own correct chains
V2	14x the readout data (42K questions, every pool question, half in the eval's option order), 11K chains (4B + 662 questions from Qwen3.5-35B-A3B via API, $6.60), stricter chain filters, lr 2e-4 from a 3-rate sweep, 1.5x faster training setup
V2 + RL	GRPO on V2's thinking mode: 60 steps (40 at 1e-5, 20 at 2.5e-5), 8 chains per question, +1 / -1 reward (annotator share for soft labels), -1.5 for no answer by 8,192 tokens, 240 questions (60% never trained on, mostly reasoning-heavy tasksource sources)
2B	MiniCPM5-2B picked over Qwen3.5-2B by a pre-registered probe rule; V2 recipe, and a readout-only variant

Headline results (our eval: 5,442 questions; thinking on a 1,498-question subset)

Model	Fast readout	Thinking	Looped chains	Routed 30% to thinking	Thinking tokens per question at 30%
Base Qwen3.5-4B	62.7%	84.0%	0.9%	73.0%	1,069
v1	69.5%	84.4%	1.4%	78.0%	1,166
V2	72.7%	81.2%	5.1%	81.8%	1,749
V2 + RL (step 60)	72.5%	83.1%	0.2%	81.8%	729
V2 + RL (step 40, 1e-5 only)	72.7%	83.6%	2.0%	82.0%	1,079

(Routing columns use the 1,498-question subset: fast readout there is 61.8 / 70.3 / 76.0 / 75.9 / 76.0%.)

• The fast readout gained 10 points over the base model (62.7 -> 72.5%), most of it from data: V2's 42K questions took it from v1's 69.5% to 72.7% and its held-out dev score from 65% (1,500 questions) to 81.3%.
• V2's chain training brought loops into thinking mode (5.1% of chains ran to the 32K cap with no answer), costing 3.2 points; RL removed them (0.2%) and recovered most of the loss, at half the thinking tokens.
• The routed system is the deployable unit: V2 + RL reaches 81.8% at 30% routed with 729 thinking tokens per question, vs v1's 78.0% at 1,166 and the base model's 73.0%. It recovers 79% of the gap between fast and thinking mode (the plan's target was 80%; v1 recovered 52%).
• Calibration stays good: fast-readout ECE 0.030 (15 bins, after per-type temperatures fitted on calibration items); after-chain readout ECE 0.057 for V2 + RL vs 0.043 for V2, a difference within noise (+0.014, 95% CI -0.006 to +0.028).

By eval set (fast readout): procedural 46.8 -> 89.5, tasksource 60.4 -> 72.3, OpenDecisions 66.5 -> 73.9, LocalLLaMA 63.0 -> 66.3 (base -> V2 + RL). The two sets least like the training data gained least.

What the ablations and side experiments showed

• Votes vs gold labels (V2 ablations, the same 12K questions): mixing the model's own vote shares into the readout target lowered accuracy (70.5% vs 71.7% gold-only; -1.2 points, 95% CI -2.1 to -0.3) and made calibration worse on the eval sets (ECE 0.047 vs 0.031), although it looked better on in-distribution dev (0.017 vs 0.034). The votes help on familiar question types and hurt on new ones. v1's gold-only readout remains the best calibrated (0.019).
• RL learning rate: the 1e-5 phase gave the accuracy (+2.4 points over V2); the 2.5e-5 phase mostly removed the remaining loops (2.0% -> 0.2%) and 30% of the tokens. Training-time accuracy at equal difficulty was flat; the clear training-time effect was decisiveness ("Wait" per finished chain 10 -> 5).
• Speed: keeping activations for micro-batches up to 8K tokens (no recompute) gave 1.48x training throughput at equal numerics (user's call over a 72 GB guard; peak 71.5 GiB of 80, no OOM). Liger kernels failed the fp32 equivalence check and were not used.
• Data hygiene: dev and eval ids are disjoint from training; 61 of 999 dev items and 50 of 526 procedural eval items repeat a training question exactly (procedural generators); removing them changes procedural accuracy by <= 0.7 points.

The 2B model

2B variant	Fast readout	Thinking	Looped	Routed 30%	Routed 50%
MiniCPM5-2B, untrained	50.2%	79.4%	0.1%
V2 recipe (with chains)	65.1%	71.7%	7.4%	71.9%	72.0%
Readout-only (no chains)	66.0%	70.8%	2.1%	71.0%	71.0%
Readout-only fast + untrained thinking	66.0%	79.4%	0.1%	75.6% (253 tok)	78.0% (411 tok)

(Routing in this table: lowest readout margin to thinking, the thinking answer is the chain's parsed letter; the same method gives 81.9% / 742 tokens for V2 + RL at 30%.)

• MiniCPM5-2B thinks almost as well as our 4B untrained (79.4% vs 83-84%, at ~40% of the tokens), so it was picked.
• Every fine-tune damaged its thinking (-8 to -9 points): the V2 recipe by copying the 4B's and the teacher's longer, loop-prone chains; readout-only training by teaching it to answer at once (median chain 441 -> 127 tokens).
• The working 2B is two states of one model: the trained LoRA for the fast readout, the adapter off for thinking. That reaches 75.6% at 30% routed (253 thinking tokens per question) and 78.0% at 50%, within 4-6 points of the 4B.

External benchmarks

Jevals (jevals.com, suite 0.1.0, 3 boards x 300; Decision Score = Brier / RPS skill vs the label prior)

Board	Gemini 3.8 Flash	Jev	Ours, V2 + RL fast	Ours, V2 + RL thinking	Ours, 2B readout-only fast
PubMedQA (yes/no, unseen in training)	73.0	69.0	58.6 (5th of 8)	63.5 (3rd)	44.6
Banking77 (77 intents)	74.1	67.8	61.3 (6th)	59.3	53.9
HelpSteer2 (5-level score)	4.6	9.2	2.4	-20.1	-0.4

Fast mode sits mid-pack, level with mid-tier LLMs and 6-10 points behind Jev; its probabilities are among the best calibrated on the board (Banking77 ECE 2.8 points vs Jev 9.8). Thinking raises accuracy where the task is reading and reasoning (PubMedQA 86 -> 90%) but its post-chain probabilities are overconfident, which Brier punishes. Training lifted the base 4B's PubMedQA score from 48.1 (raw; 36.4 with our temperatures) to 58.6 on a dataset it never saw.

S1MB (System One Mosaic Benchmark, English v1: 137 benchmarks, 26,269 judgments; 211 complete published models)

Training took the 4B's Task Avg from 32.8 to 47.9 and the 2B's from 18.1 to 37.3; 74 of 211 published models rank above V2 + RL in the leaderboard's Borda order. All runs use the fast readout.

Model	Task Avg	Yes/no	Choice	Score	Published models above (Borda)
V2 + RL (4B)	47.9	50.7	57.2	35.8	74
v1 (4B)	43.5	50.1	52.3	28.0	100
2B readout-only	37.3	42.3	45.7	24.0	123
Qwen3.5-4B, untrained	32.8	33.8	45.7	18.9	134
MiniCPM5-2B, untrained	18.1	22.9	29.5	2.0	162
superion_s1 (best Task Avg)	74.4
KnowLine-4B-Gen5 (best published 4B)	67.0	65.7	76.5	58.8	19
KnowLine-2B-Gen1 (best published 2B)	44.8	48.4	57.4	28.7	88
Median published model	41.2

• Most of the gain came with v1 (+10.7 of +15.1). V2's data and RL then added choice (+4.9) and score (+7.8) skill but almost no yes/no skill (50.1 -> 50.7). Benchmarks: 105 of 137 up, 16 down (by more than 0.01).
• Training fixed a "no" bias and overshot on balanced sets. The untrained 4B said yes on 18% of yes/no judgments where the target is yes 28% of the time; V2 + RL says yes 32% (true recall 44% -> 70%, false recall 91% -> 83%). It now over-says yes on SciTail, Qasper, CanTTalk and Open-Jev customer-control, 4 of the 5 benchmarks that fell by more than 0.1.
• The largest gains are Open-Jev control and workflow tasks: drone control +1.0, entity alignment +0.83, invoice processing +0.65, VizDoom +0.64 (adjusted skill, 0-1).
• The 2B gained more than the 4B: readout-only training added 19.2 points; among the ~2B published models only KnowLine-2B-Gen1 (44.8) and unee-2b (44.4) score higher.
• Method. s1mb_eval.py reimplements the published SCORING.md (running their evaluator was refused by the permission check as third-party code) and is checked on 3 published models: 48 benchmark scores match to 6e-16. Per-task temperatures fitted on our own calibration items (never S1MB) change only the score task: V2 + RL is 46.9 without them. Prompts are our native format; the 904 judgments with more than 26 options use a tournament readout. V2 and the V2-recipe 2B were not run (V2's fast readout equals V2 + RL's; the V2-recipe 2B is dominated by the readout-only 2B).

S1MB with routing (V2 + RL, least certain 30% of judgments to thinking)

Task	Fast readout	30% routed	Difference [95% CI]
Yes/no	50.7	58.3	+7.6 [+4.5, +11.6]
Choice	57.2	67.6	+10.4 [+6.2, +14.9]
Score	35.8	39.8	+4.0 [-1.1, +9.3]
Task Avg	47.9	55.2	+7.3 [+4.9, +9.9]

Published models above it in Borda order: 74 → 50 (Task Avg level with KnowLine-4B-Gen2, 55.3). 92 benchmarks up, 20 down: thinking wins on computation and multi-step reasoning (Snake +0.86, logical entailment +0.62, temporal NLI +0.53, tic-tac-toe +0.51, AQuA-RAT +0.45) and loses on judgment and graded relevance (ETHICS -0.33, search-relevance score tasks -0.09 to -0.13). 943 thinking tokens per judgment; 158 of 7,881 chains (2.0%) gave no answer and kept the fast answer. Not a like-for-like leaderboard entry: S1MB measures single-pass judgments. The 2B hybrid's routed run was skipped.

Robustness and error analysis (added Nov 1)

• Bootstrap intervals (results/final_stats.md): readout gains over the base (+9.8 [8.4, 11.2]) and v1 (+3.0 [1.9, 4.2]) hold; RL's thinking gain over V2 holds (+1.9 [0.3, 3.5]); routed V2 + RL ties V2 (+0.1 [-1.3, 1.1]) at 1,020 fewer thinking tokens per question. Corrected: vote-mixed targets lowered readout accuracy (-1.2 [-2.1, -0.3]); the post-chain ECE rise after RL is within noise. Hypotheses: H2 confirmed, H3 met by V2 + RL only, H1 not confirmed, H4 undecided (79.5% [64%, 97%]).
• S1MB overlap check: gains are larger on benchmarks from datasets never trained on (+17.4 vs +8.4, difference +9.0 [3.4, 14.5]); only 104 of ~18,700 cases share most of their text with training data.
• Chains read by hand (results/examples_analysis.md): thinking fixes by computing (OpenDecisions 77.0 → 95.8%) and breaks even on judgment sets, where margin routing spends most of its budget; routing by task type could add up to 2.3 points. Chains that guess at dataset conventions are right ~55% vs ~90%. Only 49% of RL questions gave a learning signal (the fast readout's p(gold) did not predict thinking-mode difficulty).
• Yes/no bias: V2 + RL says yes on 62% of our yes/no questions (gold 50%); one calibration-fitted offset lifts their balanced accuracy 78.4 → 83.6%, but does not help S1MB.

The full write-up (23 pages, 23 figures): results/report/decision_model_report.pdf (bucket: report/).

What I would do next

1. Fix the chain recipe before more data: filter re-verification loops ("Wait, I'll consider ...") out of training chains, and only train chains a model cannot already produce better itself (MiniCPM's own were better than ours).
2. Calibrate the after-thinking readout (soft post-chain targets or a calibration term in the RL reward): it is what costs thinking mode on Brier-scored benchmarks.
3. More RL, set up better: questions screened to split 30-70%, 12-16 chains per question, a 6K-token cap, no spatial puzzles, and the speed work first (batched learning steps, the fast conv kernel) to afford hundreds of steps.
4. Ship the 2B as fast-readout LoRA + untouched thinking, and test RL on its readout-only variant to recover thinking without losing the readout.
5. Out-of-distribution gains are the weak spot (LocalLLaMA +3 points over the base, OpenDecisions +7): more diverse training sources beat more questions from the same sources.

Where things are

• Bucket: run1/ (v1), run2_v2/ (V2, ablations, RL with every sampled chain, merged v2_joint, rl60), run3_2b/ (the 2B, incl. merged v2b_ro_merged), jevals/, s1mb/ (items, our raw readouts, score table), code/, dashboard/, README.md, FINAL_REPORT.md.
• Deployable models: run2_v2/checkpoints/rl60_merged (4B, bf16, text-only Qwen3_5ForCausalLM); the 2B readout LoRA in run3_2b/results/adapters/v2b_ro on top of openbmb/MiniCPM5-2B.
• Known caveats: single seeds; the RL run was short (60 steps); Jevals and S1MB were scored with our own reimplementation of their rules (checked against their published numbers), our prompts are our native format, and our training pool contains train-split items of a few of their source datasets (Banking77, HelpSteer2; never the evaluated items).