report/REPORT.md

A decision model that answers fast and thinks when unsure

%subtitle Technical report · Qwen3.5-4B and MiniCPM5-2B · one A100 80GB, Oct 30 – Nov 1, 2026 · code, data, logs and checkpoints in the public bucket Mercity/Desicion-Model-Custom

In one paragraph. We trained a 4B model that answers typed decision questions (pick an option, say yes or no, give a score) in one forward pass, and thinks step by step only on the questions it is least sure of. The final model, V2 + RL, answers 72.5% of our 5,442 eval questions from the fast readout alone (untrained Qwen3.5-4B: 62.7%). Sending the least certain 30% to thinking lifts that to 81.8% at 729 thinking tokens per question, the same accuracy as the pre-RL model at 42% of its thinking budget. On the external S1MB benchmark (137 tasks, 211 published models), training raised Task Avg from 32.8 to 47.9, with larger gains on benchmarks from datasets we never trained on; with the same routing it reaches 55.2. A 2B version (MiniCPM5-2B) reaches 75.6% routed at a third of the 4B's thinking tokens.

%toc

---pagebreak---

1. Summary

What was built. A Jev-style typed decision model with two modes in one set of weights. The fast readout reads a probability for every lettered option from the next-token logits, in one forward pass (about 40 ms). Thinking mode writes a reasoning chain first and then reads the same letter probabilities after it. A router answers from the fast readout and sends the questions with the smallest margin between the top two options to thinking.

Three 4B generations and a 2B model were trained:

Model	What changed	Fast readout	Thinking	Routed 30%	Thinking tokens / question at 30%
Qwen3.5-4B, untrained	the base	62.7%	84.0%	73.0%	1,070
v1	3K readout questions + 2.7K own chains	69.5%	84.4%	78.0%	1,166
V2	42K readout questions, 11K chains incl. a 35B teacher	72.7%	81.2%	81.8%	1,749
V2 + RL	60 GRPO steps on thinking mode	72.5%	83.1%	81.8%	729
2B hybrid	MiniCPM5-2B readout LoRA + untrained thinking	66.0%	79.4%	75.6%	253

Fast readout on all 5,442 eval questions; thinking and routing on a fixed 1,498-question subset (fast readout there: 61.8 / 70.3 / 76.0 / 75.9% for the four 4B rows).

Main findings

• Data drove the fast readout. 14x more readout questions took it from 69.5% (v1) to 72.7% (V2); every readout gain over the base and over v1 holds under paired bootstrap (+9.8 [8.4, 11.2] and +3.0 [1.9, 4.2] points).
• Training on the model's own chains hurt thinking; RL repaired it. V2's chain training brought re-verification loops (5.1% of chains ran to 32K tokens without answering) and cost 2.9 points of thinking accuracy. Sixty GRPO steps removed the loops (0.2%), recovered most of the accuracy (+1.9 [0.3, 3.5] over V2) and more than halved the tokens.
• The routed system is what to deploy. At 30% routed, V2 + RL recovers 79% of the gap between fast and thinking accuracy (plan target 80%; 95% interval 64-97%), at 729 thinking tokens per question.
• Thinking wins by computing, not by judging. It lifts OpenDecisions from 77% to 96% (sums, dates, table joins), but on judgment-heavy sets it breaks about as many answers as it fixes. Margin routing spends most of its budget on those sets.
• External benchmarks. S1MB Task Avg 32.8 → 47.9 for the fast readout (74 of 211 published models rank higher; best published 4B: 67.0), and 55.2 with 30% routed to thinking (+7.3 [4.9, 9.9]; 50 rank higher), so the routed system transfers to an outside benchmark. Gains are larger on benchmarks from unseen datasets (+17.4 vs +8.4), so they are not memorisation. On Jevals the fast readout is mid-pack and well calibrated, 6-10 points behind Jev.
• Two plan hypotheses hold, two do not. The trained readout flags its own likely disagreements with thinking (H2, confirmed); training did not cost thinking more than a point (H3, met by V2 + RL only). Vote-mixed readout targets did not beat gold labels (H1: they were worse overall), and the 80% gap target (H4) is inside the interval but not met by the point estimate.

. Not shown: setup and data preparation on Oct 30, and the CPU-only analysis and report work afterwards.)

2. The task and the design

2.1 Typed decisions

Every question carries evidence, a criterion and a list of options, and is one of three types:

• choice: pick one of K options (2 to 26 letters; larger option sets use a tournament of readouts);
• yes/no ("noul"): is the proposition true;
• score: pick a level on an ordered scale.

Targets are a gold option, or a soft distribution over options where several annotators voted (LocalLLaMA panel shares, some tasksource sets).

2.2 Two modes, one model


• Fast readout. The prompt (thinking off) ends where the answer letter goes; the next token is restricted to the K option letters and their log-probabilities are read in one forward pass (vLLM allowed_token_ids, processed logprobs). Per-type temperatures are fitted on a held-out calibration set of 515 questions and never on eval data.
• Thinking mode. Same prompt with thinking on; the model writes a chain inside its think block, then the answer. The decision is the parsed answer letter, and we also read the letter probabilities right after the chain (the post-chain readout) from the exact generated token ids. A chain that never closes by 32,768 tokens counts as looped; in the routed system it falls back to the fast answer.
• Routing. Questions are ranked by the fast readout's margin (top-1 minus top-2 probability, after temperatures) and the lowest share goes to thinking. A learned agreement router was also fitted in v1 and did not beat the margin.

2.3 Hypotheses from the plan

The training plan fixed four hypotheses before any run: H1 a readout trained on targets that mix gold with the model's own vote shares beats a gold-only readout in distribution; H2 the trained readout's margin predicts disagreement with thinking better (AUROC +0.03); H3 training does not cost thinking more than 1 point; H4 routing 30% to thinking recovers at least 80% of the gap between fast and thinking accuracy. Section 6 gives the verdicts with intervals.

2.4 Base model

Qwen3.5-4B (hybrid: 24 Gated DeltaNet linear-attention layers and 8 full-attention layers; we use a 32K context) was chosen over the plan's Qwen3-4B after a probe on the same eval items: readout +5 to +7 points on the natural-language sets, OpenDecisions Capability 55.2 vs 31.9, and stronger chains (procedural: 97% of first chains right vs 87%). Its chains are about three times longer (median 1,801 vs 553 tokens), which shaped every later budget. Adapters: LoRA rank 32, alpha 64, on all attention, linear-attention (in_proj_qkv, in_proj_z, out_proj) and MLP projections.

3. Data

3.1 Evaluation sets

Set	Questions	What it is
Procedural	526	generated typed decisions with exact labels (tables, budgets, schedules, rules), test split
tasksource	599	typed decisions converted from tasksource datasets, held-out items
OpenDecisions	2,352	the OpenDecisions benchmark (easy and hard tiers)
LocalLLaMA	1,965	real-world decisions with panel-vote soft labels
All	5,442	choice 2,490 · yes/no 1,395 · score 1,557

Thinking mode is evaluated on a fixed 1,498-question subset (300 / 300 / 448 / 450). A 515-question calibration set (216 procedural, 299 tasksource) fits temperatures and routers; a 999-question dev set guards training. Dev and eval ids are disjoint from training; 61 dev and 50 procedural eval items repeat a training question exactly (procedural generators), and removing them moves procedural accuracy by at most 0.7 points.

3.2 Training data

• v1: 17,951 pool questions (9,000 procedural, 8,951 tasksource). The 4B wrote 4 chains each for 3,169 of them (12,676 chains, 41.0M tokens); training used each question's shortest correct chain (2,673) and a readout target mixing gold 50/50 with the model's vote share.
• V2: every pool question as a readout example: 41,994 questions (18,000 procedural, 23,994 tasksource from 717 sources; choice 26,034, yes/no 8,982, score 6,978), half shown in the option order the eval uses. Chains: the 4B's own, plus 4 chains each for 662 hard tasksource questions from Qwen3.5-35B-A3B through OpenRouter ($6.60). Stricter chain filters: at least 2 of 4 samples correct, a near-loop filter, and an invented-recall filter for chains that claim to remember the dataset ("I found a similar question online"), which dropped 317 more chains after a 50-chain audit.

Keeping each question's shortest correct chain did most of the cleaning: it cut the 4B's tasksource chains from a median 1,662 to 734 tokens and "Wait" per chain from 8.7 to 2.3. After selection the teacher's chains were only 12% shorter than the 4B's own.

4. Training

4.1 Supervised training (v1, V2)

One LoRA is trained jointly on two losses: a readout loss (cross-entropy of the K-letter softmax against the target distribution, at the answer position with thinking off) and a chain loss (next-token loss on the chain, the answer letter and the end-of-turn token). V2's learning rate came from a three-rate sweep on 1,500 questions scored on 500 dev questions, with a rule fixed before the results (lowest rate within 0.01 dev NLL of the best): 2e-4.

A 12-minute speed benchmark ran first. Keeping activations for micro-batches up to 8K tokens instead of recomputing them gave 1.48x throughput at equal numerics; Liger kernels were 11% faster but failed the fp32 loss-equivalence check. The benchmark's own memory rule (under 72 GB) rejected the fast setup at a 74.9 GB (69.8 GiB) peak; on the project owner's call it was used anyway with an out-of-memory fallback, and the run peaked at 71.5 GiB with no out-of-memory event.

The V2 joint run trained 1,309 steps on 47.8M tokens in 2 h 46 min (about 4,950 tokens/s). Two ablations trained readout-only on the same 12K questions: gold-only targets and vote-mixed targets.

4.2 Reinforcement learning on thinking mode

V2's thinking mode had lost accuracy to looping chains, so we ran GRPO (TRL 1.17.1) on it, starting from the merged V2 model with a fresh LoRA:

• Questions: 480 selected, 60% fresh tasksource questions the model had never trained on (for out-of-domain transfer), two thirds from reasoning-heavy sources, screened to a fast-readout gold probability between 0.15 and 0.85 so they are neither trivial nor hopeless. 240 were used, 151 of them (63%) never trained on.
• Rollouts: 8 chains per question, rounds of 128 chains (16 questions) feeding 4 optimizer steps; chains capped at 8,192 tokens.
• Reward: +1 right / -1 wrong (for soft labels, 2 x target share - 1); -1.5 for no answer by the cap; minus 0.25 per 4,096 tokens beyond the first 4,096.
• Objective: DAPO-style token-level loss, batch-level reward scaling, no KL term; importance-sampling correction per token (token_truncate, ratios clipped at 3). The TRL default (sequence_mask) zeroed most long chains (ratio mean 0.55 on 4.5K-token chains) and was replaced after the first round.
• Schedule: steps 1-40 at lr 1e-5; then, on the project owner's call, steps 41-60 at 2.5e-5 from the step-40 adapter with a fresh optimizer and only unused questions. 3 h 35 min in all.

Training-time accuracy at equal difficulty stayed flat; the clear training-time effect was decisiveness ("Wait" per finished chain fell from about 10 to 5). The eval effect was larger than the training curves suggested (Section 5).

Only 49% of the 240 questions gave GRPO a right-versus-wrong signal (some but not all of their 8 chains right). The fast readout's gold probability, used to screen the never-trained questions, did not predict thinking-mode difficulty: 46% of the reasoning-heavy new questions were solved by all 8 chains. Screening by thinking-mode solve rate would roughly double the useful questions per step.

5. Results on our evaluation set

5.1 Headline

• The fast readout gained 10 points over the base model (62.7% → 72.5%). V2 + RL's readout equals V2's (difference -0.2 [-0.5, 0.0]).
• Thinking: base 84.0%, v1 84.4%, V2 81.2%, V2 + RL 83.1%. V2's loss was loops; RL removed them.
• Calibration: fast-readout ECE 0.030 for V2 + RL (15 bins). After the chain the readout is less calibrated (ECE 0.057, V2 0.043; the difference is within noise).

5.2 Accuracy against thinking budget

The fairest comparison of the systems is accuracy against the thinking tokens they spend, as the routed share goes from 0 to 100%.

V2 + RL reaches V2's routed accuracy at well under half the thinking tokens (729 vs 1,749 per question at 30%; difference -1,020 [-1,262, -797]), and beats v1 at 30% by 3.9 points [1.7, 5.6]. Routing more than 30% adds little: 82.2% at 50%.

5.3 By evaluation set

Procedural gained most (46.8% → 89.5%), then tasksource (60.4 → 72.3), OpenDecisions (66.5 → 73.9) and LocalLLaMA (63.0 → 66.3). The sets least like the training data gained least.

5.4 Calibration and chain length

5.5 Yes/no bias

On our eval's yes/no questions V2 + RL says yes 62% of the time where the gold is yes 50% of the time. One logit offset fitted on the 109 yes/no calibration questions (never eval data) lifts their balanced accuracy from 78.4% to 83.6%. On S1MB the same offset does not help (Section 8.2).

6. How solid the numbers are

Every result comes from one training run per configuration, so we resampled the eval questions (paired bootstrap, 10,000 resamples, temperatures fixed at their calibration values) to put 95% intervals on every comparison. All 65 reported point estimates were reproduced from the saved predictions first.

Plan hypothesis	Result [95% CI]	Verdict
H1: vote-mixed readout targets beat gold-only, in distribution	+1.5 [-0.4, +3.4] points; on all eval -1.2 [-2.1, -0.3]	Not confirmed; worse overall
H2: the trained readout flags disagreement with thinking better (AUROC +0.03)	+0.060 [+0.030, +0.090] vs base; +0.053 [+0.026, +0.080] vs v1	Confirmed against base
H3: training costs thinking at most 1 point	V2 + RL -0.9 [-2.6, +0.7]; V2 -2.9 [-4.7, -1.0]	Met by V2 + RL on the point estimate; failed by V2
H4: routing 30% recovers at least 80% of the gap	79.5% [64%, 97%]	Undecided: 80% is inside the interval

Two claims from earlier drafts did not survive this check and are corrected here: vote-mixed targets lowered readout accuracy (not merely failed to raise it), and the higher post-chain ECE after RL is within noise.

7. What the chains show

We read 114 of V2 + RL's chains by hand (30 each where thinking fixed the readout, broke it, or both were wrong; 12 each way where RL changed V2's outcome) and counted patterns over the whole 1,498-question subset.

Set	Fast readout	Thinking	Routed 30%	Fixes / breaks	Routed questions
Procedural	91.0%	97.0%	97.0%	25 / 7	33
tasksource	71.3%	72.0%	73.7%	36 / 34	112
OpenDecisions	77.0%	95.8%	86.6%	93 / 9	99
LocalLLaMA	67.8%	68.7%	72.4%	60 / 56	205
All	75.9%	83.1%	81.8%	214 / 106	449

• Fixes come from explicit computation. In 21 of the 30 fixes read, the chain does sums, date checks, counts, table joins or rule-by-rule checks where the readout had picked a plausible distractor.
• Margin routing spends its budget where thinking helps least. LocalLLaMA takes 205 of the 449 routed questions for a net +21 answers; OpenDecisions gets 99 for +43 and would give +84 if all of it were routed. The readout's 51 confident errors (top probability at least 0.8, 30 of them OpenDecisions) are never routed, and thinking fixes 34 of them. Routing by set as well as margin could reach up to 84.1% at the same budget (an upper bound: the split was chosen on these questions).
• Weak-chain markers. About a fifth of chains guess at dataset conventions ("often in these datasets...", "I found a similar question online"); they are right 53-57% of the time against about 90% for the rest. Chains with 10 or more "Wait"s are right 52% of the time, no better than the readout on the same questions. Both probably mark an unsure model rather than cause the error, and both are cheap filters for training chains.
• Label noise limits the ceiling. 55 of LocalLLaMA's 141 thinking errors pick an option whose annotator share is within 0.15 of the gold's; counting those as right takes thinking from 83.1% to 86.8%. Of 72 wrong answers read, 26 were near-ties or had two defensible answers and 2 golds looked wrong. Real thinking errors: escalation on LocalLLaMA (12), misreading the criterion (9), over-literal reading (7), slips (3).
• What RL changed. 47 of the 91 questions that flipped from V2-wrong to right were V2 loops. In the 62 flips the other way, V2 + RL's chain was shorter 65% of the time and sometimes skipped a check V2 had made (am/pm, a rubric's own definition).

8. External benchmarks

8.1 Jevals

Jevals (jevals.com, suite 0.1.0, release 2026-09-18) has three 300-question boards scored by Decision Score: Brier skill (yes/no, choice) or RPS skill (score) against the label prior, so 0 means no better than the prior. We rebuilt the suite from its published files (hash-checked) and scored with its rules.

Board	Gemini 3.8 Flash	Jev	Ours, V2 + RL fast	Ours, V2 + RL thinking	Ours, 2B fast
PubMedQA (yes/no, unseen in training)	73.0	69.0	58.6 (5th of 8)	63.5 (3rd)	44.6
Banking77 (77 intents)	74.1	67.8	61.3 (6th)	59.3	53.9
HelpSteer2 (5-level score)	4.6	9.2	2.4	-20.1	-0.4

The fast readout sits mid-pack, 6-10 points behind Jev, and is among the best calibrated on the board (Banking77 ECE 2.8 points vs Jev's 9.8). Thinking helps where the task is reading and reasoning (PubMedQA accuracy 86% → 90%), but its overconfident post-chain probabilities are punished by Brier (HelpSteer2 -20.1). Training transferred to a dataset it never saw: the base 4B scores 48.1 on PubMedQA, V2 + RL 58.6.

8.2 S1MB

S1MB (System One Mosaic Benchmark, English v1) has 137 benchmarks (59 yes/no, 57 choice, 21 score) and 26,269 judgments; 211 complete published models form the comparison set. Running the S1MB evaluator itself was refused by this environment's permission check (third-party code), so s1mb_eval.py reimplements its published scoring rules and was checked by re-scoring 3 published models from their saved predictions: 48 benchmark scores match to 6e-16. Per-task temperatures come from our own calibration items (they affect only the score task; V2 + RL is 46.9 without them).

Model	Task Avg	Yes/no	Choice	Score	Published models above (Borda)
V2 + RL (4B)	47.9	50.7	57.2	35.8	74
v1 (4B)	43.5	50.1	52.3	28.0	100
2B readout-only	37.3	42.3	45.7	24.0	123
Qwen3.5-4B, untrained	32.8	33.8	45.7	18.9	134
MiniCPM5-2B, untrained	18.1	22.9	29.5	2.0	162
KnowLine-4B-Gen5 (best published 4B)	67.0	65.7	76.5	58.8	19
Median published model	41.2

• Training gains are solid: +15.1 [12.2, 18.1] Task Avg over the untrained 4B, +4.4 [2.4, 6.5] over v1, and +19.2 [15.5, 22.8] for the 2B (benchmarks resampled within task). Score went up on 105 of 137 benchmarks and down on 16.
• Most of the gain came with v1 (+10.7 of +15.1); V2's data and RL then added choice (+4.9) and score (+7.8) skill but almost no yes/no skill.
• The gain is not training overlap. Only 104 of about 18,700 S1MB cases share most of their text with our training data (88 of them template-built corr2cause, where we score low). 41 benchmarks come from datasets whose train splits we trained on; the gain on the other 96 is larger, +17.4 vs +8.4 (difference +9.0 [3.4, 14.5]), and holds within yes/no and choice alone.
• The largest gains are Open-Jev control and workflow tasks (drone control +1.0, entity alignment +0.83, invoice processing +0.65 adjusted skill); the drops are mostly yes/no (customer-control -0.28, SciTail -0.18, CanTTalk -0.14, Qasper -0.12).
• The yes/no drops are not one global bias on S1MB. The untrained model said yes on 18% of yes/no judgments (gold 28%); V2 + RL says yes on 32%. The calibration-fitted offset recovers 4 of the 5 benchmarks that dropped but hurts as many others (24 of 59 yes/no benchmarks up, 27 down; yes/no 50.7 → 49.9): the threshold is off in different directions on different tasks.

8.3 S1MB with routing

Running the deployed system on S1MB (fast readout on every judgment, thinking on the 30% with the smallest margin: 7,881 judgments) raised Task Avg from 47.9 to 55.2 (+7.3, 95% CI 4.9 to 9.9; benchmarks resampled within task). Yes/no +7.6 [4.5, 11.6] and choice +10.4 [6.2, 14.9] hold; score +4.0 [-1.1, +9.3] is within noise. 92 benchmarks went up and 20 down. In the Borda order 50 published models rank above it (74 for the fast readout alone); its Task Avg is level with KnowLine-4B-Gen2 (55.3).

• Thinking won where S1MB asks for computation or multi-step reasoning: Snake moves +0.86 adjusted skill, logical entailment +0.62, temporal NLI +0.53, tic-tac-toe +0.51, AQuA-RAT math +0.45, SpartQA spatial reasoning +0.44, BoardgameQA +0.42, part-of-speech tagging (UD-EWT) +0.41.
• It lost on judgment and graded relevance: ETHICS choice -0.33, and each synthetic search-relevance score task by 0.09 to 0.13: the same pattern as on our own eval (Section 7).
• Cost: 943 thinking tokens per judgment on average; S1MB's chains run longer than ours (median 2,096 tokens). 158 of the 7,881 chains (2.0%) gave no answer and kept the fast answer. The run took 2 h 46 min.
• Not a like-for-like leaderboard entry. S1MB is built to measure single-pass ("System One") judgments, and this row spends thinking compute on 30% of them. It answers our question (does the routed system transfer to an outside benchmark?), not where a fast model would rank.
• The same test for the 2B hybrid was skipped to free the GPU sooner.

9. The 2B model

On the project owner's instruction a ~2B model was trained overnight with the same pipeline. Two candidates were probed the way the 4B was and picked by a rule fixed before the probes (mean of readout accuracy and first-chain accuracy, parse rate at least 90%, Qwen wins ties within 0.02): MiniCPM5-2B (Llama architecture, 42 full-attention layers) won over Qwen3.5-2B. Training: V2 recipe, 1,265 steps, 46.0M tokens, ~95 min; readout-only variant, 549 steps, 20.8M tokens, 41 min.

2B variant	Fast readout	Thinking	Looped	Routed 30%	Routed 50%
MiniCPM5-2B, untrained	50.2%	79.4%	0.1%
V2 recipe (with chains)	65.1%	71.7%	7.4%	71.9%	72.0%
Readout-only (no chains)	66.0%	70.8%	2.1%	71.0%	71.0%
Readout-only fast + untrained thinking	66.0%	79.4%	0.1%	75.6% (253 tokens)	78.0% (411 tokens)

• MiniCPM5-2B already thinks well: 79.4% untrained against 83-84% for the 4B, at about 40% of the tokens.
• Every fine-tune damaged its thinking, by 8-9 points. The V2 recipe taught it the 4B's and the teacher's longer, loop-prone chains; readout-only training taught it to answer at once (median chain 441 → 127 tokens), and the shared LoRA carried that into thinking mode. Chain training did not even help its readout (held-out dev 76.0% readout-only vs 74.2%).
• The working 2B is one model in two states: the readout LoRA for fast answers and the adapter switched off for thinking. That reaches 75.6% at 30% routed with 253 thinking tokens per question, 4-6 points below the 4B. (This table routes by the probability gap and takes the chain's letter; under the headline rule, log-margin with fallback to the readout for unfinished chains, the hybrid is 75.8% at 245 tokens.)

10. Limitations

• Single seeds. Every configuration was trained once. The bootstrap intervals cover eval-sample noise, not training noise.
• Short RL. 60 steps on 240 questions. The step-40 checkpoint is as accurate as step 60 in thinking mode (within noise) and slightly better in fast readout (+0.3 [0.04, 0.55]); step 60 is shorter and loops less.
• Our own scorers. Jevals and S1MB were scored with reimplementations of their published rules, checked against their published numbers; prompts are our native format, not the benchmarks' reference adapters. S1MB results could not be validated by its official evaluator here.
• Some benchmark sources overlap our training sources (train splits only, never the evaluated items), measured in Section 8.2.
• Label noise on LocalLLaMA and some tasksource sets caps measurable accuracy (Section 7).
• Not run: the Jev Decision Index (needs its own harness as third-party code), the 2B hybrid's routed S1MB run, and Jevals thinking runs for the base and v1 models.
• Routing threshold and temperatures were fitted on calibration items drawn from procedural and tasksource sources only; they transfer imperfectly to other distributions (the yes/no offset is the clearest case).

11. What to do next

1. Fix the chain recipe before adding data. Filter out re-verification loops and chains that guess at dataset conventions. Only train chains that beat what the model already produces itself (MiniCPM's own chains were better than ours).
2. Route by task type as well as margin. Send computation-style questions to thinking even when the readout is confident; route judgment questions only when the margin is very small. Up to +2.3 points at the same budget on this eval (upper bound).
3. Calibrate both readouts. A per-type yes/no offset for in-distribution use; soft targets or a calibration term in the RL reward for the post-chain readout.
4. Run more RL, set up better: questions screened by thinking-mode solve rate (30-70%, so that most give a learning signal; only 49% did), 12-16 chains per question, a 6K-token cap, no spatial puzzles, and the speed work first (batched learning steps, the fast conv kernel) to afford hundreds of steps.
5. Ship the 2B as readout LoRA + untouched thinking, then try RL on its readout-only variant to recover thinking without losing the readout.
6. Add diverse sources rather than more questions from the same ones; the sets least like the training data gained least.

Glossary

Term	Meaning here
Fast readout	The model's probability for each option letter, read from one forward pass with thinking off
Thinking mode	The model writes a reasoning chain first, then answers; the post-chain readout is the letter probabilities after the chain
Margin	Top-1 minus top-2 option probability of the fast readout; small margin = unsure
Routed x%	The x% of questions with the smallest margin are answered by thinking, the rest by the fast readout
Gap recovered	(routed accuracy - fast accuracy) / (thinking accuracy - fast accuracy)
Looped chain	A chain that never reaches an answer within 32,768 tokens
LoRA	A small set of trainable low-rank weights added to a frozen model (rank 32 here)
GRPO	Group Relative Policy Optimization: RL that samples several chains per question and pushes up the better-than-average ones
ECE	Expected calibration error: the average gap between stated confidence and actual accuracy (15 bins)
Temperature	One number per question type that rescales the logits so probabilities are calibrated; fitted on calibration items only
Brier / RPS skill	Probability-error scores relative to always predicting the label prior (Jevals' Decision Score)
Task Avg	S1MB's index: per benchmark, skill over the best trivial baseline clipped to [0, 1]; averaged within yes/no, choice and score, then across the three
Borda	S1MB's leaderboard order: average rank points over all 137 benchmarks within the compared models
Paired bootstrap	Resampling the same eval questions for both models many times to get a 95% interval on their difference

---pagebreak---

Appendix A. Departures from the plan

The plan said to use judgment; all 31 departures are logged with reasons in results/DEVIATIONS.md. The ones that changed results:

#	Plan	What was done	Why
9	Base model Qwen3-4B	Qwen3.5-4B	Measured stronger on the same items before deciding
10	LoRA on attention and MLP	Also on the linear-attention projections	24 of 32 layers are Gated DeltaNet layers
2	Thinking answer = first letter after the chain	Also a letter readout after the chain	Gives thinking mode probabilities, for calibration and routing
4	Margin router only	Also a learned agreement router	Tested; it did not beat the margin
20	Chains from the 4B only	662 questions from a 35B teacher ($6.60)	Hard tasksource questions the 4B rarely solved
21, 26	Fixed training setup	Speed benchmark; 1.48x setup on the owner's call	Same numerics, 71.5 GiB peak, no OOM
22	LR sweep on 4,000 questions	1,500 questions, rule fixed before results	Time; the pick (2e-4) was clear
27	RL questions from the training pool	60% never-trained questions	Owner asked for out-of-domain transfer
28	TRL's default IS correction	Per-token truncation	The default zeroed most long chains
29	120 RL steps at 1e-5	40 at 1e-5, then 20 at 2.5e-5	Owner's call after the first 40 steps
30	2B after V2	Started overnight regardless of the RL result	Owner's instruction
31	No external benchmarks named	Jevals and S1MB with reimplemented scorers, bootstrap CIs, error analysis, routed S1MB	Rank the work and test transfer; S1MB's evaluator could not be run here

Appendix B. Compute and cost

Stage	Wall time on one A100 80GB
v1 (probe, baseline, chains, training, ablations, eval)	~9 h
V2 chains (4B)	3 h 20 min
V2 speed benchmark, sweep, joint run, ablations, eval	5 h 30 min
RL (60 steps) and evals	4 h 10 min
2B (probes, sweep, two trainings, evals)	5 h 30 min
Jevals, S1MB, routed S1MB	~5 h

API cost: $6.60 for the teacher chains (Qwen3.5-35B-A3B on OpenRouter). Software: vLLM 0.32.0, TRL 1.17.1, transformers with flash-linear-attention kernels, PEFT.

Appendix C. Where everything is

What	Where (bucket Mercity/Desicion-Model-Custom)
Deployable 4B (V2 + RL)	run2_v2/checkpoints/rl60_merged (bf16, text-only Qwen3_5ForCausalLM)
Deployable 2B	run3_2b/checkpoints/v2b_ro_merged; thinking with openbmb/MiniCPM5-2B itself
Every prediction and chain	run1/results, run2_v2/results, run3_2b/results
RL checkpoints and all sampled chains	run2_v2/checkpoints/rl_grpo, rl_grpo_b
Benchmarks	jevals/, s1mb/
Code, notebook, deviations	code/ (incl. NOTEBOOK.md, DEVIATIONS.md, analysis/)
This report	FINAL_REPORT.md, report/decision_model_report.pdf
Live dashboard	https://claude.ai/artifact/XehYsr1iD4hq2zA3ni241r