code/DEVIATIONS.md

Deviations from the plan

Every change to "Reasoning decision model: A100 training plan", with its reason. The plan says to use judgment; these are the places where I did.

#	Plan	What I did	Why
1	Readout eval in transformers	Readout runs in vLLM: next token restricted to the K option letters (allowed_token_ids), processed logprobs returned for all K. Trained models are evaluated from merged bf16 checkpoints, one engine each	Exactly the K-way letter softmax of one forward pass, batched: ~5,400 items in 1.5 minutes. LoRA serving was tried and rejected: vLLM 0.32 silently ignores LoRA adapters on Qwen3.5 (with a perturbed adapter, vLLM's outputs equalled the base model's while PEFT's moved by 0.53). A merged checkpoint in vLLM matches PEFT up to bf16 rounding, and the text-only and multimodal vLLM code paths give identical base outputs (max difference 0.0 on 40 items), so base and trained models are compared on equal terms
2	Thinking answer = first capital letter after </think>	Kept as the primary answer, and added a readout after the chain: the letter softmax at the answer position, given the prompt and the chain	Gives the reasoning mode a probability over options, and checks whether the decision is the one the reasoning leads into (letter vs post-chain readout agreement is reported)
3	OpenDecisions by tier and family	Also scored with OpenDecisions' own score.py (Capability, Intelligence, Calibration)	Puts the numbers on the public OpenDecisions leaderboard scale (4B systems: 58.9 to 77.1)
4	Margin router only	Margin router plus a learned agreement router: logistic regression on readout features (top probability, margin, entropy, K, type), fitted on the calibration set only, predicting agreement with the same model's reasoning	LEV (arXiv 2610.20951) finds a stopper that predicts agreement with the deeper answer beats confidence thresholds, since shallow readouts are overconfident on new data. Costs nothing extra
5	Generation in dataset option order	One fixed shuffled option order per item (score levels keep theirs), saved with the record	Reasoner examples and votes are not tied to the first-listed option; votes are mapped back to the original options
6	Calibration set ~400	515 items (216 procedural, 299 tasksource validation), also given one reasoning sample	Fits per-type temperatures and the learned router (which needs reasoning on its fitting items)
7	Scorer from the dataset repo	od_score.py is a reviewed, unchanged copy of OpenDecisions' MIT score.py	Downloaded code is not imported from the download directory
8	Reasoner loss on chain and letter	Chain, letter and the end-of-turn token	Keeps the trained reasoner stopping cleanly after its letter
9	Base model Qwen3-4B	Qwen3.5-4B (Qwen3-4B kept as the fallback). Measured on the same items before deciding, see the table below	Much stronger decision model: raw OpenDecisions Capability 55.2 vs 31.9, readout +5 to +7 points on the natural-language sets, reasoner 97% vs 87% on procedural items. It is the base of every public 4B decision model, so results are comparable. The plan's concern (custom inference code) did not hold: vLLM 0.32 runs its readout, thinking and post-chain readout. The user also preferred it
10	LoRA on q, k, v, o, gate, up, down	Also on the linear-attention projections in_proj_qkv, in_proj_z, out_proj	24 of Qwen3.5-4B's 32 layers are gated-delta (linear attention) layers; this follows the Inner Jev recipe for Qwen3.5
11	pip install vllm peft datasets accelerate scikit-learn	Also flash-linear-attention (pure Triton) and kernels	Without them transformers runs Qwen3.5's delta rule in a reference PyTorch implementation that ran out of memory at 16k tokens. With them: 3,600-3,900 training tokens/s, 18.4 GB peak at 16k tokens. causal_conv1d stays on its PyTorch fallback (cheap, no build)
12	Preflight with Qwen3-0.6B	Qwen3.5-0.8B	Same architecture as the base, so the preflight tests the hybrid-attention paths
14	Preflight ran every stage	Preflight found four bugs, all fixed before the run: async n=4 sampling returned one sample; vLLM ignored LoRA (see #1); a self-pointing symlink; the speed test lacked model.train(). The preflight's latency stage was stopped (20 sequential 32k-token loops of the 0.8B model) and the remaining paths were tested on few items	Testing the code, not the tiny model's patience
15	vLLM defaults	max_num_seqs=256 from 03:16 UTC on (generation and later stages). The probe and baseline ran at 128	The in-process async engine falls back to vLLM's global default of 128 running sequences (its usage context is not in the per-GPU table). Engine stats: 4.1-4.6k generated tok/s at 128 with the KV cache at ~40%, 4.7k at 256 with ~70%. A first try at 512 (02:51-03:08) could not be judged, since engine stats were off and the completion-based counter lags; my first reading of it ("no faster") was a bad comparison and is withdrawn
16	Generation budget 165 min, ending 3.2-3.4 h before the 8 h end	Generation stopped at 03:12 UTC (380 items kept, none partial) and restarted to run until 05:41 UTC: global window 8.67 h (TOTAL_H=8.67)	Qwen3.5 chains (about 12k tokens per item for 4 samples) at 3.4k tok/s give about 19 items/min, so about 2,500 items by the original deadline, and joint training on that data takes about 40 min instead of the 75 budgeted. The freed time goes to generation (about +750 items) rather than idling; the overall finish moves little
17	If N < 4,000, run only the gold-only ablation	Both ablations run	N will be about 2,800-3,000 (Qwen3.5 chains). The rule saved time; here each readout-only ablation is about 3M tokens (~15 min), and votes-only is the control for the readout copying the reasoner's mistakes
18	10-step timing probe sets how many examples fit	Kept as written; it dropped 5.3% of joint examples (318 of 5,987, whole examples)	The probe ran during first-shape kernel compilation (flash-linear-attention, Triton) and measured 2.1k tok/s; steady state was 2.9k, at which everything would have fit. A probe that skips its first steps would avoid this next time
19	Ablations on the same items as the joint model	Ablations use all 3,169 generated items; the joint readout saw 2,996 (the rest went in #18's drop)	The drop happened inside the joint run. The ablations therefore have about 5% more readout data, which biases H1 against the joint model (conservative)
13	Ablation budget ~25 min each	30 min each; generation stops at the earlier of 165 min or 3.4 h before the global end	Qwen3.5 chains are about 3x longer (median 1,801 vs 553 tokens), so later stages need the margin
20	V2 chains come from the 4B itself	662 tasksource questions from the back of the pool got 4 chains each from Qwen3.5-35B-A3B on OpenRouter (teacher_gen.py, user's key and $7 budget; $6.60 spent, 11.5 min), merged into gen.jsonl before training (source: teacher). Same prompt, option order and sampling (0.6 / 0.95 / 20), no reasoning budget, 12,288-token cost cap (3 of 2,648 hit it). Same filters as local chains. Result: 835 extra training chains from 454 questions; votes for 662 readout targets	User request. Off-policy for the 4B, but same family and tokenizer; Apache-2.0. Teacher accuracy on its tasksource items matches the 4B's on its own (64% 4/4, 21% 0/4 vs 61% / 21%), so the 0/4 tail is mostly label noise (checked by hand on 5), and the teacher's value is shorter (median 1,083 vs 1,567 tokens) and non-repetitive chains, not harder questions solved. Local generation was not shortened, so this adds ~35 min of A100-equivalent generation rather than saving time
21	Training setup fixed (all layers checkpointed, 16K micro-batches, HF kernels)	A ~12 min benchmark (bench_speed.py) runs first: current vs Liger RMSNorm+SwiGLU vs checkpointing only micro-batches above 8K padded tokens vs both (+12K variant). A setup counts only if its loss and LoRA-gradient errors against an fp32 reference are at most 1.5x the current bf16 setup's; the fastest passing one under 72 GB wins if >= 3% faster, else nothing changes	User asked for Liger/compile analysis. A smoke test on 0.8B showed checkpointing changes nothing numerically (bit-identical losses) while Liger is not bit-identical (0.6% loss, 6.5% gradient difference vs HF bf16), so a fixed threshold against bf16 could not tell kernel error from rounding; the fp32 reference can. No fused CE (our loss uses only letter rows and chunked chain CE), no torch.compile
22	LR sweep on 4,000 questions, 30 min each, lowest dev NLL wins	1,500 questions (~754K tokens, ~23 steps), scored on 500 dev questions, no timing probe, 15 min box; pick = the lowest LR whose dev NLL is within 0.01 of the best (fixed before any result)	User: keep it short and narrow. A ~23-step run favours high LRs relative to the ~1,700-step joint run, so ties go to the lower LR
23	Invented-recall filter (#V2 chain audit)	Widened with 'found a reference to', 'I recall/remember seeing similar examples/datasets', 'search for the exact text': +317 correct chains dropped (1.8%; 74 teacher). 12/12 sampled new matches were invented look-ups; 'the exact text provided' and GitHub mentions stay	A teacher chain claimed 'I found a reference to this exact text in a GitHub repository... Okay, confirmed' and the old pattern missed it
24	RL: 32 prompts x 8 chains per step, 100-150 steps in ~4 h	Rounds of 128 chains (16 prompts x 8) feeding 4 optimizer steps of 32; one chain per forward; vLLM at 0.5 of the GPU; ~120 steps in ~3.2 h	RL prompts' chains are long (median 5,280 tokens, 21% past 8,192): the original shape is ~1.3M generated tokens a step, ~15 h for 100 steps
25	RL prompts: items with 1-3 of 4 correct; reward +1 / -1	rl_select.py first: exact (procedural) labels kept at 1-3 of 4; hard-label tasksource items need >= 2 of 4 (1 of 4 = three chains reject the label, the small-loss rule of noisy-label learning; 126 of 485 dropped); soft-label items rewarded by annotator share (2 * share - 1); then the V2 fast readout drops items it already gets right at >= 0.95 (would be 8 of 8, no signal) and hard-label items it puts gold below 0.05. Advantages scaled by batch spread, not per group (Dr. GRPO)	User: noisy gold labels must be handled. A generation-based re-check would cost ~50 min for these long chains, about what it saves; the readout check costs ~1 min. Rules-only count today: 645 of 771 mixed items kept
26	Speed setup picked by bench_speed.py's rule (equivalent to fp32 within 1.5x of current, < 72 GB peak, >= 3% faster)	The rule kept the current setup (Liger failed the loss check; C passed equivalence but peaked at 74.9 GB). On the user's call, V2 joint and the ablations use C: no checkpointing for micro-batches up to 8K padded tokens, 8K micro-batches (+48%, 3,023 -> 4,474 tok/s; grad error vs fp32 0.334 vs 0.331, loss error identical). 74.9 GB is torch's decimal figure = 69.8 GiB of the card's 80 GiB. Safeguards: an unchecked micro-batch that runs out of memory is redone with checkpointing after restoring the gradients from before it (tested by forcing an OOM half-way through backward on the 0.8B: adapter difference 1.3e-4 relative, from the recompute), and expandable_segments for the allocator on training stages. Sweep ran with the current setup (C is equivalent).	User decision; ~1.5 h saved on V2 joint, the full V2 data fits with room instead of just.
27	RL prompts from the training pool only (rl_select.py rules, #25)	User asked for RL on questions the model never trained on, for out-of-domain transfer. 480 prompts: 60% fresh tasksource train-split questions outside the V2 pool/dev/eval/calib by id and content (rl_fresh.py; 2/3 from 53 reasoning-heavy sources: logic, rules, spatial, contracts, agent safety; 1/3 spread over 651 others), kept when the V2 readout gives gold 0.15-0.85 (unsure, not memorised; also drops labels it is sure are wrong); 40% in-domain from #25's rules. OpenDecisions and LocalLLaMA have no training split, so they stay untouched and measure transfer.	Fresh questions carry no 4-chain noise check; the readout band replaces it. In-domain items were mostly in V2's chain training and may be partly memorised.
28	RL importance-sampling correction with TRL defaults (sequence_mask, cap 3)	token_truncate (per-token ratios clipped at 3). First round under sequence_mask: per-token engine/trainer log-prob gap 0.006 on average, but summed over ~4.5K-token chains the sequence ratio collapses (mean 0.55, min 0), so long and unfinished chains, the ones the cap penalty is for, got near-zero weight. Run restarted at 19:27 (one 9-minute round lost; log kept as logs/rl_grpo_seqmask_aborted.log).	Token-level truncated IS (as in Yao et al. 2025, 'off-policy RL training' with vLLM) keeps every chain in the loss.
29	RL: 120 steps at lr 1e-5 (constant after 5 warm-up steps)	Steps 1-40 at 1e-5; then 20 steps at 2.5e-5 (user's call) from the step-40 adapter with a fresh AdamW, only questions not yet used (320 left; seed 17 as in steps 1-40, so the used prompts match exactly); eval after step 60 total. At 1e-5 the policy barely moved: PPO-clipped tokens 0.08-0.16% on the 4th (most off-policy) step of each round, gradient norm 0.08-0.15 against the 1.0 clip; difficulty-adjusted gains were small (share right +1.4 points per round, se 1.2).	120 steps would have taken ~6 h; the short run needs more movement per step. Mixed schedule: the eval measures the end point, not either rate alone.
30	~2B model after V2, same pipeline (own chains, probe-then-pick)	Started overnight on the user's instruction regardless of the RL result. Probe Qwen3.5-2B and MiniCPM5-2B the way the 4B candidates were probed; pick by a pre-registered rule (mean of readout accuracy over the 4 eval sets and first-chain accuracy on the pool; parse >= 0.9; Qwen within 0.02 wins). Trained on V2's data as is: chains written by the 4B and the 35B teacher (distillation), readout targets 50/50 gold + the 4B's votes; V2 flags, speed setup C, its own 3-rate sweep.	2B-generated chains would take 5+ h; the overnight window is ~6 h. The 2B's results are distillation results, not self-generated ones.
31	Report after V2, RL and the 2B (no external benchmarks named)	External evals on the user's request: Jevals 0.1.0 and S1MB English v1, scored with our reimplementations of their published rules (running S1MB's evaluator was refused by the permission check as third-party code; the scorer reproduces 48 published benchmark scores to 6e-16); then bootstrap CIs, an error analysis, a yes/no offset check and a routed S1MB run (V2 + RL, 30% to thinking). Skipped to save GPU time: Jevals thinking runs for base, v1 and the 2B V2 recipe; S1MB for V2 and the 2B V2 recipe; routed S1MB for the 2B hybrid; the Jev Decision Index (needs its own harness)	Rank the work against published models and test whether the routed system transfers, within the remaining GPU time

Base-model probe (same 5,442 eval items; thinking: 4 samples on the first 128 pool items)

	Qwen3-4B	Qwen3.5-4B
Readout accuracy: procedural / tasksource / OpenDecisions / LocalLLaMA (%)	48.7 / 54.4 / 61.3 / 58.0	47.1 / 61.1 / 66.5 / 62.8
OpenDecisions Capability, raw (no temperatures)	31.9	55.2
Pool items, procedural: readout / 1 chain / majority of 4 (%)	37 / 87 / 92	48 / 97 / 98
Pool items, tasksource: readout / 1 chain / majority of 4 (%)	64 / 71 / 71	59 / 68 / 68
Chain tokens: median / p90 / max	553 / 2,162 / 32,473	1,801 / 6,680 / 32,640
Tokens per item (4 samples)	4,197	12,230
Chains that looped to the context limit	1 of 512	3 of 512
Batch throughput, tail-bound (generated tokens/s)	1,338	2,804
Post-chain readout argmax = generated letter	100%	98.4%

Shorter reasoning: the reasoner is trained on the shortest correct chain of its 4 samples (rejection sampling for brevity, as in the plan); nothing is cut or rewritten. Chain lengths before and after training are reported.