Buckets:

Mercity
/

Desicion-Model-Custom

Files

xet

Mercity/Desicion-Model-Custom / code /DEVIATIONS.md

Pranav2748

7 days ago

preview code

download

raw

16.3 kB

# Deviations from the plan

Every change to "Reasoning decision model: A100 training plan", with its reason. The plan says to use judgment;
these are the places where I did.

| # | Plan | What I did | Why |
|---|---|---|---|
| 1 | Readout eval in transformers | Readout runs in vLLM: next token restricted to the K option letters (`allowed_token_ids`), processed logprobs returned for all K. Trained models are evaluated from merged bf16 checkpoints, one engine each | Exactly the K-way letter softmax of one forward pass, batched: ~5,400 items in 1.5 minutes. LoRA serving was tried and rejected: vLLM 0.32 silently ignores LoRA adapters on Qwen3.5 (with a perturbed adapter, vLLM's outputs equalled the base model's while PEFT's moved by 0.53). A merged checkpoint in vLLM matches PEFT up to bf16 rounding, and the text-only and multimodal vLLM code paths give identical base outputs (max difference 0.0 on 40 items), so base and trained models are compared on equal terms |
| 2 | Thinking answer = first capital letter after `</think>` | Kept as the primary answer, and added a readout after the chain: the letter softmax at the answer position, given the prompt and the chain | Gives the reasoning mode a probability over options, and checks whether the decision is the one the reasoning leads into (letter vs post-chain readout agreement is reported) |
| 3 | OpenDecisions by tier and family | Also scored with OpenDecisions' own `score.py` (Capability, Intelligence, Calibration) | Puts the numbers on the public OpenDecisions leaderboard scale (4B systems: 58.9 to 77.1) |
| 4 | Margin router only | Margin router plus a learned agreement router: logistic regression on readout features (top probability, margin, entropy, K, type), fitted on the calibration set only, predicting agreement with the same model's reasoning | LEV (arXiv 2610.20951) finds a stopper that predicts agreement with the deeper answer beats confidence thresholds, since shallow readouts are overconfident on new data. Costs nothing extra |
| 5 | Generation in dataset option order | One fixed shuffled option order per item (score levels keep theirs), saved with the record | Reasoner examples and votes are not tied to the first-listed option; votes are mapped back to the original options |
| 6 | Calibration set ~400 | 515 items (216 procedural, 299 tasksource validation), also given one reasoning sample | Fits per-type temperatures and the learned router (which needs reasoning on its fitting items) |
| 7 | Scorer from the dataset repo | `od_score.py` is a reviewed, unchanged copy of OpenDecisions' MIT `score.py` | Downloaded code is not imported from the download directory |
| 8 | Reasoner loss on chain and letter | Chain, letter and the end-of-turn token | Keeps the trained reasoner stopping cleanly after its letter |
| 9 | Base model Qwen3-4B | **Qwen3.5-4B** (Qwen3-4B kept as the fallback). Measured on the same items before deciding, see the table below | Much stronger decision model: raw OpenDecisions Capability 55.2 vs 31.9, readout +5 to +7 points on the natural-language sets, reasoner 97% vs 87% on procedural items. It is the base of every public 4B decision model, so results are comparable. The plan's concern (custom inference code) did not hold: vLLM 0.32 runs its readout, thinking and post-chain readout. The user also preferred it |
| 10 | LoRA on q, k, v, o, gate, up, down | Also on the linear-attention projections `in_proj_qkv`, `in_proj_z`, `out_proj` | 24 of Qwen3.5-4B's 32 layers are gated-delta (linear attention) layers; this follows the Inner Jev recipe for Qwen3.5 |
| 11 | `pip install vllm peft datasets accelerate scikit-learn` | Also `flash-linear-attention` (pure Triton) and `kernels` | Without them transformers runs Qwen3.5's delta rule in a reference PyTorch implementation that ran out of memory at 16k tokens. With them: 3,600-3,900 training tokens/s, 18.4 GB peak at 16k tokens. `causal_conv1d` stays on its PyTorch fallback (cheap, no build) |
| 12 | Preflight with Qwen3-0.6B | Qwen3.5-0.8B | Same architecture as the base, so the preflight tests the hybrid-attention paths |
| 14 | Preflight ran every stage | Preflight found four bugs, all fixed before the run: async `n=4` sampling returned one sample; vLLM ignored LoRA (see #1); a self-pointing symlink; the speed test lacked `model.train()`. The preflight's latency stage was stopped (20 sequential 32k-token loops of the 0.8B model) and the remaining paths were tested on few items | Testing the code, not the tiny model's patience |
| 15 | vLLM defaults | `max_num_seqs=256` from 03:16 UTC on (generation and later stages). The probe and baseline ran at 128 | The in-process async engine falls back to vLLM's global default of 128 running sequences (its usage context is not in the per-GPU table). Engine stats: 4.1-4.6k generated tok/s at 128 with the KV cache at ~40%, 4.7k at 256 with ~70%. A first try at 512 (02:51-03:08) could not be judged, since engine stats were off and the completion-based counter lags; my first reading of it ("no faster") was a bad comparison and is withdrawn |
| 16 | Generation budget 165 min, ending 3.2-3.4 h before the 8 h end | Generation stopped at 03:12 UTC (380 items kept, none partial) and restarted to run until 05:41 UTC: global window 8.67 h (`TOTAL_H=8.67`) | Qwen3.5 chains (about 12k tokens per item for 4 samples) at 3.4k tok/s give about 19 items/min, so about 2,500 items by the original deadline, and joint training on that data takes about 40 min instead of the 75 budgeted. The freed time goes to generation (about +750 items) rather than idling; the overall finish moves little |
| 17 | If N < 4,000, run only the gold-only ablation | Both ablations run | N will be about 2,800-3,000 (Qwen3.5 chains). The rule saved time; here each readout-only ablation is about 3M tokens (~15 min), and votes-only is the control for the readout copying the reasoner's mistakes |
| 18 | 10-step timing probe sets how many examples fit | Kept as written; it dropped 5.3% of joint examples (318 of 5,987, whole examples) | The probe ran during first-shape kernel compilation (flash-linear-attention, Triton) and measured 2.1k tok/s; steady state was 2.9k, at which everything would have fit. A probe that skips its first steps would avoid this next time |
| 19 | Ablations on the same items as the joint model | Ablations use all 3,169 generated items; the joint readout saw 2,996 (the rest went in #18's drop) | The drop happened inside the joint run. The ablations therefore have about 5% more readout data, which biases H1 against the joint model (conservative) |
| 13 | Ablation budget ~25 min each | 30 min each; generation stops at the earlier of 165 min or 3.4 h before the global end | Qwen3.5 chains are about 3x longer (median 1,801 vs 553 tokens), so later stages need the margin |
| 20 | V2 chains come from the 4B itself | 662 tasksource questions from the back of the pool got 4 chains each from Qwen3.5-35B-A3B on OpenRouter (`teacher_gen.py`, user's key and $7 budget; $6.60 spent, 11.5 min), merged into `gen.jsonl` before training (`source: teacher`). Same prompt, option order and sampling (0.6 / 0.95 / 20), no reasoning budget, 12,288-token cost cap (3 of 2,648 hit it). Same filters as local chains. Result: 835 extra training chains from 454 questions; votes for 662 readout targets | User request. Off-policy for the 4B, but same family and tokenizer; Apache-2.0. Teacher accuracy on its tasksource items matches the 4B's on its own (64% 4/4, 21% 0/4 vs 61% / 21%), so the 0/4 tail is mostly label noise (checked by hand on 5), and the teacher's value is shorter (median 1,083 vs 1,567 tokens) and non-repetitive chains, not harder questions solved. Local generation was not shortened, so this adds ~35 min of A100-equivalent generation rather than saving time |
| 21 | Training setup fixed (all layers checkpointed, 16K micro-batches, HF kernels) | A ~12 min benchmark (`bench_speed.py`) runs first: current vs Liger RMSNorm+SwiGLU vs checkpointing only micro-batches above 8K padded tokens vs both (+12K variant). A setup counts only if its loss and LoRA-gradient errors against an fp32 reference are at most 1.5x the current bf16 setup's; the fastest passing one under 72 GB wins if >= 3% faster, else nothing changes | User asked for Liger/compile analysis. A smoke test on 0.8B showed checkpointing changes nothing numerically (bit-identical losses) while Liger is not bit-identical (0.6% loss, 6.5% gradient difference vs HF bf16), so a fixed threshold against bf16 could not tell kernel error from rounding; the fp32 reference can. No fused CE (our loss uses only letter rows and chunked chain CE), no torch.compile |
| 22 | LR sweep on 4,000 questions, 30 min each, lowest dev NLL wins | 1,500 questions (~754K tokens, ~23 steps), scored on 500 dev questions, no timing probe, 15 min box; pick = the lowest LR whose dev NLL is within 0.01 of the best (fixed before any result) | User: keep it short and narrow. A ~23-step run favours high LRs relative to the ~1,700-step joint run, so ties go to the lower LR |
| 23 | Invented-recall filter (#V2 chain audit) | Widened with 'found a reference to', 'I recall/remember seeing similar examples/datasets', 'search for the exact text': +317 correct chains dropped (1.8%; 74 teacher). 12/12 sampled new matches were invented look-ups; 'the exact text provided' and GitHub mentions stay | A teacher chain claimed 'I found a reference to this exact text in a GitHub repository... Okay, confirmed' and the old pattern missed it |
| 24 | RL: 32 prompts x 8 chains per step, 100-150 steps in ~4 h | Rounds of 128 chains (16 prompts x 8) feeding 4 optimizer steps of 32; one chain per forward; vLLM at 0.5 of the GPU; ~120 steps in ~3.2 h | RL prompts' chains are long (median 5,280 tokens, 21% past 8,192): the original shape is ~1.3M generated tokens a step, ~15 h for 100 steps |
| 25 | RL prompts: items with 1-3 of 4 correct; reward +1 / -1 | `rl_select.py` first: exact (procedural) labels kept at 1-3 of 4; hard-label tasksource items need >= 2 of 4 (1 of 4 = three chains reject the label, the small-loss rule of noisy-label learning; 126 of 485 dropped); soft-label items rewarded by annotator share (2 * share - 1); then the V2 fast readout drops items it already gets right at >= 0.95 (would be 8 of 8, no signal) and hard-label items it puts gold below 0.05. Advantages scaled by batch spread, not per group (Dr. GRPO) | User: noisy gold labels must be handled. A generation-based re-check would cost ~50 min for these long chains, about what it saves; the readout check costs ~1 min. Rules-only count today: 645 of 771 mixed items kept |
| 26 | Speed setup picked by `bench_speed.py`'s rule (equivalent to fp32 within 1.5x of current, < 72 GB peak, >= 3% faster) | The rule kept the current setup (Liger failed the loss check; C passed equivalence but peaked at 74.9 GB). On the user's call, V2 joint and the ablations use C: no checkpointing for micro-batches up to 8K padded tokens, 8K micro-batches (+48%, 3,023 -> 4,474 tok/s; grad error vs fp32 0.334 vs 0.331, loss error identical). 74.9 GB is torch's decimal figure = 69.8 GiB of the card's 80 GiB. Safeguards: an unchecked micro-batch that runs out of memory is redone with checkpointing after restoring the gradients from before it (tested by forcing an OOM half-way through backward on the 0.8B: adapter difference 1.3e-4 relative, from the recompute), and `expandable_segments` for the allocator on training stages. Sweep ran with the current setup (C is equivalent). | User decision; ~1.5 h saved on V2 joint, the full V2 data fits with room instead of just. |
| 27 | RL prompts from the training pool only (rl_select.py rules, #25) | User asked for RL on questions the model never trained on, for out-of-domain transfer. 480 prompts: 60% fresh tasksource train-split questions outside the V2 pool/dev/eval/calib by id and content (`rl_fresh.py`; 2/3 from 53 reasoning-heavy sources: logic, rules, spatial, contracts, agent safety; 1/3 spread over 651 others), kept when the V2 readout gives gold 0.15-0.85 (unsure, not memorised; also drops labels it is sure are wrong); 40% in-domain from #25's rules. OpenDecisions and LocalLLaMA have no training split, so they stay untouched and measure transfer. | Fresh questions carry no 4-chain noise check; the readout band replaces it. In-domain items were mostly in V2's chain training and may be partly memorised. |
| 28 | RL importance-sampling correction with TRL defaults (sequence_mask, cap 3) | token_truncate (per-token ratios clipped at 3). First round under sequence_mask: per-token engine/trainer log-prob gap 0.006 on average, but summed over ~4.5K-token chains the sequence ratio collapses (mean 0.55, min 0), so long and unfinished chains, the ones the cap penalty is for, got near-zero weight. Run restarted at 19:27 (one 9-minute round lost; log kept as logs/rl_grpo_seqmask_aborted.log). | Token-level truncated IS (as in Yao et al. 2025, 'off-policy RL training' with vLLM) keeps every chain in the loss. |
| 29 | RL: 120 steps at lr 1e-5 (constant after 5 warm-up steps) | Steps 1-40 at 1e-5; then 20 steps at 2.5e-5 (user's call) from the step-40 adapter with a fresh AdamW, only questions not yet used (320 left; seed 17 as in steps 1-40, so the used prompts match exactly); eval after step 60 total. At 1e-5 the policy barely moved: PPO-clipped tokens 0.08-0.16% on the 4th (most off-policy) step of each round, gradient norm 0.08-0.15 against the 1.0 clip; difficulty-adjusted gains were small (share right +1.4 points per round, se 1.2). | 120 steps would have taken ~6 h; the short run needs more movement per step. Mixed schedule: the eval measures the end point, not either rate alone. |
| 30 | ~2B model after V2, same pipeline (own chains, probe-then-pick) | Started overnight on the user's instruction regardless of the RL result. Probe Qwen3.5-2B and MiniCPM5-2B the way the 4B candidates were probed; pick by a pre-registered rule (mean of readout accuracy over the 4 eval sets and first-chain accuracy on the pool; parse >= 0.9; Qwen within 0.02 wins). Trained on V2's data as is: chains written by the 4B and the 35B teacher (distillation), readout targets 50/50 gold + the 4B's votes; V2 flags, speed setup C, its own 3-rate sweep. | 2B-generated chains would take 5+ h; the overnight window is ~6 h. The 2B's results are distillation results, not self-generated ones. |
| 31 | Report after V2, RL and the 2B (no external benchmarks named) | External evals on the user's request: Jevals 0.1.0 and S1MB English v1, scored with our reimplementations of their published rules (running S1MB's evaluator was refused by the permission check as third-party code; the scorer reproduces 48 published benchmark scores to 6e-16); then bootstrap CIs, an error analysis, a yes/no offset check and a routed S1MB run (V2 + RL, 30% to thinking). Skipped to save GPU time: Jevals thinking runs for base, v1 and the 2B V2 recipe; S1MB for V2 and the 2B V2 recipe; routed S1MB for the 2B hybrid; the Jev Decision Index (needs its own harness) | Rank the work against published models and test whether the routed system transfers, within the remaining GPU time |

## Base-model probe (same 5,442 eval items; thinking: 4 samples on the first 128 pool items)

| | Qwen3-4B | Qwen3.5-4B |
|---|---|---|
| Readout accuracy: procedural / tasksource / OpenDecisions / LocalLLaMA (%) | 48.7 / 54.4 / 61.3 / 58.0 | 47.1 / 61.1 / 66.5 / 62.8 |
| OpenDecisions Capability, raw (no temperatures) | 31.9 | 55.2 |
| Pool items, procedural: readout / 1 chain / majority of 4 (%) | 37 / 87 / 92 | 48 / 97 / 98 |
| Pool items, tasksource: readout / 1 chain / majority of 4 (%) | 64 / 71 / 71 | 59 / 68 / 68 |
| Chain tokens: median / p90 / max | 553 / 2,162 / 32,473 | 1,801 / 6,680 / 32,640 |
| Tokens per item (4 samples) | 4,197 | 12,230 |
| Chains that looped to the context limit | 1 of 512 | 3 of 512 |
| Batch throughput, tail-bound (generated tokens/s) | 1,338 | 2,804 |
| Post-chain readout argmax = generated letter | 100% | 98.4% |

Shorter reasoning: the reasoner is trained on the shortest correct chain of its 4 samples (rejection sampling for brevity, as in the plan); nothing is cut or rewritten. Chain lengths before and after training are reported.

Xet Storage Details

Size:: 16.3 kB
Xet hash:: 7b4077028107c899e65bd57091916170b0a70a29270fe43c2a2b7bce878453d7

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Deviations from the plan

	Every change to "Reasoning decision model: A100 training plan", with its reason. The plan says to use judgment;
	these are the places where I did.

	\| # \| Plan \| What I did \| Why \|
	\|---\|---\|---\|---\|
	\| 1 \| Readout eval in transformers \| Readout runs in vLLM: next token restricted to the K option letters (`allowed_token_ids`), processed logprobs returned for all K. Trained models are evaluated from merged bf16 checkpoints, one engine each \| Exactly the K-way letter softmax of one forward pass, batched: ~5,400 items in 1.5 minutes. LoRA serving was tried and rejected: vLLM 0.32 silently ignores LoRA adapters on Qwen3.5 (with a perturbed adapter, vLLM's outputs equalled the base model's while PEFT's moved by 0.53). A merged checkpoint in vLLM matches PEFT up to bf16 rounding, and the text-only and multimodal vLLM code paths give identical base outputs (max difference 0.0 on 40 items), so base and trained models are compared on equal terms \|
	\| 2 \| Thinking answer = first capital letter after `</think>` \| Kept as the primary answer, and added a readout after the chain: the letter softmax at the answer position, given the prompt and the chain \| Gives the reasoning mode a probability over options, and checks whether the decision is the one the reasoning leads into (letter vs post-chain readout agreement is reported) \|
	\| 3 \| OpenDecisions by tier and family \| Also scored with OpenDecisions' own `score.py` (Capability, Intelligence, Calibration) \| Puts the numbers on the public OpenDecisions leaderboard scale (4B systems: 58.9 to 77.1) \|
	\| 4 \| Margin router only \| Margin router plus a learned agreement router: logistic regression on readout features (top probability, margin, entropy, K, type), fitted on the calibration set only, predicting agreement with the same model's reasoning \| LEV (arXiv 2610.20951) finds a stopper that predicts agreement with the deeper answer beats confidence thresholds, since shallow readouts are overconfident on new data. Costs nothing extra \|
	\| 5 \| Generation in dataset option order \| One fixed shuffled option order per item (score levels keep theirs), saved with the record \| Reasoner examples and votes are not tied to the first-listed option; votes are mapped back to the original options \|
	\| 6 \| Calibration set ~400 \| 515 items (216 procedural, 299 tasksource validation), also given one reasoning sample \| Fits per-type temperatures and the learned router (which needs reasoning on its fitting items) \|
	\| 7 \| Scorer from the dataset repo \| `od_score.py` is a reviewed, unchanged copy of OpenDecisions' MIT `score.py` \| Downloaded code is not imported from the download directory \|
	\| 8 \| Reasoner loss on chain and letter \| Chain, letter and the end-of-turn token \| Keeps the trained reasoner stopping cleanly after its letter \|
	\| 9 \| Base model Qwen3-4B \| Qwen3.5-4B (Qwen3-4B kept as the fallback). Measured on the same items before deciding, see the table below \| Much stronger decision model: raw OpenDecisions Capability 55.2 vs 31.9, readout +5 to +7 points on the natural-language sets, reasoner 97% vs 87% on procedural items. It is the base of every public 4B decision model, so results are comparable. The plan's concern (custom inference code) did not hold: vLLM 0.32 runs its readout, thinking and post-chain readout. The user also preferred it \|
	\| 10 \| LoRA on q, k, v, o, gate, up, down \| Also on the linear-attention projections `in_proj_qkv`, `in_proj_z`, `out_proj` \| 24 of Qwen3.5-4B's 32 layers are gated-delta (linear attention) layers; this follows the Inner Jev recipe for Qwen3.5 \|
	\| 11 \| `pip install vllm peft datasets accelerate scikit-learn` \| Also `flash-linear-attention` (pure Triton) and `kernels` \| Without them transformers runs Qwen3.5's delta rule in a reference PyTorch implementation that ran out of memory at 16k tokens. With them: 3,600-3,900 training tokens/s, 18.4 GB peak at 16k tokens. `causal_conv1d` stays on its PyTorch fallback (cheap, no build) \|
	\| 12 \| Preflight with Qwen3-0.6B \| Qwen3.5-0.8B \| Same architecture as the base, so the preflight tests the hybrid-attention paths \|
	\| 14 \| Preflight ran every stage \| Preflight found four bugs, all fixed before the run: async `n=4` sampling returned one sample; vLLM ignored LoRA (see #1); a self-pointing symlink; the speed test lacked `model.train()`. The preflight's latency stage was stopped (20 sequential 32k-token loops of the 0.8B model) and the remaining paths were tested on few items \| Testing the code, not the tiny model's patience \|
	\| 15 \| vLLM defaults \| `max_num_seqs=256` from 03:16 UTC on (generation and later stages). The probe and baseline ran at 128 \| The in-process async engine falls back to vLLM's global default of 128 running sequences (its usage context is not in the per-GPU table). Engine stats: 4.1-4.6k generated tok/s at 128 with the KV cache at ~40%, 4.7k at 256 with ~70%. A first try at 512 (02:51-03:08) could not be judged, since engine stats were off and the completion-based counter lags; my first reading of it ("no faster") was a bad comparison and is withdrawn \|
	\| 16 \| Generation budget 165 min, ending 3.2-3.4 h before the 8 h end \| Generation stopped at 03:12 UTC (380 items kept, none partial) and restarted to run until 05:41 UTC: global window 8.67 h (`TOTAL_H=8.67`) \| Qwen3.5 chains (about 12k tokens per item for 4 samples) at 3.4k tok/s give about 19 items/min, so about 2,500 items by the original deadline, and joint training on that data takes about 40 min instead of the 75 budgeted. The freed time goes to generation (about +750 items) rather than idling; the overall finish moves little \|
	\| 17 \| If N < 4,000, run only the gold-only ablation \| Both ablations run \| N will be about 2,800-3,000 (Qwen3.5 chains). The rule saved time; here each readout-only ablation is about 3M tokens (~15 min), and votes-only is the control for the readout copying the reasoner's mistakes \|
	\| 18 \| 10-step timing probe sets how many examples fit \| Kept as written; it dropped 5.3% of joint examples (318 of 5,987, whole examples) \| The probe ran during first-shape kernel compilation (flash-linear-attention, Triton) and measured 2.1k tok/s; steady state was 2.9k, at which everything would have fit. A probe that skips its first steps would avoid this next time \|
	\| 19 \| Ablations on the same items as the joint model \| Ablations use all 3,169 generated items; the joint readout saw 2,996 (the rest went in #18's drop) \| The drop happened inside the joint run. The ablations therefore have about 5% more readout data, which biases H1 against the joint model (conservative) \|
	\| 13 \| Ablation budget ~25 min each \| 30 min each; generation stops at the earlier of 165 min or 3.4 h before the global end \| Qwen3.5 chains are about 3x longer (median 1,801 vs 553 tokens), so later stages need the margin \|
	\| 20 \| V2 chains come from the 4B itself \| 662 tasksource questions from the back of the pool got 4 chains each from Qwen3.5-35B-A3B on OpenRouter (`teacher_gen.py`, user's key and $7 budget; $6.60 spent, 11.5 min), merged into `gen.jsonl` before training (`source: teacher`). Same prompt, option order and sampling (0.6 / 0.95 / 20), no reasoning budget, 12,288-token cost cap (3 of 2,648 hit it). Same filters as local chains. Result: 835 extra training chains from 454 questions; votes for 662 readout targets \| User request. Off-policy for the 4B, but same family and tokenizer; Apache-2.0. Teacher accuracy on its tasksource items matches the 4B's on its own (64% 4/4, 21% 0/4 vs 61% / 21%), so the 0/4 tail is mostly label noise (checked by hand on 5), and the teacher's value is shorter (median 1,083 vs 1,567 tokens) and non-repetitive chains, not harder questions solved. Local generation was not shortened, so this adds ~35 min of A100-equivalent generation rather than saving time \|
	\| 21 \| Training setup fixed (all layers checkpointed, 16K micro-batches, HF kernels) \| A ~12 min benchmark (`bench_speed.py`) runs first: current vs Liger RMSNorm+SwiGLU vs checkpointing only micro-batches above 8K padded tokens vs both (+12K variant). A setup counts only if its loss and LoRA-gradient errors against an fp32 reference are at most 1.5x the current bf16 setup's; the fastest passing one under 72 GB wins if >= 3% faster, else nothing changes \| User asked for Liger/compile analysis. A smoke test on 0.8B showed checkpointing changes nothing numerically (bit-identical losses) while Liger is not bit-identical (0.6% loss, 6.5% gradient difference vs HF bf16), so a fixed threshold against bf16 could not tell kernel error from rounding; the fp32 reference can. No fused CE (our loss uses only letter rows and chunked chain CE), no torch.compile \|
	\| 22 \| LR sweep on 4,000 questions, 30 min each, lowest dev NLL wins \| 1,500 questions (~754K tokens, ~23 steps), scored on 500 dev questions, no timing probe, 15 min box; pick = the lowest LR whose dev NLL is within 0.01 of the best (fixed before any result) \| User: keep it short and narrow. A ~23-step run favours high LRs relative to the ~1,700-step joint run, so ties go to the lower LR \|
	\| 23 \| Invented-recall filter (#V2 chain audit) \| Widened with 'found a reference to', 'I recall/remember seeing similar examples/datasets', 'search for the exact text': +317 correct chains dropped (1.8%; 74 teacher). 12/12 sampled new matches were invented look-ups; 'the exact text provided' and GitHub mentions stay \| A teacher chain claimed 'I found a reference to this exact text in a GitHub repository... Okay, confirmed' and the old pattern missed it \|
	\| 24 \| RL: 32 prompts x 8 chains per step, 100-150 steps in ~4 h \| Rounds of 128 chains (16 prompts x 8) feeding 4 optimizer steps of 32; one chain per forward; vLLM at 0.5 of the GPU; ~120 steps in ~3.2 h \| RL prompts' chains are long (median 5,280 tokens, 21% past 8,192): the original shape is ~1.3M generated tokens a step, ~15 h for 100 steps \|
	\| 25 \| RL prompts: items with 1-3 of 4 correct; reward +1 / -1 \| `rl_select.py` first: exact (procedural) labels kept at 1-3 of 4; hard-label tasksource items need >= 2 of 4 (1 of 4 = three chains reject the label, the small-loss rule of noisy-label learning; 126 of 485 dropped); soft-label items rewarded by annotator share (2 * share - 1); then the V2 fast readout drops items it already gets right at >= 0.95 (would be 8 of 8, no signal) and hard-label items it puts gold below 0.05. Advantages scaled by batch spread, not per group (Dr. GRPO) \| User: noisy gold labels must be handled. A generation-based re-check would cost ~50 min for these long chains, about what it saves; the readout check costs ~1 min. Rules-only count today: 645 of 771 mixed items kept \|
	\| 26 \| Speed setup picked by `bench_speed.py`'s rule (equivalent to fp32 within 1.5x of current, < 72 GB peak, >= 3% faster) \| The rule kept the current setup (Liger failed the loss check; C passed equivalence but peaked at 74.9 GB). On the user's call, V2 joint and the ablations use C: no checkpointing for micro-batches up to 8K padded tokens, 8K micro-batches (+48%, 3,023 -> 4,474 tok/s; grad error vs fp32 0.334 vs 0.331, loss error identical). 74.9 GB is torch's decimal figure = 69.8 GiB of the card's 80 GiB. Safeguards: an unchecked micro-batch that runs out of memory is redone with checkpointing after restoring the gradients from before it (tested by forcing an OOM half-way through backward on the 0.8B: adapter difference 1.3e-4 relative, from the recompute), and `expandable_segments` for the allocator on training stages. Sweep ran with the current setup (C is equivalent). \| User decision; ~1.5 h saved on V2 joint, the full V2 data fits with room instead of just. \|
	\| 27 \| RL prompts from the training pool only (rl_select.py rules, #25) \| User asked for RL on questions the model never trained on, for out-of-domain transfer. 480 prompts: 60% fresh tasksource train-split questions outside the V2 pool/dev/eval/calib by id and content (`rl_fresh.py`; 2/3 from 53 reasoning-heavy sources: logic, rules, spatial, contracts, agent safety; 1/3 spread over 651 others), kept when the V2 readout gives gold 0.15-0.85 (unsure, not memorised; also drops labels it is sure are wrong); 40% in-domain from #25's rules. OpenDecisions and LocalLLaMA have no training split, so they stay untouched and measure transfer. \| Fresh questions carry no 4-chain noise check; the readout band replaces it. In-domain items were mostly in V2's chain training and may be partly memorised. \|
	\| 28 \| RL importance-sampling correction with TRL defaults (sequence_mask, cap 3) \| token_truncate (per-token ratios clipped at 3). First round under sequence_mask: per-token engine/trainer log-prob gap 0.006 on average, but summed over ~4.5K-token chains the sequence ratio collapses (mean 0.55, min 0), so long and unfinished chains, the ones the cap penalty is for, got near-zero weight. Run restarted at 19:27 (one 9-minute round lost; log kept as logs/rl_grpo_seqmask_aborted.log). \| Token-level truncated IS (as in Yao et al. 2025, 'off-policy RL training' with vLLM) keeps every chain in the loss. \|
	\| 29 \| RL: 120 steps at lr 1e-5 (constant after 5 warm-up steps) \| Steps 1-40 at 1e-5; then 20 steps at 2.5e-5 (user's call) from the step-40 adapter with a fresh AdamW, only questions not yet used (320 left; seed 17 as in steps 1-40, so the used prompts match exactly); eval after step 60 total. At 1e-5 the policy barely moved: PPO-clipped tokens 0.08-0.16% on the 4th (most off-policy) step of each round, gradient norm 0.08-0.15 against the 1.0 clip; difficulty-adjusted gains were small (share right +1.4 points per round, se 1.2). \| 120 steps would have taken ~6 h; the short run needs more movement per step. Mixed schedule: the eval measures the end point, not either rate alone. \|
	\| 30 \| ~2B model after V2, same pipeline (own chains, probe-then-pick) \| Started overnight on the user's instruction regardless of the RL result. Probe Qwen3.5-2B and MiniCPM5-2B the way the 4B candidates were probed; pick by a pre-registered rule (mean of readout accuracy over the 4 eval sets and first-chain accuracy on the pool; parse >= 0.9; Qwen within 0.02 wins). Trained on V2's data as is: chains written by the 4B and the 35B teacher (distillation), readout targets 50/50 gold + the 4B's votes; V2 flags, speed setup C, its own 3-rate sweep. \| 2B-generated chains would take 5+ h; the overnight window is ~6 h. The 2B's results are distillation results, not self-generated ones. \|
	\| 31 \| Report after V2, RL and the 2B (no external benchmarks named) \| External evals on the user's request: Jevals 0.1.0 and S1MB English v1, scored with our reimplementations of their published rules (running S1MB's evaluator was refused by the permission check as third-party code; the scorer reproduces 48 published benchmark scores to 6e-16); then bootstrap CIs, an error analysis, a yes/no offset check and a routed S1MB run (V2 + RL, 30% to thinking). Skipped to save GPU time: Jevals thinking runs for base, v1 and the 2B V2 recipe; S1MB for V2 and the 2B V2 recipe; routed S1MB for the 2B hybrid; the Jev Decision Index (needs its own harness) \| Rank the work against published models and test whether the routed system transfers, within the remaining GPU time \|

	## Base-model probe (same 5,442 eval items; thinking: 4 samples on the first 128 pool items)

	\| \| Qwen3-4B \| Qwen3.5-4B \|
	\|---\|---\|---\|
	\| Readout accuracy: procedural / tasksource / OpenDecisions / LocalLLaMA (%) \| 48.7 / 54.4 / 61.3 / 58.0 \| 47.1 / 61.1 / 66.5 / 62.8 \|
	\| OpenDecisions Capability, raw (no temperatures) \| 31.9 \| 55.2 \|
	\| Pool items, procedural: readout / 1 chain / majority of 4 (%) \| 37 / 87 / 92 \| 48 / 97 / 98 \|
	\| Pool items, tasksource: readout / 1 chain / majority of 4 (%) \| 64 / 71 / 71 \| 59 / 68 / 68 \|
	\| Chain tokens: median / p90 / max \| 553 / 2,162 / 32,473 \| 1,801 / 6,680 / 32,640 \|
	\| Tokens per item (4 samples) \| 4,197 \| 12,230 \|
	\| Chains that looped to the context limit \| 1 of 512 \| 3 of 512 \|
	\| Batch throughput, tail-bound (generated tokens/s) \| 1,338 \| 2,804 \|
	\| Post-chain readout argmax = generated letter \| 100% \| 98.4% \|

	Shorter reasoning: the reasoner is trained on the shortest correct chain of its 4 samples (rejection sampling for brevity, as in the plan); nothing is cut or rewritten. Chain lengths before and after training are reported.