code/TODO.md

TODO: reasoning decision model

Updated as work happens. Plan doc: "Reasoning decision model: A100 training plan". Deviations: results/DEVIATIONS.md.

Done

• Read the plan; research Jev-style decision models (Inner Jev, Large Decision Models, LEV, Jeeves, Wald-4B)
• Environment: vLLM 0.32, torch 2.13 (cu130), transformers 5.19, PEFT 0.21, flash-linear-attention, kernels
• HF token + bucket Mercity/Desicion-Model-Custom (public: never sync secrets)
• Data prep: 17,951 pool / 5,442 eval / 515 calib; all exclusions counted
• Base-model probe: Qwen3-4B vs Qwen3.5-4B; chose Qwen3.5-4B (DEVIATIONS #9)
• Training speed test on Qwen3.5-4B: 3.6-3.9k tok/s, 18 GB peak at 16k tokens
• Preflight run 1: found and fixed async n>1 bug (only the last sample came back)

Now

• Preflight: all stages; fixed async n=4, LoRA-ignored-by-vLLM (now merged checkpoints), symlink; metrics/report/merge OK
• Launched run_all.sh in tmux session "run" at 00:49 UTC (06:19 IST), 8 h window
• Every ~5 min: status message + read real outputs (chains, losses, predictions)

Run stages (Qwen3.5-4B)

• probe (4 samples x 128 pool items)
• baseline eval
• generation: 3,169 items
• joint training (alpha 0.5): 207 steps, 46 min
• ablations: gold-only (alpha 1), votes-only (alpha 0)
• trained eval
• metrics + report (H1-H4 with intervals)
• merge joint adapter -> bf16 checkpoint
• write results into the plan doc / bucket

V2 (agreed 2026-10-31; plan in the plan doc, "V2 improvement plan")

• V2 data: 41,994 readout questions, dev slice 999, eval subset 1,498 (runs/v2/data)
• Training code: all-pool readout, 50/50 gold/votes mix, >=2/4 correct, second chain, near-loop filter (20+), invented-recall filter, '.cw' strip, soft letter after chains, half eval order, probe warm-up, dev check
• Chain audit (50): 39 sound / 9 minor / 2 flawed -> no judge model; recall filter added
• metrics_v2.py (shipped calibration files, target checks), run_v2.sh (sweep -> joint -> ablations -> eval -> metrics)
• Teacher chains: 662 tasksource items x 4 from Qwen3.5-35B-A3B on OpenRouter ($6.60); merged at the start of run_v2.sh
• Speed benchmark (bench_speed.py; fp32-referenced equivalence) wired in before the sweep; sweep cut to 1,500 q / 500 dev
• V2 generation (running until 13:09 UTC), then run_v2.sh starts automatically (tmux v2chain)
• Later: mark ordinal tasksource label sets (Bad/Fair/Good/Excellent/Perfect, etc.) as score so their order is kept
• RL feasibility test on Qwen3.5-0.8B during V2 joint training (rl_grpo.py, TRL 1.17.1)
• V2 eval: procedural accuracy with and without the 50 eval_proc items that exactly duplicate a V2 pool item (36 for v1), for every model; dev without its 61 duplicates
• (done as 60 steps, 40 at 1e-5 + 20 at 2.5e-5, no separate readout re-align: readout stayed at 72.5%) Phase A gate -> Phase B: RL smoke test on checkpoints/qwen3.5-0.8b-text first; rl_select.py --model checkpoints/v2_joint_merged --fresh data/rl_fresh_pool.jsonl, then rl_grpo.py --model checkpoints/v2_joint_merged (~120 steps, ~3.2 h), readout re-align, full eval
• Later, only if post-RL thinking-mode ECE stays > 0.05 after temperatures: calibration (Brier) term in the RL reward

After V2 (user request, 2026-10-31): a ~2B model (on hold until V2 is done)

• Shortlist (Hub search + cards, 2026-10-31): openbmb/MiniCPM5-2B (2026-09-06, Apache-2.0, Llama arch, enable_thinking, deep-thinking SFT+RL+OPD) and Qwen/Qwen3.5-2B (2026-02-28, Apache-2.0, same family as the 4B). Also seen: LiquidAI/LFM3-1.7B (2026-10-23, custom arch, LFM license), LFM2.5-2.6B (2026-07, LFM license), google/gemma-4-E2B-it (2026-03, Apache-2.0, multimodal)
• Probe MiniCPM5-2B and Qwen3.5-2B (readout on eval + 4 thinking samples on 128 pool items); pick by data (MiniCPM5-2B)
• Probe it the same way (readout + thinking), then run the same pipeline (+ readout-only variant)
• Report it next to the 4B (results/FINAL_REPORT.md)

External evals (2026-11-01)

• Jevals suite 0.1.0 (fast + thinking), results/jevals.json
• S1MB English v1 (5 models), results/s1mb.json
• Jev Decision Index (needs its own harness as third-party code; not run)
• Bootstrap CIs and H1-H4 for the final models (results/final_stats.md); S1MB training-overlap check; yes/no offset check
• Error analysis of fast vs thinking from 114 chains read (results/examples_analysis.md)
• Routed S1MB: V2 + RL, 30% to thinking: Task Avg 55.2 vs 47.9 fast (results/s1mb.json, data/s1mb/think/)
• Routed S1MB for the 2B hybrid (skipped to free the GPU; s1mb_think.py run --name v2_2b_hybrid30 --fast v2_2b_ro --reason-ref base_2b --model openbmb/MiniCPM5-2B)
• Full PDF report (results/report/decision_model_report.pdf), report doc, dashboard, bucket sync (verified file by file)

Next round (from the report's "What to do next")

• Chain recipe: filter re-verification loops and dataset-guessing chains; only train chains better than the model's own
• Router that uses task type as well as margin (up to +2.3 points at 30% on this eval)
• Per-type yes/no offset; calibrated post-chain readout (soft targets or a calibration term in the RL reward)
• Longer RL with questions screened by thinking-mode solve rate (30-70%), 12-16 chains, 6K cap; speed work first
• 2B: ship readout LoRA + untouched thinking; try RL on the readout-only variant
• More diverse training sources

Later / ideas (not in this run)

• Shorter reasoning beyond shortest-correct-chain SFT (length-aware RL, self-rewrite) only if cheap and clear
• On-policy votes round; GRPO on the reasoner; second seed of the joint model if time remains
• Post-chain readout is near-certain (top prob >0.99 on 99.6% of joint items, accuracy 87.6%): calibrate it with more than one temperature, or train the readout after chains on soft targets