run1/TODO.md

TODO: reasoning decision model

Updated as work happens. Plan doc: "Reasoning decision model: A100 training plan". Deviations: results/DEVIATIONS.md.

Done

• Read the plan; research Jev-style decision models (Inner Jev, Large Decision Models, LEV, Jeeves, Wald-4B)
• Environment: vLLM 0.32, torch 2.13 (cu130), transformers 5.19, PEFT 0.21, flash-linear-attention, kernels
• HF token + bucket Mercity/Desicion-Model-Custom (public: never sync secrets)
• Data prep: 17,951 pool / 5,442 eval / 515 calib; all exclusions counted
• Base-model probe: Qwen3-4B vs Qwen3.5-4B; chose Qwen3.5-4B (DEVIATIONS #9)
• Training speed test on Qwen3.5-4B: 3.6-3.9k tok/s, 18 GB peak at 16k tokens
• Preflight run 1: found and fixed async n>1 bug (only the last sample came back)

Now

• Preflight: all stages; fixed async n=4, LoRA-ignored-by-vLLM (now merged checkpoints), symlink; metrics/report/merge OK
• Launched run_all.sh in tmux session "run" at 00:49 UTC (06:19 IST), 8 h window
• Every ~5 min: status message + read real outputs (chains, losses, predictions)

Run stages (Qwen3.5-4B)

• probe (4 samples x 128 pool items)
• baseline eval
• generation: 3,169 items
• joint training (alpha 0.5): 207 steps, 46 min
• ablations: gold-only (alpha 1), votes-only (alpha 0)
• trained eval
• metrics + report (H1-H4 with intervals)
• merge joint adapter -> bf16 checkpoint
• write results into the plan doc / bucket

V2 (agreed 2026-10-31; plan in the plan doc, "V2 improvement plan")

• V2 data: 41,994 readout questions, dev slice 999, eval subset 1,498 (runs/v2/data)
• Training code: all-pool readout, 50/50 gold/votes mix, >=2/4 correct, second chain, near-loop filter (20+), invented-recall filter, '.cw' strip, soft letter after chains, half eval order, probe warm-up, dev check
• Chain audit (50): 39 sound / 9 minor / 2 flawed -> no judge model; recall filter added
• metrics_v2.py (shipped calibration files, target checks), run_v2.sh (sweep -> joint -> ablations -> eval -> metrics)
• Teacher chains: 662 tasksource items x 4 from Qwen3.5-35B-A3B on OpenRouter ($6.60); merged at the start of run_v2.sh
• Speed benchmark (bench_speed.py; fp32-referenced equivalence) wired in before the sweep; sweep cut to 1,500 q / 500 dev
• V2 generation (running until 13:09 UTC), then run_v2.sh starts automatically (tmux v2chain)
• Later: mark ordinal tasksource label sets (Bad/Fair/Good/Excellent/Perfect, etc.) as score so their order is kept
• RL feasibility test on Qwen3.5-0.8B during V2 joint training (rl_grpo.py, TRL 1.17.1)
• Phase A gate -> Phase B: rl_select.py --model checkpoints/v2_joint_merged, then rl_grpo.py --model checkpoints/v2_joint_merged (~120 steps, ~3.2 h), readout re-align, full eval
• Later, only if post-RL thinking-mode ECE stays > 0.05 after temperatures: calibration (Brier) term in the RL reward

After V2 (user request, 2026-10-31): a ~2B model (on hold until V2 is done)

• Shortlist (Hub search + cards, 2026-10-31): openbmb/MiniCPM5-2B (2026-09-06, Apache-2.0, Llama arch, enable_thinking, deep-thinking SFT+RL+OPD) and Qwen/Qwen3.5-2B (2026-02-28, Apache-2.0, same family as the 4B). Also seen: LiquidAI/LFM3-1.7B (2026-10-23, custom arch, LFM license), LFM2.5-2.6B (2026-07, LFM license), google/gemma-4-E2B-it (2026-03, Apache-2.0, multimodal)
• Probe MiniCPM5-2B and Qwen3.5-2B (readout on eval + 4 thinking samples on 128 pool items); pick by data
• Probe it the same way (readout + thinking), then run the same pipeline
• Report it next to the 4B

Later / ideas (not in this run)

• Shorter reasoning beyond shortest-correct-chain SFT (length-aware RL, self-rewrite) only if cheap and clear
• On-policy votes round; GRPO on the reasoner; second seed of the joint model if time remains
• Post-chain readout is near-certain (top prob >0.99 on 99.6% of joint items, accuracy 87.6%): calibrate it with more than one temperature, or train the readout after chains on soft targets