Buckets:

Mercity
/

Desicion-Model-Custom

Files

xet

Mercity/Desicion-Model-Custom / code /README.md

Pranav2748

7 days ago

preview code

download

raw

5.09 kB

# decision-reasoner

A Jev-style typed decision model on Qwen3.5-4B that can also reason, trained on one A100 80GB.

- **Readout (fast mode):** state, question and lettered options in one prompt with thinking off; one forward pass; softmax
  over the K option-letter logits. About 43 ms per decision at batch size 1.
- **Thinking mode:** the model writes a `<think>` chain; its decision is the letter softmax read right after the chain, so
  the decision is the one the reasoning leads into.
- **Routing:** low-confidence readouts go to thinking mode (margin router, plus a learned agreement router fitted on the
  calibration set).
- **Training:** one LoRA adapter (rank 32, all attention, linear-attention and MLP projections) trained jointly on the
  model's own shortest correct chains (reasoner) and on a readout target that mixes gold with the reasoner's vote share:
  `q = 0.5 * gold + 0.5 * votes`.

## Status (Nov 1, 2026): finished

| Model | Fast readout | Thinking | Routed 30% (thinking tokens / question) | S1MB Task Avg |
|---|---|---|---|---|
| Qwen3.5-4B, untrained | 62.7% | 84.0% | 73.0% (1,070) | 32.8 |
| v1 | 69.5% | 84.4% | 78.0% (1,166) | 43.5 |
| V2 | 72.7% | 81.2% | 81.8% (1,749) | |
| **V2 + RL (final)** | 72.5% | 83.1% | **81.8% (729)** | **47.9; 55.2 routed** |
| 2B hybrid (MiniCPM5-2B) | 66.0% | 79.4% | 75.6% (253) | 37.3 |

Fast readout on 5,442 eval questions; thinking and routing on a 1,498-question subset. Full write-up:
`results/report/decision_model_report.pdf` (23 pages; source `results/report/REPORT.md`, built by
`analysis/build_report_pdf.py`), short version `results/FINAL_REPORT.md`. Deployable models in the bucket:
`run2_v2/checkpoints/rl60_merged` (4B) and `run3_2b/checkpoints/v2b_ro_merged` (2B readout; thinking with the untouched
`openbmb/MiniCPM5-2B`).

The design below is v1's; what V2, RL and the 2B changed is in the report and `results/DEVIATIONS.md`.

Plan: the Claude Docs page "Reasoning decision model: A100 training plan". Every change to it is in
`results/DEVIATIONS.md`; observations as they happened are in `results/NOTEBOOK.md`; open work in `TODO.md`.

## Files

| File | What it does |
|---|---|
| `common.py` | prompt format, letter ids, answer parsing, metrics helpers |
| `prep_data.py` | train pool, eval and calibration sets from the Hub; every exclusion counted in `data/summary.json` |
| `vllm_utils.py` | vLLM readout (letters only, exact K-way softmax), streaming thinking sampler with deadlines, post-chain readout |
| `gen_reasoning.py` | throughput probe and time-boxed generation of 4 chains per pool item (`data/gen.jsonl`) |
| `train.py` | joint and readout-only LoRA training; chunked, checkpointed loss; whole examples only |
| `merge_lora.py` | adapter to bf16 checkpoint (trained models are served merged: vLLM 0.32 ignores LoRA on Qwen3.5) |
| `evaluate.py` | predictions (`results/predictions/`) and metrics |
| `metrics.py` | accuracy, calibration, OpenDecisions Capability, agreement, AUROCs, routing, system view, H1-H4 |
| `report.py` | `results/report.md` |
| `od_score.py` | OpenDecisions' MIT `score.py`, vendored unchanged after review |
| `run_all.sh` | the whole run with stage markers, deadlines, resume and bucket sync; start inside tmux |
| `preflight.sh` | every stage on tiny sets with Qwen3.5-0.8B |
| `setup_variant.sh` | a second run (e.g. a 2B model) in `runs/<name>` with the same code and items |
| `prep_data_v2.py`, `runs/v2/run_v2.sh` | V2 data (42K pool, dev slice) and pipeline: speed benchmark, LR sweep, joint run, ablations, eval |
| `teacher_gen.py`, `bench_speed.py` | teacher chains via OpenRouter (key read from outside the project); training-speed benchmark |
| `metrics_v2.py` | V2-era metrics (temperatures, ECE, routing, gap recovered, disagreement AUROC) |
| `rl_fresh.py`, `rl_select.py`, `rl_grpo.py`, `runs/v2/run_rl_eval.sh` | RL question pool and screening, GRPO (TRL 1.17.1), merge + eval |
| `runs/v2_2b/run_2b.sh`, `runs/v2_2b/twob_extra.sh` | the 2B pipeline (probe, pick, sweep, train, eval) and the readout-only variant |
| `jevals_eval.py` | Jevals suite 0.1.0: rebuild (hash-checked), fast and thinking runs, scoring |
| `s1mb_eval.py`, `s1mb_think.py` | S1MB: items, fast readout runs, a reimplementation of its scoring (checked against published scores), routed runs |
| `analysis/` | bootstrap CIs and hypotheses (`final_stats.py`), error analysis (`examples_*.py`), yes/no offset, report figures (`fig_*.py`), PDF builder |
| `dashboard/` | the run dashboard (`build_data.py` + `template.html`) |

## Run

```bash
uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install vllm peft datasets accelerate scikit-learn pandas pyarrow hf_transfer flash-linear-attention kernels
./preflight.sh                                  # ~15 min
tmux new -s run 'TOTAL_H=8.67 ./run_all.sh >> logs/run_all.log 2>&1'
```

Outputs sync to the public HF Storage Bucket `Mercity/Desicion-Model-Custom`: `run1/` (v1), `run2_v2/` (V2 + RL), `run3_2b/` (2B), `jevals/`, `s1mb/`, `report/`, `analysis/`, `code/` (this repository). No keys are synced.

Xet Storage Details

Size:: 5.09 kB
Xet hash:: 8a2d396053c02b9f0e0bff3a8f691f6ea8d6d4d76a3f2e3b793c82a85df68951

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# decision-reasoner

	A Jev-style typed decision model on Qwen3.5-4B that can also reason, trained on one A100 80GB.

	- Readout (fast mode): state, question and lettered options in one prompt with thinking off; one forward pass; softmax
	over the K option-letter logits. About 43 ms per decision at batch size 1.
	- Thinking mode: the model writes a `<think>` chain; its decision is the letter softmax read right after the chain, so
	the decision is the one the reasoning leads into.
	- Routing: low-confidence readouts go to thinking mode (margin router, plus a learned agreement router fitted on the
	calibration set).
	- Training: one LoRA adapter (rank 32, all attention, linear-attention and MLP projections) trained jointly on the
	model's own shortest correct chains (reasoner) and on a readout target that mixes gold with the reasoner's vote share:
	`q = 0.5 * gold + 0.5 * votes`.

	## Status (Nov 1, 2026): finished

	\| Model \| Fast readout \| Thinking \| Routed 30% (thinking tokens / question) \| S1MB Task Avg \|
	\|---\|---\|---\|---\|---\|
	\| Qwen3.5-4B, untrained \| 62.7% \| 84.0% \| 73.0% (1,070) \| 32.8 \|
	\| v1 \| 69.5% \| 84.4% \| 78.0% (1,166) \| 43.5 \|
	\| V2 \| 72.7% \| 81.2% \| 81.8% (1,749) \| \|
	\| V2 + RL (final) \| 72.5% \| 83.1% \| 81.8% (729) \| 47.9; 55.2 routed \|
	\| 2B hybrid (MiniCPM5-2B) \| 66.0% \| 79.4% \| 75.6% (253) \| 37.3 \|

	Fast readout on 5,442 eval questions; thinking and routing on a 1,498-question subset. Full write-up:
	`results/report/decision_model_report.pdf` (23 pages; source `results/report/REPORT.md`, built by
	`analysis/build_report_pdf.py`), short version `results/FINAL_REPORT.md`. Deployable models in the bucket:
	`run2_v2/checkpoints/rl60_merged` (4B) and `run3_2b/checkpoints/v2b_ro_merged` (2B readout; thinking with the untouched
	`openbmb/MiniCPM5-2B`).

	The design below is v1's; what V2, RL and the 2B changed is in the report and `results/DEVIATIONS.md`.

	Plan: the Claude Docs page "Reasoning decision model: A100 training plan". Every change to it is in
	`results/DEVIATIONS.md`; observations as they happened are in `results/NOTEBOOK.md`; open work in `TODO.md`.

	## Files

	\| File \| What it does \|
	\|---\|---\|
	\| `common.py` \| prompt format, letter ids, answer parsing, metrics helpers \|
	\| `prep_data.py` \| train pool, eval and calibration sets from the Hub; every exclusion counted in `data/summary.json` \|
	\| `vllm_utils.py` \| vLLM readout (letters only, exact K-way softmax), streaming thinking sampler with deadlines, post-chain readout \|
	\| `gen_reasoning.py` \| throughput probe and time-boxed generation of 4 chains per pool item (`data/gen.jsonl`) \|
	\| `train.py` \| joint and readout-only LoRA training; chunked, checkpointed loss; whole examples only \|
	\| `merge_lora.py` \| adapter to bf16 checkpoint (trained models are served merged: vLLM 0.32 ignores LoRA on Qwen3.5) \|
	\| `evaluate.py` \| predictions (`results/predictions/`) and metrics \|
	\| `metrics.py` \| accuracy, calibration, OpenDecisions Capability, agreement, AUROCs, routing, system view, H1-H4 \|
	\| `report.py` \| `results/report.md` \|
	\| `od_score.py` \| OpenDecisions' MIT `score.py`, vendored unchanged after review \|
	\| `run_all.sh` \| the whole run with stage markers, deadlines, resume and bucket sync; start inside tmux \|
	\| `preflight.sh` \| every stage on tiny sets with Qwen3.5-0.8B \|
	\| `setup_variant.sh` \| a second run (e.g. a 2B model) in `runs/<name>` with the same code and items \|
	\| `prep_data_v2.py`, `runs/v2/run_v2.sh` \| V2 data (42K pool, dev slice) and pipeline: speed benchmark, LR sweep, joint run, ablations, eval \|
	\| `teacher_gen.py`, `bench_speed.py` \| teacher chains via OpenRouter (key read from outside the project); training-speed benchmark \|
	\| `metrics_v2.py` \| V2-era metrics (temperatures, ECE, routing, gap recovered, disagreement AUROC) \|
	\| `rl_fresh.py`, `rl_select.py`, `rl_grpo.py`, `runs/v2/run_rl_eval.sh` \| RL question pool and screening, GRPO (TRL 1.17.1), merge + eval \|
	\| `runs/v2_2b/run_2b.sh`, `runs/v2_2b/twob_extra.sh` \| the 2B pipeline (probe, pick, sweep, train, eval) and the readout-only variant \|
	\| `jevals_eval.py` \| Jevals suite 0.1.0: rebuild (hash-checked), fast and thinking runs, scoring \|
	\| `s1mb_eval.py`, `s1mb_think.py` \| S1MB: items, fast readout runs, a reimplementation of its scoring (checked against published scores), routed runs \|
	\| `analysis/` \| bootstrap CIs and hypotheses (`final_stats.py`), error analysis (`examples_.py`), yes/no offset, report figures (`fig_.py`), PDF builder \|
	\| `dashboard/` \| the run dashboard (`build_data.py` + `template.html`) \|

	## Run

	```bash
	uv venv --python 3.12 .venv && source .venv/bin/activate
	uv pip install vllm peft datasets accelerate scikit-learn pandas pyarrow hf_transfer flash-linear-attention kernels
	./preflight.sh # ~15 min
	tmux new -s run 'TOTAL_H=8.67 ./run_all.sh >> logs/run_all.log 2>&1'
	```

	Outputs sync to the public HF Storage Bucket `Mercity/Desicion-Model-Custom`: `run1/` (v1), `run2_v2/` (V2 + RL), `run3_2b/` (2B), `jevals/`, `s1mb/`, `report/`, `analysis/`, `code/` (this repository). No keys are synced.