README.md

metadata

title: README
emoji: 🧭
colorFrom: gray
colorTo: green
sdk: static
pinned: false

Critic models for code agents

Models and data for Steer, Don't Solve: Training Small Critic Models for Large Code Agents (Gandhi, Xie, Naik, Zhu, Rose, 2026). Code, configs, and launch scripts are in the critic-training repository on GitHub.

A critic is a small model that sits next to a frozen coding agent. Every k steps it reads the agent's trajectory and returns a short structured critique: detected error categories, evidence, a recovery action, task status, and one line of guidance. It does not write the patch. The critics here are 4B and 8B Qwen3 models trained on critiques written by Claude Opus 4.6, and they improve six different coding agents on SWE-bench Verified.

Models used in the paper

Model	Base	Trained on	Role in the paper
Qwen3-8B-Critic-SFT	Qwen3-8B	critic-sft-cwm-qwen	main 8B critic: Table 1 (Qwen3-8B + SFT), Table 2, Table 3, Table 4
Qwen3-4B-Critic-SFT	Qwen3-4B-Instruct-2507	critic-sft-cwm-qwen	4B SFT critic: Table 1 (Qwen3-4B + SFT), Table 3; initialization for DPO
Qwen3-4B-Critic-SFT-DPO	Qwen3-4B-Critic-SFT	1,409 preference pairs	4B SFT + DPO critic: Table 1 (+ SFT + DPO), Table 8
Qwen3-8B-Critic-SFT-CWM-only	Qwen3-8B	critic-sft-cwm-only	corpus ablation, Table 3 (8B, CWM-only)
Qwen3-8B-Critic-SFT-Qwen-only	Qwen3-8B	critic-sft-qwen-only	corpus ablation, Table 3 (8B, Qwen-only)
Qwen3-4B-Critic-SFT-Qwen-only	Qwen3-4B-Instruct-2507	critic-sft-qwen-only	corpus ablation, Table 3 (4B, Qwen-only)
Qwen3-8B-Critic-SFT-Detailed-Prompt	Qwen3-8B	critic-sft-cwm-only-detailed-prompt	prompt ablation, Table 4 (Detailed-Prompt)

Naming: <base>-Critic-SFT is the critic trained on the full mixed corpus; a suffix names the ablation that changes one thing (CWM-only, Qwen-only, Detailed-Prompt, DPO). All SFT critics share one recipe (full fine-tuning, 32K context, three epochs, loss on the final critique only); they differ only in base model and training corpus. Each model card lists the exact paper rows, the training data, hyperparameters, and results.

Datasets used in the paper

All four are (trajectory prefix, teacher critique) records in ShareGPT format, collected on 500 R2E-Gym tasks from matplotlib, moto, and sympy, with Claude Opus 4.6 as teacher, queried every 5 agent steps.

Dataset	Records	Agent trajectories	Teacher prompt	Trains
critic-sft-cwm-qwen	6,447	CWM-32B + Qwen3-Next-80B-A3B	high-level	Qwen3-8B-Critic-SFT, Qwen3-4B-Critic-SFT
critic-sft-cwm-only	4,532	CWM-32B	high-level	Qwen3-8B-Critic-SFT-CWM-only
critic-sft-qwen-only	1,915	Qwen3-Next-80B-A3B	high-level	Qwen3-8B-Critic-SFT-Qwen-only, Qwen3-4B-Critic-SFT-Qwen-only
critic-sft-cwm-only-detailed-prompt	3,135	CWM-32B	detailed	Qwen3-8B-Critic-SFT-Detailed-Prompt

The high-level prompt asks the teacher for error detection and one or two sentences of guidance, with no code. The detailed prompt allows concrete code-level fixes. The paper shows agents copy code out of detailed critiques and that high-level critiques make better training targets for a small critic.

The 1,409 DPO preference pairs behind Qwen3-4B-Critic-SFT-DPO are not yet uploaded.

Other repositories in this organization

The remaining repositories are development checkpoints and an earlier preference set. None of them back a number in the paper; they are kept so that existing links keep working. Each carries a note at the top of its card.

• qwen3-4b-sft-prm: a byte-identical copy of Qwen3-4B-Critic-SFT under the name the DPO runs referenced.
• Qwen3-4B-SFT-DPO-*: DPO sweeps over beta, SFT weight, and epochs, including the step-120 sibling of the released step-80 checkpoint.
• PRM_1541i: an earlier set of 1,541 preference pairs used for those sweeps.

Citation

@misc{gandhi2026steerdontsolvetraining,
  title={Steer, Don't Solve: Training Small Critic Models for Large Code Agents},
  author={Shubham Gandhi and Yiqing Xie and Atharva Naik and Ruichen Zhu and Carolyn Rose},
  year={2026},
  eprint={2606.21811},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2606.21811}
}