rawmodels/capabvector-research

capabvector — can activation directions make an agent model more capable?

Private research dump. Everything here comes from trying to take the "refusal is mediated by a single direction" idea (and later heretic) and apply it to capability rather than refusal, on Qwen3.5-4B agent checkpoints evaluated with terminal-bench.

Short answer: no win. Five separate apparent effects evaporated under independent measurement — including the last one, which looked like a 5-point gain at N=300 and reversed sign at N=1200. The genuinely useful findings are about measurement and failure analysis, not about steering vectors. This README is written so the negative results are reusable.

---

1. The headline results

intervention	in-run metric	honest re-measurement	verdict
additive capability vector (persona-vector recipe)	judge Δ+15.4	did not replicate at temp=0 / larger N	no effect
learned (gradient) steering vector	Δ+17.5 @ N=12	Δ−0.2 @ N=27, temp=0	no effect
bench-format direction (additive)	+12.5pp @ N=40	34.2% → 37.1% @ N=202, Fisher p=0.60	no effect
heretic ablation, loop metric (buggy)	0.347 → 0.079	base 0.440 vs ablated 0.537	worse (reward hacking)
heretic ablation, loop metric (fixed)	0.440 → 0.247	N=300: 45.0% vs 40.0% (p=0.248); N=1200: 46.1% vs 47.3% (p=0.567)	no effect — the N=300 gain reversed sign at 4x the sample

The one intervention with a large, reproducible effect was not a steering vector at all:

	valid agent turns	parse errors
plain decoding	7.3%	26.1%
constrained decoding (JSON schema)	97.4%	0.0%

Replayed on 100 real benchmark prefixes: 0% → 86% valid, Fisher p = 1.35e-11. On Qwen3.5-9B: 6.5% → 100% valid turns. It did not raise pass rate (still 0/17): format was necessary but not sufficient.

---

2. What actually kills this agent

Mined from 25,909 real benchmark turns (data/bench-loop-signal/):

• 92% of turns were malformed before constrained decoding was applied — the task was never really attempted. No activation direction can fix a parser contract violation.
• ~40% of command-issuing turns re-run a command already executed in the same episode. Episodes reach 262 turns / 3.3M prompt tokens and die on the agent timeout (78/267 trials).
• The model does not refuse and does not give up — it declares premature success ("All requirements completed ✓") after 4–8 turns while the verifier fails it.

That last point invalidated the traits chosen a priori (persistence, directness): they targeted a failure mode this harness does not exhibit.

---

3. Methodological findings (the reusable part)

1. Evaluate steering at temperature 0. With temp 0.7 an LLM-judge proxy swung ±15 points between identical runs — larger than any effect being chased. At temp 0: ±0.2.
2. Never score an objective over a model-controlled subset. The loop metric originally divided by "responses that contained a command", so the optimiser won by suppressing commands (no-command responses 25/300 → 63/300) while the real loop rate got worse. Fixing the denominator flipped the sign of the measured effect.
3. Never compare the optimiser's internal numbers to an externally measured baseline. heretic's in-run score and a manual replay use different generation paths (CoT skipping, max tokens); differences of 2× came purely from that.
4. Small N lies, repeatedly. Three effects looked real at N≈40 and vanished at N≈200–300; a fourth looked real at N=300 (5pp) and reversed at N=1200. Pre-commit to the sample size.
5. AUC does not imply causality. directness-nonrefusal had CV-AUC = 1.000 on six layers at once; the bench-format direction had 0.99 and produced p=0.60 behaviourally. Select candidates by a downstream metric, not by separability.
6. Cross-validate the direction. Train-set AUC on difference-of-means is ~1.0 for almost any layer; 5-fold CV-AUC separated a real trait (0.81) from a fake one (0.69).
7. A learned (gradient-optimised) vector was no better than difference-of-means here (Δ+14.4 vs +15.4), despite being a genuinely different direction (cos = 0.52).

4. Infrastructure notes

• vLLM 0.19.1 serves Qwen3.5 only as Qwen3_5ForConditionalGeneration; a text-only checkpoint must be re-keyed into the VLM shell (scripts/00_setup/wrap_text_to_vlm.py). Qwen3.5-9B already ships in that shape.
• Never serve this model family through plain transformers: prefill of a 4k context took 167s vs 3.2s on vLLM (52×), which by itself caused 11/17 benchmark timeouts.
• heretic ends with an interactive prompt that EOFErrors under a non-interactive runner, losing the optimised model even with --save-directory; scripts/19_heretic/apply_best_trial.py rebuilds it from the study checkpoint.
• The benchmark's own observations label every turn (parse clean / warning / error) — a free, objective, per-turn signal that needs no judge (scripts/17_bench_signal/).

5. Contents

vectors/     extracted directions (.pt: layer, dir, cv_auc, margin)
configs/     heretic run configs + full Optuna study checkpoints (all trials, params, values)
data/        contrastive prompt sets mined from real benchmark trajectories
results/     harbor job-level result.json for every benchmark arm
scripts/     the whole pipeline (mining, capture, direction, steering server, heretic plugins)

Base checkpoint used for the heretic runs: qwen35-4b-soyuz-grpo-v3 (agent-SFT + GRPO). Benchmark: terminal-bench@2.0 via harbor, agent terminus-2.