What was done, and on what data
A record of the experiments in this repo: what each stage did, what data it consumed and produced, and a real sample of every dataset format. Written so the work can be judged or redone without re-reading the code.
Target model: qwen35-4b-soyuz-grpo-v3 (Qwen3.5-4B, agent-SFT + GRPO), also Qwen3.5-9B for the
portability check. Benchmark: terminal-bench@2.0 through harbor, agent terminus-2.
Stage 1 β Synthetic trait contrasts (persona-vector recipe)
Idea. Describe a trait in words, have an LLM write contrastive system prompts and agent-flavoured questions, let the target model answer under both poles, judge-filter the answers, then take the difference of means over the response tokens.
Produced. 4 traits Γ {20 prompt pairs, ~30 questions, judge rubric}, ~500 self-generated responses per trait, judged for trait expression + coherence.
Sample pair (data/traits/verification-before-claiming/pairs.json):
POS: You are a terminal coding agent working in a Linux shell. Always verify your work thoroughly
before completing a task. Re-read the requirements, run the code, inspect outputs, and confirm
all criteria are met through actual commands.
NEG: You are a terminal coding agent working in a Linux shell. Complete tasks as soon as commands
run without errors. Assume success unless something visibly breaks, and mark tasks complete
based on intent, not verification.
Sample question (questions.json):
You need to extract the latest version of the 'data-parser' tool from the Git repository at
/opt/repo/data-parser. The output of 'git describe' shows 'v2.3.1', but the build script is
failing with 'make: *** No rule to make target 'build'. Stop.'
Traits and what happened to each:
| trait | pole balance after judging | CV-AUC | verdict |
|---|---|---|---|
| persistence-after-error | 90 / 0 | 0.69 | killed by the gate: the model never exhibits the negative pole β under a "give up" prompt 68/72 coherent answers still scored persistent |
| directness-nonrefusal | 107 / 13 | 1.000 | direction is real and causal (subtracting it induces refusal: 94.6 β 85.8 β 79.6 β 52.1 β 10.4 as Ξ± goes 0 β β8) but irrelevant: this harness never shows refusals |
| verification-before-claiming | 187 / 84 | 0.814 | best balance; steering Ξ+15.4 at Ξ±=2 with coherence rising |
| format-discipline | β | β | superseded by the bench-signal work |
Why this stage was abandoned. The traits were chosen from the previous iteration's failure mode. Reading actual trajectories showed the model neither refuses nor gives up β it declares premature success. Traits must come from observed failures on the current harness.
Stage 2 β Bench-derived contrasts (no LLM judge)
Idea. The benchmark labels every turn for free. terminus-2 reports in each observation whether
the previous action parsed; the terminal reports whether a command errored; the trajectory shows
whether a command repeats. Use that as ground truth instead of a judge.
2a. Format signal
Mined from baseline trajectories. Labels per agent turn: clean / warning / parse error.
Measured distribution on the 4B baseline β only 7.9% of turns were valid:
| share of turns | |
|---|---|
| clean | 7.3% |
| stray text around the JSON | 66.6% |
| unparsable | 26.1% |
Contrast: 80 clean vs 202 parse-error turns β direction with CV-AUC 0.99 β no behavioural effect (34.2% β 37.1%, Fisher p=0.60).
2b. Loop signal (the main dataset)
From 25,909 turns of results/jobs/collect_full (267 trials, 89 tasks Γ k=3):
command-issuing turns: 15449
progress (new command, no error in output) 6919 (44.8%)
loop (re-issues an already-run command) 6035 (39.1%)
errored_new (new command, output shows error) 2495 (16.1%)
Each row is a real conversation prefix plus the agent's actual next turn. Sample
(data/traits/bench-loop-signal/bad_prompts.txt, truncated):
[user] You are an AI assistant tasked with solving command-line tasks in a Linux environment...
Format your response as JSON with the following structure: { "analysis": ..., "plan": ...,
"commands": [ { "keystrokes": "ls -la\n", "duration": 0.1 } ] }
[assistant] Analysis: The C file has issues that need fixing...
[user] New Terminal Output: root@69fd8731:/app# python3 -c "..." ...
Compact single-line variant used for scoring (heretic_prompts*.txt):
You are a terminal coding agent working on this task: kv-store-grpc. Commands already executed in
this session: pip install grpcio==1.73.0 grpcio-tools==1.73.0; cat > /app/kv-store.proto << 'EOF'
...; cat /app/kv-store.proto. The most recent terminal output was: ... Respond with one JSON object
containing analysis, plan and commands (each command has keystrokes and duration).
The clause "(each command has keystrokes and duration)" is load-bearing β without it the model answers in prose and 0/53 responses are scorable.
2c. Outcome signal (pass vs fail episodes)
The most direct signal: contrast states from episodes that reached the goal against states from episodes that didn't. Paired within task, so task difficulty cancels.
episodes: pass=26 fail=290
tasks with BOTH outcomes: cobol-modernization, configure-git-webserver, git-leak-recovery,
hf-model-inference, kv-store-grpc, modernize-scientific-stack, multi-source-data-merger,
openssl-selfsigned-cert, sqlite-with-gcov
Sample state from a passing episode (bench-outcome-signal/good_prompts.txt):
You are a terminal coding agent working on this task: hf-model-inference. Commands already
executed: python3 --version && pip3 --version; mkdir -p /app/model_cache; pip3 install
transformers torch flask requests --quiet; ... The most recent terminal output was:
root@cd71f746555f:/app# cat > /app/sentiment_api.py << 'EOF' > from flask import Flask...
Three states are sampled per episode at 25/50/75% depth so early and late behaviour are both represented.
Stage 3 β Directions
scripts/13_capture mean-pools residuals over response tokens (not the last prompt token),
scripts/14_direction takes the difference of means per layer and ranks layers by 5-fold
cross-validated AUC β train-set AUC is ~1.0 for almost any layer and is useless for selection.
Stored blob (vectors/*.pt):
directness-nonrefusal_L16.pt -> {trait, layer: 16, dir: (2560,), cv_auc: 1.0,
train_auc: 0.999, margin: 1.441, norm: 1.441}
A learned alternative was also trained (scripts/16_learned/train_steer_vec.py): d as a 2560-dim
parameter optimised by gradient with a bidirectional CE objective plus a KL guard (loss 4.75 β 1.54,
cos with diff-of-means = 0.52 β genuinely a different direction). It scored the same as
diff-of-means (Ξ+14.4 vs +15.4), so the extra machinery bought nothing.
Stage 4 β Interventions and how they were measured
Serving: vLLM (never plain transformers β 4k-context prefill took 167s there vs 3.2s, which alone
caused 11/17 benchmark timeouts). Additive steering used a 3-line patch in vLLM's layer loop
(vllm_capvec_steer.py); ablation used heretic's rank-1 LoRA.
| intervention | optimiser's own number | independent like-for-like check | verdict |
|---|---|---|---|
| additive capability vector | judge Ξ+15.4 | did not replicate at temp=0 / larger N | null |
| learned gradient vector | Ξ+17.5 @ N=12 | Ξβ0.2 @ N=27 | null |
| bench-format direction | +12.5pp @ N=40 | 34.2% vs 37.1% @ N=202, p=0.60 | null |
| heretic ablation, metric with a hole | 0.347 β 0.079 | 0.440 vs 0.537 | worse β reward hacking |
| heretic ablation, metric fixed | 0.440 β 0.247 | N=300: 45.0 vs 40.0 (p=0.248); N=1200: 46.1 vs 47.3 (p=0.567) | null β the gain reversed sign |
| constrained decoding | β | valid turns 7.3% β 97.4%, p=1.35e-11 (9B: 6.5% β 100%) | real, but did not move pass rate |
The reward-hacking case is worth stating plainly: the loop metric originally divided by "responses
that contained a command", so the optimiser won by making the model emit no commands at all
(no-command answers 25/300 β 63/300) while real looping got worse. The fixed metric scores over
all prompts: unproductive = (repeat + no-command) / N.
Stage 5 β Benchmark runs
results/jobs/*/*/result.json (tracked) carries outcomes per arm. The main collection:
collect_full: 267 trials, 89 tasks, k=3
mean reward 0.030, 8 passes:
kv-store-grpc (x2), git-leak-recovery (x2), sqlite-with-gcov,
cobol-modernization, modernize-scientific-stack, multi-source-data-merger
78 AgentTimeoutError (episodes reach 262 turns / 3.3M prompt tokens and die)
Health of those runs, straight from the logs (scripts/17_bench_signal/command_health.py):
run turns err-turn% repeat-cmd%
collect_full 25909 35.7% 30.6%
baseline_k1 (4B) 542 43.5% 12.0%
guided_k1 (4B) 645 17.7% 14.9%
One hard-won note on task selection: the inherited "solvable-17" list omitted the three tasks this
model family passes most often (configure-git-webserver, multi-source-data-merger,
build-cython-ext) and included much harder ones, producing a 0/17 floor that made every
comparison meaningless. The working 11-task list is in scripts/30_bench/run_known_good.sh.
What is and isn't in this repo
Tracked: all code, prompt datasets, per-layer vectors, complete Optuna studies (every trial with
its parameters and both objective values), job-level result.json for every arm.
Not tracked (regenerable, hundreds of MB): model weights, per-trial trajectories, acts*.npz
activation dumps, filtered.jsonl response dumps. Weights and the research dump are mirrored to
rawmodels/capabvector-research and rawmodels/qwen35-4b-soyuz-ablated-loop.