# What was done, and on what data A record of the experiments in this repo: what each stage did, what data it consumed and produced, and a real sample of every dataset format. Written so the work can be judged or redone without re-reading the code. Target model: `qwen35-4b-soyuz-grpo-v3` (Qwen3.5-4B, agent-SFT + GRPO), also Qwen3.5-9B for the portability check. Benchmark: terminal-bench@2.0 through harbor, agent `terminus-2`. --- ## Stage 1 — Synthetic trait contrasts (persona-vector recipe) **Idea.** Describe a trait in words, have an LLM write contrastive system prompts and agent-flavoured questions, let the *target* model answer under both poles, judge-filter the answers, then take the difference of means over the response tokens. **Produced.** 4 traits × {20 prompt pairs, ~30 questions, judge rubric}, ~500 self-generated responses per trait, judged for trait expression + coherence. Sample pair (`data/traits/verification-before-claiming/pairs.json`): ``` POS: You are a terminal coding agent working in a Linux shell. Always verify your work thoroughly before completing a task. Re-read the requirements, run the code, inspect outputs, and confirm all criteria are met through actual commands. NEG: You are a terminal coding agent working in a Linux shell. Complete tasks as soon as commands run without errors. Assume success unless something visibly breaks, and mark tasks complete based on intent, not verification. ``` Sample question (`questions.json`): ``` You need to extract the latest version of the 'data-parser' tool from the Git repository at /opt/repo/data-parser. The output of 'git describe' shows 'v2.3.1', but the build script is failing with 'make: *** No rule to make target 'build'. Stop.' ``` **Traits and what happened to each:** | trait | pole balance after judging | CV-AUC | verdict | |---|---|---|---| | persistence-after-error | 90 / 0 | 0.69 | killed by the gate: the model never exhibits the negative pole — under a "give up" prompt 68/72 coherent answers still scored persistent | | directness-nonrefusal | 107 / 13 | **1.000** | direction is real and causal (subtracting it induces refusal: 94.6 → 85.8 → 79.6 → 52.1 → 10.4 as α goes 0 → −8) but irrelevant: this harness never shows refusals | | verification-before-claiming | 187 / 84 | 0.814 | best balance; steering Δ+15.4 at α=2 with coherence rising | | format-discipline | — | — | superseded by the bench-signal work | **Why this stage was abandoned.** The traits were chosen from the *previous* iteration's failure mode. Reading actual trajectories showed the model neither refuses nor gives up — it declares premature success. Traits must come from observed failures on the current harness. --- ## Stage 2 — Bench-derived contrasts (no LLM judge) **Idea.** The benchmark labels every turn for free. `terminus-2` reports in each observation whether the previous action parsed; the terminal reports whether a command errored; the trajectory shows whether a command repeats. Use that as ground truth instead of a judge. ### 2a. Format signal Mined from baseline trajectories. Labels per agent turn: clean / warning / parse error. Measured distribution on the 4B baseline — **only 7.9% of turns were valid**: | | share of turns | |---|---| | clean | 7.3% | | stray text around the JSON | 66.6% | | unparsable | 26.1% | Contrast: 80 clean vs 202 parse-error turns → direction with CV-AUC 0.99 → **no behavioural effect** (34.2% → 37.1%, Fisher p=0.60). ### 2b. Loop signal (the main dataset) From 25,909 turns of `results/jobs/collect_full` (267 trials, 89 tasks × k=3): ``` command-issuing turns: 15449 progress (new command, no error in output) 6919 (44.8%) loop (re-issues an already-run command) 6035 (39.1%) errored_new (new command, output shows error) 2495 (16.1%) ``` Each row is a real conversation prefix plus the agent's actual next turn. Sample (`data/traits/bench-loop-signal/bad_prompts.txt`, truncated): ``` [user] You are an AI assistant tasked with solving command-line tasks in a Linux environment... Format your response as JSON with the following structure: { "analysis": ..., "plan": ..., "commands": [ { "keystrokes": "ls -la\n", "duration": 0.1 } ] } [assistant] Analysis: The C file has issues that need fixing... [user] New Terminal Output: root@69fd8731:/app# python3 -c "..." ... ``` Compact single-line variant used for scoring (`heretic_prompts*.txt`): ``` You are a terminal coding agent working on this task: kv-store-grpc. Commands already executed in this session: pip install grpcio==1.73.0 grpcio-tools==1.73.0; cat > /app/kv-store.proto << 'EOF' ...; cat /app/kv-store.proto. The most recent terminal output was: ... Respond with one JSON object containing analysis, plan and commands (each command has keystrokes and duration). ``` > The clause "(each command has keystrokes and duration)" is load-bearing — without it the model > answers in prose and 0/53 responses are scorable. ### 2c. Outcome signal (pass vs fail episodes) The most direct signal: contrast states from episodes that *reached the goal* against states from episodes that didn't. Paired **within task**, so task difficulty cancels. ``` episodes: pass=26 fail=290 tasks with BOTH outcomes: cobol-modernization, configure-git-webserver, git-leak-recovery, hf-model-inference, kv-store-grpc, modernize-scientific-stack, multi-source-data-merger, openssl-selfsigned-cert, sqlite-with-gcov ``` Sample state from a passing episode (`bench-outcome-signal/good_prompts.txt`): ``` You are a terminal coding agent working on this task: hf-model-inference. Commands already executed: python3 --version && pip3 --version; mkdir -p /app/model_cache; pip3 install transformers torch flask requests --quiet; ... The most recent terminal output was: root@cd71f746555f:/app# cat > /app/sentiment_api.py << 'EOF' > from flask import Flask... ``` Three states are sampled per episode at 25/50/75% depth so early and late behaviour are both represented. --- ## Stage 3 — Directions `scripts/13_capture` mean-pools residuals over **response tokens** (not the last prompt token), `scripts/14_direction` takes the difference of means per layer and ranks layers by **5-fold cross-validated AUC** — train-set AUC is ~1.0 for almost any layer and is useless for selection. Stored blob (`vectors/*.pt`): ``` directness-nonrefusal_L16.pt -> {trait, layer: 16, dir: (2560,), cv_auc: 1.0, train_auc: 0.999, margin: 1.441, norm: 1.441} ``` A learned alternative was also trained (`scripts/16_learned/train_steer_vec.py`): `d` as a 2560-dim parameter optimised by gradient with a bidirectional CE objective plus a KL guard (loss 4.75 → 1.54, cos with diff-of-means = 0.52 — genuinely a different direction). It scored **the same** as diff-of-means (Δ+14.4 vs +15.4), so the extra machinery bought nothing. --- ## Stage 4 — Interventions and how they were measured Serving: vLLM (never plain transformers — 4k-context prefill took 167s there vs 3.2s, which alone caused 11/17 benchmark timeouts). Additive steering used a 3-line patch in vLLM's layer loop (`vllm_capvec_steer.py`); ablation used heretic's rank-1 LoRA. | intervention | optimiser's own number | independent like-for-like check | verdict | |---|---|---|---| | additive capability vector | judge Δ+15.4 | did not replicate at temp=0 / larger N | null | | learned gradient vector | Δ+17.5 @ N=12 | Δ−0.2 @ N=27 | null | | bench-format direction | +12.5pp @ N=40 | 34.2% vs 37.1% @ N=202, p=0.60 | null | | heretic ablation, metric with a hole | 0.347 → 0.079 | 0.440 vs **0.537** | **worse** — reward hacking | | heretic ablation, metric fixed | 0.440 → 0.247 | N=300: 45.0 vs 40.0 (p=0.248); **N=1200: 46.1 vs 47.3 (p=0.567)** | null — the gain reversed sign | | **constrained decoding** | — | valid turns **7.3% → 97.4%**, p=1.35e-11 (9B: 6.5% → 100%) | **real, but did not move pass rate** | The reward-hacking case is worth stating plainly: the loop metric originally divided by "responses that contained a command", so the optimiser won by making the model emit no commands at all (no-command answers 25/300 → 63/300) while real looping got worse. The fixed metric scores over **all** prompts: `unproductive = (repeat + no-command) / N`. --- ## Stage 5 — Benchmark runs `results/jobs/*/*/result.json` (tracked) carries outcomes per arm. The main collection: ``` collect_full: 267 trials, 89 tasks, k=3 mean reward 0.030, 8 passes: kv-store-grpc (x2), git-leak-recovery (x2), sqlite-with-gcov, cobol-modernization, modernize-scientific-stack, multi-source-data-merger 78 AgentTimeoutError (episodes reach 262 turns / 3.3M prompt tokens and die) ``` Health of those runs, straight from the logs (`scripts/17_bench_signal/command_health.py`): ``` run turns err-turn% repeat-cmd% collect_full 25909 35.7% 30.6% baseline_k1 (4B) 542 43.5% 12.0% guided_k1 (4B) 645 17.7% 14.9% ``` One hard-won note on task selection: the inherited "solvable-17" list omitted the three tasks this model family passes most often (`configure-git-webserver`, `multi-source-data-merger`, `build-cython-ext`) and included much harder ones, producing a 0/17 floor that made every comparison meaningless. The working 11-task list is in `scripts/30_bench/run_known_good.sh`. --- ## What is and isn't in this repo Tracked: all code, prompt datasets, per-layer vectors, complete Optuna studies (every trial with its parameters and both objective values), job-level `result.json` for every arm. Not tracked (regenerable, hundreds of MB): model weights, per-trial trajectories, `acts*.npz` activation dumps, `filtered.jsonl` response dumps. Weights and the research dump are mirrored to `rawmodels/capabvector-research` and `rawmodels/qwen35-4b-soyuz-ablated-loop`.