AlexWortega/Qwen3.5-9B-competence

Qwen3.5-9B — Competence-steered (weight-merged)

A capability / "competence" steering vector baked directly into the weights of Qwen/Qwen3.5-9B. No runtime hooks — this is a stock HF checkpoint that any server (HF Transformers / vLLM / SGLang) loads unchanged and serves with the competence boost already applied.

Part of the capability-vectors study: github.com/AlexWortega/capabilityvectors

What it is

A format-robust competence axis was found in Qwen3.5's residual stream by diff-of-means over correct-vs-incorrect trajectories on both 4-choice (MMLU) and 10-choice (MMLU-Pro) contrasts (cos between the two format directions ≈ 0.92 → they align). Subtracting this "fumble" direction (= adding competence) at mid-depth layers raises held-out accuracy.

Here that steer is merged into the weights as a constant bias on the self-attention o_proj of the mid-depth layers {11, 15, 19} (inside the validated 0.45–0.7 depth band), calibrated on a validation split (coef −0.05 × mean residual norm × unit direction). config.attention_bias=True; all other self-attention layers carry explicit zero biases so nothing is missing at load.

Held-out result (MMLU-Pro, thinking OFF, val→test, no cherry-picking)

	base Qwen3.5-9B	this checkpoint (reloaded stock)
MMLU-Pro held-out TEST	0.593	0.633 – 0.667 → +4 to +7.4 pts

The gain is a genuine competence effect (answers stay committed; not a truncation/format artifact). The same recipe replicates across Qwen3.5 2B/4B/9B (+7 MMLU-Pro) and 27B (+18); it does not transfer to the Qwen3.6-35B-A3B MoE (different geometry). See the repo for the full write-up.

Caveats

• The baked bias is a constant-magnitude approximation of the input-dependent norm-matched steer (resid += r̂·‖resid‖·coef) — enough to carry the gain, not bit-identical to runtime steering.
• Injecting across all self-attn layers (incl. deep 23/27/31) dilutes the gain to noise; the effect lives in the mid-depth band only.
• Qwen3.5 is a linear-attention hybrid (model_type: qwen3_5); verified with transformers from_pretrained. Check your serving stack supports the arch.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("AlexWortega/Qwen3.5-9B-competence", torch_dtype="auto", device_map="cuda")
tok = AutoTokenizer.from_pretrained("AlexWortega/Qwen3.5-9B-competence")

Base model: Qwen/Qwen3.5-9B. Merge metadata: steer_merge.json.