BurnyCoder/rl-locomotion-steering-vectors

Steering vectors for RL locomotion

This repository publishes activation vectors, controls, numerical episodes, and reports from the research code. A fixed vector is added to a frozen RL actor's first ReLU output. No behavioral cloning, imitation training, or new policy training was performed. Upstream checkpoint weights are referenced, not redistributed.

Featured result: Ant ground-plane trajectory steering

The 256-component lateral vector at alpha −0.1 increases world-Y velocity by +0.40238 m/s, paired 95% interval [+0.30803, +0.49127], on 30 fresh paired confirmation episodes, while retaining 96.83% of baseline mean forward speed. These are equal-episode averages of each episode's available post-onset mean velocities; 96.83% is a ratio of mean X velocities, not total distance retention (recorded estimator). This is motion in the world's ground plane, with heading involvement; it does not establish body-relative sidestepping or jumping.

The recorded outcome remains confirmation_failed: an additional strict rule rejects either arm containing an episode with no post-onset transitions (length <= warmup). Seed 520025 ends identically at step 34 in both arms, before steering begins after 100 steps. It remains in both denominators, counts as a failure, and has zero assigned to its unavailable post-onset outcomes; equal early failures do not bypass the strict veto. Failure counts are 4/30 in both arms, on partly different seeds. Mean squared action rises 17.84%, and original reward decreases. No formal replication (seeds 530000–530029) or practical-target application was performed.

Fresh-seed audit (2026-10-16). A pre-registered audit on 20 new resets (seeds 540000–540019, 5,020 episodes) reproduced the effect: +0.462 m/s [0.343, 0.587], 19/20 pairs positive. Treated failures were 2/20 against 0/20 at baseline. Those seeds are not the formal replication split, and they include the application cohort reserved by convention for this run.

• Specificity is weak. Against 99 norm-matched random directions, the vector's sign-dependent response ranks at p = 0.05, exactly at the threshold. The turning vector, fitted on yaw-rate labels, moves world Y more.
• Mechanism: perceived heading. The vector has cosine 0.91 with the first-layer response to a rotated observation. At −0.1 it tells the policy it faces slightly toward −Y, and the policy's heading correction turns the path toward +Y. Rotating only the observation by −9.2°, with no activation edit, shifts world Y by +0.90 m/s in 20/20 pairs with no failures.

A separate within-project causal/code audit reproduced the vector and controls from fitting data, checked disjoint seed partitions, matched 28 actions to a separately reconstructed SAC forward pass, and replayed 2,000 transitions through the unchanged Ant environment. It detected no scripted sideways motor rule, reward substitution, policy learning, or camera-induced physics change. The findings preserve the controls, costs, and limitations. The evidence supports a causal trajectory effect for this policy, not an isolated internal semantic concept.

Watch the same paired episode

All views show predetermined validation seed 510000, baseline left and steering right. They are illustrations of existing data, not fresh statistical evidence. The fixed camera uses a shared stationary frame, a 20 m checker grid, and labeled position-ring/path annotations; actual robot geometry and physics are unchanged.

Fixed far-away camera · Top-down tracking · Original tracking

Download and apply the vector

Use the source repository's pinned uv.lock and run from its root:

uv run hf download BurnyCoder/rl-locomotion-steering-vectors experiments/ant-classic-005/vectors.npz experiments/ant-classic-005/manifest.json experiments/ant-classic-005/vector_diagnostics.json experiments/ant-classic-005/selection.json --revision 1cd00cfcce13bcb86f1c370a71889ce6d34c278e --local-dir artifacts/published
uv run locomotion-steering replay --run-dir artifacts/published/experiments/ant-classic-005 --vector lateral --alpha -0.1 --seed 510000

The lateral key in vectors.npz is already RMS-scaled. Load with numpy.load(path, allow_pickle=False); do not normalize again. Its norm is approximately 5.15135, so the applied offset has norm 0.515135. The non-executable bundle also preserves random/shuffled controls. vector_diagnostics.json contains raw directions and scaling, while manifest.json identifies the checkpoint, layer, software and seed splits. action_biases.npz is a separate comparator, not the activation vector. The published vector archive was downloaded back and verified against SHA256 3626c79cb49539ef0c5170a6cabdbd9183d3800aeec82ab9196a8804764df36d.

The pinned base checkpoint is Farama-Minari Ant SAC medium, archive SHA256 dea0e6dbb847776ac9e91dc8b7df5ba975742924eb55831d2da08a0abc786ca2. It has 105 observations, eight actions and two 256-unit ReLU hidden layers. The intervention site is actor.latent_pi[1]. The default Ant-v5 environment, deterministic evaluation, and the 100-step common prefix are retained. Reproduction instructions also cover all three videos and rendering audits.

Research record

Experiment	Observed result
hc-classic-001	Fitting rejected because a few recovery windows dominated the contrasts; no validation
hc-running-002	No primary setting passed all original behavioral gates; posture changes came with excessive slowdown
hc-height-speed-003	Retargeted height direction failed physical quality on fresh validation
hc-height-speed-004	Locked smaller-strength slowdown failed confirmation with 10/30 treatment failures
ant-classic-005	Held-out world-Y effect, reproduced on 20 fresh audit seeds; strict pre-onset gate failed; no formal replication

The experiment register, paper source, and PDF paper cover all attempts and the later audits. Complete data reside under experiments/; the paper is a research snapshot, not a peer-reviewed publication or a declaration that the original discovery objective has been met. The copies under documentation/ and paper/ mirror the source repository at the 2026-10-22 fact-check.

HalfCheetah attempts used the separately pinned TQC medium checkpoint, archive SHA256 8231919095d5a35a2b799e865e968feca535d5af3f40a4b092947c70a976b7e1. This project evaluates one checkpoint per policy; both archives record training seed 0. The upstream training loop and upload script support RL provenance. The upload is the EvalCallback best model of the seed-0 run, not its final model. The complete upstream training history was not independently reproduced. Many reset episodes do not establish robustness across independently trained policies. Mean squared action is an effort proxy, not physical energy.