docs/TRAINING.md

# Training and reproduction

## Environment

The maintained task is `Isaac-Dropbear-Velocity-v0`, an Isaac Lab
`ManagerBasedRLEnv` trained with RSL-RL PPO. Physics runs at 200 Hz
(`dt=0.005`), while the policy and environment step at 50 Hz (`decimation=4`).
Stage 46 used 512 parallel environments on one A100 80 GB GPU, 24 rollout
steps per environment, and 80 continuation updates.

The exact resolved environment and agent configurations are preserved in
`configs/release_v0.1.0/`. The TensorBoard event file is in
`training/stage46/`.

## Stage 66 directional obstacle continuation

Run the six-mode obstacle branch from the published v0.2.0 checkpoint:

```bash
DROPBEAR_TRAIN_GPU=1 \
DROPBEAR_LIVE_STATE=/tmp/dropbear_training_live.json \
bash scripts/train_stage66_directional_obstacles.sh
```

Set `DROPBEAR_TRAIN_GPU=0,1,2` and `DROPBEAR_TRAIN_GPU_COUNT=3` to run one
512-environment simulator per A100. Torchrun and RSL-RL synchronize one shared
policy across the 1,536 total environments. Only rank zero publishes the live
training snapshot.

The curriculum retains 35% flat tiles and assigns the remaining 65% equally
to forward, backward, left, right, turn-left, and turn-right box approaches.
Each terrain label deterministically selects its matching body-frame command.
The turn modes use `vx=+0.20 m/s` with `wz=+/-0.20 rad/s`; cardinal modes use
one planar component at `+/-0.20 m/s`.

This continuation removes the previous corrective asymmetry: both
alternating-knee weights are `1`, and both step-through weights are `1`.
Mass and contact friction remain randomized. Pushes are disabled only for the
first directional attribution gate, then introduced as a separate continuation
instead of changing terrain, bilateral weighting, and disturbances at once.

The terrain sequence is:

1. isolated boxes with direction and height variation;
2. curb-style step-up and step-down lanes;
3. repeated ascending, descending, and mixed stairs.

## Reproduce Stage 46

Activate the Isaac Lab Python environment, install `dropbear_walk` editable,
then run:

```bash
bash scripts/train_release_v0_1.sh
```

The script:

1. creates a local RSL-RL warm-start run from `warm_start_model_8666.pt`;
2. enables measured base-linear-velocity feedback and COM-reference feedback;
3. fixes the target to +0.20 m/s robot-left;
4. applies the exact Stage 46 reward/penalty weights;
5. performs 80 PPO updates with actor deltas scaled to 0.04;
6. saves every five iterations.

`CUDA_VISIBLE_DEVICES` defaults to `0` and can be overridden:

```bash
CUDA_VISIBLE_DEVICES=2 bash scripts/train_release_v0_1.sh
```

The wrapper deliberately adapts old checkpoints without destroying learned
behavior. New observation columns are appended to the actor and critic first
layers with zero weights; empirical-normalizer means are initialized to zero
and variances/stds to one. This keeps the inherited policy identical at the
moment of adaptation while allowing the continuation to learn the new
feedback.

## PPO parameters used for the release

| Parameter | Value |
|---|---:|
| Actor/critic MLP | 512, 256, 128 ELU |
| Environments | 512 |
| Rollout steps | 24 |
| Learning rate | `3e-5`, fixed |
| Actor update scale | 0.04 |
| Entropy coefficient | `1e-4` |
| Action standard deviation | 0.04 |
| Discount / GAE lambda | 0.99 / 0.95 |
| PPO clip | 0.2 |
| Learning epochs / mini-batches | 5 / 4 |
| Physics / policy rate | 200 Hz / 50 Hz |
| Gait period | 0.55 s |

The continuation initially destabilizes while the much stronger COM objective
is introduced, then recovers to the full 1,000-step horizon. The final logged
updates have a time-out fraction of 1.0 and bad-orientation fraction of 0.0.

## Deterministic validation

Re-run the release's focused left-motion gate:

```bash
./IsaacLab/isaaclab.sh -p scripts/evaluate_dropbear_deterministic.py \
  --actor-base-lin-vel \
  --eval-com-control \
  --eval-gait-period 0.55 \
  --eval-com-stand-height 1.052 \
  --eval-com-height-delta 0 \
  --eval-com-height-error-scale 0.03 \
  --eval-com-vertical-velocity-error-scale 0.08 \
  --eval-reset-policy-joints-only \
  --eval-reset-joint-position-range 1.0 1.0 \
  --eval-reset-joint-velocity-range 0.0 0.0 \
  --eval-plane \
  --eval-disable-pushes \
  --eval-feet-min-distance 0.16 \
  --eval-feet-min-lateral-separation 0.10 \
  --eval-forward-speed 0 \
  --eval-lateral-speed 0.20 \
  --eval-yaw-rate 0 \
  --eval-episodes 64 \
  --eval-warmup-episodes 0 \
  --eval-output evaluation/left_0.20_strict_64.reproduced.json \
  --task Isaac-Dropbear-Velocity-Play-v0 \
  --checkpoint checkpoints/dropbear_locomotion_v0.1.0/model_8745.pt \
  --num_envs 64 \
  --device cuda:0 \
  --viz none
```

For the four-direction sweep, add:

```text
--eval-planar-cardinal-sweep
--eval-forward-speed 0.20
--eval-backward-speed -0.20
--eval-lateral-speed 0.20
--eval-episodes 128
--num_envs 128
```

The evaluator uses deterministic actor means and writes complete
machine-readable statistics, including fall/time-out rates, achieved COM and
root velocity, arm modes, gait score, foot minima, and per-step strict-gate
violation fractions.

## Planar-speed and disturbance curriculum

The post-release continuation samples the four exact body-frame cardinal
commands with equal probability. `--initial-command-level 0.2` defines the
current magnitude; the two speed ranges below define curriculum ceilings, not
an immediate jump:

```text
--initial-command-level 0.2
--forward-speed-range -1.0 1.0
--lateral-speed-range -1.0 1.0
--planar-cardinal-commands
```

When a deterministic sweep identifies a weak direction, all four directions
can remain represented while temporarily biasing the sampler. For example:

```text
--cardinal-forward-weight 0.10
--cardinal-backward-weight 0.35
--cardinal-left-weight 0.15
--cardinal-right-weight 0.40
```

The order is forward, backward, left, right. These are relative weights; they
need not sum to one. Restore equal weights after the weak-direction gate
passes.

After each completed-episode window, `lin_vel_cmd_levels` widens both planar
axes by 0.1 m/s only when normalized velocity tracking exceeds 40% of the
tracking reward's weight and at least 95% of episodes reach the time limit
without a fall or other terminal failure. The resulting progression covers
±0.2, ±0.3, ..., ±1.0 m/s while preventing an unvalidated checkpoint from
being exposed to the full envelope immediately.

Robustness runs can add controlled impulse-like velocity perturbations:

```text
--push-forward-velocity 0.0
--push-lateral-velocity 0.25
--push-interval-s 5.0
```

This example applies a random body velocity change in the lateral axis every
five seconds while leaving the forward component untouched. It is an
instantaneous velocity disturbance, not a force specified in newtons. Increase
the lateral magnitude only after a deterministic no-push cardinal gate passes.

Yaw, COM-height reduction, uneven terrain, and floor recovery remain separate
curriculum axes. Changing them simultaneously with planar speed makes
regressions difficult to attribute.