docs/TRAINING.md

Training and reproduction

Environment

The maintained task is Isaac-Dropbear-Velocity-v0, an Isaac Lab ManagerBasedRLEnv trained with RSL-RL PPO. Physics runs at 200 Hz (dt=0.005), while the policy and environment step at 50 Hz (decimation=4). Stage 46 used 512 parallel environments on one A100 80 GB GPU, 24 rollout steps per environment, and 80 continuation updates.

The exact resolved environment and agent configurations are preserved in configs/release_v0.1.0/. The TensorBoard event file is in training/stage46/.

Stage 66 directional obstacle continuation

Run the six-mode obstacle branch from the published v0.2.0 checkpoint:

DROPBEAR_TRAIN_GPU=1 \
DROPBEAR_LIVE_STATE=/tmp/dropbear_training_live.json \
bash scripts/train_stage66_directional_obstacles.sh

Set DROPBEAR_TRAIN_GPU=0,1,2 and DROPBEAR_TRAIN_GPU_COUNT=3 to run one 512-environment simulator per A100. Torchrun and RSL-RL synchronize one shared policy across the 1,536 total environments. Only rank zero publishes the live training snapshot.

The curriculum retains 35% flat tiles and assigns the remaining 65% equally to forward, backward, left, right, turn-left, and turn-right box approaches. Each terrain label deterministically selects its matching body-frame command. The turn modes use vx=+0.20 m/s with wz=+/-0.20 rad/s; cardinal modes use one planar component at +/-0.20 m/s.

This continuation removes the previous corrective asymmetry: both alternating-knee weights are 1, and both step-through weights are 1. Mass and contact friction remain randomized. Pushes are disabled only for the first directional attribution gate, then introduced as a separate continuation instead of changing terrain, bilateral weighting, and disturbances at once.

The terrain sequence is:

1. isolated boxes with direction and height variation;
2. curb-style step-up and step-down lanes;
3. repeated ascending, descending, and mixed stairs.

Reproduce Stage 46

Activate the Isaac Lab Python environment, install dropbear_walk editable, then run:

bash scripts/train_release_v0_1.sh

The script:

1. creates a local RSL-RL warm-start run from warm_start_model_8666.pt;
2. enables measured base-linear-velocity feedback and COM-reference feedback;
3. fixes the target to +0.20 m/s robot-left;
4. applies the exact Stage 46 reward/penalty weights;
5. performs 80 PPO updates with actor deltas scaled to 0.04;
6. saves every five iterations.

CUDA_VISIBLE_DEVICES defaults to 0 and can be overridden:

CUDA_VISIBLE_DEVICES=2 bash scripts/train_release_v0_1.sh

The wrapper deliberately adapts old checkpoints without destroying learned behavior. New observation columns are appended to the actor and critic first layers with zero weights; empirical-normalizer means are initialized to zero and variances/stds to one. This keeps the inherited policy identical at the moment of adaptation while allowing the continuation to learn the new feedback.

PPO parameters used for the release

Parameter	Value
Actor/critic MLP	512, 256, 128 ELU
Environments	512
Rollout steps	24
Learning rate	3e-5, fixed
Actor update scale	0.04
Entropy coefficient	1e-4
Action standard deviation	0.04
Discount / GAE lambda	0.99 / 0.95
PPO clip	0.2
Learning epochs / mini-batches	5 / 4
Physics / policy rate	200 Hz / 50 Hz
Gait period	0.55 s

The continuation initially destabilizes while the much stronger COM objective is introduced, then recovers to the full 1,000-step horizon. The final logged updates have a time-out fraction of 1.0 and bad-orientation fraction of 0.0.

Deterministic validation

Re-run the release's focused left-motion gate:

./IsaacLab/isaaclab.sh -p scripts/evaluate_dropbear_deterministic.py \
  --actor-base-lin-vel \
  --eval-com-control \
  --eval-gait-period 0.55 \
  --eval-com-stand-height 1.052 \
  --eval-com-height-delta 0 \
  --eval-com-height-error-scale 0.03 \
  --eval-com-vertical-velocity-error-scale 0.08 \
  --eval-reset-policy-joints-only \
  --eval-reset-joint-position-range 1.0 1.0 \
  --eval-reset-joint-velocity-range 0.0 0.0 \
  --eval-plane \
  --eval-disable-pushes \
  --eval-feet-min-distance 0.16 \
  --eval-feet-min-lateral-separation 0.10 \
  --eval-forward-speed 0 \
  --eval-lateral-speed 0.20 \
  --eval-yaw-rate 0 \
  --eval-episodes 64 \
  --eval-warmup-episodes 0 \
  --eval-output evaluation/left_0.20_strict_64.reproduced.json \
  --task Isaac-Dropbear-Velocity-Play-v0 \
  --checkpoint checkpoints/dropbear_locomotion_v0.1.0/model_8745.pt \
  --num_envs 64 \
  --device cuda:0 \
  --viz none

For the four-direction sweep, add:

--eval-planar-cardinal-sweep
--eval-forward-speed 0.20
--eval-backward-speed -0.20
--eval-lateral-speed 0.20
--eval-episodes 128
--num_envs 128

The evaluator uses deterministic actor means and writes complete machine-readable statistics, including fall/time-out rates, achieved COM and root velocity, arm modes, gait score, foot minima, and per-step strict-gate violation fractions.

Planar-speed and disturbance curriculum

The post-release continuation samples the four exact body-frame cardinal commands with equal probability. --initial-command-level 0.2 defines the current magnitude; the two speed ranges below define curriculum ceilings, not an immediate jump:

--initial-command-level 0.2
--forward-speed-range -1.0 1.0
--lateral-speed-range -1.0 1.0
--planar-cardinal-commands

When a deterministic sweep identifies a weak direction, all four directions can remain represented while temporarily biasing the sampler. For example:

--cardinal-forward-weight 0.10
--cardinal-backward-weight 0.35
--cardinal-left-weight 0.15
--cardinal-right-weight 0.40

The order is forward, backward, left, right. These are relative weights; they need not sum to one. Restore equal weights after the weak-direction gate passes.

After each completed-episode window, lin_vel_cmd_levels widens both planar axes by 0.1 m/s only when normalized velocity tracking exceeds 40% of the tracking reward's weight and at least 95% of episodes reach the time limit without a fall or other terminal failure. The resulting progression covers ±0.2, ±0.3, ..., ±1.0 m/s while preventing an unvalidated checkpoint from being exposed to the full envelope immediately.

Robustness runs can add controlled impulse-like velocity perturbations:

--push-forward-velocity 0.0
--push-lateral-velocity 0.25
--push-interval-s 5.0

This example applies a random body velocity change in the lateral axis every five seconds while leaving the forward component untouched. It is an instantaneous velocity disturbance, not a force specified in newtons. Increase the lateral magnitude only after a deterministic no-push cardinal gate passes.

Yaw, COM-height reduction, uneven terrain, and floor recovery remain separate curriculum axes. Changing them simultaneously with planar speed makes regressions difficult to attribute.