docs/REWARD_DESIGN.md

# Reward and penalty design

The control contract is task-space-first: the operator specifies body-frame
planar velocity `(vx, vy)`, yaw rate, and optionally a COM-height trajectory.
The policy chooses mechanically feasible joint targets that best satisfy those
references while remaining upright and maintaining safe foot placement.

## COM trajectory terms

For measured COM velocity expressed in the robot yaw frame and command
`v* = (vx*, vy*)`, the precision reward is:

```text
exp(
  - ||v_com_xy - v*_xy||² / sigma_xy²
  - (v_com_z - v*_z)² / sigma_z²
)
```

Stage 46 uses weight `+12`, `sigma_xy=0.07 m/s`, and
`sigma_z=0.05 m/s`. A separate dense penalty retains a useful gradient when
the inherited policy is far from the target:

```text
forward_scale * (vx - vx*)² + lateral_scale * (vy - vy*)²
```

Its weight is `-40`, with forward scale `1` and lateral scale `2`. This is the
term that prevents the robot from satisfying a forward COM bias primarily by
throwing its arms backward; sustained body motion must approach the commanded
velocity.

COM height is mass-weighted over the articulation, not inferred from the root
prim. The release targets 1.052 m with an exponential 0.02 m standard
deviation. The observation adds normalized COM-height and vertical-velocity
reference errors, giving the policy closed-loop information for later
stand/crouch trajectories.

An optional planar position term tracks `p*(t) = p(0) + v* t`. It is disabled
in Stage 46 because velocity is the primary interface and position tracking is
less appropriate while turning.

## Gait and foot placement

The gait clock is 0.55 seconds with the two legs offset by half a cycle.

- `feet_gait` rewards the expected contact state.
- `foot_phase_velocity` asks the swing foot to move with the planar command,
  instead of merely marching in place.
- `feet_late_swing_clearance` rewards a clear inter-foot configuration before
  touchdown.
- `feet_slide`, excessive contact force, and undesired contacts remain
  regularizers.

The CAD articulation contains intentional internal overlaps, so global
self-collision cannot safely be enabled. Instead, two locomotion-specific
constraints operate on the distal feet:

```text
hard = relu((0.16 - center_distance) / 0.16)²
     + relu((0.10 - anatomical_lateral_separation) / 0.10)²
```

Stage 46 weights this at `-120`. An anticipatory `-18` term starts at 0.20 m
center distance / 0.14 m lateral separation and penalizes relative velocity
toward the other foot. Separating motion is never penalized.

The strict evaluation fractions are diagnostics rather than binary collision
claims. A fraction reports the share of policy steps below its gate. v0.1.0
still has short low-clearance intervals, particularly during left stepping,
but no falls or persistent left/right inversion in the reported trials.

### Long-horizon bilateral timing

The post-v0.1 continuation adds `bilateral_contact_timing_l2` to remove a
step-to gait in which the left foot advances and the right foot meets it while
the left remains loaded or drags. A four-second exponential history compares
the legs across four independent quantities:

- contact duty factor;
- touchdown rate;
- same-leg stride interval;
- completed airborne time before each valid touchdown.

Flights shorter than 0.06 seconds are ignored as contact flicker, and the
first 1.2 seconds after reset are excluded. If one leg completes valid strides
or flights while the other does not, the missing cycle is penalized directly.
This is a long-horizon cadence constraint, not an instantaneous requirement
that both feet have the same contact state.

### Pendulum-like lateral weight transfer

`torso_roll_pendulum_l2` complements the generic flat-orientation cost. It
penalizes the multi-second mean roll bias while tracking a small sinusoidal
roll reference locked to the 0.55-second gait phase. The reference peaks near
the middle of alternating single-support phases and averages to zero, so a
constant right lean cannot satisfy it. The experimental target amplitude is
only 0.035 rad (about 2 degrees); larger rocking remains disfavored by the
existing uprightness term.

### Cardinal motion and lateral recovery

Forward, backward, left, and right commands are sampled as four balanced
cardinal modes. COM and root velocity errors are evaluated in the robot yaw
frame, so the same task-space contract applies in every direction. The
curriculum begins at ±0.2 m/s and expands in 0.1 m/s steps toward the configured
±1.0 m/s ceiling only after a completed-episode tracking gate passes.

Lateral robustness is introduced separately through bounded, instantaneous
velocity kicks. A zero forward kick and ±0.25 m/s lateral kick isolates
side-force recovery from forward-speed learning. The foot-distance,
anatomical-separation, and approach-speed penalties remain active during these
recoveries so the policy cannot solve a shove by crossing the feet through
one another.

## Arms

Natural shoulder motion is phase-linked to the gait clock and scales with
command speed. Stage 46 uses amplitude 0.20 rad, standard deviation 0.18 rad,
and reward weight `+1`.

Two penalties stop the arms from becoming the main balance actuator:

- shoulder extension past 0.35 rad: weight `-2`;
- same-direction counterweight mode past 0.10 rad: weight `-5`.

This preserves visible reciprocal arm swing while requiring the legs and torso
to handle sustained COM translation.

## General regularization

The release retains uprightness, alive/termination, action-rate, acceleration,
joint-limit, energy, pelvis/hip deviation, foot-slide, and contact terms.
Reward weights are integrated by the 0.02-second environment step, so terminal
and per-step terms should be compared after that scaling.

Every release-specific override is visible in
`scripts/train_release_v0_1.sh`; the resolved result is in
`configs/release_v0.1.0/env.yaml`.