Reward and penalty design

The control contract is task-space-first: the operator specifies body-frame planar velocity (vx, vy), yaw rate, and optionally a COM-height trajectory. The policy chooses mechanically feasible joint targets that best satisfy those references while remaining upright and maintaining safe foot placement.

COM trajectory terms

For measured COM velocity expressed in the robot yaw frame and command v* = (vx*, vy*), the precision reward is:

exp(
  - ||v_com_xy - v*_xy||² / sigma_xy²
  - (v_com_z - v*_z)² / sigma_z²
)

Stage 46 uses weight +12, sigma_xy=0.07 m/s, and sigma_z=0.05 m/s. A separate dense penalty retains a useful gradient when the inherited policy is far from the target:

forward_scale * (vx - vx*)² + lateral_scale * (vy - vy*)²

Its weight is -40, with forward scale 1 and lateral scale 2. This is the term that prevents the robot from satisfying a forward COM bias primarily by throwing its arms backward; sustained body motion must approach the commanded velocity.

COM height is mass-weighted over the articulation, not inferred from the root prim. The release targets 1.052 m with an exponential 0.02 m standard deviation. The observation adds normalized COM-height and vertical-velocity reference errors, giving the policy closed-loop information for later stand/crouch trajectories.

An optional planar position term tracks p*(t) = p(0) + v* t. It is disabled in Stage 46 because velocity is the primary interface and position tracking is less appropriate while turning.

Gait and foot placement

The gait clock is 0.55 seconds with the two legs offset by half a cycle.

feet_gait rewards the expected contact state.
foot_phase_velocity asks the swing foot to move with the planar command, instead of merely marching in place.
feet_late_swing_clearance rewards a clear inter-foot configuration before touchdown.
feet_slide, excessive contact force, and undesired contacts remain regularizers.

The CAD articulation contains intentional internal overlaps, so global self-collision cannot safely be enabled. Instead, two locomotion-specific constraints operate on the distal feet:

hard = relu((0.16 - center_distance) / 0.16)²
     + relu((0.10 - anatomical_lateral_separation) / 0.10)²

Stage 46 weights this at -120. An anticipatory -18 term starts at 0.20 m center distance / 0.14 m lateral separation and penalizes relative velocity toward the other foot. Separating motion is never penalized.

The strict evaluation fractions are diagnostics rather than binary collision claims. A fraction reports the share of policy steps below its gate. v0.1.0 still has short low-clearance intervals, particularly during left stepping, but no falls or persistent left/right inversion in the reported trials.

Long-horizon bilateral timing

The post-v0.1 continuation adds bilateral_contact_timing_l2 to remove a step-to gait in which the left foot advances and the right foot meets it while the left remains loaded or drags. A four-second exponential history compares the legs across four independent quantities:

contact duty factor;
touchdown rate;
same-leg stride interval;
completed airborne time before each valid touchdown.

Flights shorter than 0.06 seconds are ignored as contact flicker, and the first 1.2 seconds after reset are excluded. If one leg completes valid strides or flights while the other does not, the missing cycle is penalized directly. This is a long-horizon cadence constraint, not an instantaneous requirement that both feet have the same contact state.

Pendulum-like lateral weight transfer

torso_roll_pendulum_l2 complements the generic flat-orientation cost. It penalizes the multi-second mean roll bias while tracking a small sinusoidal roll reference locked to the 0.55-second gait phase. The reference peaks near the middle of alternating single-support phases and averages to zero, so a constant right lean cannot satisfy it. The experimental target amplitude is only 0.035 rad (about 2 degrees); larger rocking remains disfavored by the existing uprightness term.

Cardinal motion and lateral recovery

Forward, backward, left, and right commands are sampled as four balanced cardinal modes. COM and root velocity errors are evaluated in the robot yaw frame, so the same task-space contract applies in every direction. The curriculum begins at ±0.2 m/s and expands in 0.1 m/s steps toward the configured ±1.0 m/s ceiling only after a completed-episode tracking gate passes.

Lateral robustness is introduced separately through bounded, instantaneous velocity kicks. A zero forward kick and ±0.25 m/s lateral kick isolates side-force recovery from forward-speed learning. The foot-distance, anatomical-separation, and approach-speed penalties remain active during these recoveries so the policy cannot solve a shove by crossing the feet through one another.

Arms

Natural shoulder motion is phase-linked to the gait clock and scales with command speed. Stage 46 uses amplitude 0.20 rad, standard deviation 0.18 rad, and reward weight +1.

Two penalties stop the arms from becoming the main balance actuator:

shoulder extension past 0.35 rad: weight -2;
same-direction counterweight mode past 0.10 rad: weight -5.

This preserves visible reciprocal arm swing while requiring the legs and torso to handle sustained COM translation.

General regularization

The release retains uprightness, alive/termination, action-rate, acceleration, joint-limit, energy, pelvis/hip deviation, foot-slide, and contact terms. Reward weights are integrated by the 0.02-second environment step, so terminal and per-step terms should be compared after that scaling.

Every release-specific override is visible in scripts/train_release_v0_1.sh; the resolved result is in configs/release_v0.1.0/env.yaml.