SO-101 tabletop with MJWarp 4

A native batched EnvHub environment using MuJoCo 4.0.1, MuJoCo Warp 4.0.1 and Warp 1.19.0. One mujoco_warp.Data holds all worlds; simulation and camera rendering run together on the selected Warp device.

The scene and robot assets come from lerobot/so101-tabletop-mujoco revision 844dee6c058960c2ef42b65e2c68b47a5cf7c4f3. The task is to put four cubes in the green box. Reward is the fraction in the box, success terminates the episode, and the default time limit is 3,600 steps at 30 Hz.

Install

Use Python 3.12 and a working NVIDIA driver:

pip install -r requirements.txt

MJWarp 4 is the only supported runtime. The CPU device is available for development and tests.

Use

import numpy as np
from env import make_env

env = make_env(n_envs=32, device="cuda:0")["so101_tabletop"][0]
obs, info = env.reset(seed=42)
actions = np.tile([0, -60, 60, 35, 0, 30], (32, 1))
obs, reward, terminated, truncated, info = env.step(actions)
done = terminated | truncated
if done.any():
    obs, info = env.reset(options={"reset_mask": done})
env.close()

Observations contain agent_pos and uint8 RGB images in pixels.front and pixels.wrist (480ร—640 by default). Six absolute joint targets use SO-101 follower units: degrees for the five arm joints and 0โ€“100 for the gripper.

Shadows, skybox, textures and the original scene's lights/materials are enabled by default. Set use_shadows=False or render_skybox=False explicitly when needed. The renderer preserves the scene rather than applying color or lighting calibration. Its output remains different from classic MuJoCo OpenGL; the matched camera comparison in render-study/comparison.png illustrates the remaining differences. Reproduce it with:

MUJOCO_GL=egl python render-study/run.py

Reset sampling follows the original environment. reset(seed=int) seeds world i with seed + i; a list of seeds is also accepted. A boolean reset_mask selectively resets worlds. There is no autoreset. The batch can continue stepping after individual worlds finish for LeRobot evaluation, but callers must discard those worlds' subsequent transitions or reset them.

Options include fps, episode_length, initial_joints, randomize, cameras, camera_fovy, observation_height, observation_width, nconmax, njmax, device, use_shadows, and render_skybox. render_pixels=False removes images. Tune contact and constraint capacities for the intended workload.

LeRobot evaluation

Use a LeRobot checkout with native EnvHub batch support and install requirements.txt into its environment. For in-process evaluation:

lerobot-eval \
  --env.type=envhub \
  --env.hub_path=lerobot/so101-tabletop-mjwarp \
  --env.trust_remote_code=true \
  --env.use_native_batch=true \
  --policy.path=<SO-101-checkpoint> \
  --policy.device=cuda \
  --eval.batch_size=32 \
  --eval.n_episodes=32 \
  --eval.use_async_envs=false

Pin the Hub revision for reproducible runs. A local checkout path can replace the Hub ID. Native batching is required, and use_async_envs must be false. The simulator-server / SimRobot adapter does not support this native batch.

The Gym interface returns NumPy arrays. Pixels are unpacked to uint8 on the Warp device before transfer, and eval videos reuse the latest rendered front-camera frames. Policy preprocessing and actions still cross the CPU/GPU boundary. CUDA graph capture is not implemented. Physics uses float32 and can diverge numerically from CPU MuJoCo; rendered-image similarity does not establish policy parity.

GPU tensor interface (experimental)

observation_backend="torch" returns image/state tensors on the Warp device and accepts tensor actions. It requires a LeRobot checkout with tensor observation preparation and metadata["tensor_actions"] support; use the NumPy default with other checkouts. Set --env.make_env_kwargs='{"observation_backend":"torch"}' for evaluation.

Image tensors are borrowed views valid until the next step() or reset(). Clone them when retaining raw frames. Warp kernels run on PyTorch's current CUDA stream so rendering, preprocessing, inference and control are ordered without downloading camera observations. Rewards and episode flags remain small CPU arrays. Videos download frames only when requested. The same interface works on the CPU device for tests. The environment package does not install PyTorch; use the policy runtime's installation.

Tests

pip install -r requirements-dev.txt
python -m pytest -q tests
SO101_TEST_DEVICE=cuda:0 python -m pytest -q tests

Tests cover seeded reset parity, independent camera outputs, selective reset, truncation, moving-object rendering, RGB unpacking, cached video frames and preservation of lights/materials. GPU validation uses the pinned MJWarp 4 runtime.

The environment code and scene are Apache-2.0. robotstudio_so101 comes from MuJoCo Menagerie and includes its own Apache-2.0 license.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Space using lerobot/so101-tabletop-mjwarp 1