lerobot/so101-tabletop-isaac

SO-101 tabletop (Isaac Sim)

A LeRobot EnvHub environment built natively in NVIDIA Isaac Sim 6.2: an SO-101 arm in the turquoise of the LeRobot kits, four painted wooden 3 cm cubes and a green open box on a table in a simple room, simulated by PhysX and rendered with RTX.

The layout, cameras, observations, actions and LEROBOT_SIM match the MuJoCo version, lerobot/so101-tabletop-mujoco, so the same policies and commands run against both.

Observation	pixels: front (scene camera) and wrist (gripper camera), 480×640 RGB; agent_pos: 6 joint positions
Action	6 absolute joint position targets: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll in degrees, gripper 0-100 (the units of LeRobot's so101_follower with use_degrees=True)
Control rate	30 fps (8 PhysX steps per control step)
Task	"Put all the cubes in the green box."
Reward	the fraction of cubes in the box at each step (0, 0.25, ... 1); info["cubes_in_box"] gives the count
Episode end	success (terminated) when all four cubes are in the box; truncated after 3600 steps (2 minutes), set with episode_length
Reset	seeded cube positions and yaws, joint noise

Isaac Sim starts once per process, in make_env: about 20 s, plus a few minutes of shader compilation the first time on a machine or in a container. It holds one world, so make_env takes n_envs=1. A step takes about 80 ms on an RTX 3080 Laptop GPU, so lockstep evaluation is exact, while realtime runs slower than 30 fps. The arm asset is bundled, so the environment needs no network access at runtime.

With the LeRobot simulator server

From a LeRobot checkout, start the server. The image installs packages.txt (system libraries) and requirements.txt (Isaac Sim 6.2 from PyPI and NVIDIA's index, Python 3.12); it needs an NVIDIA RTX GPU and the NVIDIA Container Toolkit:

lerobot-env-server --docker --envhub=lerobot/so101-tabletop-isaac

Then run a policy against it, for example:

lerobot-rollout --robot.type=remote --robot.endpoint=tcp/127.0.0.1:7448 \
  --robot.deployment=envhub --robot.task=so101_tabletop \
  --policy.path=<your SO-101 checkpoint> --fps=30

Observations reach policies as observation.images.front, observation.images.wrist and observation.state; use --rename_map if your checkpoint names its cameras differently.

In-process

Install Isaac Sim 6.2 next to LeRobot (uv pip install -r requirements.txt with UV_INDEX_STRATEGY=unsafe-best-match, or pip install -r requirements.txt), then:

from lerobot.envs.factory import make_env

envs = make_env("lerobot/so101-tabletop-isaac", n_envs=1, trust_remote_code=True)

How the scene is built

env.py is a compact example of the Isaac Sim 6 Python API:

Part	Isaac Sim API
Arm	NVIDIA's RobotStudio/so101_new_calib asset (bundled in so101_new_calib/), referenced with stage_utils.add_reference_to_stage, driven through Articulation
Room, table, cubes, box, lights	GroundPlane, scaled Cube prims, DomeLight, DistantLight; GeomPrim colliders, RigidPrim bodies, RigidBodyMaterial
Looks	OmniPbrMaterial for the wooden cubes (textures/wood.png tinted per cube, with projected UVs since USD cubes have none) and for the arm's color (an override of the asset's own material), PreviewSurfaceMaterial for the table and the glossy box; colors are the MuJoCo scene's sRGB values, converted to Isaac's linear space
Cameras	RtxCamera + CameraSensor (RGB annotator); the wrist camera is parented to the arm's gripper link
Stepping	timeline paused, SimulationManager.step for physics, a Replicator capture step for the cameras

Licenses

• This environment is Apache-2.0, including textures/wood.png (generated procedurally for it).
• so101_new_calib/ is NVIDIA's SO-101 asset from the Isaac Sim 6.2 asset library (Isaac/Robots/RobotStudio/so101_new_calib, thumbnails left out), unchanged (env.py overrides its color on the stage), under the Apache-2.0 license in that folder.
• Isaac Sim is distributed under the NVIDIA Isaac Sim license. env.py sets OMNI_KIT_ACCEPT_EULA=YES; running this environment means you accept it.