README.md

metadata

title: Nova Sim
colorFrom: yellow
colorTo: green
sdk: docker
app_port: 7860

Nova Sim

A unified MuJoCo-based robot simulation platform with web interface for multiple robot types.

Outline

1. Overview
2. Highlights
3. Supported Robots
4. Project Structure
5. Quick Start
6. Docker Deployment
7. Controls
8. Architecture
9. API
10. Wandelbots Nova API Integration
11. Testing
12. License

Overview

Nova Sim combines MuJoCo physics, a Flask/WebSocket server, and a browser UI so you can explore locomotion and manipulation robots from a single web interface. The platform streams MJPEG video of the rendered scene, exposes HTTP endpoints for trainers and RL clients, and lets you switch robots and control inputs without restarting the server.

Highlights

• Real-time MuJoCo physics simulation
• Web-based video streaming interface with TypeScript frontend
• WebSocket-based state/command communication
• Simple venv-based setup with Python and npm
• Gym-style WebSocket API for RL/IL clients
• Interactive camera controls (rotate, zoom, pan)
• Robot switching without restart
• Keyboard and button controls for locomotion

Supported Robots

Unitree G1 (Humanoid)

• 29 degrees of freedom
• RL-based walking policy from unitree_rl_gym
• Full body control with arms and waist

Boston Dynamics Spot (Quadruped)

• 12 degrees of freedom (3 per leg)
• Multiple gait controllers:
  • MPC Gait: Feedback-based balance control with trot pattern (default)
  • PyMPC Gait: Uses standalone gait generator (no external dependencies needed)
  • Trot Gait: Simple open-loop trot pattern
• Note: Spot controllers are now fully self-contained with extracted dependencies from gym_quadruped and Quadruped-PyMPC. See robots/spot/DEPENDENCIES.md for details.

Universal Robots UR5e (Robot Arm)

• 6 degrees of freedom (6-axis industrial arm)
• Robotiq 2F-85 gripper (in regular scene only; T-push scene uses a push stick tool)
• Two control modes:
  • IK Mode: Set target XYZ position and orientation (Roll/Pitch/Yaw), inverse kinematics computes joint angles using damped least squares with 6-DOF Jacobian
  • Joint Mode: Direct control of individual joint positions
• End-effector position and orientation tracking
• Full 6-DOF IK with orientation control (can be toggled on/off)
• Optional Wandelbots Nova API integration for real robot state streaming and cloud-based IK

Project Structure

nova_sim/
├── mujoco_server.py          # Main Flask server with WebSocket
├── Dockerfile                # Docker build configuration
├── docker-compose.yml        # CPU/OSMesa configuration
├── docker-compose.gpu.yml    # GPU/EGL configuration
├── requirements.txt          # Python dependencies
├── frontend/                 # Web UI (TypeScript + Vite)
│   ├── src/
│   │   ├── index.html        # HTML template
│   │   ├── main.ts           # Main TypeScript entry point
│   │   ├── styles.css        # CSS styles
│   │   ├── api/
│   │   │   └── client.ts     # WebSocket client
│   │   └── types/
│   │       └── protocol.ts   # TypeScript protocol types
│   ├── package.json          # Node.js dependencies
│   ├── tsconfig.json         # TypeScript configuration
│   └── vite.config.ts        # Vite bundler configuration
├── templates/                # Flask templates
│   └── index.html            # Main HTML template
├── robots/
│   ├── g1/                   # Unitree G1 humanoid
│   │   ├── g1_env.py         # Gymnasium environment
│   │   ├── scene.xml         # MuJoCo scene
│   │   ├── g1_29dof.xml      # Robot model
│   │   ├── meshes/           # 3D mesh files
│   │   ├── policy/           # RL policy weights
│   │   └── controllers/      # G1 controllers
│   │       ├── rl_policy.py      # RL walking policy
│   │       ├── pd_standing.py    # Standing controller
│   │       └── keyframe.py       # Keyframe controller
│   ├── spot/                 # Boston Dynamics Spot quadruped
│   │   ├── spot_env.py       # Gymnasium environment
│   │   ├── model/            # MuJoCo model files
│   │   └── controllers/      # Spot controllers
│   │       ├── mpc_gait.py           # MPC-inspired gait (default)
│   │       ├── quadruped_pympc_controller.py  # PyMPC gait
│   │       ├── trot_gait.py          # Simple trot gait
│   │       └── pd_standing.py        # Standing controller
│   └── ur5/                  # Universal Robots UR5e arm
│       ├── ur5_env.py        # Gymnasium environment
│       ├── model/            # MuJoCo model files
│       │   ├── scene.xml     # Combined UR5e + Robotiq scene
│       │   └── assets/       # Mesh files
│       └── controllers/
│           └── ik_controller.py  # Damped least-squares IK
└── README.md

Quick Start

Setup

# Create and activate a virtualenv
python3 -m venv .venv
source .venv/bin/activate

# Install Python dependencies
pip install -r requirements.txt

# Install frontend dependencies
cd frontend && npm install && cd ..

# Build frontend
cd frontend && npm run build && cd ..

# Start the server
python mujoco_server.py

# Open browser at http://localhost:5000/nova-sim

Reward Threshold: Episodes automatically terminate when the robot reaches within the specified distance of the target. See REWARD_THRESHOLD.md for details.

Docker Deployment

Quick Start

# Build and run (CPU/software rendering)
docker-compose up --build

# Or with GPU acceleration (requires NVIDIA GPU)
docker-compose -f docker-compose.gpu.yml up --build

# Access at: http://localhost:3004/nova-sim

Docker Commands

# Build and run (CPU mode)
docker-compose up --build

# Run with GPU support
docker-compose -f docker-compose.gpu.yml up --build

# View logs
docker-compose logs -f

# Stop containers
docker-compose down

# Or with docker run directly
docker run -p 3004:5000 nova-sim

Configuration & Tuning

Performance Optimization

Docker uses software rendering (OSMesa) which is slower than native GPU rendering. Configure these environment variables to optimize performance:

Variable	Default	Description
RENDER_WIDTH	640	Render width in pixels
RENDER_HEIGHT	360	Render height in pixels
TARGET_FPS	30	Target frame rate
SIM_STEPS_PER_FRAME	10	Physics steps per rendered frame
OMP_NUM_THREADS	4	OpenMP thread count
MKL_NUM_THREADS	4	MKL thread count

docker-compose.yml (CPU - Default)

services:
  nova-sim:
    build: .
    ports:
      - "3004:5000"  # Host:Container
    environment:
      - MUJOCO_GL=osmesa
      - PYOPENGL_PLATFORM=osmesa
      - RENDER_WIDTH=640      # Lower resolution for speed
      - RENDER_HEIGHT=360
      - TARGET_FPS=30

docker-compose.gpu.yml (NVIDIA GPU)

For significantly better performance, use GPU acceleration:

# Requires: NVIDIA GPU + nvidia-container-toolkit
docker-compose -f docker-compose.gpu.yml up
# Or: make docker-gpu

services:
  nova-sim:
    build: .
    ports:
      - "3004:5000"  # Host:Container
    environment:
      - MUJOCO_GL=egl         # GPU rendering
      - PYOPENGL_PLATFORM=egl
      - RENDER_WIDTH=1280     # Full resolution
      - RENDER_HEIGHT=720
      - TARGET_FPS=60
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Performance Comparison

Mode	Resolution	Expected FPS	Notes
Native (macOS/Linux)	1280x720	60+	Best performance
Docker + GPU (EGL)	1280x720	60	Requires NVIDIA GPU
Docker + CPU (OSMesa)	640x360	15-20	Slower, but compatible
Docker + CPU (OSMesa)	320x180	30+	Very low quality

Custom Docker Run

# Ultra-low resolution for maximum speed
docker run -p 3004:5000 \
  -e RENDER_WIDTH=320 \
  -e RENDER_HEIGHT=180 \
  -e MUJOCO_GL=osmesa \
  nova-sim

# High quality with GPU
docker run --gpus all -p 3004:5000 \
  -e RENDER_WIDTH=1920 \
  -e RENDER_HEIGHT=1080 \
  -e MUJOCO_GL=egl \
  nova-sim

Controls

Locomotion Robots (G1, Spot)

• W / Arrow Up: Walk forward
• S / Arrow Down: Walk backward
• A / Arrow Left: Turn left
• D / Arrow Right: Turn right
• Q: Strafe left
• E: Strafe right

Robot Arm (UR5)

• IK Mode: Use XYZ sliders to set end-effector target position, RPY sliders for orientation
• Orientation Control: Toggle checkbox to enable/disable 6-DOF orientation tracking
• Joint Mode: Use J1-J6 sliders to control individual joints
• Gripper: Open/Close buttons for Robotiq gripper
• Home: Return to home position (joint mode)
• Keyboard Teleop: With the UR5 selected, W/A/S/D jogs the tool in the XY plane, R/F nudges along Z, and every keystroke streams a teleop_action event so the browser panel and any trainer see the same velocity delta.

Common Controls

• Mouse drag: Rotate camera
• Scroll wheel: Zoom camera
• Robot selector: Switch between G1, Spot, and UR5
• Reset button: Reset robot to default pose

UR5 Scene & Camera Hints

• The UI now selects between exactly two UR5 options: the gripper-ready scene and the T-push scene (both enumerated via /nova-sim/api/v1/metadata). Scene-specific camera feeds are available via the /env endpoint, so trainers can build dashboards based on the available streams.
• Auxiliary camera tiles now appear only when you add cameras via the dynamic camera API (see API section below). The UI refreshes when a new camera is announced.
• The T-shape target stays anchored at its configured pose across resets, which keeps the training objective consistent even when you hit Reset from the UI.

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                              Browser                                     │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐     │
│  │   Video Player  │    │  Control Panel  │    │   State Display │     │
│  │   (MJPEG img)   │    │  (buttons/keys) │    │   (joint data)  │     │
│  └────────┬────────┘    └────────┬────────┘    └────────▲────────┘     │
└───────────┼──────────────────────┼──────────────────────┼──────────────┘
            │ HTTP GET             │ WebSocket            │ WebSocket
            │ /video_feed          │ command/reset        │ state
            ▼                      ▼                      │
┌─────────────────────────────────────────────────────────────────────────┐
│                         mujoco_server.py                                │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐     │
│  │  Flask + WSGI   │    │   WebSocket     │    │  Render Thread  │     │
│  │  HTTP endpoints │    │  cmd, reset,    │    │  60 FPS loop    │     │
│  │                 │    │  switch_robot   │    │  MJPEG encode   │     │
│  └─────────────────┘    └────────┬────────┘    └────────┬────────┘     │
│                                  │                      │               │
│                                  ▼                      ▼               │
│                         ┌───────────────────────────────────┐          │
│                         │      Active Environment (env)      │          │
│                         │ G1Env, SpotEnv, or UR5Env (Gym)    │          │
│                         └───────────────────────────────────┘          │
└─────────────────────────────────────────────────────────────────────────┘
                                       │
                                       │ env.step(action)
                                       ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         Robot Environments                              │
│                                                                         │
│  ┌────────────────────────────┐    ┌────────────────────────────┐     │
│  │         G1Env              │    │         SpotEnv            │     │
│  │  ┌──────────────────────┐  │    │  ┌──────────────────────┐  │     │
│  │  │ MuJoCo Model (XML)   │  │    │  │ MuJoCo Model (XML)   │  │     │
│  │  │ g1_29dof.xml         │  │    │  │ spot.xml             │  │     │
│  │  └──────────────────────┘  │    │  └──────────────────────┘  │     │
│  │  ┌──────────────────────┐  │    │  ┌──────────────────────┐  │     │
│  │  │ Controller           │  │    │  │ Controller           │  │     │
│  │  │ rl_policy.py         │  │    │  │ (selectable)         │  │     │
│  │  │ (PyTorch .pt file)   │  │    │  │ - mpc_gait.py        │  │     │
│  │  └──────────────────────┘  │    │  │ - pympc_controller   │  │     │
│  │                            │    │  │ - trot_gait.py       │  │     │
│  │  29 DOF Humanoid           │    │  └──────────────────────┘  │     │
│  │  - Torso, arms, legs       │    │                            │     │
│  │  - RL locomotion policy    │    │  12 DOF Quadruped          │     │
│  └────────────────────────────┘    │  - 4 legs × 3 joints       │     │
│                                    │  - Gait-based locomotion   │     │
│                                    └────────────────────────────┘     │
│                                                                         │
│  ┌────────────────────────────┐                                        │
│  │         UR5Env             │    6 DOF Robot Arm                     │
│  │  ┌──────────────────────┐  │    - IK or direct joint control        │
│  │  │ scene.xml (UR5e +    │  │    - Robotiq 2F-85 gripper             │
│  │  │ Robotiq 2F-85)       │  │    - End-effector tracking             │
│  │  └──────────────────────┘  │                                        │
│  │  ┌──────────────────────┐  │                                        │
│  │  │ ik_controller.py     │  │                                        │
│  │  │ (Damped LS IK)       │  │                                        │
│  │  └──────────────────────┘  │                                        │
│  └────────────────────────────┘                                        │
└─────────────────────────────────────────────────────────────────────────┘
                                       │
                                       │ mujoco.mj_step()
                                       ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         MuJoCo Physics Engine                           │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐     │
│  │  Rigid Body     │    │   Collision     │    │   Rendering     │     │
│  │  Dynamics       │    │   Detection     │    │   (OpenGL/      │     │
│  │                 │    │                 │    │    OSMesa/EGL)  │     │
│  └─────────────────┘    └─────────────────┘    └─────────────────┘     │
└─────────────────────────────────────────────────────────────────────────┘

Data Flow

1. User Input → Browser captures keyboard/button events
2. WebSocket → Commands sent as {vx, vy, vyaw} velocity targets
3. Controller → Converts velocity actions to joint position targets
4. MuJoCo → Simulates physics at 500Hz (0.002s timestep)
5. Renderer → Captures frames at 60 FPS (native) or 30 FPS (Docker)
6. MJPEG Stream → Frames encoded as JPEG and streamed to browser
7. State Broadcast → Robot state sent via WebSocket at render rate

Controller Architecture

G1 (Humanoid): Uses a pre-trained RL policy (PyTorch) that maps observations (body orientation, joint positions/velocities, actions) to joint position targets.

Spot (Quadruped): Uses gait-based controllers:

• MPC Gait: Phase-based trot with feedback balance control
• PyMPC: Integrates Quadruped-PyMPC's gait generator for timing
• Trot: Simple open-loop sinusoidal gait pattern

All controllers output joint position targets; MuJoCo's built-in PD control tracks these targets.

API

All endpoints are prefixed with /nova-sim/api/v1.

WebSocket

Connect using standard WebSocket:

const ws = new WebSocket('ws://localhost:3004/nova-sim/api/v1/ws');

// Send message
ws.send(JSON.stringify({type: 'action', data: {vx: 0.5, vy: 0, vyaw: 0}}));

// Receive messages
ws.onmessage = (event) => {
    const msg = JSON.parse(event.data);
    if (msg.type === 'state') {
      console.log(msg.data);
    }
  };

Nova-Sim uses /ws as the shared control channel for the browser UI, trainers, and any RL clients. Every UI interaction (teleop, camera controls, robot switching) and the trainer handshake/notifications flows through this single socket; the UI state messages shown below now also carry the action velocities, integrated reward, and trainer connection status that RL agents need.

HTTP Endpoints

Nova-Sim provides a minimal HTTP API for static information:

Endpoint	Method	Description
/env	GET	Returns static environment information: robot, scene, has_gripper, action_space, observation_space, camera_feeds
/metadata	GET	Returns available robots, scenes, actions, and system configuration
/video_feed	GET	MJPEG video stream of the main camera
/camera/<name>/video_feed	GET	MJPEG video stream of auxiliary cameras (added via /cameras)
/cameras	POST	Register a new auxiliary camera for the current robot/scene
/homing	GET	Blocking homing for UR5 via Nova jogging (pauses joggers while executing)

Example /env response:

{
  "robot": "ur5",
  "scene": "scene",
  "has_gripper": true,
  "control_mode": "ik",
  "action_space": {...},
  "observation_space": {...},
  "camera_feeds": [
    {
      "name": "main",
      "label": "Main Camera",
      "url": "/nova-sim/api/v1/video_feed",
      "pose": {
        "position": {"x": 0.75, "y": -0.75, "z": 0.9},
        "orientation": {"w": 0.92, "x": -0.16, "y": -0.36, "z": 0.04}
      },
      "intrinsics": {"fx": 869.1, "fy": 869.1, "cx": 640.0, "cy": 360.0, "width": 1280, "height": 720, "fovy_degrees": 45.0}
    },
    {
      "name": "aux_side",
      "label": "Aux Side View",
      "url": "/nova-sim/api/v1/camera/aux_side/video_feed",
      "pose": {
        "position": {"x": 0.5, "y": 0.2, "z": 0.9},
        "orientation": {"w": 0.99, "x": -0.08, "y": 0.02, "z": 0.02}
      },
      "intrinsics": {"fx": 193.1, "fy": 193.1, "cx": 120.0, "cy": 80.0, "width": 240, "height": 160, "fovy_degrees": 45.0}
    }
  ],
  "home_pose": [-1.57, -1.57, 1.57, -1.57, -1.57, 0.0]
}

The /env endpoint returns scene-specific information including camera feeds available for the current robot/scene configuration. Dynamic cameras added via POST /cameras appear here immediately.

All dynamic operations (reset, switching robots, sending actions) are performed via WebSocket messages. Training data (observations, rewards, etc.) come from the /ws state stream.

Add a camera (POST /cameras)

{
  "name": "aux_side",
  "label": "Aux Side View",
  "lookat": [0.55, -0.1, 0.42],
  "distance": 0.9,
  "azimuth": -45,
  "elevation": -30,
  "replace": true
}

Notes:

• Cameras are scoped to the current robot + scene and show up in /env under camera_feeds.
• The UI listens for a state message containing camera_event and refreshes tiles automatically.

Blocking homing (GET /homing)

/nova-sim/api/v1/homing?timeout_s=30&tolerance=0.01&poll_interval_s=0.1

Notes:

• This endpoint is blocking: it returns only after the robot reaches home_pose or a timeout occurs.
• While homing is active, jogging commands are paused (incoming jog/teleop actions are ignored).

Client → Server WebSocket Messages

action - Send velocity actions to all robots:

{"type": "action", "data": {"vx": 0.5, "vy": 0.0, "vyaw": 0.0}}

For locomotion robots (G1, Spot):

• vx: Forward/backward velocity [-1, 1]
• vy: Left/right strafe velocity [-1, 1]
• vyaw: Turn rate [-1, 1]

For robot arms (UR5):

• vx, vy, vz: Cartesian translation velocities (m/s)
• vrx, vry, vrz: Cartesian rotation velocities (rad/s)
• j1-j6: Joint velocities (rad/s)
• gripper: Gripper position [0-255]

Note: Old message type command is still accepted for backward compatibility.

reset - Reset the environment:

{"type": "reset", "data": {"seed": 42}}

• seed: Optional random seed for environment reset

switch_robot - Switch to a different robot/scene:

{"type": "switch_robot", "data": {"robot": "ur5", "scene": "scene"}}

• robot: Required. One of: "g1", "spot", "ur5", "ur5_t_push"
• scene: Optional scene name (see /metadata for available scenes)

teleop_action - Send teleoperation action (primarily for UR5 keyboard control):

{"type": "teleop_action", "data": {"vx": 0.01, "vy": 0.0, "vz": 0.0}}

• For UR5: vx, vy, vz represent Cartesian velocity deltas (m/s)
• Backward compatible: old dx, dy, dz format is auto-mapped to vx, vy, vz
• Note: Old message type teleop_command is still accepted for backward compatibility.

camera:

// Rotate camera
{"type": "camera", "data": {"action": "rotate", "dx": 10, "dy": 5}}

// Zoom camera
{"type": "camera", "data": {"action": "zoom", "dz": -50}}

// Pan camera
{"type": "camera", "data": {"action": "pan", "dx": 10, "dy": 5}}

// Set absolute distance
{"type": "camera", "data": {"action": "set_distance", "distance": 3.0}}

camera_follow:

{"type": "camera_follow", "data": {"follow": true}}

UR5-Specific Messages

arm_target (IK mode - set end-effector target position):

{"type": "arm_target", "data": {"x": 0.4, "y": 0.0, "z": 0.6}}

• x, y, z: Target position in meters

arm_orientation (IK mode - set end-effector target orientation):

{"type": "arm_orientation", "data": {"roll": 0.0, "pitch": 1.57, "yaw": 0.0}}

• roll, pitch, yaw: Target orientation in radians (XYZ Euler convention)
• Default pointing down: pitch = 1.57 (π/2)

use_orientation (Toggle orientation control):

{"type": "use_orientation", "data": {"enabled": true}}

• enabled: true for 6-DOF IK (position + orientation), false for 3-DOF IK (position only)

joint_positions (Joint mode - direct joint control):

{"type": "joint_positions", "data": {"positions": [-1.57, -1.57, 1.57, -1.57, -1.57, 0.0]}}

• positions: Array of 6 joint angles in radians [J1, J2, J3, J4, J5, J6]

control_mode (Switch control mode):

{"type": "control_mode", "data": {"mode": "ik"}}

• mode: "ik" (end-effector target) or "joint" (direct joint positions)

gripper (Control gripper):

{"type": "gripper", "data": {"action": "close"}}

• action: "open" or "close"

set_nova_mode (Configure Nova API integration):

{"type": "set_nova_mode", "data": {"state_streaming": true, "ik": false}}

• enabled: (Optional, legacy) Enable/disable all Nova features
• state_streaming: (Optional) Enable/disable Nova state streaming
• ik: (Optional) Enable/disable Nova IK computation

Client Identity & Notification Messages

client_identity (Client handshake):

{"type": "client_identity", "data": {"client_id": "my_client_v1"}}

• client_id: Unique identifier for the external client
• Note: Old trainer_identity type is still supported for backward compatibility

client_notification (Send client notification to all connected clients):

{"type": "client_notification", "data": {"message": "Starting epoch 5", "level": "info"}}

• message: Notification text
• level: "info", "warning", or "error"
• Note: Old notification type is still supported for backward compatibility

episode_control (Control episode state):

{"type": "episode_control", "data": {"action": "terminate"}}

• action: "terminate" (episode ended successfully) or "truncate" (episode cut short)

Server → Client Messages

state (broadcast at ~10 Hz):

For locomotion robots (G1, Spot):

{
  "type": "state",
  "data": {
    "observation": {
      "position": {"x": 0.0, "y": 0.0, "z": 0.46},
      "orientation": {"w": 1.0, "x": 0.0, "y": 0.0, "z": 0.0}
    },
    "steps": 1234,
    "reward": 0.0,
    "teleop_action": {"vx": 0.5, "vy": 0.0, "vz": 0.0, "vyaw": 0.0, "vrx": 0.0, "vry": 0.0, "vrz": 0.0, "j1": 0.0, "j2": 0.0, "j3": 0.0, "j4": 0.0, "j5": 0.0, "j6": 0.0, "gripper": 0.0},
    "trainer_connected": true
  }
}

For robot arm (UR5):

{
  "type": "state",
  "data": {
    "observation": {
      "end_effector": {"x": 0.4, "y": 0.0, "z": 0.6},
      "ee_orientation": {"w": 0.5, "x": 0.5, "y": 0.5, "z": 0.5},
      "ee_target": {"x": 0.4, "y": 0.0, "z": 0.6},
      "ee_target_orientation": {"roll": 0.0, "pitch": 1.57, "yaw": 0.0},
      "gripper": 128,
      "joint_positions": [-1.57, -1.57, 1.57, -1.57, -1.57, 0.0],
      "joint_targets": [-1.57, -1.57, 1.57, -1.57, -1.57, 0.0]
    },
    "control_mode": "ik",
    "steps": 1234,
    "reward": -0.25,
    "teleop_action": {"vx": 0.02, "vy": 0.0, "vz": 0.0, "vyaw": 0.0, "vrx": 0.0, "vry": 0.0, "vrz": 0.0, "j1": 0.0, "j2": 0.0, "j3": 0.0, "j4": 0.0, "j5": 0.0, "j6": 0.0, "gripper": 128.0},
    "trainer_connected": true,
    "nova_api": {
      "connected": true,
      "state_streaming": true,
      "ik": false
    }
  }
}

Field descriptions:

Common fields (all robots):

• observation: Contains robot-specific sensor data and state information
• steps: Number of simulation steps since last reset
• reward: The integrated task reward from the simulator that external clients can consume
• teleop_action: The canonical action/velocity stream that drives locomotion or arm movement; the UI and every external client should read this field as the unified action record. Always present with zero values when idle
  • Cartesian velocities: vx (forward/back or X-axis), vy (strafe or Y-axis), vz (vertical for UR5), vyaw (rotation for locomotion)
  • Cartesian rotation velocities (UR5 only): vrx, vry, vrz (rad/s)
  • Joint velocities (UR5 only): j1, j2, j3, j4, j5, j6 (rad/s)
  • Gripper: gripper (0-255 for UR5, 0 for others)
  • Locomotion robots: Use vx, vy, vyaw (other fields are 0)
  • UR5: Use vx/vy/vz for Cartesian translation, vrx/vry/vrz for rotation, j1-j6 for joint velocities, and gripper for gripper control
• connected_clients: List of connected external client IDs (e.g., ["trainer_v1", "monitor"])
• scene_objects: List of scene objects with their positions and orientations. Each object has:
  • name: Object identifier (e.g., "t_object", "t_target", "box")
  • position: Object position in meters (x, y, z)
  • orientation: Object orientation as quaternion (w, x, y, z)

Locomotion observation fields (inside observation):

• position: Robot base position in world coordinates (x, y, z) in meters
• orientation: Robot base orientation as quaternion (w, x, y, z)

UR5 observation fields (inside observation):

• end_effector: Current end-effector position in meters (x, y, z)
• ee_orientation: Current end-effector orientation as quaternion (w, x, y, z)
• ee_target: Target end-effector position in meters (x, y, z)
• ee_target_orientation: Target end-effector orientation as Euler angles in radians (roll, pitch, yaw)
• gripper: Gripper position (0-255, where 0 is open and 255 is closed)
• joint_positions: Current joint angles in radians
• joint_targets: Target joint angles in radians (in joint control mode)

UR5-specific fields (outside observation):

• control_mode: Current control mode ("ik" for inverse kinematics or "joint" for joint control)
• nova_api: Nova API integration status (displays which controllers are active)
  • connected: Whether Nova API client is connected
  • state_streaming: Whether using Nova API for robot state streaming (vs. internal)
  • ik: Whether using Nova API for inverse kinematics (vs. internal)

Note: Static environment information (robot name, scene name, has_gripper, action/observation spaces, camera feeds) has been moved to the GET /env endpoint and is no longer included in the state stream.

connected_clients (broadcast to all clients):

{
  "type": "connected_clients",
  "data": {
    "clients": ["trainer_v1", "monitor"]
  }
}

• clients: Array of connected external client IDs

client_notification (broadcast to all clients):

{
  "type": "client_notification",
  "data": {
    "message": "Starting epoch 5",
    "level": "info"
  }
}

• message: Notification text from external client
• level: "info", "warning", or "error"

State broadcasts and client notifications

Every /ws client receives a state message roughly every 100 ms. The examples above show the locomotion (spot) and arm (ur5) payloads; the payload also now includes:

• teleop_action: The latest action/teleoperation stream (includes vx, vy, vz, vyaw, vrx, vry, vrz, j1-j6, gripper) so external clients and the UI read a single canonical action payload. Always present with zero values when idle.
• reward: The integrated task reward that external clients can consume without sending a separate step.
• connected_clients: Array of connected external client IDs (used to display which clients are connected).

External clients (trainers, monitors, etc.) announce themselves by sending a client_identity payload when the socket opens. The server mirrors that information into the connected_clients broadcasts (connected_clients messages flow to all clients) and lets external clients emit client_notification payloads that all clients receive.

HTTP Endpoints

Endpoint	Method	Description
/nova-sim/api/v1	GET	Web interface (HTML/JS)
/nova-sim/api/v1/env	GET	Static environment info (robot, scene, spaces, camera feeds)
/nova-sim/api/v1/metadata	GET	Available robots, scenes, actions, and system configuration
/nova-sim/api/v1/video_feed	GET	MJPEG video stream (main camera)
/nova-sim/api/v1/camera/<name>/video_feed	GET	MJPEG video stream (auxiliary cameras)

Video stream usage:

<img src="http://localhost:3004/nova-sim/api/v1/video_feed" />

Note: Robot switching, reset, and other dynamic operations are performed via WebSocket messages, not HTTP endpoints. See the WebSocket Messages section for details.

Metadata & Camera Feeds

• GET /nova-sim/api/v1/metadata returns JSON describing every available robot/scene pair and the supported actions
• GET /nova-sim/api/v1/env returns scene-specific camera feeds - the camera_feeds array lists all available video streams for the current robot/scene configuration including the main camera and any auxiliary cameras you registered via POST /nova-sim/api/v1/cameras
• GET /nova-sim/api/v1/camera/<name>/video_feed streams MJPEG for a specific camera feed
• pytest tests/ exercises the HTTP metadata/video endpoints, the /ws control socket, and every camera feed. Keep Nova-Sim running at http://localhost:3004 when you run it so the suite can talk to the live server.

Wandelbots Nova API Integration

The UR5 environment supports optional integration with the Wandelbots Nova API v2 for real-time robot state streaming, cloud-based inverse kinematics, and robot jogging control.

Nova API integration provides three key features:

• Robot State Streaming: Real-time synchronization of joint positions from physical robots via WebSocket
• Inverse Kinematics: Cloud-based IK computation using Nova's kinematic solver
• Robot Jogging: Direct velocity control of robot joints and cartesian motion via unified v2 API

Quick Setup

1. Copy the environment template:

   cp .env.example .env.local
   

2. Configure your Nova credentials in .env.local:

   NOVA_INSTANCE_URL=https://your-instance.wandelbots.io
   NOVA_ACCESS_TOKEN=your_access_token
   NOVA_CELL_ID=cell
   NOVA_CONTROLLER_ID=your_controller_id
   NOVA_MOTION_GROUP_ID=your_motion_group_id
   NOVA_MOTION_GROUP_MODEL=ur5e  # or your robot model
   NOVA_TCP_NAME=Flange
   NOVA_RESPONSE_RATE_MS=200
   

3. Discover available robots in your Nova instance:

   python3 scripts/discover_nova_simple.py
   

4. Test your connection:

   cd nova-sim
   python3 scripts/test_nova_connection.py
   

Usage Modes

Nova API integration supports three distinct modes:

1. Digital Twin Mode (State Streaming Only - Recommended): The simulation mirrors a real robot's state. Robot movements come from the physical robot via WebSocket. Important: In this mode, keyboard controls, jogging buttons, and internal target controls are disabled because the robot state is being synchronized from the real robot. This is the default when .env.local is detected.

2. Nova IK Mode (IK Only): Use Nova's cloud-based IK solver while controlling the simulation robot locally. Useful for validating IK solutions or when you want to use Nova's kinematic model. Keyboard controls and jogging work normally in this mode.

3. Hybrid Mode (Both): Uses Nova state streaming and Nova IK. Generally not recommended since the real robot is controlled externally. Keyboard controls are disabled in this mode.

Usage Examples

Digital Twin Mode (State Streaming Only - Default)

Mirror a real robot's state in the simulation:

import os
from pathlib import Path
from nova_sim.robots.ur5.ur5_env import UR5Env

# Load environment variables
def load_env(filepath):
    with open(filepath) as f:
        for line in f:
            if line.strip() and not line.startswith('#') and '=' in line:
                key, val = line.split('=', 1)
                os.environ[key.strip()] = val.strip()

load_env(Path("nova-sim/.env.local"))

# Create environment with state streaming enabled
env = UR5Env(
    render_mode="human",
    scene_name="scene",
    nova_api_config={
        "use_state_stream": True,  # Mirror real robot state
        "use_ik": False  # Real robot controlled externally
    }
)

obs, info = env.reset()
for _ in range(1000):
    obs = env.step_with_controller(dt=0.002)
    env.render()
env.close()

Nova IK Mode (IK Only)

Use Nova's cloud-based IK solver while controlling the simulation robot locally:

env = UR5Env(
    render_mode="human",
    nova_api_config={
        "use_state_stream": False,
        "use_ik": True  # Use Nova IK
    }
)

# IK will be computed by Nova API
env.set_target(x=0.4, y=0.2, z=0.6)

Hybrid Mode (Not Recommended)

Enable both state streaming and Nova IK. Generally not useful since the real robot is controlled externally:

env = UR5Env(
    render_mode="human",
    nova_api_config={
        "use_state_stream": True,  # Mirror real robot
        "use_ik": True  # Also use Nova IK (usually not needed)
    }
)

Note: When state streaming is enabled, the simulation becomes a digital twin that mirrors the real robot's state. The robot can still be controlled via Nova API jogging commands (when use_jogging is enabled). Local IK target updates are ignored in favor of the real robot's state, but the robot can be moved through Nova API commands.

Environment Variables Reference

Variable	Required	Default	Description
NOVA_INSTANCE_URL	Yes	-	Nova instance URL (e.g., https://nova.wandelbots.io)
NOVA_ACCESS_TOKEN	Yes	-	Nova API access token
NOVA_CELL_ID	No	cell	Cell ID
NOVA_CONTROLLER_ID	Yes	-	Controller ID
NOVA_MOTION_GROUP_ID	Yes	-	Motion group ID
NOVA_MOTION_GROUP_MODEL	No	Auto-detected	Motion group model (e.g., ur5e, KUKA_KR16_R2010_2)
NOVA_TCP_NAME	No	Flange	Tool Center Point name
NOVA_RESPONSE_RATE_MS	No	200	State stream response rate in milliseconds

Note: NOVA_API is also supported as an alias for NOVA_INSTANCE_URL for backward compatibility.

How It Works

State Streaming

When use_state_stream is enabled:

1. A background thread connects to Nova's WebSocket state stream
2. Robot state (joint positions) is continuously received at the configured rate
3. Each time the environment queries robot state, the latest data from Nova is synced
4. The simulation updates its joint positions to match the real robot
5. MuJoCo forward kinematics computes the end-effector pose

Note: When state streaming is active, the simulation becomes a digital twin that mirrors the real robot's state. The robot can still be moved via Nova API commands (e.g., jogging).

Inverse Kinematics

When use_ik is enabled:

1. Target poses are sent to Nova's IK API endpoint: POST /api/v2/cells/{cell_id}/kinematic/inverse
2. Nova computes joint angles using the robot's kinematic model
3. Joint targets are returned and applied to the simulation
4. If Nova IK fails, the system automatically falls back to local IK computation

Getting Nova Credentials

To obtain your Nova instance credentials:

1. Log in to the Wandelbots Portal
2. Navigate to your Nova instance
3. Generate an API access token in the settings
4. Note your controller ID and motion group ID from the API documentation
5. Use the discovery script to find available robots:

   python3 scripts/discover_nova_simple.py
   

For more details, see the Nova API Documentation.

Troubleshooting

"Missing required environment variables"

Ensure all required variables are set in .env.local. Verify loading with:

import os
print("NOVA_INSTANCE_URL:", os.getenv("NOVA_INSTANCE_URL"))

"websockets is required for Nova state streaming"

Install the websockets package:

pip install websockets

"Nova API error 401"

Your access token may be invalid or expired. Generate a new token from the Wandelbots Portal.

State Stream Errors

• Verify your instance URL is correct
• Check that controller and motion group IDs are valid
• Ensure your network can reach the Nova instance
• Confirm your access token has not expired

IK Falls Back to Local Solver

If you see "Nova IK failed, falling back to local IK":

• Check that the motion group model matches your robot
• Verify the TCP name matches a TCP defined in your Nova instance
• Ensure the target pose is reachable by the robot

Performance Considerations

• State Stream Rate: Default is 200ms (5Hz). Lower values provide more responsive streaming but increase bandwidth usage
• IK Latency: Nova IK requires a network round-trip (~50-200ms depending on connection quality)
• Simulation Rate: When using state streaming, simulation rate matches the Nova stream rate

Dependencies

• websockets (for state streaming): pip install websockets
• python-dotenv (optional, for easier .env loading): pip install python-dotenv

Helpful Scripts

Located in scripts/:

• discover_nova_simple.py: Discover available controllers and motion groups in your Nova instance
• test_nova_connection.py: Verify your Nova API configuration is working correctly

Implementation Details

The Nova API integration is implemented in:

• robots/ur5/nova_api.py - API client and configuration
• robots/ur5/ur5_env.py - Environment integration (lines 123-203, 499-520)

Testing

1. Start the Nova Sim server (e.g. python nova-sim/mujoco_server.py or via docker-compose).
2. Keep it running at http://localhost:3004 so the HTTP/websocket endpoints stay reachable.
3. Run pytest nova-sim/tests to exercise:
   • API endpoints (/metadata, /camera/<name>/video_feed, /video_feed)
   • Unified WebSocket control (/ws)
   • Auxiliary MJPEG overlays after switching to the T-push UR5 scene

The tests assume the server is accessible via http://localhost:3004/nova-sim/api/v1 and will skip automatically if the API is unreachable.

License

This project uses models from:

• MuJoCo Menagerie (BSD-3-Clause)
• unitree_mujoco (BSD-3-Clause)
• unitree_rl_gym
• Quadruped-PyMPC (BSD-3-Clause) teleop_action:

{"type": "teleop_action", "data": {"vx": 0.01, "vy": 0.0, "vz": -0.01}}

• vx, vy, vz: Velocity/delta values for UR5 Cartesian movement (WASD + RF from the UI) or locomotion robot velocity (vx, vy, vyaw)
• These values appear in the teleop_action field of each /ws state broadcast, which is the canonical action stream for both the UI and any RL trainer
• The field always contains zero values when idle (never null)