Georg commited on
Commit ·
1227fda
1
Parent(s): d5b839c
Update README.md with project outline and detailed sections; add .env.off.local to .gitignore for environment configuration
Browse files- .gitignore +1 -0
- README.md +87 -67
.gitignore
CHANGED
|
@@ -46,3 +46,4 @@ Thumbs.db
|
|
| 46 |
# MuJoCo compiled
|
| 47 |
*.mjb
|
| 48 |
.env.local
|
|
|
|
|
|
| 46 |
# MuJoCo compiled
|
| 47 |
*.mjb
|
| 48 |
.env.local
|
| 49 |
+
.env.off.local
|
README.md
CHANGED
|
@@ -5,6 +5,34 @@ A unified MuJoCo-based robot simulation platform with web interface for multiple
|
|
| 5 |

|
| 6 |

|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
## Supported Robots
|
| 9 |
|
| 10 |
### Unitree G1 (Humanoid)
|
|
@@ -29,15 +57,43 @@ A unified MuJoCo-based robot simulation platform with web interface for multiple
|
|
| 29 |
- Full 6-DOF IK with orientation control (can be toggled on/off)
|
| 30 |
- **Optional [Wandelbots Nova API](#wandelbots-nova-api-integration) integration** for real robot state streaming and cloud-based IK
|
| 31 |
|
| 32 |
-
##
|
| 33 |
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
-
|
| 39 |
-
-
|
| 40 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
## Quick Start
|
| 43 |
|
|
@@ -68,22 +124,9 @@ python mujoco_server.py
|
|
| 68 |
If you see `ModuleNotFoundError: No module named 'cv2'`, make sure the venv is activated
|
| 69 |
and `opencv-python` is installed in it.
|
| 70 |
|
| 71 |
-
##
|
| 72 |
|
| 73 |
-
|
| 74 |
-
It supports `reset`, `step`, `configure`, and `get_spaces`.
|
| 75 |
-
|
| 76 |
-
Example request payloads:
|
| 77 |
-
|
| 78 |
-
```json
|
| 79 |
-
{"type": "configure", "data": {"robot": "ur5_t_push"}}
|
| 80 |
-
{"type": "reset"}
|
| 81 |
-
{"type": "step", "data": {"action": [0,0,0,0,0,0,0], "render": false}}
|
| 82 |
-
```
|
| 83 |
-
|
| 84 |
-
The server responds with `gym_reset`, `gym_step`, `gym_spaces`, or `gym_configured` messages.
|
| 85 |
-
|
| 86 |
-
### Docker
|
| 87 |
|
| 88 |
```bash
|
| 89 |
# Build and run (CPU/software rendering)
|
|
@@ -93,9 +136,9 @@ docker-compose up --build
|
|
| 93 |
docker run -p 3004:3004 nova-sim
|
| 94 |
```
|
| 95 |
|
| 96 |
-
##
|
| 97 |
|
| 98 |
-
### Performance Optimization
|
| 99 |
|
| 100 |
Docker uses software rendering (OSMesa) which is slower than native GPU rendering. Configure these environment variables to optimize performance:
|
| 101 |
|
|
@@ -108,7 +151,7 @@ Docker uses software rendering (OSMesa) which is slower than native GPU renderin
|
|
| 108 |
| `OMP_NUM_THREADS` | 4 | OpenMP thread count |
|
| 109 |
| `MKL_NUM_THREADS` | 4 | MKL thread count |
|
| 110 |
|
| 111 |
-
### docker-compose.yml (CPU - Default)
|
| 112 |
|
| 113 |
```yaml
|
| 114 |
services:
|
|
@@ -124,7 +167,7 @@ services:
|
|
| 124 |
- TARGET_FPS=30
|
| 125 |
```
|
| 126 |
|
| 127 |
-
### docker-compose.gpu.yml (NVIDIA GPU)
|
| 128 |
|
| 129 |
For significantly better performance, use GPU acceleration:
|
| 130 |
|
|
@@ -154,7 +197,7 @@ services:
|
|
| 154 |
capabilities: [gpu]
|
| 155 |
```
|
| 156 |
|
| 157 |
-
### Performance Comparison
|
| 158 |
|
| 159 |
| Mode | Resolution | Expected FPS | Notes |
|
| 160 |
|------|------------|--------------|-------|
|
|
@@ -163,7 +206,7 @@ services:
|
|
| 163 |
| Docker + CPU (OSMesa) | 640x360 | 15-20 | Slower, but compatible |
|
| 164 |
| Docker + CPU (OSMesa) | 320x180 | 30+ | Very low quality |
|
| 165 |
|
| 166 |
-
### Custom Docker Run
|
| 167 |
|
| 168 |
```bash
|
| 169 |
# Ultra-low resolution for maximum speed
|
|
@@ -302,44 +345,6 @@ docker run --gpus all -p 3004:3004 \
|
|
| 302 |
|
| 303 |
All controllers output joint position targets; MuJoCo's built-in PD control tracks these targets.
|
| 304 |
|
| 305 |
-
## Project Structure
|
| 306 |
-
|
| 307 |
-
```
|
| 308 |
-
nova_sim/
|
| 309 |
-
├── mujoco_server.py # Main Flask server with WebSocket
|
| 310 |
-
├── Dockerfile # Docker build configuration
|
| 311 |
-
├── docker-compose.yml # CPU/OSMesa configuration
|
| 312 |
-
├── docker-compose.gpu.yml # GPU/EGL configuration
|
| 313 |
-
├── requirements.txt # Python dependencies
|
| 314 |
-
├── robots/
|
| 315 |
-
│ ├── g1/ # Unitree G1 humanoid
|
| 316 |
-
│ │ ├── g1_env.py # Gymnasium environment
|
| 317 |
-
│ │ ├── scene.xml # MuJoCo scene
|
| 318 |
-
│ │ ├── g1_29dof.xml # Robot model
|
| 319 |
-
│ │ ├── meshes/ # 3D mesh files
|
| 320 |
-
│ │ ├── policy/ # RL policy weights
|
| 321 |
-
│ │ └── controllers/ # G1 controllers
|
| 322 |
-
│ │ ├── rl_policy.py # RL walking policy
|
| 323 |
-
│ │ ├── pd_standing.py # Standing controller
|
| 324 |
-
│ │ └── keyframe.py # Keyframe controller
|
| 325 |
-
│ ├── spot/ # Boston Dynamics Spot quadruped
|
| 326 |
-
│ │ ├── spot_env.py # Gymnasium environment
|
| 327 |
-
│ │ ├── model/ # MuJoCo model files
|
| 328 |
-
│ │ └── controllers/ # Spot controllers
|
| 329 |
-
│ │ ├── mpc_gait.py # MPC-inspired gait (default)
|
| 330 |
-
│ │ ├── quadruped_pympc_controller.py # PyMPC gait
|
| 331 |
-
│ │ ├── trot_gait.py # Simple trot gait
|
| 332 |
-
│ │ └── pd_standing.py # Standing controller
|
| 333 |
-
│ └── ur5/ # Universal Robots UR5e arm
|
| 334 |
-
│ ├── ur5_env.py # Gymnasium environment
|
| 335 |
-
│ ├── model/ # MuJoCo model files
|
| 336 |
-
│ │ ├── scene.xml # Combined UR5e + Robotiq scene
|
| 337 |
-
│ │ └── assets/ # Mesh files
|
| 338 |
-
│ └── controllers/
|
| 339 |
-
│ └── ik_controller.py # Damped least-squares IK
|
| 340 |
-
└── README.md
|
| 341 |
-
```
|
| 342 |
-
|
| 343 |
## Spot Controller Options
|
| 344 |
|
| 345 |
The Spot robot supports multiple controller types via the `controller_type` parameter:
|
|
@@ -521,6 +526,21 @@ For robot arm (UR5):
|
|
| 521 |
- `state_streaming`: Whether using Nova API for robot state streaming (vs. internal)
|
| 522 |
- `ik`: Whether using Nova API for inverse kinematics (vs. internal)
|
| 523 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 524 |
### HTTP Endpoints
|
| 525 |
|
| 526 |
| Endpoint | Method | Description |
|
|
|
|
| 5 |

|
| 6 |

|
| 7 |
|
| 8 |
+
## Outline
|
| 9 |
+
1. [Overview](#overview)
|
| 10 |
+
2. [Highlights](#highlights)
|
| 11 |
+
3. [Supported Robots](#supported-robots)
|
| 12 |
+
4. [Project Structure](#project-structure)
|
| 13 |
+
5. [Quick Start](#quick-start)
|
| 14 |
+
6. [Docker Deployment](#docker-deployment)
|
| 15 |
+
7. [Controls](#controls)
|
| 16 |
+
8. [Architecture](#architecture)
|
| 17 |
+
9. [Spot Controller Options](#spot-controller-options)
|
| 18 |
+
10. [API](#api)
|
| 19 |
+
11. [Wandelbots Nova API Integration](#wandelbots-nova-api-integration)
|
| 20 |
+
12. [License](#license)
|
| 21 |
+
|
| 22 |
+
## Overview
|
| 23 |
+
|
| 24 |
+
Nova Sim combines MuJoCo physics, a Flask/WebSocket server, and a browser UI so you can explore locomotion and manipulation robots from a single web interface. The platform streams MJPEG video of the rendered scene, exposes a gym-style API, and lets you switch robots and control inputs without restarting the server.
|
| 25 |
+
|
| 26 |
+
## Highlights
|
| 27 |
+
|
| 28 |
+
- Real-time MuJoCo physics simulation
|
| 29 |
+
- Web-based video streaming interface
|
| 30 |
+
- WebSocket-based state/command communication
|
| 31 |
+
- Gym-style WebSocket API for RL/IL clients
|
| 32 |
+
- Interactive camera controls (rotate, zoom, pan)
|
| 33 |
+
- Robot switching without restart
|
| 34 |
+
- Keyboard and button controls for locomotion
|
| 35 |
+
|
| 36 |
## Supported Robots
|
| 37 |
|
| 38 |
### Unitree G1 (Humanoid)
|
|
|
|
| 57 |
- Full 6-DOF IK with orientation control (can be toggled on/off)
|
| 58 |
- **Optional [Wandelbots Nova API](#wandelbots-nova-api-integration) integration** for real robot state streaming and cloud-based IK
|
| 59 |
|
| 60 |
+
## Project Structure
|
| 61 |
|
| 62 |
+
```
|
| 63 |
+
nova_sim/
|
| 64 |
+
├── mujoco_server.py # Main Flask server with WebSocket
|
| 65 |
+
├── Dockerfile # Docker build configuration
|
| 66 |
+
├── docker-compose.yml # CPU/OSMesa configuration
|
| 67 |
+
├── docker-compose.gpu.yml # GPU/EGL configuration
|
| 68 |
+
├── requirements.txt # Python dependencies
|
| 69 |
+
├── robots/
|
| 70 |
+
│ ├── g1/ # Unitree G1 humanoid
|
| 71 |
+
│ │ ├── g1_env.py # Gymnasium environment
|
| 72 |
+
│ │ ├── scene.xml # MuJoCo scene
|
| 73 |
+
│ │ ├── g1_29dof.xml # Robot model
|
| 74 |
+
│ │ ├── meshes/ # 3D mesh files
|
| 75 |
+
│ │ ├── policy/ # RL policy weights
|
| 76 |
+
│ │ └── controllers/ # G1 controllers
|
| 77 |
+
│ │ ├── rl_policy.py # RL walking policy
|
| 78 |
+
│ │ ├── pd_standing.py # Standing controller
|
| 79 |
+
│ │ └── keyframe.py # Keyframe controller
|
| 80 |
+
│ ├── spot/ # Boston Dynamics Spot quadruped
|
| 81 |
+
│ │ ├── spot_env.py # Gymnasium environment
|
| 82 |
+
│ │ ├── model/ # MuJoCo model files
|
| 83 |
+
│ │ └── controllers/ # Spot controllers
|
| 84 |
+
│ │ ├── mpc_gait.py # MPC-inspired gait (default)
|
| 85 |
+
│ │ ├── quadruped_pympc_controller.py # PyMPC gait
|
| 86 |
+
│ │ ├── trot_gait.py # Simple trot gait
|
| 87 |
+
│ │ └── pd_standing.py # Standing controller
|
| 88 |
+
│ └── ur5/ # Universal Robots UR5e arm
|
| 89 |
+
│ ├── ur5_env.py # Gymnasium environment
|
| 90 |
+
│ ├── model/ # MuJoCo model files
|
| 91 |
+
│ │ ├── scene.xml # Combined UR5e + Robotiq scene
|
| 92 |
+
│ │ └── assets/ # Mesh files
|
| 93 |
+
│ └── controllers/
|
| 94 |
+
│ └── ik_controller.py # Damped least-squares IK
|
| 95 |
+
└── README.md
|
| 96 |
+
```
|
| 97 |
|
| 98 |
## Quick Start
|
| 99 |
|
|
|
|
| 124 |
If you see `ModuleNotFoundError: No module named 'cv2'`, make sure the venv is activated
|
| 125 |
and `opencv-python` is installed in it.
|
| 126 |
|
| 127 |
+
## Docker Deployment
|
| 128 |
|
| 129 |
+
### Getting Started
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
```bash
|
| 132 |
# Build and run (CPU/software rendering)
|
|
|
|
| 136 |
docker run -p 3004:3004 nova-sim
|
| 137 |
```
|
| 138 |
|
| 139 |
+
### Configuration & Tuning
|
| 140 |
|
| 141 |
+
#### Performance Optimization
|
| 142 |
|
| 143 |
Docker uses software rendering (OSMesa) which is slower than native GPU rendering. Configure these environment variables to optimize performance:
|
| 144 |
|
|
|
|
| 151 |
| `OMP_NUM_THREADS` | 4 | OpenMP thread count |
|
| 152 |
| `MKL_NUM_THREADS` | 4 | MKL thread count |
|
| 153 |
|
| 154 |
+
#### docker-compose.yml (CPU - Default)
|
| 155 |
|
| 156 |
```yaml
|
| 157 |
services:
|
|
|
|
| 167 |
- TARGET_FPS=30
|
| 168 |
```
|
| 169 |
|
| 170 |
+
#### docker-compose.gpu.yml (NVIDIA GPU)
|
| 171 |
|
| 172 |
For significantly better performance, use GPU acceleration:
|
| 173 |
|
|
|
|
| 197 |
capabilities: [gpu]
|
| 198 |
```
|
| 199 |
|
| 200 |
+
#### Performance Comparison
|
| 201 |
|
| 202 |
| Mode | Resolution | Expected FPS | Notes |
|
| 203 |
|------|------------|--------------|-------|
|
|
|
|
| 206 |
| Docker + CPU (OSMesa) | 640x360 | 15-20 | Slower, but compatible |
|
| 207 |
| Docker + CPU (OSMesa) | 320x180 | 30+ | Very low quality |
|
| 208 |
|
| 209 |
+
#### Custom Docker Run
|
| 210 |
|
| 211 |
```bash
|
| 212 |
# Ultra-low resolution for maximum speed
|
|
|
|
| 345 |
|
| 346 |
All controllers output joint position targets; MuJoCo's built-in PD control tracks these targets.
|
| 347 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 348 |
## Spot Controller Options
|
| 349 |
|
| 350 |
The Spot robot supports multiple controller types via the `controller_type` parameter:
|
|
|
|
| 526 |
- `state_streaming`: Whether using Nova API for robot state streaming (vs. internal)
|
| 527 |
- `ik`: Whether using Nova API for inverse kinematics (vs. internal)
|
| 528 |
|
| 529 |
+
### Gym WebSocket API (RL/IL)
|
| 530 |
+
|
| 531 |
+
The gym-style API is exposed at `ws://localhost:3004/nova-sim/api/v1/gym/ws`.
|
| 532 |
+
It supports `reset`, `step`, `configure`, and `get_spaces`.
|
| 533 |
+
|
| 534 |
+
Example request payloads:
|
| 535 |
+
|
| 536 |
+
```json
|
| 537 |
+
{"type": "configure", "data": {"robot": "ur5_t_push"}}
|
| 538 |
+
{"type": "reset"}
|
| 539 |
+
{"type": "step", "data": {"action": [0,0,0,0,0,0,0], "render": false}}
|
| 540 |
+
```
|
| 541 |
+
|
| 542 |
+
The server responds with `gym_reset`, `gym_step`, `gym_spaces`, or `gym_configured` messages.
|
| 543 |
+
|
| 544 |
### HTTP Endpoints
|
| 545 |
|
| 546 |
| Endpoint | Method | Description |
|