File size: 5,014 Bytes
b60a439 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | # SimplerEnv
Framework for evaluating real-world robot manipulation policies (RT-1, RT-1-X, Octo) in simulation. Replicates common setups like Google Robot and WidowX+Bridge, with GPU-accelerated simulations (10-15x speedup). Offers visual matching and variant aggregation evaluation methods for robust policy assessment.
For more information, see the [official repository](https://github.com/simpler-env/SimplerEnv).
---
# Fine-tune Simpler Env bridge dataset (WidowX robot)
To reproduce our finetune results, use the following commands to setup dataset and launch finetune experiments. Please remember to set `WANDB_API_KEY` since `--use-wandb` is turned on by default. If you don't have a WANDB account, please remove this argument:
```bash
uv run hf download \
--repo-type dataset IPEC-COMMUNITY/bridge_orig_lerobot \
--local-dir examples/SimplerEnv/bridge_orig_lerobot/
# Copy the patches and run the finetune script
cp examples/SimplerEnv/bridge_modality.json examples/SimplerEnv/bridge_orig_lerobot/meta/modality.json
```
```bash
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=1024 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
--base-model-path nvidia/GR00T-N1.7-3B \
--dataset-path examples/SimplerEnv/bridge_orig_lerobot/ \
--embodiment-tag SIMPLER_ENV_WIDOWX \
--output-dir /tmp/bridge_finetune \
--state-dropout-prob 0.8
```
# Fine-tune Simpler Env fractal dataset (Google robot)
```bash
uv run hf download \
--repo-type dataset IPEC-COMMUNITY/fractal20220817_data_lerobot \
--local-dir examples/SimplerEnv/fractal20220817_data_lerobot/
# Copy the patches and run the finetune script
cp -r examples/SimplerEnv/fractal_modality.json examples/SimplerEnv/fractal20220817_data_lerobot/meta/modality.json
uv run python examples/SimplerEnv/convert_av1_to_h264.py examples/SimplerEnv/fractal20220817_data_lerobot --jobs 16 # (Optional) if AV1 doesn't work on your machine
```
```bash
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=1024 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
--base-model-path nvidia/GR00T-N1.7-3B \
--dataset-path examples/SimplerEnv/fractal20220817_data_lerobot/ \
--embodiment-tag SIMPLER_ENV_GOOGLE \
--output-dir /tmp/fractal_finetune \
--state-dropout-prob 0.5
```
# Evaluate checkpoint
First, setup the evaluation simulation environment. This only needs to run once for each simulation benchmark. After it's done, we only need to launch server and client.
```bash
sudo apt update
sudo apt install libegl1-mesa-dev libglu1-mesa
bash gr00t/eval/sim/SimplerEnv/setup_SimplerEnv.sh
```
Then, run client server evaluation under the project root directory in separate terminals:
## Fractal (Google Robot) Evaluation
**Terminal 1 - Server:**
You can use either a local finetuned checkpoint path or the remote finetuned checkpoint (provided by us):
**Option 1: Local finetuned checkpoint**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path /tmp/fractal_finetune/checkpoint-30000 \
--embodiment-tag SIMPLER_ENV_GOOGLE \
--use-sim-policy-wrapper
```
**Option 2: Remote finetuned checkpoint (directly runnable)**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path nvidia/GR00T-N1.7-SimplerEnv-Fractal \
--embodiment-tag SIMPLER_ENV_GOOGLE \
--use-sim-policy-wrapper
```
**Terminal 2 - Client:**
```bash
gr00t/eval/sim/SimplerEnv/simpler_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
--n-episodes 10 \
--policy-client-host 127.0.0.1 \
--policy-client-port 5555 \
--max-episode-steps 300 \
--env-name simpler_env_google/google_robot_pick_coke_can \
--n-action-steps 1 \
--n-envs 5
```
## Bridge (WidowX) Evaluation
**Terminal 1 - Server:**
**Option 1: Local finetuned checkpoint**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path /tmp/bridge_finetune/checkpoint-30000 \
--embodiment-tag SIMPLER_ENV_WIDOWX \
--use-sim-policy-wrapper
```
**Option 2: Remote finetuned checkpoint (directly runnable)**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path nvidia/GR00T-N1.7-SimplerEnv-Bridge \
--embodiment-tag SIMPLER_ENV_WIDOWX \
--use-sim-policy-wrapper
```
**Terminal 2 - Client:**
```bash
gr00t/eval/sim/SimplerEnv/simpler_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
--n-episodes 10 \
--policy-client-host 127.0.0.1 \
--policy-client-port 5555 \
--max-episode-steps 300 \
--env-name simpler_env_widowx/widowx_spoon_on_towel \
--n-action-steps 4 \
--n-envs 5
```
Other supported tasks are:
```
simpler_env_google/google_robot_pick_object
simpler_env_google/google_robot_move_near
simpler_env_google/google_robot_open_drawer
...
simpler_env_widowx/widowx_spoon_on_towel
simpler_env_widowx/widowx_carrot_on_plate
simpler_env_widowx/widowx_stack_cube
```
you can replace the env_name with the corresponding tasks listed in the SimplerEnv fork this repo pins at `external_dependencies/SimplerEnv` (see `.gitmodules`).
|