Add BridgeWAM model card
Browse files
README.md
CHANGED
|
@@ -114,8 +114,88 @@ python experiments/libero/run_libero_manager.py \
|
|
| 114 |
```
|
| 115 |
|
| 116 |
The default manager evaluates all 40 tasks across LIBERO-Spatial, LIBERO-Object,
|
| 117 |
-
LIBERO-Goal and LIBERO-Long (`libero_10`).
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
### RoboTwin
|
| 121 |
|
|
|
|
| 114 |
```
|
| 115 |
|
| 116 |
The default manager evaluates all 40 tasks across LIBERO-Spatial, LIBERO-Object,
|
| 117 |
+
LIBERO-Goal and LIBERO-Long (`libero_10`).
|
| 118 |
+
|
| 119 |
+
### LIBERO-Plus
|
| 120 |
+
|
| 121 |
+
Use the same `libero/model.pt` and `libero/dataset_stats.json` for LIBERO-Plus;
|
| 122 |
+
no separate Plus checkpoint is required. Install the external LIBERO-Plus
|
| 123 |
+
environment, its extra dependencies and simulator assets. Sensor-noise tasks
|
| 124 |
+
also require ImageMagick. Replace `/path/to/LIBERO-plus` below with that checkout.
|
| 125 |
+
|
| 126 |
+
The full protocol evaluates **10,030 task variants with one trial per variant**.
|
| 127 |
+
Each worker loads the model once and evaluates a shard of up to 50 variants.
|
| 128 |
+
|
| 129 |
+
```bash
|
| 130 |
+
export ROOT_DIR="$PWD"
|
| 131 |
+
export LIBERO_PLUS_REPO=/path/to/LIBERO-plus
|
| 132 |
+
export LIBERO_ROOT="$LIBERO_PLUS_REPO"
|
| 133 |
+
export PYTHONPATH="$PWD/src:$PWD:$LIBERO_PLUS_REPO"
|
| 134 |
+
|
| 135 |
+
LIBERO_PLUS_DETACH=true \
|
| 136 |
+
bash experiments/libero-plus/run_stage1.sh \
|
| 137 |
+
task=libero_uncond_2cam224_lbqs_only_2layer_alternating_cross_self_fullfinetune_1e-4 \
|
| 138 |
+
ckpt=./checkpoints/bridgewam/libero/model.pt \
|
| 139 |
+
EVALUATION.dataset_stats_path=./checkpoints/bridgewam/libero/dataset_stats.json \
|
| 140 |
+
EVALUATION.output_dir="./runs/eval/libero-plus/$(date +%Y%m%d_%H%M%S)" \
|
| 141 |
+
EVALUATION.num_trials=1 \
|
| 142 |
+
EVALUATION.num_inference_steps=10 \
|
| 143 |
+
EVALUATION.save_rollout_video=true \
|
| 144 |
+
LIBERO_PLUS.repo_path="$LIBERO_PLUS_REPO" \
|
| 145 |
+
LIBERO_PLUS.expected_num_tasks=10030 \
|
| 146 |
+
model.skip_dit_load_from_pretrain=true \
|
| 147 |
+
seed=42 \
|
| 148 |
+
MULTIRUN.tasks_per_worker=50 \
|
| 149 |
+
MULTIRUN.num_gpus=8 \
|
| 150 |
+
MULTIRUN.max_tasks_per_gpu=5
|
| 151 |
+
```
|
| 152 |
+
|
| 153 |
+
This launcher requires `tmux` and detaches the manager by default. It prints the
|
| 154 |
+
session name and log location; returning to the terminal does not mean evaluation
|
| 155 |
+
has finished. Wait for completion before starting another benchmark on the same
|
| 156 |
+
GPUs. Results include `libero_plus_task_results.csv` and
|
| 157 |
+
`libero_plus_summary.json` in the selected output directory.
|
| 158 |
+
|
| 159 |
+
### LIBERO-Pro
|
| 160 |
+
|
| 161 |
+
Use the same LIBERO checkpoint and statistics for LIBERO-Pro. Install the external
|
| 162 |
+
LIBERO-PRO checkout with its benchmark task map, BDDL files, initial states and
|
| 163 |
+
simulator assets. The current adapter expects these resources under
|
| 164 |
+
`LIBERO-PRO/libero/libero/{bddl_files,init_files,assets}`. Use the same generated
|
| 165 |
+
resource release and environment variant for comparisons between models.
|
| 166 |
+
|
| 167 |
+
The full protocol selects **200 tasks across five perturbation categories**
|
| 168 |
+
(environment, position, object, language and task), with **50 trials per task**
|
| 169 |
+
for 10,000 episodes. These are separate perturbation categories, not their runtime
|
| 170 |
+
combinations. A task-count check alone does not verify resource equivalence.
|
| 171 |
+
|
| 172 |
+
```bash
|
| 173 |
+
export LIBERO_PRO_REPO=/path/to/LIBERO-PRO
|
| 174 |
+
export LIBERO_ROOT="$LIBERO_PRO_REPO"
|
| 175 |
+
export PYTHONPATH="$PWD/src:$PWD:$LIBERO_PRO_REPO"
|
| 176 |
+
|
| 177 |
+
python experiments/libero-pro/run_libero_pro_manager.py \
|
| 178 |
+
task=libero_uncond_2cam224_lbqs_only_2layer_alternating_cross_self_fullfinetune_1e-4 \
|
| 179 |
+
ckpt=./checkpoints/bridgewam/libero/model.pt \
|
| 180 |
+
EVALUATION.dataset_stats_path=./checkpoints/bridgewam/libero/dataset_stats.json \
|
| 181 |
+
EVALUATION.output_dir="./runs/eval/libero-pro/$(date +%Y%m%d_%H%M%S)" \
|
| 182 |
+
EVALUATION.num_trials=50 \
|
| 183 |
+
EVALUATION.num_inference_steps=10 \
|
| 184 |
+
EVALUATION.save_rollout_video=true \
|
| 185 |
+
LIBERO_PRO.repo_path="$LIBERO_PRO_REPO" \
|
| 186 |
+
'LIBERO_PRO.perturbations=[environment,position,object,language,task]' \
|
| 187 |
+
LIBERO_PRO.expected_num_tasks=200 \
|
| 188 |
+
model.skip_dit_load_from_pretrain=true \
|
| 189 |
+
seed=42 \
|
| 190 |
+
MULTIRUN.num_gpus=8 \
|
| 191 |
+
MULTIRUN.max_tasks_per_gpu=5
|
| 192 |
+
```
|
| 193 |
+
|
| 194 |
+
The Pro manager runs in the foreground and waits for all workers; run it inside
|
| 195 |
+
`tmux` if it must survive a terminal disconnect. Each run writes an isolated
|
| 196 |
+
environment configuration, `task_manifest.jsonl`, task logs, `summary.json` and
|
| 197 |
+
`manager_runtime.json`. Use a fresh output directory and check that the summary
|
| 198 |
+
is complete; an existing summary file alone does not establish completion.
|
| 199 |
|
| 200 |
### RoboTwin
|
| 201 |
|