BridgeWAM commited on
Commit
fa5d408
·
verified ·
1 Parent(s): fc097ce

Add BridgeWAM model card

Browse files
Files changed (1) hide show
  1. README.md +82 -2
README.md CHANGED
@@ -114,8 +114,88 @@ python experiments/libero/run_libero_manager.py \
114
  ```
115
 
116
  The default manager evaluates all 40 tasks across LIBERO-Spatial, LIBERO-Object,
117
- LIBERO-Goal and LIBERO-Long (`libero_10`). LIBERO-Plus and LIBERO-Pro have separate
118
- environment setup instructions and evaluation entrypoints in the code repository.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
 
120
  ### RoboTwin
121
 
 
114
  ```
115
 
116
  The default manager evaluates all 40 tasks across LIBERO-Spatial, LIBERO-Object,
117
+ LIBERO-Goal and LIBERO-Long (`libero_10`).
118
+
119
+ ### LIBERO-Plus
120
+
121
+ Use the same `libero/model.pt` and `libero/dataset_stats.json` for LIBERO-Plus;
122
+ no separate Plus checkpoint is required. Install the external LIBERO-Plus
123
+ environment, its extra dependencies and simulator assets. Sensor-noise tasks
124
+ also require ImageMagick. Replace `/path/to/LIBERO-plus` below with that checkout.
125
+
126
+ The full protocol evaluates **10,030 task variants with one trial per variant**.
127
+ Each worker loads the model once and evaluates a shard of up to 50 variants.
128
+
129
+ ```bash
130
+ export ROOT_DIR="$PWD"
131
+ export LIBERO_PLUS_REPO=/path/to/LIBERO-plus
132
+ export LIBERO_ROOT="$LIBERO_PLUS_REPO"
133
+ export PYTHONPATH="$PWD/src:$PWD:$LIBERO_PLUS_REPO"
134
+
135
+ LIBERO_PLUS_DETACH=true \
136
+ bash experiments/libero-plus/run_stage1.sh \
137
+ task=libero_uncond_2cam224_lbqs_only_2layer_alternating_cross_self_fullfinetune_1e-4 \
138
+ ckpt=./checkpoints/bridgewam/libero/model.pt \
139
+ EVALUATION.dataset_stats_path=./checkpoints/bridgewam/libero/dataset_stats.json \
140
+ EVALUATION.output_dir="./runs/eval/libero-plus/$(date +%Y%m%d_%H%M%S)" \
141
+ EVALUATION.num_trials=1 \
142
+ EVALUATION.num_inference_steps=10 \
143
+ EVALUATION.save_rollout_video=true \
144
+ LIBERO_PLUS.repo_path="$LIBERO_PLUS_REPO" \
145
+ LIBERO_PLUS.expected_num_tasks=10030 \
146
+ model.skip_dit_load_from_pretrain=true \
147
+ seed=42 \
148
+ MULTIRUN.tasks_per_worker=50 \
149
+ MULTIRUN.num_gpus=8 \
150
+ MULTIRUN.max_tasks_per_gpu=5
151
+ ```
152
+
153
+ This launcher requires `tmux` and detaches the manager by default. It prints the
154
+ session name and log location; returning to the terminal does not mean evaluation
155
+ has finished. Wait for completion before starting another benchmark on the same
156
+ GPUs. Results include `libero_plus_task_results.csv` and
157
+ `libero_plus_summary.json` in the selected output directory.
158
+
159
+ ### LIBERO-Pro
160
+
161
+ Use the same LIBERO checkpoint and statistics for LIBERO-Pro. Install the external
162
+ LIBERO-PRO checkout with its benchmark task map, BDDL files, initial states and
163
+ simulator assets. The current adapter expects these resources under
164
+ `LIBERO-PRO/libero/libero/{bddl_files,init_files,assets}`. Use the same generated
165
+ resource release and environment variant for comparisons between models.
166
+
167
+ The full protocol selects **200 tasks across five perturbation categories**
168
+ (environment, position, object, language and task), with **50 trials per task**
169
+ for 10,000 episodes. These are separate perturbation categories, not their runtime
170
+ combinations. A task-count check alone does not verify resource equivalence.
171
+
172
+ ```bash
173
+ export LIBERO_PRO_REPO=/path/to/LIBERO-PRO
174
+ export LIBERO_ROOT="$LIBERO_PRO_REPO"
175
+ export PYTHONPATH="$PWD/src:$PWD:$LIBERO_PRO_REPO"
176
+
177
+ python experiments/libero-pro/run_libero_pro_manager.py \
178
+ task=libero_uncond_2cam224_lbqs_only_2layer_alternating_cross_self_fullfinetune_1e-4 \
179
+ ckpt=./checkpoints/bridgewam/libero/model.pt \
180
+ EVALUATION.dataset_stats_path=./checkpoints/bridgewam/libero/dataset_stats.json \
181
+ EVALUATION.output_dir="./runs/eval/libero-pro/$(date +%Y%m%d_%H%M%S)" \
182
+ EVALUATION.num_trials=50 \
183
+ EVALUATION.num_inference_steps=10 \
184
+ EVALUATION.save_rollout_video=true \
185
+ LIBERO_PRO.repo_path="$LIBERO_PRO_REPO" \
186
+ 'LIBERO_PRO.perturbations=[environment,position,object,language,task]' \
187
+ LIBERO_PRO.expected_num_tasks=200 \
188
+ model.skip_dit_load_from_pretrain=true \
189
+ seed=42 \
190
+ MULTIRUN.num_gpus=8 \
191
+ MULTIRUN.max_tasks_per_gpu=5
192
+ ```
193
+
194
+ The Pro manager runs in the foreground and waits for all workers; run it inside
195
+ `tmux` if it must survive a terminal disconnect. Each run writes an isolated
196
+ environment configuration, `task_manifest.jsonl`, task logs, `summary.json` and
197
+ `manager_runtime.json`. Use a fresh output directory and check that the summary
198
+ is complete; an existing summary file alone does not establish completion.
199
 
200
  ### RoboTwin
201