| # m1_mix — Evaluation Summary |
| |
| A single Mem-0 execution-module checkpoint (`checkpoint/m1_mix_final_step50000.pt`), |
| trained jointly on all five RMBench **M1** tasks (`m1_mix` dataset), evaluated on each |
| task in turn. Same checkpoint and same `m1_mix` normalization stats are used for every |
| task — only `--task_name` and the per-task `--global_task` instruction change. |
|
|
| ## Results (task_config = `demo_clean`, instruction_type = `unseen`, 100 episodes/task) |
| |
| | Task | Success Rate | Reward | Eval timestamp | |
| |---------------------|:------------:|:------:|-----------------------| |
| | put_back_block | **1.00** | 1.00 | 2026-06-22 20:00:54 | |
| | rearrange_blocks | **0.86** | 0.86 | 2026-06-22 09:35:33 | |
| | swap_blocks | **0.81** | 0.81 | 2026-06-22 09:34:12 | |
| | swap_T | **0.13** | 0.13 | 2026-06-22 20:01:49 | |
| | observe_and_pickup | **0.03** | 0.00 | 2026-06-23 05:03:56 | |
| | **Average** | **0.566** | — | | |
|
|
| Block-manipulation tasks (put_back / rearrange / swap_blocks) are strong. The two weak |
| tasks are **swap_T** (fine T-block pose alignment — both position and orientation) and |
| **observe_and_pickup** (cross-view target identification after occlusion, then pickup). |
| |
| ## Provenance — identical across all five evaluations |
| |
| - **Checkpoint:** `checkpoints/m1_mix/final_step50000.pt` (global_step 50000) |
| - **Normalization stats:** `policy/Mem-0/assets/m1_mix/norm_stats.json` (min-max → [-1, 1]) |
| - **Action horizon:** 30 |
| - **Entry point:** `script/eval_policy.py --config policy/Mem-0/deploy_policy.yml --overrides ...` |
| - **vLLM / planner:** not used (M1 tasks set a single global instruction directly) |
|
|
| The exact `--global_task` string for each task is in `../task_instructions.json` and is |
| reproduced verbatim (including the original `traies` spelling in `swap_blocks`). |
|
|
| ## Per-task artifacts |
|
|
| Each `eval_results/<task>/` folder contains: |
|
|
| - `_result.txt` — final success rate and reward |
| - `eval_log.txt` — per-episode `episode_id, seed, result=Success/Fail` (seeds start at 100000) |
| - `episode<N>.mp4` — head-camera rollout video for all 100 evaluated episodes |
|
|
| > Note: the evaluation loop counts an episode only after it passes the simulator's |
| > expert-feasibility check, so the 100 episodes correspond to seeds drawn from 100000 |
| > upward until 100 feasible inits are collected. |
|
|
