eval_results/summary.md

m1_mix — Evaluation Summary

A single Mem-0 execution-module checkpoint (checkpoint/m1_mix_final_step50000.pt), trained jointly on all five RMBench M1 tasks (m1_mix dataset), evaluated on each task in turn. Same checkpoint and same m1_mix normalization stats are used for every task — only --task_name and the per-task --global_task instruction change.

Results (task_config = demo_clean, instruction_type = unseen, 100 episodes/task)

Task	Success Rate	Reward	Eval timestamp
put_back_block	1.00	1.00	2026-06-22 20:00:54
rearrange_blocks	0.86	0.86	2026-06-22 09:35:33
swap_blocks	0.81	0.81	2026-06-22 09:34:12
swap_T	0.13	0.13	2026-06-22 20:01:49
observe_and_pickup	0.03	0.00	2026-06-23 05:03:56
Average	0.566	—

Block-manipulation tasks (put_back / rearrange / swap_blocks) are strong. The two weak tasks are swap_T (fine T-block pose alignment — both position and orientation) and observe_and_pickup (cross-view target identification after occlusion, then pickup).

Provenance — identical across all five evaluations

• Checkpoint: checkpoints/m1_mix/final_step50000.pt (global_step 50000)
• Normalization stats: policy/Mem-0/assets/m1_mix/norm_stats.json (min-max → [-1, 1])
• Action horizon: 30
• Entry point: script/eval_policy.py --config policy/Mem-0/deploy_policy.yml --overrides ...
• vLLM / planner: not used (M1 tasks set a single global instruction directly)

The exact --global_task string for each task is in ../task_instructions.json and is reproduced verbatim (including the original traies spelling in swap_blocks).

Per-task artifacts

Each eval_results/<task>/ folder contains:

• _result.txt — final success rate and reward
• eval_log.txt — per-episode episode_id, seed, result=Success/Fail (seeds start at 100000)
• episode<N>.mp4 — head-camera rollout video for all 100 evaluated episodes

Note: the evaluation loop counts an episode only after it passes the simulator's expert-feasibility check, so the 100 episodes correspond to seeds drawn from 100000 upward until 100 feasible inits are collected.