diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..c0b33e1e21a5ce859b3fd06232529dd9ef026297 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,47 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode19.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode31.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode34.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode75.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode48.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode85.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode65.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode24.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode51.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode73.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode83.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode79.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode53.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/blocks_ranking_try/episode30.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode80.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode77.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode68.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode70.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode13.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode62.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode1.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode57.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode71.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode42.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode72.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode97.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode27.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode74.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode18.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode98.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode32.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode33.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode67.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode35.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode89.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode36.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode86.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode20.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode84.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode59.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode21.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode16.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode60.mp4 filter=lfs diff=lfs merge=lfs -text +eval_results/cover_blocks/episode95.mp4 filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..599cdad20256b7ec58ccc225f8c263e53bd5cc14 --- /dev/null +++ b/README.md @@ -0,0 +1,94 @@ +--- +license: apache-2.0 +--- + +# Mem-0 Execution Module — Mn tasks (RMBench / RoboTwin 2.0) + +**Per-task Mem-0 execution-module checkpoints** for four RMBench **Mn** tasks: +`battery_try`, `blocks_ranking_try`, `cover_blocks`, `press_button`. +Each task has its **own** executor — trained only on that task's 50-episode +`demo_clean` dataset — and its **own** normalization stats (unlike the +[`m1_mix` release](https://huggingface.co/qiuly/Mem-0-m1mix-RMBench), which used one +joint checkpoint for all five M1 tasks). + +- **Backbone:** Qwen3-VL-2B-Instruct (fine-tuned; weights bundled in each checkpoint) +- **Action head:** DiT-B flow-matching policy (16-D action) +- **Memory:** MemoryBank (instant + anchor memory fusion across the episode) +- **Aux head:** subtask-end classifier +- ≈ 2.67 B params per checkpoint; 30,000 training steps per task + +## Results + +`task_config = demo_clean`, `instruction_type = unseen`, 100 episodes per task. +`action_horizon` is the per-task inference-time setting used for these numbers. + +| Task | action_horizon | Success Rate | Reward | +|--------------------|:--------------:|:------------:|:------:| +| battery_try | 15 | **0.21** | 0.00 | +| blocks_ranking_try | 15 | **0.15** | 0.00 | +| cover_blocks | 8 | **0.76** | 0.82 | +| press_button | 30 | **0.02** | 0.00 | +| **Average** | — | **0.285** | — | + +Rollout videos (100 per task) and raw score files are under `eval_results/` +(see `eval_results/results.md`). + +## Contents + +``` +mn_submit/ +├── README.md +├── checkpoints/ +│ └── /final_step30000.pt (+ .sha256) # one full ckpt per task, ~15.3 GB each +├── norm_stats/ +│ └── /norm_stats.json # per-task min-max state/action stats +└── eval_results/ + ├── results.md # scores + per-task action_horizon + ├── success_rate.png + └── / # _result.txt, episode*.mp4 (×100) +``` + +## Checkpoints + +Each `checkpoints//final_step30000.pt` is a **single, unsplit** full training +checkpoint (~15.3 GB): `model_state_dict` + `optimizer_state_dict` + +`scheduler_state_dict`, `global_step` 30000. Verify integrity with the `.sha256` +file next to it: + +```bash +sha256sum -c final_step30000.pt.sha256 +``` + +The `model_state_dict` is self-contained — it already includes the fine-tuned +Qwen3-VL-2B backbone weights. Optional inference-only slimming (~15.3 GB → ~6 GB): + +```python +import torch +ck = torch.load("final_step30000.pt", map_location="cpu", weights_only=False) +torch.save({"model_state_dict": ck["model_state_dict"], "global_step": ck["global_step"]}, + "final_step30000_inference.pt") +``` + +## Base VLM + +Model instantiation requires the official `Qwen/Qwen3-VL-2B-Instruct` directory +(architecture + tokenizer/processor; Apache-2.0). Its weights are overwritten by the +checkpoint at load time, but the directory must exist locally: + +```bash +huggingface-cli download Qwen/Qwen3-VL-2B-Instruct \ + --local-dir policy/Mem-0/checkpoints/Qwen3-VL-2B-Instruct +``` + +## Normalization + +State and action are min-max normalized to `[-1, 1]` using the per-task stats in +`norm_stats//norm_stats.json`. Always pair a checkpoint with **its own task's** +stats at inference; predicted actions are denormalized with the same file. + +## License & attribution + +Base VLM **Qwen3-VL-2B-Instruct** is © the Qwen team, licensed **Apache-2.0**; the +checkpoints embed fine-tuned Qwen weights, so that license applies to the +corresponding components. RMBench / RoboTwin and the Mem-0 policy code are governed +by their respective upstream licenses. diff --git a/eval_results/battery_try/_result.txt b/eval_results/battery_try/_result.txt new file mode 100644 index 0000000000000000000000000000000000000000..2a5996c527883e1094acb1d97e3b1050f8d22b35 --- /dev/null +++ b/eval_results/battery_try/_result.txt @@ -0,0 +1,7 @@ +Timestamp: 2026-07-04 12:48:49 + +Instruction Type: unseen + +Success Rate: 0.21 + +Reward: 0.0 diff --git a/eval_results/blocks_ranking_try/_result.txt b/eval_results/blocks_ranking_try/_result.txt new file mode 100644 index 0000000000000000000000000000000000000000..efbe9b9f5ba2ff241af22ae1218dcc34d02d1846 --- /dev/null +++ b/eval_results/blocks_ranking_try/_result.txt @@ -0,0 +1,7 @@ +Timestamp: 2026-07-07 09:06:03 + +Instruction Type: unseen + +Success Rate: 0.15 + +Reward: 0.0 diff --git a/eval_results/blocks_ranking_try/episode19.mp4 b/eval_results/blocks_ranking_try/episode19.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..692ef7fb4d7ae3252a7e790a2a5379b8d469038a --- /dev/null +++ b/eval_results/blocks_ranking_try/episode19.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:541f9a09363b4ae3601adfb66de92eb3be898c6243b6b9d3ffcfceff98b22648 +size 3487453 diff --git a/eval_results/blocks_ranking_try/episode24.mp4 b/eval_results/blocks_ranking_try/episode24.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..888453c84666cc1029154776e665fe83da24f3f0 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode24.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1b251dad0a6e15491137e0d974c48430820e059ddb6bc266b019508d0e96cf7f +size 3022093 diff --git a/eval_results/blocks_ranking_try/episode30.mp4 b/eval_results/blocks_ranking_try/episode30.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..6f1d00e0e26d1c93447921b72aadafc249f2e3a5 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode30.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cbc1bcab40aaba37a2830dca7612e8c8cae64faf7f1350b85af79780c859c6af +size 3030288 diff --git a/eval_results/blocks_ranking_try/episode31.mp4 b/eval_results/blocks_ranking_try/episode31.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..ab60c1d6c09ca7950fd6ae70899eaba85d366cc8 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode31.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:008f3e3670166bc87c1d1501d79cc7749eb1f08742892ef96e4e83e51805b992 +size 3580679 diff --git a/eval_results/blocks_ranking_try/episode34.mp4 b/eval_results/blocks_ranking_try/episode34.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..67fe2a7531b14f81660e0cc9503d0d2d6200ea30 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode34.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:14aebf9c0bd0f263d071a928a76ba3e3fe35d6235201ae4cebe75243d040b65c +size 3509299 diff --git a/eval_results/blocks_ranking_try/episode48.mp4 b/eval_results/blocks_ranking_try/episode48.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..9f20cd8e183b292d4625f1f164f7eb19d1db0cd3 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode48.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7c6b53a8b145b4650f77c128a5111cea1329ea6bac5b90f8fbaeec867516012c +size 3167200 diff --git a/eval_results/blocks_ranking_try/episode51.mp4 b/eval_results/blocks_ranking_try/episode51.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..9b80679917c63c007ed5c100f45d7f91fd37afde --- /dev/null +++ b/eval_results/blocks_ranking_try/episode51.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:07726b3bbc35d7698d6e41bcc2d0a1a37428dd9ad1591552f5a9ebcf19a5cefa +size 3001315 diff --git a/eval_results/blocks_ranking_try/episode53.mp4 b/eval_results/blocks_ranking_try/episode53.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..ec940db0bba84c3e3a9bb0ce41116b0a12aa5fbb --- /dev/null +++ b/eval_results/blocks_ranking_try/episode53.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4b2125f058721315d8f8b73007effc7855fb34c8929637e56af26781dd162d84 +size 3584937 diff --git a/eval_results/blocks_ranking_try/episode65.mp4 b/eval_results/blocks_ranking_try/episode65.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..5be20883403de93f02ad6cfbd77f6e0a366b747c --- /dev/null +++ b/eval_results/blocks_ranking_try/episode65.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f507fe7f3ca5c3df4fc6c45aafc2378b23abaee10ad09778f4d0841ef9a941f4 +size 3508203 diff --git a/eval_results/blocks_ranking_try/episode73.mp4 b/eval_results/blocks_ranking_try/episode73.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..a08f737ad12c6b267aa5a1458a13a8f2e392e624 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode73.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b8583abd00e640d10d697f5443ea0f000eaed494ced8a350285b64b4cbc1d9e9 +size 2724360 diff --git a/eval_results/blocks_ranking_try/episode75.mp4 b/eval_results/blocks_ranking_try/episode75.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..b0cdae3533b2ae4275648bcb3024dc32d5f0cbc6 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode75.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7d5656574b35b4bd551abcb21eecd04a53b9a4ccd9c4ec7c0ca209c3be30e25c +size 3572764 diff --git a/eval_results/blocks_ranking_try/episode79.mp4 b/eval_results/blocks_ranking_try/episode79.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..1b3361f6d4570c285d9ee39247bc7e196eff662a --- /dev/null +++ b/eval_results/blocks_ranking_try/episode79.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e2fac999fd4c8c8f312ad279286c554f67be19469b80bd2fab37e6ab17876090 +size 3448833 diff --git a/eval_results/blocks_ranking_try/episode83.mp4 b/eval_results/blocks_ranking_try/episode83.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..1d27b958969e0451ca5f5a50cdfebed6beaedde0 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode83.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c40f0efb81746eede82beba4c4549c17f64662288670907ac1574c61a6a81b48 +size 3509188 diff --git a/eval_results/blocks_ranking_try/episode85.mp4 b/eval_results/blocks_ranking_try/episode85.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..f06484a8329ace2fd7ad8d159e4b193326a51498 --- /dev/null +++ b/eval_results/blocks_ranking_try/episode85.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:69534f3d1a9614588df33547b241340b56661d0451947f7f3d8a366051c973f5 +size 3503945 diff --git a/eval_results/cover_blocks/_result.txt b/eval_results/cover_blocks/_result.txt new file mode 100644 index 0000000000000000000000000000000000000000..db384899edaadf9fd3fdc32c82d30f6cf399b2f2 --- /dev/null +++ b/eval_results/cover_blocks/_result.txt @@ -0,0 +1,7 @@ +Timestamp: 2026-07-02 23:49:59 + +Instruction Type: unseen + +Success Rate: 0.76 + +Reward: 0.8199999999999998 diff --git a/eval_results/cover_blocks/episode1.mp4 b/eval_results/cover_blocks/episode1.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..93e638aab7693093acbd4e837b288b24f2797ddf --- /dev/null +++ b/eval_results/cover_blocks/episode1.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4bd5162c17db5179e52f6891df9ab5484b456a415a154846a2bba0e260ca8ee2 +size 940446 diff --git a/eval_results/cover_blocks/episode13.mp4 b/eval_results/cover_blocks/episode13.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..8760a6251d89c6d9a85fad7cc397d91ec65fc7ef --- /dev/null +++ b/eval_results/cover_blocks/episode13.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:199c66a1c28983c1f0e9418ead88b88c71855639a5264254bcee06ae0ea0d2ad +size 904800 diff --git a/eval_results/cover_blocks/episode16.mp4 b/eval_results/cover_blocks/episode16.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..e7cd13c92649f6a89122b10fc964c04aaa2a2762 --- /dev/null +++ b/eval_results/cover_blocks/episode16.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:042626497ef1c7c0d2f2395d9d35f61dd12c0cddc9ea0c8c0827f25b645b5503 +size 931238 diff --git a/eval_results/cover_blocks/episode18.mp4 b/eval_results/cover_blocks/episode18.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..f36ad86c5f7a73469a3ebd454bcf83e12879500f --- /dev/null +++ b/eval_results/cover_blocks/episode18.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b2749f1524f22a0836da4af5c2dd4603c740801805ec8e672a83dc930bd8403b +size 929733 diff --git a/eval_results/cover_blocks/episode20.mp4 b/eval_results/cover_blocks/episode20.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..b031c93c3ec89169f0fca75d5fc3dc69bfb58780 --- /dev/null +++ b/eval_results/cover_blocks/episode20.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9903cac03b020eb95590f9931c39bf454f8f52c6f5d556b92d3b4127e8756044 +size 890095 diff --git a/eval_results/cover_blocks/episode21.mp4 b/eval_results/cover_blocks/episode21.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..fa482b054566d4a3d95450c02da32258556be23b --- /dev/null +++ b/eval_results/cover_blocks/episode21.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2ce1fe72d9df2198cf998341ac4ec352db968dc109fac1fa49bf962a25a0b40a +size 875854 diff --git a/eval_results/cover_blocks/episode27.mp4 b/eval_results/cover_blocks/episode27.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..21b54752b759c99d934cd77439cc825d6cc072a9 --- /dev/null +++ b/eval_results/cover_blocks/episode27.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:45dea7feb725abc4fb1ad7259d1ddb88bb6e05563f443444e2eec2431e6e17c1 +size 864780 diff --git a/eval_results/cover_blocks/episode32.mp4 b/eval_results/cover_blocks/episode32.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..5a60c1d8508b21a99b9f9dab90555cf48c9285af --- /dev/null +++ b/eval_results/cover_blocks/episode32.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8caddfa6ae490c57a632ddebb8449f62cc3e22f8ff06e31b4e2f9c2620113b41 +size 1110181 diff --git a/eval_results/cover_blocks/episode33.mp4 b/eval_results/cover_blocks/episode33.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..f40527ae1654ad6fdd3858dd6f4f50afe14e6925 --- /dev/null +++ b/eval_results/cover_blocks/episode33.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c3276558667fd2af2493aacc736dd9c565a8c148c4b62b5fd3a8dc957522bc7f +size 926191 diff --git a/eval_results/cover_blocks/episode35.mp4 b/eval_results/cover_blocks/episode35.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..472170ff62c76c319c04cd0f71addc47c1531001 --- /dev/null +++ b/eval_results/cover_blocks/episode35.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d2a4c5e9b3a5da510c5476be10758baf8855f1e5c157351d799e93ff664acbc4 +size 898708 diff --git a/eval_results/cover_blocks/episode36.mp4 b/eval_results/cover_blocks/episode36.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..df2615d4c2ad552d9fe120f04877d1f4c521aa9f --- /dev/null +++ b/eval_results/cover_blocks/episode36.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e541c5ef5e8fbf4f8c10dd5766e6bcf7cb39dfaca2f8cf84e6c2129e7c1ce98f +size 927481 diff --git a/eval_results/cover_blocks/episode42.mp4 b/eval_results/cover_blocks/episode42.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..19bd533b93ffd035e3e214f87630d0d1f55210f5 --- /dev/null +++ b/eval_results/cover_blocks/episode42.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:91f7e109325d0b5371e521e70c4e2f950fbb83e428ed9b8634f0adb2588b8982 +size 916918 diff --git a/eval_results/cover_blocks/episode57.mp4 b/eval_results/cover_blocks/episode57.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..6cb210b786527004b79863740883ac8e679dd606 --- /dev/null +++ b/eval_results/cover_blocks/episode57.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9c4ea72c219773975ed5ff9d9eb4499a6f663128dcc4e477901ee97eb0908098 +size 945467 diff --git a/eval_results/cover_blocks/episode59.mp4 b/eval_results/cover_blocks/episode59.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..1ed7820efd0e020884c4f9d49e3dda9a9fd13cf3 --- /dev/null +++ b/eval_results/cover_blocks/episode59.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ed6445cc35757dd9663204a34893e397d6fe68237d0ab8adbb116efbd0fcf8dd +size 1068205 diff --git a/eval_results/cover_blocks/episode60.mp4 b/eval_results/cover_blocks/episode60.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..8d975ba7a9cd392efdb2f8a06dd960fdf0b3fcc9 --- /dev/null +++ b/eval_results/cover_blocks/episode60.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6ce9e8a97d729f72fbc5a48361829b8f9ef3196659cbcc9433c438ae0b154be7 +size 890276 diff --git a/eval_results/cover_blocks/episode62.mp4 b/eval_results/cover_blocks/episode62.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..41a3f3b2517d3fa709495081779a13d5af1b1781 --- /dev/null +++ b/eval_results/cover_blocks/episode62.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:54d5f75ea9afda979b3eb54565fff4c66b97f3e6768d4846b64b8c64337d580b +size 1077784 diff --git a/eval_results/cover_blocks/episode67.mp4 b/eval_results/cover_blocks/episode67.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..9e5f1d6104a9908f518db9b828da7d4b660cd1c7 --- /dev/null +++ b/eval_results/cover_blocks/episode67.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:92657f022613b27b86222e48bbdda68df57d2a36dbf7abe2b6357c2c90ded703 +size 848296 diff --git a/eval_results/cover_blocks/episode68.mp4 b/eval_results/cover_blocks/episode68.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..1d95664fa9071e1c1e2b315b8266f44e70c60b73 --- /dev/null +++ b/eval_results/cover_blocks/episode68.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4c1e90e3aa5ff4abe88d8683fd7c4fbfa021063a0d76da9ff480aeb6b7e94c33 +size 859488 diff --git a/eval_results/cover_blocks/episode70.mp4 b/eval_results/cover_blocks/episode70.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..8eae5c969b83fb4ac2668fca39875a0bbff333fe --- /dev/null +++ b/eval_results/cover_blocks/episode70.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9cde06e1b46a34635fbd5e55411ba75593a4479b3051ed0ebcc7a4f5984178cb +size 1077940 diff --git a/eval_results/cover_blocks/episode71.mp4 b/eval_results/cover_blocks/episode71.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..17935ad5191ec95fbfb1df9bd33805bc61bebfc4 --- /dev/null +++ b/eval_results/cover_blocks/episode71.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d7d757259892421b6875314f8ee4d428b39ae3d52bdb55816d7c88d8b8a2e1f4 +size 938379 diff --git a/eval_results/cover_blocks/episode72.mp4 b/eval_results/cover_blocks/episode72.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..7df6c39b98aca8d68b023ba907bddc41ab2b0696 --- /dev/null +++ b/eval_results/cover_blocks/episode72.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3ba5b8f587c37937f8d599940f737e567360828496ce5b3367a2fa5bdc42b162 +size 915580 diff --git a/eval_results/cover_blocks/episode74.mp4 b/eval_results/cover_blocks/episode74.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..124c023b0f13d3c813b251a54fa014dd1000cbb2 --- /dev/null +++ b/eval_results/cover_blocks/episode74.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3487f3672c2804dd477063be22de3287a7608d04527f598ad0d8eb72e56a19c5 +size 945642 diff --git a/eval_results/cover_blocks/episode77.mp4 b/eval_results/cover_blocks/episode77.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..b4017a72a09f75d58c737a52d9786194da72a013 --- /dev/null +++ b/eval_results/cover_blocks/episode77.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8c2afe53b7b66c50a82d0c4f5b2678647a8d25edbc8041dd941a9bfea9840a30 +size 896655 diff --git a/eval_results/cover_blocks/episode80.mp4 b/eval_results/cover_blocks/episode80.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..3da2b6e0d1e3f2f10111053a773cae6c05b8ec2b --- /dev/null +++ b/eval_results/cover_blocks/episode80.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7ddecd93c7f827f2c471ed255f1b0a8360b994dd9c2082dc4f29dc81d74090e8 +size 917542 diff --git a/eval_results/cover_blocks/episode84.mp4 b/eval_results/cover_blocks/episode84.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..7eb0657ed1fd9fa88f89f106f3174bb5853e7a69 --- /dev/null +++ b/eval_results/cover_blocks/episode84.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:691cbb4a7cd724f8a2d558eaff5317916e879bccb544213a73140bc10e49abfb +size 1109291 diff --git a/eval_results/cover_blocks/episode86.mp4 b/eval_results/cover_blocks/episode86.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..85eafffb21b2cd4fd5624525260ae1ce0e950d97 --- /dev/null +++ b/eval_results/cover_blocks/episode86.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9b1af21d5957ec401a0fe321860a5dca01db97a35a8814e4c718be2baa8ff867 +size 919599 diff --git a/eval_results/cover_blocks/episode89.mp4 b/eval_results/cover_blocks/episode89.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..6f8da2d7fe289d0ac5c5c5799b7a954d9fbc7856 --- /dev/null +++ b/eval_results/cover_blocks/episode89.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fe0e5744221958038dd5d79255823713cd0bbf62d6b617bebe6b2f69b787b1b2 +size 865050 diff --git a/eval_results/cover_blocks/episode95.mp4 b/eval_results/cover_blocks/episode95.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..a78fd069fe18df59d7b37cc34d0576dcfdea3455 --- /dev/null +++ b/eval_results/cover_blocks/episode95.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d59de5fd71e531bcb7105e53c6e5e4343d1e4a2265c005b2b722e467d4fe8a8f +size 907587 diff --git a/eval_results/cover_blocks/episode97.mp4 b/eval_results/cover_blocks/episode97.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..fa42fff012d4d388be9734f4c7c079ae52ee54ea --- /dev/null +++ b/eval_results/cover_blocks/episode97.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:afa97c667a2a33d59b50c55f712fecf6c8593a907db8b09f4d32868fb26762dc +size 1112058 diff --git a/eval_results/cover_blocks/episode98.mp4 b/eval_results/cover_blocks/episode98.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..39bc9bbb646e28573b9aa1ad3403d62a8cc78254 --- /dev/null +++ b/eval_results/cover_blocks/episode98.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5e37d86893fd34a98cd1ebe7e55eb5b1c28938ea9ee3e672918c291c386dd06b +size 912031 diff --git a/eval_results/press_button/_result.txt b/eval_results/press_button/_result.txt new file mode 100644 index 0000000000000000000000000000000000000000..1b0c5b62a0e9f2ef2da7a3b496c1e110035e84a0 --- /dev/null +++ b/eval_results/press_button/_result.txt @@ -0,0 +1,7 @@ +Timestamp: 2026-07-10 14:24:34 + +Instruction Type: unseen + +Success Rate: 0.02 + +Reward: 0.0 diff --git a/eval_results/results.md b/eval_results/results.md new file mode 100644 index 0000000000000000000000000000000000000000..1ab9aea30ebdbdf1431d1850813aa20b89b91b5e --- /dev/null +++ b/eval_results/results.md @@ -0,0 +1,29 @@ +# Mn Evaluation Results + +`task_config = demo_clean`, `instruction_type = unseen`, 100 episodes per task. +Each task is evaluated with its own separately trained execution-module checkpoint +and its own normalization stats. + +| Task | action_horizon | Success Rate | Reward | +|--------------------|:--------------:|:------------:|:------:| +| battery_try | 15 | **0.21** | 0.00 | +| blocks_ranking_try | 15 | **0.15** | 0.00 | +| cover_blocks | 8 | **0.76** | 0.82 | +| press_button | 30 | **0.02** | 0.00 | +| **Average** | — | **0.285** | — | + +## Action horizon + +Per-task inference-time `action_horizon` settings: + +- `battery_try` — **15** +- `blocks_ranking_try` — **15** +- `cover_blocks` — **8** +- `press_button` — **30** + +All results above were produced with these values. + +## Files + +- `/episode0.mp4 … episode99.mp4` — full rollout videos (100 per task) +- `/_result.txt` — raw score file for that run