Publish Codex Benchmark evaluation evidence

README.md

@@ -10,18 +10,17 @@ pinned: false
 
 # Codex Benchmark
 
-62 completed ordinary Codex tasks across RoboCasa, LIBERO Long, and RoboDojo. Results are reported separately by simulator family.
+71 published results across 4 simulator families. This edition adds 9 RoboTwin results; Lift pot is deferred by the user. Results are reported separately by family.
 
-| Family | Native successes | Tasks | Success rate |
+| Family | Native successes | Completed | Selected slots |
 | --- | ---: | ---: | ---: |
-| RoboCasa | 2 | 10 | 20.00% |
-| LIBERO Long | 9 | 10 | 90.00% |
-| RoboDojo | 31 | 42 | 73.81% |
+| RoboCasa | 2 | 10 | 10 |
+| LIBERO Long | 9 | 10 | 10 |
+| RoboTwin | 6 | 9 | 10 |
+| RoboDojo | 31 | 42 | 42 |
 
-Stock Codex CLI 0.160.0, GPT-6 Astra / high / default, seed 0, fresh independent workspace. RoboCasa and LIBERO use 6,000 steps at 20 Hz; LIBERO uses initial state 0. RoboDojo retains 7,500 frames at 25 Hz. All use stepped simulation and an 8-hour safety limit. Each task has one formal episode.
+RoboTwin uses stock Codex CLI 0.160.0, GPT-6 Astra / high / default, one independent seed-0 attempt, 7,500 native frames at 25 Hz, stepped simulation, native stop-on-success and an 8-hour safety timeout. No automatic episode retries. Current catalog instructions and native verifiers are unchanged by this campaign. Kinex source is frozen as preparation provenance; the launched agent is Codex.
 
-RoboCasa and LIBERO use the latest reviewed catalog instructions and the same native simulator source, assets, and task settings as their Kinex references. LIBERO Kinex was fully rerun at caac19a; RoboCasa retains its current a1eaece results. The intervening Kinex commits affect session timing and footer display. See [comparison.json](comparison.json) for paired results, usage, timing, and evidence boundaries.
-
-Native spectator videos are presented at 4x speed and 20 FPS. Per-task pages retain full sessions, observed images, saved tools and memos, native verdicts, instructions, protocol, usage, and source identities. Native failures are valid results; detailed failure mechanisms are unreviewed unless supported by a separate review. The four historical RoboDojo provider interruptions remain in [attempt-history.json](attempt-history.json).
+Native spectator recordings are shown at 4x speed and 20 FPS, with full sessions, observed images, resources, native verdicts, usage and source identities. No trajectory replay is claimed. Valid native failures remain results. Failure mechanisms are unreviewed unless separately supported by evidence. Previously published simulator results and interruption histories are retained.
 
 [Results](episodes.csv) · [Protocol](protocol.json) · [Comparison](comparison.json)

REPORT.md

@@ -1,17 +1,16 @@
 # Codex Benchmark
 
-62 completed ordinary Codex tasks across RoboCasa, LIBERO Long, and RoboDojo. Results are reported separately by simulator family.
+71 published results across 4 simulator families. This edition adds 9 RoboTwin results; Lift pot is deferred by the user. Results are reported separately by family.
 
-| Family | Native successes | Tasks | Success rate |
+| Family | Native successes | Completed | Selected slots |
 | --- | ---: | ---: | ---: |
-| RoboCasa | 2 | 10 | 20.00% |
-| LIBERO Long | 9 | 10 | 90.00% |
-| RoboDojo | 31 | 42 | 73.81% |
+| RoboCasa | 2 | 10 | 10 |
+| LIBERO Long | 9 | 10 | 10 |
+| RoboTwin | 6 | 9 | 10 |
+| RoboDojo | 31 | 42 | 42 |
 
-Stock Codex CLI 0.160.0, GPT-6 Astra / high / default, seed 0, fresh independent workspace. RoboCasa and LIBERO use 6,000 steps at 20 Hz; LIBERO uses initial state 0. RoboDojo retains 7,500 frames at 25 Hz. All use stepped simulation and an 8-hour safety limit. Each task has one formal episode.
+RoboTwin uses stock Codex CLI 0.160.0, GPT-6 Astra / high / default, one independent seed-0 attempt, 7,500 native frames at 25 Hz, stepped simulation, native stop-on-success and an 8-hour safety timeout. No automatic episode retries. Current catalog instructions and native verifiers are unchanged by this campaign. Kinex source is frozen as preparation provenance; the launched agent is Codex.
 
-RoboCasa and LIBERO use the latest reviewed catalog instructions and the same native simulator source, assets, and task settings as their Kinex references. LIBERO Kinex was fully rerun at caac19a; RoboCasa retains its current a1eaece results. The intervening Kinex commits affect session timing and footer display. See [comparison.json](comparison.json) for paired results, usage, timing, and evidence boundaries.
-
-Native spectator videos are presented at 4x speed and 20 FPS. Per-task pages retain full sessions, observed images, saved tools and memos, native verdicts, instructions, protocol, usage, and source identities. Native failures are valid results; detailed failure mechanisms are unreviewed unless supported by a separate review. The four historical RoboDojo provider interruptions remain in [attempt-history.json](attempt-history.json).
+Native spectator recordings are shown at 4x speed and 20 FPS, with full sessions, observed images, resources, native verdicts, usage and source identities. No trajectory replay is claimed. Valid native failures remain results. Failure mechanisms are unreviewed unless separately supported by evidence. Previously published simulator results and interruption histories are retained.
 
 [Results](episodes.csv) · [Protocol](protocol.json) · [Comparison](comparison.json)

THIRD_PARTY_NOTICES.md

@@ -14,3 +14,8 @@ LIBERO source: https://github.com/Lifelong-Robot-Learning/LIBERO/tree/8f1084e313
 Instruction clarifications for three LIBERO Long tasks draw on [BenchMend](https://benchmend-gallery.static.hf.space/gallery/index.html). Attribution: the BenchMend contributors. Additional current catalog refinements are recorded in each task instruction and source provenance.
 
 RoboDojo source: https://github.com/RoboDojo-Benchmark/RoboDojo/tree/726e9aabfaa642203722eb126f5eaf0f37f3e1ad . The pinned source carries the RoboDojo Non-Commercial Research License; see licenses/RoboDojo-LICENSE.txt. Attribution: the RoboDojo authors and the Isaac Sim, robotics and scene-asset contributors. This release presents non-commercial evaluation results, rendered views and agent evidence. No simulator asset packs, saved-layout archives or Isaac Sim software are redistributed. Underlying components retain their own terms.
+
+
+## RoboTwin source and assets
+
+RoboTwin native source is pinned to ea8b21121ebb3cd201ff5b3fe361944ac94eda3f. Simulator assets and runtime images are retained locally and are not redistributed in this result edition. See the retained source notices and per-episode source identities.

comparison.json

@@ -2297,5 +2297,791 @@
     "LIBERO Kinex uses caac19a. The retained RoboCasa Kinex reference uses a1eaece; the two intervening Kinex commits change session timing and footer display.",
     "The ordinary Codex baseline uses its native exec tools and the public RoboEnv SDK/CLI. Kinex uses its robotics tools.",
     "RoboDojo retains its previously published protocol and historical source differences."
-  ]
+  ],
+  "robotwin": {
+    "reference_repo": "RLE-Bench/kinex-v0.10.0-benchmark",
+    "reference_heads": {
+      "dataset": {
+        "sha": "b8b577c704c297f0bcf10f0bb195cb14576db4b9",
+        "files": 9026,
+        "private": false,
+        "host": null,
+        "bytes": 5217283103
+      },
+      "space": {
+        "sha": "f59eb8b6114a517c9629a3f7ae10cbd1c7ca19c2",
+        "files": 528,
+        "private": false,
+        "host": "https://rle-bench-kinex-v0-10-0-benchmark.static.hf.space",
+        "bytes": 125334443
+      }
+    },
+    "tasks": [
+      {
+        "task_key": "task03/01",
+        "codex_success": true,
+        "kinex_success": true,
+        "codex_steps": 803,
+        "kinex_steps": 1940,
+        "same_instruction": false,
+        "same_instruction_policy": false,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9529955328334399,
+          "cache_reported_input_tokens": 640003,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 640003,
+          "cached_input_tokens": 609920,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 640003,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 609920,
+          "known_input_tokens": 640003,
+          "known_output_tokens": 6267,
+          "known_reasoning_output_tokens": 1323,
+          "output_tokens": 6267,
+          "reasoning_output_tokens": 1323,
+          "reasoning_reported_output_tokens": 6267,
+          "reported_responses": {
+            "cache_reported_input_tokens": 22,
+            "cache_write_input_tokens": 22,
+            "cache_write_reported_input_tokens": 22,
+            "cached_input_tokens": 22,
+            "input_tokens": 22,
+            "output_tokens": 22,
+            "reasoning_output_tokens": 22,
+            "reasoning_reported_output_tokens": 22
+          },
+          "response_count": 22,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 30083,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 1875971,
+          "output_tokens": 11332,
+          "cached_input_tokens": 1778560,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 3668,
+          "cache_reported_input_tokens": 1875971,
+          "cache_write_reported_input_tokens": 1875971,
+          "reasoning_reported_output_tokens": 11332,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 56,
+          "reported_responses": {
+            "input_tokens": 56,
+            "output_tokens": 56,
+            "cached_input_tokens": 56,
+            "cache_write_input_tokens": 56,
+            "reasoning_output_tokens": 56,
+            "cache_reported_input_tokens": 56,
+            "cache_write_reported_input_tokens": 56,
+            "reasoning_reported_output_tokens": 56
+          },
+          "known_cached_input_tokens": 1778560,
+          "known_input_tokens": 1875971,
+          "known_output_tokens": 11332,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 3668,
+          "cache_hit_rate": 0.9480743572262045,
+          "uncached_input_tokens": 97411,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 56,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      },
+      {
+        "task_key": "task03/02",
+        "codex_success": true,
+        "kinex_success": true,
+        "codex_steps": 776,
+        "kinex_steps": 836,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9590275594676726,
+          "cache_reported_input_tokens": 1106988,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 1106988,
+          "cached_input_tokens": 1061632,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 1106988,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 1061632,
+          "known_input_tokens": 1106988,
+          "known_output_tokens": 9915,
+          "known_reasoning_output_tokens": 3875,
+          "output_tokens": 9915,
+          "reasoning_output_tokens": 3875,
+          "reasoning_reported_output_tokens": 9915,
+          "reported_responses": {
+            "cache_reported_input_tokens": 31,
+            "cache_write_input_tokens": 31,
+            "cache_write_reported_input_tokens": 31,
+            "cached_input_tokens": 31,
+            "input_tokens": 31,
+            "output_tokens": 31,
+            "reasoning_output_tokens": 31,
+            "reasoning_reported_output_tokens": 31
+          },
+          "response_count": 31,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 45356,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 1396364,
+          "output_tokens": 8842,
+          "cached_input_tokens": 1312384,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 3143,
+          "cache_reported_input_tokens": 1396364,
+          "cache_write_reported_input_tokens": 1396364,
+          "reasoning_reported_output_tokens": 8842,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 44,
+          "reported_responses": {
+            "input_tokens": 44,
+            "output_tokens": 44,
+            "cached_input_tokens": 44,
+            "cache_write_input_tokens": 44,
+            "reasoning_output_tokens": 44,
+            "cache_reported_input_tokens": 44,
+            "cache_write_reported_input_tokens": 44,
+            "reasoning_reported_output_tokens": 44
+          },
+          "known_cached_input_tokens": 1312384,
+          "known_input_tokens": 1396364,
+          "known_output_tokens": 8842,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 3143,
+          "cache_hit_rate": 0.9398580885786227,
+          "uncached_input_tokens": 83980,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 44,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true,
+          "accounting": "reported-responses"
+        }
+      },
+      {
+        "task_key": "task03/03",
+        "codex_success": false,
+        "kinex_success": true,
+        "codex_steps": 2790,
+        "kinex_steps": 2064,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9849180997132441,
+          "cache_reported_input_tokens": 3977947,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 3977947,
+          "cached_input_tokens": 3917952,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 3977947,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 3917952,
+          "known_input_tokens": 3977947,
+          "known_output_tokens": 19047,
+          "known_reasoning_output_tokens": 8202,
+          "output_tokens": 19047,
+          "reasoning_output_tokens": 8202,
+          "reasoning_reported_output_tokens": 19047,
+          "reported_responses": {
+            "cache_reported_input_tokens": 97,
+            "cache_write_input_tokens": 97,
+            "cache_write_reported_input_tokens": 97,
+            "cached_input_tokens": 97,
+            "input_tokens": 97,
+            "output_tokens": 97,
+            "reasoning_output_tokens": 97,
+            "reasoning_reported_output_tokens": 97
+          },
+          "response_count": 97,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 59995,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 1997820,
+          "output_tokens": 6541,
+          "cached_input_tokens": 1914368,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 1414,
+          "cache_reported_input_tokens": 1997820,
+          "cache_write_reported_input_tokens": 1997820,
+          "reasoning_reported_output_tokens": 6541,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 73,
+          "reported_responses": {
+            "input_tokens": 73,
+            "output_tokens": 73,
+            "cached_input_tokens": 73,
+            "cache_write_input_tokens": 73,
+            "reasoning_output_tokens": 73,
+            "cache_reported_input_tokens": 73,
+            "cache_write_reported_input_tokens": 73,
+            "reasoning_reported_output_tokens": 73
+          },
+          "known_cached_input_tokens": 1914368,
+          "known_input_tokens": 1997820,
+          "known_output_tokens": 6541,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 1414,
+          "cache_hit_rate": 0.958228469031244,
+          "uncached_input_tokens": 83452,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 73,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      },
+      {
+        "task_key": "task03/04",
+        "codex_success": false,
+        "kinex_success": false,
+        "codex_steps": 1277,
+        "kinex_steps": 1634,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": true,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9479087578186759,
+          "cache_reported_input_tokens": 1663485,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 1663485,
+          "cached_input_tokens": 1576832,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 1663485,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 1576832,
+          "known_input_tokens": 1663485,
+          "known_output_tokens": 11054,
+          "known_reasoning_output_tokens": 3714,
+          "output_tokens": 11054,
+          "reasoning_output_tokens": 3714,
+          "reasoning_reported_output_tokens": 11054,
+          "reported_responses": {
+            "cache_reported_input_tokens": 46,
+            "cache_write_input_tokens": 46,
+            "cache_write_reported_input_tokens": 46,
+            "cached_input_tokens": 46,
+            "input_tokens": 46,
+            "output_tokens": 46,
+            "reasoning_output_tokens": 46,
+            "reasoning_reported_output_tokens": 46
+          },
+          "response_count": 46,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 86653,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 2492378,
+          "output_tokens": 10795,
+          "cached_input_tokens": 2383360,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 4000,
+          "cache_reported_input_tokens": 2492378,
+          "cache_write_reported_input_tokens": 2492378,
+          "reasoning_reported_output_tokens": 10795,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 66,
+          "reported_responses": {
+            "input_tokens": 66,
+            "output_tokens": 66,
+            "cached_input_tokens": 66,
+            "cache_write_input_tokens": 66,
+            "reasoning_output_tokens": 66,
+            "cache_reported_input_tokens": 66,
+            "cache_write_reported_input_tokens": 66,
+            "reasoning_reported_output_tokens": 66
+          },
+          "known_cached_input_tokens": 2383360,
+          "known_input_tokens": 2492378,
+          "known_output_tokens": 10795,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 4000,
+          "cache_hit_rate": 0.9562594437922337,
+          "uncached_input_tokens": 109018,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "accounting": "reported-responses",
+          "usage_source": "provider-raw",
+          "request_attempts": 66,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      },
+      {
+        "task_key": "task03/05",
+        "codex_success": false,
+        "kinex_success": true,
+        "codex_steps": 3263,
+        "kinex_steps": 2172,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": true,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9794733490044869,
+          "cache_reported_input_tokens": 3072737,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 3072737,
+          "cached_input_tokens": 3009664,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 3072737,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 3009664,
+          "known_input_tokens": 3072737,
+          "known_output_tokens": 20636,
+          "known_reasoning_output_tokens": 8283,
+          "output_tokens": 20636,
+          "reasoning_output_tokens": 8283,
+          "reasoning_reported_output_tokens": 20636,
+          "reported_responses": {
+            "cache_reported_input_tokens": 71,
+            "cache_write_input_tokens": 71,
+            "cache_write_reported_input_tokens": 71,
+            "cached_input_tokens": 71,
+            "input_tokens": 71,
+            "output_tokens": 71,
+            "reasoning_output_tokens": 71,
+            "reasoning_reported_output_tokens": 71
+          },
+          "response_count": 71,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 63073,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 2470735,
+          "output_tokens": 13511,
+          "cached_input_tokens": 2363776,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 4117,
+          "cache_reported_input_tokens": 2470735,
+          "cache_write_reported_input_tokens": 2470735,
+          "reasoning_reported_output_tokens": 13511,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 70,
+          "reported_responses": {
+            "input_tokens": 70,
+            "output_tokens": 70,
+            "cached_input_tokens": 70,
+            "cache_write_input_tokens": 70,
+            "reasoning_output_tokens": 70,
+            "cache_reported_input_tokens": 70,
+            "cache_write_reported_input_tokens": 70,
+            "reasoning_reported_output_tokens": 70
+          },
+          "known_cached_input_tokens": 2363776,
+          "known_input_tokens": 2470735,
+          "known_output_tokens": 13511,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 4117,
+          "cache_hit_rate": 0.956709643081917,
+          "uncached_input_tokens": 106959,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "accounting": "reported-responses",
+          "usage_source": "provider-raw",
+          "request_attempts": 70,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      },
+      {
+        "task_key": "task03/06",
+        "codex_success": true,
+        "kinex_success": true,
+        "codex_steps": 905,
+        "kinex_steps": 2602,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.961291401007713,
+          "cache_reported_input_tokens": 921294,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 921294,
+          "cached_input_tokens": 885632,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 921294,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 885632,
+          "known_input_tokens": 921294,
+          "known_output_tokens": 6531,
+          "known_reasoning_output_tokens": 1389,
+          "output_tokens": 6531,
+          "reasoning_output_tokens": 1389,
+          "reasoning_reported_output_tokens": 6531,
+          "reported_responses": {
+            "cache_reported_input_tokens": 28,
+            "cache_write_input_tokens": 28,
+            "cache_write_reported_input_tokens": 28,
+            "cached_input_tokens": 28,
+            "input_tokens": 28,
+            "output_tokens": 28,
+            "reasoning_output_tokens": 28,
+            "reasoning_reported_output_tokens": 28
+          },
+          "response_count": 28,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 35662,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 1123532,
+          "output_tokens": 9317,
+          "cached_input_tokens": 1035136,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 2828,
+          "cache_reported_input_tokens": 1123532,
+          "cache_write_reported_input_tokens": 1123532,
+          "reasoning_reported_output_tokens": 9317,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 34,
+          "reported_responses": {
+            "input_tokens": 34,
+            "output_tokens": 34,
+            "cached_input_tokens": 34,
+            "cache_write_input_tokens": 34,
+            "reasoning_output_tokens": 34,
+            "cache_reported_input_tokens": 34,
+            "cache_write_reported_input_tokens": 34,
+            "reasoning_reported_output_tokens": 34
+          },
+          "known_cached_input_tokens": 1035136,
+          "known_input_tokens": 1123532,
+          "known_output_tokens": 9317,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 2828,
+          "cache_hit_rate": 0.921323113182357,
+          "uncached_input_tokens": 88396,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 34,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      },
+      {
+        "task_key": "task03/07",
+        "codex_success": true,
+        "kinex_success": true,
+        "codex_steps": 870,
+        "kinex_steps": 3537,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9695526652578946,
+          "cache_reported_input_tokens": 1343960,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 1343960,
+          "cached_input_tokens": 1303040,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 1343960,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 1303040,
+          "known_input_tokens": 1343960,
+          "known_output_tokens": 10223,
+          "known_reasoning_output_tokens": 3099,
+          "output_tokens": 10223,
+          "reasoning_output_tokens": 3099,
+          "reasoning_reported_output_tokens": 10223,
+          "reported_responses": {
+            "cache_reported_input_tokens": 39,
+            "cache_write_input_tokens": 39,
+            "cache_write_reported_input_tokens": 39,
+            "cached_input_tokens": 39,
+            "input_tokens": 39,
+            "output_tokens": 39,
+            "reasoning_output_tokens": 39,
+            "reasoning_reported_output_tokens": 39
+          },
+          "response_count": 39,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 40920,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 6184301,
+          "output_tokens": 24218,
+          "cached_input_tokens": 6022016,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 9092,
+          "cache_reported_input_tokens": 6184301,
+          "cache_write_reported_input_tokens": 6184301,
+          "reasoning_reported_output_tokens": 24218,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 132,
+          "reported_responses": {
+            "input_tokens": 132,
+            "output_tokens": 132,
+            "cached_input_tokens": 132,
+            "cache_write_input_tokens": 132,
+            "reasoning_output_tokens": 132,
+            "cache_reported_input_tokens": 132,
+            "cache_write_reported_input_tokens": 132,
+            "reasoning_reported_output_tokens": 132
+          },
+          "known_cached_input_tokens": 6022016,
+          "known_input_tokens": 6184301,
+          "known_output_tokens": 24218,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 9092,
+          "cache_hit_rate": 0.9737585541195359,
+          "uncached_input_tokens": 162285,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 133,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true,
+          "accounting": "reported-responses"
+        }
+      },
+      {
+        "task_key": "task03/08",
+        "codex_success": true,
+        "kinex_success": true,
+        "codex_steps": 433,
+        "kinex_steps": 1399,
+        "same_instruction": true,
+        "same_instruction_policy": true,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9536282677980344,
+          "cache_reported_input_tokens": 683067,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 683067,
+          "cached_input_tokens": 651392,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 683067,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 651392,
+          "known_input_tokens": 683067,
+          "known_output_tokens": 5483,
+          "known_reasoning_output_tokens": 994,
+          "output_tokens": 5483,
+          "reasoning_output_tokens": 994,
+          "reasoning_reported_output_tokens": 5483,
+          "reported_responses": {
+            "cache_reported_input_tokens": 22,
+            "cache_write_input_tokens": 22,
+            "cache_write_reported_input_tokens": 22,
+            "cached_input_tokens": 22,
+            "input_tokens": 22,
+            "output_tokens": 22,
+            "reasoning_output_tokens": 22,
+            "reasoning_reported_output_tokens": 22
+          },
+          "response_count": 22,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 31675,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 1051290,
+          "output_tokens": 6748,
+          "cached_input_tokens": 972160,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 1687,
+          "cache_reported_input_tokens": 1051290,
+          "cache_write_reported_input_tokens": 1051290,
+          "reasoning_reported_output_tokens": 6748,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 36,
+          "reported_responses": {
+            "input_tokens": 36,
+            "output_tokens": 36,
+            "cached_input_tokens": 36,
+            "cache_write_input_tokens": 36,
+            "reasoning_output_tokens": 36,
+            "cache_reported_input_tokens": 36,
+            "cache_write_reported_input_tokens": 36,
+            "reasoning_reported_output_tokens": 36
+          },
+          "known_cached_input_tokens": 972160,
+          "known_input_tokens": 1051290,
+          "known_output_tokens": 6748,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 1687,
+          "cache_hit_rate": 0.9247305691103311,
+          "uncached_input_tokens": 79130,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 36,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      },
+      {
+        "task_key": "task03/09",
+        "codex_success": true,
+        "kinex_success": true,
+        "codex_steps": 1097,
+        "kinex_steps": 2705,
+        "same_instruction": false,
+        "same_instruction_policy": false,
+        "same_roboenv_source": false,
+        "codex_usage": {
+          "accounting": "reported-responses",
+          "audit_complete": true,
+          "cache_hit_rate": 0.9593260437553782,
+          "cache_reported_input_tokens": 1228329,
+          "cache_write_input_tokens": 0,
+          "cache_write_reported_input_tokens": 1228329,
+          "cached_input_tokens": 1178368,
+          "completed_turns": 1,
+          "cost_usd": null,
+          "failed_turns": 0,
+          "input_tokens": 1228329,
+          "known_cache_write_input_tokens": 0,
+          "known_cached_input_tokens": 1178368,
+          "known_input_tokens": 1228329,
+          "known_output_tokens": 8401,
+          "known_reasoning_output_tokens": 2418,
+          "output_tokens": 8401,
+          "reasoning_output_tokens": 2418,
+          "reasoning_reported_output_tokens": 8401,
+          "reported_responses": {
+            "cache_reported_input_tokens": 34,
+            "cache_write_input_tokens": 34,
+            "cache_write_reported_input_tokens": 34,
+            "cached_input_tokens": 34,
+            "input_tokens": 34,
+            "output_tokens": 34,
+            "reasoning_output_tokens": 34,
+            "reasoning_reported_output_tokens": 34
+          },
+          "response_count": 34,
+          "response_ids_complete": true,
+          "schema": "rlebench/token-usage/1",
+          "source": "Codex token_usage_record per response",
+          "uncached_input_tokens": 49961,
+          "unidentified_usage_records": 0
+        },
+        "kinex_usage": {
+          "input_tokens": 1739536,
+          "output_tokens": 8301,
+          "cached_input_tokens": 1646592,
+          "cache_write_input_tokens": 0,
+          "reasoning_output_tokens": 2232,
+          "cache_reported_input_tokens": 1739536,
+          "cache_write_reported_input_tokens": 1739536,
+          "reasoning_reported_output_tokens": 8301,
+          "schema": "rlebench/token-usage/1",
+          "source": "Provider terminal response usage",
+          "response_count": 53,
+          "reported_responses": {
+            "input_tokens": 53,
+            "output_tokens": 53,
+            "cached_input_tokens": 53,
+            "cache_write_input_tokens": 53,
+            "reasoning_output_tokens": 53,
+            "cache_reported_input_tokens": 53,
+            "cache_write_reported_input_tokens": 53,
+            "reasoning_reported_output_tokens": 53
+          },
+          "known_cached_input_tokens": 1646592,
+          "known_input_tokens": 1739536,
+          "known_output_tokens": 8301,
+          "known_cache_write_input_tokens": 0,
+          "known_reasoning_output_tokens": 2232,
+          "cache_hit_rate": 0.9465696599552984,
+          "uncached_input_tokens": 92944,
+          "cost_usd": null,
+          "response_ids_complete": true,
+          "usage_source": "provider-raw",
+          "request_attempts": 53,
+          "rejected_attempts": 0,
+          "unresolved_attempts": [],
+          "audit_complete": true
+        }
+      }
+    ],
+    "limits": [
+      "Single seed and one attempt per task. Historical Kinex references may have different source or instruction policy; equality is reported per task.",
+      "GPU concurrency differs; wall time is not a controlled speed comparison.",
+      "Lift pot is deferred and excluded from the new Codex results."
+    ]
+  }
 }

episodes.csv

@@ -1,63 +1,73 @@
-task_key,native_id,status,run_status,success,steps,input_tokens,cached_input_tokens,output_tokens
-task01/01,robocasa/load-condiments-in-fridge,completed,finished,False,5496,7077229,6989312,20496
-task01/02,robocasa/filter-microwavable-item,completed,finished,False,5903,4438106,4354176,22048
-task01/03,robocasa/store-dumplings,completed,finished,False,5053,11115392,11003904,29598
-task01/04,robocasa/divide-buffet-trays,completed,finished,False,5435,11714063,11596032,32111
-task01/05,robocasa/make-cheesecake-filling,completed,finished,True,5224,12355153,12213888,35229
-task01/06,robocasa/multistep-steaming,completed,finished,False,5416,9630097,9514624,24897
-task01/07,robocasa/scale-portioning,completed,finished,False,1426,1217225,1176320,8600
-task01/08,robocasa/scrub-cutting-board,completed,finished,False,1212,775611,745088,6607
-task01/09,robocasa/prepare-veggie-dip,completed,finished,False,2216,1973478,1920896,11695
-task01/10,robocasa/prepare-vegetable-roasting,completed,finished,True,3969,2989000,2922496,13452
-task02/01,libero/10-0,completed,finished,True,969,832121,791296,6802
-task02/02,libero/10-1,completed,finished,True,722,784518,746496,6225
-task02/03,libero/10-2,completed,finished,True,2013,2225094,2152064,13673
-task02/04,libero/bowl-into-bottom-drawer,completed,finished,False,3687,3356377,3271040,23446
-task02/05,libero/10-4,completed,finished,True,634,509662,483840,5212
-task02/06,libero/10-5,completed,finished,True,325,707210,676224,6550
-task02/07,libero/10-6,completed,finished,True,1016,701841,665856,6754
-task02/08,libero/10-7,completed,finished,True,1115,1406070,1350400,9396
-task02/09,libero/10-8,completed,finished,True,560,714527,682496,5355
-task02/10,libero/mug-into-microwave,completed,finished,True,1210,1349079,1287040,10556
-task04/01,robodojo/make-toast,completed,finished,False,6884,13262282,13103360,42295
-task04/02,robodojo/classify-objects-by-language,completed,finished,True,1642,1667619,1598848,9371
-task04/03,robodojo/store-laptop-and-headphones,completed,finished,False,5696,18189657,17666048,45612
-task04/04,robodojo/cover-blocks,completed,finished,True,2184,2163308,2107392,9584
-task04/05,robodojo/play-xylophone,completed,finished,True,818,1397747,1321344,9368
-task04/06,robodojo/store-tools-in-toolbox,completed,finished,False,7421,15593052,15416704,57648
-task04/07,robodojo/insert-tubes,completed,finished,True,4162,11924908,11788288,45245
-task04/08,robodojo/deposit-coin,completed,finished,True,1432,2957393,2867840,15930
-task04/09,robodojo/fasten-screws,completed,finished,True,7340,5986459,5888000,23452
-task04/10,robodojo/play-stacking-toy,completed,finished,True,2460,4468354,4390912,20623
-task04/11,robodojo/align-blocks,completed,finished,True,905,1207263,1170304,7588
-task04/12,robodojo/arrange-largest-number,completed,finished,True,2040,2403089,2343936,14542
-task04/14,robodojo/build-tower,completed,finished,True,3359,4541614,4394368,24568
-task04/15,robodojo/classify-objects,completed,finished,True,2596,4079082,3972608,15725
-task04/16,robodojo/fill-egg-holder,completed,finished,True,3772,10433608,10253056,29150
-task04/17,robodojo/fill-pen-holder,completed,finished,False,7440,29201838,28982912,80788
-task04/18,robodojo/fold-clothes,completed,finished,True,2826,4706237,4633088,20162
-task04/20,robodojo/general-pickup,completed,finished,True,260,890742,818688,5727
-task04/21,robodojo/hang-mugs,completed,finished,True,3649,8564328,8449664,31426
-task04/23,robodojo/imitate-sorting-sequence,completed,finished,True,2522,2056606,1965312,10680
-task04/24,robodojo/insert-key,completed,finished,True,6895,30834346,30588160,88978
-task04/25,robodojo/make-kong,completed,finished,False,2476,4991204,4908544,24592
-task04/27,robodojo/match-and-pick-from-conveyor,completed,finished,True,484,1059308,1022976,5961
-task04/28,robodojo/organize-table,completed,finished,False,4143,6791820,6688768,30892
-task04/29,robodojo/pack-objects-into-box,completed,finished,True,6433,12341824,12166656,39288
-task04/31,robodojo/pick-from-conveyor-by-image,completed,finished,False,863,3797181,3703168,23800
-task04/32,robodojo/play-tic-tac-toe,completed,finished,True,2665,3137952,3077888,12655
-task04/33,robodojo/plug-in-charger,completed,finished,True,964,2129540,2076032,9894
-task04/34,robodojo/pour-balls-into-vase,completed,finished,True,5938,10284206,10160640,35628
-task04/35,robodojo/pour-by-language,completed,finished,False,1616,2677076,2588032,11360
-task04/36,robodojo/pour-liquid-into-cup,completed,finished,True,1071,2577693,2518144,15029
-task04/38,robodojo/press-by-number,completed,finished,True,952,1030503,987136,6991
-task04/39,robodojo/push-t,completed,finished,False,535,787536,751232,7374
-task04/41,robodojo/put-bottles-into-dustbin,completed,finished,True,2440,4211396,4139776,19241
-task04/42,robodojo/solve-equation,completed,finished,True,376,899491,850304,7148
-task04/43,robodojo/sort-nesting-dolls-by-size,completed,finished,True,3036,4182694,4058752,19444
-task04/45,robodojo/stack-blocks,completed,finished,True,843,1114064,1055104,6855
-task04/46,robodojo/stack-blocks-by-language,completed,finished,True,1265,1362092,1320064,9240
-task04/48,robodojo/stack-bowls,completed,finished,True,1209,972152,934144,6498
-task04/51,robodojo/swap-t,completed,finished,True,1046,2182726,2127488,11222
-task04/52,robodojo/swap-blocks,completed,finished,False,1447,1785448,1735168,11732
-task04/53,robodojo/sweep-blocks,completed,finished,False,6753,14232661,14088064,48099
+task_key,native_id,status,success,steps,input_tokens,cached_input_tokens,output_tokens
+task01/01,robocasa/load-condiments-in-fridge,completed,False,5496,7077229,6989312,20496
+task01/02,robocasa/filter-microwavable-item,completed,False,5903,4438106,4354176,22048
+task01/03,robocasa/store-dumplings,completed,False,5053,11115392,11003904,29598
+task01/04,robocasa/divide-buffet-trays,completed,False,5435,11714063,11596032,32111
+task01/05,robocasa/make-cheesecake-filling,completed,True,5224,12355153,12213888,35229
+task01/06,robocasa/multistep-steaming,completed,False,5416,9630097,9514624,24897
+task01/07,robocasa/scale-portioning,completed,False,1426,1217225,1176320,8600
+task01/08,robocasa/scrub-cutting-board,completed,False,1212,775611,745088,6607
+task01/09,robocasa/prepare-veggie-dip,completed,False,2216,1973478,1920896,11695
+task01/10,robocasa/prepare-vegetable-roasting,completed,True,3969,2989000,2922496,13452
+task02/01,libero/10-0,completed,True,969,832121,791296,6802
+task02/02,libero/10-1,completed,True,722,784518,746496,6225
+task02/03,libero/10-2,completed,True,2013,2225094,2152064,13673
+task02/04,libero/bowl-into-bottom-drawer,completed,False,3687,3356377,3271040,23446
+task02/05,libero/10-4,completed,True,634,509662,483840,5212
+task02/06,libero/10-5,completed,True,325,707210,676224,6550
+task02/07,libero/10-6,completed,True,1016,701841,665856,6754
+task02/08,libero/10-7,completed,True,1115,1406070,1350400,9396
+task02/09,libero/10-8,completed,True,560,714527,682496,5355
+task02/10,libero/mug-into-microwave,completed,True,1210,1349079,1287040,10556
+task03/01,robotwin/handover-block-aloha-agilex,completed,True,803,640003,609920,6267
+task03/02,robotwin/place-dual-shoes-aloha-agilex,completed,True,776,1106988,1061632,9915
+task03/03,robotwin/put-bottles-dustbin-aloha-agilex,completed,False,2790,3977947,3917952,19047
+task03/04,robotwin/scan-object-aloha-agilex,completed,False,1277,1663485,1576832,11054
+task03/05,robotwin/stack-bowls-three-aloha-agilex,completed,False,3263,3072737,3009664,20636
+task03/06,robotwin/stack-blocks-three-aloha-agilex,completed,True,905,921294,885632,6531
+task03/07,robotwin/hanging-mug-aloha-agilex,completed,True,870,1343960,1303040,10223
+task03/08,robotwin/put-object-cabinet-aloha-agilex,completed,True,433,683067,651392,5483
+task03/09,robotwin/blocks-ranking-size-aloha-agilex,completed,True,1097,1228329,1178368,8401
+task03/10,robotwin/lift-pot-aloha-agilex,pending,,,,,
+task04/01,robodojo/make-toast,completed,False,6884,13262282,13103360,42295
+task04/02,robodojo/classify-objects-by-language,completed,True,1642,1667619,1598848,9371
+task04/03,robodojo/store-laptop-and-headphones,completed,False,5696,18189657,17666048,45612
+task04/04,robodojo/cover-blocks,completed,True,2184,2163308,2107392,9584
+task04/05,robodojo/play-xylophone,completed,True,818,1397747,1321344,9368
+task04/06,robodojo/store-tools-in-toolbox,completed,False,7421,15593052,15416704,57648
+task04/07,robodojo/insert-tubes,completed,True,4162,11924908,11788288,45245
+task04/08,robodojo/deposit-coin,completed,True,1432,2957393,2867840,15930
+task04/09,robodojo/fasten-screws,completed,True,7340,5986459,5888000,23452
+task04/10,robodojo/play-stacking-toy,completed,True,2460,4468354,4390912,20623
+task04/11,robodojo/align-blocks,completed,True,905,1207263,1170304,7588
+task04/12,robodojo/arrange-largest-number,completed,True,2040,2403089,2343936,14542
+task04/14,robodojo/build-tower,completed,True,3359,4541614,4394368,24568
+task04/15,robodojo/classify-objects,completed,True,2596,4079082,3972608,15725
+task04/16,robodojo/fill-egg-holder,completed,True,3772,10433608,10253056,29150
+task04/17,robodojo/fill-pen-holder,completed,False,7440,29201838,28982912,80788
+task04/18,robodojo/fold-clothes,completed,True,2826,4706237,4633088,20162
+task04/20,robodojo/general-pickup,completed,True,260,890742,818688,5727
+task04/21,robodojo/hang-mugs,completed,True,3649,8564328,8449664,31426
+task04/23,robodojo/imitate-sorting-sequence,completed,True,2522,2056606,1965312,10680
+task04/24,robodojo/insert-key,completed,True,6895,30834346,30588160,88978
+task04/25,robodojo/make-kong,completed,False,2476,4991204,4908544,24592
+task04/27,robodojo/match-and-pick-from-conveyor,completed,True,484,1059308,1022976,5961
+task04/28,robodojo/organize-table,completed,False,4143,6791820,6688768,30892
+task04/29,robodojo/pack-objects-into-box,completed,True,6433,12341824,12166656,39288
+task04/31,robodojo/pick-from-conveyor-by-image,completed,False,863,3797181,3703168,23800
+task04/32,robodojo/play-tic-tac-toe,completed,True,2665,3137952,3077888,12655
+task04/33,robodojo/plug-in-charger,completed,True,964,2129540,2076032,9894
+task04/34,robodojo/pour-balls-into-vase,completed,True,5938,10284206,10160640,35628
+task04/35,robodojo/pour-by-language,completed,False,1616,2677076,2588032,11360
+task04/36,robodojo/pour-liquid-into-cup,completed,True,1071,2577693,2518144,15029
+task04/38,robodojo/press-by-number,completed,True,952,1030503,987136,6991
+task04/39,robodojo/push-t,completed,False,535,787536,751232,7374
+task04/41,robodojo/put-bottles-into-dustbin,completed,True,2440,4211396,4139776,19241
+task04/42,robodojo/solve-equation,completed,True,376,899491,850304,7148
+task04/43,robodojo/sort-nesting-dolls-by-size,completed,True,3036,4182694,4058752,19444
+task04/45,robodojo/stack-blocks,completed,True,843,1114064,1055104,6855
+task04/46,robodojo/stack-blocks-by-language,completed,True,1265,1362092,1320064,9240
+task04/48,robodojo/stack-bowls,completed,True,1209,972152,934144,6498
+task04/51,robodojo/swap-t,completed,True,1046,2182726,2127488,11222
+task04/52,robodojo/swap-blocks,completed,False,1447,1785448,1735168,11732
+task04/53,robodojo/sweep-blocks,completed,False,6753,14232661,14088064,48099

index.html

@@ -1,5 +1,5 @@
 <!doctype html>
-<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="description" content="Codex robotics benchmark: original and revised instructions, seed 0, GPT-6 Astra high. Explore task videos, sessions and evidence across RoboCasa, LIBERO and RoboDojo."><title>Codex Benchmark · RLE-Bench</title><link rel="icon" href="favicon.svg"><link rel="stylesheet" href="style.css"><script defer src="app.js"></script></head>
-<body><header class="hero"><div class="wrap"><div class="topbar"><a class="brand" href="#"><span class="brand-icon" aria-hidden="true">◇</span> RLE-Bench</a><nav aria-label="Primary"><a href="#tasks">Tasks</a><a href="https://huggingface.co/datasets/RLE-Bench/codex-benchmark" target="_blank" rel="noopener">Dataset ↗</a><a href="REPORT.md" target="_blank" rel="noopener">Report ↗</a></nav></div><div class="hero-content"><div class="eyebrow">ROBOTICS EVALUATION / STOCK CODEX</div><h1>Codex Benchmark</h1><p>GPT-6 Astra · high · seed 0 · original &amp; modified instructions</p><div class="hero-bottom"><span>62 tasks across three environments. Native success criteria.</span><a class="hero-button" href="#task04">Explore the results <span>↗</span></a></div></div></div></header>
-<main class="wrap"><div id="loading" role="status">Loading benchmark…</div><div id="browse" hidden><div class="edition"><span id="edition-count">Loading results…</span><span class="chip" id="edition-label">Benchmark results</span></div><section aria-labelledby="overview-heading"><div class="section-top"><h2 id="overview-heading">Task overview</h2><a href="summary.json" target="_blank" rel="noopener">Summary JSON ↗</a></div><div class="table-wrap"><table><thead><tr><th>Environment</th><th>Coverage</th><th>Native success</th><th>Input tokens</th><th>Output tokens</th><th>Usage coverage</th></tr></thead><tbody id="overview"></tbody></table></div><p class="footnote">Current recorded result per task · Seed 0 · Native success criteria. Each task shows its latest result and the instruction used for that recording. Rates include valid completed results only; pending tasks are excluded. No pooled score across environments. Input includes cached tokens; output includes reasoning tokens.</p></section><section id="failure-overview" lang="zh-CN" aria-label="失败原因"></section><p id="readiness-note" class="readiness-note" hidden></p><div id="tasks" class="task-controls"><nav id="family-nav" aria-label="Task families"></nav><div class="filters"><label class="search-label"><span class="sr-only">Search tasks</span><input id="search" type="search" placeholder="Search 62 tasks…"></label><label><span class="sr-only">Filter by status</span><select id="status-filter"><option value="all">All tasks</option><option value="completed">Results available</option><option value="pending">Pending</option></select></label><label><span class="sr-only">Filter by failure category</span><select id="failure-filter" aria-label="Filter by failure category"><option value="all">All failure categories</option></select></label><span id="filter-count" aria-live="polite"></span></div></div><div id="families"></div><p id="empty" hidden>No tasks match this search.</p><section class="protocol-box"><h2>The evaluation protocol</h2><div class="protocol-grid"><div><b>Instruction versions</b><p>Modified instructions are labeled, with changes highlighted. Each recording retains the instruction it used; native success predicates are unchanged.</p></div><div><b>Single seed</b><p>One independent episode, seed 0. GPT-6 Astra with high reasoning effort. Stock Codex CLI 0.160.0; each task starts with a fresh workspace and no inherited trajectory.</p></div><div><b>Native control budgets</b><p>RoboCasa and LIBERO: 6,000 steps at 20 Hz. RoboDojo: 7,500 frames at 25 Hz. Stepped simulation, with an eight-hour wall-clock limit.</p></div></div></section></div><article id="detail" hidden></article></main>
+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="description" content="Codex robotics benchmark: original and revised instructions, seed 0, GPT-6 Astra high. Explore task videos, sessions and evidence across RoboCasa, LIBERO, RoboTwin and RoboDojo."><title>Codex Benchmark · RLE-Bench</title><link rel="icon" href="favicon.svg"><link rel="stylesheet" href="style.css"><script defer src="app.js"></script></head>
+<body><header class="hero"><div class="wrap"><div class="topbar"><a class="brand" href="#"><span class="brand-icon" aria-hidden="true">◇</span> RLE-Bench</a><nav aria-label="Primary"><a href="#tasks">Tasks</a><a href="https://huggingface.co/datasets/RLE-Bench/codex-benchmark" target="_blank" rel="noopener">Dataset ↗</a><a href="REPORT.md" target="_blank" rel="noopener">Report ↗</a></nav></div><div class="hero-content"><div class="eyebrow">ROBOTICS EVALUATION / STOCK CODEX</div><h1>Codex Benchmark</h1><p>GPT-6 Astra · high · seed 0 · original &amp; modified instructions</p><div class="hero-bottom"><span>72 selected slots across four environments. Native success criteria.</span><a class="hero-button" href="#task04">Explore the results <span>↗</span></a></div></div></div></header>
+<main class="wrap"><div id="loading" role="status">Loading benchmark…</div><div id="browse" hidden><div class="edition"><span id="edition-count">Loading results…</span><span class="chip" id="edition-label">Benchmark results</span></div><section aria-labelledby="overview-heading"><div class="section-top"><h2 id="overview-heading">Task overview</h2><a href="summary.json" target="_blank" rel="noopener">Summary JSON ↗</a></div><div class="table-wrap"><table><thead><tr><th>Environment</th><th>Coverage</th><th>Native success</th><th>Input tokens</th><th>Output tokens</th><th>Usage coverage</th></tr></thead><tbody id="overview"></tbody></table></div><p class="footnote">Current recorded result per task · Seed 0 · Native success criteria. Each task shows its latest result and the instruction used for that recording. Rates include valid completed results only; pending tasks are excluded. No pooled score across environments. Input includes cached tokens; output includes reasoning tokens.</p></section><section id="failure-overview" lang="zh-CN" aria-label="失败原因"></section><p id="readiness-note" class="readiness-note" hidden></p><div id="tasks" class="task-controls"><nav id="family-nav" aria-label="Task families"></nav><div class="filters"><label class="search-label"><span class="sr-only">Search tasks</span><input id="search" type="search" placeholder="Search 72 tasks…"></label><label><span class="sr-only">Filter by status</span><select id="status-filter"><option value="all">All tasks</option><option value="completed">Results available</option><option value="pending">Pending</option></select></label><label><span class="sr-only">Filter by failure category</span><select id="failure-filter" aria-label="Filter by failure category"><option value="all">All failure categories</option></select></label><span id="filter-count" aria-live="polite"></span></div></div><div id="families"></div><p id="empty" hidden>No tasks match this search.</p><section class="protocol-box"><h2>The evaluation protocol</h2><div class="protocol-grid"><div><b>Instruction versions</b><p>Modified instructions are labeled, with changes highlighted. Each recording retains the instruction it used; native success predicates are unchanged.</p></div><div><b>Single seed</b><p>One independent episode, seed 0. GPT-6 Astra with high reasoning effort. Stock Codex CLI 0.160.0; each task starts with a fresh workspace and no inherited trajectory.</p></div><div><b>Native control budgets</b><p>RoboCasa and LIBERO: 6,000 steps at 20 Hz. RoboDojo: 7,500 frames at 25 Hz. Stepped simulation, with an eight-hour wall-clock limit.</p></div></div></section></div><article id="detail" hidden></article></main>
 <footer class="wrap"><span>RLE-Bench · Codex Benchmark</span><nav aria-label="Downloads"><a href="data.json">Data JSON</a><a href="episodes.csv">CSV</a><a href="DATA_FORMAT.md">Data format</a><a href="THIRD_PARTY_NOTICES.md">Attribution</a><a href="https://huggingface.co/datasets/RLE-Bench/codex-benchmark">Dataset ↗</a></nav></footer></body></html>

licenses/RoboTwin-LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Tianxing Chen (陈天行)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

manifest.json

@@ -1,24 +1,25 @@
 {
   ".gitattributes": "51dec59a208c8656ce7d6614b7b547022f7a4720b1cdad6a20aeeda717ec090b",
   "DATA_FORMAT.md": "e2fd4f6b5c8be9d9e008db7d0c1f5d52b419d68328a1c86625c9f67c49b03403",
-  "README.md": "43d89b6c59fdee0a1f5bddf3bd21cf0d809fee832f115771da95389f40f421b8",
-  "REPORT.md": "696454f04ac34467d2313e28b1e58b7095b02dc09e2b33b4b597594e10e3b59d",
-  "THIRD_PARTY_NOTICES.md": "8f38aa1ab2e03203b438e095a6325b3914541efe434290a9a2f786087d8f657d",
+  "README.md": "9f31727c4b1580261125888e71b3306440d298f9b8c1c3b2ee8819e6f1d0caa2",
+  "REPORT.md": "7326a5571a2a558395d005902b6ce46068252619aef6f9539bfb697503f39ba5",
+  "THIRD_PARTY_NOTICES.md": "0a7c3cf1144ffdabc16a3c70028ee708cbb0cf553fc401fe2461a76d9ee19442",
   "app.js": "eb013cc9d24382410953a1a8c81e446a6592a81b32887749d17bfc863936556b",
   "attempt-history.json": "8555f458c1dd362f5a07447342994fddf265ff23bbc40e9bd6f9984a892ddb4e",
-  "comparison.json": "012c7a64ecfde3352d258e332eae3b52de3d0c253663841e1a7dc9af3b3dfab7",
-  "data.json": "ca7388d97a0ed696d3b173eef5f8264bc76451b864baf2678df3144943bfe53e",
-  "episodes.csv": "ef6dc2beac16fbf02f603a770de4f3045eede1696d04c25cb0de789745fd4c69",
-  "episodes.json": "4dfa2559eee240954b0c2bf1dc8e15085f03d34dde749a21497c805cb7f57129",
+  "comparison.json": "3bb3d037b6a3c0ad14c768398286e15ba49f7d1928bf571106addde484e63e10",
+  "data.json": "fbc0782433e35a6e549438b584b2ffa408b85e5deb217ab3c91eb5969ba15ff7",
+  "episodes.csv": "0a597620179ff2d145cde0828975d4aab94da10f183450d4ad8da3b1953b0891",
+  "episodes.json": "28a0a1009c9a2410487ce36144dbca05502a53f104f892b15fc943c3712fc725",
   "failure-reviews.json": "c66a997c9ecd6f7f43f130a8a33cd64c7a5b2049033e9ccde653ee30e4796f23",
   "favicon.svg": "46063791218655f70c828ef71a1232a34fd38181da938338badec929ad226819",
-  "index.html": "80694c5b55656fd3392b354a0d122586bb86a2a1466ca23d2031092299d72202",
+  "index.html": "26e9fe65ac5958cdf3dd96a42c628b9a6658e7a2a7ff8c093a03058415938391",
   "licenses/LIBERO-LICENSE.txt": "e2885fd30a08381b799c4a33385522b23d637b4051b8f9a7f9f2519944b68ff6",
   "licenses/RoboDojo-LICENSE.txt": "7794bb06af8fe5485ca912454ad1665ccd7846c45f9c2b1258658e480948cdbe",
+  "licenses/RoboTwin-LICENSE.txt": "c695d421718e54e6f3a60858f0c601125c3ecc60c199332c15947360561e2a9f",
   "licenses/robocasa-LICENSE.txt": "5da18670b3f00c59847b1ded9c28dee59940d963b1e03b528b0108d9c5a09885",
   "licenses/robosuite-LICENSE.txt": "177978cbece0a4c454c2aaec5b3f145b39270814874c43109da9e829c39d9cba",
-  "protocol.json": "8368960a967825a8d721cf982e61d028c5d72091530b6d54316e7626955e6dbd",
+  "protocol.json": "ff8ba4e83a3e7d8679471b1ccf62c85aac4d7b1ddcd407d4f9e5ec6a0d0a187f",
   "style.css": "dd31b30951c220ebb82bc96224b5e640913945679e42b2e0a42e2c373a329e6c",
-  "summary.json": "752f4bccc2f906bc94f05113a9fd906d8bc1ad512d3a4057ae885479b1b94611",
-  "task-index.json": "c5bee5272347efefba25529144c9e752bf3ca2d336942c6b57a34036d1774517"
+  "summary.json": "25846760503d28793d17551fde029efb618f4170b03e86c218f8d9b146a95018",
+  "task-index.json": "a0399dcc6f7ba8b7b77caecbe3b4b83f63e5fb8151ac0083ce7d01fe69292055"
 }

protocol.json

@@ -50,6 +50,27 @@
       "modified_results": 5,
       "original_results": 5
     },
+    {
+      "id": "task03",
+      "name": "RoboTwin",
+      "total": 10,
+      "completed": 9,
+      "pending": 1,
+      "successes": 6,
+      "valid_results": 9,
+      "success_rate": 0.6666666666666666,
+      "input_tokens": 14637810,
+      "cached_input_tokens": 14194432,
+      "output_tokens": 97557,
+      "usage_complete": 9,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "preflight_results": 0,
+      "formal_results": 9,
+      "modified_results": 6,
+      "original_results": 3,
+      "deferred_tasks": 1
+    },
     {
       "id": "task04",
       "name": "RoboDojo",
@@ -92,6 +113,19 @@
       "seed": 0,
       "max_steps": 7500,
       "control_frequency_hz": 25
+    },
+    "task03": {
+      "episodes": 1,
+      "seed": 0,
+      "max_steps": 7500,
+      "control_frequency_hz": 25,
+      "safety_timeout_sec": 28800,
+      "execution_mode": "stepped",
+      "stop_on_success": true,
+      "deferred_tasks": [
+        "task03/10"
+      ]
     }
-  }
+  },
+  "robotwin_harness": "Stock Codex CLI 0.160.0, gpt-6-astra, high, default service tier; current RoboEnv SDK/CLI; no Kinex agent runtime."
 }

publication-manifest.json

@@ -10,16 +10,16 @@
       "bytes": 1129
     },
     "README.md": {
-      "sha256": "43d89b6c59fdee0a1f5bddf3bd21cf0d809fee832f115771da95389f40f421b8",
-      "bytes": 1688
+      "sha256": "9f31727c4b1580261125888e71b3306440d298f9b8c1c3b2ee8819e6f1d0caa2",
+      "bytes": 1373
     },
     "REPORT.md": {
-      "sha256": "696454f04ac34467d2313e28b1e58b7095b02dc09e2b33b4b597594e10e3b59d",
-      "bytes": 1565
+      "sha256": "7326a5571a2a558395d005902b6ce46068252619aef6f9539bfb697503f39ba5",
+      "bytes": 1250
     },
     "THIRD_PARTY_NOTICES.md": {
-      "sha256": "8f38aa1ab2e03203b438e095a6325b3914541efe434290a9a2f786087d8f657d",
-      "bytes": 2319
+      "sha256": "0a7c3cf1144ffdabc16a3c70028ee708cbb0cf553fc401fe2461a76d9ee19442",
+      "bytes": 2604
     },
     "app.js": {
       "sha256": "eb013cc9d24382410953a1a8c81e446a6592a81b32887749d17bfc863936556b",
@@ -30,20 +30,20 @@
       "bytes": 36661
     },
     "comparison.json": {
-      "sha256": "012c7a64ecfde3352d258e332eae3b52de3d0c253663841e1a7dc9af3b3dfab7",
-      "bytes": 84718
+      "sha256": "3bb3d037b6a3c0ad14c768398286e15ba49f7d1928bf571106addde484e63e10",
+      "bytes": 114939
     },
     "data.json": {
-      "sha256": "ca7388d97a0ed696d3b173eef5f8264bc76451b864baf2678df3144943bfe53e",
-      "bytes": 907017
+      "sha256": "fbc0782433e35a6e549438b584b2ffa408b85e5deb217ab3c91eb5969ba15ff7",
+      "bytes": 1046698
     },
     "episodes.csv": {
-      "sha256": "ef6dc2beac16fbf02f603a770de4f3045eede1696d04c25cb0de789745fd4c69",
-      "bytes": 5479
+      "sha256": "0a597620179ff2d145cde0828975d4aab94da10f183450d4ad8da3b1953b0891",
+      "bytes": 5779
     },
     "episodes.json": {
-      "sha256": "4dfa2559eee240954b0c2bf1dc8e15085f03d34dde749a21497c805cb7f57129",
-      "bytes": 667599
+      "sha256": "28a0a1009c9a2410487ce36144dbca05502a53f104f892b15fc943c3712fc725",
+      "bytes": 771432
     },
     "failure-reviews.json": {
       "sha256": "c66a997c9ecd6f7f43f130a8a33cd64c7a5b2049033e9ccde653ee30e4796f23",
@@ -54,8 +54,8 @@
       "bytes": 227
     },
     "index.html": {
-      "sha256": "80694c5b55656fd3392b354a0d122586bb86a2a1466ca23d2031092299d72202",
-      "bytes": 4340
+      "sha256": "26e9fe65ac5958cdf3dd96a42c628b9a6658e7a2a7ff8c093a03058415938391",
+      "bytes": 4358
     },
     "licenses/LIBERO-LICENSE.txt": {
       "sha256": "e2885fd30a08381b799c4a33385522b23d637b4051b8f9a7f9f2519944b68ff6",
@@ -65,6 +65,10 @@
       "sha256": "7794bb06af8fe5485ca912454ad1665ccd7846c45f9c2b1258658e480948cdbe",
       "bytes": 1091
     },
+    "licenses/RoboTwin-LICENSE.txt": {
+      "sha256": "c695d421718e54e6f3a60858f0c601125c3ecc60c199332c15947360561e2a9f",
+      "bytes": 1081
+    },
     "licenses/robocasa-LICENSE.txt": {
       "sha256": "5da18670b3f00c59847b1ded9c28dee59940d963b1e03b528b0108d9c5a09885",
       "bytes": 1418
@@ -74,24 +78,24 @@
       "bytes": 1474
     },
     "protocol.json": {
-      "sha256": "8368960a967825a8d721cf982e61d028c5d72091530b6d54316e7626955e6dbd",
-      "bytes": 2574
+      "sha256": "ff8ba4e83a3e7d8679471b1ccf62c85aac4d7b1ddcd407d4f9e5ec6a0d0a187f",
+      "bytes": 3537
     },
     "style.css": {
       "sha256": "dd31b30951c220ebb82bc96224b5e640913945679e42b2e0a42e2c373a329e6c",
       "bytes": 16560
     },
     "summary.json": {
-      "sha256": "752f4bccc2f906bc94f05113a9fd906d8bc1ad512d3a4057ae885479b1b94611",
-      "bytes": 1922
+      "sha256": "25846760503d28793d17551fde029efb618f4170b03e86c218f8d9b146a95018",
+      "bytes": 2338
     },
     "task-index.json": {
-      "sha256": "c5bee5272347efefba25529144c9e752bf3ca2d336942c6b57a34036d1774517",
-      "bytes": 192726
+      "sha256": "a0399dcc6f7ba8b7b77caecbe3b4b83f63e5fb8151ac0083ce7d01fe69292055",
+      "bytes": 221024
     },
     "manifest.json": {
-      "sha256": "65df6bd8f0503fc521d5101cd2d41f2c4d1bd408d3f8ea672fd8363211d68e14",
-      "bytes": 1979
+      "sha256": "4b384aadb0df61924a5c52b93a1b53a24454f70d7ae76cd0d57f5fed86d52f82",
+      "bytes": 2082
     }
   }
 }

summary.json

@@ -1,7 +1,7 @@
 {
-  "planned_tasks": 62,
-  "published_results": 62,
-  "pending_tasks": 0,
+  "planned_tasks": 72,
+  "published_results": 71,
+  "pending_tasks": 1,
   "families": [
     {
       "id": "task01",
@@ -43,6 +43,27 @@
       "modified_results": 5,
       "original_results": 5
     },
+    {
+      "id": "task03",
+      "name": "RoboTwin",
+      "total": 10,
+      "completed": 9,
+      "pending": 1,
+      "successes": 6,
+      "valid_results": 9,
+      "success_rate": 0.6666666666666666,
+      "input_tokens": 14637810,
+      "cached_input_tokens": 14194432,
+      "output_tokens": 97557,
+      "usage_complete": 9,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "preflight_results": 0,
+      "formal_results": 9,
+      "modified_results": 6,
+      "original_results": 3,
+      "deferred_tasks": 1
+    },
     {
       "id": "task04",
       "name": "RoboDojo",
@@ -64,16 +85,10 @@
       "original_results": 8
     }
   ],
-  "progress": {
-    "finished": 62,
-    "running": 0,
-    "queued": 0,
-    "needs_review": 0,
-    "interrupted": 0,
-    "native_successes": 42,
-    "native_failures": 20
-  },
   "interrupted_attempts": 4,
   "usage_incomplete_tasks": [],
-  "execution_incomplete_tasks": []
+  "execution_incomplete_tasks": [],
+  "deferred_tasks": [
+    "task03/10"
+  ]
 }

task-index.json

@@ -1462,6 +1462,782 @@
     "attempt_history": [],
     "status_note": "Evaluation and native evidence complete."
   },
+  {
+    "key": "task03/01",
+    "family": "task03",
+    "slot": "01",
+    "native_id": "robotwin/handover-block-aloha-agilex",
+    "title": "Handover block aloha agilex",
+    "catalog_instruction": "use the left arm to grasp the red block on the table, handover it to the right arm and place it upright on the blue pad",
+    "native_instruction": "use the left arm to grasp the red block on the table, handover it to the right arm and place it on the blue pad",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "completed",
+    "episode_id": "task03-01-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/handover-block-aloha-agilex",
+    "native_identity": {
+      "task_name": "handover_block",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "use the left arm to grasp the red block on the table, handover it to the right arm and place it on the blue pad",
+      "instruction": "use the left arm to grasp the red block on the table, handover it to the right arm and place it upright on the blue pad",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "equal",
+          "original": "use the left arm to grasp the red block on the table, handover it to the right arm and place it ",
+          "modified": "use the left arm to grasp the red block on the table, handover it to the right arm and place it "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "upright "
+        },
+        {
+          "op": "equal",
+          "original": "on the blue pad",
+          "modified": "on the blue pad"
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/01-handover-block/task.yaml"
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/02",
+    "family": "task03",
+    "slot": "02",
+    "native_id": "robotwin/place-dual-shoes-aloha-agilex",
+    "title": "Place dual shoes aloha agilex",
+    "catalog_instruction": "use both arms to pick up the two shoes on the table and put them flat in the shoebox, with the shoe tips pointing to the left. Put the shoe initially on the left in the front half (closer to the robot), and the other shoe in the back half. Release both shoes and withdraw the open grippers.",
+    "native_instruction": "use both arms to pick up the two shoes on the table and put them in the shoebox, with the shoe tip pointing to the left",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "completed",
+    "episode_id": "task03-02-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/place-dual-shoes-aloha-agilex",
+    "native_identity": {
+      "task_name": "place_dual_shoes",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "use both arms to pick up the two shoes on the table and put them in the shoebox, with the shoe tip pointing to the left",
+      "instruction": "use both arms to pick up the two shoes on the table and put them flat in the shoebox, with the shoe tips pointing to the left. Put the shoe initially on the left in the front half (closer to the robot), and the other shoe in the back half. Release both shoes and withdraw the open grippers.",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "equal",
+          "original": "use both arms to pick up the two shoes on the table and put them ",
+          "modified": "use both arms to pick up the two shoes on the table and put them "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "flat "
+        },
+        {
+          "op": "equal",
+          "original": "in the shoebox, with the shoe ",
+          "modified": "in the shoebox, with the shoe "
+        },
+        {
+          "op": "replace",
+          "original": "tip",
+          "modified": "tips"
+        },
+        {
+          "op": "equal",
+          "original": " pointing to the ",
+          "modified": " pointing to the "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "left. Put the shoe initially on the "
+        },
+        {
+          "op": "equal",
+          "original": "left",
+          "modified": "left"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": " in the front half (closer to the robot), and the other shoe in the back half. Release both shoes and withdraw the open grippers."
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/02-place-dual-shoes/task.yaml"
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/03",
+    "family": "task03",
+    "slot": "03",
+    "native_id": "robotwin/put-bottles-dustbin-aloha-agilex",
+    "title": "Put bottles dustbin aloha agilex",
+    "catalog_instruction": "use arms to grab the bottles and put them into the dustbin to the left of the table",
+    "native_instruction": "use arms to grab the bottles and put them into the dustbin to the left of the table",
+    "instruction_source": "runtime native task.instruction",
+    "status": "completed",
+    "episode_id": "task03-03-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "original_native"
+    },
+    "catalog_id": "robotwin/put-bottles-dustbin-aloha-agilex",
+    "native_identity": {
+      "task_name": "put_bottles_dustbin",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/04",
+    "family": "task03",
+    "slot": "04",
+    "native_id": "robotwin/scan-object-aloha-agilex",
+    "title": "Scan object aloha agilex",
+    "catalog_instruction": "Pick up the scanner and the object simultaneously with separate arms. Hold the scanner\u2019s scanning face close to and directly facing the object\u2019s center, keeping both grippers closed around the items.",
+    "native_instruction": "simultaneously pick up the scanner and the object with separate arms, then scan the object",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "completed",
+    "episode_id": "task03-04-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/scan-object-aloha-agilex",
+    "native_identity": {
+      "task_name": "scan_object",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "simultaneously pick up the scanner and the object with separate arms, then scan the object",
+      "instruction": "Pick up the scanner and the object simultaneously with separate arms. Hold the scanner\u2019s scanning face close to and directly facing the object\u2019s center, keeping both grippers closed around the items.",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "replace",
+          "original": "simultaneously p",
+          "modified": "P"
+        },
+        {
+          "op": "equal",
+          "original": "ick up the scanner and the object ",
+          "modified": "ick up the scanner and the object "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "simultaneously "
+        },
+        {
+          "op": "equal",
+          "original": "with separate arms",
+          "modified": "with separate arms"
+        },
+        {
+          "op": "replace",
+          "original": ",",
+          "modified": ". Hold"
+        },
+        {
+          "op": "equal",
+          "original": " the",
+          "modified": " the"
+        },
+        {
+          "op": "delete",
+          "original": "n",
+          "modified": ""
+        },
+        {
+          "op": "equal",
+          "original": " scan",
+          "modified": " scan"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "ner\u2019s scanning face close to and directly facing"
+        },
+        {
+          "op": "equal",
+          "original": " the object",
+          "modified": " the object"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "\u2019s center, keeping both grippers closed around the items."
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/04-scan-object/task.yaml",
+      "native_predicates_unchanged": true,
+      "evaluation_status": "Evaluated with this modified instruction.",
+      "review_reason": "Clarify the close, centered face-to-face scan pose and keeping both items grasped; native scan geometry thresholds are unchanged."
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/05",
+    "family": "task03",
+    "slot": "05",
+    "native_id": "robotwin/stack-bowls-three-aloha-agilex",
+    "title": "Stack bowls three aloha agilex",
+    "catalog_instruction": "nest the three bowls into one compact, vertically aligned stack resting on the table. Release the bowls and leave both grippers open.",
+    "native_instruction": "stack the three bowls on top of each other",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "completed",
+    "episode_id": "task03-05-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/stack-bowls-three-aloha-agilex",
+    "native_identity": {
+      "task_name": "stack_bowls_three",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "stack the three bowls on top of each other",
+      "instruction": "nest the three bowls into one compact, vertically aligned stack resting on the table. Release the bowls and leave both grippers open.",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "ne"
+        },
+        {
+          "op": "equal",
+          "original": "st",
+          "modified": "st"
+        },
+        {
+          "op": "delete",
+          "original": "ack",
+          "modified": ""
+        },
+        {
+          "op": "equal",
+          "original": " the three bowls ",
+          "modified": " the three bowls "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "into one compact, vertically aligned stack resting "
+        },
+        {
+          "op": "equal",
+          "original": "on t",
+          "modified": "on t"
+        },
+        {
+          "op": "replace",
+          "original": "op",
+          "modified": "he"
+        },
+        {
+          "op": "equal",
+          "original": " ",
+          "modified": " "
+        },
+        {
+          "op": "replace",
+          "original": "of",
+          "modified": "table."
+        },
+        {
+          "op": "equal",
+          "original": " ",
+          "modified": " "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "Rel"
+        },
+        {
+          "op": "equal",
+          "original": "ea",
+          "modified": "ea"
+        },
+        {
+          "op": "replace",
+          "original": "c",
+          "modified": "se t"
+        },
+        {
+          "op": "equal",
+          "original": "h",
+          "modified": "h"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "e"
+        },
+        {
+          "op": "equal",
+          "original": " ",
+          "modified": " "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "bowls and leave b"
+        },
+        {
+          "op": "equal",
+          "original": "oth",
+          "modified": "oth"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": " gripp"
+        },
+        {
+          "op": "equal",
+          "original": "er",
+          "modified": "er"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "s open."
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/05-stack-bowls-three/task.yaml",
+      "native_predicates_unchanged": true,
+      "evaluation_status": "Evaluated with this modified instruction.",
+      "review_reason": "Retain the existing concise instruction describing the compact bowl stack, release and open grippers; rerun using the current Kinex source."
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/06",
+    "family": "task03",
+    "slot": "06",
+    "native_id": "robotwin/stack-blocks-three-aloha-agilex",
+    "title": "Stack blocks three aloha agilex",
+    "catalog_instruction": "build a three-block stack at the center by placing the red block first, the green block on red, and the blue block on green",
+    "native_instruction": "build a three-block stack at the center by placing the red block first, the green block on red, and the blue block on green",
+    "instruction_source": "runtime native task.instruction",
+    "status": "completed",
+    "episode_id": "task03-06-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "original_native"
+    },
+    "catalog_id": "robotwin/stack-blocks-three-aloha-agilex",
+    "native_identity": {
+      "task_name": "stack_blocks_three",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/07",
+    "family": "task03",
+    "slot": "07",
+    "native_id": "robotwin/hanging-mug-aloha-agilex",
+    "title": "Hanging mug aloha agilex",
+    "catalog_instruction": "Use the left arm to pick up the mug on the table, rotate it and put it down in the middle of the table, then use the right arm to hang the mug by its handle on the rack's peg. Seat the handle near the middle of the peg and open the right gripper to leave the mug hanging.",
+    "native_instruction": "Use left arm to pick the mug on the table, rotate the mug and put the mug down in the middle of the table, use the right arm to pick the mug and hang it onto the rack.",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "completed",
+    "episode_id": "task03-07-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/hanging-mug-aloha-agilex",
+    "native_identity": {
+      "task_name": "hanging_mug",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "Use left arm to pick the mug on the table, rotate the mug and put the mug down in the middle of the table, use the right arm to pick the mug and hang it onto the rack.",
+      "instruction": "Use the left arm to pick up the mug on the table, rotate it and put it down in the middle of the table, then use the right arm to hang the mug by its handle on the rack's peg. Seat the handle near the middle of the peg and open the right gripper to leave the mug hanging.",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "equal",
+          "original": "Use",
+          "modified": "Use"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": " the"
+        },
+        {
+          "op": "equal",
+          "original": " left arm to pick",
+          "modified": " left arm to pick"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": " up"
+        },
+        {
+          "op": "equal",
+          "original": " the mug on the table, rotate ",
+          "modified": " the mug on the table, rotate "
+        },
+        {
+          "op": "replace",
+          "original": "the mug",
+          "modified": "it"
+        },
+        {
+          "op": "equal",
+          "original": " and put ",
+          "modified": " and put "
+        },
+        {
+          "op": "replace",
+          "original": "the mug",
+          "modified": "it"
+        },
+        {
+          "op": "equal",
+          "original": " down in the middle of the table, ",
+          "modified": " down in the middle of the table, "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "then "
+        },
+        {
+          "op": "equal",
+          "original": "use the right arm to ",
+          "modified": "use the right arm to "
+        },
+        {
+          "op": "replace",
+          "original": "pick",
+          "modified": "hang"
+        },
+        {
+          "op": "equal",
+          "original": " the mug ",
+          "modified": " the mug "
+        },
+        {
+          "op": "replace",
+          "original": "and",
+          "modified": "by"
+        },
+        {
+          "op": "equal",
+          "original": " ",
+          "modified": " "
+        },
+        {
+          "op": "replace",
+          "original": "hang",
+          "modified": "its"
+        },
+        {
+          "op": "equal",
+          "original": " ",
+          "modified": " "
+        },
+        {
+          "op": "replace",
+          "original": "it",
+          "modified": "handle"
+        },
+        {
+          "op": "equal",
+          "original": " ",
+          "modified": " "
+        },
+        {
+          "op": "replace",
+          "original": "onto",
+          "modified": "on"
+        },
+        {
+          "op": "equal",
+          "original": " the ",
+          "modified": " the "
+        },
+        {
+          "op": "replace",
+          "original": "rack.",
+          "modified": "rack's peg. Seat the handle near the middle of the peg and open the right gripper to leave the mug hanging."
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/07-hanging-mug/task.yaml"
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/08",
+    "family": "task03",
+    "slot": "08",
+    "native_id": "robotwin/put-object-cabinet-aloha-agilex",
+    "title": "Put object cabinet aloha agilex",
+    "catalog_instruction": "use one arm to hold the object and the other arm to open the cabinet drawer, then place the object inside",
+    "native_instruction": "use one arm to hold the object and the other arm to open the cabinet drawer, then place the object inside",
+    "instruction_source": "runtime native task.instruction",
+    "status": "completed",
+    "episode_id": "task03-08-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "original_native"
+    },
+    "catalog_id": "robotwin/put-object-cabinet-aloha-agilex",
+    "native_identity": {
+      "task_name": "put_object_cabinet",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/09",
+    "family": "task03",
+    "slot": "09",
+    "native_id": "robotwin/blocks-ranking-size-aloha-agilex",
+    "title": "Blocks ranking size aloha agilex",
+    "catalog_instruction": "there are three blocks on the table, the color of the blocks is random, move the blocks to the center of the table, and arrange them in a left-to-right row from largest to smallest",
+    "native_instruction": "there are three blocks on the table, the color of the blocks is random, move the blocks to the center of the table, and arrange them from largest to smallest, from left to right",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "completed",
+    "episode_id": "task03-09-seed0-formal",
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/blocks-ranking-size-aloha-agilex",
+    "native_identity": {
+      "task_name": "blocks_ranking_size",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "there are three blocks on the table, the color of the blocks is random, move the blocks to the center of the table, and arrange them from largest to smallest, from left to right",
+      "instruction": "there are three blocks on the table, the color of the blocks is random, move the blocks to the center of the table, and arrange them in a left-to-right row from largest to smallest",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "equal",
+          "original": "there are three blocks on the table, the color of the blocks is random, move the blocks to the center of the table, and arrange them ",
+          "modified": "there are three blocks on the table, the color of the blocks is random, move the blocks to the center of the table, and arrange them "
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": "in a left-to-right row "
+        },
+        {
+          "op": "equal",
+          "original": "from largest to ",
+          "modified": "from largest to "
+        },
+        {
+          "op": "replace",
+          "original": "smallest, from left to right",
+          "modified": "smallest"
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/09-blocks-ranking-size/task.yaml"
+    },
+    "attempt_history": [],
+    "run_status": "finished",
+    "status_note": "Native result, execution and usage complete."
+  },
+  {
+    "key": "task03/10",
+    "family": "task03",
+    "slot": "10",
+    "native_id": "robotwin/lift-pot-aloha-agilex",
+    "title": "Lift pot \u00b7 coordinated bimanual lifting",
+    "catalog_instruction": "Use both arms to lift the pot well clear of the tabletop, grasping the left handle with the left gripper and the right handle with the right gripper. Keep the pot upright and keep each gripper centered on its handle while holding the pot up.",
+    "native_instruction": "use BOTH!!! arms to lift the pot",
+    "instruction_source": "Current reviewed benchmark instruction; each recording retains its measured instruction.",
+    "status": "pending",
+    "episode_id": null,
+    "planned_protocol": {
+      "episodes": 1,
+      "seed": 0,
+      "control_frequency_hz": 25,
+      "max_control_steps": 7500,
+      "timeout_s": 28800,
+      "mode": "stepped",
+      "instruction_policy": "modified"
+    },
+    "catalog_id": "robotwin/lift-pot-aloha-agilex",
+    "native_identity": {
+      "task_name": "lift_pot",
+      "embodiment": "aloha_agilex",
+      "max_steps": 7500,
+      "task_config": "demo_clean",
+      "allow_seed_substitution": false
+    },
+    "instruction_revision": {
+      "native_instruction": "use BOTH!!! arms to lift the pot",
+      "instruction": "Use both arms to lift the pot well clear of the tabletop, grasping the left handle with the left gripper and the right handle with the right gripper. Keep the pot upright and keep each gripper centered on its handle while holding the pot up.",
+      "native_instruction_kind": "source",
+      "diff": [
+        {
+          "op": "replace",
+          "original": "u",
+          "modified": "U"
+        },
+        {
+          "op": "equal",
+          "original": "se ",
+          "modified": "se "
+        },
+        {
+          "op": "replace",
+          "original": "BOTH!!!",
+          "modified": "both"
+        },
+        {
+          "op": "equal",
+          "original": " arms to lift the pot",
+          "modified": " arms to lift the pot"
+        },
+        {
+          "op": "insert",
+          "original": "",
+          "modified": " well clear of the tabletop, grasping the left handle with the left gripper and the right handle with the right gripper. Keep the pot upright and keep each gripper centered on its handle while holding the pot up."
+        }
+      ],
+      "changed": true,
+      "preserve_whitespace": true,
+      "label": "Modified instruction",
+      "source_url": "https://huggingface.co/datasets/RLE-Bench/codex-benchmark/resolve/main/catalog/task03/10-lift-pot/task.yaml",
+      "native_predicates_unchanged": true,
+      "evaluation_status": "Evaluated with this modified instruction.",
+      "review_reason": "Retain the existing instruction describing two-handle grasping, upright orientation and sufficient lift; rerun using the current Kinex source."
+    },
+    "attempt_history": [],
+    "run_status": "deferred",
+    "status_note": "Deferred by the user before launch; no Codex episode or model call."
+  },
   {
     "key": "task04/01",
     "family": "task04",