| # task-5 `setting_mousetraps` — evaluation videos |
| |
| Rollout videos and per-instance scores from the stock BEHAVIOR v3.9.1 evaluator |
| (`omnigibson.eval.eval`, commit `9a03ca3f`), non-fast path, on the 20 official |
| `public_test` instances (ids 301–320). |
|
|
| ## Layout |
|
|
| ``` |
| merged39999/ 20 videos + results.json mean q 0.6417 success 1/20 |
| merged20000/ 20 videos + results.json mean q 0.6167 success 0/20 |
| merged15000/ results.json only mean q 0.6250 success 2/20 |
| ``` |
|
|
| Filenames encode the outcome: `inst<id>_idx<NN>_q<score>.mp4`. |
| Each video is h264 672x448 — head plus both wrist cameras tiled into one frame. |
| `merged15000` has no videos because that sweep predated `--write-video`. |
|
|
| ## Eval configuration |
|
|
| Two settings differ from a stock submission run and both matter when reading these numbers: |
|
|
| - **`B1K_STATE_2026=1`** — the 61-dim `PROP_2026` state extraction. Without it the |
| 2025 extractor's slices (`base_qvel` 253:256) run off the end of v3.9.x's 61-dim |
| proprioception and silently yield a 2-element state, so the policy receives a |
| constant input and scores 0. This flag is required, not optional. |
| - **correction rules disabled** (`--no-apply-eval-tricks`) — they index `state[14]` |
| and raise `IndexError` on the collapsed state; disabling them was the only way to |
| complete rollouts while diagnosing. Re-enabling is untested. |
|
|
| Also applied: 100-task / 1120-stage embedding tables, which these checkpoints |
| require and the 50-task solution repos do not provide. |
|
|
| ## What the scores say |
|
|
| ``` |
| merged15000 0.6250 merged20000 0.6167 merged39999 0.6417 |
| ``` |
|
|
| A 0.025 spread across 15k–40k training steps, identical medians (0.6667), and a |
| paired sign test of 39999 vs 15000 giving p = 0.754. There is no evidence these |
| checkpoints differ; training further did not help. |
|
|
| The consistent feature is a ceiling: **14 of 20 instances score exactly 0.6667** |
| and 19 of 20 run to the 15,295-step cap. `q_score` counts goal predicates that |
| flipped false→true divided by the total in that option, so a predicate already |
| true at reset counts in the denominator but can never earn credit. Whether the |
| 0.6667 wall is an unachieved condition or an uncreditable pre-satisfied one is |
| not resolved by these files. |
|
|
