task10_taskfinetune/EVAL_RESULTS.md

# Rollout evaluation — `task10_taskfinetune`

Closed-loop rollout evaluation on **task 10 = `set_up_a_coffee_station_in_your_kitchen`**,
against the official BEHAVIOR-1K solution checkpoint for the same task.

This supersedes the "no rollout evaluation has been run" note in the model card.

## Result

| | official `checkpoint_1` | `task10_taskfinetune` |
|---|---|---|
| **Success rate** | **5.0 %** (1/20) | **0.0 %** (0/20) |
| **Avg q_score** | 0.1750 | **0.1917** |
| Median q | 0.167 | 0.167 |
| Episodes at q = 0 | 8 | 7 |
| Best episode | **1.000** | 0.500 |

**The fine-tune shows no reliable gain.** It edges the official checkpoint on mean q
(+0.0167) but loses the only solved episode in either arm.
Paired sign test over the 20 instances: **8 better / 7 worse / 5 tied, p = 1.000** —
indistinguishable from chance.

The single most informative row is **instance 310**: the official checkpoint scores **1.000**
(the only solve anywhere in this study), the fine-tune **0.167**. The fine-tune is marginally more
consistent (7 zeros vs 8, never above 0.500); the official checkpoint is spikier and
occasionally completes the task.

## Per-instance q_score

| instance | official `checkpoint_1` | `task10_taskfinetune` | delta |
|---|---|---|---|
| 301 | 0.167 | 0.500 | +0.333 |
| 302 | 0.167 | 0.000 | -0.167 |
| 303 | 0.000 | 0.333 | +0.333 |
| 304 | 0.000 | 0.000 | +0.000 |
| 305 | 0.000 | 0.000 | +0.000 |
| 306 | 0.167 | 0.000 | -0.167 |
| 307 | 0.500 | 0.333 | -0.167 |
| 308 | 0.000 | 0.167 | +0.167 |
| 309 | 0.000 | 0.167 | +0.167 |
| 310 | 1.000 | 0.167 | -0.833 |
| 311 | 0.000 | 0.500 | +0.500 |
| 312 | 0.000 | 0.333 | +0.333 |
| 313 | 0.333 | 0.000 | -0.333 |
| 314 | 0.167 | 0.167 | +0.000 |
| 315 | 0.167 | 0.500 | +0.333 |
| 316 | 0.333 | 0.167 | -0.167 |
| 317 | 0.000 | 0.000 | +0.000 |
| 318 | 0.167 | 0.000 | -0.167 |
| 319 | 0.167 | 0.333 | +0.167 |
| 320 | 0.167 | 0.167 | +0.000 |

## Evaluation setup

Both arms ran on the same day, same nodes, same evaluator — the comparison is internally controlled.

| | |
|---|---|
| task | `set_up_a_coffee_station_in_your_kitchen` (task_id 10) |
| instances | 20, `--mode public_test` (ids 301–320) |
| horizon | 1.5x mean human-demo length = **9,400 steps** (no `--max-steps` override) |
| control rate | 30 Hz -> 313 s = 5.2 min simulated per episode |
| runner | `omnigibson.eval.eval_fast` + chunk-aware policy server |
| render | trailing4 + flush3; per episode `full=470`, `tail_render=1410`, `fast=8930` |
| `base_qvel` frame | **robot** (`B1K_BASE_QVEL_FRAME=robot`) |
| norm stats | qvelfix, sha256 `ee2e1a97...` — **byte-identical between both arms** |
| dtype | float32 |
| challenge year | 2026 |
| official baseline | `IliaLarchenko/behavior_submission` **checkpoint_1**, per `b50_tasks.csv` (`10,set_up_a_coffee_station_in_your_kitchen,checkpoint_1`), with the qvelfix norm-stats overlay so both arms share one input contract |
| evaluator build | `eval_fast.py` mtime 2026-08-14 16:47 |

## Caveats

1. **n = 20 cannot resolve differences this small.** Re-running the *same* checkpoint under an
   identical configuration has been observed to swing a single instance's q_score by 0.333, so
   run-to-run noise is comparable to the effect being measured. Repeats across seeds would be
   needed to separate 0.175 from 0.192.
2. **Not comparable to numbers produced before 2026-08-14.** `eval_fast.py` gained the
   trailing-render + annotator-flush behaviour on that date; its own source comment attributes a
   3-5x q_score effect to the flush. Only runs sharing an evaluator build may be compared.
3. **Step 29,999 is not necessarily the best checkpoint.** Training used a flat 5e-6 LR with no
   decay phase, so there is no annealing into a minimum. Intermediate steps were retained.
4. `q_score` and `success` can disagree — an episode scoring q = 1.000 has been observed with
   `success = False` on a related task. Report both.