Add rollout eval results for task10_taskfinetune; retarget README to the new folder name

README.md

@@ -27,7 +27,11 @@ using the **200 human demonstrations** from the 2026 BEHAVIOR-1K challenge set a
 | task space | `B1K_TASK_SPACE=100`; `setting_mousetraps` is index 5 in both the 2025 and 2026 tables, so its task/stage embeddings are **pretrained**, not randomly initialised |
 | final logged | `action_loss = 0.0209` at step 21,775 |
 
-## `task10_coffee_station_29999`
+## `task10_taskfinetune`
+
+> **Rollout evaluation:** see [`task10_taskfinetune/EVAL_RESULTS.md`](task10_taskfinetune/EVAL_RESULTS.md) —
+> 0.0 % success / avg q 0.1917 over 20 `public_test` instances, vs 5.0 % / 0.1750 for the
+> official `checkpoint_1`. The difference is not statistically significant (sign test p = 1.000).
 
 Single-task fine-tune of the 50-task meta checkpoint on **task 10 =
 `set_up_a_coffee_station_in_your_kitchen`**, using the **200 human demonstrations** from the

task10_taskfinetune/EVAL_RESULTS.md

@@ -0,0 +1,84 @@
+# Rollout evaluation — `task10_taskfinetune`
+
+Closed-loop rollout evaluation on **task 10 = `set_up_a_coffee_station_in_your_kitchen`**,
+against the official BEHAVIOR-1K solution checkpoint for the same task.
+
+This supersedes the "no rollout evaluation has been run" note in the model card.
+
+## Result
+
+| | official `checkpoint_1` | `task10_taskfinetune` |
+|---|---|---|
+| **Success rate** | **5.0 %** (1/20) | **0.0 %** (0/20) |
+| **Avg q_score** | 0.1750 | **0.1917** |
+| Median q | 0.167 | 0.167 |
+| Episodes at q = 0 | 8 | 7 |
+| Best episode | **1.000** | 0.500 |
+
+**The fine-tune shows no reliable gain.** It edges the official checkpoint on mean q
+(+0.0167) but loses the only solved episode in either arm.
+Paired sign test over the 20 instances: **8 better / 7 worse / 5 tied, p = 1.000** —
+indistinguishable from chance.
+
+The single most informative row is **instance 310**: the official checkpoint scores **1.000**
+(the only solve anywhere in this study), the fine-tune **0.167**. The fine-tune is marginally more
+consistent (7 zeros vs 8, never above 0.500); the official checkpoint is spikier and
+occasionally completes the task.
+
+## Per-instance q_score
+
+| instance | official `checkpoint_1` | `task10_taskfinetune` | delta |
+|---|---|---|---|
+| 301 | 0.167 | 0.500 | +0.333 |
+| 302 | 0.167 | 0.000 | -0.167 |
+| 303 | 0.000 | 0.333 | +0.333 |
+| 304 | 0.000 | 0.000 | +0.000 |
+| 305 | 0.000 | 0.000 | +0.000 |
+| 306 | 0.167 | 0.000 | -0.167 |
+| 307 | 0.500 | 0.333 | -0.167 |
+| 308 | 0.000 | 0.167 | +0.167 |
+| 309 | 0.000 | 0.167 | +0.167 |
+| 310 | 1.000 | 0.167 | -0.833 |
+| 311 | 0.000 | 0.500 | +0.500 |
+| 312 | 0.000 | 0.333 | +0.333 |
+| 313 | 0.333 | 0.000 | -0.333 |
+| 314 | 0.167 | 0.167 | +0.000 |
+| 315 | 0.167 | 0.500 | +0.333 |
+| 316 | 0.333 | 0.167 | -0.167 |
+| 317 | 0.000 | 0.000 | +0.000 |
+| 318 | 0.167 | 0.000 | -0.167 |
+| 319 | 0.167 | 0.333 | +0.167 |
+| 320 | 0.167 | 0.167 | +0.000 |
+
+## Evaluation setup
+
+Both arms ran on the same day, same nodes, same evaluator — the comparison is internally controlled.
+
+| | |
+|---|---|
+| task | `set_up_a_coffee_station_in_your_kitchen` (task_id 10) |
+| instances | 20, `--mode public_test` (ids 301–320) |
+| horizon | 1.5x mean human-demo length = **9,400 steps** (no `--max-steps` override) |
+| control rate | 30 Hz -> 313 s = 5.2 min simulated per episode |
+| runner | `omnigibson.eval.eval_fast` + chunk-aware policy server |
+| render | trailing4 + flush3; per episode `full=470`, `tail_render=1410`, `fast=8930` |
+| `base_qvel` frame | **robot** (`B1K_BASE_QVEL_FRAME=robot`) |
+| norm stats | qvelfix, sha256 `ee2e1a97...` — **byte-identical between both arms** |
+| dtype | float32 |
+| challenge year | 2026 |
+| official baseline | `IliaLarchenko/behavior_submission` **checkpoint_1**, per `b50_tasks.csv` (`10,set_up_a_coffee_station_in_your_kitchen,checkpoint_1`), with the qvelfix norm-stats overlay so both arms share one input contract |
+| evaluator build | `eval_fast.py` mtime 2026-08-14 16:47 |
+
+## Caveats
+
+1. **n = 20 cannot resolve differences this small.** Re-running the *same* checkpoint under an
+   identical configuration has been observed to swing a single instance's q_score by 0.333, so
+   run-to-run noise is comparable to the effect being measured. Repeats across seeds would be
+   needed to separate 0.175 from 0.192.
+2. **Not comparable to numbers produced before 2026-08-14.** `eval_fast.py` gained the
+   trailing-render + annotator-flush behaviour on that date; its own source comment attributes a
+   3-5x q_score effect to the flush. Only runs sharing an evaluator build may be compared.
+3. **Step 29,999 is not necessarily the best checkpoint.** Training used a flat 5e-6 LR with no
+   decay phase, so there is no annealing into a minimum. Intermediate steps were retained.
+4. `q_score` and `success` can disagree — an episode scoring q = 1.000 has been observed with
+   `success = False` on a related task. Report both.