add README for task54_putting_away_toys

task54_putting_away_toys/README.md

@@ -0,0 +1,134 @@
+# Task 54 — `putting_away_toys` (PiBehavior / pi0.5, no DA3)
+
+Single-task fine-tune for the [BEHAVIOR-1K](https://behavior.stanford.edu/) 2026 challenge.
+Part of [`JackLiu0406/b1k-checkpoints`](https://huggingface.co/JackLiu0406/b1k-checkpoints).
+
+---
+
+## Quick facts
+
+| | |
+|---|---|
+| architecture | PiBehavior / pi0.5 — `gemma_2b` VLM + `gemma_300m` action expert, 3,826.8 M params |
+| initialised from | [`IliaLarchenko/behavior_50t_checkpoint`](https://huggingface.co/IliaLarchenko/behavior_50t_checkpoint) (50-task meta) |
+| task | **54** = `putting_away_toys` |
+| data | 200 episodes, **2,259,351 frame-samples** |
+| checkpoint | **19999** (full 20,000-step schedule completed) |
+| task space | 100 tasks |
+| DA3 | not used |
+| training data | 224x224 **GOP=8** re-encode — [`JackLiu0406/b1k-224-gop8`](https://huggingface.co/datasets/JackLiu0406/b1k-224-gop8) |
+
+### Final metrics (step 19,975)
+
+| metric | value |
+|---|---|
+| `action_loss` | **0.0443** (from 0.8591 at step 0) |
+| `fast_accuracy` | **0.8143** (from 0.7523) |
+| `subtask_accuracy` | 0.9989 |
+| `grad_norm` | 0.1308 |
+
+No nan/inf at any point. 20,000 steps at batch 256 over 2,259,351 samples is ~2.3 epochs.
+
+---
+
+## This is a NEW task — both embedding rows started from random init
+
+`putting_away_toys` is **not** in the upstream 50, so nothing about it transferred:
+
+```
+task_embeddings[54]                    random init
+task_stage_embeddings[646-658]        random init (13 stages, pretrained table ends at 596)
+```
+
+Confirmed from the run's own log:
+
+```
+Expanded task_embeddings/embedding       (50, 2048) -> (100, 2048)
+Expanded task_stage_embeddings/embedding (596, 1024) -> (1120, 1024)
+```
+
+`B1K_TASK_SPACE=100` is required — the tables were widened to 100 rows and a 50-task model
+fails on a shape mismatch at load.
+
+---
+
+## Normalisation — use the bundled `norm_stats.json`
+
+`assets/IliaLarchenko/behavior_224_rgb/norm_stats.json` ships with this checkpoint and is
+the file it trained against: the upstream 2025 **50-task** stats with `state[0:3]`
+(`base_qvel`) corrected to the robot frame — identical to
+[`norm-stats-fixed/`](../norm-stats-fixed) in this repo.
+
+**Do not substitute per-task statistics.** That was tested directly, and it is worse. A
+controlled A/B on task 56 — same seed, same data, same recipe, only `assets` differing
+(`subtask_accuracy` identical at 0.2148, confirming identical init):
+
+| step | `fast_accuracy` with task-only stats | with these 50-task stats |
+|---|---|---|
+| 0 | 0.4824 | **0.7523** |
+| 70 | 0.5679 | **0.7792** |
+
+`fast_loss` at step 0 was 2.3997 vs 0.9398. The FAST tokenizer is a fixed artifact
+calibrated on the 2025 action distribution, and per-task normalisation rescales actions
+1.45-1.60x (up to 3.2x on trunk), pushing them outside its bins.
+
+Note `action_loss` is *lower* under per-task stats (0.7842 vs 0.9493) — that is a change of
+units, not accuracy: per-task stats give unit-variance targets by construction. Only metrics
+invariant to the normalisation (`fast_accuracy`, `fast_loss`, `subtask_accuracy`) are
+comparable, and all favour these.
+
+---
+
+## Running it
+
+Observation contract, wrapper behaviour and the full loop:
+[`EVAL.md`](../task85_putting_dirty_dishes_in_sink/EVAL.md) — applies to every checkpoint here.
+
+```bash
+huggingface-cli download JackLiu0406/b1k-checkpoints \
+  --include "task54_putting_away_toys/19999/*" --local-dir ./ckpts
+
+export B1K_TASK_SPACE=100
+export XLA_PYTHON_CLIENT_MEM_FRACTION=0.9
+
+uv run scripts/serve_b1k.py \
+  --policy.config pi_behavior_b1k_fast \
+  --policy.dir   ./ckpts/task54_putting_away_toys/19999 \
+  --task_id 54 \
+  --port 8000
+```
+
+Point `assets_base_dir` at the **bundled** `assets/` so both `norm_stats.json` and
+`fast_tokenizer/` resolve under `<assets>/IliaLarchenko/behavior_224_rgb/`.
+
+For a clean measurement of the policy itself add `--apply_eval_tricks False`; it defaults to
+`True` and can rewrite actions and force stage corrections.
+
+---
+
+## Training setup
+
+| | |
+|---|---|
+| GPUs | 8 x H200, global batch **256** (32/GPU), FSDP **off** |
+| steps | 20,000 |
+| LR | ramp `5e-7` -> `5e-5` over 2,000 warmup, cosine to `1e-6` at 20,000 |
+| data | 224x224 GOP=8, decoded without resize (`B1K_DECODE_RESIZE=0`) |
+| throughput | ~1.7 s/it, GPU-idle 0/30 samples (compute-limited) |
+
+Inherited from the base config: `action_horizon=30`, `action_dim=32`, correlated noise
+(`beta=0.5`), FAST auxiliary (weight 0.05, vocab 1024), `subtask_loss_weight=0.1`, frozen
+vision backbone, delta joint actions, per-timestamp normalisation. The FAST tokenizer was
+**not** retrained.
+
+---
+
+## Caveats
+
+1. **Inference only** — `train_state/` is not published; you cannot resume training.
+2. **Single-task.** Only task 54 was fine-tuned. Other indices in 50..99 not named in
+   this repo are at random init and will produce garbage.
+3. **All metrics are training-set.** No validation split, no rollout evaluation — none of
+   these numbers speak to generalisation.
+4. Episode length varies several-fold across tasks, so `action_loss` is not comparable
+   between checkpoints in this repo.