# TinyVLA — corrected evaluation and ablations

Measured independently of the numbers in the model card, on a re-implemented
harness that separates the three rates and logs them instead of inferring them.
All closed-loop numbers below are **paired**: every configuration runs the same
(task, init state) episodes, and the flow-matching noise is seeded per
(episode, step), so two rows differ only by the thing being ablated.

Hardware: 4x Tesla V100-SXM2-32GB (sm_70), fp32, driver 535 / CUDA 12.2,
torch 2.6.0+cu124, transformers 5.5.4, lerobot 0.6.1. Suite: `libero_spatial`.
Generated 2026-08-25.

**The headline dual-rate comparison in the model card is not reproduced.** With the
clock fixed and 500 paired episodes per configuration, no refresh rate differs
significantly from any other. What does survive is the efficiency claim, and it
survives causally: see sections 3 and 4.

---

## 1. The refresh schedule was derived from the wrong frame rate

`HuggingFaceVLA/libero` declares `fps = 10`, but its frames are native 20 Hz. The
dataset has 1693 episodes and 273,465 frames, i.e. **161.5 frames per episode**;
LIBERO demos run about 8 s, giving 20.2 Hz, which matches the environment
(`lerobot.envs.configs` sets `fps: int = 20  # Must match robosuite's default
control_freq (20 Hz)` and `LiberoEnv(control_freq=20)`).

Every place that expresses staleness in seconds divides by the declared 10:

| Site | What it controls | Effect |
|---|---|---|
| `scripts/eval_libero.py` (`fps = meta.fps`) | closed-loop refresh schedule | refreshes **2x more often** than the flag says |
| `src/tinyvla/data/mixture.py` (`staleness_max_s * ds.fps`) | training staleness augmentation | nominal 2 s is **0.99 real seconds** |
| `scripts/eval_offline.py` (`int(stale_s * ds.fps)`) | offline staleness curve | `--stale-s 2` is **1 real second** |

Consequences for claims in the card:

* the offline result "stale-latent 2 s == fresh (0.772 vs 0.772)" was measured at
  **1 real second**;
* the released checkpoint has never been trained on staleness beyond ~1 real
  second, so 2 s is outside its training distribution.

Verified by frame count, not by reading: with the declared fps a nominal
`staleness_max_s=2.0` samples a maximum delay of 20 frames (1.00 s at 20 Hz);
with the true rate it samples 40 frames (2.00 s).

### The published command does not enable dual-rate

With `--n-action-steps 10 --refresh-s 1.0`, the refresh period works out to 10
environment steps — exactly the chunk replan interval. Counting the actual calls
over two episodes:

| `--refresh-s` | `_semantic_latent` calls | chunk replans | ratio |
|---|---|---|---|
| 1.0 | 20 | 20 | **1.00** |
| 4.0 | 5 | 20 | 0.25 |

So the two rows of the card's headline table run the semantic path at the same
cadence and differ only in the random draw.

---

## 2. Reproducing the shipped configuration

The published command, re-run through the new harness with its rates measured
(100 episodes, init states 0-9):


| | |
|---|---|
| success | **87/100 = 87.0%** |
| environment | 20.0 Hz |
| fast path (chunk replan) | 2.079 Hz |
| semantic refresh | 2.079 Hz |
| mean latent age | 0.0 s |

That is at or slightly above the higher of the card's two published numbers
(86 and 79), while running both paths at ~2 Hz, which is what the flags actually
produce. The conventions are therefore preserved end to end; what changed is that
the rates are now measured rather than assumed.

---

## 3. Rates separated, 500 paired episodes each

`libero_spatial` has 50 init states per task, so 10 tasks x 50 states = 500
episodes is the full deterministic sweep of the suite. Reference row is
full-rate; `p` is an exact two-sided McNemar test on the discordant pairs.

| Configuration | Success | 95% CI | fast Hz | sem Hz | mean age | sem calls/ep | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|---|---|---|---|
| **full-rate** | **426/500 = 85.2%** | 81.8–88.0 | 20.00 | 20.00 | 0.00 s | 171.0 | — | — | — | — |
| **dual-rate 5 Hz** | **437/500 = 87.4%** | 84.2–90.0 | 20.00 | 5.06 | 0.07 s | 41.1 | +2.2 | +0.0…+4.4 | +21/-10 | 0.071 |
| **dual-rate 2 Hz** | **425/500 = 85.0%** | 81.6–87.9 | 20.00 | 2.07 | 0.22 s | 17.7 | -0.2 | -2.8…+2.2 | +20/-21 | 1.000 |
| **dual-rate 1 Hz** | **422/500 = 84.4%** | 81.0–87.3 | 20.00 | 1.08 | 0.46 s | 9.2 | -0.8 | -3.2…+1.6 | +17/-21 | 0.627 |
| **dual-rate 0.5 Hz** | **426/500 = 85.2%** | 81.8–88.0 | 20.00 | 0.58 | 0.93 s | 4.7 | +0.0 | -3.0…+3.0 | +30/-30 | 1.000 |
| **open-loop chunk** | **437/500 = 87.4%** | 84.2–90.0 | 2.08 | 2.08 | 0.00 s | 16.5 | +2.2 | -1.0…+5.6 | +41/-30 | 0.235 |

"flips" counts episodes this configuration wins that full-rate lost, and the reverse.

**No difference is significant.** The largest gap (dual-rate 5 Hz, +2.2 points)
has p = 0.071. Per-task success agrees within +/-2 episodes on all ten tasks. The
"dual-rate beats full-rate" claim, and the temporal-regularization explanation
offered for it, are not supported at this sample size.

**What is supported:** dropping the semantic path from 20 Hz to 0.5 Hz cuts
semantic calls from 171 to 4.7 per episode, a **36x reduction**, at a mean latent
age of 0.93 s, and costs exactly nothing (85.2% vs 85.2%, 30 flips each way,
p = 1.0). That is the useful half of the architectural claim, now measured
causally rather than inferred from a two-row table.

---

## 4. Success against measured compute

Amortized with measured per-call cost and the rates each run actually ran at.

| Configuration | Success | fast Hz | sem Hz | GFLOPs / robot-second | vs full-rate | compute ms / robot-second | real time on one V100 |
|---|---|---|---|---|---|---|---|
| full-rate | 85.2% | 20.00 | 20.00 | 4493 | 1.00x | 6140 | no |
| dual-rate 5 Hz | 87.4% | 20.00 | 5.06 | 3134 | 0.70x | 3632 | no |
| dual-rate 2 Hz | 85.0% | 20.00 | 2.07 | 2862 | 0.64x | 3130 | no |
| dual-rate 1 Hz | 84.4% | 20.00 | 1.08 | 2771 | 0.62x | 2963 | no |
| dual-rate 0.5 Hz | 85.2% | 20.00 | 0.58 | 2726 | 0.61x | 2879 | no |
| open-loop chunk | 87.4% | 2.08 | 2.08 | 467 | 0.10x | 638 | yes |

Two things fall out of this table.

**Caching the semantic latent saves at most ~39% of compute, not 97%,** because the
floor is set by the fast path (section 6).

**Plain action chunking beats the dual-rate decomposition on both axes.** Executing
10 actions per replan is 9.6x cheaper than the best dual-rate
configuration, scores 2.2 points higher, and is the only configuration that runs in
real time on this hardware. Since the open-loop baseline exists precisely to
separate "the gain comes from dual-rate" from "the gain comes from chunking", on
this benchmark the answer is chunking.

**Neither the semantic rate (20 -> 0.5 Hz) nor the replan rate (20 -> 2 Hz)
changes success.** On its own that is ambiguous: it could mean semantics tolerate
staleness, or it could mean nothing in this scene changes fast enough to matter.
Section 5 separates the two, and the answer is the first one - the suite is not
quasi-static at all, it is specifically insensitive to *semantic* staleness.

---

## 5. The asymmetry: staleness on each path, separately

Constant-age probes rather than a periodic cache, so the age on the x-axis is the
age, not the mean of a sawtooth. 150 paired episodes per cell, compared against the
same episodes of the full-rate run. Proprioception is fresh throughout.

**Staling the semantic latent while geometry stays fresh:**

| latent age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| 0.25 s | 128/150 = 85.3% | 78.8–90.1 | +1.3 | -4.0…+6.7 | +9/-7 | 0.8036 |
| 0.5 s | 129/150 = 86.0% | 79.5–90.7 | +2.0 | -3.3…+7.3 | +10/-7 | 0.6291 |
| 1 s | 128/150 = 85.3% | 78.8–90.1 | +1.3 | -4.0…+6.7 | +9/-7 | 0.8036 |
| 2 s | 126/150 = 84.0% | 77.3–89.0 | +0.0 | -5.3…+5.3 | +8/-8 | 1.0000 |
| 4 s | 125/150 = 83.3% | 76.6–88.4 | -0.7 | -6.0…+4.7 | +8/-9 | 1.0000 |

**Staling the geometry while the semantic latent stays fresh:**

| frame age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| 0.25 s | 35/150 = 23.3% | 17.3–30.7 | -60.7 | -68.7…-52.0 | +2/-93 | 0.0000*** |
| 0.5 s | 8/150 = 5.3% | 2.7–10.2 | -78.7 | -85.3…-72.0 | +0/-118 | 0.0000*** |
| 1 s | 4/150 = 2.7% | 1.0–6.7 | -81.3 | -87.3…-74.7 | +1/-123 | 0.0000*** |
| 2 s | 2/150 = 1.3% | 0.4–4.7 | -82.7 | -88.7…-76.0 | +0/-124 | 0.0000*** |
| 4 s | 0/150 = 0.0% | 0.0–2.5 | -84.0 | -89.3…-78.0 | +0/-126 | 0.0000*** |

**Both stale together:**

| both aged | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| 0.5 s | 6/150 = 4.0% | 1.8–8.5 | -80.0 | -86.7…-72.7 | +2/-122 | 0.0000*** |
| 2 s | 1/150 = 0.7% | 0.1–3.7 | -83.3 | -89.3…-77.3 | +0/-125 | 0.0000*** |

**Controls - is the semantic path used at all?**

| control | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| latent zeroed | 16/150 = 10.7% | 6.7–16.6 | -73.3 | -80.7…-66.0 | +1/-111 | 0.0000*** |
| latent from a pixel-shuffled frame | 127/150 = 84.7% | 78.0–89.6 | +0.7 | -6.0…+7.3 | +13/-12 | 1.0000 |

### Reading

At four seconds of staleness the geometric path costs **84.0 points**
while the semantic path costs **0.7**, which is not distinguishable from
zero - no semantic cell reaches significance at any age out to 4 s, and their
paired intervals all straddle 0. (A ratio of the two is not quoted: the denominator
is consistent with zero, so it would be arbitrarily large and meaningless.) Every
geometric cell is significant at p < 0.0001, and a quarter of a second - five frames
at 20 Hz - already costs 60.7 points. The both-stale
column tracks the geometry column, so geometric staleness accounts for the damage
and semantic staleness adds nothing on top.

The control rules out the obvious alternative explanation. Zeroing the latent
costs 73.3 points, so the policy is not ignoring the semantic path - it needs
it, it just does not need it recent. That is the claim this architecture rests on,
and here it is separated from "the benchmark is static" and from "the latent is
decorative".

**A second control narrows what the semantic path is actually doing.** Shuffling
the pixels of the frame fed to it - destroying all spatial structure while keeping
the same pixel statistics - costs 0.7 points and is not significant
(p = 1.00). Zeroing the latent outright costs 73.3. So whatever the
semantic path contributes is not fine-grained scene layout: something coarse
(a colour or object cue, or the interaction with the language tokens) survives a
full spatial scramble, but nothing survives replacing the latent with zeros.

### Is the geometric collapse about old pixels, or a self-inconsistent observation?

The grid above stales the pixels while proprioception stays fresh, which hands the
policy an observation whose parts disagree in time. If *that* disagreement were the
problem, holding proprioception back by the same amount - keeping the observation
internally consistent, which is what caching the vision tower on a real robot would
actually look like - should help.

| age | pixels-only stale | pixels + proprio stale | delta | flips | McNemar p |
|---|---|---|---|---|---|
| 0.25 s | 23.3% | 14.0% | -9.3 | +6/-20 | 0.0094** |
| 0.5 s | 5.3% | 0.0% | -5.3 | +0/-8 | 0.0078** |
| 1.0 s | 2.7% | 0.7% | -2.0 | +1/-4 | 0.3750 |

All 3 ages tested, all in the same direction: making the observation internally consistent does not help, and at 2/3 of them it is significantly *worse*. So the collapse is not explained by the observation disagreeing with itself — staling proprioception on top of the pixels adds damage rather than removing it. Holding proprioception back to make a cached vision tower's input "agree with itself" is not a viable mitigation: geometric staleness costs, and adding more of it costs more.

---

## 6. System profile

Measured on an idle machine, batch 1, fp32, 30 iterations (5 on CPU). No edge
device is attached, so CPU stands in as the constrained-device proxy; no
Jetson or Raspberry Pi number is reported because none was measured.

| Path | Params | GFLOPs / call | V100 p50 | V100 p95 | CPU p50 |
|---|---|---|---|---|---|
| semantic (slow) | 604.5M | 91 | 168.0 ms | 171.67 ms | 412.63 ms |
| fast (spatial + expert) | 38.9M | 134 | 138.11 ms | 141.23 ms | 577.49 ms |
| both (no cache) | — | 225 | 324.06 ms | 329.96 ms | 982.55 ms |
| vision tower, one camera | 100.6M | 45 | 18.95 ms | 19.26 ms | 146.0 ms |

Peak VRAM 2.61 GB.

**Parameter count and FLOPs point in opposite directions.** The 38.9M fast path
costs more per call (134 GFLOPs) than the 604.5M semantic path
(91 GFLOPs), because the fast path pushes **both cameras through the shared
100.6M vision tower on every step** (2 x 45
GFLOPs). The tower's parameters are counted under the semantic branch; its FLOPs
are paid by the fast branch.

So the "~140M online per tick" figure is accurate as a parameter count and
misleading as a cost: at 20 Hz control the fast path alone needs
2782 ms of compute per robot second on a V100. The remaining
headroom is in the visual encoder, not in the language stack.

---

## 7. Adaptive refresh — threshold calibration

A drift gate that refreshes when the pooled cam0 spatial tokens have moved away
from those at the last refresh (plus a proprio term and age bounds). The signal
is free: `_encode_spatial` already caches that projection. Calibration so the gate
can be compared against fixed schedules at a **matched number of semantic calls**
rather than at a matched frequency:

| threshold tau | measured semantic Hz |
|---|---|
| 0.05 | 5.24 |
| 0.15 | 2.79 |
| 0.30 | 1.55 |
| 0.50 | 1.16 |

### Fixed vs. adaptive, at a matched budget

150 paired episodes per gate, same episodes as `full_rate`'s first 150. `random`
uses the same expected call budget as `fixed` (both target ~1 Hz) - the point of
including it is that an adaptive gate beating a *fixed* schedule is a weak result
on its own, since simply refreshing at different points in the episode could look
like it "learned something" even from noise; `random` is the actual null hypothesis
a real adaptive gate has to beat.

One bug found and fixed while building this table, worth recording: the model's
own `_conditioning` silently recomputes the semantic latent internally whenever
`batch["semantic_latent"]` is absent (which is exactly the state before an
episode's first scheduled refresh), and that internal recompute was being counted
by the same instrumentation that counts the harness's own scheduled calls. Every
schedule except `random` happens to force a refresh within its first `max_period_s`
regardless (so this was invisible for them), but `random` has no such cap and
could go arbitrarily long without a hit - one unlucky early stretch of an episode
meant every single step silently double-counted as a "semantic call", which is
what produced 3.5-4x inflated rates on the first pass. Fixed by forcing an
explicit bootstrap call at step 0 of every episode (matching what every other
schedule already guaranteed) and capping `random`'s tail at 4 s like the others.
Regression-checked against the periodic-schedule numbers in section 3 (bit-for-bit
identical) before any of the numbers below were produced.

| gate | measured Hz | calls/ep | Success | 95% CI | delta | flips | McNemar p |
|---|---|---|---|---|---|---|---|
| fixed 1 Hz | 1.08 | 9.1 | 128/150 = 85.3% | [78.8,90.1] | +1.3 | +9/-7 | 0.8036 |
| random ~1 Hz (null hypothesis) | 1.18 | 10.0 | 129/150 = 86.0% | [79.5,90.7] | +2.0 | +8/-5 | 0.5811 |
| drift, tau=0.30 | 1.43 | 10.2 | 122/150 = 81.3% | [74.3,86.8] | -2.7 | +6/-10 | 0.4545 |
| drift, tau=0.50 | 0.95 | 6.7 | 127/150 = 84.7% | [78.0,89.6] | +0.7 | +8/-7 | 1.0000 |
| oracle (gripper transitions) | 0.51 | 5.7 | 124/150 = 82.7% | [75.8,87.9] | -1.3 | +6/-8 | 0.7905 |

flips: episodes this gate wins that `full_rate` lost, and the reverse.

None of the five is distinguishable from full-rate at this sample size, matching
section 3's finding that semantic refresh rate does not move success on this
suite - but every one of them does it at 5.7-10.2 semantic calls per episode
against full-rate's 171, a 17-30x reduction. Since even `random` matches `fixed`
here, the honest reading is that nothing about *when* the semantic path refreshes
matters on `libero_spatial` at all (consistent with section 5's causal finding
that only spatial staleness costs anything) - which also means this suite cannot
show whether the drift signal is doing anything useful; that question needs the
perturbation suite (dynamic scene changes), not this one.

---

## 8. Not yet included

Running or pending, and deliberately not summarized here until finished: the
perturbation suite (instruction swap, paraphrase, mid-episode instruction change,
occlusion, latency ramp), retrained staleness and latency-metadata variants, and
same-harness baselines beyond the checkpoints in section 9. Also running now:
`tinyvla2/pretrain_C` fine-tuned on LIBERO with the same recipe as section 10's
`pretrain_B` run (the morphology-conditioning caveat from section 9 applies here
too - this fine-tune only exercises the embodiment-ID path, `morph_encoder` is
still dropped on load). The fresh/stale grid (section 5), the consistent-lag check
(section 5, three ages), and the gate comparison (section 7) are all done: 12 grid
cells, two controls, three consistency cells, five gates - 500 + 12x150 + 2x150 +
3x150 + 5x150 = 3,500 closed-loop episodes.

## 9. Other checkpoints in the same repo, same harness

Three checkpoints that predate the LIBERO fine-tune, evaluated closed-loop on
`libero_spatial` with the exact same conventions and no ablation (full-rate,
100 episodes = init states 0-9). None of them were fine-tuned on LIBERO, so this
sizes how much of the 85.2% full-rate number is the fine-tune rather than the
shared pretrain.

| Checkpoint | What it is | Success | 95% CI |
|---|---|---|---|
| `checkpoints/stage2_final` | SO-100/101 + Bridge mixture, frozen backbone | 0/100 = 0.0% | 0.0-3.7 |
| `checkpoints/stage3_final` | + spatial distillation | 0/100 = 0.0% | 0.0-3.7 |
| `tinyvla2/pretrain_A` | TinyVLA-2 ablation, variant A (native action space) | 0/100 = 0.0% | 0.0-3.7 |
| `tinyvla2/pretrain_B` | TinyVLA-2 ablation, variant B (canonical action space) | 0/100 = 0.0% | 0.0-3.7 |

**`tinyvla2/pretrain_A` scores 0.0% on LIBERO** (0/100), consistent with stage2/stage3: none of the three pre-fine-tune checkpoints do anything useful on this suite. This is a different model from the one the rest of this document ablates — it is the *control* arm of a separate cross-embodiment ablation (variants A/B/C), not a LIBERO checkpoint. Its own model card already reports it failing cross-embodiment transfer on held-out `jaco` (endpoint error 1392 mm vs. a 128 mm zero-prediction floor, n=8) — a result from the author, not independently reproduced here since the held-out-embodiment harness this document uses is LIBERO/MetaWorld/SO-101, not the jaco eval that number came from. `conditioning`, `morph_tokens`, and `action_space` in its config are not implemented in `modeling_tinyvla.py` as of this checkpoint family's release — they were added to `configuration_tinyvla.py` here as pass-through fields so the checkpoint would load, since variant A's own documentation states it is architecturally identical to the embodiment-ID scheme already implemented (`EmbodimentEmbedding`). Whether that also holds for variants B and C is unverified.

**`tinyvla2/pretrain_B` scores 0.0% on LIBERO** (0/100). Same story as A: this is not a LIBERO checkpoint either, and pre-fine-tune performance on this suite tells you nothing about the A-vs-B question, which is about zero-shot transfer to held-out embodiments, not about LIBERO. B differs from A only in `action_space` (canonical vs. native) - same seed, mixture, weights, and step count per both checkpoints' own model cards. B's card reports a mixed zero-shot result: canonical action space is decisive on `jaco` (137 mm vs. A's catastrophic 1392 mm) but *loses* to native on `stretch` (195 mm vs. 150 mm) - so canonicalization is not a strict win, and the authors flag the comparison itself as confounded by differing normalization, deferring the decisive test to few-shot adaptation efficiency once variant C lands. Not independently reproduced here for the same reason as A: this document's harness is LIBERO/MetaWorld/SO-101, not the jaco/stretch eval those numbers came from.

**`tinyvla2/pretrain_C` was not run through this harness.** Its checkpoint carries four trained tensors (`morph_encoder.mlp.{0,2}.{weight,bias}`, a 16-to-512-to-1024 MLP producing the 2 morphology-conditioning tokens the README describes) that `modeling_tinyvla.py` as published has no forward-pass code path for, and the morphology descriptor file its README points to (`configs/morphology/descriptors.yaml`) is not in the repository. Loading it anyway was tested directly: the loader logs `Unexpected key(s) ... morph_encoder.mlp.*` and silently drops all four tensors, and inference would fall back to `embodiment_emb(emb_id)` - the exact conditioning path the README states C replaces. Reporting a LIBERO number from that would measure a model C was never designed to be, not the checkpoint as released. So this document cites only what the author has already published (`tinyvla2/RESULTS.md`, `tinyvla2/fewshot_results.json`), unverified here:

| jaco FT episodes | A (native+ID) | B (canonical+ID) | C (canonical+morph) |
|---|---|---|---|
| 0 (zero-shot) | 1377 mm | 155 mm | 178 mm |
| 100 | 6919 mm | 215 mm | 218 mm |
| 500 | 5667 mm | 198 mm | 197 mm |

(GT trajectories ~210 mm, zero-prediction floor ~128 mm, n=20 held-out jaco episodes.) The author's own verdict: canonical action space (B, C) beats native (A) by 10-45x at every budget and is decisive; morphology tokens (C) show no measurable advantage over a learned ID (B) once actions are canonical - "the win is action-space unification, not morphology conditioning." Native additionally *diverges* under few-shot fine-tuning (A gets worse with more FT episodes, not better), which the author attributes to re-learning a fresh embodiment-ID against a native action distribution.

---

## 10. Fine-tuning pretrain_B and pretrain_C on LIBERO

Section 9's `pretrain_B` row asks what a cross-embodiment pretrain is worth on
LIBERO with no fine-tuning at all (nothing - 0%, same as every other pre-fine-tune
checkpoint). This section asks the useful version of that question: swap the
*starting checkpoint* for `ft4`'s recipe and hold everything else fixed - what does
LIBERO fine-tuning look like starting from the canonical cross-embodiment pretrain
instead of `stage3_distill`?

**Recipe, unmodified from `configs/libero_ft4.yaml` except the resume point:**
unfreeze vision tower + language stack, backbone at 0.1x lr, 40,000 steps,
`resume_from: tinyvla2/pretrain_B` (and separately `pretrain_C`).
`num_embodiments=16` and each pretrain's
`conditioning`/`morph_tokens`/`action_space` fields carried through so the resume
loads with 0 missing / 0 unexpected keys (verified directly before launching).
fp16 instead of bf16 (V100, sm_70); effective batch 48 via 4-way data-parallel
instead of one A6000, matching ft4's effective batch. Final training loss 0.178,
comparable to ft4's reported 0.066 - 500-episode full-rate closed-loop eval,
paired against the same episodes as ft4's own full-rate row (section 3).

`pretrain_C` got the identical treatment. One caveat specific to C: its
`morph_encoder` conditioning has **no forward-pass implementation** in this
codebase, so a zero-shot evaluation of C would silently measure a *different*
model than the one the checkpoint describes, and none is reported here. Fine-tuning
is still legitimate — the backbone unfreezes either way, so `embodiment_emb`
receives real gradient — and that is what the C row below measures.

| | Success | 95% CI | delta vs. ft4 | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| ft4 (from stage3_distill) | 426/500 = 85.2% | — | — | — | — | — |
| **B fine-tuned (from pretrain_B)** | 345/500 = 69.0% | [64.8,72.9] | -16.2 | [-21.0,-11.4] | +40/-121 | 0.0000*** |
| **C fine-tuned (from pretrain_C)** | 372/500 = 74.4% | [70.4,78.0] | -10.8 | [-15.2,-6.6] | +36/-90 | 0.0000*** |

**Neither cross-embodiment pretrain makes a better starting point than
stage3_distill for LIBERO specifically - both make a worse one, by wide and
significant margins.** Head-to-head on the same 500 paired episodes, **C beats B by +5.4 pp** (paired 95% CI [+0.0,+10.8], McNemar p=0.0623, +111/-84 flips) - which does **not** clear significance at 0.05, so the two cross-embodiment pretrains are not separated by this evidence. This isn't
one catastrophic task dragging an average down: 7/10 tasks score lower for the B-derived fine-tune than for ft4 (the other 3 are within a couple of episodes either way). Worst-hit: task8 at 9/50 vs. ft4's 39/50. Best-preserved: task5 at 49/50 vs. ft4's 42/50. Both checkpoints have identical architecture and parameter/FLOP counts (confirmed via the same profiling harness as section 6) - this is purely a pretraining-lineage effect, not a compute or capacity difference.

Plausible reason: `stage3_distill` was trained specifically as *this* model's
LIBERO-adjacent lineage (stage 1-3, spatial distillation against LIBERO-style
labels per the model card), whereas `pretrain_B` is a from-scratch TinyVLA-2 run
on a disjoint mixture (SO-100/101 + Bridge + RT-1, no LIBERO exposure anywhere in
its lineage, canonical-EE-space objective) built for a different question entirely
(cross-embodiment transfer to `jaco`/`stretch`, not LIBERO manipulation quality).
40k fine-tuning steps evidently were not enough to close that gap. This is a data
point against "any reasonable pretrain fine-tunes to a similar place on LIBERO",
not evidence about canonical vs. native action spaces (both `ft4`'s lineage and
`pretrain_B` ultimately regress against LIBERO's own action convention once
fine-tuning starts, per section 1's frame-rate check applying identically to
both - the difference is what the backbone learned before LIBERO ever entered
the picture).


---

## 11. Zero-shot transfer to RoboCasa (a genuinely different domain)

Everything above is LIBERO: one fixed tabletop Panda, one scene layout family.
This section asks whether `ft4` transfers with no fine-tuning at all to
[RoboCasa365](https://robocasa.ai/) — PandaOmron (Franka arm on a holonomic
mobile base) in procedurally-varied kitchens, a different robot, different
scenes, and a different observation layout.

**Conventions were verified empirically, not assumed** (`verify_robocasa.py`,
`verify_robocasa_axes.py`), because a silent frame mismatch here would be
indistinguishable from a policy failure:

* **Gripper**: RoboCasa's -1=open/+1=closed matches LIBERO's polarity directly
  (measured qpos 0.039 at -1, 0.001 at +1). No sign flip.
* **Action**: both use the same OSC_POSE delta controller at the same physical
  scaling (±0.05 m, ±0.5 rad/step), so LIBERO's unnormalized
  `[dpos(3), drot(3), grip(1)]` maps onto RoboCasa's `ee_pos/ee_rot/gripper`
  slots. `base_motion(4)` and `control_mode(1)` held at zero — arm-only.
* **State**: RoboCasa's native 16-dim `agent_pos` is a *different layout*
  (base pose + relative EE) from the trained convention (absolute EE pose +
  gripper), converted via `ee_state_from_agent_pos` before normalizing. Feeding
  the raw vector through would put a base quaternion where the model expects an
  EE rotvec.
* **Cameras**: `robot0_agentview_left` + `robot0_eye_in_hand`, used raw. A
  crosshair projection through the camera's own MuJoCo extrinsics shows +y and
  +z already match LIBERO's screen convention (+y→left, +z→up), while +x is a
  *diagonal* blend of both screen axes rather than LIBERO's clean "+x→down" —
  this camera looks into the kitchen at an angle instead of down a tabletop.
  **No correction is applied**: a reflection that fixed x would break the y/z
  agreement. This is a documented residual mismatch, not a fixed one.

Each task runs at RoboCasa's own registered horizon (`get_task_horizon`), not a
truncated budget: 900 steps for CloseFridge, 750 for OpenDrawer, 450-600 for the
rest. Semantic refresh 1 Hz, `n_action_steps=1`, 15 episodes/task.

| Task | Horizon | Object registry | Success | 95% Wilson | Note |
|---|---|---|---|---|---|
| `CloseFridge` | 900 | lightwheel | 0/15 | 0.0-20.4 | — |
| `OpenDrawer` | 750 | lightwheel | 0/15 | 0.0-20.4 | — |
| `NavigateKitchen` | 450 | lightwheel | 0/15 | 0.0-20.4 | requires mobile-base motion; harness holds base_motion at zero |
| `TurnOnMicrowave` | 450 | lightwheel+objaverse+aigen | 0/15 | 0.0-20.4 | — |
| `PickPlaceCounterToStove` | 600 | lightwheel+objaverse+aigen | 0/15 | 0.0-20.4 | — |

**Zero-shot transfer is a complete failure: 0/60 across the 4 arm-only tasks (95% Wilson 0.0-6.0%).** Not a partial-credit
result to be tuned up — nothing succeeds anywhere.

Two failure modes are visible in the saved videos (`results/robocasa_ft4/`), and
they are *behavioural*, not random flailing: the arm drifts steadily upward until
it clips the ceiling, and the wrist camera rotates until it faces the floor. Both
are consistent with the one convention that could **not** be pinned down
empirically — the correspondence between LIBERO's rotation axes and RoboCasa's
`ee_rot`. A systematic rotation-frame error produces exactly this: a constant
spurious delta integrated over hundreds of steps.

**Section 12 tests that hypothesis and does not confirm it.** Fine-tuning on
RoboCasa's own data fixes the trained task completely but moves none of the
held-out ones — which a repaired shared rotation frame should have done. Treat
the paragraph above as the motivating hypothesis, not a finding.

Two honest caveats on the table above:

* `NavigateKitchen` is **not a measurement of the policy**. The task requires
  driving the mobile base across the kitchen, and this harness pins
  `base_motion` to zero. Its 0/15 is guaranteed by construction, so it is
  excluded from the aggregate.
* The object registry differs by row. `TurnOnMicrowave` and
  `PickPlaceCounterToStove` sample object categories that do not exist in the
  `lightwheel` registry and **cannot be constructed** without objaverse/aigen
  assets, so a single registry across all five tasks was not possible without
  re-running the first two. The fixtures each task acts on (fridge, drawer,
  microwave, stove) are scene fixtures, not sampled objects; the registry
  affects distractor content.

This is the motivation for section 12: rather than hand-tuning the one
unverifiable frame convention, fine-tune on RoboCasa's own recorded data and let
the model learn whatever axis mapping the dataset actually contains.


---

## 12. Fine-tuning on RoboCasa's own data

Section 11 leaves one convention unverifiable by construction — the mapping
between LIBERO's rotation axes and RoboCasa's `ee_rot` — and blames the zero-shot
collapse on it. Rather than hand-tune that mapping, this section removes the
question: fine-tune on RoboCasa's *own recorded actions*, so the model learns
whatever axis convention the data actually contains.

**Data**: [`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge),
513 episodes / 155,443 frames at a correctly-declared 20 fps (no repeat of
section 1's frame-rate bug). RoboCasa's native 16-dim `agent_pos` and 12-dim flat
action are converted to this repo's canonical 8-dim state / 7-dim action schema by
`robocasa_train_source.py`, and normalized with statistics computed over the whole
dataset (`compute_robocasa_stats.py`), **not** LIBERO's. Feeding the raw vectors
through `CanonicalSource` would have silently truncated the 12-dim action to its
first 8 entries — `base_motion` + `control_mode` + a partial `ee_pos` — which is
why a dedicated source class exists rather than a config flag.

**Recipe**: `ft4` as the starting point, 8,000 steps at effective batch 24
(2xV100, fp16), lr 1e-4, backbone unfrozen at 0.1x — the ft4 recipe on a shorter
schedule (~1.24 epochs) because this is one task with 155k frames rather than
LIBERO's 500-task mixture. Final training loss ~0.35.

Evaluation is the *same harness, same seeds, same object registry* as section 11's
zero-shot row, with one necessary difference: the fine-tuned checkpoint is
unnormalized with its own RoboCasa statistics via `--stats-json`. Scoring it with
LIBERO's statistics would push actions through the wrong mean/std and report a
failure that belongs to the harness, not the policy.

| | Success | 95% Wilson | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| `ft4` zero-shot | 0/15 = 0.0% | — | — | — | — | — |
| **RoboCasa fine-tune** | 7/15 = 46.7% | [24.8,69.9] | +46.7 | [+20.0,+73.3] | +7/-0 | 0.0156 |

15 paired episodes (same seeds, same object registry). Pairing verified directly: the sampled instruction string matches on 15/15 episode indices, so the two runs saw the same scenes.

**Fine-tuning works, and the effect is unambiguous**: every flip goes one way
(+7/-0), so this is not variance around a shared mean. The two
zero-shot failure modes from section 11 — upward drift into the ceiling, wrist
camera rotating to face the floor — are absent from the fine-tuned videos
(`results/robocasa_ftrc/`).

### The residual failures are structured, not random

| Scene variant | Instruction | Success | 95% Wilson |
|---|---|---|---|
| single-door fridge | `Close the fridge door.` | 7/10 | [39.7,89.2] |
| double-door fridge | `Close the fridge doors.` | 0/5 | [0.0,43.4] |

Fisher exact p=0.0256. **This is not a training-data gap**: both instruction variants are
well represented in the fine-tuning set (84,341 frames for the single-door
phrasing, 71,102 for the double-door one — 54% / 46%). The model saw plenty of
double-door episodes and still closes none of them. Closing two doors requires
sequencing two separate contacts; closing one does not. With n=5 double-door
episodes this is a suggestive split, not a settled result — but it is the correct
place to look next, and it is invisible in the 7/15 aggregate.

### It did not fix a global convention error

The obvious reading of section 11 is that one bad rotation mapping broke every
task, in which case fine-tuning on any single task should repair the shared
representation and lift the others too. It does not. Same fine-tuned checkpoint,
same harness, tasks never seen during fine-tuning:

| Held-out task | Success | 95% Wilson |
|---|---|---|
| `OpenDrawer` | 0/15 | [0.0,20.4] |
| `TurnOnMicrowave` | 0/15 | [0.0,20.4] |
| `PickPlaceCounterToStove` | 0/15 | [0.0,20.4] |

**no measurable transfer**: 0/45 (95% Wilson 0.0-7.9%).

So the honest conclusion is narrower than "fine-tuning fixes RoboCasa transfer."
Fine-tuning demonstrably solves *the task it was trained on*, moving it from
categorically broken to roughly half-successful. It provides **no evidence** that
the zero-shot collapse was purely a frame-convention artifact — had it been, the
held-out tasks would have moved. What the model acquired looks task-specific
rather than a repaired shared geometry. Distinguishing "memorized one task" from
"the other tasks need capabilities beyond the convention" needs a multi-task
fine-tune, which is the next experiment, not a claim this data supports.


---

## Reproduction

```bash
# the published configuration, with its rates measured
python eval_libero_rates.py --n-action-steps 10 --semantic-hz 2 \
    --n-init-states 10 --out-json accept.json

# full-rate and dual-rate, full 500-episode paired sweep
./run_sharded.sh full_rate 50 --n-action-steps 1 --semantic-hz 20
./run_sharded.sh dual_1hz  50 --n-action-steps 1 --semantic-hz 1

# paired statistics
python stats.py results/rates/full_rate.json results/rates/dual_1hz.json
```

RoboCasa (sections 11-12) runs in a separate venv — robosuite 1.5.2, versus the
1.4.0 that LIBERO's eval depends on and that must not be disturbed:

```bash
source .venv_robocasa/bin/activate

# zero-shot: LIBERO-trained checkpoint, LIBERO normalization (the default)
python eval_robocasa.py --checkpoint tinyvla     --tasks CloseFridge OpenDrawer --episodes 15     --n-action-steps 1 --refresh-s 1.0     --out-json results/rates/robocasa_ft4.json --tag robocasa_ft4

# fine-tune on RoboCasa's own data (main venv, 2-GPU DDP)
accelerate launch --multi_gpu --num_processes 2 --mixed_precision fp16     tinyvla/scripts/train.py --config tinyvla/configs/robocasa_ft_closefridge.yaml

# evaluate it with ITS OWN normalization — LIBERO's stats would measure noise
python eval_robocasa.py --checkpoint outputs/robocasa_ft_closefridge/final     --stats-json robocasa_stats.json     --tasks CloseFridge --episodes 15 --n-action-steps 1 --refresh-s 1.0     --out-json results/rates/robocasa_ftrc.json --tag robocasa_ftrc
```

Tasks whose scenes sample objaverse/aigen object categories (`TurnOnMicrowave`,
`PickPlaceCounterToStove`) need those registries downloaded
(`python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse objs_aigen`)
and passed via `--obj-registries lightwheel objaverse aigen`; without them the
environment cannot be constructed at all.