TinyVLA — corrected evaluation and ablations
Measured independently of the numbers in the model card, on a re-implemented harness that separates the three rates and logs them instead of inferring them. All closed-loop numbers below are paired: every configuration runs the same (task, init state) episodes, and the flow-matching noise is seeded per (episode, step), so two rows differ only by the thing being ablated.
Hardware: 4x Tesla V100-SXM2-32GB (sm_70), fp32, driver 535 / CUDA 12.2,
torch 2.6.0+cu124, transformers 5.5.4, lerobot 0.6.1. Suite: libero_spatial.
Generated 2026-08-25.
The headline dual-rate comparison in the model card is not reproduced. With the clock fixed and 500 paired episodes per configuration, no refresh rate differs significantly from any other. What does survive is the efficiency claim, and it survives causally: see sections 3 and 4.
1. The refresh schedule was derived from the wrong frame rate
HuggingFaceVLA/libero declares fps = 10, but its frames are native 20 Hz. The
dataset has 1693 episodes and 273,465 frames, i.e. 161.5 frames per episode;
LIBERO demos run about 8 s, giving 20.2 Hz, which matches the environment
(lerobot.envs.configs sets fps: int = 20 # Must match robosuite's default control_freq (20 Hz) and LiberoEnv(control_freq=20)).
Every place that expresses staleness in seconds divides by the declared 10:
| Site | What it controls | Effect |
|---|---|---|
scripts/eval_libero.py (fps = meta.fps) |
closed-loop refresh schedule | refreshes 2x more often than the flag says |
src/tinyvla/data/mixture.py (staleness_max_s * ds.fps) |
training staleness augmentation | nominal 2 s is 0.99 real seconds |
scripts/eval_offline.py (int(stale_s * ds.fps)) |
offline staleness curve | --stale-s 2 is 1 real second |
Consequences for claims in the card:
- the offline result "stale-latent 2 s == fresh (0.772 vs 0.772)" was measured at 1 real second;
- the released checkpoint has never been trained on staleness beyond ~1 real second, so 2 s is outside its training distribution.
Verified by frame count, not by reading: with the declared fps a nominal
staleness_max_s=2.0 samples a maximum delay of 20 frames (1.00 s at 20 Hz);
with the true rate it samples 40 frames (2.00 s).
The published command does not enable dual-rate
With --n-action-steps 10 --refresh-s 1.0, the refresh period works out to 10
environment steps — exactly the chunk replan interval. Counting the actual calls
over two episodes:
--refresh-s |
_semantic_latent calls |
chunk replans | ratio |
|---|---|---|---|
| 1.0 | 20 | 20 | 1.00 |
| 4.0 | 5 | 20 | 0.25 |
So the two rows of the card's headline table run the semantic path at the same cadence and differ only in the random draw.
2. Reproducing the shipped configuration
The published command, re-run through the new harness with its rates measured (100 episodes, init states 0-9):
| success | 87/100 = 87.0% |
| environment | 20.0 Hz |
| fast path (chunk replan) | 2.079 Hz |
| semantic refresh | 2.079 Hz |
| mean latent age | 0.0 s |
That is at or slightly above the higher of the card's two published numbers (86 and 79), while running both paths at ~2 Hz, which is what the flags actually produce. The conventions are therefore preserved end to end; what changed is that the rates are now measured rather than assumed.
3. Rates separated, 500 paired episodes each
libero_spatial has 50 init states per task, so 10 tasks x 50 states = 500
episodes is the full deterministic sweep of the suite. Reference row is
full-rate; p is an exact two-sided McNemar test on the discordant pairs.
| Configuration | Success | 95% CI | fast Hz | sem Hz | mean age | sem calls/ep | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|---|---|---|---|
| full-rate | 426/500 = 85.2% | 81.8–88.0 | 20.00 | 20.00 | 0.00 s | 171.0 | — | — | — | — |
| dual-rate 5 Hz | 437/500 = 87.4% | 84.2–90.0 | 20.00 | 5.06 | 0.07 s | 41.1 | +2.2 | +0.0…+4.4 | +21/-10 | 0.071 |
| dual-rate 2 Hz | 425/500 = 85.0% | 81.6–87.9 | 20.00 | 2.07 | 0.22 s | 17.7 | -0.2 | -2.8…+2.2 | +20/-21 | 1.000 |
| dual-rate 1 Hz | 422/500 = 84.4% | 81.0–87.3 | 20.00 | 1.08 | 0.46 s | 9.2 | -0.8 | -3.2…+1.6 | +17/-21 | 0.627 |
| dual-rate 0.5 Hz | 426/500 = 85.2% | 81.8–88.0 | 20.00 | 0.58 | 0.93 s | 4.7 | +0.0 | -3.0…+3.0 | +30/-30 | 1.000 |
| open-loop chunk | 437/500 = 87.4% | 84.2–90.0 | 2.08 | 2.08 | 0.00 s | 16.5 | +2.2 | -1.0…+5.6 | +41/-30 | 0.235 |
"flips" counts episodes this configuration wins that full-rate lost, and the reverse.
No difference is significant. The largest gap (dual-rate 5 Hz, +2.2 points) has p = 0.071. Per-task success agrees within +/-2 episodes on all ten tasks. The "dual-rate beats full-rate" claim, and the temporal-regularization explanation offered for it, are not supported at this sample size.
What is supported: dropping the semantic path from 20 Hz to 0.5 Hz cuts semantic calls from 171 to 4.7 per episode, a 36x reduction, at a mean latent age of 0.93 s, and costs exactly nothing (85.2% vs 85.2%, 30 flips each way, p = 1.0). That is the useful half of the architectural claim, now measured causally rather than inferred from a two-row table.
4. Success against measured compute
Amortized with measured per-call cost and the rates each run actually ran at.
| Configuration | Success | fast Hz | sem Hz | GFLOPs / robot-second | vs full-rate | compute ms / robot-second | real time on one V100 |
|---|---|---|---|---|---|---|---|
| full-rate | 85.2% | 20.00 | 20.00 | 4493 | 1.00x | 6140 | no |
| dual-rate 5 Hz | 87.4% | 20.00 | 5.06 | 3134 | 0.70x | 3632 | no |
| dual-rate 2 Hz | 85.0% | 20.00 | 2.07 | 2862 | 0.64x | 3130 | no |
| dual-rate 1 Hz | 84.4% | 20.00 | 1.08 | 2771 | 0.62x | 2963 | no |
| dual-rate 0.5 Hz | 85.2% | 20.00 | 0.58 | 2726 | 0.61x | 2879 | no |
| open-loop chunk | 87.4% | 2.08 | 2.08 | 467 | 0.10x | 638 | yes |
Two things fall out of this table.
Caching the semantic latent saves at most ~39% of compute, not 97%, because the floor is set by the fast path (section 6).
Plain action chunking beats the dual-rate decomposition on both axes. Executing 10 actions per replan is 9.6x cheaper than the best dual-rate configuration, scores 2.2 points higher, and is the only configuration that runs in real time on this hardware. Since the open-loop baseline exists precisely to separate "the gain comes from dual-rate" from "the gain comes from chunking", on this benchmark the answer is chunking.
Neither the semantic rate (20 -> 0.5 Hz) nor the replan rate (20 -> 2 Hz) changes success. On its own that is ambiguous: it could mean semantics tolerate staleness, or it could mean nothing in this scene changes fast enough to matter. Section 5 separates the two, and the answer is the first one - the suite is not quasi-static at all, it is specifically insensitive to semantic staleness.
5. The asymmetry: staleness on each path, separately
Constant-age probes rather than a periodic cache, so the age on the x-axis is the age, not the mean of a sawtooth. 150 paired episodes per cell, compared against the same episodes of the full-rate run. Proprioception is fresh throughout.
Staling the semantic latent while geometry stays fresh:
| latent age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| 0.25 s | 128/150 = 85.3% | 78.8–90.1 | +1.3 | -4.0…+6.7 | +9/-7 | 0.8036 |
| 0.5 s | 129/150 = 86.0% | 79.5–90.7 | +2.0 | -3.3…+7.3 | +10/-7 | 0.6291 |
| 1 s | 128/150 = 85.3% | 78.8–90.1 | +1.3 | -4.0…+6.7 | +9/-7 | 0.8036 |
| 2 s | 126/150 = 84.0% | 77.3–89.0 | +0.0 | -5.3…+5.3 | +8/-8 | 1.0000 |
| 4 s | 125/150 = 83.3% | 76.6–88.4 | -0.7 | -6.0…+4.7 | +8/-9 | 1.0000 |
Staling the geometry while the semantic latent stays fresh:
| frame age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| 0.25 s | 35/150 = 23.3% | 17.3–30.7 | -60.7 | -68.7…-52.0 | +2/-93 | 0.0000*** |
| 0.5 s | 8/150 = 5.3% | 2.7–10.2 | -78.7 | -85.3…-72.0 | +0/-118 | 0.0000*** |
| 1 s | 4/150 = 2.7% | 1.0–6.7 | -81.3 | -87.3…-74.7 | +1/-123 | 0.0000*** |
| 2 s | 2/150 = 1.3% | 0.4–4.7 | -82.7 | -88.7…-76.0 | +0/-124 | 0.0000*** |
| 4 s | 0/150 = 0.0% | 0.0–2.5 | -84.0 | -89.3…-78.0 | +0/-126 | 0.0000*** |
Both stale together:
| both aged | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| 0.5 s | 6/150 = 4.0% | 1.8–8.5 | -80.0 | -86.7…-72.7 | +2/-122 | 0.0000*** |
| 2 s | 1/150 = 0.7% | 0.1–3.7 | -83.3 | -89.3…-77.3 | +0/-125 | 0.0000*** |
Controls - is the semantic path used at all?
| control | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
|---|---|---|---|---|---|---|
| latent zeroed | 16/150 = 10.7% | 6.7–16.6 | -73.3 | -80.7…-66.0 | +1/-111 | 0.0000*** |
| latent from a pixel-shuffled frame | 127/150 = 84.7% | 78.0–89.6 | +0.7 | -6.0…+7.3 | +13/-12 | 1.0000 |
Reading
At four seconds of staleness the geometric path costs 84.0 points while the semantic path costs 0.7, which is not distinguishable from zero - no semantic cell reaches significance at any age out to 4 s, and their paired intervals all straddle 0. (A ratio of the two is not quoted: the denominator is consistent with zero, so it would be arbitrarily large and meaningless.) Every geometric cell is significant at p < 0.0001, and a quarter of a second - five frames at 20 Hz - already costs 60.7 points. The both-stale column tracks the geometry column, so geometric staleness accounts for the damage and semantic staleness adds nothing on top.
The control rules out the obvious alternative explanation. Zeroing the latent costs 73.3 points, so the policy is not ignoring the semantic path - it needs it, it just does not need it recent. That is the claim this architecture rests on, and here it is separated from "the benchmark is static" and from "the latent is decorative".
A second control narrows what the semantic path is actually doing. Shuffling the pixels of the frame fed to it - destroying all spatial structure while keeping the same pixel statistics - costs 0.7 points and is not significant (p = 1.00). Zeroing the latent outright costs 73.3. So whatever the semantic path contributes is not fine-grained scene layout: something coarse (a colour or object cue, or the interaction with the language tokens) survives a full spatial scramble, but nothing survives replacing the latent with zeros.
Is the geometric collapse about old pixels, or a self-inconsistent observation?
The grid above stales the pixels while proprioception stays fresh, which hands the policy an observation whose parts disagree in time. If that disagreement were the problem, holding proprioception back by the same amount - keeping the observation internally consistent, which is what caching the vision tower on a real robot would actually look like - should help.
| age | pixels-only stale | pixels + proprio stale | delta | flips | McNemar p |
|---|---|---|---|---|---|
| 0.25 s | 23.3% | 14.0% | -9.3 | +6/-20 | 0.0094** |
| 0.5 s | 5.3% | 0.0% | -5.3 | +0/-8 | 0.0078** |
| 1.0 s | 2.7% | 0.7% | -2.0 | +1/-4 | 0.3750 |
All 3 ages tested, all in the same direction: making the observation internally consistent does not help, and at 2/3 of them it is significantly worse. So the collapse is not explained by the observation disagreeing with itself — staling proprioception on top of the pixels adds damage rather than removing it. Holding proprioception back to make a cached vision tower's input "agree with itself" is not a viable mitigation: geometric staleness costs, and adding more of it costs more.
6. System profile
Measured on an idle machine, batch 1, fp32, 30 iterations (5 on CPU). No edge device is attached, so CPU stands in as the constrained-device proxy; no Jetson or Raspberry Pi number is reported because none was measured.
| Path | Params | GFLOPs / call | V100 p50 | V100 p95 | CPU p50 |
|---|---|---|---|---|---|
| semantic (slow) | 604.5M | 91 | 168.0 ms | 171.67 ms | 412.63 ms |
| fast (spatial + expert) | 38.9M | 134 | 138.11 ms | 141.23 ms | 577.49 ms |
| both (no cache) | — | 225 | 324.06 ms | 329.96 ms | 982.55 ms |
| vision tower, one camera | 100.6M | 45 | 18.95 ms | 19.26 ms | 146.0 ms |
Peak VRAM 2.61 GB.
Parameter count and FLOPs point in opposite directions. The 38.9M fast path costs more per call (134 GFLOPs) than the 604.5M semantic path (91 GFLOPs), because the fast path pushes both cameras through the shared 100.6M vision tower on every step (2 x 45 GFLOPs). The tower's parameters are counted under the semantic branch; its FLOPs are paid by the fast branch.
So the "~140M online per tick" figure is accurate as a parameter count and misleading as a cost: at 20 Hz control the fast path alone needs 2782 ms of compute per robot second on a V100. The remaining headroom is in the visual encoder, not in the language stack.
7. Adaptive refresh — threshold calibration
A drift gate that refreshes when the pooled cam0 spatial tokens have moved away
from those at the last refresh (plus a proprio term and age bounds). The signal
is free: _encode_spatial already caches that projection. Calibration so the gate
can be compared against fixed schedules at a matched number of semantic calls
rather than at a matched frequency:
| threshold tau | measured semantic Hz |
|---|---|
| 0.05 | 5.24 |
| 0.15 | 2.79 |
| 0.30 | 1.55 |
| 0.50 | 1.16 |
Fixed vs. adaptive, at a matched budget
150 paired episodes per gate, same episodes as full_rate's first 150. random
uses the same expected call budget as fixed (both target ~1 Hz) - the point of
including it is that an adaptive gate beating a fixed schedule is a weak result
on its own, since simply refreshing at different points in the episode could look
like it "learned something" even from noise; random is the actual null hypothesis
a real adaptive gate has to beat.
One bug found and fixed while building this table, worth recording: the model's
own _conditioning silently recomputes the semantic latent internally whenever
batch["semantic_latent"] is absent (which is exactly the state before an
episode's first scheduled refresh), and that internal recompute was being counted
by the same instrumentation that counts the harness's own scheduled calls. Every
schedule except random happens to force a refresh within its first max_period_s
regardless (so this was invisible for them), but random has no such cap and
could go arbitrarily long without a hit - one unlucky early stretch of an episode
meant every single step silently double-counted as a "semantic call", which is
what produced 3.5-4x inflated rates on the first pass. Fixed by forcing an
explicit bootstrap call at step 0 of every episode (matching what every other
schedule already guaranteed) and capping random's tail at 4 s like the others.
Regression-checked against the periodic-schedule numbers in section 3 (bit-for-bit
identical) before any of the numbers below were produced.
| gate | measured Hz | calls/ep | Success | 95% CI | delta | flips | McNemar p |
|---|---|---|---|---|---|---|---|
| fixed 1 Hz | 1.08 | 9.1 | 128/150 = 85.3% | [78.8,90.1] | +1.3 | +9/-7 | 0.8036 |
| random ~1 Hz (null hypothesis) | 1.18 | 10.0 | 129/150 = 86.0% | [79.5,90.7] | +2.0 | +8/-5 | 0.5811 |
| drift, tau=0.30 | 1.43 | 10.2 | 122/150 = 81.3% | [74.3,86.8] | -2.7 | +6/-10 | 0.4545 |
| drift, tau=0.50 | 0.95 | 6.7 | 127/150 = 84.7% | [78.0,89.6] | +0.7 | +8/-7 | 1.0000 |
| oracle (gripper transitions) | 0.51 | 5.7 | 124/150 = 82.7% | [75.8,87.9] | -1.3 | +6/-8 | 0.7905 |
flips: episodes this gate wins that full_rate lost, and the reverse.
None of the five is distinguishable from full-rate at this sample size, matching
section 3's finding that semantic refresh rate does not move success on this
suite - but every one of them does it at 5.7-10.2 semantic calls per episode
against full-rate's 171, a 17-30x reduction. Since even random matches fixed
here, the honest reading is that nothing about when the semantic path refreshes
matters on libero_spatial at all (consistent with section 5's causal finding
that only spatial staleness costs anything) - which also means this suite cannot
show whether the drift signal is doing anything useful; that question needs the
perturbation suite (dynamic scene changes), not this one.
8. Not yet included
Running or pending, and deliberately not summarized here until finished: the
perturbation suite (instruction swap, paraphrase, mid-episode instruction change,
occlusion, latency ramp), retrained staleness and latency-metadata variants, and
same-harness baselines beyond the checkpoints in section 9. Also running now:
tinyvla2/pretrain_C fine-tuned on LIBERO with the same recipe as section 10's
pretrain_B run (the morphology-conditioning caveat from section 9 applies here
too - this fine-tune only exercises the embodiment-ID path, morph_encoder is
still dropped on load). The fresh/stale grid (section 5), the consistent-lag check
(section 5, three ages), and the gate comparison (section 7) are all done: 12 grid
cells, two controls, three consistency cells, five gates - 500 + 12x150 + 2x150 +
3x150 + 5x150 = 3,500 closed-loop episodes.
9. Other checkpoints in the same repo, same harness
Three checkpoints that predate the LIBERO fine-tune, evaluated closed-loop on
libero_spatial with the exact same conventions and no ablation (full-rate,
100 episodes = init states 0-9). None of them were fine-tuned on LIBERO, so this
sizes how much of the 85.2% full-rate number is the fine-tune rather than the
shared pretrain.
| Checkpoint | What it is | Success | 95% CI |
|---|---|---|---|
checkpoints/stage2_final |
SO-100/101 + Bridge mixture, frozen backbone | 0/100 = 0.0% | 0.0-3.7 |
checkpoints/stage3_final |
+ spatial distillation | 0/100 = 0.0% | 0.0-3.7 |
tinyvla2/pretrain_A |
TinyVLA-2 ablation, variant A (native action space) | 0/100 = 0.0% | 0.0-3.7 |
tinyvla2/pretrain_B |
TinyVLA-2 ablation, variant B (canonical action space) | 0/100 = 0.0% | 0.0-3.7 |
tinyvla2/pretrain_A scores 0.0% on LIBERO (0/100), consistent with stage2/stage3: none of the three pre-fine-tune checkpoints do anything useful on this suite. This is a different model from the one the rest of this document ablates — it is the control arm of a separate cross-embodiment ablation (variants A/B/C), not a LIBERO checkpoint. Its own model card already reports it failing cross-embodiment transfer on held-out jaco (endpoint error 1392 mm vs. a 128 mm zero-prediction floor, n=8) — a result from the author, not independently reproduced here since the held-out-embodiment harness this document uses is LIBERO/MetaWorld/SO-101, not the jaco eval that number came from. conditioning, morph_tokens, and action_space in its config are not implemented in modeling_tinyvla.py as of this checkpoint family's release — they were added to configuration_tinyvla.py here as pass-through fields so the checkpoint would load, since variant A's own documentation states it is architecturally identical to the embodiment-ID scheme already implemented (EmbodimentEmbedding). Whether that also holds for variants B and C is unverified.
tinyvla2/pretrain_B scores 0.0% on LIBERO (0/100). Same story as A: this is not a LIBERO checkpoint either, and pre-fine-tune performance on this suite tells you nothing about the A-vs-B question, which is about zero-shot transfer to held-out embodiments, not about LIBERO. B differs from A only in action_space (canonical vs. native) - same seed, mixture, weights, and step count per both checkpoints' own model cards. B's card reports a mixed zero-shot result: canonical action space is decisive on jaco (137 mm vs. A's catastrophic 1392 mm) but loses to native on stretch (195 mm vs. 150 mm) - so canonicalization is not a strict win, and the authors flag the comparison itself as confounded by differing normalization, deferring the decisive test to few-shot adaptation efficiency once variant C lands. Not independently reproduced here for the same reason as A: this document's harness is LIBERO/MetaWorld/SO-101, not the jaco/stretch eval those numbers came from.
tinyvla2/pretrain_C was not run through this harness. Its checkpoint carries four trained tensors (morph_encoder.mlp.{0,2}.{weight,bias}, a 16-to-512-to-1024 MLP producing the 2 morphology-conditioning tokens the README describes) that modeling_tinyvla.py as published has no forward-pass code path for, and the morphology descriptor file its README points to (configs/morphology/descriptors.yaml) is not in the repository. Loading it anyway was tested directly: the loader logs Unexpected key(s) ... morph_encoder.mlp.* and silently drops all four tensors, and inference would fall back to embodiment_emb(emb_id) - the exact conditioning path the README states C replaces. Reporting a LIBERO number from that would measure a model C was never designed to be, not the checkpoint as released. So this document cites only what the author has already published (tinyvla2/RESULTS.md, tinyvla2/fewshot_results.json), unverified here:
| jaco FT episodes | A (native+ID) | B (canonical+ID) | C (canonical+morph) |
|---|---|---|---|
| 0 (zero-shot) | 1377 mm | 155 mm | 178 mm |
| 100 | 6919 mm | 215 mm | 218 mm |
| 500 | 5667 mm | 198 mm | 197 mm |
(GT trajectories ~210 mm, zero-prediction floor ~128 mm, n=20 held-out jaco episodes.) The author's own verdict: canonical action space (B, C) beats native (A) by 10-45x at every budget and is decisive; morphology tokens (C) show no measurable advantage over a learned ID (B) once actions are canonical - "the win is action-space unification, not morphology conditioning." Native additionally diverges under few-shot fine-tuning (A gets worse with more FT episodes, not better), which the author attributes to re-learning a fresh embodiment-ID against a native action distribution.
10. Fine-tuning pretrain_B and pretrain_C on LIBERO
Section 9's pretrain_B row asks what a cross-embodiment pretrain is worth on
LIBERO with no fine-tuning at all (nothing - 0%, same as every other pre-fine-tune
checkpoint). This section asks the useful version of that question: swap the
starting checkpoint for ft4's recipe and hold everything else fixed - what does
LIBERO fine-tuning look like starting from the canonical cross-embodiment pretrain
instead of stage3_distill?
Recipe, unmodified from configs/libero_ft4.yaml except the resume point:
unfreeze vision tower + language stack, backbone at 0.1x lr, 40,000 steps,
resume_from: tinyvla2/pretrain_B (and separately pretrain_C).
num_embodiments=16 and each pretrain's
conditioning/morph_tokens/action_space fields carried through so the resume
loads with 0 missing / 0 unexpected keys (verified directly before launching).
fp16 instead of bf16 (V100, sm_70); effective batch 48 via 4-way data-parallel
instead of one A6000, matching ft4's effective batch. Final training loss 0.178,
comparable to ft4's reported 0.066 - 500-episode full-rate closed-loop eval,
paired against the same episodes as ft4's own full-rate row (section 3).
pretrain_C got the identical treatment. One caveat specific to C: its
morph_encoder conditioning has no forward-pass implementation in this
codebase, so a zero-shot evaluation of C would silently measure a different
model than the one the checkpoint describes, and none is reported here. Fine-tuning
is still legitimate — the backbone unfreezes either way, so embodiment_emb
receives real gradient — and that is what the C row below measures.
| Success | 95% CI | delta vs. ft4 | paired 95% CI | flips | McNemar p | |
|---|---|---|---|---|---|---|
| ft4 (from stage3_distill) | 426/500 = 85.2% | — | — | — | — | — |
| B fine-tuned (from pretrain_B) | 345/500 = 69.0% | [64.8,72.9] | -16.2 | [-21.0,-11.4] | +40/-121 | 0.0000*** |
| C fine-tuned (from pretrain_C) | 372/500 = 74.4% | [70.4,78.0] | -10.8 | [-15.2,-6.6] | +36/-90 | 0.0000*** |
Neither cross-embodiment pretrain makes a better starting point than stage3_distill for LIBERO specifically - both make a worse one, by wide and significant margins. Head-to-head on the same 500 paired episodes, C beats B by +5.4 pp (paired 95% CI [+0.0,+10.8], McNemar p=0.0623, +111/-84 flips) - which does not clear significance at 0.05, so the two cross-embodiment pretrains are not separated by this evidence. This isn't one catastrophic task dragging an average down: 7/10 tasks score lower for the B-derived fine-tune than for ft4 (the other 3 are within a couple of episodes either way). Worst-hit: task8 at 9/50 vs. ft4's 39/50. Best-preserved: task5 at 49/50 vs. ft4's 42/50. Both checkpoints have identical architecture and parameter/FLOP counts (confirmed via the same profiling harness as section 6) - this is purely a pretraining-lineage effect, not a compute or capacity difference.
Plausible reason: stage3_distill was trained specifically as this model's
LIBERO-adjacent lineage (stage 1-3, spatial distillation against LIBERO-style
labels per the model card), whereas pretrain_B is a from-scratch TinyVLA-2 run
on a disjoint mixture (SO-100/101 + Bridge + RT-1, no LIBERO exposure anywhere in
its lineage, canonical-EE-space objective) built for a different question entirely
(cross-embodiment transfer to jaco/stretch, not LIBERO manipulation quality).
40k fine-tuning steps evidently were not enough to close that gap. This is a data
point against "any reasonable pretrain fine-tunes to a similar place on LIBERO",
not evidence about canonical vs. native action spaces (both ft4's lineage and
pretrain_B ultimately regress against LIBERO's own action convention once
fine-tuning starts, per section 1's frame-rate check applying identically to
both - the difference is what the backbone learned before LIBERO ever entered
the picture).
11. Zero-shot transfer to RoboCasa (a genuinely different domain)
Everything above is LIBERO: one fixed tabletop Panda, one scene layout family.
This section asks whether ft4 transfers with no fine-tuning at all to
RoboCasa365 — PandaOmron (Franka arm on a holonomic
mobile base) in procedurally-varied kitchens, a different robot, different
scenes, and a different observation layout.
Conventions were verified empirically, not assumed (verify_robocasa.py,
verify_robocasa_axes.py), because a silent frame mismatch here would be
indistinguishable from a policy failure:
- Gripper: RoboCasa's -1=open/+1=closed matches LIBERO's polarity directly (measured qpos 0.039 at -1, 0.001 at +1). No sign flip.
- Action: both use the same OSC_POSE delta controller at the same physical
scaling (±0.05 m, ±0.5 rad/step), so LIBERO's unnormalized
[dpos(3), drot(3), grip(1)]maps onto RoboCasa'see_pos/ee_rot/gripperslots.base_motion(4)andcontrol_mode(1)held at zero — arm-only. - State: RoboCasa's native 16-dim
agent_posis a different layout (base pose + relative EE) from the trained convention (absolute EE pose + gripper), converted viaee_state_from_agent_posbefore normalizing. Feeding the raw vector through would put a base quaternion where the model expects an EE rotvec. - Cameras:
robot0_agentview_left+robot0_eye_in_hand, used raw. A crosshair projection through the camera's own MuJoCo extrinsics shows +y and +z already match LIBERO's screen convention (+y→left, +z→up), while +x is a diagonal blend of both screen axes rather than LIBERO's clean "+x→down" — this camera looks into the kitchen at an angle instead of down a tabletop. No correction is applied: a reflection that fixed x would break the y/z agreement. This is a documented residual mismatch, not a fixed one.
Each task runs at RoboCasa's own registered horizon (get_task_horizon), not a
truncated budget: 900 steps for CloseFridge, 750 for OpenDrawer, 450-600 for the
rest. Semantic refresh 1 Hz, n_action_steps=1, 15 episodes/task.
| Task | Horizon | Object registry | Success | 95% Wilson | Note |
|---|---|---|---|---|---|
CloseFridge |
900 | lightwheel | 0/15 | 0.0-20.4 | — |
OpenDrawer |
750 | lightwheel | 0/15 | 0.0-20.4 | — |
NavigateKitchen |
450 | lightwheel | 0/15 | 0.0-20.4 | requires mobile-base motion; harness holds base_motion at zero |
TurnOnMicrowave |
450 | lightwheel+objaverse+aigen | 0/15 | 0.0-20.4 | — |
PickPlaceCounterToStove |
600 | lightwheel+objaverse+aigen | 0/15 | 0.0-20.4 | — |
Zero-shot transfer is a complete failure: 0/60 across the 4 arm-only tasks (95% Wilson 0.0-6.0%). Not a partial-credit result to be tuned up — nothing succeeds anywhere.
Two failure modes are visible in the saved videos (results/robocasa_ft4/), and
they are behavioural, not random flailing: the arm drifts steadily upward until
it clips the ceiling, and the wrist camera rotates until it faces the floor. Both
are consistent with the one convention that could not be pinned down
empirically — the correspondence between LIBERO's rotation axes and RoboCasa's
ee_rot. A systematic rotation-frame error produces exactly this: a constant
spurious delta integrated over hundreds of steps.
Section 12 tests that hypothesis and does not confirm it. Fine-tuning on RoboCasa's own data fixes the trained task completely but moves none of the held-out ones — which a repaired shared rotation frame should have done. Treat the paragraph above as the motivating hypothesis, not a finding.
Two honest caveats on the table above:
NavigateKitchenis not a measurement of the policy. The task requires driving the mobile base across the kitchen, and this harness pinsbase_motionto zero. Its 0/15 is guaranteed by construction, so it is excluded from the aggregate.- The object registry differs by row.
TurnOnMicrowaveandPickPlaceCounterToStovesample object categories that do not exist in thelightwheelregistry and cannot be constructed without objaverse/aigen assets, so a single registry across all five tasks was not possible without re-running the first two. The fixtures each task acts on (fridge, drawer, microwave, stove) are scene fixtures, not sampled objects; the registry affects distractor content.
This is the motivation for section 12: rather than hand-tuning the one unverifiable frame convention, fine-tune on RoboCasa's own recorded data and let the model learn whatever axis mapping the dataset actually contains.
12. Fine-tuning on RoboCasa's own data
Section 11 leaves one convention unverifiable by construction — the mapping
between LIBERO's rotation axes and RoboCasa's ee_rot — and blames the zero-shot
collapse on it. Rather than hand-tune that mapping, this section removes the
question: fine-tune on RoboCasa's own recorded actions, so the model learns
whatever axis convention the data actually contains.
Data: pepijn223/robocasa_CloseFridge,
513 episodes / 155,443 frames at a correctly-declared 20 fps (no repeat of
section 1's frame-rate bug). RoboCasa's native 16-dim agent_pos and 12-dim flat
action are converted to this repo's canonical 8-dim state / 7-dim action schema by
robocasa_train_source.py, and normalized with statistics computed over the whole
dataset (compute_robocasa_stats.py), not LIBERO's. Feeding the raw vectors
through CanonicalSource would have silently truncated the 12-dim action to its
first 8 entries — base_motion + control_mode + a partial ee_pos — which is
why a dedicated source class exists rather than a config flag.
Recipe: ft4 as the starting point, 8,000 steps at effective batch 24
(2xV100, fp16), lr 1e-4, backbone unfrozen at 0.1x — the ft4 recipe on a shorter
schedule (~1.24 epochs) because this is one task with 155k frames rather than
LIBERO's 500-task mixture. Final training loss ~0.35.
Evaluation is the same harness, same seeds, same object registry as section 11's
zero-shot row, with one necessary difference: the fine-tuned checkpoint is
unnormalized with its own RoboCasa statistics via --stats-json. Scoring it with
LIBERO's statistics would push actions through the wrong mean/std and report a
failure that belongs to the harness, not the policy.
| Success | 95% Wilson | delta | paired 95% CI | flips | McNemar p | |
|---|---|---|---|---|---|---|
ft4 zero-shot |
0/15 = 0.0% | — | — | — | — | — |
| RoboCasa fine-tune | 7/15 = 46.7% | [24.8,69.9] | +46.7 | [+20.0,+73.3] | +7/-0 | 0.0156 |
15 paired episodes (same seeds, same object registry). Pairing verified directly: the sampled instruction string matches on 15/15 episode indices, so the two runs saw the same scenes.
Fine-tuning works, and the effect is unambiguous: every flip goes one way
(+7/-0), so this is not variance around a shared mean. The two
zero-shot failure modes from section 11 — upward drift into the ceiling, wrist
camera rotating to face the floor — are absent from the fine-tuned videos
(results/robocasa_ftrc/).
The residual failures are structured, not random
| Scene variant | Instruction | Success | 95% Wilson |
|---|---|---|---|
| single-door fridge | Close the fridge door. |
7/10 | [39.7,89.2] |
| double-door fridge | Close the fridge doors. |
0/5 | [0.0,43.4] |
Fisher exact p=0.0256. This is not a training-data gap: both instruction variants are well represented in the fine-tuning set (84,341 frames for the single-door phrasing, 71,102 for the double-door one — 54% / 46%). The model saw plenty of double-door episodes and still closes none of them. Closing two doors requires sequencing two separate contacts; closing one does not. With n=5 double-door episodes this is a suggestive split, not a settled result — but it is the correct place to look next, and it is invisible in the 7/15 aggregate.
It did not fix a global convention error
The obvious reading of section 11 is that one bad rotation mapping broke every task, in which case fine-tuning on any single task should repair the shared representation and lift the others too. It does not. Same fine-tuned checkpoint, same harness, tasks never seen during fine-tuning:
| Held-out task | Success | 95% Wilson |
|---|---|---|
OpenDrawer |
0/15 | [0.0,20.4] |
TurnOnMicrowave |
0/15 | [0.0,20.4] |
PickPlaceCounterToStove |
0/15 | [0.0,20.4] |
no measurable transfer: 0/45 (95% Wilson 0.0-7.9%).
So the honest conclusion is narrower than "fine-tuning fixes RoboCasa transfer." Fine-tuning demonstrably solves the task it was trained on, moving it from categorically broken to roughly half-successful. It provides no evidence that the zero-shot collapse was purely a frame-convention artifact — had it been, the held-out tasks would have moved. What the model acquired looks task-specific rather than a repaired shared geometry. Distinguishing "memorized one task" from "the other tasks need capabilities beyond the convention" needs a multi-task fine-tune, which is the next experiment, not a claim this data supports.
Reproduction
# the published configuration, with its rates measured
python eval_libero_rates.py --n-action-steps 10 --semantic-hz 2 \
--n-init-states 10 --out-json accept.json
# full-rate and dual-rate, full 500-episode paired sweep
./run_sharded.sh full_rate 50 --n-action-steps 1 --semantic-hz 20
./run_sharded.sh dual_1hz 50 --n-action-steps 1 --semantic-hz 1
# paired statistics
python stats.py results/rates/full_rate.json results/rates/dual_1hz.json
RoboCasa (sections 11-12) runs in a separate venv — robosuite 1.5.2, versus the 1.4.0 that LIBERO's eval depends on and that must not be disturbed:
source .venv_robocasa/bin/activate
# zero-shot: LIBERO-trained checkpoint, LIBERO normalization (the default)
python eval_robocasa.py --checkpoint tinyvla --tasks CloseFridge OpenDrawer --episodes 15 --n-action-steps 1 --refresh-s 1.0 --out-json results/rates/robocasa_ft4.json --tag robocasa_ft4
# fine-tune on RoboCasa's own data (main venv, 2-GPU DDP)
accelerate launch --multi_gpu --num_processes 2 --mixed_precision fp16 tinyvla/scripts/train.py --config tinyvla/configs/robocasa_ft_closefridge.yaml
# evaluate it with ITS OWN normalization — LIBERO's stats would measure noise
python eval_robocasa.py --checkpoint outputs/robocasa_ft_closefridge/final --stats-json robocasa_stats.json --tasks CloseFridge --episodes 15 --n-action-steps 1 --refresh-s 1.0 --out-json results/rates/robocasa_ftrc.json --tag robocasa_ftrc
Tasks whose scenes sample objaverse/aigen object categories (TurnOnMicrowave,
PickPlaceCounterToStove) need those registries downloaded
(python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse objs_aigen)
and passed via --obj-registries lightwheel objaverse aigen; without them the
environment cannot be constructed at all.