tinyvla2/RESULTS.md

TinyVLA-2 A/B/C — final results (held-out jaco, integrated-trajectory endpoint error)

Test split jaco[956:976] (n=20), GT trajectories ~210mm, zero-prediction floor ~128mm.

jaco FT episodes	A (native+ID)	B (canonical+ID)	C (canonical+morph)
0 (zero-shot)	1377 mm	155 mm	178 mm
100	6919 mm	215 mm	218 mm
500	5667 mm	198 mm	197 mm

Verdict

1. Canonical action space ≫ native (10-45×) at every budget — decisive. FK-unified base-frame EE actions are what enable cross-embodiment transfer. Native is not just worse, it DIVERGES under few-shot FT (fresh embodiment-ID + native action dist).
2. Morphology tokens ≈ learned ID (C ≈ B) — no measurable advantage once actions are canonical. The morphology hypothesis (C>B) is NOT supported.
3. Short few-shot FT (5k steps) barely helps — pretrained canonical already transfers zero-shot (~155mm vs 128mm floor); the fresh-ID re-learn even hurts B at N=100. Protocol caveat: warm-start the new ID, more steps, tune LR.

Honest takeaway: the win is action-space unification, not morphology conditioning.

TidyBot (wheeled mobile manipulator) — B vs C morphology test (endpoint mm)

Held-out mobile morphology (is_mobile=1, holonomic), test eps [95:110], n=15.

N eps	B (canon+ID)	C (canon+morph)
0 (zero-shot)	352	419
30	332	383
80	335	329

C starts WORSE (OOD descriptor extrapolation) but adapts far steeper (419→329, −90mm) vs B's plateau (352→335, −17mm), OVERTAKING B by N=80 on both endpoint (329<335) and rotation (24°<28°). First support for morphology conditioning: it helps on a genuinely NEW morphology (mobile), unlike held-out arms (jaco: C≈B, both flat). Margin small (~2%, n=15) — crossover needs confirmation (more N, seeds).

Morphology-diversity continue-train (user idea 3) — held-out jaco zero-shot

model	training morphologies	morph→fast	morph→slow	jaco endpoint
C	3 (arms only)	✓	✗	178 mm
C-diverse	8 (arms + mobile)	✓	✗	150 mm (−16%)
C-diverse-slow	8	✓	✓	(training)

Morphology diversity in continue-training improves held-out-arm transfer 178→150mm: the descriptor→conditioning MLP only generalizes once it has seen diverse morphologies (arm-only C's encoder was useless/harmful on new morphologies). Caveat: TidyBot (331mm) is now IN C-diverse training, so it's not a clean OOD-mobile test; a strict OOD-mobile claim needs a held-out mobile robot (leave-one-out or a new dataset).

Slow-path morphology injection (idea: feed morphology to Qwen prefix too)

model	morph→fast	morph→slow	jaco endpoint
C-diverse	✓	✗	150 mm
C-diverse-slow	✓	✓	162 mm
No benefit (slightly worse, within n=20 noise). Fresh morph_to_lm projector (17K
params) likely underfit in 25k steps; injecting into the frozen LM prefix perturbs
the semantic latent without clear gain on a held-out arm. Idea sound, didn't pay off here.

TinyVLA-2 overall verdict

1. Canonical action space >> native for cross-embodiment transfer (10-45x). DECISIVE.
2. Morphology conditioning ~= learned ID on held-out ARMS — UNLESS the descriptor encoder is trained on DIVERSE morphologies, which improves held-out transfer (178->150mm). On a mobile morphology (TidyBot) C adapts steeper than B (overtakes by N=80).
3. Slow-path morphology injection: no clear benefit here. Takeaway: unify actions (big win); morphology conditioning helps only with morphological diversity in training.

ZERO-SHOT new-robot via descriptor (no fine-tuning) — the core promise

Descriptor sensitivity: swapping the descriptor (same observations) changes predictions by ~0.5 RMS (normalized) → the model genuinely READS the descriptor zero-shot. Held-out jaco, C-diverse, NO fine-tuning, endpoint error by descriptor given:

descriptor fed	endpoint
jaco (correct)	159.6 mm
so101 (wrong arm)	166.0 mm
zeros (none)	167.6 mm
uav (alien)	178.2 mm
Correct new-robot descriptor → best zero-shot control (no FT). Describing a robot
correctly beats a wrong/absent/alien descriptor. Modest (~5-11%) but consistent.

LeKiwi (wheeled mobile manipulator, SO-100 arm + kiwi base) — ZERO-SHOT, no fine-tuning

Arm-EE endpoint error on held-out LeKiwi split (C-diverse, descriptor swapped, NO FT):

descriptor fed	endpoint
zero-floor	241 mm
lekiwi (correct, is_mobile=1)	295 mm
uav (alien)	363 mm
so101 (same arm but is_mobile=-1)	533 mm
Correct descriptor beats the SAME-ARM-but-non-mobile descriptor by 45% (295 vs 533) —
the model genuinely uses the mobility bit zero-shot (mobile base couples arm-in-frame
motion). BUT correct-descriptor (295) still > zero-floor (241): zero-shot control of a
genuinely new mobile platform isn't achieved — needs base dims in the action head
(our 8-dim output has no vx/vy/omega) + some adaptation. Descriptor makes it "less
wrong", not yet working. LeKiwi has no sim env, so this is offline; closed-loop would
need building a MuJoCo LeKiwi scene.

Text robot-prompt (Qwen slow-path) x numeric descriptor ablation — held-out LeKiwi

C-mega (9 max-diverse robots, resume from C-diverse, 20k steps), zero-shot on LeKiwi:

condition	endpoint
zero-floor	241 mm
numeric=lekiwi + text=NONE	298.5 mm (best; ~= prior C-diverse 295mm)
numeric=lekiwi + text=lekiwi	314.6 mm (text HURTS slightly, +16mm)
numeric=none + text=none	413.4 mm
numeric=none + text=lekiwi	522.8 mm (text alone WORSE than nothing)
numeric=so101(wrong) + text=lekiwi	565.2 mm (conflicting signals, worst)

Text prompt does NOT help, and actively hurts when used alone or combined. Numeric descriptor remains the only working conditioning channel. Likely cause: the text channel is genuinely new (this run), LM is frozen, and 20k continue-train steps weren't enough to learn to use it — it just perturbs the readout-token aggregation as noise. The numeric MLP path has accumulated training since the original C pretrain and works. Conclusion: keep numeric morphology descriptors; the text-prompt idea needs either LM unfreezing, far more steps, or a different injection point to pay off (not validated here).

9-robot diversity (adding jaco+lekiwi-adjacent training) did not improve over the 8-robot C-diverse baseline on LeKiwi (298 vs 295mm, within noise) — diminishing returns past ~8 morphologies at this training budget.

Morphology via shared Qwen (text, unfrozen LM) — held-out LeKiwi

Replaced the MLP-over-numbers morphology encoder with the SAME Qwen doing a text-only pass over a robot description ("Robot: LeKiwi, a 5-DOF SO-100 arm on a three-wheel holonomic mobile base."), AND unfroze the LM (249M params, 0.1x lr), resumed from C-diverse (8 robots), 20k steps.

condition	endpoint
zero-floor	241 mm
desc=none (empty)	294.1 mm (best)
desc=lekiwi (correct)	314.1 mm
desc=so101 (wrong, non-mobile)	464.8 mm

Absolute level unchanged vs prior best (294-314 vs 295-298mm) — neither LM-unfreeze nor Qwen-encoding gave a breakthrough at this budget. BUT the correct-vs-wrong gap widened to 32% (314 vs 465mm) vs the numeric-MLP's ~10% (150 vs 166mm on jaco) — the Qwen text encoder is more sensitive to descriptor CONTENT, a sign real language understanding is engaged (unlike the frozen-LM C-mega experiment where wrong beat none, i.e. noise). Oddity: desc=none beats the correct description — LeKiwi's specific phrasing/combination of words may be out-of-distribution for the 8 training descriptions the (still fairly short, 20k-step) fine-tune has seen. Net: promising SENSITIVITY signal, not yet a WIN on absolute error. Would need either more diverse text descriptions in training, more steps, or investigating the desc=none anomaly to turn this into a clear improvement.

In-context few-shot demo conditioning (K=3 real exemplars, NO gradient updates) — held-out LeKiwi

DemoConditioner: K=3 (obs,action) pairs -> spatial-encoder mean-pool + action summary -> MLP -> tokens, composed with morph_qwen text channel. Trained 20k steps (resumed from C-qwen-morph).

condition	endpoint
REAL LeKiwi demos + desc=lekiwi	324.7 mm
REAL LeKiwi demos + desc=none	290.6 mm
GARBAGE/noise demos + desc=lekiwi	322.5 mm
GARBAGE/noise demos + desc=none	288.5 mm

Real demos ≈ garbage demos (within noise) — the demo conditioner learned to IGNORE support-example content entirely. Gradient reaches demo_encoder during training (verified), but the model found it easier to route around this channel than to use it, likely because during training support pairs were sampled from the SAME source as the query — redundant with the already-available text/numeric identity signal, so no incentive to learn content-dependence. Net: dead channel, clear negative result. Would need support examples that carry information NOT available elsewhere (e.g. from a genuinely different episode/task, or examples that vary the dynamics being asked about) to give the model a reason to attend to them.

Demo conditioning v2 — exemplars from a DIFFERENT TASK of the same robot

Fix attempt for v1's dead channel: sample support pairs cross-task (so they carry robot-dynamics info the identity channels don't duplicate). Held-out LeKiwi, no FT:

condition	endpoint
zero-floor	241 mm
GARBAGE demos + desc=none	260.9 mm (best)
REAL LeKiwi demos + desc=none	263.6 mm
OTHER-ROBOT (jaco) demos + desc=none	265.9 mm
REAL demos + desc=lekiwi	294.0 mm
GARBAGE demos + desc=lekiwi	294.2 mm

TWO findings:

1. Absolute level improved a lot: 261-266mm vs 288-295mm for every prior variant — closest to the 241mm floor in the whole series. But this came from TRAINING WITH the extra (noisy) channel acting as regularization/augmentation, NOT from using demo content.
2. The channel is still content-blind: real (263.6) ≈ garbage (260.9) ≈ other-robot (265.9), all within noise. Cross-task sampling did NOT break the shortcut — the model still ignores WHAT is in the exemplars. Conclusion: in-context demo conditioning does not work in this architecture even with informative exemplars. Its only benefit is incidental regularization. Two plausible remaining causes: (a) the mean-pool + 3-step action summary destroys the information, (b) a single MLP token per exemplar is too weak a pathway — proper in-context learning likely needs the exemplars inside the attention stream (e.g. as extra tokens the expert cross-attends over), not squeezed through an MLP.

Demo conditioning v3 — high-capacity encoder (24x params) — held-out LeKiwi

Widened every stage: spatial group-pools (not global mean), 8 action keyframes (not 3), 3-layer 4x-wide MLP + LayerNorm, 4 tokens/exemplar; 0.7M -> 16.9M params.

condition	v3	v2 (for reference)
REAL LeKiwi demos + desc=none	294.8 mm	263.6 mm
GARBAGE demos + desc=none	291.3 mm	260.9 mm
OTHER-ROBOT demos + desc=none	293.2 mm	265.9 mm

Capacity was NOT the problem. Third independent attempt (v1 same-task, v2 cross-task, v3 high-capacity) and the channel is still content-blind: real ≈ garbage ≈ other-robot within noise. v3 is also worse in absolute terms than v2 (294 vs 261mm) because it had to resume from C-qwen-morph (v2 weights were shape-incompatible with the new encoder), losing v2's incidental regularization.

Conclusion after three attempts: the failure is ARCHITECTURAL PLACEMENT, not capacity or sampling. Demo tokens sit in the expert's conditioning set alongside ~150 other tokens; the flow head predicts actions from the CURRENT observation and has no mechanism to COMPARE the current scene against exemplars. Real in-context learning needs exemplars in the attention stream where query tokens can attend to them selectively (RoboCat-style separate demo cross-attention, or explicit retrieval), not "placed alongside" where the network marginalizes them as constant noise. BEST CHECKPOINT OF THE SERIES REMAINS v2 (261mm on LeKiwi zero-shot).

DEMO-ONLY — all identity channels removed (5th and final in-context attempt)

Removed embodiment-ID, morphology descriptor AND robot text; the K=4 demo exemplars in the LM attention stream were the ONLY cue for which robot this is (per-source action normalization creates the pressure: correct prediction requires knowing the robot's action scale, available only from demos). Held-out LeKiwi:

condition	step 5k	step 10k
REAL LeKiwi demos	376.1 mm	323.6 mm
GARBAGE/noise demos	393.4 mm	322.1 mm
OTHER-ROBOT (jaco) demos	330.3 mm	294.8 mm

At 5k a spread finally appeared (63mm — the first time any demo variant showed content-sensitivity) but by 10k it COLLAPSED back: real (323.6) ≈ garbage (322.1). Other-robot demos "win" only because they regress toward the training distribution, not because demos are read. Training stopped at 10k (no point burning 7 more hours).

In-context demo conditioning: FINAL VERDICT after 5 architectures

1. MLP encoder, same-task exemplars — dead (real ≈ garbage)
2. MLP encoder, cross-task exemplars — dead (but +regularization: 295→261mm)
3. High-capacity encoder (24x params, spatial groups) — dead
4. Demos inside the LM attention stream (rich-slow, dual-rate preserved) — dead
5. Demos as the ONLY identity channel — brief flicker at 5k, dead by 10k Neither sampling, capacity, architectural placement, nor removing all alternatives makes this model read demonstration content. Plausible remaining explanation: the flow head predicts from the CURRENT observation, and nothing in the training objective ever requires comparing the current scene to an exemplar — the robot is visually identifiable in-frame (an identity leak we cannot remove), so the model always has a cheaper route than reading demos. Working mechanism remains the numeric morphology descriptor (C-diverse: 150mm on held-out jaco). Best LeKiwi zero-shot: C-incontext-v2 at 261mm.

Reality check: absolute performance across robots (C-diverse, best morphology ckpt)

Endpoint error vs the per-robot zero-prediction floor (ratio<1 = better than predicting mean motion). Absolute mm are NOT comparable across robots (different motion scales) — the ratio is.

robot	status	endpoint	floor	ratio
stretch (mobile)	in training	110 mm	139 mm	0.79
xarm	in training	171 mm	179 mm	0.95
dlr_edan (wheelchair)	in training	326 mm	340 mm	0.96
ur5	in training	229 mm	183 mm	1.25
jaco (arm)	HELD-OUT	161 mm	146 mm	1.10
LeKiwi (mobile)	HELD-OUT	294 mm	239 mm	1.23
mean, in-training				0.99
mean, held-out				1.17

IMPORTANT CAVEAT FOR THE WHOLE TinyVLA-2 SERIES: this model barely beats the trivial "predict mean motion" baseline even on robots it trained on (mean ratio 0.99). Every morphology comparison in this document (178 vs 150mm, canonical vs native, demo channels) is therefore a comparison between variants of a WEAK model. The RELATIVE conclusions stand — they were measured under identical conditions — but nobody should read these numbers as "this is a working cross-embodiment controller". For reference, TinyVLA v1 reached 86% LIBERO success with 40k steps on ONE domain with an unfrozen LM; here we spread 40k+20k steps across 8 morphologies with a mostly-frozen backbone. Getting absolute competence would need far more compute per embodiment, not more conditioning tricks. Interesting detail: held-out jaco (1.10) beats in-training ur5 (1.25) — robot SIMILARITY matters more than mere presence in the training mix, which is mildly supportive of the morphology-conditioning idea.

SCALED C — the fix that actually mattered (data + budget, not conditioning tricks)

Full Bridge (53K eps) + full RT-1 (87K eps) → pool 4.4M → 9.1M frames; 60k steps (was 20k); LM unfrozen at 0.1x lr. Same C setup (canonical actions + numeric morphology descriptor). Ratio = endpoint error / per-robot zero-prediction floor (<1.0 beats "predict mean motion").

robot	C-diverse	20k	40k	60k final
jaco	1.10	0.80	0.69	0.69
xarm	0.95	0.66	0.62	0.64
dlr_edan	0.96	0.77	0.71	0.69
stretch	0.79	0.71	0.66	0.69
ur5	1.25	1.27	1.24	1.22
LeKiwi (HELD-OUT mobile)	1.23	1.24	1.13	1.07
mean in-training	0.99	0.85	0.81	0.81
mean HELD-OUT	1.17	1.02	0.91	0.88

This is the main positive result of TinyVLA-2. The model went from barely matching the trivial baseline (0.99) to clearly beating it (0.81 / 0.88). Five conditioning tricks (text prompts, Qwen-encoded morphology, demo conditioning x3 architectures) moved nothing; scaling data + steps + unfreezing the LM moved everything. In-training saturates by 40k (0.81) — further gains would need unfreezing the vision tower or a bigger expert, not more steps. Held-out is still improving at 60k (0.91→0.88), and LeKiwi — the only true unseen morphology — has not saturated (1.24→1.13→1.07), suggesting cross-morphology transfer is the slowest-converging capability and would keep improving with more budget. ur5 stays hardest (1.22): its canonicalization is fine, it is simply the fastest robot in the mix (9.9mm/step vs 5.4 for jaco, 15x the per-step rotation).

---

CORRECTION (2026-08-28): the offline eval fed the model the wrong proprioception

Every offline number above this line was produced by an eval harness that passed observation.state raw, truncated to 6 dims. Training normalizes it with the dataset's own mean/std and pads to max_state_dim (mixture.CanonicalSource. __getitem__ → _norm → _pad). For joint-angle datasets the raw values are O(100) where the model expects O(1), so the policy was evaluated with a proprioceptive input tens of sigma outside its training distribution, and mobile robots additionally lost their base channels to the [:6] truncation.

The fix is src/tinyvla/data/eval_utils.py::StateAdapter, now used by every eval script. Controlled re-measurement (same checkpoint, same episodes, same sampled frames — only the state input differs) is in scripts/ and reproduces below.

Corrected: tv2_C_scaled (60k) across robots

robot	status	as-published	corrected	Δ
stretch (mobile)	in training	0.69	0.11	−0.58
xarm	in training	0.64	0.13	−0.51
dlr_edan (wheelchair)	in training	0.69	0.23	−0.46
ur5	in training	1.22	0.90	−0.32
jaco (arm)	HELD-OUT	0.69	0.28	−0.41
LeKiwi (mobile)	HELD-OUT	1.07	1.67	+0.60
mean, in-training		0.81	0.34	−0.47
mean, HELD-OUT		0.88	0.97	+0.09

What this changes about the project's conclusions

The "weak model" caveat was wrong. The model is far more competent than reported: 0.34 on robots it trained on, not 0.81. It was never barely beating the "predict mean motion" baseline — it beats it by ~3x. The paragraph above claiming "nobody should read these numbers as a working cross-embodiment controller" was an artefact of the harness, not a property of the model.

The held-out claim splits in two, and only half survives. Transfer to an unseen arm is genuinely strong: jaco 0.28, better than in-training ur5 (0.90). Transfer to an unseen mobile base got worse under correct measurement: LeKiwi 1.67, i.e. worse than predicting mean motion. The honest statement is now: morphology conditioning transfers across arms of different geometry, and fails on a base type never seen in training.

The mechanism is consistent: with garbage proprioception the model had to ignore the state channel entirely, which flattered LeKiwi (whose proprio it cannot interpret anyway) and penalised every robot whose state was informative. Correct proprio helps wherever the model learned to read it and actively misleads it on the one morphology class it never saw.

Relative conclusions between conditioning variants are NOT automatically safe. They were all measured under the same bug, so the comparison was internally consistent, but the bug suppressed the state channel — precisely one of the inputs a conditioning mechanism could have exploited. The in-context/demo experiments are being re-measured; see below.

Re-measured after the fix: does any conditioning conclusion change?

experiment	condition	as-published	corrected
C-qwen-morph (LeKiwi)	desc=lekiwi (correct)	294 mm	503.9 mm
	desc=so101 (wrong robot)	302 mm	519.4 mm
	desc=none	294 mm	756.4 mm
C-incontext-v3 (LeKiwi)	REAL demos K=3	439 mm	439.0 mm
	GARBAGE demos	431 mm	429.4 mm
	OTHER-ROBOT (jaco) demos	—	437.0 mm
demo-only (LeKiwi, 10k)	REAL demos K=4	261 mm	490.5 mm
	GARBAGE/noise demos	259 mm	518.7 mm
	OTHER-ROBOT (jaco) demos	—	489.5 mm

The in-context negative result survives the fix, and is now better supported. Real exemplars remain indistinguishable from garbage (439.0 vs 429.4; garbage is marginally better), and — the sharper test — real LeKiwi demos are indistinguishable from demos of a completely different robot (490.5 vs 489.5). The model is reading demonstration format, never demonstration content.

One thing the fix does buy: the descriptor channel now clearly matters. With correct proprioception, dropping the morphology descriptor costs 503.9 → 756.4 mm, where under the bug it cost nothing (294 → 294). So the numeric descriptor is a working channel that the broken harness had rendered invisible.

tv2_C_incontext and tv2_C_incontext_v2 could NOT be re-measured: widening the demo encoder ("increase MLP capacity") changed its weight shapes (512x1048 → 2048x4160), so those checkpoints no longer load against current code. Their published numbers stand as measured under the bug and should not be compared with anything in this section.

LIBERO closed-loop benchmark of the canonical (v2) line

The only benchmark comparable to v1 (86% spatial) and to the literature. The v2 model predicts base-frame EE deltas, not LIBERO's OSC action units, so it needs a decoder — and the decoder needs its own correctness gate.

Pipeline

• Sidecar: LIBERO state is [pos(3), rotvec(3), grip(2)] — neither euler nor quaternion, so build_ee_sidecar.py gained --rot-format rotvec. 273,465 frames.
• Decoder fit from ground truth: per-dim scale [80.4, 76.9, 78.8] (position) and [7.57, 7.67, 8.60] (rotation); correlations with the GT action 0.93 / 0.96 / 0.96 / 0.95 / 0.84 / 0.98. Gripper is absolute openness, decoded by a threshold at 0.879 (92.5% sign agreement vs 86.8% for a linear map).

Round-trip gate (GT actions through encode → decode → env)

decode	success
raw GT actions (ceiling)	4/5
scaled: per-step delta x fitted gain	1/5
servo, kp=60	3/5
servo, kp=80	3/5
servo, kp=120	4/5
servo, kp=160	3/5

The naive decode loses three quarters of the ceiling. Reason: a canonical delta is achieved motion, while an OSC action is a command to a P-controller that never fully closes the gap in one step, so replaying achieved motion as a command systematically under-shoots. Integrating the deltas into an absolute desired pose and commanding the error against the measured pose removes the lag and recovers the full ceiling. Anyone deploying a canonical-action policy on a position-servo robot needs this; the scaled decode looks reasonable offline and fails on hardware.

Results (30k fine-tune complete)

Pipeline ceilings first, so the model number can be read against what the action space and decoder actually permit:

what is being replayed	success
raw GT actions (env ceiling)	9/10
canonical encode→decode, GT gripper	9/10
canonical encode→decode, decoded gripper	8/10

The canonical motion representation is lossless — routing GT through base-frame EE deltas and back through the servo decoder costs nothing (9/10 either way). The whole ~10-point loss comes from the gripper: canonical stores absolute openness, and turning that back into LIBERO's binary open/close command is only 92.5% accurate per step. That is the one place the canonical space is lossy, and it is fixable (predict the command channel directly rather than openness).

model	LIBERO-spatial (n=50)
zero-shot, no LIBERO training (offline ratio to zero-floor)	0.79
canonical FT 5k steps	40%
canonical FT 30k, full rate	70%
canonical FT 30k, dual-rate (latent @1 Hz)	68%
decoded-pipeline ceiling	80%
v1, native actions, 40k steps (reference)	86%

70% against an 80% decoder ceiling: the policy captures ~88% of what the pipeline allows. Against v1's 86% in the native action space, canonical actions cost 16 points, of which ~10 are the gripper decode and ~6 are the policy.

Dual-rate survives the change of action space (68% vs 70%, indistinguishable at n=50). The v1 claim — a semantic latent refreshed at 1 Hz is enough, the fast path carries the rest — holds for a cross-embodiment canonical policy too, which is the result that makes the v2 line deployable at v1's compute profile.

Zero-shot 0.79 means the cross-embodiment checkpoint beat the trivial baseline on a robot and domain it had never seen, before any LIBERO training at all.

RoboCasa continue-train, re-measured (step_10000, corrected harness)

RoboCasa365 (sim kitchens, PandaOmron mobile manipulator, 1.5M frames) at weight 0.50, old robots replayed at 0.50, continued from C-scaled. Both columns measured with the corrected StateAdapter; the run was stopped at ~18k and only step_10000 was kept.

robot	C-scaled (base)	+RoboCasa 10k	Δ
RoboCasa (NEW domain)	—	0.47	new
xarm	0.13	0.17	+0.04
dlr_edan	0.23	0.26	+0.03
stretch	0.11	0.25	+0.14
ur5	0.90	0.96	+0.06
jaco (held-out arm)	0.28	0.32	+0.04
LeKiwi (held-out base)	1.67	1.60	−0.07
mean, old in-training	0.34	0.41	+0.07

This reverses what we reported under the buggy harness. The old story was "RoboCasa integrates well, old robots improve 0.81→0.76, but held-out LeKiwi collapses 1.07→1.58, so weight 0.5 is too high". Correctly measured, LeKiwi does not collapse — it is unchanged to slightly better (1.67→1.60) — and the cost lands instead as mild, uniform forgetting on the robots that were already learned (mean 0.34→0.41, worst case stretch 0.11→0.25, which more than doubles).

So the trade-off is real but benign and ordinary: a genuinely new domain reaches 0.47 for ~20% relative degradation spread across the old robots, with no effect on cross-morphology transfer. Weight 0.5 is defensible; the earlier "too high, catastrophic" conclusion was an artefact.

Calibrate 0.47 honestly: it is mid-pack, not best-in-project. Corrected in-training ranking is xarm 0.17 < stretch 0.25 < dlr_edan 0.26 < RoboCasa 0.47 < ur5 0.96. RoboCasa only looked best-in-project under the buggy harness, where every other robot sat at 0.7-1.2. It is also the least-trained entry here — 10k continue-train steps on a 1.5M-frame domain, versus 60k for the rest — so 0.47 is a floor on what this domain can reach, not a converged value.

Joint training with Unitree humanoid/arm + navigation packs (tv2_joint, 40k)

3.25M new samples added to the 9.1M-frame canonical pool: Unitree G1/Z1 with native joint-position actions (7/14/16/26/28 channels) and five navigation packs with 3-dim waypoints/step-deltas. One flow head serves all three action spaces via the per-sample action_dim_mask. Weights: old pool 0.50, unitree 0.30, nav 0.20.

Resumed from a dimension-expanded checkpoint (state 16→32, action 8→32, embodiments 16→24, all new rows/columns zero), which is also the control column below: zero-initialised output rows mean "no capability on the new dims yet".

What the new data bought (ratio to each space's trivial baseline)

Nav actions are deltas → floor is predicting zero motion. Unitree actions are ABSOLUTE joint positions → integrating is meaningless, so the floor is "hold still", i.e. predict the current joint configuration for the whole chunk.

embodiment	space	before	after
G1 Dex3 ToastedBread	joint	2.76	0.65
G1 Dex1 DualArm	joint	2.47	0.78
Z1 StackBox	joint	1.09	0.53
scand (social nav)	nav	1.08	0.33
tartandrive (ATV)	nav	0.91	0.38
sacson (indoor nav)	nav	2.18	0.73
go-stanford (nav)	nav	1.17	0.76
recon (outdoor nav)	nav	1.24	1.04
mean, joint		2.11	0.65
mean, nav		1.32	0.65

Both new modalities are genuinely learned: a humanoid controlled in 28-channel joint space and five navigation robots all beat their own trivial baselines from a model that started at or below those baselines.

recon is the exception and the interesting one: it is the LARGEST nav pack (598k samples, the biggest nav weight) and it is the only embodiment that barely moved (1.24 → 1.04). Size did not help it. Outdoor exploration from a single camera plausibly has the weakest action-from-vision determinism in the pool — the same "hardest robot" slot ur5 occupies among the arms.

What it cost the canonical line

robot	C-scaled	+joint	Δ
xarm	0.13	0.12	−0.01
stretch	0.11	0.14	+0.03
dlr_edan	0.23	0.28	+0.05
ur5	0.90	1.02	+0.12
jaco (HELD-OUT arm)	0.28	0.35	+0.07
LeKiwi (HELD-OUT base)	1.67	2.10	+0.43
mean, in-training	0.34	0.39	+0.05
mean, HELD-OUT	0.97	1.22	+0.25

In-training degradation is mild (0.34 → 0.39) and matches what RoboCasa cost at a comparable new-domain weight. Held-out is where it hurts, and one result there is worth stating plainly:

Adding 1.45M frames across five mobile navigation robots made zero-shot transfer to a mobile manipulator WORSE (LeKiwi 1.67 → 2.10). The intuition that "more wheeled bases should help the wheeled held-out robot" is wrong here. LeKiwi is an SO-100 arm on a kiwi base, and what the nav packs teach is base motion with no arm at all (their descriptors are literally arm_dof 0, reach 0, gripper 0). The model appears to be learning "mobile ⇒ do not manipulate", which is the opposite of what LeKiwi needs. This is the third consecutive measurement in which LeKiwi degrades as the pool grows (1.07 buggy → 1.67 corrected → 2.10 joint), and the first where the added data was specifically of its own base class.

So the honest summary: joint training is a clear win for the new embodiments and a measurable, not catastrophic, tax on the old ones — except for cross-morphology transfer to a mobile manipulator, which it actively damages.

GR00T / physical_ai_ft (B200) — first evaluation of a run that was never measured

AlexWortega/tinyvla-b200-artifacts shipped a 60k-step, 8×B300 finetune on the nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim collection (163 LeRobot v2.x datasets, 214.7M source frames, streamed from the hub). It had no training log for the successful run (the only log in the artifacts is a SIGSEGV from a different launch), no loss curve, and no metric of any kind. This is its first evaluation: scripts/eval_physical_ai.py, streaming, nothing written to disk.

Regime differs from the whole v2 line — conditioning: id, action_space: native, max_state_dim: 256, max_action_dim: 64 — and state_proj / action_in / action_out were RE-INITIALISED at resume. It is a new action head on the C-scaled trunk, so no number here is comparable to our 0.34/0.28.

Three things found before any number was produced

1. The published artifact is a trap. 167 of 489 tensors carry torch.compile's expert._orig_mod. prefix. LeRobot loads with strict=False and only logs missing keys, and FlowMatchingExpert zero-inits action_out — so loading it the normal way yields an expert that returns the sampler's own initial Gaussian noise. Not a crash: a plausible-looking bad number. Measured: un-rekeyed error 7.74 vs rekeyed 3.33 on the same batch. Fixed locally (scripts/rekey_checkpoint.py, 489/489 strict=True); the hub artifact is left as-is and documented here.

2. The realized training mixture is 96.5% one family, not the intended 49%. HubEpisodeStream samples a dataset per episode and then streams the whole episode. r1_pro episodes are ~37× longer than everyone else's, so the sqrt(frames) weights were multiplied by episode length:

family	intended (weight)	realized (samples)	P(episode unseen)
r1_pro	49.1%	96.5%	0.0%
single_panda	19.1%	1.4%	94.0%
gr1	16.6%	1.0%	96.2%
g1	6.4%	0.3%	91.2%
bi_panda_hand	4.6%	0.4%	88.6%
bi_panda_grip	4.2%	0.3%	91.1%

This is a sampler bug, not a config choice: weighting by episode when episode lengths differ by 37× silently converts a diversity mixture into single-task training. It also hands us a real held-out set for free — five of six families were evaluated on episodes ~90% likely never streamed.

3. Action semantics are per-dim, not per-family. r1_pro's action[3:6] is state[6:9] exactly (offset by 3 in the state layout); gr1 mixes 13 absolute dims with 10 deltas and 21 constants; single_panda is pure delta with 5 dead dims. The baseline is therefore searched per dim in raw units over lags 0-2, not assumed. It matters enormously: on r1_pro's absolute dims, hold-still scores 1.08 while predict-the-mean scores 3.79 — using the latter would have manufactured a 3.5× "win" out of nothing.

Results (displacement error ÷ strongest trivial floor; <1 beats it)

4-seed averaged ODE. n = episodes, CI = bootstrap over episodes.

family	subset	dims	err	B0 mean	B1 hold-still	B2 oracle	ratio	95% CI
gr1	abs	13	1.68	3.40	3.11	2.96	0.54	[0.48, 0.65]
gr1	delta	10	19.7	57.6	—	48.9	0.34	[0.30, 0.40]
bi_panda_grip	delta	14	27.8	46.8	—	93.4	0.59	[0.57, 0.61]
bi_panda_hand	abs	4	0.69	1.73	1.64	1.52	0.42	[0.28, 0.45]
bi_panda_hand	delta	16	33.0	64.4	—	84.1	0.51	[0.44, 0.56]
single_panda	delta	6	25.7	37.1	—	52.4	0.69	[0.63, 0.76]
r1_pro (in-training)	abs	17	0.74	3.79	1.08	1.06	0.69	[0.62, 0.76]
r1_pro (in-training)	delta	4	13.1	45.2	—	26.4	0.29	[0.25, 0.35]
g1	abs	16	2.16	3.68	3.88	3.72	0.59	[0.57, 0.81]
g1	delta	22	69.2	93.9	—	77.1	0.74	[0.63, 0.85]

The checkpoint works. Every family beats its own trivial baseline, and for five of six that is on episodes ~90% likely never seen in training. Best transfer is on the delta subsets (0.29-0.59); worst is single_panda delta (0.69) and g1 delta (0.74).

Read the absolute-dim rows with suspicion. On every ABS subset the oracle "repeat the first action" lands within 10% of the trivial floor (r1_pro 1.06 vs 1.08; gr1 2.96 vs 3.11; g1 3.72 vs 3.68). There is almost no headroom between doing nothing and the best any dynamics-free predictor can do, so a ratio of 0.6-0.7 there is a much weaker claim than the same number on a delta subset. r1_pro's 0.69 — the family that got 96.5% of training — is the least meaningful number in the table.

Controls

• Rekey (B): as-published 7.74 vs rekeyed 3.33 — the rename is load-bearing.
• Flow loss (E): L/L₀ = 0.124 on gr1. The expert learned the velocity field, independent of the ODE and of the metric design. This is the closest thing to the missing training log.
• Vision vs proprioception (D): swapping in a donor observation moves the prediction by 0.184 for state and only 0.042 for images, against a prediction scale of 0.404 — and the donor states differed by just 0.047. The model is roughly 4× more sensitive to proprioception than to visual content. Swapping the embodiment id is the biggest perturbation of all (ratio 0.245 → 0.747), so under conditioning: id the identity token is doing much of the work.
• ODE sampling (C): single-draw seed sensitivity S = 1.22 — two draws disagree with each other as much as either disagrees with truth. The 10-step Euler integration does not contract the initial noise, so a single sample is noise-dominated. Averaging helps and saturates: gr1 abs 0.539 → 0.475 → 0.423 → 0.419 and gr1 delta 0.236 → 0.198 → 0.176 → 0.168 at 1 / 4 / 16 / 64 seeds. The headline table above uses 4 seeds and is therefore ~12% pessimistic; the converged values are ~0.42 and ~0.17 for gr1.

Caveats

• r1_pro starts are confined to the first 600 strided frames of an episode (prefix decode, to avoid pulling 190 MB and decoding 26k frames per episode), so its numbers cover the approach phase of the task, not the whole task.
• Nothing is strictly held out — no exclusion list exists — the hold-out claim is the probabilistic one in the table above, and it assumes the 8-GPU launch recorded in metadata/commands.txt. On a 1-GPU run every P(unseen) rises (r1_pro 0.0% → 0.8%).
• Normalized units throughout: per-dataset z-scores make absolute magnitudes incomparable across datasets, but the ratio is comparable. No mm are reported for the panda families — their native layouts are an unknown mix and no FK exists.

GR00T checkpoint on held-out LeKiwi — zero-shot to an unseen embodiment

LeKiwi (SO-100 arm on a 3-wheel kiwi base, 9 native dims: 5 joint targets + gripper

• x_mm/y_mm/theta) is not one of the six GR00T families and has no embodiment id. Under conditioning: id + action_space: native there is no reason its dim k means what any GR00T robot's dim k means, so the checkpoint was given every advantage: each trained id 10..15 tried and the best reported, plus id 0 as the "no matching identity" reference. Same 4-seed ODE averaging, same per-dim baseline search, held-out tail episodes.

What id 0 actually is matters, and it is not what it first looks like. tv2_C_scaled is variant C = conditioning: morph, so its embodiment_emb table never received a gradient in that run: rows 0-9 have per-row std 0.93-1.05, norm ~22.6 (= sqrt(512)) and pairwise cosine -0.03, i.e. they are exactly nn.Embedding's default N(0,1) initialisation. The GR00T run then moved only rows 10-15, and only by 0.006-0.036 mean-abs on entries of scale ~1. So id 0 is a pure random 512-d vector, and ids 10-15 are random vectors plus a small trained nudge.

The dim search recovers the layout correctly without being told: 5 arm joints and x_mm/y_mm map to their own state dims (ABS), gripper and theta come out DELTA.

emb id	abs (7 dims)	delta (2 dims)
10 gr1	1.22	1.07
11 bi_panda_grip	1.21	1.09
12 bi_panda_hand	1.33	1.56
13 single_panda	1.24	1.08
14 r1_pro	1.23	1.07
15 g1	1.18	0.88
0 (never trained)	1.22	0.92
oracle repeat-A₀	0.91	0.54

It does not transfer. On the absolute dims every id lands at 1.18-1.33 — worse than simply holding the current pose for 5 seconds. Only the delta pair (gripper, theta) creeps under 1.0, and even there a dynamics-free predictor that repeats the first ground-truth action scores 0.54 against the model's 0.88. The model loses to a trivial predictor on both subsets.

The control is the interesting part: a pure random identity vector (id 0: 1.22 / 0.92) does as well as the best trained one (id 15: 1.18 / 0.88).

That is NOT because the id channel is inert. On an in-distribution gr1 batch the channel is clearly functional — error vs ground truth is 0.223 with the correct id 10, 0.263 with another trained id, 0.299 and 0.331 with random ids 0 and 1. The model reads the code and the right code is measurably better.

The channel is a lookup table with no entry for LeKiwi and no way to synthesise one. Swapping between two random codes changes the prediction by 0.174 and between two trained codes by 0.194 — comparable magnitudes, so there is no special "trained subspace" a new robot could be projected into. For an unseen robot every available code is somebody else's, and a random one is no worse.

That looks like the argument for the morphology descriptor — a 16-number physical descriptor can be computed for an unseen robot where a learned id can only be looked up. So it was tested, and the argument does not survive the test.

Converting the GR00T checkpoint to morphology conditioning does NOT fix LeKiwi

The checkpoint has no morph_encoder at all (the run discarded C-scaled's, since its config is conditioning: id). Two steps were tried:

1. Graft only — set conditioning: morph, transplant tv2_C_scaled's trained morph encoder. Result 1.30 / 1.11, worse than id. Expected: the expert spent 60k steps reading a 1-token id and has never seen 2 morph tokens from another model's encoder. (All six ids give identical numbers here, which confirms the morph path is actually active and the id is ignored.)
2. Train it to read the descriptor — 6k streaming steps with the six GR00T family descriptors (scripts/train_groot_morph.py), everything else already trained.

conditioning	abs (7 dims)	delta (2 dims)
id, best of six (oracle)	1.18	0.88
morph, grafted only	1.30	1.11
morph, 1k steps	1.13	0.83
morph, 2k steps	1.16	1.00
morph, 4k steps	1.12	0.87
morph, 6k steps	1.17	0.93
oracle repeat-A₀	0.91	0.54

The improvement is 1.18 -> 1.12 at best, and it is not monotonic (1.13, 1.16, 1.12, 1.17 across checkpoints) — that spread is noise, not convergence. Both conditioning schemes stay above 1.0 on the absolute dims, i.e. worse than holding the current pose, and both lose to a dynamics-free predictor that repeats the first ground-truth action.

So the honest conclusion is narrower than the one above: being computable is necessary but not sufficient. A descriptor gives you a well-defined input for an unseen robot, which an id cannot; it does not by itself give you transfer. This model was trained on six morphologies — three panda variants, two humanoids and an R1-Pro — none of them a small 5-DOF arm on a wheeled base, and no conditioning scheme extracts a LeKiwi policy from that.

Confound, stated rather than buried: the morph run had to reweight the mixture, because the per-episode sampler starved the loader (every draw pulled a ~190 MB r1_pro episode and the first batch never arrived). The correction over-shot — realized shares became gr1 0.67 / panda_single 0.23 / g1 0.04 / bimanual 0.035 / r1_pro 0.02 — so the morph model saw a different mixture than the id model. This is not a matched pair, and a matched-pair version would need retraining the id model on the same reweighted stream.

That id 15 (g1: a humanoid arm commanded in joint space) is the best of the six is the one sensible signal in the table — it is the closest morphology to an SO-100 arm also commanded in joint space.

Not comparable to our LeKiwi numbers (C-scaled 1.67, +joint 2.10): those are integrated endpoint error in canonical base-frame EE space, these are native joint space. The only shared, comparable statement is that both lines fail on LeKiwi, each against its own trivial floor.

Closed-loop eval was single-draw, and that was costing more than any ablation

predict_action_chunk starts from torch.randn and nothing seeded it, so every closed-loop rollout was a lottery. Caught by accident: the SAME checkpoint with the SAME settings scored 4/20 and then 8/20 on consecutive runs. Every previously reported closed-loop number was one draw from that distribution.

Fixed by averaging the flow ODE over 4 noise draws per decision and seeding each decision deterministically (--pred-seeds, default 4). The eval is now reproducible — three consecutive runs returned exactly 70.0%.

How much it was costing depends on how converged the checkpoint is:

checkpoint	single draw	4-seed averaged
new run, 5k steps (n=20)	20% / 40% (two runs)	70%
old run, 5k steps (n=20)	40%	60%
old run, 30k steps (n=50)	70%	68%

At 30k the averaging changes nothing (70 vs 68, within noise) — the flow field is sharp enough that one draw is already near the conditional mean. At 5k it is worth 20-30 points, because an under-trained field is diffuse and the noise compounds over a 500-step rollout. So the published 70% for the converged model stands, but every intermediate-checkpoint number in this document was noise-dominated, and the dual-rate comparison (70 vs 68) was never precise enough to support a claim either way.

Gripper command channel — the honest verdict so far

The ceiling gain is real and proven: GT round-trip 8/10 with a decoded command vs 9/10 with the carried command, matching raw GT exactly.

Whether the policy converts that ceiling into success is not yet established:

comparison (all 4-seed averaged)	result
new run @5k vs old run @5k, n=20	70% vs 60%
within the new checkpoint: decode canon vs threshold, n=20	70% vs 70%

+10 points at a matched step is two episodes out of twenty — not significant. And the decode-mode ablation on the same checkpoint shows no difference at all, which argues the channel is not the binding constraint at this stage: the model's predicted openness is already good enough that thresholding it matches reading the command.

The decisive number is the new run at 30k against the old run's 68% at n=50, and that needs the training to finish (currently ~10k/30k).