Upload folder using huggingface_hub

.gitattributes

@@ -39,3 +39,15 @@ videos/ep02_task2_succ.mp4 filter=lfs diff=lfs merge=lfs -text
 videos/ep03_task3_fail.mp4 filter=lfs diff=lfs merge=lfs -text
 videos/ep04_task4_succ.mp4 filter=lfs diff=lfs merge=lfs -text
 videos/ep05_task5_fail.mp4 filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/base_motor_holder_so101_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/base_so101_v2.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/motor_holder_so101_base_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/motor_holder_so101_wrist_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/moving_jaw_so101_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/rotation_pitch_so101_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/sts3215_03a_no_horn_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/sts3215_03a_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/under_arm_so101_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/upper_arm_so101_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/wrist_roll_follower_so101_v1.stl filter=lfs diff=lfs merge=lfs -text
+tinyvla2/assets/so_arm/assets/wrist_roll_pitch_so101_v2.stl filter=lfs diff=lfs merge=lfs -text

tinyvla2/CLAUDE.md

@@ -0,0 +1,153 @@
+# TinyVLA — project context
+
+Small vision-language-action models. Two lines of work:
+
+- **v1 (shipped)** — dual-rate VLA for LIBERO: slow Qwen3.5 semantic path (cacheable,
+  0.5–2 Hz) + fast flow-matching expert (10–20 Hz). **86% LIBERO-spatial, 84% 4-suite
+  average** — beats OpenVLA-7B on every suite at 0.64B params / 140M online.
+- **v2 (research)** — cross-embodiment: canonical action space + morphology
+  conditioning across 10 robots. Best: `tv2_C_scaled` (ratio-to-baseline **0.81**
+  in-training / **0.88** held-out; was 0.99/1.17 before scaling).
+
+Everything (checkpoints, results, wiring) lives in the **private** HF repo
+`AlexWortega/tinyvla` (v1 at the root, v2 under `tinyvla2/`).
+
+## Environment
+
+```bash
+source ~/tinyvla_venv/bin/activate     # NOT physics_venv
+```
+
+Pinned, non-obvious, and hard-won:
+- lerobot **0.6.1**, torch **2.10.0+cu128**, transformers **5.5.4** (lerobot's
+  `transformers-dep` extra; older breaks `eo1`).
+- **torchcodec 0.10 CPU build from PyPI** — the cu128 wheel clashes with the system
+  NPP libs. Video decode is the training bottleneck, use 12+ dataloader workers.
+- `placo` for SO-101 forward kinematics; needs `assets/so_arm/so101_new_calib.urdf`
+  **plus the 13 STL meshes** (placo loads collision geometry, it won't run without them).
+- `fla-core` + `causal-conv1d` give Qwen3.5 linear-attention its fast path (~5×);
+  they require CUDA — any CPU-only smoke test of the model will crash inside
+  `causal_conv1d`.
+- Data lives in `~/tinyvla_data` (gitignored), HF cache in `~/.cache/huggingface`.
+
+## Architecture
+
+```
+SLOW  (cacheable @1Hz)                    FAST (10–20 Hz, ~140M online)
+image → Qwen3.5-0.8B vision tower ─┬────► 64 spatial tokens/cam ──┐
+instruction ──────────────────────┤                              │
+                                  ▼                       proprio ┤
+                    LM (first 12 of 24 layers)      morphology/ID ┤
+                                  ▼                               ▼
+                   8–16 readout latents ────────────────► flow-matching expert
+                                                          (38M, chunk 50, 10 Euler)
+```
+
+Variant switches in `TinyVLAConfig`:
+- `conditioning`: `id` (learned embedding) | `morph` (16-number descriptor → MLP) |
+  `morph_qwen` (robot text through the shared LM)
+- `action_space`: `native` | `canonical` (base-frame EE deltas)
+- `vlm_native`: demos + robot text go INTO the LM sequence (dual-rate preserved)
+- `use_demo_conditioning`, `demo_only`: in-context exemplar channels (all dead, see below)
+
+## Data pipeline
+
+```
+LeRobot dataset → scripts/build_ee_sidecar.py (offline, once, NO video decode)
+                    SO-101: FK(joints) via placo;  others: EE pose from state
+                  → ~/tinyvla_data/ee_sidecar/<name>.parquet
+                  → src/tinyvla/data/canonical.py  (10 Hz resample, base-frame
+                    left-delta rotvec, gripper q1–q99, quantile norm to [-1,1])
+                  → CanonicalSource → WeightedMixtureDataset → policy
+```
+
+`docs/WIRING.md` has the full embodiment registry (id ↔ morph_key ↔ dataset ↔
+canonicalization source ↔ weight ↔ train/held-out).
+
+Robot pool: SO-101 (40 community sets, FK), Bridge/WidowX 53K eps, RT-1/Google 87K eps,
+UR5, xArm, DLR-EDAN, TidyBot, Stretch, Jaco, LeKiwi (held out), RoboCasa365/PandaOmron.
+
+## Running things
+
+```bash
+# training (any variant — the config picks the mode)
+PYTHONUNBUFFERED=1 python scripts/train.py --config configs/tv2_C_scaled.yaml
+
+# physical-space eval across robots (the metric that matters)
+python scripts/eval_similar_robots.py          # ratio = endpoint err ÷ zero-floor
+
+# LIBERO closed-loop (v1)
+MUJOCO_GL=egl python scripts/eval_libero.py --checkpoint <ckpt> \
+    --suite libero_spatial --episodes 100 --n-action-steps 10 --action-repeat 1
+```
+
+Long runs: launch with `nohup`, watch via checkpoints (stdout is block-buffered
+under nohup — use `PYTHONUNBUFFERED=1` or you see nothing for hours).
+
+## Evaluation — read this before trusting any number
+
+1. **Always run the zero-prediction control.** Per-step action error is degenerate
+   (jaco moves ~3.6 mm/step; predicting zeros scores ~4 mm). Use the
+   **integrated-trajectory endpoint error over the 50-step chunk**, and divide by the
+   per-robot **zero-floor** (what "predict mean motion" achieves). Only the ratio is
+   comparable across robots; ratio < 1.0 means the model beats the trivial baseline.
+2. **Check both held-out robots, not one.** "canonical ≫ native" looked decisive on
+   jaco and reversed on stretch.
+3. Absolute mm are meaningless across robots — different motion scales.
+
+## Results
+
+### v1 — LIBERO (closed-loop success)
+| suite | TinyVLA | OpenVLA-7B |
+|---|---|---|
+| spatial | **86%** (dual-rate @1 Hz; 79% full-rate) | 84.7% |
+| object | **94%** | 88.4% |
+| goal | **86%** | 79.2% |
+| long | **70%** | 53.7% |
+| **avg** | **84.0%** | 76.5% |
+
+Dual-rate (semantic latent refreshed at 1 Hz) matched or beat full-rate — the core
+architectural claim, verified closed-loop.
+
+### v2 — cross-embodiment (ratio to zero-floor, lower is better)
+| | in-training | held-out |
+|---|---|---|
+| C-diverse (8 robots, 20k steps, frozen LM) | 0.99 | 1.17 |
+| **C-scaled (9.1M frames, 60k steps, LM unfrozen)** | **0.81** | **0.88** |
+
+Per-robot final: jaco 0.69, xarm 0.64, dlr_edan 0.69, stretch 0.69, LeKiwi (held-out)
+1.07, ur5 1.22 (hardest — the fastest robot, 9.9 mm/step; its canonicalization is fine).
+
+**Negative results (all documented in `tinyvla2/RESULTS.md`):**
+- Robot description as text in the prompt (frozen LM) — hurts.
+- Morphology via the shared Qwen + unfrozen LM — 3× more content-sensitive, no
+  absolute gain.
+- In-context demo conditioning — **five architectures**, all dead (real vs garbage
+  exemplars indistinguishable): MLP same-task → MLP cross-task → 24× capacity →
+  demos inside the LM attention stream → demos as the only identity channel.
+
+## Lessons
+
+1. **Check against the trivial baseline before running ablations.** Five conditioning
+   mechanisms moved nothing while the model sat at ratio 0.99 — we were comparing
+   variants of an underfit model and measuring noise. Data + steps + unfreezing the LM
+   moved everything.
+2. **Silent data-convention bugs cost the most time.** LIBERO went 0% → 60% purely by
+   fixing three of them: robosuite renders are **180°-rotated** (a vertical flip alone
+   leaves a mirror), eef rotvec is antipodal vs the dataset, and the dataset's
+   `fps: 10` metadata is wrong (frames are native 20 Hz). A **GT-action replay
+   harness** (`scripts/replay_libero.py`) diagnoses all of these independently of the
+   policy — build it first.
+3. **Unfreezing the LM is the single biggest lever** (LIBERO 60% → 86%).
+4. Cross-morphology transfer converges slowest — LeKiwi was still improving at 60k
+   (1.24 → 1.13 → 1.07) when in-training had plateaued at 40k.
+5. Some community datasets are broken at source (meta frame count ≠ parquet rows);
+   validate `len(reader.hf_dataset) == meta.total_frames` and quarantine mismatches.
+
+## State
+
+Best checkpoints: **v1** `outputs/libero_ft4/final` (86% LIBERO) · **v2**
+`outputs/tv2_C_scaled/final` (0.81/0.88). Currently training: `tv2_robocasa` —
+continue-train from C-scaled on RoboCasa365 (sim kitchens, PandaOmron mobile
+manipulator, 1.5M frames) at weight 0.5 with the old robots replayed at 0.5, to
+separate new-domain integration from catastrophic forgetting.

tinyvla2/REPORT.md

@@ -0,0 +1,171 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-0.8B
+tags:
+  - robotics
+  - vla
+  - lerobot
+  - libero
+  - dual-rate
+pipeline_tag: robotics
+---
+
+# TinyVLA: a dual-rate VLA that beats OpenVLA-7B on LIBERO-Spatial at 0.64B params
+
+**Headline result: 86% success on LIBERO-Spatial (n=100) in dual-rate mode (semantic latent
+refreshed at 1 Hz), 79% full-rate — vs OpenVLA-7B's 84.7% — with 140M parameters on the
+online control path.** Built and trained end-to-end in ~5 days on a single RTX A6000.
+
+## 1. Architecture
+
+```
+                SLOW / SMART path (0.5–2 Hz, cached)
+instruction ─────────────────────┐
+camera → Qwen3.5-0.8B vision tower → Qwen3.5 language stack (first 12 of 24 layers)
+                  │                       │
+                  │            8 readout tokens → linear → semantic latent (8×512)
+                  │                       │ cached between refreshes
+──────────────────┼───────────────────────┼──────────────────────────
+                  │       FAST / MOTOR path (10–20 Hz, ~140M online)
+                  └── 64 spatial tokens per camera ─┐
+proprio (padded 16d) ───────────────────────────────┤
+embodiment-id embedding ────────────────────────────┤
+                                                    ▼
+                    flow-matching action expert (38M, 12 blocks,
+                    interleaved cross-/self-attention, AdaLN-free)
+                                                    ▼
+                              50-step action chunk (padded 8d + validity mask)
+```
+
+- **Single shared encoder**: the Qwen3.5-0.8B native ViT (100.6M) feeds both paths.
+- **Slow path**: image + instruction + 8 trailing pad-token "readout" slots through the
+  truncated language stack (L/2 layer-skip a la SmolVLA); their hidden states → latent 8×512.
+- **Fast path**: raw vision-tower spatial tokens (bypassing the LM) + proprio + embodiment
+  token, cross-attended by the flow expert. 10 Euler steps at inference.
+- **Flow matching**: linear-OT path, t ~ Beta(1.5, 1), velocity target, masked MSE.
+- Totals: 643M stored / 39.5M–575M trainable depending on stage / ~140M online per tick.
+- Implemented as an **out-of-tree LeRobot policy plugin**
+  (`--policy.type=tinyvla --policy.discover_packages_path=tinyvla`, lerobot 0.6.1).
+
+## 2. Training recipe
+
+| Stage | Data | Steps | Trainable | Result |
+|---|---|---|---|---|
+| 1. Overfit smoke | svla_so100_stacking (56 eps) | 5k | expert+proj (40M) | loss 1.885→0.099 |
+| 2. Mixture pretrain | 40 SO100/101 community datasets (~10.7K eps, weight 0.65, per-dataset proportional) + Bridge V2 10K eps (0.35) | 40k, bs 64×2, lr 1e-4 | 39.5M (backbone frozen) | held-out RMSE@t0 0.30σ; **stale-latent 2s == fresh** |
+| 2b. Staleness aug | same | from step 20k | — | latent from frame t−k, k~U(0,2s), p=0.5 |
+| 3. Spatial distillation | + 132K teacher labels (Qwen3.5-4B, center-point → 32×32 grid), CE ×0.15 on cam0 tokens | 20k, lr 5e-5 | same + head | head top-1 20.8%, top-5 44.8% (random 0.1%) |
+| 4. LIBERO FT (ft4) | HuggingFaceVLA/libero (1693 eps, all suites) | 40k, bs 48, lr 1e-4 / backbone 0.1× | **575M** (LM layers + vision unfrozen; vocab embeddings frozen) | loss 0.066 |
+
+Ablation FTs: ft1 (all frozen) → 0% closed-loop; ft2 (vision unfrozen) → 60%;
+ft3 (= ft2 from distilled ckpt) → 61%; **ft4 (LM+vision unfrozen) → 79/86%**.
+Unfreezing the LM was the single largest gain; distillation was neutral on LIBERO
+(likely overwritten by vision unfreezing during FT — bake it into pretraining next time).
+
+Cross-embodiment handling: canonical schema (cam0=scene/cam1=wrist via regex mapping,
+per-dataset mean/std normalization, action pad-to-8 + validity mask, state pad-to-16,
+learned embodiment-id token). Weighted mixture sampler with per-source retry on corrupt
+frames (IndexError/KeyError/decode RuntimeError).
+
+## 3. Results
+
+### LIBERO — all four suites (dual-rate, latent @ 1 Hz)
+
+| Suite | TinyVLA | OpenVLA-7B (published) |
+|---|---|---|
+| Spatial | **86%** (n=100; full-rate 79%) | 84.7% |
+| Object | **94%** (n=50) | 88.4% |
+| Goal | **86%** (n=50) | 79.2% |
+| Long (libero_10) | **70%** (n=50) | 53.7% |
+| **Average** | **84.0%** | 76.5% |
+
+Beats OpenVLA-7B on every suite at 1/11 the parameters and ~1/50 the online compute.
+XS-VLA (90.3 avg) remains ahead — their latent flow matching is future work for us.
+
+### LIBERO-Spatial (100 episodes, 10 per task, init states 0-9)
+
+| Mode | Success |
+|---|---|
+| Full-rate (latent every chunk) | 79/100 = **79%** |
+| **Dual-rate (latent @ 1 Hz)** | 86/100 = **86%** |
+
+Dual-rate ≥ full-rate consistently across all evals (also 65% vs 58% at ft3, n=50) —
+a stable cached latent appears to act as temporal regularization.
+
+### Context (published numbers, their own eval protocols)
+
+| Model | Params | LIBERO-Spatial | 4-suite avg |
+|---|---|---|---|
+| MolmoAct2-LIBERO | 5B | 98.4% (**97.2% re-measured on our rig**, n=500) | **98.25%** |
+| π0 | 3.3B | 98% | — |
+| SmolVLA-2.25B | 2.25B | 93% | — |
+| XS-VLA | 0.26B | ~93% | 90.3% |
+| MolmoAct-7B (v1) | 7B | 87.0% | 86.6% |
+| **TinyVLA (this)** | **0.64B (140M online)** | **86% dual-rate** | **84.0%** |
+| OpenVLA | 7B | 84.7% | 76.5% |
+
+**MolmoAct2 context** (from their cards/repo, no numeric tables published outside the
+LIBERO-LeRobot card): 5B params on Molmo2-ER backbone + flow expert; claims to beat π0.5;
+Molmo2-ER reportedly surpasses GPT-5 / Gemini-Robotics-ER-1.5 on 13 embodied-reasoning
+benchmarks; #1 VLA on the MolmoSpace leaderboard. Inference needs ~26GB fp32 (16GB bf16),
+cuda-graphs for ~2× speedup — H100-class serving vs our Raspberry-Pi-class 140M fast path.
+Notably they ship **MolmoAct2-SO100_101** (absolute joint control) — a 5B competitor on our
+target embodiment; a head-to-head offline eval on SO101 held-out data is an obvious next test.
+
+Honesty notes: our n=100 (±8pp) vs their 500; single suite; eval covers init states 0–9
+per task (demos of same tasks/init states are in training data — this is the standard
+LIBERO protocol used by all listed models). Multi-suite evals (object/goal/long) and an
+identically-protocoled SmolVLA baseline run are in progress.
+
+### Dual-rate validation (the architectural claim)
+
+- Offline: held-out chunk MSE with latent from a frame 2s earlier == fresh (0.772 vs 0.772).
+- Closed-loop: 86% @ 1 Hz refresh vs 79% full-rate.
+- Deployment implication (Raspberry Pi 5): the 0.5B slow path can run at 0.5–1 Hz
+  (llama.cpp GGUF Q4 territory), only the 140M fast path needs real-time rates.
+
+## 4. Pitfalls we hit (read before reproducing)
+
+1. **robosuite/LIBERO renders are 180°-rotated** vs recorded datasets. A vertical flip
+   alone leaves a horizontal MIRROR — the policy reaches to the wrong side and scores 0%.
+   Use `img[::-1, ::-1]`. (lerobot's `LiberoEnv.render()` flips for visualization but
+   `_format_raw_obs` feeds policies raw rotated frames — check your eval path.)
+2. **eef orientation antipodal ambiguity**: env quat → rotvec gives −π-ish x-component,
+   datasets store +π-ish. Canonicalize (flip antipodal representation when rotvec_x < 0)
+   or your normalized state input is off by ~17σ.
+3. **HuggingFaceVLA/libero meta says fps=10 but frames are native 20 Hz** (~161
+   frames/episode ≈ 8s demos). Do NOT action-repeat at eval.
+4. **GT-replay harness is the decisive convention test**: replay dataset actions in the env
+   (first episode per task ↔ init_state 0). 4/5 success = conventions right; policy-independent.
+5. HuggingFaceVLA community_dataset v1/v3 are aggregates of v2.1 sub-datasets → convert with
+   lerobot's `convert_dataset_v21_to_v30.py`; 4/44 sources have meta≠parquet row counts
+   (broken at source) — validate `len(reader.hf_dataset) == meta.total_frames`.
+6. Qwen3.5 custom forwards need `mm_token_type_ids` for M-RoPE; patchify must match the
+   official processor bit-exact (verify!). Linear-attention needs fla-core + causal-conv1d
+   or you get a 5× slower torch fallback.
+7. Flow-matching loss can drop while conditioning is ignored — A/B with shuffled
+   observations to verify the model actually uses them.
+
+## 5. Repository layout
+
+```
+src/tinyvla/            # LeRobot plugin: configuration/modeling/processor + modules
+scripts/train.py        # accelerate loop: weighted mixture, staleness aug, distill loss, resume
+scripts/eval_libero.py  # closed-loop LIBERO eval (all convention fixes, dual-rate, video)
+scripts/replay_libero.py, diag_obs_gap.py   # convention diagnostics
+scripts/prepare_community_v3.py, label_spatial.py, eval_offline.py
+configs/                # stage2/stage3/libero_ft* yaml
+checkpoints/            # stage2_final, stage3_final, libero_ft4_final (main)
+videos/                 # sample eval episodes
+```
+
+## 6. Quick start
+
+```python
+from tinyvla.modeling_tinyvla import TinyVLAPolicy
+policy = TinyVLAPolicy.from_pretrained("AlexWortega/tinyvla")  # ft4 checkpoint at root
+# closed-loop LIBERO: python scripts/eval_libero.py --checkpoint <ckpt> \
+#   --suite libero_spatial --episodes 100 --n-action-steps 10 --refresh-s 1.0
+```
+
+Trained 2026-08-14 → 2026-08-19 on 1× RTX A6000 (48GB), lerobot 0.6.1, torch 2.10, transformers 5.5.4.

tinyvla2/assets/so_arm/assets/base_motor_holder_so101_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8cd2f241037ea377af1191fffe0dd9d9006beea6dcc48543660ed41647072424
+size 1877084

tinyvla2/assets/so_arm/assets/base_so101_v2.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb12b7026575e1f70ccc7240051f9d943553bf34e5128537de6cd86fae33924d
+size 471584

tinyvla2/assets/so_arm/assets/motor_holder_so101_base_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31242ae6fb59d8b15c66617b88ad8e9bded62d57c35d11c0c43a70d2f4caa95b
+size 1129384

tinyvla2/assets/so_arm/assets/motor_holder_so101_wrist_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:887f92e6013cb64ea3a1ab8675e92da1e0beacfd5e001f972523540545e08011
+size 1052184

tinyvla2/assets/so_arm/assets/moving_jaw_so101_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:785a9dded2f474bc1d869e0d3dae398a3dcd9c0c345640040472210d2861fa9d
+size 1413584

tinyvla2/assets/so_arm/assets/rotation_pitch_so101_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9be900cc2a2bf718102841ef82ef8d2873842427648092c8ed2ca1e2ef4ffa34
+size 883684

tinyvla2/assets/so_arm/assets/sts3215_03a_no_horn_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75ef3781b752e4065891aea855e34dc161a38a549549cd0970cedd07eae6f887
+size 865884

tinyvla2/assets/so_arm/assets/sts3215_03a_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a37c871fb502483ab96c256baf457d36f2e97afc9205313d9c5ab275ef941cd0
+size 954084

tinyvla2/assets/so_arm/assets/under_arm_so101_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d01d1f2de365651dcad9d6669e94ff87ff7652b5bb2d10752a66a456a86dbc71
+size 1975884

tinyvla2/assets/so_arm/assets/upper_arm_so101_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:475056e03a17e71919b82fd88ab9a0b898ab50164f2a7943652a6b2941bb2d4f
+size 1303484

tinyvla2/assets/so_arm/assets/wrist_roll_follower_so101_v1.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b17b410a12d64ec39554abc3e8054d8a97384b2dc4a8d95a5ecb2a93670f5f4
+size 1439884

tinyvla2/assets/so_arm/assets/wrist_roll_pitch_so101_v2.stl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c7ec5525b4d8b9e397a30ab4bb0037156a5d5f38a4adf2c7d943d6c56eda5ae
+size 2699784

tinyvla2/assets/so_arm/so101_new_calib.urdf

@@ -0,0 +1,453 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generated using onshape-to-robot -->
+<!-- Onshape https://cad.onshape.com/documents/7715cc284bb430fe6dab4ffd/w/4fd0791b683777b02f8d975a/e/826c553ede3b7592eb9ca800 -->
+<robot name="so101_new_calib">
+
+  <!-- Materials -->
+  <material name="3d_printed">
+    <color rgba="1.0 0.82 0.12 1.0"/>
+  </material>
+  <material name="sts3215">
+    <color rgba="0.1 0.1 0.1 1.0"/>
+  </material>
+
+  <!-- Link base -->
+  <link name="base_link">
+    <inertial>
+      <origin xyz="0.0137179 -5.19711e-05 0.0334843" rpy="0 0 0"/>
+      <mass value="0.147"/>
+      <inertia ixx="0.000114686" ixy="-4.59787e-07" ixz="4.97151e-06" iyy="0.000136117" iyz="9.75275e-08" izz="0.000130364"/>
+    </inertial>
+    <!-- Part base_motor_holder_so101_v1 -->
+    <visual>
+      <origin xyz="-0.00636471 -9.94414e-05 -0.0024" rpy="1.5708 -1.67685e-15 1.5708"/>
+      <geometry>
+        <mesh filename="assets/base_motor_holder_so101_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.00636471 -9.94414e-05 -0.0024" rpy="1.5708 -1.67685e-15 1.5708"/>
+      <geometry>
+        <mesh filename="assets/base_motor_holder_so101_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part base_so101_v2 -->
+    <visual>
+      <origin xyz="-0.00636471 -8.97657e-09 -0.0024" rpy="1.5708 -2.78073e-29 1.5708"/>
+      <geometry>
+        <mesh filename="assets/base_so101_v2.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.00636471 -8.97657e-09 -0.0024" rpy="1.5708 -2.78073e-29 1.5708"/>
+      <geometry>
+        <mesh filename="assets/base_so101_v2.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part sts3215_03a_v1 -->
+    <visual>
+      <origin xyz="0.0263353 -8.97657e-09 0.0437" rpy="-8.21148e-16 7.84513e-18 1.249e-15"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="0.0263353 -8.97657e-09 0.0437" rpy="-8.21148e-16 7.84513e-18 1.249e-15"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part waveshare_mounting_plate_so101_v2 -->
+    <visual>
+      <origin xyz="-0.0309827 -0.000199441 0.0474" rpy="1.5708 -1.35493e-14 1.5708"/>
+      <geometry>
+        <mesh filename="assets/waveshare_mounting_plate_so101_v2.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0309827 -0.000199441 0.0474" rpy="1.5708 -1.35493e-14 1.5708"/>
+      <geometry>
+        <mesh filename="assets/waveshare_mounting_plate_so101_v2.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link shoulder -->
+  <link name="shoulder_link">
+    <inertial>
+      <origin xyz="-0.0307604 -1.66727e-05 -0.0252713" rpy="0 0 0"/>
+      <mass value="0.100006"/>
+      <inertia ixx="8.3759e-05" ixy="7.55525e-08" ixz="-1.16342e-06" iyy="8.10403e-05" iyz="1.54663e-07" izz="2.39783e-05"/>
+    </inertial>
+    <!-- Part sts3215_03a_v1_2 -->
+    <visual>
+      <origin xyz="-0.0303992 0.000422241 -0.0417" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0303992 0.000422241 -0.0417" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part motor_holder_so101_base_v1 -->
+    <visual>
+      <origin xyz="-0.0675992 -0.000177759 0.0158499" rpy="1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/motor_holder_so101_base_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0675992 -0.000177759 0.0158499" rpy="1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/motor_holder_so101_base_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part rotation_pitch_so101_v1 -->
+    <visual>
+      <origin xyz="0.0122008 2.22413e-05 0.0464" rpy="-1.5708 2.35221e-33 0"/>
+      <geometry>
+        <mesh filename="assets/rotation_pitch_so101_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="0.0122008 2.22413e-05 0.0464" rpy="-1.5708 2.35221e-33 0"/>
+      <geometry>
+        <mesh filename="assets/rotation_pitch_so101_v1.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link upper_arm -->
+  <link name="upper_arm_link">
+    <inertial>
+      <origin xyz="-0.0898471 -0.00838224 0.0184089" rpy="0 0 0"/>
+      <mass value="0.103"/>
+      <inertia ixx="4.08002e-05" ixy="-1.97819e-05" ixz="-4.03016e-08" iyy="0.000147318" iyz="8.97326e-09" izz="0.000142487"/>
+    </inertial>
+    <!-- Part sts3215_03a_v1_3 -->
+    <visual>
+      <origin xyz="-0.11257 -0.0155 0.0187" rpy="-3.14159 -5.27356e-16 -1.5708"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.11257 -0.0155 0.0187" rpy="-3.14159 -5.27356e-16 -1.5708"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part upper_arm_so101_v1 -->
+    <visual>
+      <origin xyz="-0.065085 0.012 0.0182" rpy="3.14159 -0 -1.30911e-30"/>
+      <geometry>
+        <mesh filename="assets/upper_arm_so101_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.065085 0.012 0.0182" rpy="3.14159 -0 -1.30911e-30"/>
+      <geometry>
+        <mesh filename="assets/upper_arm_so101_v1.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link lower_arm -->
+  <link name="lower_arm_link">
+    <inertial>
+      <origin xyz="-0.0980701 0.00324376 0.0182831" rpy="0 0 0"/>
+      <mass value="0.104"/>
+      <inertia ixx="2.87438e-05" ixy="7.41152e-06" ixz="1.26409e-06" iyy="0.000159844" iyz="-4.90188e-08" izz="0.00014529"/>
+    </inertial>
+    <!-- Part under_arm_so101_v1 -->
+    <visual>
+      <origin xyz="-0.0648499 -0.032 0.0182" rpy="3.14159 -0 6.67202e-31"/>
+      <geometry>
+        <mesh filename="assets/under_arm_so101_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0648499 -0.032 0.0182" rpy="3.14159 -0 6.67202e-31"/>
+      <geometry>
+        <mesh filename="assets/under_arm_so101_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part motor_holder_so101_wrist_v1 -->
+    <visual>
+      <origin xyz="-0.0648499 -0.032 0.018" rpy="-3.14159 -2.55351e-15 -1.83387e-30"/>
+      <geometry>
+        <mesh filename="assets/motor_holder_so101_wrist_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0648499 -0.032 0.018" rpy="-3.14159 -2.55351e-15 -1.83387e-30"/>
+      <geometry>
+        <mesh filename="assets/motor_holder_so101_wrist_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part sts3215_03a_v1_4 -->
+    <visual>
+      <origin xyz="-0.1224 0.0052 0.0187" rpy="-3.14159 -7.88861e-31 -3.14159"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.1224 0.0052 0.0187" rpy="-3.14159 -7.88861e-31 -3.14159"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link wrist -->
+  <link name="wrist_link">
+    <inertial>
+      <origin xyz="-0.000103312 -0.0386143 0.0281156" rpy="0 0 0"/>
+      <mass value="0.079"/>
+      <inertia ixx="3.68263e-05" ixy="1.7893e-08" ixz="-5.28128e-08" iyy="2.5391e-05" iyz="3.6412e-06" izz="2.1e-05"/>
+    </inertial>
+    <!-- Part sts3215_03a_no_horn_v1 -->
+    <visual>
+      <origin xyz="8.32667e-17 -0.0424 0.0306" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_no_horn_v1.stl"/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="8.32667e-17 -0.0424 0.0306" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_no_horn_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part wrist_roll_pitch_so101_v2 -->
+    <visual>
+      <origin xyz="0 -0.028 0.0181" rpy="-1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/wrist_roll_pitch_so101_v2.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="0 -0.028 0.0181" rpy="-1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename="assets/wrist_roll_pitch_so101_v2.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link gripper -->
+  <link name="gripper_link">
+    <inertial>
+      <origin xyz="0.000213627 0.000245138 -0.025187" rpy="0 0 0"/>
+      <mass value="0.087"/>
+      <inertia ixx="2.75087e-05" ixy="-3.35241e-07" ixz="-5.7352e-06" iyy="4.33657e-05" iyz="-5.17847e-08" izz="3.45059e-05"/>
+    </inertial>
+    <!-- Part sts3215_03a_v1_5 -->
+    <visual>
+      <origin xyz="0.0077 0.0001 -0.0234" rpy="-1.5708 -5.19179e-17 -1.66533e-16"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="0.0077 0.0001 -0.0234" rpy="-1.5708 -5.19179e-17 -1.66533e-16"/>
+      <geometry>
+        <mesh filename="assets/sts3215_03a_v1.stl"/>
+      </geometry>
+    </collision>
+    <!-- Part wrist_roll_follower_so101_v1 -->
+    <visual>
+      <origin xyz="8.32667e-17 -0.000218214 0.000949706" rpy="-3.14159 -5.55112e-17 0"/>
+      <geometry>
+        <mesh filename="assets/wrist_roll_follower_so101_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="8.32667e-17 -0.000218214 0.000949706" rpy="-3.14159 -5.55112e-17 0"/>
+      <geometry>
+        <mesh filename="assets/wrist_roll_follower_so101_v1.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Gripper frame (dummy link + fixed joint) -->
+  <link name="gripper_frame_link">
+    <origin xyz="0 0 0" rpy="0 -0 0"/>
+    <inertial>
+      <origin xyz="0 0 0" rpy="0 0 0"/>
+      <mass value="1e-9"/>
+      <inertia ixx="0" ixy="0" ixz="0" iyy="0" iyz="0" izz="0"/>
+    </inertial>
+  </link>
+
+  <joint name="gripper_frame_joint" type="fixed">
+    <origin xyz="-0.0079 -0.000218121 -0.0981274" rpy="0 3.14159 0"/>
+    <parent link="gripper_link"/>
+    <child link="gripper_frame_link"/>
+    <axis xyz="0 0 0"/>
+  </joint>
+
+  <!-- Link moving_jaw_so101_v1 -->
+  <link name="moving_jaw_so101_v1_link">
+    <inertial>
+      <origin xyz="-0.00157495 -0.0300244 0.0192755" rpy="0 0 0"/>
+      <mass value="0.012"/>
+      <inertia ixx="6.61427e-06" ixy="-3.19807e-07" ixz="-5.90717e-09" iyy="1.89032e-06" iyz="-1.09945e-07" izz="5.28738e-06"/>
+    </inertial>
+    <!-- Part moving_jaw_so101_v1 -->
+    <visual>
+      <origin xyz="-5.55112e-17 -5.55112e-17 0.0189" rpy="9.53145e-17 6.93889e-18 1.24077e-24"/>
+      <geometry>
+        <mesh filename="assets/moving_jaw_so101_v1.stl"/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-5.55112e-17 -5.55112e-17 0.0189" rpy="9.53145e-17 6.93889e-18 1.24077e-24"/>
+      <geometry>
+        <mesh filename="assets/moving_jaw_so101_v1.stl"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Joint from gripper to moving_jaw_so101_v1 -->
+  <joint name="gripper" type="revolute">
+    <origin xyz="0.0202 0.0188 -0.0234" rpy="1.5708 -5.24284e-08 -1.41553e-15"/>
+    <parent link="gripper_link"/>
+    <child link="moving_jaw_so101_v1_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-0.174533" upper="1.74533"/>
+  </joint>
+
+  <transmission name="gripper_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="gripper">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor6">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from wrist to gripper -->
+  <joint name="wrist_roll" type="revolute">
+    <origin xyz="5.55112e-17 -0.0611 0.0181" rpy="1.5708 0.0486795 3.14159"/>
+    <parent link="wrist_link"/>
+    <child link="gripper_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-2.74385" upper="2.84121"/>
+  </joint>
+
+  <transmission name="wrist_roll_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="wrist_roll">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor5">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from lower_arm to wrist -->
+  <joint name="wrist_flex" type="revolute">
+    <origin xyz="-0.1349 0.0052 3.62355e-17" rpy="4.02456e-15 8.67362e-16 -1.5708"/>
+    <parent link="lower_arm_link"/>
+    <child link="wrist_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.65806" upper="1.65806"/>
+  </joint>
+
+  <transmission name="wrist_flex_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="wrist_flex">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor4">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from upper_arm to lower_arm -->
+  <!-- Note: 5-degree calibration offset applied to joint limits -->
+  <joint name="elbow_flex" type="revolute">
+    <origin xyz="-0.11257 -0.028 1.73763e-16" rpy="-3.63608e-16 8.74301e-16 1.5708"/>
+    <parent link="upper_arm_link"/>
+    <child link="lower_arm_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.69" upper="1.69"/>
+  </joint>
+
+  <transmission name="elbow_flex_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="elbow_flex">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor3">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from shoulder to upper_arm -->
+  <joint name="shoulder_lift" type="revolute">
+    <origin xyz="-0.0303992 -0.0182778 -0.0542" rpy="-1.5708 -1.5708 0"/>
+    <parent link="shoulder_link"/>
+    <child link="upper_arm_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.74533" upper="1.74533"/>
+  </joint>
+
+  <transmission name="shoulder_lift_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="shoulder_lift">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor2">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from base to shoulder -->
+  <joint name="shoulder_pan" type="revolute">
+    <origin xyz="0.0388353 -8.97657e-09 0.0624" rpy="3.14159 4.18253e-17 -3.14159"/>
+    <parent link="base_link"/>
+    <child link="shoulder_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.91986" upper="1.91986"/>
+  </joint>
+
+  <transmission name="shoulder_pan_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="shoulder_pan">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor1">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission> 
+
+</robot>

tinyvla2/assets/so_arm/so101_nomesh.urdf

@@ -0,0 +1,453 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generated using onshape-to-robot -->
+<!-- Onshape https://cad.onshape.com/documents/7715cc284bb430fe6dab4ffd/w/4fd0791b683777b02f8d975a/e/826c553ede3b7592eb9ca800 -->
+<robot name="so101_new_calib">
+
+  <!-- Materials -->
+  <material name="3d_printed">
+    <color rgba="1.0 0.82 0.12 1.0"/>
+  </material>
+  <material name="sts3215">
+    <color rgba="0.1 0.1 0.1 1.0"/>
+  </material>
+
+  <!-- Link base -->
+  <link name="base_link">
+    <inertial>
+      <origin xyz="0.0137179 -5.19711e-05 0.0334843" rpy="0 0 0"/>
+      <mass value="0.147"/>
+      <inertia ixx="0.000114686" ixy="-4.59787e-07" ixz="4.97151e-06" iyy="0.000136117" iyz="9.75275e-08" izz="0.000130364"/>
+    </inertial>
+    <!-- Part base_motor_holder_so101_v1 -->
+    <visual>
+      <origin xyz="-0.00636471 -9.94414e-05 -0.0024" rpy="1.5708 -1.67685e-15 1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.00636471 -9.94414e-05 -0.0024" rpy="1.5708 -1.67685e-15 1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part base_so101_v2 -->
+    <visual>
+      <origin xyz="-0.00636471 -8.97657e-09 -0.0024" rpy="1.5708 -2.78073e-29 1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.00636471 -8.97657e-09 -0.0024" rpy="1.5708 -2.78073e-29 1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part sts3215_03a_v1 -->
+    <visual>
+      <origin xyz="0.0263353 -8.97657e-09 0.0437" rpy="-8.21148e-16 7.84513e-18 1.249e-15"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="0.0263353 -8.97657e-09 0.0437" rpy="-8.21148e-16 7.84513e-18 1.249e-15"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part waveshare_mounting_plate_so101_v2 -->
+    <visual>
+      <origin xyz="-0.0309827 -0.000199441 0.0474" rpy="1.5708 -1.35493e-14 1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0309827 -0.000199441 0.0474" rpy="1.5708 -1.35493e-14 1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link shoulder -->
+  <link name="shoulder_link">
+    <inertial>
+      <origin xyz="-0.0307604 -1.66727e-05 -0.0252713" rpy="0 0 0"/>
+      <mass value="0.100006"/>
+      <inertia ixx="8.3759e-05" ixy="7.55525e-08" ixz="-1.16342e-06" iyy="8.10403e-05" iyz="1.54663e-07" izz="2.39783e-05"/>
+    </inertial>
+    <!-- Part sts3215_03a_v1_2 -->
+    <visual>
+      <origin xyz="-0.0303992 0.000422241 -0.0417" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0303992 0.000422241 -0.0417" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part motor_holder_so101_base_v1 -->
+    <visual>
+      <origin xyz="-0.0675992 -0.000177759 0.0158499" rpy="1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0675992 -0.000177759 0.0158499" rpy="1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part rotation_pitch_so101_v1 -->
+    <visual>
+      <origin xyz="0.0122008 2.22413e-05 0.0464" rpy="-1.5708 2.35221e-33 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="0.0122008 2.22413e-05 0.0464" rpy="-1.5708 2.35221e-33 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link upper_arm -->
+  <link name="upper_arm_link">
+    <inertial>
+      <origin xyz="-0.0898471 -0.00838224 0.0184089" rpy="0 0 0"/>
+      <mass value="0.103"/>
+      <inertia ixx="4.08002e-05" ixy="-1.97819e-05" ixz="-4.03016e-08" iyy="0.000147318" iyz="8.97326e-09" izz="0.000142487"/>
+    </inertial>
+    <!-- Part sts3215_03a_v1_3 -->
+    <visual>
+      <origin xyz="-0.11257 -0.0155 0.0187" rpy="-3.14159 -5.27356e-16 -1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.11257 -0.0155 0.0187" rpy="-3.14159 -5.27356e-16 -1.5708"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part upper_arm_so101_v1 -->
+    <visual>
+      <origin xyz="-0.065085 0.012 0.0182" rpy="3.14159 -0 -1.30911e-30"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.065085 0.012 0.0182" rpy="3.14159 -0 -1.30911e-30"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link lower_arm -->
+  <link name="lower_arm_link">
+    <inertial>
+      <origin xyz="-0.0980701 0.00324376 0.0182831" rpy="0 0 0"/>
+      <mass value="0.104"/>
+      <inertia ixx="2.87438e-05" ixy="7.41152e-06" ixz="1.26409e-06" iyy="0.000159844" iyz="-4.90188e-08" izz="0.00014529"/>
+    </inertial>
+    <!-- Part under_arm_so101_v1 -->
+    <visual>
+      <origin xyz="-0.0648499 -0.032 0.0182" rpy="3.14159 -0 6.67202e-31"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0648499 -0.032 0.0182" rpy="3.14159 -0 6.67202e-31"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part motor_holder_so101_wrist_v1 -->
+    <visual>
+      <origin xyz="-0.0648499 -0.032 0.018" rpy="-3.14159 -2.55351e-15 -1.83387e-30"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.0648499 -0.032 0.018" rpy="-3.14159 -2.55351e-15 -1.83387e-30"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part sts3215_03a_v1_4 -->
+    <visual>
+      <origin xyz="-0.1224 0.0052 0.0187" rpy="-3.14159 -7.88861e-31 -3.14159"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="-0.1224 0.0052 0.0187" rpy="-3.14159 -7.88861e-31 -3.14159"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link wrist -->
+  <link name="wrist_link">
+    <inertial>
+      <origin xyz="-0.000103312 -0.0386143 0.0281156" rpy="0 0 0"/>
+      <mass value="0.079"/>
+      <inertia ixx="3.68263e-05" ixy="1.7893e-08" ixz="-5.28128e-08" iyy="2.5391e-05" iyz="3.6412e-06" izz="2.1e-05"/>
+    </inertial>
+    <!-- Part sts3215_03a_no_horn_v1 -->
+    <visual>
+      <origin xyz="8.32667e-17 -0.0424 0.0306" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="8.32667e-17 -0.0424 0.0306" rpy="1.5708 1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part wrist_roll_pitch_so101_v2 -->
+    <visual>
+      <origin xyz="0 -0.028 0.0181" rpy="-1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="0 -0.028 0.0181" rpy="-1.5708 -1.5708 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Link gripper -->
+  <link name="gripper_link">
+    <inertial>
+      <origin xyz="0.000213627 0.000245138 -0.025187" rpy="0 0 0"/>
+      <mass value="0.087"/>
+      <inertia ixx="2.75087e-05" ixy="-3.35241e-07" ixz="-5.7352e-06" iyy="4.33657e-05" iyz="-5.17847e-08" izz="3.45059e-05"/>
+    </inertial>
+    <!-- Part sts3215_03a_v1_5 -->
+    <visual>
+      <origin xyz="0.0077 0.0001 -0.0234" rpy="-1.5708 -5.19179e-17 -1.66533e-16"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="sts3215"/>
+    </visual>
+    <collision>
+      <origin xyz="0.0077 0.0001 -0.0234" rpy="-1.5708 -5.19179e-17 -1.66533e-16"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+    <!-- Part wrist_roll_follower_so101_v1 -->
+    <visual>
+      <origin xyz="8.32667e-17 -0.000218214 0.000949706" rpy="-3.14159 -5.55112e-17 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="8.32667e-17 -0.000218214 0.000949706" rpy="-3.14159 -5.55112e-17 0"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Gripper frame (dummy link + fixed joint) -->
+  <link name="gripper_frame_link">
+    <origin xyz="0 0 0" rpy="0 -0 0"/>
+    <inertial>
+      <origin xyz="0 0 0" rpy="0 0 0"/>
+      <mass value="1e-9"/>
+      <inertia ixx="0" ixy="0" ixz="0" iyy="0" iyz="0" izz="0"/>
+    </inertial>
+  </link>
+
+  <joint name="gripper_frame_joint" type="fixed">
+    <origin xyz="-0.0079 -0.000218121 -0.0981274" rpy="0 3.14159 0"/>
+    <parent link="gripper_link"/>
+    <child link="gripper_frame_link"/>
+    <axis xyz="0 0 0"/>
+  </joint>
+
+  <!-- Link moving_jaw_so101_v1 -->
+  <link name="moving_jaw_so101_v1_link">
+    <inertial>
+      <origin xyz="-0.00157495 -0.0300244 0.0192755" rpy="0 0 0"/>
+      <mass value="0.012"/>
+      <inertia ixx="6.61427e-06" ixy="-3.19807e-07" ixz="-5.90717e-09" iyy="1.89032e-06" iyz="-1.09945e-07" izz="5.28738e-06"/>
+    </inertial>
+    <!-- Part moving_jaw_so101_v1 -->
+    <visual>
+      <origin xyz="-5.55112e-17 -5.55112e-17 0.0189" rpy="9.53145e-17 6.93889e-18 1.24077e-24"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+      <material name="3d_printed"/>
+    </visual>
+    <collision>
+      <origin xyz="-5.55112e-17 -5.55112e-17 0.0189" rpy="9.53145e-17 6.93889e-18 1.24077e-24"/>
+      <geometry>
+        <mesh filename=""/>
+      </geometry>
+    </collision>
+  </link>
+
+  <!-- Joint from gripper to moving_jaw_so101_v1 -->
+  <joint name="gripper" type="revolute">
+    <origin xyz="0.0202 0.0188 -0.0234" rpy="1.5708 -5.24284e-08 -1.41553e-15"/>
+    <parent link="gripper_link"/>
+    <child link="moving_jaw_so101_v1_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-0.174533" upper="1.74533"/>
+  </joint>
+
+  <transmission name="gripper_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="gripper">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor6">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from wrist to gripper -->
+  <joint name="wrist_roll" type="revolute">
+    <origin xyz="5.55112e-17 -0.0611 0.0181" rpy="1.5708 0.0486795 3.14159"/>
+    <parent link="wrist_link"/>
+    <child link="gripper_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-2.74385" upper="2.84121"/>
+  </joint>
+
+  <transmission name="wrist_roll_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="wrist_roll">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor5">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from lower_arm to wrist -->
+  <joint name="wrist_flex" type="revolute">
+    <origin xyz="-0.1349 0.0052 3.62355e-17" rpy="4.02456e-15 8.67362e-16 -1.5708"/>
+    <parent link="lower_arm_link"/>
+    <child link="wrist_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.65806" upper="1.65806"/>
+  </joint>
+
+  <transmission name="wrist_flex_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="wrist_flex">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor4">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from upper_arm to lower_arm -->
+  <!-- Note: 5-degree calibration offset applied to joint limits -->
+  <joint name="elbow_flex" type="revolute">
+    <origin xyz="-0.11257 -0.028 1.73763e-16" rpy="-3.63608e-16 8.74301e-16 1.5708"/>
+    <parent link="upper_arm_link"/>
+    <child link="lower_arm_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.69" upper="1.69"/>
+  </joint>
+
+  <transmission name="elbow_flex_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="elbow_flex">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor3">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from shoulder to upper_arm -->
+  <joint name="shoulder_lift" type="revolute">
+    <origin xyz="-0.0303992 -0.0182778 -0.0542" rpy="-1.5708 -1.5708 0"/>
+    <parent link="shoulder_link"/>
+    <child link="upper_arm_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.74533" upper="1.74533"/>
+  </joint>
+
+  <transmission name="shoulder_lift_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="shoulder_lift">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor2">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission>
+
+  <!-- Joint from base to shoulder -->
+  <joint name="shoulder_pan" type="revolute">
+    <origin xyz="0.0388353 -8.97657e-09 0.0624" rpy="3.14159 4.18253e-17 -3.14159"/>
+    <parent link="base_link"/>
+    <child link="shoulder_link"/>
+    <axis xyz="0 0 1"/>
+    <limit effort="10" velocity="10" lower="-1.91986" upper="1.91986"/>
+  </joint>
+
+  <transmission name="shoulder_pan_trans">
+    <type>transmission_interface/SimpleTransmission</type>
+    <joint name="shoulder_pan">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+    </joint>
+    <actuator name="motor1">
+      <hardwareInterface>hardware_interface/PositionJointInterface</hardwareInterface>
+      <mechanicalReduction>1</mechanicalReduction>
+    </actuator>
+  </transmission> 
+
+</robot>

tinyvla2/configs/ablation_nolatent.yaml

@@ -0,0 +1,25 @@
+# Stage-2 gate baseline: same mixture, 5k steps, nolatent ablation
+output_dir: /home/alexw/tinyvla/outputs/ablation_nolatent
+wandb: null
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.65
+  - repo_id: nvidia/BridgeData2_LeRobot_v3
+    root: /home/alexw/tinyvla_data/bridge_v3_10k
+    revision: main
+    episodes: 10000
+    weight: 0.35
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  use_semantic_latent: false
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 500
+steps: 5000
+log_freq: 50
+save_freq: 2500
+mixed_precision: bf16

tinyvla2/configs/ablation_nospatial.yaml

@@ -0,0 +1,25 @@
+# Stage-2 gate baseline: same mixture, 5k steps, nospatial ablation
+output_dir: /home/alexw/tinyvla/outputs/ablation_nospatial
+wandb: null
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.65
+  - repo_id: nvidia/BridgeData2_LeRobot_v3
+    root: /home/alexw/tinyvla_data/bridge_v3_10k
+    revision: main
+    episodes: 10000
+    weight: 0.35
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  use_spatial_tokens: false
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 500
+steps: 5000
+log_freq: 50
+save_freq: 2500
+mixed_precision: bf16

tinyvla2/configs/libero_ft.yaml

@@ -0,0 +1,31 @@
+# LIBERO fine-tune from the Stage-2 pretrained checkpoint (XS-VLA/SmolVLA protocol).
+# LIBERO = new embodiment (id 2): Franka 7-DoF EE @10fps, two 256^2 cameras.
+
+output_dir: /home/alexw/tinyvla/outputs/libero_ft
+wandb: null
+
+datasets:
+  - repo_id: HuggingFaceVLA/libero
+    weight: 1.0
+    embodiment_id: 2
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+
+resume_from: /home/alexw/tinyvla/outputs/stage2_mixture/final
+resume_step: 0
+
+batch_size: 64
+grad_accum: 1
+num_workers: 12
+lr: 5.0e-5
+warmup_steps: 500
+steps: 15000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 2500
+mixed_precision: bf16

tinyvla2/configs/libero_ft2.yaml

@@ -0,0 +1,33 @@
+# LIBERO fine-tune v2: unfreeze the vision tower (sim renders are OOD for the
+# real-world-pretrained encoder), longer schedule.
+
+output_dir: /home/alexw/tinyvla/outputs/libero_ft2
+wandb: null
+
+datasets:
+  - repo_id: HuggingFaceVLA/libero
+    weight: 1.0
+    embodiment_id: 2
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: false
+
+backbone_lr_mult: 0.1
+
+resume_from: /home/alexw/tinyvla/outputs/stage2_mixture/final
+resume_step: 0
+
+batch_size: 48
+grad_accum: 1
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 30000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/libero_ft3.yaml

@@ -0,0 +1,33 @@
+# LIBERO fine-tune v2: unfreeze the vision tower (sim renders are OOD for the
+# real-world-pretrained encoder), longer schedule.
+
+output_dir: /home/alexw/tinyvla/outputs/libero_ft3
+wandb: null
+
+datasets:
+  - repo_id: HuggingFaceVLA/libero
+    weight: 1.0
+    embodiment_id: 2
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: false
+
+backbone_lr_mult: 0.1
+
+resume_from: /home/alexw/tinyvla/outputs/stage3_distill/final
+resume_step: 0
+
+batch_size: 48
+grad_accum: 1
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 30000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/libero_ft4.yaml

@@ -0,0 +1,33 @@
+# LIBERO fine-tune v4: unfreeze BOTH vision tower and language stack
+# (12 truncated layers; vocab embeddings stay frozen) at 0.1x lr, longer schedule.
+
+output_dir: /home/alexw/tinyvla/outputs/libero_ft4
+wandb: null
+
+datasets:
+  - repo_id: HuggingFaceVLA/libero
+    weight: 1.0
+    embodiment_id: 2
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: false
+
+backbone_lr_mult: 0.1
+
+resume_from: /home/alexw/tinyvla/outputs/stage3_distill/final
+resume_step: 0
+
+batch_size: 48
+grad_accum: 1
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 40000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/morphology/descriptors.yaml

@@ -0,0 +1,277 @@
+# Per-embodiment morphology descriptors (variant C).
+# Fields hand-normalized to O(1); order MUST match MORPH_FIELDS in
+# src/tinyvla/modules/embodiment.py. Values: raw → normalized applied at load.
+# arm_dof/10, reach_m*2, gripper_width_m*10, num_cameras/3, is_mobile ±1,
+# control_hz/30, joint_lo_mean/pi, joint_hi_mean/pi, workspace_xyz*2,
+# payload_kg/5, ee_type_parallel ±1, ee_type_multi ±1, base_holonomic ±1, reserved.
+
+so101:        # SO-100/101 5-DOF + gripper
+  arm_dof: 5
+  reach_m: 0.35
+  gripper_width_m: 0.03
+  num_cameras: 2
+  is_mobile: -1
+  control_hz: 30
+  joint_lo_mean: -1.9
+  joint_hi_mean: 1.9
+  workspace_x: 0.30
+  workspace_y: 0.30
+  workspace_z: 0.25
+  payload_kg: 0.25
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+bridge:       # WidowX 250 6-DOF
+  arm_dof: 6
+  reach_m: 0.60
+  gripper_width_m: 0.06
+  num_cameras: 1
+  is_mobile: -1
+  control_hz: 5
+  joint_lo_mean: -2.6
+  joint_hi_mean: 2.6
+  workspace_x: 0.40
+  workspace_y: 0.40
+  workspace_z: 0.35
+  payload_kg: 0.25
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+rt1:          # Google robot (mobile manipulator, 7-DOF arm)
+  arm_dof: 7
+  reach_m: 0.90
+  gripper_width_m: 0.08
+  num_cameras: 1
+  is_mobile: 1
+  control_hz: 3
+  joint_lo_mean: -2.9
+  joint_hi_mean: 2.9
+  workspace_x: 0.70
+  workspace_y: 0.70
+  workspace_z: 0.60
+  payload_kg: 1.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: 1
+  reserved: 0
+
+droid:        # Franka Panda 7-DOF
+  arm_dof: 7
+  reach_m: 0.85
+  gripper_width_m: 0.08
+  num_cameras: 2
+  is_mobile: -1
+  control_hz: 15
+  joint_lo_mean: -2.8
+  joint_hi_mean: 2.8
+  workspace_x: 0.60
+  workspace_y: 0.60
+  workspace_z: 0.55
+  payload_kg: 3.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+libero:       # Franka Panda (sim) — same morphology as droid
+  arm_dof: 7
+  reach_m: 0.85
+  gripper_width_m: 0.08
+  num_cameras: 2
+  is_mobile: -1
+  control_hz: 20
+  joint_lo_mean: -2.8
+  joint_hi_mean: 2.8
+  workspace_x: 0.60
+  workspace_y: 0.60
+  workspace_z: 0.55
+  payload_kg: 3.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+# --- held-out (never in pretraining; descriptors ready for zero-shot variant C) ---
+jaco:         # Kinova Jaco 6-DOF, 3-finger gripper
+  arm_dof: 6
+  reach_m: 0.90
+  gripper_width_m: 0.10
+  num_cameras: 1
+  is_mobile: -1
+  control_hz: 10
+  joint_lo_mean: -3.1
+  joint_hi_mean: 3.1
+  workspace_x: 0.60
+  workspace_y: 0.60
+  workspace_z: 0.55
+  payload_kg: 1.6
+  ee_type_parallel: -1
+  ee_type_multi: 1
+  base_holonomic: -1
+  reserved: 0
+
+fanuc:        # Fanuc Mate 6-DOF industrial
+  arm_dof: 6
+  reach_m: 0.70
+  gripper_width_m: 0.05
+  num_cameras: 1
+  is_mobile: -1
+  control_hz: 10
+  joint_lo_mean: -3.1
+  joint_hi_mean: 3.1
+  workspace_x: 0.50
+  workspace_y: 0.50
+  workspace_z: 0.45
+  payload_kg: 4.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+lekiwi:       # LeKiwi — SO-100 (5-DOF) arm on a 3-wheel holonomic kiwi base (mobile manipulator)
+  arm_dof: 5
+  reach_m: 0.35
+  gripper_width_m: 0.03
+  num_cameras: 2
+  is_mobile: 1
+  control_hz: 30
+  joint_lo_mean: -1.9
+  joint_hi_mean: 1.9
+  workspace_x: 1.5
+  workspace_y: 1.5
+  workspace_z: 0.30
+  payload_kg: 0.25
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: 1
+  reserved: 0
+
+ur5:          # Universal Robots UR5e — 6-DOF industrial arm, fixed base
+  arm_dof: 6
+  reach_m: 0.85
+  gripper_width_m: 0.085
+  num_cameras: 1
+  is_mobile: -1
+  control_hz: 5
+  joint_lo_mean: -3.1
+  joint_hi_mean: 3.1
+  workspace_x: 0.70
+  workspace_y: 0.70
+  workspace_z: 0.60
+  payload_kg: 5.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+xarm:         # UFactory xArm — 6-DOF arm, fixed base
+  arm_dof: 6
+  reach_m: 0.70
+  gripper_width_m: 0.08
+  num_cameras: 1
+  is_mobile: -1
+  control_hz: 5
+  joint_lo_mean: -3.1
+  joint_hi_mean: 3.1
+  workspace_x: 0.55
+  workspace_y: 0.55
+  workspace_z: 0.50
+  payload_kg: 3.5
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+dlr_edan:     # DLR EDAN — 7-DOF assistive arm on a wheelchair (mobile assistive)
+  arm_dof: 7
+  reach_m: 0.80
+  gripper_width_m: 0.07
+  num_cameras: 1
+  is_mobile: 1
+  control_hz: 5
+  joint_lo_mean: -2.9
+  joint_hi_mean: 2.9
+  workspace_x: 0.65
+  workspace_y: 0.65
+  workspace_z: 0.55
+  payload_kg: 2.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+tidybot:      # holonomic wheeled mobile manipulator (Kinova arm on omni base), whole-body EE control
+  arm_dof: 7
+  reach_m: 0.90
+  gripper_width_m: 0.08
+  num_cameras: 2
+  is_mobile: 1
+  control_hz: 10
+  joint_lo_mean: -2.9
+  joint_hi_mean: 2.9
+  workspace_x: 2.0
+  workspace_y: 2.0
+  workspace_z: 1.0
+  payload_kg: 1.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: 1
+  reserved: 0
+
+uav:          # quadrotor drone — free-flying 6-DOF body, no arm, no gripper (max morphology gap)
+  arm_dof: 0
+  reach_m: 0.0
+  gripper_width_m: 0.0
+  num_cameras: 6
+  is_mobile: 1
+  control_hz: 30
+  joint_lo_mean: 0.0
+  joint_hi_mean: 0.0
+  workspace_x: 5.0
+  workspace_y: 5.0
+  workspace_z: 5.0
+  payload_kg: 0.5
+  ee_type_parallel: -1
+  ee_type_multi: -1
+  base_holonomic: 1
+  reserved: 1
+
+hello_stretch:  # Hello Robot Stretch — mobile base + prismatic lift
+  arm_dof: 4
+  reach_m: 0.52
+  gripper_width_m: 0.08
+  num_cameras: 1
+  is_mobile: 1
+  control_hz: 10
+  joint_lo_mean: -1.5
+  joint_hi_mean: 1.5
+  workspace_x: 0.50
+  workspace_y: 0.30
+  workspace_z: 1.10
+  payload_kg: 1.5
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: -1
+  reserved: 0
+
+panda_omron:  # RoboCasa PandaOmron — Franka 7-DOF arm on a holonomic omni mobile base (kitchen scenes)
+  arm_dof: 7
+  reach_m: 0.85
+  gripper_width_m: 0.08
+  num_cameras: 3
+  is_mobile: 1
+  control_hz: 20
+  joint_lo_mean: -2.8
+  joint_hi_mean: 2.8
+  workspace_x: 2.0
+  workspace_y: 2.0
+  workspace_z: 1.2
+  payload_kg: 3.0
+  ee_type_parallel: 1
+  ee_type_multi: -1
+  base_holonomic: 1
+  reserved: 0

tinyvla2/configs/morphology/robot_prompts.yaml

@@ -0,0 +1,13 @@
+# Natural-language robot descriptions prepended to the task instruction (Qwen slow path).
+# Complements the numeric morphology descriptor (C scheme) with a text channel.
+so101: "Robot: SO-101, a 5-DOF tabletop arm with a parallel-jaw gripper on a fixed base."
+bridge: "Robot: WidowX 250, a 6-DOF tabletop arm with a parallel gripper on a fixed base."
+rt1: "Robot: Google Robot, a 7-DOF arm mounted on a wheeled mobile base."
+ur5: "Robot: UR5e, a 6-DOF industrial arm with a parallel gripper on a fixed base."
+xarm: "Robot: xArm, a 6-DOF arm with a parallel gripper on a fixed base."
+dlr_edan: "Robot: DLR EDAN, a 7-DOF assistive arm mounted on a wheelchair mobile base."
+tidybot: "Robot: TidyBot, a 7-DOF arm on a holonomic omnidirectional wheeled base; whole-body end-effector control."
+hello_stretch: "Robot: Hello Stretch, an arm on a telescoping prismatic lift atop a mobile base."
+jaco: "Robot: Kinova Jaco, a 6-DOF arm with a three-finger gripper on a fixed base."
+lekiwi: "Robot: LeKiwi, a 5-DOF SO-100 arm mounted on a three-wheel holonomic mobile base."
+panda_omron: "Robot: RoboCasa PandaOmron, a 7-DOF Franka arm on a holonomic omnidirectional mobile base, operating in kitchen scenes."

tinyvla2/configs/stage2_mixture.yaml

@@ -0,0 +1,37 @@
+# Stage 2: small-scale mixture pretraining on one A6000.
+# SO-100/101 community slice (converted v3.0) 0.65 + Bridge V2 0.35.
+# Fractal dropped for MVP (v2.0, unconvertible in lerobot 0.6.1).
+
+output_dir: /home/alexw/tinyvla/outputs/stage2_mixture
+wandb: null  # set a project name to enable
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.65
+  - repo_id: nvidia/BridgeData2_LeRobot_v3
+    root: /home/alexw/tinyvla_data/bridge_v3_10k
+    revision: main
+    episodes: 10000
+    weight: 0.35
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+
+batch_size: 64
+grad_accum: 2
+num_workers: 14
+resume_from: /home/alexw/tinyvla/outputs/stage2_mixture/step_2000
+resume_step: 2000
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 40000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 2000
+mixed_precision: bf16
+staleness_start_step: 20000
+staleness_prob: 0.5

tinyvla2/configs/stage3_distill.yaml

@@ -0,0 +1,40 @@
+# Stage 3: continue SO101+Bridge pretraining with the spatial-distillation
+# aux CE loss (teacher labels from Qwen3.5-4B) and staleness aug from the start.
+
+output_dir: /home/alexw/tinyvla/outputs/stage3_distill
+wandb: null
+
+spatial_labels_dir: /home/alexw/tinyvla_data/spatial_labels
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.65
+  - repo_id: nvidia/BridgeData2_LeRobot_v3
+    root: /home/alexw/tinyvla_data/bridge_v3_10k
+    revision: main
+    episodes: 10000
+    weight: 0.35
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  spatial_loss_weight: 0.15
+
+resume_from: /home/alexw/tinyvla/outputs/stage2_mixture/final
+resume_step: 0
+
+batch_size: 64
+grad_accum: 2
+num_workers: 14
+lr: 5.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 2500
+mixed_precision: bf16
+staleness_start_step: 0
+staleness_prob: 0.5

tinyvla2/configs/tv2_A.yaml

@@ -0,0 +1,42 @@
+# TinyVLA-2 variant A: conditioning=id, action_space=native
+# Identical data/weights/seed across A/B/C (matched-pairs design).
+output_dir: /home/alexw/tinyvla/outputs/tv2_A_id_native
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.40
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 20000
+    weight: 0.30
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 20000
+    weight: 0.30
+    morph_key: rt1
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: id
+  action_space: native
+
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 40000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_B.yaml

@@ -0,0 +1,42 @@
+# TinyVLA-2 variant B: conditioning=id, action_space=canonical
+# Identical data/weights/seed across A/B/C (matched-pairs design).
+output_dir: /home/alexw/tinyvla/outputs/tv2_B_id_canon
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.40
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 20000
+    weight: 0.30
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 20000
+    weight: 0.30
+    morph_key: rt1
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: id
+  action_space: canonical
+
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 40000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C.yaml

@@ -0,0 +1,42 @@
+# TinyVLA-2 variant C: conditioning=morph, action_space=canonical
+# Identical data/weights/seed across A/B/C (matched-pairs design).
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_morph_canon
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.40
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 20000
+    weight: 0.30
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 20000
+    weight: 0.30
+    morph_key: rt1
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph
+  action_space: canonical
+
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 40000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_diverse.yaml

@@ -0,0 +1,67 @@
+# Continue-train variant C on an EXPANDED, morphology-diverse mixture to fix the
+# OOD descriptor extrapolation (original C's morph-encoder saw only 3 arm-ish
+# embodiments → failed zero-shot on mobile TidyBot). Now 8 embodiments spanning
+# fixed arms (SO101, WidowX, UR5, xArm) + mobile (Google-robot, dlr_edan wheelchair,
+# TidyBot holonomic, Stretch). jaco held out entirely for the transfer test.
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_diverse
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_morph_canon/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph
+  action_space: canonical
+
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 25000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_diverse_slow.yaml

@@ -0,0 +1,68 @@
+# Continue-train variant C on an EXPANDED, morphology-diverse mixture to fix the
+# OOD descriptor extrapolation (original C's morph-encoder saw only 3 arm-ish
+# embodiments → failed zero-shot on mobile TidyBot). Now 8 embodiments spanning
+# fixed arms (SO101, WidowX, UR5, xArm) + mobile (Google-robot, dlr_edan wheelchair,
+# TidyBot holonomic, Stretch). jaco held out entirely for the transfer test.
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_diverse_slow
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_diverse/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph
+  morph_to_slow: true
+  action_space: canonical
+
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 25000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_incontext.yaml

@@ -0,0 +1,72 @@
+# Variant: in-context few-shot DEMO conditioning — a few (obs, action) exemplar
+# pairs from the target robot's own data, encoded via the shared spatial-aware
+# vision encoder + a small MLP, WITHOUT any gradient update to adapt to that robot.
+# Composed with the text-via-Qwen morphology channel (both active). Resumes from
+# tv2_C_qwen_morph/final (LM already unfrozen, morph_out_proj already trained).
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_incontext
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph_qwen
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+  use_demo_conditioning: true
+  n_support: 3
+
+backbone_lr_mult: 0.1
+batch_size: 40
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_incontext_v2.yaml

@@ -0,0 +1,73 @@
+# Variant: in-context few-shot DEMO conditioning — a few (obs, action) exemplar
+# pairs from the target robot's own data, encoded via the shared spatial-aware
+# vision encoder + a small MLP, WITHOUT any gradient update to adapt to that robot.
+# Composed with the text-via-Qwen morphology channel (both active). Resumes from
+# tv2_C_qwen_morph/final (LM already unfrozen, morph_out_proj already trained).
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_incontext_v2
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph_qwen
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+  use_demo_conditioning: true
+  n_support: 3
+  support_other_task: true
+
+backbone_lr_mult: 0.1
+batch_size: 40
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_incontext_v3.yaml

@@ -0,0 +1,77 @@
+# Variant: in-context few-shot DEMO conditioning — a few (obs, action) exemplar
+# pairs from the target robot's own data, encoded via the shared spatial-aware
+# vision encoder + a small MLP, WITHOUT any gradient update to adapt to that robot.
+# Composed with the text-via-Qwen morphology channel (both active). Resumes from
+# tv2_C_qwen_morph/final (LM already unfrozen, morph_out_proj already trained).
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_incontext_v3
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph_qwen
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+  use_demo_conditioning: true
+  n_support: 3
+  support_other_task: true
+  demo_tokens_per_example: 4
+  demo_visual_groups: 4
+  demo_action_keys: 8
+  demo_hidden_mult: 4
+
+backbone_lr_mult: 0.1
+batch_size: 40
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_mega.yaml

@@ -0,0 +1,71 @@
+# Continue-train C on the MAXIMALLY diverse robot set (9 embodiments) with BOTH
+# channels describing the robot: numeric morphology descriptor (C scheme) AND a
+# natural-language robot description in the Qwen prompt (slow path). Held out: LeKiwi
+# (mobile SO-100 + kiwi base) — we have its zero-shot baseline (295mm) to compare.
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_mega
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_diverse/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.20
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 12000
+    weight: 0.15
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 12000
+    weight: 0.15
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.06
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.09
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.08
+    morph_key: hello_stretch
+  - repo_id: heldout_jaco_play
+    root: /home/alexw/tinyvla_data/heldout/heldout_jaco_play
+    weight: 0.08
+    morph_key: jaco
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: true
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph
+  action_space: canonical
+
+batch_size: 64
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_qwen_morph.yaml

@@ -0,0 +1,73 @@
+# Variant: morphology conditioning via the SAME shared Qwen (text-only pass over a
+# robot description, e.g. "Robot: LeKiwi, a 5-DOF SO-100 arm on a three-wheel
+# holonomic mobile base."), replacing the from-scratch MLP-over-numbers encoder.
+# ALSO unfreezes the LM (freeze_lm: false) — per user request, since the frozen LM
+# was the likely reason the earlier text-prompt-in-instruction experiment failed to
+# learn to use new text. Resumes from C-diverse (8 robots, our best checkpoint:
+# 150mm on held-out jaco). LeKiwi held out again for direct comparison (prior
+# baselines: C-diverse numeric-only zero-shot 295mm; C-mega text-in-prompt 522mm).
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_diverse/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph_qwen
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+
+backbone_lr_mult: 0.1
+batch_size: 48
+grad_accum: 2
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_C_scaled.yaml

@@ -0,0 +1,76 @@
+# SCALED C: the setup that works (canonical actions + numeric morphology
+# descriptor), given a real budget. The reality check showed the model barely beat
+# the trivial baseline (mean ratio 0.99 in-training) because 40k+20k steps were
+# spread across 8 morphologies with a frozen backbone. This run scales:
+#   data : full Bridge (53K eps) + full RT-1 (87K eps) instead of 12-15K subsets
+#          -> pool 4.4M -> ~9.1M frames
+#   steps: 60k (was 20k) at eff batch 128 -> 7.7M samples (~0.85 epoch)
+#   model: LM UNFROZEN at 0.1x lr (the single biggest lever in TinyVLA v1:
+#          frozen 60% -> unfrozen 86% on LIBERO)
+# Held out: LeKiwi (mobile) + jaco stays in for training here, so held-out
+# comparison uses LeKiwi; jaco numbers become in-distribution reference.
+output_dir: /home/alexw/tinyvla/outputs/tv2_C_scaled
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_diverse/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.22
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset          # FULL 53,192 eps (1.89M frames)
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    weight: 0.22
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset            # FULL 87,212 eps (3.79M frames)
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    weight: 0.22
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.09
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.08
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.04
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.05
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.04
+    morph_key: hello_stretch
+  - repo_id: heldout_jaco_play
+    root: /home/alexw/tinyvla_data/heldout/heldout_jaco_play
+    weight: 0.04
+    morph_key: jaco
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false          # the biggest known lever
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph       # numeric descriptor: the channel that actually works
+  action_space: canonical
+
+backbone_lr_mult: 0.1
+batch_size: 48
+grad_accum: 3               # eff batch 144
+num_workers: 12
+lr: 1.0e-4
+warmup_steps: 1000
+steps: 60000
+grad_clip: 10.0
+log_freq: 100
+save_freq: 10000
+mixed_precision: bf16

tinyvla2/configs/tv2_demoonly.yaml

@@ -0,0 +1,89 @@
+# DEMO-ONLY: every robot-identity channel removed (no embodiment ID, no
+# morphology descriptor, no robot text) — the K demo exemplars in the LM
+# attention stream are the ONLY cue for which robot this is. Per-source action
+# normalization creates the pressure: to predict correctly the model MUST infer
+# the robot's action scale, and only the demos carry it.
+# (was: RICH SLOW PATH) demos + robot text + instruction + image all inside the LM
+# sequence (demos in the ATTENTION STREAM), LM emits latent tokens conditioning
+# the fast path. Dual-rate PRESERVED (fast path unchanged, latent cacheable).
+# (was: in-context few-shot DEMO conditioning) — a few (obs, action) exemplar
+# pairs from the target robot's own data, encoded via the shared spatial-aware
+# vision encoder + a small MLP, WITHOUT any gradient update to adapt to that robot.
+# Composed with the text-via-Qwen morphology channel (both active). Resumes from
+# tv2_C_qwen_morph/final (LM already unfrozen, morph_out_proj already trained).
+output_dir: /home/alexw/tinyvla/outputs/tv2_demoonly
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph_qwen
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+  use_demo_conditioning: false  # demos now go INTO the LM sequence, not the MLP encoder
+  n_support: 4
+  support_other_task: true
+  demo_tokens_per_example: 4
+  demo_visual_groups: 4
+  demo_action_keys: 8
+  demo_hidden_mult: 4
+  vlm_native: true
+  num_action_readout: 16
+  demo_action_keys_lm: 4
+  demo_only: true
+
+backbone_lr_mult: 0.1
+batch_size: 24
+grad_accum: 3
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_richslow.yaml

@@ -0,0 +1,83 @@
+# RICH SLOW PATH: demos + robot text + instruction + image all inside the LM
+# sequence (demos in the ATTENTION STREAM), LM emits latent tokens conditioning
+# the fast path. Dual-rate PRESERVED (fast path unchanged, latent cacheable).
+# (was: in-context few-shot DEMO conditioning) — a few (obs, action) exemplar
+# pairs from the target robot's own data, encoded via the shared spatial-aware
+# vision encoder + a small MLP, WITHOUT any gradient update to adapt to that robot.
+# Composed with the text-via-Qwen morphology channel (both active). Resumes from
+# tv2_C_qwen_morph/final (LM already unfrozen, morph_out_proj already trained).
+output_dir: /home/alexw/tinyvla/outputs/tv2_richslow
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph_qwen
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+  use_demo_conditioning: false  # demos now go INTO the LM sequence, not the MLP encoder
+  n_support: 3
+  support_other_task: true
+  demo_tokens_per_example: 4
+  demo_visual_groups: 4
+  demo_action_keys: 8
+  demo_hidden_mult: 4
+  vlm_native: true
+  num_action_readout: 16
+  demo_action_keys_lm: 4
+
+backbone_lr_mult: 0.1
+batch_size: 24
+grad_accum: 3
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/configs/tv2_robocasa.yaml

@@ -0,0 +1,77 @@
+# Continue-train the best checkpoint (C-scaled, 60k) on RoboCasa365 — the first
+# genuinely NEW domain in this project: simulated kitchen scenes (everything before
+# was real-world tabletops) with PandaOmron, a Franka arm on a holonomic mobile base.
+# 7356 eps / 1.5M frames / 20fps / 3 cameras.
+#
+# RoboCasa gets half the sampling weight; the other half replays the previous robots
+# so we can tell integration from catastrophic forgetting (all old robots stay
+# measurable with the same eval). LeKiwi remains fully held out.
+output_dir: /home/alexw/tinyvla/outputs/tv2_robocasa
+wandb: null
+seed: 42
+morphology_descriptors: /home/alexw/tinyvla/configs/morphology/descriptors.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_scaled/final
+resume_step: 0
+
+datasets:
+  - repo_id: robocasa365                       # NEW domain: sim kitchens, PandaOmron
+    root: /home/alexw/tinyvla_data/robocasa365
+    weight: 0.50
+    morph_key: panda_omron
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.12
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    weight: 0.11
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    weight: 0.11
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.05
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.04
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.02
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.02
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.02
+    morph_key: hello_stretch
+  - repo_id: heldout_jaco_play
+    root: /home/alexw/tinyvla_data/heldout/heldout_jaco_play
+    weight: 0.01
+    morph_key: jaco
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: morph
+  action_space: canonical
+
+backbone_lr_mult: 0.1
+batch_size: 48
+grad_accum: 3
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 30000
+grad_clip: 10.0
+log_freq: 100
+save_freq: 10000
+mixed_precision: bf16

tinyvla2/configs/tv2_vlm_native.yaml

@@ -0,0 +1,83 @@
+# LLaVA-style: ALL modalities as tokens in one LM sequence, policy conditions on
+# latent readout tokens. Demos live IN the attention stream (fix for v1-v3 failure).
+# Trades away the dual-rate fast path.
+# (was: in-context few-shot DEMO conditioning) — a few (obs, action) exemplar
+# pairs from the target robot's own data, encoded via the shared spatial-aware
+# vision encoder + a small MLP, WITHOUT any gradient update to adapt to that robot.
+# Composed with the text-via-Qwen morphology channel (both active). Resumes from
+# tv2_C_qwen_morph/final (LM already unfrozen, morph_out_proj already trained).
+output_dir: /home/alexw/tinyvla/outputs/tv2_vlm_native
+wandb: null
+seed: 42
+robot_prompts: /home/alexw/tinyvla/configs/morphology/robot_prompts.yaml
+resume_from: /home/alexw/tinyvla/outputs/tv2_C_qwen_morph/final
+resume_step: 0
+
+datasets:
+  - root_glob: /home/alexw/tinyvla_data/so101_v3/*
+    weight: 0.25
+    morph_key: so101
+  - repo_id: MolmoAct2-Bridge-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-Bridge-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: bridge
+  - repo_id: MolmoAct2-RT-1-Dataset
+    root: /home/alexw/tinyvla_data/MolmoAct2-RT-1-Dataset
+    episodes: 15000
+    weight: 0.18
+    morph_key: rt1
+  - repo_id: div_ur5
+    root: /home/alexw/tinyvla_data/diverse/div_ur5
+    weight: 0.10
+    morph_key: ur5
+  - repo_id: div_xarm
+    root: /home/alexw/tinyvla_data/diverse/div_xarm
+    weight: 0.09
+    morph_key: xarm
+  - repo_id: div_dlr_edan
+    root: /home/alexw/tinyvla_data/diverse/div_dlr_edan
+    weight: 0.05
+    morph_key: dlr_edan
+  - repo_id: heldout_tidybot
+    root: /home/alexw/tinyvla_data/heldout/heldout_tidybot
+    weight: 0.08
+    morph_key: tidybot
+  - repo_id: heldout_cmu_stretch
+    root: /home/alexw/tinyvla_data/heldout/heldout_cmu_stretch
+    weight: 0.07
+    morph_key: hello_stretch
+
+policy:
+  chunk_size: 50
+  n_action_steps: 50
+  image_size: 256
+  freeze_lm: false
+  freeze_vision_encoder: true
+  num_embodiments: 16
+  conditioning: id  # unused in vlm_native; robot text goes into the LM sequence
+  num_morph_readout: 4
+  morph_text_max_len: 32
+  action_space: canonical
+  use_demo_conditioning: true
+  n_support: 3
+  support_other_task: true
+  demo_tokens_per_example: 4
+  demo_visual_groups: 4
+  demo_action_keys: 8
+  demo_hidden_mult: 4
+  vlm_native: true
+  num_action_readout: 16
+  demo_action_keys_lm: 4
+
+backbone_lr_mult: 0.1
+batch_size: 20
+grad_accum: 4
+num_workers: 12
+lr: 7.0e-5
+warmup_steps: 500
+steps: 20000
+grad_clip: 10.0
+log_freq: 50
+save_freq: 5000
+mixed_precision: bf16

tinyvla2/scripts/_eval_richslow_10k.py

@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+"""Auto-eval for in-context demo conditioning on held-out LeKiwi.
+
+The key test: give the model K=3 REAL (obs, action) example pairs from LeKiwi's
+OWN data at test time — NO gradient update, NO fine-tuning — and see if that beats
+zero-shot with no examples, wrong examples, or the numeric/text descriptor alone.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import torch
+import yaml
+from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
+from transformers import AutoTokenizer
+
+from tinyvla.data.canonical import CanonicalChunkStore, quantile_normalize
+from tinyvla.modeling_tinyvla import TinyVLAPolicy
+
+CKPT = "outputs/tv2_richslow/step_10000"
+DS = "lekiwi_cleanup"
+ROOT = f"/home/alexw/tinyvla_data/lekiwi/{DS}"
+K = 3
+
+
+@torch.no_grad()
+def main():
+    pol = TinyVLAPolicy.from_pretrained(CKPT).cuda().eval()
+    cfg = pol.config
+    tok = AutoTokenizer.from_pretrained(cfg.lm_model_name)
+    prompts = yaml.safe_load(open("configs/morphology/robot_prompts.yaml"))
+
+    m = LeRobotDatasetMetadata(DS, root=ROOT)
+    ds = LeRobotDataset(DS, root=ROOT, delta_timestamps={"action": [t / m.fps for t in range(50)]},
+                        video_backend="torchcodec")
+    store = CanonicalChunkStore(DS, src_fps=m.fps, chunk=50)
+    st = store.compute_stats()
+    q01, q99 = np.asarray(st["q01"]), np.asarray(st["q99"])
+    span = np.maximum(q99 - q01, 0.01 * np.median(np.abs(np.concatenate([q01, q99])) + 1e-6))
+    mid = 0.5 * (q01 + q99)
+
+    imk = sorted(k for k in ds.meta.features if k.startswith("observation.images"))
+    prim = next((k for k in imk if "front" in k or "base" in k), imk[0])
+
+    def integ(d):
+        return np.cumsum(d[:, :3], 0)
+
+    def img_at(idx, ep, ep_start):
+        it = ds[idx]
+        im = torch.nn.functional.interpolate(it[prim][None], size=(256, 256), mode="bilinear")[0]
+        raw = quantile_normalize(store.chunk_for(ep, idx - ep_start), q01, q99)
+        act = torch.from_numpy(raw[:, :7]).float()
+        act = torch.nn.functional.pad(act, (0, cfg.max_action_dim - 7))
+        return im, act
+
+    # REAL LeKiwi demo pairs from early episodes (support pool), disjoint from test
+    support_pool_eps = range(0, min(20, m.total_episodes - 15))
+    test = range(max(0, m.total_episodes - 15), m.total_episodes)
+
+    def get_real_support(k):
+        imgs, acts = [], []
+        for _ in range(k):
+            ep = int(np.random.choice(list(support_pool_eps)))
+            s = int(m.episodes["dataset_from_index"][ep])
+            e = int(m.episodes["dataset_to_index"][ep])
+            idx = int(np.random.randint(s, max(s + 1, e - 1)))
+            im, act = img_at(idx, ep, s)
+            imgs.append(im)
+            acts.append(act)
+        return torch.stack(imgs), torch.stack(acts)
+
+    def get_wrong_support(k):
+        # random noise images + random actions — a garbage support set control
+        return torch.rand(k, 3, 256, 256), torch.randn(k, 50, cfg.max_action_dim) * 0.3
+
+
+    # control: demos from a DIFFERENT ROBOT (jaco) — if the model truly extracts
+    # robot-specific dynamics from exemplars, these should be worse than LeKiwi's own
+    def get_otherrobot_support(k):
+        from lerobot.datasets.lerobot_dataset import LeRobotDataset as _LD, LeRobotDatasetMetadata as _LM
+        from tinyvla.data.canonical import CanonicalChunkStore as _CS
+        jr = "/home/alexw/tinyvla_data/heldout/heldout_jaco_play"
+        jm = _LM("heldout_jaco_play", root=jr)
+        jds = _LD("heldout_jaco_play", root=jr, delta_timestamps={"action": [t / jm.fps for t in range(50)]}, video_backend="torchcodec")
+        jstore = _CS("heldout_jaco_play", src_fps=jm.fps, chunk=50)
+        jst = jstore.compute_stats(200)
+        jq01, jq99 = np.asarray(jst["q01"]), np.asarray(jst["q99"])
+        jimk = sorted(kk for kk in jds.meta.features if kk.startswith("observation.images"))[0]
+        imgs, acts = [], []
+        for _ in range(k):
+            ep = int(np.random.randint(0, 50))
+            s = int(jm.episodes["dataset_from_index"][ep]); e = int(jm.episodes["dataset_to_index"][ep])
+            idx = int(np.random.randint(s, max(s + 1, e - 1)))
+            it = jds[idx]
+            im = torch.nn.functional.interpolate(it[jimk][None], size=(256, 256), mode="bilinear")[0]
+            raw = quantile_normalize(jstore.chunk_for(ep, idx - s), jq01, jq99)
+            act = torch.nn.functional.pad(torch.from_numpy(raw[:, :7]).float(), (0, cfg.max_action_dim - 7))
+            imgs.append(im); acts.append(act)
+        return torch.stack(imgs), torch.stack(acts)
+
+    def run(support_fn, desc_text):
+        ids_t = tok([desc_text] if desc_text else [""], padding="max_length", truncation=True,
+                    max_length=cfg.morph_text_max_len, return_tensors="pt")
+        errs = []
+        for ep in test:
+            s = int(m.episodes["dataset_from_index"][ep])
+            e = int(m.episodes["dataset_to_index"][ep])
+            for idx in range(s, e - 1, 30):
+                item = ds[idx]
+                task = item.get("task") or ""
+                t = tok([task], padding=True, truncation=True, max_length=48, return_tensors="pt")
+                img = torch.nn.functional.interpolate(item[prim][None], size=(256, 256), mode="bilinear")[0]
+                stt = torch.nn.functional.pad(item["observation.state"].float()[:6], (0, cfg.max_state_dim - 6))
+                sup_img, sup_act = support_fn(K)
+                b = {"observation.images.cam0": img[None].cuda(),
+                     "observation.images.cam1": torch.zeros_like(img)[None].cuda(),
+                     "observation.state": stt[None].cuda(),
+                     "observation.language.tokens": t["input_ids"].cuda(),
+                     "observation.language.attention_mask": t["attention_mask"].bool().cuda(),
+                     "morph_text_ids": ids_t["input_ids"].cuda(),
+                     "morph_text_mask": ids_t["attention_mask"].bool().cuda(),
+                     "support_images": sup_img[None].cuda(), "support_actions": sup_act[None].cuda(),
+                     "embodiment_id": torch.tensor([0]).cuda()}
+                with torch.autocast("cuda", torch.bfloat16):
+                    pr = pol.predict_action_chunk(b)[0].cpu().float().numpy()
+                gu = quantile_normalize(store.chunk_for(ep, idx - s), q01, q99)[:, :7] * span / 2 + mid
+                pu = pr[:, :7] * span / 2 + mid
+                errs.append(np.linalg.norm(integ(pu)[-1] - integ(gu)[-1]) * 1000)
+        return np.mean(errs)
+
+    print("=== RICH-SLOW (demos in LM attention stream, no FT) on held-out LeKiwi ===")
+    print(f"REAL LeKiwi demos (K={K}) + desc=lekiwi   endpoint {run(get_real_support, prompts['lekiwi']):.1f}mm")
+    print(f"REAL LeKiwi demos (K={K}) + desc=none     endpoint {run(get_real_support, None):.1f}mm")
+    print(f"WRONG/garbage demos      + desc=lekiwi   endpoint {run(get_wrong_support, prompts['lekiwi']):.1f}mm")
+    print(f"WRONG/garbage demos      + desc=none     endpoint {run(get_wrong_support, None):.1f}mm")
+    print(f"OTHER-ROBOT (jaco) demos + desc=none     endpoint {run(get_otherrobot_support, None):.1f}mm")
+    print("\nprior baselines: C-diverse 295mm | C-qwen-morph desc=none 294mm | v1 demo(same-task) real=290.6 garbage=288.5 (dead)")
+
+
+if __name__ == "__main__":
+    main()

tinyvla2/scripts/audit_so101_fk.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python
+"""Gate 1: per-dataset SO101 FK calibration audit.
+
+Same recorded degrees can map to different physical poses across lerobot
+zero-conventions → FK garbage that would poison B/C canonical labels while
+leaving A untouched (a confound AGAINST the hypothesis). Drop datasets whose
+FK produces non-physical EE trajectories.
+
+Checks per dataset (sampled frames across episodes):
+  - reach in [0.02, 0.40] m (SO101 max reach ~0.35)
+  - z above a floor (> -0.20 m; base frame origin at arm mount)
+  - trajectory smoothness (median consecutive EE step < 0.05 m at 30fps)
+  - FK(state) vs FK(action-target) diffs correlate (both go through FK, deltas
+    should track since action = commanded target of the same arm)
+
+Writes ~/tinyvla_data/so101_fk_audit.json with per-dataset verdict.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+
+DATA_ROOT = Path.home() / "tinyvla_data" / "so101_v3"
+OUT = Path.home() / "tinyvla_data" / "so101_fk_audit.json"
+
+
+def audit_one(name, root, fk, n_frames=200):
+    from lerobot.datasets.lerobot_dataset import LeRobotDataset
+
+    ds = LeRobotDataset(name, root=root)
+    anames = ds.meta.features["action"]["shape"][0]
+    if anames != 6:
+        return {"verdict": "SKIP", "reason": f"action dim {anames} != 6"}
+    # smoothness needs CONSECUTIVE frames → sample a few contiguous windows;
+    # reach/corr can use sparse knots across the whole dataset
+    n = len(ds)
+    eps = ds.meta.episodes
+    sparse = np.linspace(0, n - 1, min(n_frames, n)).astype(int)
+    pos_state, pos_act = [], []
+    for i in sparse:
+        item = ds[int(i)]
+        pos_state.append(fk.ee_pose(item["observation.state"].numpy())[:3, 3])
+        pos_act.append(fk.ee_pose(item["action"].numpy())[:3, 3])
+    ps = np.array(pos_state)
+    pa = np.array(pos_act)
+    reach = np.linalg.norm(ps, axis=1)
+
+    # consecutive-frame EE steps within the first episode (real per-frame motion)
+    e0, e1 = int(eps["dataset_from_index"][0]), int(eps["dataset_to_index"][0])
+    consec = []
+    prev = None
+    for i in range(e0, min(e1, e0 + 150)):
+        p = fk.ee_pose(ds[i]["observation.state"].numpy())[:3, 3]
+        if prev is not None:
+            consec.append(np.linalg.norm(p - prev))
+        prev = p
+    steps = np.array(consec) if consec else np.array([0.0])
+    # correlation of state-vs-target displacement over sampled knots
+    d_state = np.diff(ps, axis=0).flatten()
+    d_act = np.diff(pa, axis=0).flatten()
+    corr = float(np.corrcoef(d_state, d_act)[0, 1]) if d_state.std() > 1e-9 else 0.0
+
+    ok_reach = bool(0.02 < reach.mean() < 0.40 and reach.max() < 0.50)
+    ok_z = bool(ps[:, 2].min() > -0.20)
+    ok_smooth = bool(np.median(steps) < 0.06)
+    ok_corr = bool(corr > 0.5)
+    verdict = "KEEP" if (ok_reach and ok_z and ok_smooth and ok_corr) else "DROP"
+    return {
+        "verdict": verdict,
+        "reach_mean": round(float(reach.mean()), 3),
+        "reach_max": round(float(reach.max()), 3),
+        "z_min": round(float(ps[:, 2].min()), 3),
+        "step_median": round(float(np.median(steps)), 4),
+        "corr_state_target": round(corr, 3),
+        "flags": {"reach": ok_reach, "z": ok_z, "smooth": ok_smooth, "corr": ok_corr},
+    }
+
+
+def main():
+    from tinyvla.data.kinematics_so101 import SO101FK
+
+    fk = SO101FK()
+    results = {}
+    roots = sorted(DATA_ROOT.iterdir())
+    for r in roots:
+        if not (r / "meta" / "info.json").exists():
+            continue
+        try:
+            res = audit_one(r.name, r, fk)
+        except Exception as e:
+            res = {"verdict": "ERROR", "reason": f"{type(e).__name__}: {str(e)[:120]}"}
+        results[r.name] = res
+        print(f"{res['verdict']:6} {r.name}: {res}")
+    OUT.write_text(json.dumps(results, indent=1))
+    keep = sum(1 for v in results.values() if v["verdict"] == "KEEP")
+    print(f"\nKEEP {keep}/{len(results)} -> {OUT}")
+
+
+if __name__ == "__main__":
+    main()

tinyvla2/scripts/build_ee_sidecar.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python
+"""Build per-frame canonical EE-pose sidecars (offline, once).
+
+For each frame stores the absolute end-effector pose implied by BOTH the
+observation state and the action target, in the robot base frame:
+  cols: episode_index, frame_index,
+        s_px s_py s_pz s_qx s_qy s_qz s_qw   (state EE pose)
+        a_px a_py a_pz a_qx a_qy a_qz a_qw   (action-target EE pose)
+        grip                                  (raw gripper channel, for q1-q99 later)
+
+Canonical chunk deltas are assembled at train time from these poses (anchor,
+frequency, rotation convention are all decided there — this file is convention-free).
+
+Source kinds:
+  so101  : FK(joint_deg) for state and action (both are joint vectors).
+  ee     : state/action already EE; reconstruct absolute pose. Bridge/LIBERO/DROID
+           store EE pose in observation.state; action is a delta → target = state ⊕ action.
+           (implemented per-source as adapters are verified.)
+
+Usage:
+  python scripts/build_ee_sidecar.py --kind so101 --glob '~/tinyvla_data/so101_v3/*'
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.parquet as pq
+
+OUT_DIR = Path.home() / "tinyvla_data" / "ee_sidecar"
+
+
+def _read_cols(name, root):
+    """State/action/ep/frame columns WITHOUT video decode (raw parquet)."""
+    from lerobot.datasets.lerobot_dataset import LeRobotDataset
+
+    ds = LeRobotDataset(name, root=root)
+    hf = ds.reader.hf_dataset.with_format("numpy")
+    return (
+        np.asarray(hf["observation.state"], dtype=np.float64),
+        np.asarray(hf["action"], dtype=np.float64),
+        np.asarray(hf["episode_index"]).astype(int),
+        np.asarray(hf["frame_index"]).astype(int),
+    )
+
+
+def build_so101(name, root, fk):
+    from scipy.spatial.transform import Rotation
+
+    state, action, ep, fr = _read_cols(name, root)
+    n = len(state)
+    cols = ["episode_index", "frame_index",
+            "s_px", "s_py", "s_pz", "s_qx", "s_qy", "s_qz", "s_qw",
+            "a_px", "a_py", "a_pz", "a_qx", "a_qy", "a_qz", "a_qw", "grip"]
+    buf = {k: np.empty(n, dtype=np.float64) for k in cols}
+    buf["episode_index"] = ep.astype(np.float64)
+    buf["frame_index"] = fr.astype(np.float64)
+    for i in range(n):
+        Ts = fk.ee_pose(state[i])
+        Ta = fk.ee_pose(action[i])
+        sp, sq = Ts[:3, 3], Rotation.from_matrix(Ts[:3, :3]).as_quat()
+        ap, aq = Ta[:3, 3], Rotation.from_matrix(Ta[:3, :3]).as_quat()
+        for j, k in enumerate(("s_px", "s_py", "s_pz")): buf[k][i] = sp[j]
+        for j, k in enumerate(("s_qx", "s_qy", "s_qz", "s_qw")): buf[k][i] = sq[j]
+        for j, k in enumerate(("a_px", "a_py", "a_pz")): buf[k][i] = ap[j]
+        for j, k in enumerate(("a_qx", "a_qy", "a_qz", "a_qw")): buf[k][i] = aq[j]
+        buf["grip"][i] = action[i, 5]
+    return pa.table(buf)
+
+
+def build_ee_from_state(name, root):
+    """EE-native sources (Bridge, RT-1): state carries absolute EE pose.
+
+    Bridge state = [x y z roll pitch yaw pad gripper]; RT-1 state =
+    [x y z rx ry rz rw gripper] (quat). We store the STATE pose as canonical
+    absolute pose (both s_* and a_* set to state pose; targets reconstructed at
+    train time as state ⊕ action-delta if needed, but state-derived deltas are
+    the uniform choice per plan → a_* == next-frame not needed here).
+    """
+    from lerobot.datasets.lerobot_dataset import LeRobotDataset
+    from scipy.spatial.transform import Rotation
+
+    ds = LeRobotDataset(name, root=root)
+    snames = ds.meta.features["observation.state"].get("names", {})
+    flat = snames.get("motors", snames) if isinstance(snames, dict) else snames
+    state, _, ep, fr = _read_cols(name, root)
+    n = len(state)
+    # RoboCasa/PandaOmron: state = base_pos(3) base_quat(4) ee_pos(3) ee_quat(4) grip(2)
+    # -> the EE block is not at the front, slice it out first
+    if state.shape[1] == 16:
+        state = np.concatenate([state[:, 7:14], state[:, 14:15]], axis=1)  # pos,quat,grip
+    # detect quaternion: named "rw", or 8-dim state whose dims 3:7 are unit-norm
+    is_quat = ("rw" in flat) or (
+        state.shape[1] >= 8 and abs(np.linalg.norm(state[:200, 3:7], axis=1).mean() - 1.0) < 1e-2
+    )
+    cols = ["episode_index", "frame_index",
+            "s_px", "s_py", "s_pz", "s_qx", "s_qy", "s_qz", "s_qw",
+            "a_px", "a_py", "a_pz", "a_qx", "a_qy", "a_qz", "a_qw", "grip"]
+    buf = {k: np.empty(n, dtype=np.float64) for k in cols}
+    buf["episode_index"] = ep.astype(np.float64)
+    buf["frame_index"] = fr.astype(np.float64)
+    pos = state[:, :3]
+    if is_quat:  # [x y z rx ry rz rw gripper]
+        quat = state[:, 3:7]
+        grip = state[:, 7]
+    else:  # [x y z roll pitch yaw (pad) (gripper)]
+        quat = Rotation.from_euler("xyz", state[:, 3:6]).as_quat()
+        grip = state[:, 7] if state.shape[1] > 7 else np.zeros(n)  # free-body (UAV): no gripper
+    for j, k in enumerate(("s_px", "s_py", "s_pz")): buf[k] = pos[:, j]
+    for j, k in enumerate(("s_qx", "s_qy", "s_qz", "s_qw")): buf[k] = quat[:, j]
+    for j, k in enumerate(("a_px", "a_py", "a_pz")): buf[k] = pos[:, j]
+    for j, k in enumerate(("a_qx", "a_qy", "a_qz", "a_qw")): buf[k] = quat[:, j]
+    buf["grip"] = grip
+    return pa.table(buf)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--kind", choices=["so101", "ee"], required=True)
+    parser.add_argument("--glob", required=True)
+    parser.add_argument("--audit", default=str(Path.home() / "tinyvla_data/so101_fk_audit.json"),
+                        help="skip datasets not marked KEEP")
+    args = parser.parse_args()
+
+    import glob as _glob
+    import json
+
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    keep = None
+    audit_path = Path(args.audit).expanduser()
+    if args.kind == "so101" and audit_path.exists():
+        audit = json.loads(audit_path.read_text())
+        keep = {k for k, v in audit.items() if v.get("verdict") == "KEEP"}
+
+    fk = None
+    if args.kind == "so101":
+        from tinyvla.data.kinematics_so101 import SO101FK
+        fk = SO101FK()
+
+    roots = sorted(Path(p) for p in _glob.glob(str(Path(args.glob).expanduser())))
+    done = 0
+    for r in roots:
+        if not (r / "meta" / "info.json").exists():
+            continue
+        if keep is not None and r.name not in keep:
+            print(f"skip {r.name} (not KEEP)")
+            continue
+        out = OUT_DIR / f"{r.name}.parquet"
+        if out.exists():
+            done += 1
+            continue
+        try:
+            if args.kind == "so101":
+                tbl = build_so101(r.name, r, fk)
+            else:
+                tbl = build_ee_from_state(r.name, r)
+            pq.write_table(tbl, out)
+            done += 1
+            print(f"[{done}] {r.name}: {tbl.num_rows} frames -> {out}")
+        except Exception as e:
+            print(f"FAIL {r.name}: {type(e).__name__}: {str(e)[:150]}")
+    print(f"done {done}")
+
+
+if __name__ == "__main__":
+    main()

tinyvla2/scripts/diag_obs_gap.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python
+"""Diagnose the env-obs adapter: compare policy predictions from env-rendered
+observations vs dataset observations at the SAME init state.
+
+If pred(dataset obs) is close to GT but pred(env obs) differs, the observation
+adapter (image orientation/cameras/state) is the remaining gap.
+"""
+
+from __future__ import annotations
+
+import argparse
+
+import numpy as np
+import torch
+from scipy.spatial.transform import Rotation
+
+
+@torch.no_grad()
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--checkpoint", default="outputs/libero_ft2/step_15000")
+    parser.add_argument("--suite", default="libero_spatial")
+    parser.add_argument("--tasks", type=int, default=3)
+    args = parser.parse_args()
+
+    from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
+    from lerobot.envs.factory import make_env, make_env_config
+    from transformers import AutoTokenizer
+    from tinyvla.data.mixture import CanonicalSource
+    from tinyvla.modeling_tinyvla import TinyVLAPolicy
+
+    policy = TinyVLAPolicy.from_pretrained(args.checkpoint).cuda().eval()
+    cfg = policy.config
+    tok = AutoTokenizer.from_pretrained(cfg.lm_model_name)
+
+    meta = LeRobotDatasetMetadata("HuggingFaceVLA/libero")
+    ds = LeRobotDataset(
+        "HuggingFaceVLA/libero",
+        delta_timestamps={"action": [t / meta.fps for t in range(cfg.chunk_size)]},
+        video_backend="torchcodec",
+    )
+    src = CanonicalSource(ds, 2, cfg.image_size, cfg.max_state_dim, cfg.max_action_dim)
+    s_stats = meta.stats["observation.state"]
+    s_mean = torch.as_tensor(s_stats["mean"]).flatten().float()
+    s_std = torch.as_tensor(s_stats["std"]).flatten().float().clamp(min=1e-6)
+
+    env_cfg = make_env_config("libero", task=args.suite)
+    task_envs = make_env(env_cfg, n_envs=1)[args.suite]
+    env_by_task = {}
+    for tid, env in task_envs.items():
+        desc = env.get_attr("task_description")[0]
+        env_by_task[desc.strip().lower()] = (tid, env)
+
+    eps_meta = ds.meta.episodes
+    first_ep_by_task = {}
+    for ep in range(ds.num_episodes):
+        start = int(eps_meta["dataset_from_index"][ep])
+        task = ds[start]["task"].strip().lower()
+        if task in env_by_task and task not in first_ep_by_task:
+            first_ep_by_task[task] = ep
+
+    def tok_batch(task_text):
+        t = tok([task_text], padding=True, truncation=True,
+                max_length=cfg.tokenizer_max_length, return_tensors="pt")
+        return t["input_ids"].cuda(), t["attention_mask"].bool().cuda()
+
+    def env_to_batch(obs, task_text):
+        imgs = {}
+        for slot, key in (("cam0", "image"), ("cam1", "image2")):
+            x = torch.as_tensor(np.asarray(obs["pixels"][key]))[0].flip(0).flip(1)
+            x = x.permute(2, 0, 1).float() / 255.0
+            x = torch.nn.functional.interpolate(x[None], size=(cfg.image_size, cfg.image_size),
+                                                mode="bilinear", align_corners=False)[0]
+            imgs[slot] = x
+        rs = obs["robot_state"]
+        pos = np.asarray(rs["eef"]["pos"]).flatten()
+        quat = np.asarray(rs["eef"]["quat"]).flatten()
+        rotvec = Rotation.from_quat(quat).as_rotvec()
+        if rotvec[0] < 0:
+            th = np.linalg.norm(rotvec)
+            rotvec = rotvec * (th - 2 * np.pi) / th
+        grip = np.asarray(rs["gripper"]["qpos"]).flatten()
+        state = torch.tensor(np.concatenate([pos, rotvec, grip]), dtype=torch.float32)
+        state = (state - s_mean) / s_std
+        state = torch.nn.functional.pad(state, (0, cfg.max_state_dim - state.shape[-1]))
+        ids, mask = tok_batch(task_text)
+        return {
+            "observation.images.cam0": imgs["cam0"][None].cuda(),
+            "observation.images.cam1": imgs["cam1"][None].cuda(),
+            "observation.state": state[None].cuda(),
+            "observation.language.tokens": ids,
+            "observation.language.attention_mask": mask,
+            "embodiment_id": torch.tensor([2], device="cuda"),
+        }
+
+    def ds_to_batch(item):
+        ids, mask = tok_batch(item.pop("task"))
+        b = {k: v[None].cuda() for k, v in item.items() if torch.is_tensor(v)}
+        b["observation.language.tokens"] = ids
+        b["observation.language.attention_mask"] = mask
+        return b
+
+    for task, ep in list(first_ep_by_task.items())[: args.tasks]:
+        tid, env = env_by_task[task]
+        obs, _ = env.reset(seed=0)
+        start = int(eps_meta["dataset_from_index"][ep])
+        item = src[start]
+        gt = item["action"].clone()[None].cuda()
+
+        env_b = env_to_batch(obs, task)
+        ds_b = ds_to_batch(dict(item))
+
+        torch.manual_seed(0)
+        pred_env = policy.predict_action_chunk(env_b)
+        torch.manual_seed(0)
+        pred_ds = policy.predict_action_chunk(ds_b)
+
+        m = item["action_dim_mask"]
+        d_env_gt = ((pred_env[0, :, m].cpu() - gt[0, :, m].cpu()) ** 2).mean().sqrt().item()
+        d_ds_gt = ((pred_ds[0, :, m].cpu() - gt[0, :, m].cpu()) ** 2).mean().sqrt().item()
+        d_env_ds = ((pred_env[0, :, m] - pred_ds[0, :, m]) ** 2).mean().sqrt().item()
+
+        # also compare the raw inputs
+        img_gap = (env_b["observation.images.cam0"] - ds_b["observation.images.cam0"]).abs().mean().item()
+        st_gap = (env_b["observation.state"] - ds_b["observation.state"]).abs().max().item()
+        print(f"task {tid}: RMSE(pred_env, GT)={d_env_gt:.3f} | RMSE(pred_ds, GT)={d_ds_gt:.3f} "
+              f"| RMSE(pred_env, pred_ds)={d_env_ds:.3f} | img L1 gap={img_gap:.4f} | state max gap={st_gap:.2f}")
+
+
+if __name__ == "__main__":
+    main()

tinyvla2/scripts/download_subsets.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python
+"""Download Stage-2 dataset subsets under a hard disk budget.
+
+Usage:
+    python scripts/download_subsets.py --root ~/tinyvla_data [--dry-run]
+
+Downloads metadata first, measures actual on-disk size of a small episode
+sample, then extrapolates before committing to the full subset download.
+"""
+
+from __future__ import annotations
+
+import argparse
+import shutil
+from pathlib import Path
+
+DISK_BUDGET_GB = 400
+
+# (repo_id, num_episodes or None for all)
+SUBSETS = [
+    ("HuggingFaceVLA/community_dataset_v1", None),
+    ("nvidia/BridgeData2_LeRobot_v3", 10_000),
+    ("IPEC-COMMUNITY/fractal20220817_data_lerobot", 8_000),
+]
+
+
+def free_gb(path: Path) -> float:
+    return shutil.disk_usage(path).free / 1e9
+
+
+def used_gb(path: Path) -> float:
+    return sum(f.stat().st_size for f in path.rglob("*") if f.is_file()) / 1e9
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--root", type=Path, default=Path.home() / "tinyvla_data")
+    parser.add_argument("--dry-run", action="store_true")
+    parser.add_argument("--only", type=str, default=None, help="substring filter on repo_id")
+    args = parser.parse_args()
+
+    args.root.mkdir(parents=True, exist_ok=True)
+
+    from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
+
+    for repo_id, n_eps in SUBSETS:
+        if args.only and args.only not in repo_id:
+            continue
+        print(f"\n=== {repo_id} (episodes: {n_eps or 'all'}) ===")
+        meta = LeRobotDatasetMetadata(repo_id)
+        total_eps = meta.total_episodes
+        print(f"total episodes: {total_eps}, fps: {meta.fps}")
+
+        episodes = list(range(min(n_eps, total_eps))) if n_eps else None
+
+        # probe with 1% to estimate size
+        probe_n = max(10, (len(episodes) if episodes else total_eps) // 100)
+        probe_dir = args.root / "_probe" / repo_id.replace("/", "__")
+        if not args.dry_run and not probe_dir.exists():
+            LeRobotDataset(repo_id, root=probe_dir, episodes=list(range(probe_n)))
+            probe_gb = used_gb(probe_dir)
+            est_gb = probe_gb / probe_n * (len(episodes) if episodes else total_eps)
+            print(f"probe: {probe_n} eps = {probe_gb:.2f}GB -> estimated full subset {est_gb:.0f}GB")
+            if est_gb > free_gb(args.root) - 100 or used_gb(args.root) + est_gb > DISK_BUDGET_GB:
+                print(f"SKIP {repo_id}: would exceed budget ({DISK_BUDGET_GB}GB) or disk")
+                continue
+
+        if args.dry_run:
+            continue
+
+        target = args.root / repo_id.replace("/", "__")
+        LeRobotDataset(repo_id, root=target, episodes=episodes)
+        print(f"downloaded {repo_id}: {used_gb(target):.1f}GB, total used {used_gb(args.root):.1f}GB")
+
+
+if __name__ == "__main__":
+    main()

tinyvla2/scripts/eval_canonical.py

@@ -0,0 +1,186 @@
+#!/usr/bin/env python
+"""Physical-space canonical-action eval (variants A/B/C, TinyVLA-2 headline metric).
+
+Compares A/B/C in ONE fair space: canonical base-frame EE deltas, unnormalized to
+physical units — position error (mm), rotation error (deg), gripper error ([0,1]).
+
+- B/C predict canonical directly → unnormalize with dataset canonical stats.
+- A predicts NATIVE actions → for SO101 map joint predictions through FK to EE deltas;
+  for EE-native sources A's native deltas are already comparable (convention aside).
+- Held-out embodiment: A/B have no ID row → oracle over trained IDs (best), making any
+  C win conservative. C uses the written descriptor (zero-shot by construction).
+
+Usage:
+  python scripts/eval_canonical.py --checkpoint outputs/tv2_C_morph_canon/final \
+      --dataset heldout_jaco_play --root ~/tinyvla_data/heldout/heldout_jaco_play \
+      --morph-key jaco --episodes 20
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+import torch
+
+
+@torch.no_grad()
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--checkpoint", type=Path, required=True)
+    ap.add_argument("--dataset", required=True)
+    ap.add_argument("--root", required=True)
+    ap.add_argument("--morph-key", default=None, help="descriptor key for variant C zero-shot")
+    ap.add_argument("--oracle-ids", type=int, default=8, help="A/B: try IDs 0..N-1, report best")
+    ap.add_argument("--episodes", type=int, default=20)
+    ap.add_argument("--ep-start", type=int, default=0, help="first episode index (few-shot: eval on held-out test split after FT episodes)")
+    ap.add_argument("--stride", type=int, default=30)
+    args = ap.parse_args()
+
+    import yaml
+    from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
+    from scipy.spatial.transform import Rotation
+    from transformers import AutoTokenizer
+
+    from tinyvla.data.canonical import CanonicalChunkStore, quantile_normalize
+    from tinyvla.modeling_tinyvla import TinyVLAPolicy
+    from tinyvla.modules.embodiment import MORPH_FIELDS
+
+    policy = TinyVLAPolicy.from_pretrained(args.checkpoint).cuda().eval()
+    cfg = policy.config
+    tok = AutoTokenizer.from_pretrained(cfg.lm_model_name)
+    chunk = cfg.chunk_size
+
+    meta = LeRobotDatasetMetadata(args.dataset, root=args.root)
+    ds = LeRobotDataset(args.dataset, root=args.root,
+                        delta_timestamps={"action": [t / meta.fps for t in range(chunk)]},
+                        video_backend="torchcodec")
+    store = CanonicalChunkStore(args.dataset, src_fps=ds.fps, chunk=chunk)
+    stats = store.compute_stats()
+    q01, q99 = np.asarray(stats["q01"]), np.asarray(stats["q99"])
+
+    # morphology descriptor (variant C)
+    morph = None
+    if cfg.conditioning == "morph" and args.morph_key:
+        raw = yaml.safe_load(open("configs/morphology/descriptors.yaml"))[args.morph_key]
+        _sc = {"arm_dof": 0.1, "reach_m": 2, "gripper_width_m": 10, "num_cameras": 1 / 3,
+               "control_hz": 1 / 30, "joint_lo_mean": 1 / 3.1416, "joint_hi_mean": 1 / 3.1416,
+               "workspace_x": 2, "workspace_y": 2, "workspace_z": 2, "payload_kg": 0.2}
+        morph = torch.tensor([raw.get(f, 0) * _sc.get(f, 1) for f in MORPH_FIELDS],
+                             dtype=torch.float32)
+
+    image_keys = sorted(k for k in ds.meta.features if k.startswith("observation.images"))
+
+    def to_batch(item, emb_id, morph_vec):
+        import re
+        from tinyvla.data.mixture import PRIMARY_RE, WRIST_RE
+        prim = next((k for k in image_keys if PRIMARY_RE.search(k)), image_keys[0])
+        wr = next((k for k in image_keys if k != prim and WRIST_RE.search(k)), None)
+
+        def img(x):
+            if x.shape[-2:] != (cfg.image_size, cfg.image_size):
+                x = torch.nn.functional.interpolate(x[None], size=(cfg.image_size, cfg.image_size),
+                                                    mode="bilinear", align_corners=False)[0]
+            return x
+        cam0 = img(item[prim])
+        cam1 = img(item[wr]) if wr else torch.zeros_like(cam0)
+        state = item["observation.state"].float()
+        state = torch.nn.functional.pad(state, (0, cfg.max_state_dim - state.shape[-1]))
+        t = tok([item.get("task") or ""], padding=True, truncation=True,
+                max_length=cfg.tokenizer_max_length, return_tensors="pt")
+        b = {"observation.images.cam0": cam0[None].cuda(),
+             "observation.images.cam1": cam1[None].cuda(),
+             "observation.state": state[None].cuda(),
+             "observation.language.tokens": t["input_ids"].cuda(),
+             "observation.language.attention_mask": t["attention_mask"].bool().cuda(),
+             "embodiment_id": torch.tensor([emb_id], device="cuda")}
+        if morph_vec is not None:
+            b["morphology"] = morph_vec[None].cuda()
+        return b
+
+    # native action stats (for variant A unnormalization) — physical EE deltas
+    native_stats = ds.meta.stats.get("action", {})
+    nat_mean = np.asarray(native_stats.get("mean", np.zeros(7)))
+    nat_std = np.asarray(native_stats.get("std", np.ones(7)))
+    is_native = cfg.action_space == "native"
+
+    def _unnorm_canon(x):
+        span = np.maximum(q99 - q01, 0.01 * np.median(np.abs(np.concatenate([q01, q99])) + 1e-6))
+        mid = 0.5 * (q01 + q99)
+        return x[:, :7] * span / 2 + mid
+
+    def _integrate(deltas):
+        """(T,6+) physical per-step EE deltas -> (T,3) cumulative positions,
+        list of cumulative rotations. Amplifies per-step differences into a
+        trajectory with real dynamic range (per-step motion is tiny)."""
+        pos = np.cumsum(deltas[:, :3], axis=0)
+        R = Rotation.identity()
+        rots = []
+        for k in range(len(deltas)):
+            R = Rotation.from_rotvec(deltas[k, 3:6]) * R
+            rots.append(R)
+        return pos, rots
+
+    def canonical_phys_err(pred_norm, gt_canon):
+        """Integrated-trajectory error (the per-step delta floor ~3.6mm has no
+        dynamic range — see control). Returns:
+          ep_mm  : endpoint position error over the 5s chunk (mm)
+          path_mm: mean cumulative-position error along the chunk (mm)
+          ep_deg : endpoint cumulative-rotation error (deg)
+        """
+        gu = _unnorm_canon(gt_canon)
+        if is_native:
+            d = pred_norm.shape[-1]
+            pu = pred_norm[:, : min(d, 7)] * nat_std[: min(d, 7)] + nat_mean[: min(d, 7)]
+            if pu.shape[-1] < 7:
+                pu = np.concatenate([pu, gu[:, pu.shape[-1]:7]], axis=1)
+        else:
+            pu = _unnorm_canon(pred_norm)
+        gp, gr = _integrate(gu)
+        pp, pr = _integrate(pu)
+        ep_mm = np.linalg.norm(pp[-1] - gp[-1]) * 1000
+        path_mm = np.linalg.norm(pp - gp, axis=1).mean() * 1000
+        ep_deg = np.degrees((pr[-1] * gr[-1].inv()).magnitude())
+        return ep_mm, path_mm, ep_deg
+        pos_mm = np.linalg.norm(pu[:, :3] - gu[:, :3], axis=1).mean() * 1000
+        # rotation error: geodesic between rotvec deltas
+        rp = Rotation.from_rotvec(pu[:, 3:6])
+        rg = Rotation.from_rotvec(gu[:, 3:6])
+        rot_deg = np.degrees((rp * rg.inv()).magnitude()).mean()
+        grip = np.abs(pu[:, 6] - gu[:, 6]).mean()
+        return pos_mm, rot_deg, grip
+
+    eps = list(range(args.ep_start, min(args.ep_start + args.episodes, ds.num_episodes)))
+    # candidate embodiment conditionings
+    if cfg.conditioning == "morph":
+        candidates = [("morph", morph)]
+    else:
+        candidates = [(f"id{i}", i) for i in range(args.oracle_ids)]
+
+    best = None
+    for label, cand in candidates:
+        errs = []
+        for ep in eps:
+            start = int(ds.meta.episodes["dataset_from_index"][ep])
+            end = int(ds.meta.episodes["dataset_to_index"][ep])
+            for idx in range(start, end - 1, args.stride):
+                gt = quantile_normalize(store.chunk_for(ep, idx - start), q01, q99)
+                if cfg.conditioning == "morph":
+                    b = to_batch(ds[idx], 0, cand)
+                else:
+                    b = to_batch(ds[idx], cand, None)
+                pred = policy.predict_action_chunk(b)[0].cpu().numpy()
+                errs.append(canonical_phys_err(pred, gt))
+        errs = np.array(errs)
+        m = errs.mean(0)
+        if best is None or m[0] < best[1][0]:
+            best = (label, m)
+        print(f"  {label}: endpoint {m[0]:.1f}mm  path {m[1]:.1f}mm  rot {m[2]:.1f}deg")
+
+    print(f"\n=== {args.dataset} | {args.checkpoint.name} | cond={cfg.conditioning} ===")
+    print(f"BEST ({best[0]}): endpoint {best[1][0]:.1f}mm  path {best[1][1]:.1f}mm  rot {best[1][2]:.1f}deg")
+
+
+if __name__ == "__main__":
+    main()