README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
tags:
  - robotics
  - vla
  - lerobot
  - libero
  - dual-rate
pipeline_tag: robotics
---

# TinyVLA: a dual-rate VLA that beats OpenVLA-7B on LIBERO-Spatial at 0.64B params

**Headline result: 86% success on LIBERO-Spatial (n=100) in dual-rate mode (semantic latent
refreshed at 1 Hz), 79% full-rate — vs OpenVLA-7B's 84.7% — with 140M parameters on the
online control path.** Built and trained end-to-end in ~5 days on a single RTX A6000.

## 1. Architecture

```
                SLOW / SMART path (0.5–2 Hz, cached)
instruction ─────────────────────┐
camera → Qwen3.5-0.8B vision tower → Qwen3.5 language stack (first 12 of 24 layers)
                  │                       │
                  │            8 readout tokens → linear → semantic latent (8×512)
                  │                       │ cached between refreshes
──────────────────┼───────────────────────┼──────────────────────────
                  │       FAST / MOTOR path (10–20 Hz, ~140M online)
                  └── 64 spatial tokens per camera ─┐
proprio (padded 16d) ───────────────────────────────┤
embodiment-id embedding ────────────────────────────┤
                                                    ▼
                    flow-matching action expert (38M, 12 blocks,
                    interleaved cross-/self-attention, AdaLN-free)
                                                    ▼
                              50-step action chunk (padded 8d + validity mask)
```

- **Single shared encoder**: the Qwen3.5-0.8B native ViT (100.6M) feeds both paths.
- **Slow path**: image + instruction + 8 trailing pad-token "readout" slots through the
  truncated language stack (L/2 layer-skip a la SmolVLA); their hidden states → latent 8×512.
- **Fast path**: raw vision-tower spatial tokens (bypassing the LM) + proprio + embodiment
  token, cross-attended by the flow expert. 10 Euler steps at inference.
- **Flow matching**: linear-OT path, t ~ Beta(1.5, 1), velocity target, masked MSE.
- Totals: 643M stored / 39.5M–575M trainable depending on stage / ~140M online per tick.
- Implemented as an **out-of-tree LeRobot policy plugin**
  (`--policy.type=tinyvla --policy.discover_packages_path=tinyvla`, lerobot 0.6.1).

## 2. Training recipe

| Stage | Data | Steps | Trainable | Result |
|---|---|---|---|---|
| 1. Overfit smoke | svla_so100_stacking (56 eps) | 5k | expert+proj (40M) | loss 1.885→0.099 |
| 2. Mixture pretrain | 40 SO100/101 community datasets (~10.7K eps, weight 0.65, per-dataset proportional) + Bridge V2 10K eps (0.35) | 40k, bs 64×2, lr 1e-4 | 39.5M (backbone frozen) | held-out RMSE@t0 0.30σ; **stale-latent 2s == fresh** |
| 2b. Staleness aug | same | from step 20k | — | latent from frame t−k, k~U(0,2s), p=0.5 |
| 3. Spatial distillation | + 132K teacher labels (Qwen3.5-4B, center-point → 32×32 grid), CE ×0.15 on cam0 tokens | 20k, lr 5e-5 | same + head | head top-1 20.8%, top-5 44.8% (random 0.1%) |
| 4. LIBERO FT (ft4) | HuggingFaceVLA/libero (1693 eps, all suites) | 40k, bs 48, lr 1e-4 / backbone 0.1× | **575M** (LM layers + vision unfrozen; vocab embeddings frozen) | loss 0.066 |

Ablation FTs: ft1 (all frozen) → 0% closed-loop; ft2 (vision unfrozen) → 60%;
ft3 (= ft2 from distilled ckpt) → 61%; **ft4 (LM+vision unfrozen) → 79/86%**.
Unfreezing the LM was the single largest gain; distillation was neutral on LIBERO
(likely overwritten by vision unfreezing during FT — bake it into pretraining next time).

Cross-embodiment handling: canonical schema (cam0=scene/cam1=wrist via regex mapping,
per-dataset mean/std normalization, action pad-to-8 + validity mask, state pad-to-16,
learned embodiment-id token). Weighted mixture sampler with per-source retry on corrupt
frames (IndexError/KeyError/decode RuntimeError).

## 3. Results

### LIBERO — all four suites (dual-rate, latent @ 1 Hz)

| Suite | TinyVLA | OpenVLA-7B (published) |
|---|---|---|
| Spatial | **86%** (n=100; full-rate 79%) | 84.7% |
| Object | **94%** (n=50) | 88.4% |
| Goal | **86%** (n=50) | 79.2% |
| Long (libero_10) | **70%** (n=50) | 53.7% |
| **Average** | **84.0%** | 76.5% |

Beats OpenVLA-7B on every suite at 1/11 the parameters and ~1/50 the online compute.
XS-VLA (90.3 avg) remains ahead — their latent flow matching is future work for us.

### LIBERO-Spatial (100 episodes, 10 per task, init states 0-9)

| Mode | Success |
|---|---|
| Full-rate (latent every chunk) | 79/100 = **79%** |
| **Dual-rate (latent @ 1 Hz)** | 86/100 = **86%** |

Dual-rate ≥ full-rate consistently across all evals (also 65% vs 58% at ft3, n=50) —
a stable cached latent appears to act as temporal regularization.

### Context (published numbers, their own eval protocols)

| Model | Params | LIBERO-Spatial | 4-suite avg |
|---|---|---|---|
| MolmoAct2-LIBERO | 5B | 98.4% (**97.2% re-measured on our rig**, n=500) | **98.25%** |
| π0 | 3.3B | 98% | — |
| SmolVLA-2.25B | 2.25B | 93% | — |
| XS-VLA | 0.26B | ~93% | 90.3% |
| MolmoAct-7B (v1) | 7B | 87.0% | 86.6% |
| **TinyVLA (this)** | **0.64B (140M online)** | **86% dual-rate** | **84.0%** |
| OpenVLA | 7B | 84.7% | 76.5% |

**MolmoAct2 context** (from their cards/repo, no numeric tables published outside the
LIBERO-LeRobot card): 5B params on Molmo2-ER backbone + flow expert; claims to beat π0.5;
Molmo2-ER reportedly surpasses GPT-5 / Gemini-Robotics-ER-1.5 on 13 embodied-reasoning
benchmarks; #1 VLA on the MolmoSpace leaderboard. Inference needs ~26GB fp32 (16GB bf16),
cuda-graphs for ~2× speedup — H100-class serving vs our Raspberry-Pi-class 140M fast path.
Notably they ship **MolmoAct2-SO100_101** (absolute joint control) — a 5B competitor on our
target embodiment; a head-to-head offline eval on SO101 held-out data is an obvious next test.

Honesty notes: our n=100 (±8pp) vs their 500; single suite; eval covers init states 0–9
per task (demos of same tasks/init states are in training data — this is the standard
LIBERO protocol used by all listed models). Multi-suite evals (object/goal/long) and an
identically-protocoled SmolVLA baseline run are in progress.

### Dual-rate validation (the architectural claim)

- Offline: held-out chunk MSE with latent from a frame 2s earlier == fresh (0.772 vs 0.772).
- Closed-loop: 86% @ 1 Hz refresh vs 79% full-rate.
- Deployment implication (Raspberry Pi 5): the 0.5B slow path can run at 0.5–1 Hz
  (llama.cpp GGUF Q4 territory), only the 140M fast path needs real-time rates.

## 4. Pitfalls we hit (read before reproducing)

1. **robosuite/LIBERO renders are 180°-rotated** vs recorded datasets. A vertical flip
   alone leaves a horizontal MIRROR — the policy reaches to the wrong side and scores 0%.
   Use `img[::-1, ::-1]`. (lerobot's `LiberoEnv.render()` flips for visualization but
   `_format_raw_obs` feeds policies raw rotated frames — check your eval path.)
2. **eef orientation antipodal ambiguity**: env quat → rotvec gives −π-ish x-component,
   datasets store +π-ish. Canonicalize (flip antipodal representation when rotvec_x < 0)
   or your normalized state input is off by ~17σ.
3. **HuggingFaceVLA/libero meta says fps=10 but frames are native 20 Hz** (~161
   frames/episode ≈ 8s demos). Do NOT action-repeat at eval.
4. **GT-replay harness is the decisive convention test**: replay dataset actions in the env
   (first episode per task ↔ init_state 0). 4/5 success = conventions right; policy-independent.
5. HuggingFaceVLA community_dataset v1/v3 are aggregates of v2.1 sub-datasets → convert with
   lerobot's `convert_dataset_v21_to_v30.py`; 4/44 sources have meta≠parquet row counts
   (broken at source) — validate `len(reader.hf_dataset) == meta.total_frames`.
6. Qwen3.5 custom forwards need `mm_token_type_ids` for M-RoPE; patchify must match the
   official processor bit-exact (verify!). Linear-attention needs fla-core + causal-conv1d
   or you get a 5× slower torch fallback.
7. Flow-matching loss can drop while conditioning is ignored — A/B with shuffled
   observations to verify the model actually uses them.

## 5. Repository layout

```
src/tinyvla/            # LeRobot plugin: configuration/modeling/processor + modules
scripts/train.py        # accelerate loop: weighted mixture, staleness aug, distill loss, resume
scripts/eval_libero.py  # closed-loop LIBERO eval (all convention fixes, dual-rate, video)
scripts/replay_libero.py, diag_obs_gap.py   # convention diagnostics
scripts/prepare_community_v3.py, label_spatial.py, eval_offline.py
configs/                # stage2/stage3/libero_ft* yaml
checkpoints/            # stage2_final, stage3_final, libero_ft4_final (main)
videos/                 # sample eval episodes
```

## 6. Quick start

```python
from tinyvla.modeling_tinyvla import TinyVLAPolicy
policy = TinyVLAPolicy.from_pretrained("AlexWortega/tinyvla")  # ft4 checkpoint at root
# closed-loop LIBERO: python scripts/eval_libero.py --checkpoint <ckpt> \
#   --suite libero_spatial --episodes 100 --n-action-steps 10 --refresh-s 1.0
```

Trained 2026-08-14 → 2026-08-19 on 1× RTX A6000 (48GB), lerobot 0.6.1, torch 2.10, transformers 5.5.4.