| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: robotics |
| base_model: Dream-org/Dream-VLA-7B |
| base_model_relation: adapter |
| tags: |
| - vision-language-action |
| - robotics |
| - robosuite |
| - lora |
| - flow-matching |
| - diffusion-language-model |
| - simulation |
| --- |
| # Jocasta-diff-VLA |
|
|
| **Code and loader:** [alanrbtx/Diffusion-VLA-tuning](https://github.com/alanrbtx/Diffusion-VLA-tuning) |
|
|
| Jocasta-diff-VLA is a Dream-VLA-7B adaptation to eight simulated Panda manipulation task families in robosuite. It uses the Dream-VLX authors' continuous Flow Matching action head and a diffusion language-model backbone. One jointly trained policy achieved **332/512 sustained successes (64.8%)** on the fixed development evaluation. |
|
|
| This repository contains an **inference checkpoint**, not just a LoRA language adapter: the LoRA weights, trained action head, noisy-action projector, proprioception projector, dataset normalization, processor files and training configuration are included. The pinned original Dream-VLA base weights are required separately. The public loader downloads both when needed. No optimizer or training-resume state is distributed. |
|
|
| ## Load |
|
|
| Use the accompanying repository with its pinned submodules and patch: |
|
|
| ```bash |
| git clone --recurse-submodules https://github.com/alanrbtx/Diffusion-VLA-tuning.git |
| cd Diffusion-VLA-tuning |
| git -C third_party/Dream-VLX apply ../../patches/dream-vlx-panda.patch |
| python -m pip install -r requirements.lock -r requirements-multitask.txt |
| ``` |
|
|
| Run inference on one visible CUDA GPU: |
|
|
| ```python |
| from load_policy import load_policy, predict_action |
| |
| policy, processor, cfg = load_policy('RobotLearningLab/Jocasta-diff-VLA') |
| # primary and wrist: uint8 RGB arrays, each (224,224,3) |
| # proprio: float32 (8,), xyz + axis-angle orientation + two gripper joints |
| actions = predict_action(policy, processor, cfg, primary, wrist, proprio, |
| 'pick up the red cube', seed=84907) |
| # actions.shape == (8,7) |
| ``` |
|
|
| For reproducibility, pass the desired Hub commit with `revision=...`. The base is fixed to `Dream-org/Dream-VLA-7B@0cd0e25c1771c5b468c74585b5cf35c4a4aa315c`; `base_model_path=...` reuses a local copy. This is a custom VLA package; `AutoModel.from_pretrained` or loading only the adapter will not reconstruct the trained action policy. |
|
|
| Action columns are world-frame translation deltas in metres, extrinsic XYZ Euler deltas in radians, and gripper openness. They require the documented conversion to normalized Panda OSC commands; see `panda_adapter.py`. Proprioception uses the recorded quaternion-to-axis-angle convention in `robot_state.py`. In simulation, keep CUDA inference and EGL rendering on separate physical GPUs. Model inference itself creates no EGL context. |
|
|
| ## Training |
|
|
| | Setting | Value | |
| |---|---| |
| | Base | Dream-VLA-7B, pinned revision above | |
| | Upstream code | DreamLM/Dream-VLX `a72fa4106eae97a923b531342fb692742809782f` | |
| | Objective | Author Flow Matching velocity MSE | |
| | Optimizer | AdamW, learning rate1e-4 | |
| | Updates |100,000; fixed final checkpoint | |
| | Global batch |64 (4 GPUs × microbatch8 × accumulation2) | |
| | LoRA |rank32, alpha16, dropout0, all linear layers | |
| | Training/sampling seeds |84907 /190019 | |
| | Inputs |two224RGB cameras,8D proprioception, instruction | |
| | Inference |4 Flow Matching steps; action chunk8 | |
| | Training data |8,140 successful trajectories;4,841,083 timesteps | |
| | Task/language diversity |8 families,30 goals,510 train instruction strings | |
|
|
| The model was initialized from the original Dream-VLA base with fresh action modules, not from the earlier single-task Lift model. Training used family/goal balancing and the author's image augmentation. A separate120-string held-out-language development bank was excluded from training; the results below use canonical instructions and do not constitute a held-out-language evaluation. Raw demonstration data are not included; the code documents the exact cache schema. |
|
|
| ## Evaluation |
|
|
| 64 fixed development scenes per family, seeds30000–30063, original physical scene configuration and horizons,10 settling steps, `hard_reset=False`. Success requires the original task predicate to hold for10 consecutive physical steps. All512 episodes were independently audited, including45,254 policy calls and365,450 physical steps. One training seed; no reserved test scenes, checkpoint selection or metric-driven reruns. |
|
|
| | Task | Successes | Rate | |
| |---|---:|---:| |
| | Lift |64/64|100.0%| |
| | PickPlace |40/64|62.5%| |
| | Stack |43/64|67.2%| |
| | Door |34/64|53.1%| |
| | NutAssembly |32/64|50.0%| |
| | ToolHang |9/64|14.1%| |
| | Push |61/64|95.3%| |
| | Drawer |49/64|76.6%| |
| | Total |332/512|64.8%| |
|
|
| Original pretrained Dream-VLA achieved **0/64** on the same PickPlace scenes; Jocasta-diff-VLA achieved **40/64**. The original policy used one image, one action,4 discrete-diffusion steps and native Bridge statistics. The adapted policy uses two images/proprio,8 actions,4 Flow Matching steps and Panda statistics. This is a comparison of complete transferred policies, not an isolated backbone or diffusion-mechanism effect. No baseline result is claimed for the other seven tasks. See `evaluation-results.json` for counts and per-family Wilson intervals. |
|
|
| ## Scope and limitations |
|
|
| These are development results on our robosuite/Panda task definitions, not official LIBERO scores. ToolHang is weak (9/64). Other robots, unseen task families, real hardware and the reserved test split have not been validated. This release implements manipulation skills, not autonomous replication or learned mission-memory inheritance. |
|
|
| The release loader is checked against the original checkpoint on saved observations from all eight task families. `validation.json` records the outcome. `release.json` records file hashes; the original adapter and action-component weight bytes are retained unchanged. Reference inference environment: Python3.10.18, PyTorch2.5.1+cu124, Transformers4.51.3, BF16/SDPA on H200. No task evaluation was rerun for packaging. |
|
|
| ## Attribution |
|
|
| Architecture, Flow Matching loss/sampler, action heads and projectors originate from Dream-VLX, with upstream OpenVLA/OpenVLA-OFT attribution retained in the code. Panda adaptations and release: RobotLearningLab. Upstream code and model terms remain applicable; this adaptation is released under Apache-2.0. |
|
|
| ```bibtex |
| @article{ye2025dreamvla, |
| title={Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone}, |
| author={Ye, Jiacheng and Gong, Shansan and Gao, Jiahui and Fan, Junming and Wu, Shuang and Bi, Wei and Bai, Haoli and Shang, Lifeng and Kong, Lingpeng}, |
| journal={arXiv preprint arXiv:2512.22615}, |
| year={2025} |
| } |
| ``` |
|
|
| Validated code revision: [c21ef87ef2d749b1eb2d06f7eb91e4c2b843bcd6](https://github.com/alanrbtx/Diffusion-VLA-tuning/tree/c21ef87ef2d749b1eb2d06f7eb91e4c2b843bcd6). |
|
|
