README.md

---
license: apache-2.0
library_name: diffusers
pipeline_tag: robotics
base_model: robbyant/lingbot-va-base
tags:
- robotics
- vision-language-action
- lingbot-va
- lora
- franka
- dual-arm
- real-robot
---

# LingBot-VA Dual-Franka Five-Task LoRA — 10k

English | [中文说明](#中文说明)

This repository contains a LoRA adapter trained from [`robbyant/lingbot-va-base`](https://huggingface.co/robbyant/lingbot-va-base) on five real-robot manipulation tasks collected with a dual Franka FR3 setup.

The repository is not a standalone full model. Load the base LingBot-VA checkpoint first, then apply `adapter_model.safetensors` using `adapter_config.json`.

## Tasks and prompts

| Task ID | Training prompt | Episodes used |
|---|---|---:|
| `place_bowl_on_cabinet` | `Place the green bowl on top of the cabinet with the left arm.` | 49 |
| `place_bowl_on_plate_pt2` | `Place the green bowl on the blue plate with the left arm.` | 18 |
| `place_bowl_on_plate` | `Place the gray bowl on the green plate with the left arm.` | 49 |
| `place_cup_on_tray` | `Place the cup on the tray with the left arm.` | 50 |
| `tape_on_basket` | `Place the tape in the green basket with the left arm.` | 50 |

Total training episodes: 216. The dataset is imbalanced because only 18 `place_bowl_on_plate_pt2` trajectories passed the strict LingBot-VA conversion/alignment checks.

## Inputs and outputs

- Visual inputs: chest, left-wrist, and right-wrist RGB streams
- Language conditioning: exact task prompt listed above
- Training representation: video features pre-encoded as Wan2.2 VAE visual latents, with text embeddings
- Robot-state input: no `observation.state` was supplied to this LingBot-VA training run
- Source action: 16 dimensions — left 7 joints + left gripper + right 7 joints + right gripper
- LingBot action layout: the 16 source dimensions are scattered into a 30-dimensional canonical layout; unused Cartesian channels are masked/padded

## Training configuration

| Setting | Value |
|---|---|
| Base model | `robbyant/lingbot-va-base` |
| Method | LoRA SFT |
| Trainable parameters | 47,185,920 |
| LoRA rank / alpha / dropout | 32 / 32 / 0.05 |
| Target modules | self- and cross-attention Q/K/V/output projections |
| Training steps | 10,000 |
| Hardware | 4 × NVIDIA A100 80GB |
| Per-device / global batch | 1 / 4 |
| Gradient accumulation | 1 |
| Learning rate | `5e-5`, 10-step warmup |
| Adam betas / weight decay | 0.9, 0.95 / 0.1 |
| Precision | bfloat16 |
| Checkpoint interval | 1,000 steps |
| Final checkpoint | `checkpoint_step_10000` |

At step 9,999, the logged losses were: latent `0.0806`, action `0.00481`, total `0.0854`. These are training losses, not task success rates.

## Repository contents

- `adapter_model.safetensors`: LoRA weights
- `adapter_config.json`: rank, alpha, dropout, target modules, and base-model reference
- `training_state.pt`: optimizer/scheduler/RNG state for the original training environment

`training_state.pt` may contain Python-serialized objects. Only load it with `torch.load` if you trust this repository and need training resumption. It is not required for inference.

## Download and loading

```bash
hf download robbyant/lingbot-va-base --local-dir ./lingbot-va-base
hf download THU98/lingbotva_franka_5task_lora_10k \
  --local-dir ./lingbotva_franka_5task_lora_10k
```

Use the official LingBot-VA repository and its post-training/inference code:

```bash
git clone https://github.com/Robbyant/lingbot-va.git
```

Configure the base path as `./lingbot-va-base` and the adapter path as:

```text
./lingbotva_franka_5task_lora_10k/adapter_model.safetensors
```

The exact adapter targets are stored in `adapter_config.json`. Use the dual-Franka joint configuration and the same 16-to-30-dimensional action-channel mapping used during training. Generic PEFT loading is not guaranteed because the training code uses the LingBot-VA LoRA integration.

## Evaluation status

No controlled real-robot success-rate evaluation is published with this adapter. No held-out validation set was used to select the final checkpoint. Loss reduction alone does not demonstrate safe or successful robot execution.

## Intended use and limitations

- Research use with the official LingBot-VA base model and a matching dual Franka setup.
- The data are small, task-specific, imbalanced, and dominated by left-arm behaviors.
- The adapter was trained without proprioceptive `observation.state` input; behavior depends heavily on visual calibration and temporal alignment.
- It is not validated for different robots, viewpoints, objects, control frequencies, action conventions, or prompt paraphrases.
- Validate action-channel mapping, normalization, camera order, control rate, latency, collision avoidance, and emergency stops before deployment.
- Human supervision and independent robot safety controls are required.

## 中文说明

这是基于 `robbyant/lingbot-va-base` 训练的双 Franka FR3 五任务 LoRA adapter，不是可以独立加载的完整模型。推理时需要先加载 LingBot-VA base，再加载本仓库中的 `adapter_model.safetensors`。

训练使用三路相机的视觉 latent 和 Language Instruction，没有把 16 维 `observation.state` 作为模型输入。源动作是 16 维双臂关节与夹爪动作，并映射到 LingBot-VA 的 30 维动作布局。五个任务共使用 216 条轨迹，其中 `place_bowl_on_plate_pt2` 只有 18 条通过严格数据校验，因此数据分布不均衡。

当前没有发布正式真机成功率。部署前必须核对 base 版本、LoRA target、相机顺序、动作通道、归一化、控制频率以及真机安全限制。

## Sources and citation

- LingBot-VA repository: https://github.com/Robbyant/lingbot-va
- Base model: https://huggingface.co/robbyant/lingbot-va-base
- Paper: https://huggingface.co/papers/2601.21998

```bibtex
@article{lingbotva2026,
  title={Causal World Modeling for Robot Control},
  journal={arXiv preprint arXiv:2601.21998},
  year={2026}
}
```