| --- |
| library_name: lerobot |
| pipeline_tag: robotics |
| tags: |
| - lerobot |
| - robotics |
| - imitation-learning |
| - x7s |
| - smolvla |
| - vision-language-action |
| - offline-evaluation |
| datasets: |
| - LightwheelAI/Lightwheel-Tasks-X7S |
| base_model: |
| - lerobot/smolvla_base |
| --- |
| |
| # LW-Bench SmolVLA 5K for X7s |
|
|
| LeRobot SmolVLA policy fine-tuned from `lerobot/smolvla_base` on the audited X7s strict compositional split in LW-Compositional-Bench. |
|
|
| ## Training contract |
|
|
| - Dataset: `LightwheelAI/Lightwheel-Tasks-X7S` |
| - Train split: 91 clean tasks; tasks 7, 119, and 149 held out |
| - Inputs: three RGB cameras, 25D robot state, and canonical task language |
| - Output: native 21D X7s absolute action |
| - Hardware: 4 GPUs |
| - Seed: 1000 |
| - Effective global batch: 64 |
| - Checkpoint: 5,000 global optimizer updates |
| - Evaluation horizon: first 16 valid action steps |
|
|
| ## Offline 5K results |
|
|
| | Metric | Value | |
| | --- | ---: | |
| | H16 MAE | 0.117617 | |
| | H16 RMSE | 0.227890 | |
| | Gripper balanced accuracy | 87.08% | |
| | Boundary-proxy MAE | 0.147732 | |
| | Interior MAE | 0.111848 | |
| | Boundary / interior MAE | 1.321x | |
|
|
| These results cover three strict held-out tasks and are descriptive. Offline action error does not replace closed-loop success evaluation. |
|
|
| ## Load with LeRobot |
|
|
| ```python |
| from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy |
| |
| policy = SmolVLAPolicy.from_pretrained("Stevenshuqing/LW-Bench-SmolVLA") |
| policy.eval() |
| ``` |
|
|
| Use LeRobot 0.4.3 and the included preprocessor/postprocessor files. The repository contains the complete `pretrained_model` directory produced by LeRobot. |
|
|
| ## Provenance |
|
|
| - Base model: https://huggingface.co/lerobot/smolvla_base |
| - Code and protocol: https://github.com/stevenqing/LW-Compositional-Bench |
| - Full report: https://github.com/stevenqing/LW-Compositional-Bench/blob/main/artifacts/model-eval/x7s/rotation-v0/budget-5k/report.md |
| - Code revision: `2d0e26dd2fe2e91046f416bd65bc369f8568d7f0` |
| - `model.safetensors` SHA256: `f70959474aad109134f799e49dd313abf209380c8e1c808d8650e6d92287e242` |
| |
| ## Limitations |
| |
| This checkpoint is specific to the X7s observation/action schema. The benchmark is offline because the available A100 GPUs cannot provide Isaac Sim RT camera rendering. Transition boundaries are action-space proxies based on gripper sign changes and robust continuous-action peaks, not human semantic annotations. |
| |
