| --- |
| library_name: pytorch |
| license: apache-2.0 |
| tags: |
| - foundation |
| - amd |
| - rocm |
| - robotics |
| pipeline_tag: robotics |
| --- |
| |
|  |
|
|
| # SmolVLA: Optimized for AMD ROCm |
|
|
| SmolVLA (vision-language-action) is a behavior-cloning policy from Hugging Face LeRobot for 6-DOF robot arm control. This repository packages inference for robot arm action prediction using **PyTorch**, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs and CPUs. |
|
|
| This is based on the implementation of SmolVLA found [here](https://huggingface.co/lerobot/smolvla_base). |
| This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [smolVLA AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/smolVLA) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline). |
|
|
| --- |
|
|
| ## Task Overview |
|
|
| **Task:** Robot arm action prediction (vision-language-action) |
|
|
| **Dataset:** BlankHead/so101_redcube_greencloth_3cams (LeRobot format) |
| |
| **Output metrics:** MAE, RMSE (per-joint and per-episode) |
| |
| > **PyTorch note:** CPU runs FP32; GPU runs BF16. No NPU (VitisAI) path is available — `make setup-npu`, `make benchmark-npu`, and `make evaluate-npu` print an informational note and exit cleanly. |
| |
| --- |
| |
| ## AMD ROCm Optimization |
| |
| This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs. Key points: |
| |
| - Validated backend: **PyTorch** (native ROCm HIP kernels), FP32 on CPU, BF16 on GPU. |
| - No code changes required versus the upstream SmolVLA implementation — only environment/runtime configuration differs. |
| - No NPU (VitisAI) fallback path is available for this model. |
| |
| | Runtime | Precision | Backend | Hardware | Notes | |
| |---|---|---|---|---| |
| | PyTorch | FP32 | HIP (ROCm) | AMD CPU | — | |
| | PyTorch | BF16 | HIP (ROCm) | AMD Instinct™ / Radeon™ GPU | No NPU path available | |
| |
| --- |
| |
| ## Getting Started |
| |
| For setup instructions, evaluation scripts, and custom configuration options, see the [smolVLA on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/smolVLA). |
| |
| --- |
| |
| ## Model Details |
| |
| **Model Type:** Vision-language-action policy for robot arm control |
| |
| **Base Model:** `lerobot/smolvla_base` |
|
|
| **Model Stats:** |
| - Precision tested: FP32 (CPU), BF16 (GPU) |
| - Configurable runtime knobs: `rtc_config.enabled` (Real-Time Chunking), `num_steps` (flow-matching denoising passes per chunk), `n_action_steps` (actions consumed per chunk) |
|
|
| --- |
|
|
| ## Accuracy Pipeline |
|
|
| Open-loop offline evaluation is fully implemented: `make evaluate-<device>` runs inference on recorded dataset episodes and computes per-joint and per-episode MAE / RMSE against the recorded ground-truth actions. Lower is better for both metrics. |
|
|
| ### Metrics Explained |
|
|
| | Metric | Description | |
| |--------|-------------| |
| | MAE | Mean Absolute Error — average absolute difference between predicted and ground-truth joint positions across all timesteps. Lower is better. | |
| | RMSE | Root Mean Squared Error — penalizes large deviations more heavily than MAE. Lower is better. | |
|
|
| ### Accuracy Results |
|
|
| **Published Results** — Dataset: `BlankHead/so101_redcube_greencloth_3cams` (13 episodes, chunked_rtc mode): |
| |
| <!-- accuracy-table-start --> |
| | Metric | Value | |
| |--------|-------| |
| | Average MAE | 3.9521 | |
| | Average RMSE | 7.7251 | |
| <!-- accuracy-table-end --> |
| |
| **Per-joint breakdown:** |
| |
| | Joint | Avg MAE | Avg RMSE | |
| |-------|---------|----------| |
| | shoulder_pan | 3.3654 | 5.1317 | |
| | shoulder_lift | 7.8224 | 14.0630 | |
| | elbow_flex | 4.9989 | 8.7409 | |
| | wrist_flex | 2.4138 | 3.4653 | |
| | wrist_roll | 2.6728 | 3.9475 | |
| | gripper | 2.4394 | 4.5340 | |
|
|
| --- |
|
|
| ## Dig Deeper |
|
|
| Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples? |
|
|
| 📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/smolVLA)** |
|
|
| The GitHub repository includes: |
| - Setup and prerequisites for ROCm environments |
| - Open-loop dataset evaluation with trajectory plots and comparison videos |
| - Latency benchmarking with Chrome trace output |
| - PEFT/LoRA-aware checkpoint loading and runtime chunking configuration |
|
|
