Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+collage_small.gif filter=lfs diff=lfs merge=lfs -text

Finetune_SmolVLA_notebook.ipynb

@@ -0,0 +1,214 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "NQUk3Y0WwYZ4"
+      },
+      "source": [
+        "# 🤗 x 🦾: Training SmolVLA with LeRobot Notebook\n",
+        "\n",
+        "Welcome to the **LeRobot SmolVLA training notebook**! This notebook provides a ready-to-run setup for training imitation learning policies using the [🤗 LeRobot](https://github.com/huggingface/lerobot) library.\n",
+        "\n",
+        "In this example, we train an `SmolVLA` policy using a dataset hosted on the [Hugging Face Hub](https://huggingface.co/), and optionally track training metrics with [Weights & Biases (wandb)](https://wandb.ai/).\n",
+        "\n",
+        "## ⚙️ Requirements\n",
+        "- A Hugging Face dataset repo ID containing your training data (`--dataset.repo_id=YOUR_USERNAME/YOUR_DATASET`)\n",
+        "- Optional: A [wandb](https://wandb.ai/) account if you want to enable training visualization\n",
+        "- Recommended: GPU runtime (e.g., NVIDIA A100) for faster training\n",
+        "\n",
+        "## ⏱️ Expected Training Time\n",
+        "Training with the `SmolVLA` policy for 20,000 steps typically takes **about 5 hours on an NVIDIA A100** GPU. On less powerful GPUs or CPUs, training may take significantly longer!\n",
+        "\n",
+        "## Example Output\n",
+        "Model checkpoints, logs, and training plots will be saved to the specified `--output_dir`. If `wandb` is enabled, progress will also be visualized in your wandb project dashboard.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MOJyX0CnwA5m"
+      },
+      "source": [
+        "## Install conda\n",
+        "This cell uses `condacolab` to bootstrap a full Conda environment inside Google Colab.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QlKjL1X5t_zM"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install -q condacolab\n",
+        "import condacolab\n",
+        "condacolab.install()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DxCc3CARwUjN"
+      },
+      "source": [
+        "## Install LeRobot\n",
+        "This cell clones the `lerobot` repository from Hugging Face, installs FFmpeg (version 7.1.1), and installs the package in editable mode.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dgLu7QT5tUik"
+      },
+      "outputs": [],
+      "source": [
+        "!git clone https://github.com/huggingface/lerobot.git\n",
+        "!conda install ffmpeg=7.1.1 -c conda-forge\n",
+        "!cd lerobot && pip install -e ."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Q8Sn2wG4wldo"
+      },
+      "source": [
+        "## Weights & Biases login\n",
+        "This cell logs you into Weights & Biases (wandb) to enable experiment tracking and logging."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PolVM_movEvp"
+      },
+      "outputs": [],
+      "source": [
+        "!wandb login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zTWQAgX9xseE"
+      },
+      "source": [
+        "## Install SmolVLA dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DiHs0BKwxseE"
+      },
+      "outputs": [],
+      "source": [
+        "!cd lerobot && pip install -e \".[smolvla]\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "IkzTo4mNwxaC"
+      },
+      "source": [
+        "## Start training SmolVLA with LeRobot\n",
+        "\n",
+        "This cell runs the `train.py` script from the `lerobot` library to train a robot control policy.  \n",
+        "\n",
+        "Make sure to adjust the following arguments to your setup:\n",
+        "\n",
+        "1. `--dataset.repo_id=YOUR_HF_USERNAME/YOUR_DATASET`:  \n",
+        "   Replace this with the Hugging Face Hub repo ID where your dataset is stored, e.g., `pepijn223/il_gym0`.\n",
+        "\n",
+        "2. `--batch_size=64`: means the model processes 64 training samples in parallel before doing one gradient update. Reduce this number if you have a GPU with low memory.\n",
+        "\n",
+        "3. `--output_dir=outputs/train/...`:  \n",
+        "   Directory where training logs and model checkpoints will be saved.\n",
+        "\n",
+        "4. `--job_name=...`:  \n",
+        "   A name for this training job, used for logging and Weights & Biases.\n",
+        "\n",
+        "5. `--policy.device=cuda`:  \n",
+        "   Use `cuda` if training on an NVIDIA GPU. Use `mps` for Apple Silicon, or `cpu` if no GPU is available.\n",
+        "\n",
+        "6. `--wandb.enable=true`:  \n",
+        "   Enables Weights & Biases for visualizing training progress. You must be logged in via `wandb login` before running this."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZO52lcQtxseE"
+      },
+      "outputs": [],
+      "source": [
+        "!cd lerobot && python lerobot/scripts/train.py \\\n",
+        "  --policy.path=lerobot/smolvla_base \\\n",
+        "  --dataset.repo_id=${HF_USER}/mydataset \\\n",
+        "  --batch_size=64 \\\n",
+        "  --steps=20000 \\\n",
+        "  --output_dir=outputs/train/my_smolvla \\\n",
+        "  --job_name=my_smolvla_training \\\n",
+        "  --policy.device=cuda \\\n",
+        "  --wandb.enable=true"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2PBu7izpxseF"
+      },
+      "source": [
+        "## Login into Hugging Face Hub\n",
+        "Now after training is done login into the Hugging Face hub and upload the last checkpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8yu5khQGIHi6"
+      },
+      "outputs": [],
+      "source": [
+        "!huggingface-cli login"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zFMLGuVkH7UN"
+      },
+      "outputs": [],
+      "source": [
+        "!huggingface-cli upload ${HF_USER}/my_smolvla \\\n",
+        "  /content/lerobot/outputs/train/my_smolvla/checkpoints/last/pretrained_model"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "gpuType": "A100",
+      "machine_shape": "hm",
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

README.md

@@ -0,0 +1,146 @@
+---
+language:
+- en
+library_name: lerobot
+pipeline_tag: robotics
+tags:
+- vision-language-action
+- imitation-learning
+- lerobot
+inference: false
+---
+
+# SmolVLA (LeRobot)
+
+SmolVLA is a compact, efficient Vision-Language-Action (VLA) model designed for affordable robotics, trainable on a single GPU and deployable on consumer hardware, while matching the performance of much larger VLAs through community-driven data.
+
+**Original paper:** (SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics)[https://arxiv.org/abs/2506.01844]
+**Reference implementation:** https://github.com/huggingface/lerobot
+
+
+## Model description
+
+- **Inputs:** images (multi-view), proprio/state, optional language instruction
+- **Outputs:** continuous actions
+- **Training objective:** flow matching
+- **Action representation:** continuous
+- **Intended use:** Base model to fine tune on your specific use case
+
+
+## Quick start (inference on a real batch)
+
+### Installation
+
+```bash
+pip install "lerobot[smolvla]"
+```
+For full installation details (including optional video dependencies such as ffmpeg for torchcodec), see the official documentation: https://huggingface.co/docs/lerobot/installation
+
+### Load model + dataset, run `select_action`
+
+```python
+import torch
+from lerobot.datasets.lerobot_dataset import LeRobotDataset
+from lerobot.policies.factory import make_pre_post_processors
+
+# Swap this import per-policy
+from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
+
+# load a policy
+model_id = "lerobot/smolvla_base"  # <- swap checkpoint
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+policy = SmolVLAPolicy.from_pretrained(model_id).to(device).eval()
+
+preprocess, postprocess = make_pre_post_processors(
+    policy.config,
+    model_id,
+    preprocessor_overrides={"device_processor": {"device": str(device)}},
+)
+# load a lerobotdataset
+dataset = LeRobotDataset("lerobot/libero")
+
+# pick an episode
+episode_index = 0
+
+# each episode corresponds to a contiguous range of frame indices
+from_idx = dataset.meta.episodes["dataset_from_index"][episode_index]
+to_idx   = dataset.meta.episodes["dataset_to_index"][episode_index]
+
+# get a single frame from that episode (e.g. the first frame)
+frame_index = from_idx
+frame = dict(dataset[frame_index])
+
+batch = preprocess(frame)
+with torch.inference_mode():
+    pred_action = policy.select_action(frame)
+    # use your policy postprocess, this post process the action
+    # for instance unnormalize the actions, detokenize it etc..
+    pred_action = postprocess(pred_action)
+```
+
+
+## Training step (loss + backward)
+
+If you’re training / fine-tuning, you typically call `forward(...)` to get a loss and then:
+
+```python
+policy.train()
+batch = dict(dataset[0])
+batch = preprocess(batch)
+
+loss, outputs = policy.forward(batch)
+loss.backward()
+
+```
+
+> Notes:
+> 
+> - Some policies expose `policy(**batch)` or return a dict; keep this snippet aligned with the policy API.
+> - Use your trainer script (`lerobot-train`) for full training loops.
+
+
+## How to train / fine-tune
+
+```bash
+lerobot-train \
+  --dataset.repo_id=${HF_USER}/<dataset> \
+  --output_dir=./outputs/[RUN_NAME] \
+  --job_name=[RUN_NAME] \
+  --policy.repo_id=${HF_USER}/<desired_policy_repo_id> \
+  --policy.path=lerobot/[BASE_CHECKPOINT] \
+  --policy.dtype=bfloat16 \
+  --policy.device=cuda \
+  --steps=100000 \
+  --batch_size=4
+```
+
+Add policy-specific flags below:
+
+- `-policy.chunk_size=...`
+- `-policy.n_action_steps=...`
+- `-policy.max_action_tokens=...`
+- `-policy.gradient_checkpointing=true`
+
+
+## Real-World Inference & Evaluation
+
+You can use the `record` script from [**`lerobot-record`**](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/lerobot_record.py) with a policy checkpoint as input, to run inference and evaluate your policy. 
+
+For instance, run this command or API example to run inference and record 10 evaluation episodes:
+
+```
+lerobot-record  \
+  --robot.type=so100_follower \
+  --robot.port=/dev/ttyACM1 \
+  --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}, side: {type: intelrealsense, serial_number_or_name: 233522074606, width: 640, height: 480, fps: 30}}" \
+  --robot.id=my_awesome_follower_arm \
+  --display_data=false \
+  --dataset.repo_id=${HF_USER}/eval_so100 \
+  --dataset.single_task="Put lego brick into the transparent box" \
+  # <- Teleop optional if you want to teleoperate in between episodes \
+  # --teleop.type=so100_leader \
+  # --teleop.port=/dev/ttyACM0 \
+  # --teleop.id=my_awesome_leader_arm \
+  --policy.path=${HF_USER}/my_policy
+```

config.json

@@ -0,0 +1,96 @@
+{
+    "type": "smolvla",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.camera1": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        },
+        "observation.images.camera2": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        },
+        "observation.images.camera3": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "push_to_hub": true,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "chunk_size": 50,
+    "n_action_steps": 50,
+    "normalization_mapping": {
+        "VISUAL": "IDENTITY",
+        "STATE": "MEAN_STD",
+        "ACTION": "MEAN_STD"
+    },
+    "max_state_dim": 32,
+    "max_action_dim": 32,
+    "resize_imgs_with_padding": [
+        512,
+        512
+    ],
+    "empty_cameras": 0,
+    "adapt_to_pi_aloha": false,
+    "use_delta_joint_actions_aloha": false,
+    "tokenizer_max_length": 48,
+    "num_steps": 10,
+    "use_cache": true,
+    "freeze_vision_encoder": true,
+    "train_expert_only": true,
+    "train_state_proj": true,
+    "optimizer_lr": 0.0001,
+    "optimizer_betas": [
+        0.9,
+        0.95
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 1e-10,
+    "optimizer_grad_clip_norm": 10,
+    "scheduler_warmup_steps": 1000,
+    "scheduler_decay_steps": 30000,
+    "scheduler_decay_lr": 2.5e-06,
+    "vlm_model_name": "HuggingFaceTB/SmolVLM2-500M-Video-Instruct",
+    "load_vlm_weights": true,
+    "add_image_special_tokens": false,
+    "attention_mode": "cross_attn",
+    "prefix_length": 0,
+    "pad_language_to": "max_length",
+    "num_expert_layers": 0,
+    "num_vlm_layers": 16,
+    "self_attn_every_n_layers": 2,
+    "expert_width_multiplier": 0.75,
+    "min_period": 0.004,
+    "max_period": 4.0
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cd549ac2351fb069c0ddb3c34ad2d09cfc92b56a15dccdfc2e41467aaca01eb
+size 906712520

policy_postprocessor.json

@@ -0,0 +1,32 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:490ab239d96e263687c0b2e386a0afbc235a2eceb9857c36ed32f2f162a3e7c8
+size 640

policy_preprocessor.json

@@ -0,0 +1,87 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "smolvla_new_line_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "tokenizer_processor",
+      "config": {
+        "max_length": 48,
+        "task_key": "task",
+        "padding_side": "right",
+        "padding": "max_length",
+        "truncation": true,
+        "tokenizer_name": "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
+      }
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              6
+            ]
+          },
+          "observation.image2": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "observation.image": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "observation.image3": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_preprocessor_step_5_normalizer_processor.safetensors"
+    }
+  ]
+}

policy_preprocessor_step_5_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:490ab239d96e263687c0b2e386a0afbc235a2eceb9857c36ed32f2f162a3e7c8
+size 640