Add checkpoint at step 100000

checkpoints/100000/README.md

@@ -0,0 +1,196 @@
+---
+license: apache-2.0
+library_name: lerobot
+pipeline_tag: robotics
+tags:
+  - robotics
+  - lerobot
+  - imitation-learning
+  - act
+  - b-spline
+  - so101
+  - bspline_act
+datasets:
+  - aryankakad/CUPSTACKING
+model_name: cupstack_bspline_act
+---
+
+# cupstack_bspline_act
+
+An [ACT](https://huggingface.co/papers/2304.13705) policy that predicts **B-spline
+trajectory segments** instead of discrete action chunks, trained on SO-101 cup
+stacking. This is the "Reg.+BSP" variant from
+[B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations](https://arxiv.org/abs/2607.09648).
+
+The practical consequence: **you can run this checkpoint faster without retraining it.**
+The policy outputs a curve, so executing `a(n·t)` replays the same trajectory geometry
+n times faster. Speed-up is an inference flag.
+
+> [!WARNING]
+> **This checkpoint has never been evaluated on held-out data or on hardware.**
+> It was trained on all 50 episodes with no validation split, so its numbers measure
+> fit, not generalization. Treat first hardware runs as untested — keep the e-stop
+> within reach. See [Limitations](#limitations).
+
+## Usage
+
+```bash
+lerobot-rollout \
+    --strategy.type=base \
+    --policy.path=aryankakad/cupstack_bspline_act \
+    --policy.speed_up=1.0 \
+    --device=cuda \
+    --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=FOLLOWER \
+    --robot.cameras='{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, gripper: {type: opencv, index_or_path: 4, width: 640, height: 480, fps: 30}}' \
+    --task="stack the cups" \
+    --fps=30 --duration=30 --display_data=true
+```
+
+Raise `--policy.speed_up` to 2.0 or 4.0 to execute faster. Start at 1.0 and step up
+only once the task succeeds.
+
+### Camera naming is not optional
+
+The policy requires exactly these observation keys, so the `--robot.cameras` dict keys
+must be **`front`** and **`gripper`**, lowercase:
+
+| Key | Shape |
+|---|---|
+| `observation.images.front` | `(3, 480, 640)` |
+| `observation.images.gripper` | `(3, 480, 640)` |
+| `observation.state` | `(6,)` |
+
+Joint order is `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper`.
+Swapping the two cameras produces confident but wrong behaviour rather than an error.
+
+### Requirements
+
+This policy type is not in upstream LeRobot. You need a checkout containing the
+`bspline_act` policy, installed with the scipy extra:
+
+```bash
+uv pip install -e ".[feetech,bspline]"
+```
+
+`scipy>=1.15` is required for `interpolate.generate_knots`.
+
+### No dataset needed at inference
+
+Unnormalization statistics are stored as buffers inside `model.safetensors`, so the
+checkpoint is self-contained. You do not need the training dataset on the robot machine.
+
+## Training
+
+| | |
+|---|---|
+| Steps | **100,000** |
+| Batch size | 32 |
+| Epochs | 133.09 |
+| Dataset | [`aryankakad/CUPSTACKING`](https://huggingface.co/datasets/aryankakad/CUPSTACKING) → B-spline converted |
+| Episodes / frames | 50 / 24,044 @ 30 fps |
+| Optimizer | AdamW, lr 2e-5 (backbone 1e-5), wd 1e-4 |
+| Precision | bf16 autocast + `channels_last` |
+| Hardware | 1× RTX 4090, 5 h 24 min |
+| Seed | 1000 |
+
+Learning rate is 2e-5 rather than ACT's default 1e-5, sqrt-scaled for batch 32.
+
+### Loss
+
+| Step | loss | l1_loss | kld_loss |
+|---|---|---|---|
+| 500 | 3.027 | 0.390 | 0.264 |
+| 10k | 0.166 | 0.104 | 0.007 |
+| 50k | ~0.111 | 0.041 | 0.007 |
+| **100k** | **0.095** | **0.026** | **0.007** |
+
+Training loss only — there was no validation split.
+
+## Architecture
+
+Standard ACT with one change: the decoder emits a B-spline **parameter matrix** rather
+than an action chunk.
+
+```
+(n_knots, 1 + action_dim) = (16, 7) = 112 values per prediction
+   column 0   knot vector, in source-frame units, 0 = "now"
+   columns 1: control points, one per joint
+```
+
+| | |
+|---|---|
+| Params | 52 M |
+| Vision backbone | ResNet18 (ImageNet init) |
+| dim_model / chunk_size | 512 / 16 |
+| VAE | enabled, kl_weight 10.0 |
+| B-spline degree | 3 (cubic, C² continuous) |
+| bspline_chunk_size | 10 |
+| Fitting tolerance ε | 0.2 (degrees) |
+
+Knot spacing is fitted adaptively per episode, so a fixed 16 rows covers a **variable**
+time horizon — 0.53 s to 1.47 s per segment on this dataset, 0.84 s mean. The network
+only runs when a segment is exhausted, which decouples policy rate from control rate.
+
+### On the fitting tolerance
+
+ε = 0.2 is not the paper's value. The paper uses 0.002 for metre-scale end-effector
+actions; SO-101 stores joint targets in **degrees** (~±120), roughly 100× larger. At
+ε = 0.002 compression is 1.06× — one knot per frame, which defeats the representation.
+
+| ε | Compression | p99 reconstruction | Segment span |
+|---|---|---|---|
+| 0.05 | 1.4× | 0.05° | 0.44 s |
+| **0.2** | **2.7×** | **0.19°** | **0.84 s** |
+| 0.5 | 4.1× | 0.48° | 1.24 s |
+
+2.7× sits inside the paper's reported 1.12×–3.34× range.
+
+## Evaluation
+
+**Offline only. No hardware evaluation has been performed.**
+
+Open-loop prediction error — decoded trajectory vs ground-truth actions, measured on
+**training data**:
+
+| Checkpoint | Mean | Median | p90 | Max |
+|---|---|---|---|---|
+| 20k | 5.99° | 4.12° | 8.78° | 45.2° |
+| 50k | 3.34° | 2.28° | 6.53° | 30.8° |
+| **100k** | **2.10°** | **1.61°** | **3.86°** | **18.6°** |
+
+Error was still falling at 100k and 2.10° is far from zero, which argues against
+outright memorization — but this is training data, so it is not evidence of
+generalization.
+
+The representation is not the bottleneck: B-spline fitting reconstructs to 0.19° p99,
+so essentially all of the 2.10° is policy prediction error.
+
+Temporal rescaling is **exact**. On a real predicted segment (0.76 s span), `a(2t)` and
+`a(4t)` reproduce the 1× samples to 0.00e+00, consuming the segment in 22 / 11 / 5
+control ticks.
+
+## Limitations
+
+- **No validation split.** All 50 episodes were used for training (`eval_steps: 0`).
+  Generalization is unmeasured.
+- **No hardware evaluation.** Task success rate is unknown.
+- **Speed-up has a hardware ceiling.** The paper reaches 4× on cube picking but only 2×
+  on speed stacking before the low-level controller loses tracking. Cup stacking is the
+  same precise-placement regime, so expect degradation near 2×. Retiming does not make
+  the arm faster — past the tracking limit it overshoots rather than stopping.
+- **Single task, single scene.** 50 demonstrations of one cup-stacking setup; no
+  robustness to lighting, camera placement, or cup position changes should be assumed.
+- **Camera assignment is silent when wrong.** Swapped views degrade behaviour without
+  raising an error.
+
+## Citation
+
+```bibtex
+@article{han2026b,
+  title={B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations},
+  author={Han, Xiaoshen and Xiong, Haoyu and Chen, Haonan and Liu, Chaoqi and
+          Torralba, Antonio and Zhu, Yuke and Du, Yilun},
+  journal={arXiv preprint arXiv:2607.09648},
+  year={2026}
+}
+```

checkpoints/100000/config.json

@@ -0,0 +1,83 @@
+{
+    "type": "bspline_act",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.front": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.gripper": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                112
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "use_peft": false,
+    "push_to_hub": false,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": null,
+    "pretrained_revision": null,
+    "chunk_size": 16,
+    "n_action_steps": 16,
+    "normalization_mapping": {
+        "VISUAL": "MEAN_STD",
+        "STATE": "MEAN_STD",
+        "ACTION": "MEAN_STD"
+    },
+    "vision_backbone": "resnet18",
+    "pretrained_backbone_weights": "ResNet18_Weights.IMAGENET1K_V1",
+    "replace_final_stride_with_dilation": false,
+    "pre_norm": false,
+    "dim_model": 512,
+    "n_heads": 8,
+    "dim_feedforward": 3200,
+    "feedforward_activation": "relu",
+    "n_encoder_layers": 4,
+    "n_decoder_layers": 1,
+    "use_vae": true,
+    "latent_dim": 32,
+    "n_vae_encoder_layers": 4,
+    "temporal_ensemble_coeff": null,
+    "dropout": 0.1,
+    "kl_weight": 10.0,
+    "optimizer_lr": 2e-05,
+    "optimizer_weight_decay": 0.0001,
+    "optimizer_lr_backbone": 1e-05,
+    "bspline_degree": 3,
+    "bspline_chunk_size": 10,
+    "relative_knots": false,
+    "speed_up": 1.0,
+    "n_decoded_steps": 32,
+    "align_segments": true,
+    "align_error_threshold": 0.5,
+    "align_max_fraction": 0.2,
+    "align_exclude_last_dims": 1,
+    "replan_margin": 1.0,
+    "channels_last": true
+}

checkpoints/100000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a77d68bc97ef70d0034826afe7e58465472f1573a40581c60defae1cbe52b24c
+size 206360820

checkpoints/100000/policy_postprocessor.json

@@ -0,0 +1,12 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

checkpoints/100000/policy_preprocessor.json

@@ -0,0 +1,64 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              6
+            ]
+          },
+          "observation.images.front": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.images.gripper": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              112
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
+    }
+  ]
+}

checkpoints/100000/policy_preprocessor_step_3_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02568f8cfc013582b7eb3e22a63efb41cf4125b6adb6ad83935e8dbc2eb6e42c
+size 11592

checkpoints/100000/train_config.json

@@ -0,0 +1,230 @@
+{
+    "dataset": {
+        "repo_id": "aryankakad/CUPSTACKING_bspline",
+        "root": null,
+        "episodes": null,
+        "image_transforms": {
+            "enable": false,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "affine": {
+                    "weight": 1.0,
+                    "type": "RandomAffine",
+                    "kwargs": {
+                        "degrees": [
+                            -5.0,
+                            5.0
+                        ],
+                        "translate": [
+                            0.05,
+                            0.05
+                        ]
+                    }
+                }
+            }
+        },
+        "revision": null,
+        "use_imagenet_stats": true,
+        "video_backend": "torchcodec",
+        "return_uint8": false,
+        "depth_output_unit": "mm",
+        "streaming": false,
+        "eval_split": 0.0
+    },
+    "env": null,
+    "policy": {
+        "type": "bspline_act",
+        "n_obs_steps": 1,
+        "input_features": {
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            },
+            "observation.images.front": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            },
+            "observation.images.gripper": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    112
+                ]
+            }
+        },
+        "device": "cuda",
+        "use_amp": false,
+        "use_peft": false,
+        "push_to_hub": false,
+        "repo_id": null,
+        "private": null,
+        "tags": null,
+        "license": null,
+        "pretrained_path": null,
+        "pretrained_revision": null,
+        "chunk_size": 16,
+        "n_action_steps": 16,
+        "normalization_mapping": {
+            "VISUAL": "MEAN_STD",
+            "STATE": "MEAN_STD",
+            "ACTION": "MEAN_STD"
+        },
+        "vision_backbone": "resnet18",
+        "pretrained_backbone_weights": "ResNet18_Weights.IMAGENET1K_V1",
+        "replace_final_stride_with_dilation": false,
+        "pre_norm": false,
+        "dim_model": 512,
+        "n_heads": 8,
+        "dim_feedforward": 3200,
+        "feedforward_activation": "relu",
+        "n_encoder_layers": 4,
+        "n_decoder_layers": 1,
+        "use_vae": true,
+        "latent_dim": 32,
+        "n_vae_encoder_layers": 4,
+        "temporal_ensemble_coeff": null,
+        "dropout": 0.1,
+        "kl_weight": 10.0,
+        "optimizer_lr": 2e-05,
+        "optimizer_weight_decay": 0.0001,
+        "optimizer_lr_backbone": 1e-05,
+        "bspline_degree": 3,
+        "bspline_chunk_size": 10,
+        "relative_knots": false,
+        "speed_up": 1.0,
+        "n_decoded_steps": 32,
+        "align_segments": true,
+        "align_error_threshold": 0.5,
+        "align_max_fraction": 0.2,
+        "align_exclude_last_dims": 1,
+        "replan_margin": 1.0,
+        "channels_last": true
+    },
+    "reward_model": null,
+    "output_dir": "/home/sra/mimosa/lerobot/outputs/cupstack_bspline_act",
+    "job_name": "cupstack_bspline_act",
+    "resume": false,
+    "seed": 1000,
+    "cudnn_deterministic": false,
+    "num_workers": 8,
+    "batch_size": 32,
+    "prefetch_factor": 4,
+    "persistent_workers": true,
+    "steps": 100000,
+    "env_eval_freq": 20000,
+    "log_freq": 500,
+    "eval_steps": 0,
+    "max_eval_samples": 0,
+    "tolerance_s": 0.0001,
+    "save_checkpoint": true,
+    "save_freq": 10000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adamw",
+        "lr": 2e-05,
+        "weight_decay": 0.0001,
+        "grad_clip_norm": 10.0,
+        "betas": [
+            0.9,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": null,
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 22,
+        "use_async_envs": true,
+        "recording": false,
+        "recording_repo_id": null,
+        "recording_private": false
+    },
+    "wandb": {
+        "enable": false,
+        "disable_artifact": false,
+        "project": "lerobot",
+        "entity": null,
+        "notes": null,
+        "run_id": null,
+        "mode": null,
+        "add_tags": true
+    },
+    "peft": null,
+    "job": {
+        "target": null,
+        "image": "huggingface/lerobot-gpu:latest",
+        "timeout": "2d",
+        "detach": false,
+        "tags": []
+    },
+    "save_checkpoint_to_hub": false,
+    "sample_weighting": null,
+    "rename_map": {},
+    "checkpoint_path": null
+}