| --- |
| library_name: lerobot |
| license: apache-2.0 |
| pipeline_tag: robotics |
| tags: |
| - lerobot |
| - robotics |
| - act |
| - robosynchallenge |
| - handle_basket |
| datasets: [] |
| model_name: act |
| --- |
| |
| # ACT_sim_handle_basket (RoboSynChallenge submission checkpoint) |
| |
| ACT (Action Chunking with Transformers) checkpoint for the RoboSynChallenge |
| **`handle_basket`** simulation task, re-packaged in the **LeRobot 0.4.4** layout so |
| that it loads without any legacy code path. |
| |
| This checkpoint is the imitation-learning component of our preliminary-round |
| submission. It is used both |
| |
| * as a stand-alone ACT baseline for `handle_basket`, and |
| * as the action-chunk generator that the submitted hybrid policy hands over to |
| after its analytic grasp/place stage. |
|
|
| ## Provenance |
|
|
| | Item | Value | |
| | --- | --- | |
| | Upstream weights | [`RoboSynChallenge/ACT_sim_handle_basket`](https://huggingface.co/RoboSynChallenge/ACT_sim_handle_basket) | |
| | Upstream format | legacy LeRobot `b883328` (normalization buffers stored inside `model.safetensors`) | |
| | This repository | same weights, migrated to the LeRobot 0.4.4 external pre/post-processor layout | |
| | Migration tool | `lerobot.processor.migrate_policy_normalization` with a key-restoration pass (see below) | |
|
|
| The migration is **format-only**: no weight is retrained, re-initialised or |
| fine-tuned. `model.safetensors` is byte-for-byte the upstream tensor payload. |
|
|
| ### Why a key-restoration pass was necessary |
|
|
| LeRobot's official migrator renames feature keys with |
| `".".join(parts[:-1]).replace("_", ".")`, which turns |
| `observation_images_cam_high` into `observation.images.cam.high` instead of |
| `observation.images.cam_high`. Those keys never match the policy features, so |
| the **image branch would silently stay un-normalised** (a mid-grey image stays |
| `0.5` instead of becoming roughly `0.066` after ImageNet normalisation). |
|
|
| The migrator was therefore followed by a pass that rewrites the state-file keys |
| back to the real feature names declared in `config.json`, and by an assertion |
| that normalisation actually applies: |
|
|
| ``` |
| image 0.5 -> ~0.066 (must NOT be ~0.5) |
| ``` |
|
|
| If you re-derive these files yourself, please re-run that assertion — a |
| checkpoint that looks fine can score 0 % purely because of this key mismatch. |
|
|
| ## Files |
|
|
| | File | Size | Role | |
| | --- | ---: | --- | |
| | `model.safetensors` | 206,560,504 B | ACT weights (upstream payload) | |
| | `config.json` | 2,117 B | ACT policy config, `chunk_size=50`, `n_action_steps=50` | |
| | `policy_preprocessor.json` | 1,780 B | LeRobot 0.4.4 preprocessing pipeline | |
| | `policy_preprocessor_step_3_normalizer_processor.safetensors` | 1,704 B | normalisation statistics | |
| | `policy_postprocessor.json` | 661 B | LeRobot 0.4.4 postprocessing pipeline | |
| | `policy_postprocessor_step_0_unnormalizer_processor.safetensors` | 1,704 B | de-normalisation statistics | |
|
|
| ## Interface |
|
|
| | | | |
| | --- | --- | |
| | Observation state | `observation.state`, shape `[14]` (`obs["robot"]["qpos"]`) | |
| | Observation images | `observation.images.cam_high`, `observation.images.cam_right_wrist`, `observation.images.cam_left_wrist`, each `[3, 480, 640]`, `float32` in `[0, 1]` | |
| | Action | `action`, shape `[14]`: `[left_arm(0-5), left_gripper(6), right_arm(7-12), right_gripper(13)]` | |
| | Action type | absolute joint positions | |
| | Chunk size | `50` predicted, `50` executed per inference (`n_action_steps`) | |
|
|
| ## Usage |
|
|
| ```python |
| from lerobot.policies.act.modeling_act import ACTPolicy |
| from lerobot.policies.factory import make_pre_post_processors |
| |
| policy = ACTPolicy.from_pretrained("<this-repo>") |
| pre, post = make_pre_post_processors( |
| policy_cfg=policy.config, |
| pretrained_path="<this-repo>", |
| preprocessor_overrides={"device_processor": {"device": policy.config.device}}, |
| postprocessor_overrides={"device_processor": {"device": "cpu"}}, |
| ) |
| batch = { |
| "observation.state": state, # [1, 14] float32 |
| "observation.images.cam_high": img_high, # [1, 3, 480, 640] float32 in [0, 1] |
| "observation.images.cam_right_wrist": img_rw, |
| "observation.images.cam_left_wrist": img_lw, |
| } |
| action = post(policy.select_action(pre(batch))) # [1, 14] |
| ``` |
|
|
| A ready-made adapter that handles both this 0.4.4 layout and the legacy |
| `b883328` layout is shipped in the submission repository alongside this card. |
|
|
| ## Evaluation protocol |
|
|
| * Task: `handle_basket`, setting `random` (domain randomisation on) |
| * Episode limit: 500 environment steps (`configs/handle_basket/random/gym_config.json`) |
| * Success criterion (official): the milk carton is inside the basket, the basket |
| is lifted and displaced more than 0.15 m along `+y`, and all of that holds for |
| 55 consecutive steps. |
|
|
| ## Limitations |
|
|
| * Simulation only. This checkpoint was trained on the released simulated |
| `handle_basket` demonstrations; it is not a real-robot policy. |
| * It is **not** a stand-alone solution for the whole task. On its own it |
| performs the grasp/place/carry sequence at the base rate of the released ACT |
| baseline; the submitted hybrid policy adds an analytic, RGB-only |
| grasp-and-place stage in front of it. |
| * Training data, seeds and hyper-parameters are those of the upstream released |
| checkpoint; only the on-disk format was changed here. |
|
|
