Card: results hold under benchmark 0.2.0's joint speed limit

bfb0ad7 verified 24 days ago

10.2 kB

---
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: robotics
tags:
  - robotics
  - vision-language-action
  - vla
  - imitation-learning
  - onnx
  - amp-toy-robotics
---
# AMP Tadpole

**A small vision-language-action model that drives a robot arm from three
camera images and a plain-English instruction.** It has 6.5M parameters
and runs on a laptop CPU (and, through ONNX, in a web browser).

![A scene from the AMP Toy Robotics Benchmark](scene.png)

AMP Tadpole is trained from scratch on the
[AMP Toy Robotics Benchmark](https://github.com/Adaptive-Machine-Patterns/amp-toy-robotics), and is
the first model of [Adaptive Machine Patterns](https://adaptive-machine-patterns.com).

**Try it in your browser:** the
[AMP Embodied AI Workbench Demo](https://huggingface.co/spaces/Adaptive-Machine-Patterns/amp-embodied-ai-workbench-demo) runs AMP
Tadpole live (through ONNX), and shows it inside, layer by layer.

> **v0.2.0 is a preview.** Later versions will be published under new
> tags (see [Versions](#versions)).

## What it does

Every 8 steps the model is given:

- the **top, side and isometric camera images** (512 × 512 RGB each);
- the **arm's state**: 6 joint angles and the gripper opening;
- **one instruction clause**, e.g. *"Pick up the lime star"*.

It returns **the next 8 steps** of joint angles and gripper opening. It is
never told where the objects are: it has to find them in the pictures and
work out which one the words mean. The benchmark's closed loop executes
the steps and asks again, until the model asks for no further motion.

Instructions: `pick up the X`, `move to the X`,
`move to the midpoint between X and Y`, `open the gripper`,
`close the gripper`, `return home`, chained with "then".

## Results

On the benchmark's standard 300 held-out scenes (never seen in training),
with the model driving the arm itself:

| clauses | pick | move | midpoint | open | close | home | all | path vs teacher |
|---|---|---|---|---|---|---|---|---|
| 898 | 93% | 94% | 75% | 100% | 100% | 100% | **93.7%** | 1.03 |

The reference teacher, a scripted controller that knows where everything
is, scores 99.4% on the same scenes.

Grounding tests (100 further held-out scenes; see the benchmark's README):

| named object | other objects | image swap | instruction swap | blank instruction |
|---|---|---|---|---|
| 0.98 | 0.31 | 1.00 | 1.00 | 0.95 |

Told "move to the X", its motion points the way the teacher's does
(0.98, where 1 is the same direction), and not toward the other objects
in the scene (0.31). When two objects swap places, or the instruction
names another object, its motion changes as much as the teacher's does
(1.00). So it uses both the cameras and the words.

These results are the same under the benchmark's version 0.2.0, which adds a
joint speed limit (no joint faster than 0.12 rad per step): it slowed 0.13% of
AMP Tadpole's steps, and no clause came out differently. AMP Tadpole takes 1.13
times the teacher's steps on the clauses both complete, and its path is 1.96
times a straight line (the teacher's: 1.90).

**How the checkpoint was chosen.** The training run checked itself every
2,500 steps on 100 held-out scenes, and those turned out to be the first
100 of the 300 above (seeds 9,010,000 to 9,010,099); v0.1.0 was picked
that way. For v0.2.0 the two final candidates (steps 300,000 and 330,000)
were compared on 300 separate validation scenes (seeds from 9,100,000),
where they tied at 94.2%; by a rule set beforehand, the tie went to the
longer-trained one. On the 200 benchmark scenes that no choice ever looked
at (608 clauses), v0.2.0 scores **93.1%**.

**These numbers are a line in the sand, not a ceiling.** They are
published as a reference point for small models on the benchmark:
something to beat.

These come from `amp-bench eval --policy amp-tadpole` on a CPU (about 40
minutes). Other hardware can differ in the last digits of the arithmetic,
so a clause or two may come out differently.

## Use it

With the benchmark (`pip install "amp-toy-robotics[packages]"`):

```bash
hf download Adaptive-Machine-Patterns/amp-tadpole --revision v0.2.1 --local-dir amp-tadpole
amp-bench eval --policy amp-tadpole
```

In Python, this repository is a model package:

```python
import importlib.util
from safetensors.torch import load_file
spec = importlib.util.spec_from_file_location("tadpole", "amp-tadpole/model.py")
tadpole = importlib.util.module_from_spec(spec); spec.loader.exec_module(tadpole)
model = tadpole.build()
model.load_state_dict(load_file("amp-tadpole/weights.safetensors"))
model.eval()
chunk = tadpole.predict(model, images, state, "Pick up the lime star")   # (8, 7)
```

With [LeRobot](https://github.com/huggingface/lerobot): install
[lerobot-policy-tadpole](https://github.com/Adaptive-Machine-Patterns/lerobot-policy-tadpole)
and load [amp-tadpole-lerobot](https://huggingface.co/Adaptive-Machine-Patterns/amp-tadpole-lerobot),
the same weights in LeRobot's format (`--policy.path=Adaptive-Machine-Patterns/amp-tadpole-lerobot`).

With ONNX Runtime (any language, or the browser): `amp-tadpole.onnx` takes
`images` uint8 (1, 3, 3, 512, 512), `tokens` int64 (1, 48), `token_mask`
bool (1, 48) and `state` float32 (1, 7), and returns `chunk` (1, 8, 7).
`instructions.json` gives the tokens for every canonical clause.

## How it works

| part | what it does |
|---|---|
| CNN, shared by the cameras | 512 × 512 pixels (plus two pixel-coordinate channels) to a 16 × 16 grid of tokens per camera |
| word embeddings | the clause's tokens (up to 48) |
| state token | the 7 numbers, with sines and cosines of the joint angles |
| 8 action tokens | learned; one per output step |
| transformer | 6 layers, width 256, 8 heads, over all 825 tokens |
| head | each action token gives one step's change; the changes are added up |

It predicts the 8 steps directly in one pass (no sampling), which suits a
teacher whose action is a fixed function of the state.

## Training

- **Data:** generated on the fly from scene seeds by the benchmark
  (`amp_toy_robotics.data`), labelled by the scripted teacher. No external
  data, no dataset files, no human demonstrations. Scenes from seed
  9,000,000 up were never used.
- **Inputs only:** the images, the state and the text, under the
  benchmark's rules. No object positions, no progress reward, no path
  penalty.
- **Recipe:** 330,000 steps, batch 64, AdamW (peak learning rate 3e-4, 1,000-step warm-up, cosine decay; the run was extended and resumed several times), the
  state hidden from half the samples (so the model must use the cameras),
  and an auxiliary loss in which each image token redraws its patch of
  the image. The redraw head is used only in training and is not included.
- **The word table** was cut to the 32 tokens the instructions use. This
  changes no output, and takes the model from 19.2M to 6.5M parameters.

## Limitations

- It knows only this simulated world: one arm, three fixed cameras, 8
  colours and 5 shapes. It has not seen a real camera.
- It reads only canonical clauses (`instructions.json`); the benchmark
  rewrites free text into canonical form first.
- The midpoint instruction is its weakest (75%; see Results). It is not
  overfitting: on teacher-labelled samples its errors are the same on
  scenes it trained on and scenes it never saw. Midpoint is simply less
  well learned. Its target is an empty spot between two objects, so there
  is nothing to aim at in the pictures, and it is 10% of the training
  clauses; its error was still shrinking when training ended.
- **What these results do not show.** The teacher is deterministic and
  unlimited and the cameras are fixed: that is why a 6.5M-parameter model
  does well here. It is not evidence against larger VLAs on benchmarks of
  human demonstrations, which are varied, finite and often use a wrist
  camera.
- All results come from one training run; it has not been repeated with
  other random seeds.

## Versions

| version | tag | what changed |
|---|---|---|
| 0.2.1 | `v0.2.1` | `model.py` now finds `instructions.json` when loaded from the Hugging Face cache; weights unchanged. AMP Tadpole for LeRobot: [amp-tadpole-lerobot](https://huggingface.co/Adaptive-Machine-Patterns/amp-tadpole-lerobot) |
| 0.2.0 | `v0.2.0` | step 330,000, the end of the training run: 93.7% (from 88.2%), midpoint 75% (from 63%); checkpoint chosen on separate validation scenes |
| 0.1.0 | `v0.1.0` | first preview: step 250,000. Its card said the 300 scenes were not used to choose the checkpoint; the training run's own checks used 100 of them (see Results) |

Pin a version with `revision="v0.2.1"` (or `--revision v0.2.1`).

## Licence

**Free for non-commercial use** under
[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (see
LICENSE and NOTICE). **For commercial use**, licences are available from
Adaptive Machine Patterns Ltd:
[adaptive-machine-patterns.com/embodied-ai-workbench](https://adaptive-machine-patterns.com/embodied-ai-workbench)
or amp@adaptive-machine-patterns.com.

**No warranty.** AMP Tadpole is a research preview, provided "as is",
without warranty of any kind, and Adaptive Machine Patterns Ltd accepts
no liability for its use, as set out in sections 5 and 6 of the licence.
It is not suitable for controlling real machinery, or for any use where
a wrong movement could cause harm or damage.

## Credits

- The token ids in `instructions.json` come from the tokenizer of
  [SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct)
  (Apache-2.0; Marafioti et al., *SmolVLM: Redefining small and efficient
  multimodal models*, arXiv:2504.05299). The model's weights are AMP's own,
  trained from scratch.
- Written with the help of AI coding assistants (Anthropic Claude, Qwen, OpenAI ChatGPT).

## Cite

```bibtex
@misc{amp_tadpole_2026,
  author    = {Whyte, Andy and {Adaptive Machine Patterns Ltd}},
  title     = {{AMP Tadpole}: a small vision-language-action model},
  year      = {2026},
  version   = {0.2.1},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/Adaptive-Machine-Patterns/amp-tadpole}
}
```

© 2026 Adaptive Machine Patterns Ltd, registered in England & Wales, no. 15618869.

	---
	license: cc-by-nc-4.0
	library_name: pytorch
	pipeline_tag: robotics
	tags:
	- robotics
	- vision-language-action
	- vla
	- imitation-learning
	- onnx
	- amp-toy-robotics
	---

	# AMP Tadpole

	**A small vision-language-action model that drives a robot arm from three
	camera images and a plain-English instruction.** It has 6.5M parameters
	and runs on a laptop CPU (and, through ONNX, in a web browser).

	![A scene from the AMP Toy Robotics Benchmark](scene.png)

	AMP Tadpole is trained from scratch on the
	[AMP Toy Robotics Benchmark](https://github.com/Adaptive-Machine-Patterns/amp-toy-robotics), and is
	the first model of [Adaptive Machine Patterns](https://adaptive-machine-patterns.com).

	Try it in your browser: the
	[AMP Embodied AI Workbench Demo](https://huggingface.co/spaces/Adaptive-Machine-Patterns/amp-embodied-ai-workbench-demo) runs AMP
	Tadpole live (through ONNX), and shows it inside, layer by layer.

	> v0.2.0 is a preview. Later versions will be published under new
	> tags (see [Versions](#versions)).

	## What it does

	Every 8 steps the model is given:

	- the top, side and isometric camera images (512 × 512 RGB each);
	- the arm's state: 6 joint angles and the gripper opening;
	- one instruction clause, e.g. "Pick up the lime star".

	It returns the next 8 steps of joint angles and gripper opening. It is
	never told where the objects are: it has to find them in the pictures and
	work out which one the words mean. The benchmark's closed loop executes
	the steps and asks again, until the model asks for no further motion.

	Instructions: `pick up the X`, `move to the X`,
	`move to the midpoint between X and Y`, `open the gripper`,
	`close the gripper`, `return home`, chained with "then".

	## Results

	On the benchmark's standard 300 held-out scenes (never seen in training),
	with the model driving the arm itself:

	\| clauses \| pick \| move \| midpoint \| open \| close \| home \| all \| path vs teacher \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| 898 \| 93% \| 94% \| 75% \| 100% \| 100% \| 100% \| 93.7% \| 1.03 \|

	The reference teacher, a scripted controller that knows where everything
	is, scores 99.4% on the same scenes.

	Grounding tests (100 further held-out scenes; see the benchmark's README):

	\| named object \| other objects \| image swap \| instruction swap \| blank instruction \|
	\|---\|---\|---\|---\|---\|
	\| 0.98 \| 0.31 \| 1.00 \| 1.00 \| 0.95 \|

	Told "move to the X", its motion points the way the teacher's does
	(0.98, where 1 is the same direction), and not toward the other objects
	in the scene (0.31). When two objects swap places, or the instruction
	names another object, its motion changes as much as the teacher's does
	(1.00). So it uses both the cameras and the words.

	These results are the same under the benchmark's version 0.2.0, which adds a
	joint speed limit (no joint faster than 0.12 rad per step): it slowed 0.13% of
	AMP Tadpole's steps, and no clause came out differently. AMP Tadpole takes 1.13
	times the teacher's steps on the clauses both complete, and its path is 1.96
	times a straight line (the teacher's: 1.90).

	How the checkpoint was chosen. The training run checked itself every
	2,500 steps on 100 held-out scenes, and those turned out to be the first
	100 of the 300 above (seeds 9,010,000 to 9,010,099); v0.1.0 was picked
	that way. For v0.2.0 the two final candidates (steps 300,000 and 330,000)
	were compared on 300 separate validation scenes (seeds from 9,100,000),
	where they tied at 94.2%; by a rule set beforehand, the tie went to the
	longer-trained one. On the 200 benchmark scenes that no choice ever looked
	at (608 clauses), v0.2.0 scores 93.1%.

	These numbers are a line in the sand, not a ceiling. They are
	published as a reference point for small models on the benchmark:
	something to beat.

	These come from `amp-bench eval --policy amp-tadpole` on a CPU (about 40
	minutes). Other hardware can differ in the last digits of the arithmetic,
	so a clause or two may come out differently.

	## Use it

	With the benchmark (`pip install "amp-toy-robotics[packages]"`):

	```bash
	hf download Adaptive-Machine-Patterns/amp-tadpole --revision v0.2.1 --local-dir amp-tadpole
	amp-bench eval --policy amp-tadpole
	```

	In Python, this repository is a model package:

	```python
	import importlib.util
	from safetensors.torch import load_file

	spec = importlib.util.spec_from_file_location("tadpole", "amp-tadpole/model.py")
	tadpole = importlib.util.module_from_spec(spec); spec.loader.exec_module(tadpole)
	model = tadpole.build()
	model.load_state_dict(load_file("amp-tadpole/weights.safetensors"))
	model.eval()

	chunk = tadpole.predict(model, images, state, "Pick up the lime star") # (8, 7)
	```

	With [LeRobot](https://github.com/huggingface/lerobot): install
	[lerobot-policy-tadpole](https://github.com/Adaptive-Machine-Patterns/lerobot-policy-tadpole)
	and load [amp-tadpole-lerobot](https://huggingface.co/Adaptive-Machine-Patterns/amp-tadpole-lerobot),
	the same weights in LeRobot's format (`--policy.path=Adaptive-Machine-Patterns/amp-tadpole-lerobot`).

	With ONNX Runtime (any language, or the browser): `amp-tadpole.onnx` takes
	`images` uint8 (1, 3, 3, 512, 512), `tokens` int64 (1, 48), `token_mask`
	bool (1, 48) and `state` float32 (1, 7), and returns `chunk` (1, 8, 7).
	`instructions.json` gives the tokens for every canonical clause.

	## How it works

	\| part \| what it does \|
	\|---\|---\|
	\| CNN, shared by the cameras \| 512 × 512 pixels (plus two pixel-coordinate channels) to a 16 × 16 grid of tokens per camera \|
	\| word embeddings \| the clause's tokens (up to 48) \|
	\| state token \| the 7 numbers, with sines and cosines of the joint angles \|
	\| 8 action tokens \| learned; one per output step \|
	\| transformer \| 6 layers, width 256, 8 heads, over all 825 tokens \|
	\| head \| each action token gives one step's change; the changes are added up \|

	It predicts the 8 steps directly in one pass (no sampling), which suits a
	teacher whose action is a fixed function of the state.

	## Training

	- Data: generated on the fly from scene seeds by the benchmark
	(`amp_toy_robotics.data`), labelled by the scripted teacher. No external
	data, no dataset files, no human demonstrations. Scenes from seed
	9,000,000 up were never used.
	- Inputs only: the images, the state and the text, under the
	benchmark's rules. No object positions, no progress reward, no path
	penalty.
	- Recipe: 330,000 steps, batch 64, AdamW (peak learning rate 3e-4, 1,000-step warm-up, cosine decay; the run was extended and resumed several times), the
	state hidden from half the samples (so the model must use the cameras),
	and an auxiliary loss in which each image token redraws its patch of
	the image. The redraw head is used only in training and is not included.
	- The word table was cut to the 32 tokens the instructions use. This
	changes no output, and takes the model from 19.2M to 6.5M parameters.

	## Limitations

	- It knows only this simulated world: one arm, three fixed cameras, 8
	colours and 5 shapes. It has not seen a real camera.
	- It reads only canonical clauses (`instructions.json`); the benchmark
	rewrites free text into canonical form first.
	- The midpoint instruction is its weakest (75%; see Results). It is not
	overfitting: on teacher-labelled samples its errors are the same on
	scenes it trained on and scenes it never saw. Midpoint is simply less
	well learned. Its target is an empty spot between two objects, so there
	is nothing to aim at in the pictures, and it is 10% of the training
	clauses; its error was still shrinking when training ended.
	- What these results do not show. The teacher is deterministic and
	unlimited and the cameras are fixed: that is why a 6.5M-parameter model
	does well here. It is not evidence against larger VLAs on benchmarks of
	human demonstrations, which are varied, finite and often use a wrist
	camera.
	- All results come from one training run; it has not been repeated with
	other random seeds.

	## Versions

	\| version \| tag \| what changed \|
	\|---\|---\|---\|
	\| 0.2.1 \| `v0.2.1` \| `model.py` now finds `instructions.json` when loaded from the Hugging Face cache; weights unchanged. AMP Tadpole for LeRobot: [amp-tadpole-lerobot](https://huggingface.co/Adaptive-Machine-Patterns/amp-tadpole-lerobot) \|
	\| 0.2.0 \| `v0.2.0` \| step 330,000, the end of the training run: 93.7% (from 88.2%), midpoint 75% (from 63%); checkpoint chosen on separate validation scenes \|
	\| 0.1.0 \| `v0.1.0` \| first preview: step 250,000. Its card said the 300 scenes were not used to choose the checkpoint; the training run's own checks used 100 of them (see Results) \|

	Pin a version with `revision="v0.2.1"` (or `--revision v0.2.1`).

	## Licence

	Free for non-commercial use under
	[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (see
	LICENSE and NOTICE). For commercial use, licences are available from
	Adaptive Machine Patterns Ltd:
	[adaptive-machine-patterns.com/embodied-ai-workbench](https://adaptive-machine-patterns.com/embodied-ai-workbench)
	or amp@adaptive-machine-patterns.com.

	No warranty. AMP Tadpole is a research preview, provided "as is",
	without warranty of any kind, and Adaptive Machine Patterns Ltd accepts
	no liability for its use, as set out in sections 5 and 6 of the licence.
	It is not suitable for controlling real machinery, or for any use where
	a wrong movement could cause harm or damage.

	## Credits

	- The token ids in `instructions.json` come from the tokenizer of
	[SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct)
	(Apache-2.0; Marafioti et al., *SmolVLM: Redefining small and efficient
	multimodal models*, arXiv:2504.05299). The model's weights are AMP's own,
	trained from scratch.
	- Written with the help of AI coding assistants (Anthropic Claude, Qwen, OpenAI ChatGPT).

	## Cite

	```bibtex
	@misc{amp_tadpole_2026,
	author = {Whyte, Andy and {Adaptive Machine Patterns Ltd}},
	title = {{AMP Tadpole}: a small vision-language-action model},
	year = {2026},
	version = {0.2.1},
	publisher = {Hugging Face},
	url = {https://huggingface.co/Adaptive-Machine-Patterns/amp-tadpole}
	}
	```

	© 2026 Adaptive Machine Patterns Ltd, registered in England & Wales, no. 15618869.