rdt2-base-p150 / code /PYTHON.md
changh95's picture
tt-model push rdt2-base-p150 (container)
9d8d25c verified
|
Raw History Blame Contribute Delete
10.8 kB
# rdt2-base-p150: Python API
RDT2-FM on one Tenstorrent Blackhole p150 (batch 1): two wrist-camera images, a language instruction and an optional
20-d proprio state in; a 24 x 20 action chunk (two UR5e arms, 24 steps at 30 Hz) out. The package is `tt_rdt2`; the
HTTP server (`tt_rdt2.server.app`) calls exactly this API.
## Install
The package needs a tt-metal build (`ttnn`) of commit `44d66500520fda9f2c7060c0f6b41ec48f7ab37e` with the two patches
of this repo's `patches/` folder applied (`tt-metal-eth-dispatch.patch`: dispatch on the Ethernet cores, which frees the
12 x 10 Tensix grid; `tt-metal-reshape-rm-sys1419.patch`: a reshape guard). `ttnn` is not on PyPI. In that environment,
from the repo root:
```bash
pip install -e . # the API: numpy<2, pillow, tokenizers, safetensors, huggingface_hub
pip install -e ".[server,test]" # + fastapi / uvicorn / pydantic and pytest / httpx
```
## Quickstart
```python
from tt_rdt2 import RDT2FM, load_sample
sample = load_sample() # shipped sample: two 384x384 cameras, instruction, zero state
with RDT2FM.from_pretrained() as model: # weights -> device -> every prompt bucket traced and warm
out = model(sample["images"], sample["instruction"], state=sample["state"], seed=0)
print(out.actions.shape, out.prompt_bucket, out.num_tokens, out.timing_ms)
```
`examples/quickstart.py` is the runnable version.
## `RDT2FM.from_pretrained`
```python
RDT2FM.from_pretrained(model_id=None, revision=None, *, device_id=0, device=None, dispatch="eth", engine=None,
precision="baseline", weights_dirs=None, local_files_only=False, warmup=True,
allow_fallback=False, log_fn=None)
```
1. **Weights first, no device.** Four pinned repositories are resolved to local directories:
| role | repository @ revision | files read |
| --- | --- | --- |
| `vlm` | `robotics-diffusion-transformer/RDT2-VQ` @ `ec7e7f3aa4259224481c3cbd5816cc3f35354bec` | `config.json`, `model.safetensors.index.json`, `model-00001-of-00004.safetensors`, `model-00002-of-00004.safetensors` (vision tower, embedding, LM layers 0..13: 9.96 GB) |
| `expert` | `robotics-diffusion-transformer/RDT2-FM` @ `f803808e3f79c77bba8bb4f3326e83d04e60783a` | `config.json`, `pytorch_model.bin` (975 MB, loaded with `torch.load(weights_only=True)`) |
| `normalizer` | `robotics-diffusion-transformer/RVQActionTokenizer` @ `a6a614ae049cc3b9fa6bd4f5fadaeee3375d4dec` | `umi_normalizer_wo_downsample_indentity_rot.pt` (28 kB) |
| `processor` | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5` | `tokenizer.json` (no weights) |
Per role: `weights_dirs[role]` > `$RDT2_VLM_DIR` / `$RDT2_EXPERT_DIR` / `$RDT2_NORMALIZER_DIR` /
`$RDT2_PROCESSOR_DIR` > (VLM only) a local directory given as `model_id` > `hf_hub_download(repo, file,
revision=<pin>)` into the Hugging Face cache. A file already cached at a pinned commit is used without network
access; on a network failure the cache is retried with `local_files_only=True`. `model_id` / `revision` replace
the VLM repository (a fine-tuned RDT2-VQ with the same layout); the other three stay pinned.
2. **Device.** `dispatch="eth"` (default) opens the chip with dispatch on the Ethernet cores: a 12 x 10 compute grid.
`dispatch="worker"` uses Tensix dispatch (11 x 10). The grid is read from the device. If ETH dispatch is not
available, `allow_fallback=True` opens with WORKER dispatch and a warning; the default raises. `device=` takes a
caller-owned device (opened with a trace region of at least 512 MB); `close()` then leaves it open.
3. **Programs.** VISION (Qwen2.5-VL ViT, 32 blocks + merger), PREFIX (embedding, image-token scatter, LM layers
0..12 + the K/V of layer 13) and EXPERT (state prologue + 5 Euler steps of the 14-block RDT expert) are loaded
with all three prompt buckets (416 / 448 / 512 tokens) resident. `engine` selects per program the persistent
megakernel (`mk`) or the ttnn op-by-op path (`ttnn`); the PREFIX also has `fused` (the ttnn graph with tuned
matmul configurations and two fused `generic_op` programs per layer: residual add + RMSNorm, and head split +
RoPE in fp32). The default is `vision=mk, prefix=fused, expert=mk` (`"prefix=ttnn"` etc. override it; `ttnn` is
the fallback of each program; `RDT2_ENGINE` sets it from the environment). `prefix=mk` is **experimental**: it
hangs the device on repeated trace replays at bucket 512, and it is refused unless `RDT2_EXPERIMENTAL_PREFIX_MK=1`
is set in the environment (development only).
4. **Warm-up** (`warmup=True`): one Metal trace per bucket is captured (the first boot JIT-compiles the kernels), then
one call per bucket runs; the first real call is as fast as later ones. `model.warmup()` is idempotent and returns
the per-bucket warm call time in ms.
`precision="baseline"` is the gated preset (ViT at HiFi2, LM and expert at HiFi4, bf16 weights, fp32 residual
streams). The other presets of `tt_rdt2/tt/common.py` are development presets and were not gated with the default
engine: `fast` fails the long-prompt gate; the bfp8-weight presets were not re-gated with `prefix=fused`; the
presets with a bf16 language-model residual (`hifi4_bf16`, `hifi2_bf16`, `hifi2_lmmlp_bfp8`, `hifi2_bfp8`,
`hifi2_bfp8_all`) and `baseline_kfused` are refused by `prefix=fused` and need `engine="prefix=ttnn"`.
## Calling the model
```python
out = model(images, instruction, state=None, seed=None, prompt_bucket=None, num_steps=None) # -> ActionChunk
```
| argument | meaning |
| --- | --- |
| `images` | exactly 2 cameras in the checkpoint's order `[left_stereo, right_stereo]` (left-arm wrist camera, right-arm wrist camera), or a mapping `{"left_stereo": ..., "right_stereo": ...}`. Each: an `HxWx3` uint8 RGB array, a PIL image, PNG / JPEG bytes or a file path |
| `instruction` (or `prompt=`) | the task in natural language; non-empty, no special-token text such as `<|im_end|>`, at most 112 tokens |
| `state` | optional raw 20-d proprio state (not normalized); default zeros, which is what the official RDT2-FM deployment sends |
| `seed` | optional integer in [0, 2**63 - 1]: the initial flow-matching noise is `torch.randn((1, 24, 20), generator=torch.Generator().manual_seed(seed))`; default 0. The same request with the same seed returns the same chunk, bit for bit |
| `prompt_bucket` | optional 416 / 448 / 512; default the smallest bucket that holds the prompt. Pad keys are masked, so the bucket changes the chunk only within rounding |
| `num_steps` | optional; must be 5 |
The pre-processing is the official RDT2 inferencer's (`thu-ml/RDT2` `RDTInferencer.step`): the two cameras are
concatenated side by side (left | right), resized to 392 x 756 with the uint8 bicubic-antialias kernel of the Qwen2.5-VL
processor, CLIP-normalized and cut into 1512 patches (378 image tokens); the prompt is the Qwen chat template with one
user turn (image, instruction) followed by `<|im_start|>assistant\n<|quad_start|>`, tokenized with the Qwen2.5-VL
tokenizer. The prompt is 400 fixed tokens plus the instruction's: buckets 416 / 448 / 512 hold instructions of up to
16 / 48 / 112 tokens. Two 384 x 384 cameras are the official setup; any pair of equal-height images whose
concatenation resizes to the same 392 x 756 grid (for example 2 x 380 x 380) is accepted and resized exactly as the
official processor does. Other geometries are refused, never padded or cropped.
`InputError` (a `ValueError`) is raised, before anything reaches the device, for: a camera count other than 2,
unequal heights, an unsupported geometry, an image that does not decode, an empty instruction or one with
special-token text, a prompt longer than the largest bucket (never truncated), a bucket that does not hold the prompt,
a state that is not 20 finite numbers, a seed out of range, `num_steps != 5`.
## Output: `ActionChunk`
| field | content |
| --- | --- |
| `actions` | (24, 20) float32: the official `RDTInferencer.step` output, un-normalized with the UMI normalizer. Columns 0..9 right arm, 10..19 left arm; per arm `[x, y, z]` end-effector position in metres, a 6-D rotation, the gripper width in metres. Positions and rotations are relative to the current end-effector pose; 24 steps at 30 Hz |
| `actions_normalized` | (24, 20) float32: the expert's normalized output (`actions = (normalized - offset) / scale`) |
| `prompt_bucket`, `num_tokens`, `seed`, `num_steps` | what ran |
| `timing_ms` | `preprocess` (request checks, image decode, tokenization, pixel and id layout), `inference` (input upload, trace replay, read-back, un-normalization), `total` |
`out.to_dict()` is the `/predict` JSON (`actions`, `shape`, `normalized: false`, `units`, `action_layout`, `chunk`,
`prompt_bucket`, `num_tokens`, `num_steps`, `seed`, `timing_ms`). The official README rescales the gripper columns
(`a[:, 9|19] / 0.088 * 0.1`) for its robot; this package returns `RDTInferencer.step`'s values without that step.
## Lifetime and threads
`close()` releases the traces and closes the device it opened (idempotent; also at interpreter exit and on `with`
exit). Calls are serialized by a lock (one chip, batch 1); one model per chip and process. `model.info` holds the
weights (repository, revision, files, source per role), engines, device (dispatch, grid), buckets, fixed
configuration and the warm-up times; it is the server's `/info`.
## Speed
On one p150 (ETH dispatch, 12 x 10 grid; 8-vCPU host), the shipped sample, warm, default engines, median of 120:
device time per call (trace replay) 69.8 ms at bucket 416, 71.5 ms at 448, 75.4 ms at 512 (VISION 27.2, PREFIX
32.0, EXPERT 13.0 ms at 416); host pre-processing 5.5 ms; `model()` on the sample 76.8 ms. `code/scripts/bench.py`
measures these, the per-stage traces and the transfers. Served numbers are in SERVING.md.
## Tests
```bash
TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_rdt2/tests/test_api_host.py code/tt_rdt2/tests/test_server_host.py
python -m pytest -q -s code/tt_rdt2/tests/test_e2e_device.py # chip: serving path vs the stored reference
python -m pytest -q -s code/tt_rdt2/tests/test_stress_device.py # chip: 500-request stress gate
python -m pytest -q -s code/tt_rdt2/tests/test_tt_device.py # chip: per-program gates (needs the golden data)
```
Host tests need no device (missing weights or golden data skip). The device suite `test_tt_device.py` and the golden
cases of the stress gate need the official golden dumps (`RDT2_GOLDEN_DIR`), which are not shipped.
## Not included
RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: this package
runs RDT2-FM, which uses RDT2-VQ only as the vision-language backbone (K/V of layers 0..13).