# rdt2-base-p150: Python API RDT2-FM on one Tenstorrent Blackhole p150 (batch 1): two wrist-camera images, a language instruction and an optional 20-d proprio state in; a 24 x 20 action chunk (two UR5e arms, 24 steps at 30 Hz) out. The package is `tt_rdt2`; the HTTP server (`tt_rdt2.server.app`) calls exactly this API. ## Install The package needs a tt-metal build (`ttnn`) of commit `44d66500520fda9f2c7060c0f6b41ec48f7ab37e` with the two patches of this repo's `patches/` folder applied (`tt-metal-eth-dispatch.patch`: dispatch on the Ethernet cores, which frees the 12 x 10 Tensix grid; `tt-metal-reshape-rm-sys1419.patch`: a reshape guard). `ttnn` is not on PyPI. In that environment, from the repo root: ```bash pip install -e . # the API: numpy<2, pillow, tokenizers, safetensors, huggingface_hub pip install -e ".[server,test]" # + fastapi / uvicorn / pydantic and pytest / httpx ``` ## Quickstart ```python from tt_rdt2 import RDT2FM, load_sample sample = load_sample() # shipped sample: two 384x384 cameras, instruction, zero state with RDT2FM.from_pretrained() as model: # weights -> device -> every prompt bucket traced and warm out = model(sample["images"], sample["instruction"], state=sample["state"], seed=0) print(out.actions.shape, out.prompt_bucket, out.num_tokens, out.timing_ms) ``` `examples/quickstart.py` is the runnable version. ## `RDT2FM.from_pretrained` ```python RDT2FM.from_pretrained(model_id=None, revision=None, *, device_id=0, device=None, dispatch="eth", engine=None, precision="baseline", weights_dirs=None, local_files_only=False, warmup=True, allow_fallback=False, log_fn=None) ``` 1. **Weights first, no device.** Four pinned repositories are resolved to local directories: | role | repository @ revision | files read | | --- | --- | --- | | `vlm` | `robotics-diffusion-transformer/RDT2-VQ` @ `ec7e7f3aa4259224481c3cbd5816cc3f35354bec` | `config.json`, `model.safetensors.index.json`, `model-00001-of-00004.safetensors`, `model-00002-of-00004.safetensors` (vision tower, embedding, LM layers 0..13: 9.96 GB) | | `expert` | `robotics-diffusion-transformer/RDT2-FM` @ `f803808e3f79c77bba8bb4f3326e83d04e60783a` | `config.json`, `pytorch_model.bin` (975 MB, loaded with `torch.load(weights_only=True)`) | | `normalizer` | `robotics-diffusion-transformer/RVQActionTokenizer` @ `a6a614ae049cc3b9fa6bd4f5fadaeee3375d4dec` | `umi_normalizer_wo_downsample_indentity_rot.pt` (28 kB) | | `processor` | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5` | `tokenizer.json` (no weights) | Per role: `weights_dirs[role]` > `$RDT2_VLM_DIR` / `$RDT2_EXPERT_DIR` / `$RDT2_NORMALIZER_DIR` / `$RDT2_PROCESSOR_DIR` > (VLM only) a local directory given as `model_id` > `hf_hub_download(repo, file, revision=)` into the Hugging Face cache. A file already cached at a pinned commit is used without network access; on a network failure the cache is retried with `local_files_only=True`. `model_id` / `revision` replace the VLM repository (a fine-tuned RDT2-VQ with the same layout); the other three stay pinned. 2. **Device.** `dispatch="eth"` (default) opens the chip with dispatch on the Ethernet cores: a 12 x 10 compute grid. `dispatch="worker"` uses Tensix dispatch (11 x 10). The grid is read from the device. If ETH dispatch is not available, `allow_fallback=True` opens with WORKER dispatch and a warning; the default raises. `device=` takes a caller-owned device (opened with a trace region of at least 512 MB); `close()` then leaves it open. 3. **Programs.** VISION (Qwen2.5-VL ViT, 32 blocks + merger), PREFIX (embedding, image-token scatter, LM layers 0..12 + the K/V of layer 13) and EXPERT (state prologue + 5 Euler steps of the 14-block RDT expert) are loaded with all three prompt buckets (416 / 448 / 512 tokens) resident. `engine` selects per program the persistent megakernel (`mk`) or the ttnn op-by-op path (`ttnn`); the PREFIX also has `fused` (the ttnn graph with tuned matmul configurations and two fused `generic_op` programs per layer: residual add + RMSNorm, and head split + RoPE in fp32). The default is `vision=mk, prefix=fused, expert=mk` (`"prefix=ttnn"` etc. override it; `ttnn` is the fallback of each program; `RDT2_ENGINE` sets it from the environment). `prefix=mk` is **experimental**: it hangs the device on repeated trace replays at bucket 512, and it is refused unless `RDT2_EXPERIMENTAL_PREFIX_MK=1` is set in the environment (development only). 4. **Warm-up** (`warmup=True`): one Metal trace per bucket is captured (the first boot JIT-compiles the kernels), then one call per bucket runs; the first real call is as fast as later ones. `model.warmup()` is idempotent and returns the per-bucket warm call time in ms. `precision="baseline"` is the gated preset (ViT at HiFi2, LM and expert at HiFi4, bf16 weights, fp32 residual streams). The other presets of `tt_rdt2/tt/common.py` are development presets and were not gated with the default engine: `fast` fails the long-prompt gate; the bfp8-weight presets were not re-gated with `prefix=fused`; the presets with a bf16 language-model residual (`hifi4_bf16`, `hifi2_bf16`, `hifi2_lmmlp_bfp8`, `hifi2_bfp8`, `hifi2_bfp8_all`) and `baseline_kfused` are refused by `prefix=fused` and need `engine="prefix=ttnn"`. ## Calling the model ```python out = model(images, instruction, state=None, seed=None, prompt_bucket=None, num_steps=None) # -> ActionChunk ``` | argument | meaning | | --- | --- | | `images` | exactly 2 cameras in the checkpoint's order `[left_stereo, right_stereo]` (left-arm wrist camera, right-arm wrist camera), or a mapping `{"left_stereo": ..., "right_stereo": ...}`. Each: an `HxWx3` uint8 RGB array, a PIL image, PNG / JPEG bytes or a file path | | `instruction` (or `prompt=`) | the task in natural language; non-empty, no special-token text such as `<|im_end|>`, at most 112 tokens | | `state` | optional raw 20-d proprio state (not normalized); default zeros, which is what the official RDT2-FM deployment sends | | `seed` | optional integer in [0, 2**63 - 1]: the initial flow-matching noise is `torch.randn((1, 24, 20), generator=torch.Generator().manual_seed(seed))`; default 0. The same request with the same seed returns the same chunk, bit for bit | | `prompt_bucket` | optional 416 / 448 / 512; default the smallest bucket that holds the prompt. Pad keys are masked, so the bucket changes the chunk only within rounding | | `num_steps` | optional; must be 5 | The pre-processing is the official RDT2 inferencer's (`thu-ml/RDT2` `RDTInferencer.step`): the two cameras are concatenated side by side (left | right), resized to 392 x 756 with the uint8 bicubic-antialias kernel of the Qwen2.5-VL processor, CLIP-normalized and cut into 1512 patches (378 image tokens); the prompt is the Qwen chat template with one user turn (image, instruction) followed by `<|im_start|>assistant\n<|quad_start|>`, tokenized with the Qwen2.5-VL tokenizer. The prompt is 400 fixed tokens plus the instruction's: buckets 416 / 448 / 512 hold instructions of up to 16 / 48 / 112 tokens. Two 384 x 384 cameras are the official setup; any pair of equal-height images whose concatenation resizes to the same 392 x 756 grid (for example 2 x 380 x 380) is accepted and resized exactly as the official processor does. Other geometries are refused, never padded or cropped. `InputError` (a `ValueError`) is raised, before anything reaches the device, for: a camera count other than 2, unequal heights, an unsupported geometry, an image that does not decode, an empty instruction or one with special-token text, a prompt longer than the largest bucket (never truncated), a bucket that does not hold the prompt, a state that is not 20 finite numbers, a seed out of range, `num_steps != 5`. ## Output: `ActionChunk` | field | content | | --- | --- | | `actions` | (24, 20) float32: the official `RDTInferencer.step` output, un-normalized with the UMI normalizer. Columns 0..9 right arm, 10..19 left arm; per arm `[x, y, z]` end-effector position in metres, a 6-D rotation, the gripper width in metres. Positions and rotations are relative to the current end-effector pose; 24 steps at 30 Hz | | `actions_normalized` | (24, 20) float32: the expert's normalized output (`actions = (normalized - offset) / scale`) | | `prompt_bucket`, `num_tokens`, `seed`, `num_steps` | what ran | | `timing_ms` | `preprocess` (request checks, image decode, tokenization, pixel and id layout), `inference` (input upload, trace replay, read-back, un-normalization), `total` | `out.to_dict()` is the `/predict` JSON (`actions`, `shape`, `normalized: false`, `units`, `action_layout`, `chunk`, `prompt_bucket`, `num_tokens`, `num_steps`, `seed`, `timing_ms`). The official README rescales the gripper columns (`a[:, 9|19] / 0.088 * 0.1`) for its robot; this package returns `RDTInferencer.step`'s values without that step. ## Lifetime and threads `close()` releases the traces and closes the device it opened (idempotent; also at interpreter exit and on `with` exit). Calls are serialized by a lock (one chip, batch 1); one model per chip and process. `model.info` holds the weights (repository, revision, files, source per role), engines, device (dispatch, grid), buckets, fixed configuration and the warm-up times; it is the server's `/info`. ## Speed On one p150 (ETH dispatch, 12 x 10 grid; 8-vCPU host), the shipped sample, warm, default engines, median of 120: device time per call (trace replay) 69.8 ms at bucket 416, 71.5 ms at 448, 75.4 ms at 512 (VISION 27.2, PREFIX 32.0, EXPERT 13.0 ms at 416); host pre-processing 5.5 ms; `model()` on the sample 76.8 ms. `code/scripts/bench.py` measures these, the per-stage traces and the transfers. Served numbers are in SERVING.md. ## Tests ```bash TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_rdt2/tests/test_api_host.py code/tt_rdt2/tests/test_server_host.py python -m pytest -q -s code/tt_rdt2/tests/test_e2e_device.py # chip: serving path vs the stored reference python -m pytest -q -s code/tt_rdt2/tests/test_stress_device.py # chip: 500-request stress gate python -m pytest -q -s code/tt_rdt2/tests/test_tt_device.py # chip: per-program gates (needs the golden data) ``` Host tests need no device (missing weights or golden data skip). The device suite `test_tt_device.py` and the golden cases of the stress gate need the official golden dumps (`RDT2_GOLDEN_DIR`), which are not shipped. ## Not included RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: this package runs RDT2-FM, which uses RDT2-VQ only as the vision-language backbone (K/V of layers 0..13).