Download code/PYTHON.md from changh95/rdt2-base-p150: direct link, hf CLI and curl.
- Browser
- Download file 10.8 kB
-
https://huggingface.co/changh95/rdt2-base-p150/resolve/main/code/PYTHON.md
- Command line
-
hf download hf://changh95/rdt2-base-p150/code/PYTHON.md
-
curl -L -o PYTHON.md https://huggingface.co/changh95/rdt2-base-p150/resolve/main/code/PYTHON.md
rdt2-base-p150: Python API
RDT2-FM on one Tenstorrent Blackhole p150 (batch 1): two wrist-camera images, a language instruction and an optional
20-d proprio state in; a 24 x 20 action chunk (two UR5e arms, 24 steps at 30 Hz) out. The package is tt_rdt2; the
HTTP server (tt_rdt2.server.app) calls exactly this API.
Install
The package needs a tt-metal build (ttnn) of commit 44d66500520fda9f2c7060c0f6b41ec48f7ab37e with the two patches
of this repo's patches/ folder applied (tt-metal-eth-dispatch.patch: dispatch on the Ethernet cores, which frees the
12 x 10 Tensix grid; tt-metal-reshape-rm-sys1419.patch: a reshape guard). ttnn is not on PyPI. In that environment,
from the repo root:
pip install -e . # the API: numpy<2, pillow, tokenizers, safetensors, huggingface_hub
pip install -e ".[server,test]" # + fastapi / uvicorn / pydantic and pytest / httpx
Quickstart
from tt_rdt2 import RDT2FM, load_sample
sample = load_sample() # shipped sample: two 384x384 cameras, instruction, zero state
with RDT2FM.from_pretrained() as model: # weights -> device -> every prompt bucket traced and warm
out = model(sample["images"], sample["instruction"], state=sample["state"], seed=0)
print(out.actions.shape, out.prompt_bucket, out.num_tokens, out.timing_ms)
examples/quickstart.py is the runnable version.
RDT2FM.from_pretrained
RDT2FM.from_pretrained(model_id=None, revision=None, *, device_id=0, device=None, dispatch="eth", engine=None,
precision="baseline", weights_dirs=None, local_files_only=False, warmup=True,
allow_fallback=False, log_fn=None)
Weights first, no device. Four pinned repositories are resolved to local directories:
role repository @ revision files read vlmrobotics-diffusion-transformer/RDT2-VQ@ec7e7f3aa4259224481c3cbd5816cc3f35354becconfig.json,model.safetensors.index.json,model-00001-of-00004.safetensors,model-00002-of-00004.safetensors(vision tower, embedding, LM layers 0..13: 9.96 GB)expertrobotics-diffusion-transformer/RDT2-FM@f803808e3f79c77bba8bb4f3326e83d04e60783aconfig.json,pytorch_model.bin(975 MB, loaded withtorch.load(weights_only=True))normalizerrobotics-diffusion-transformer/RVQActionTokenizer@a6a614ae049cc3b9fa6bd4f5fadaeee3375d4decumi_normalizer_wo_downsample_indentity_rot.pt(28 kB)processorQwen/Qwen2.5-VL-7B-Instruct@cc594898137f460bfe9f0759e9844b3ce807cfb5tokenizer.json(no weights)Per role:
weights_dirs[role]>$RDT2_VLM_DIR/$RDT2_EXPERT_DIR/$RDT2_NORMALIZER_DIR/$RDT2_PROCESSOR_DIR> (VLM only) a local directory given asmodel_id>hf_hub_download(repo, file, revision=<pin>)into the Hugging Face cache. A file already cached at a pinned commit is used without network access; on a network failure the cache is retried withlocal_files_only=True.model_id/revisionreplace the VLM repository (a fine-tuned RDT2-VQ with the same layout); the other three stay pinned.Device.
dispatch="eth"(default) opens the chip with dispatch on the Ethernet cores: a 12 x 10 compute grid.dispatch="worker"uses Tensix dispatch (11 x 10). The grid is read from the device. If ETH dispatch is not available,allow_fallback=Trueopens with WORKER dispatch and a warning; the default raises.device=takes a caller-owned device (opened with a trace region of at least 512 MB);close()then leaves it open.Programs. VISION (Qwen2.5-VL ViT, 32 blocks + merger), PREFIX (embedding, image-token scatter, LM layers 0..12 + the K/V of layer 13) and EXPERT (state prologue + 5 Euler steps of the 14-block RDT expert) are loaded with all three prompt buckets (416 / 448 / 512 tokens) resident.
engineselects per program the persistent megakernel (mk) or the ttnn op-by-op path (ttnn); the PREFIX also hasfused(the ttnn graph with tuned matmul configurations and two fusedgeneric_opprograms per layer: residual add + RMSNorm, and head split + RoPE in fp32). The default isvision=mk, prefix=fused, expert=mk("prefix=ttnn"etc. override it;ttnnis the fallback of each program;RDT2_ENGINEsets it from the environment).prefix=mkis experimental: it hangs the device on repeated trace replays at bucket 512, and it is refused unlessRDT2_EXPERIMENTAL_PREFIX_MK=1is set in the environment (development only).Warm-up (
warmup=True): one Metal trace per bucket is captured (the first boot JIT-compiles the kernels), then one call per bucket runs; the first real call is as fast as later ones.model.warmup()is idempotent and returns the per-bucket warm call time in ms.
precision="baseline" is the gated preset (ViT at HiFi2, LM and expert at HiFi4, bf16 weights, fp32 residual
streams). The other presets of tt_rdt2/tt/common.py are development presets and were not gated with the default
engine: fast fails the long-prompt gate; the bfp8-weight presets were not re-gated with prefix=fused; the
presets with a bf16 language-model residual (hifi4_bf16, hifi2_bf16, hifi2_lmmlp_bfp8, hifi2_bfp8,
hifi2_bfp8_all) and baseline_kfused are refused by prefix=fused and need engine="prefix=ttnn".
Calling the model
out = model(images, instruction, state=None, seed=None, prompt_bucket=None, num_steps=None) # -> ActionChunk
| argument | meaning |
|---|---|
images |
exactly 2 cameras in the checkpoint's order [left_stereo, right_stereo] (left-arm wrist camera, right-arm wrist camera), or a mapping {"left_stereo": ..., "right_stereo": ...}. Each: an HxWx3 uint8 RGB array, a PIL image, PNG / JPEG bytes or a file path |
instruction (or prompt=) |
the task in natural language; non-empty, no special-token text such as `< |
state |
optional raw 20-d proprio state (not normalized); default zeros, which is what the official RDT2-FM deployment sends |
seed |
optional integer in [0, 2**63 - 1]: the initial flow-matching noise is torch.randn((1, 24, 20), generator=torch.Generator().manual_seed(seed)); default 0. The same request with the same seed returns the same chunk, bit for bit |
prompt_bucket |
optional 416 / 448 / 512; default the smallest bucket that holds the prompt. Pad keys are masked, so the bucket changes the chunk only within rounding |
num_steps |
optional; must be 5 |
The pre-processing is the official RDT2 inferencer's (thu-ml/RDT2 RDTInferencer.step): the two cameras are
concatenated side by side (left | right), resized to 392 x 756 with the uint8 bicubic-antialias kernel of the Qwen2.5-VL
processor, CLIP-normalized and cut into 1512 patches (378 image tokens); the prompt is the Qwen chat template with one
user turn (image, instruction) followed by <|im_start|>assistant\n<|quad_start|>, tokenized with the Qwen2.5-VL
tokenizer. The prompt is 400 fixed tokens plus the instruction's: buckets 416 / 448 / 512 hold instructions of up to
16 / 48 / 112 tokens. Two 384 x 384 cameras are the official setup; any pair of equal-height images whose
concatenation resizes to the same 392 x 756 grid (for example 2 x 380 x 380) is accepted and resized exactly as the
official processor does. Other geometries are refused, never padded or cropped.
InputError (a ValueError) is raised, before anything reaches the device, for: a camera count other than 2,
unequal heights, an unsupported geometry, an image that does not decode, an empty instruction or one with
special-token text, a prompt longer than the largest bucket (never truncated), a bucket that does not hold the prompt,
a state that is not 20 finite numbers, a seed out of range, num_steps != 5.
Output: ActionChunk
| field | content |
|---|---|
actions |
(24, 20) float32: the official RDTInferencer.step output, un-normalized with the UMI normalizer. Columns 0..9 right arm, 10..19 left arm; per arm [x, y, z] end-effector position in metres, a 6-D rotation, the gripper width in metres. Positions and rotations are relative to the current end-effector pose; 24 steps at 30 Hz |
actions_normalized |
(24, 20) float32: the expert's normalized output (actions = (normalized - offset) / scale) |
prompt_bucket, num_tokens, seed, num_steps |
what ran |
timing_ms |
preprocess (request checks, image decode, tokenization, pixel and id layout), inference (input upload, trace replay, read-back, un-normalization), total |
out.to_dict() is the /predict JSON (actions, shape, normalized: false, units, action_layout, chunk,
prompt_bucket, num_tokens, num_steps, seed, timing_ms). The official README rescales the gripper columns
(a[:, 9|19] / 0.088 * 0.1) for its robot; this package returns RDTInferencer.step's values without that step.
Lifetime and threads
close() releases the traces and closes the device it opened (idempotent; also at interpreter exit and on with
exit). Calls are serialized by a lock (one chip, batch 1); one model per chip and process. model.info holds the
weights (repository, revision, files, source per role), engines, device (dispatch, grid), buckets, fixed
configuration and the warm-up times; it is the server's /info.
Speed
On one p150 (ETH dispatch, 12 x 10 grid; 8-vCPU host), the shipped sample, warm, default engines, median of 120:
device time per call (trace replay) 69.8 ms at bucket 416, 71.5 ms at 448, 75.4 ms at 512 (VISION 27.2, PREFIX
32.0, EXPERT 13.0 ms at 416); host pre-processing 5.5 ms; model() on the sample 76.8 ms. code/scripts/bench.py
measures these, the per-stage traces and the transfers. Served numbers are in SERVING.md.
Tests
TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_rdt2/tests/test_api_host.py code/tt_rdt2/tests/test_server_host.py
python -m pytest -q -s code/tt_rdt2/tests/test_e2e_device.py # chip: serving path vs the stored reference
python -m pytest -q -s code/tt_rdt2/tests/test_stress_device.py # chip: 500-request stress gate
python -m pytest -q -s code/tt_rdt2/tests/test_tt_device.py # chip: per-program gates (needs the golden data)
Host tests need no device (missing weights or golden data skip). The device suite test_tt_device.py and the golden
cases of the stress gate need the official golden dumps (RDT2_GOLDEN_DIR), which are not shipped.
Not included
RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: this package runs RDT2-FM, which uses RDT2-VQ only as the vision-language backbone (K/V of layers 0..13).