rdt2-base-p150 / code /PYTHON.md
changh95's picture
tt-model push rdt2-base-p150 (container)
9d8d25c verified
|
Raw History Blame Contribute Delete
10.8 kB

rdt2-base-p150: Python API

RDT2-FM on one Tenstorrent Blackhole p150 (batch 1): two wrist-camera images, a language instruction and an optional 20-d proprio state in; a 24 x 20 action chunk (two UR5e arms, 24 steps at 30 Hz) out. The package is tt_rdt2; the HTTP server (tt_rdt2.server.app) calls exactly this API.

Install

The package needs a tt-metal build (ttnn) of commit 44d66500520fda9f2c7060c0f6b41ec48f7ab37e with the two patches of this repo's patches/ folder applied (tt-metal-eth-dispatch.patch: dispatch on the Ethernet cores, which frees the 12 x 10 Tensix grid; tt-metal-reshape-rm-sys1419.patch: a reshape guard). ttnn is not on PyPI. In that environment, from the repo root:

pip install -e .                    # the API: numpy<2, pillow, tokenizers, safetensors, huggingface_hub
pip install -e ".[server,test]"     # + fastapi / uvicorn / pydantic and pytest / httpx

Quickstart

from tt_rdt2 import RDT2FM, load_sample

sample = load_sample()                    # shipped sample: two 384x384 cameras, instruction, zero state
with RDT2FM.from_pretrained() as model:   # weights -> device -> every prompt bucket traced and warm
    out = model(sample["images"], sample["instruction"], state=sample["state"], seed=0)
print(out.actions.shape, out.prompt_bucket, out.num_tokens, out.timing_ms)

examples/quickstart.py is the runnable version.

RDT2FM.from_pretrained

RDT2FM.from_pretrained(model_id=None, revision=None, *, device_id=0, device=None, dispatch="eth", engine=None,
                       precision="baseline", weights_dirs=None, local_files_only=False, warmup=True,
                       allow_fallback=False, log_fn=None)
  1. Weights first, no device. Four pinned repositories are resolved to local directories:

    role repository @ revision files read
    vlm robotics-diffusion-transformer/RDT2-VQ @ ec7e7f3aa4259224481c3cbd5816cc3f35354bec config.json, model.safetensors.index.json, model-00001-of-00004.safetensors, model-00002-of-00004.safetensors (vision tower, embedding, LM layers 0..13: 9.96 GB)
    expert robotics-diffusion-transformer/RDT2-FM @ f803808e3f79c77bba8bb4f3326e83d04e60783a config.json, pytorch_model.bin (975 MB, loaded with torch.load(weights_only=True))
    normalizer robotics-diffusion-transformer/RVQActionTokenizer @ a6a614ae049cc3b9fa6bd4f5fadaeee3375d4dec umi_normalizer_wo_downsample_indentity_rot.pt (28 kB)
    processor Qwen/Qwen2.5-VL-7B-Instruct @ cc594898137f460bfe9f0759e9844b3ce807cfb5 tokenizer.json (no weights)

    Per role: weights_dirs[role] > $RDT2_VLM_DIR / $RDT2_EXPERT_DIR / $RDT2_NORMALIZER_DIR / $RDT2_PROCESSOR_DIR > (VLM only) a local directory given as model_id > hf_hub_download(repo, file, revision=<pin>) into the Hugging Face cache. A file already cached at a pinned commit is used without network access; on a network failure the cache is retried with local_files_only=True. model_id / revision replace the VLM repository (a fine-tuned RDT2-VQ with the same layout); the other three stay pinned.

  2. Device. dispatch="eth" (default) opens the chip with dispatch on the Ethernet cores: a 12 x 10 compute grid. dispatch="worker" uses Tensix dispatch (11 x 10). The grid is read from the device. If ETH dispatch is not available, allow_fallback=True opens with WORKER dispatch and a warning; the default raises. device= takes a caller-owned device (opened with a trace region of at least 512 MB); close() then leaves it open.

  3. Programs. VISION (Qwen2.5-VL ViT, 32 blocks + merger), PREFIX (embedding, image-token scatter, LM layers 0..12 + the K/V of layer 13) and EXPERT (state prologue + 5 Euler steps of the 14-block RDT expert) are loaded with all three prompt buckets (416 / 448 / 512 tokens) resident. engine selects per program the persistent megakernel (mk) or the ttnn op-by-op path (ttnn); the PREFIX also has fused (the ttnn graph with tuned matmul configurations and two fused generic_op programs per layer: residual add + RMSNorm, and head split + RoPE in fp32). The default is vision=mk, prefix=fused, expert=mk ("prefix=ttnn" etc. override it; ttnn is the fallback of each program; RDT2_ENGINE sets it from the environment). prefix=mk is experimental: it hangs the device on repeated trace replays at bucket 512, and it is refused unless RDT2_EXPERIMENTAL_PREFIX_MK=1 is set in the environment (development only).

  4. Warm-up (warmup=True): one Metal trace per bucket is captured (the first boot JIT-compiles the kernels), then one call per bucket runs; the first real call is as fast as later ones. model.warmup() is idempotent and returns the per-bucket warm call time in ms.

precision="baseline" is the gated preset (ViT at HiFi2, LM and expert at HiFi4, bf16 weights, fp32 residual streams). The other presets of tt_rdt2/tt/common.py are development presets and were not gated with the default engine: fast fails the long-prompt gate; the bfp8-weight presets were not re-gated with prefix=fused; the presets with a bf16 language-model residual (hifi4_bf16, hifi2_bf16, hifi2_lmmlp_bfp8, hifi2_bfp8, hifi2_bfp8_all) and baseline_kfused are refused by prefix=fused and need engine="prefix=ttnn".

Calling the model

out = model(images, instruction, state=None, seed=None, prompt_bucket=None, num_steps=None)   # -> ActionChunk
argument meaning
images exactly 2 cameras in the checkpoint's order [left_stereo, right_stereo] (left-arm wrist camera, right-arm wrist camera), or a mapping {"left_stereo": ..., "right_stereo": ...}. Each: an HxWx3 uint8 RGB array, a PIL image, PNG / JPEG bytes or a file path
instruction (or prompt=) the task in natural language; non-empty, no special-token text such as `<
state optional raw 20-d proprio state (not normalized); default zeros, which is what the official RDT2-FM deployment sends
seed optional integer in [0, 2**63 - 1]: the initial flow-matching noise is torch.randn((1, 24, 20), generator=torch.Generator().manual_seed(seed)); default 0. The same request with the same seed returns the same chunk, bit for bit
prompt_bucket optional 416 / 448 / 512; default the smallest bucket that holds the prompt. Pad keys are masked, so the bucket changes the chunk only within rounding
num_steps optional; must be 5

The pre-processing is the official RDT2 inferencer's (thu-ml/RDT2 RDTInferencer.step): the two cameras are concatenated side by side (left | right), resized to 392 x 756 with the uint8 bicubic-antialias kernel of the Qwen2.5-VL processor, CLIP-normalized and cut into 1512 patches (378 image tokens); the prompt is the Qwen chat template with one user turn (image, instruction) followed by <|im_start|>assistant\n<|quad_start|>, tokenized with the Qwen2.5-VL tokenizer. The prompt is 400 fixed tokens plus the instruction's: buckets 416 / 448 / 512 hold instructions of up to 16 / 48 / 112 tokens. Two 384 x 384 cameras are the official setup; any pair of equal-height images whose concatenation resizes to the same 392 x 756 grid (for example 2 x 380 x 380) is accepted and resized exactly as the official processor does. Other geometries are refused, never padded or cropped.

InputError (a ValueError) is raised, before anything reaches the device, for: a camera count other than 2, unequal heights, an unsupported geometry, an image that does not decode, an empty instruction or one with special-token text, a prompt longer than the largest bucket (never truncated), a bucket that does not hold the prompt, a state that is not 20 finite numbers, a seed out of range, num_steps != 5.

Output: ActionChunk

field content
actions (24, 20) float32: the official RDTInferencer.step output, un-normalized with the UMI normalizer. Columns 0..9 right arm, 10..19 left arm; per arm [x, y, z] end-effector position in metres, a 6-D rotation, the gripper width in metres. Positions and rotations are relative to the current end-effector pose; 24 steps at 30 Hz
actions_normalized (24, 20) float32: the expert's normalized output (actions = (normalized - offset) / scale)
prompt_bucket, num_tokens, seed, num_steps what ran
timing_ms preprocess (request checks, image decode, tokenization, pixel and id layout), inference (input upload, trace replay, read-back, un-normalization), total

out.to_dict() is the /predict JSON (actions, shape, normalized: false, units, action_layout, chunk, prompt_bucket, num_tokens, num_steps, seed, timing_ms). The official README rescales the gripper columns (a[:, 9|19] / 0.088 * 0.1) for its robot; this package returns RDTInferencer.step's values without that step.

Lifetime and threads

close() releases the traces and closes the device it opened (idempotent; also at interpreter exit and on with exit). Calls are serialized by a lock (one chip, batch 1); one model per chip and process. model.info holds the weights (repository, revision, files, source per role), engines, device (dispatch, grid), buckets, fixed configuration and the warm-up times; it is the server's /info.

Speed

On one p150 (ETH dispatch, 12 x 10 grid; 8-vCPU host), the shipped sample, warm, default engines, median of 120: device time per call (trace replay) 69.8 ms at bucket 416, 71.5 ms at 448, 75.4 ms at 512 (VISION 27.2, PREFIX 32.0, EXPERT 13.0 ms at 416); host pre-processing 5.5 ms; model() on the sample 76.8 ms. code/scripts/bench.py measures these, the per-stage traces and the transfers. Served numbers are in SERVING.md.

Tests

TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_rdt2/tests/test_api_host.py code/tt_rdt2/tests/test_server_host.py
python -m pytest -q -s code/tt_rdt2/tests/test_e2e_device.py       # chip: serving path vs the stored reference
python -m pytest -q -s code/tt_rdt2/tests/test_stress_device.py    # chip: 500-request stress gate
python -m pytest -q -s code/tt_rdt2/tests/test_tt_device.py        # chip: per-program gates (needs the golden data)

Host tests need no device (missing weights or golden data skip). The device suite test_tt_device.py and the golden cases of the stress gate need the official golden dumps (RDT2_GOLDEN_DIR), which are not shipped.

Not included

RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: this package runs RDT2-FM, which uses RDT2-VQ only as the vision-language backbone (K/V of layers 0..13).