rdt2-base-p150 on Tenstorrent p150

Derived from robotics-diffusion-transformer/RDT2-FM (the flow-matching action expert, commit f803808e) and the vision-language backbone of robotics-diffusion-transformer/RDT2-VQ (Qwen2.5-VL-7B, commit ec7e7f3a), released by the RDT2 team with the paper arXiv:2602.03310 and the code thu-ml/RDT2, with unmodified weights. See the original model cards for training details.

RDT2-FM, a vision-language-action model for two-arm manipulation. The Qwen2.5-VL-7B backbone of RDT2-VQ reads two wrist-camera images and a language instruction; of it, the vision tower (0.68B parameters) and the language model up to layer 13 (3.58B including the embedding) run, and the key/value caches of language-model layers 0 to 13 condition a 0.49B-parameter RDT action expert (14 blocks, hidden size 1024) that turns Gaussian noise into an action chunk in 5 Euler flow-matching steps (about 4.6 TFLOP per call). In: the left_stereo and right_stereo cameras (384x384 each in the official setup, concatenated side by side and resized to 392x756), an instruction of up to 112 tokens and an optional 20-value proprio state. Out: 24 future steps at 30 Hz of 20 values for two UR5e arms (per arm: end-effector position and 6-D rotation relative to the current pose, gripper width), after the checkpoint's own action normalizer. The server includes the official RDT2 pre-processing (camera concatenation, the Qwen2.5-VL image resize and normalization, the chat-template prompt) and the action un-normalization. The vision tower and the action expert each run as one persistent device program, the language model as tt-nn operations with two fused device programs per layer (residual add with RMSNorm; head split with rotary embedding); each prompt length bucket (416, 448 or 512 tokens) is replayed from its own captured trace.

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware p150
License apache-2.0

Intended use

Research inference with the RDT2-FM checkpoint: action chunks for a bimanual UR5e setup with wrist cameras from images and an instruction, for example to inspect the policy, to compare implementations or to run offline evaluation, one request at a time.

Prerequisites

  • The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
  • Docker
  • Tenstorrent hardware: p150
  • About 16 GB of free disk: 11 GB of weights in the Hugging Face cache and 4.1 GB for the container image (a 0.9 GB download). tt serve downloads the two RDT2-VQ shards (10 GB); the server downloads the RDT2-FM action expert (1 GB), the action normalizer and the tokenizer file into the same cache on its first start
  • Network access to huggingface.co on the first start, or those three files fetched beforehand at the commits listed under Licensing

Quickstart

tt serve changh95/rdt2-base-p150

tt serve downloads the Docker image and the robotics-diffusion-transformer/RDT2-VQ at ec7e7f3aa4259224481c3cbd5816cc3f35354bec weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/rdt2-base-p150.

Send the shipped sample (two camera images of a public bimanual UR5e recording) to /predict, on the port tt serve reported:

S=https://huggingface.co/changh95/rdt2-base-p150/resolve/main/code/tt_rdt2/samples
curl -sLO $S/ur5e_bimanual_example_0_left_stereo.png
curl -sLO $S/ur5e_bimanual_example_0_right_stereo.png
printf '{"images":["%s","%s"],"instruction":"Pour water from the plastic bottle into the cup","seed":0}' \
  "$(base64 -w0 ur5e_bimanual_example_0_left_stereo.png)" "$(base64 -w0 ur5e_bimanual_example_0_right_stereo.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

The smoke test of this repo (Python 3, standard library only) sends the same request, compares the chunk with the stored output of the official RDT2 code for it, and checks a repeat, the 512-token bucket and the refusals:

hf download changh95/rdt2-base-p150 --include "code/tt_rdt2/*" --local-dir rdt2-base-p150
python3 rdt2-base-p150/code/tt_rdt2/server/smoke_test.py --url http://localhost:20000

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β€” its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one observation and returns one action chunk. GET /health and GET /info report the server, the device, the weights and the fixed configuration; GET /v1/models is a stub, not a chat API.

request field (POST /predict) meaning
images exactly 2 base64 PNG or JPEG images in the checkpoint's order: left_stereo (left-arm wrist camera), right_stereo (right-arm wrist camera). Two 384x384 images, or any equal-height pair whose side-by-side concatenation resizes to the same 392x756 input
instruction (or prompt) the task in natural language; required; at most 112 tokens; no special-token text such as <|im_end|>
state (optional) 20 floats, the raw proprio state; default zeros, as in the official RDT2-FM deployment
seed (optional) integer seed of the initial noise, 0 to 2^63-1; default 0. The same request with the same seed returns the same chunk
prompt_bucket (optional) 416, 448 or 512 tokens; default the smallest that holds the prompt (instructions of up to 16, 48 or 112 tokens)
num_steps (optional) must be 5

Response for the quickstart request (values rounded; arrays shortened):

{"actions": [[-0.00575, -0.00308, 0.00008, 0.99595, 0.00249, 0.00108, -0.00151, 0.98854, 0.00008, 0.0858,
              0.0007, 0.00139, -0.00084, 1.00048, 0.00236, -0.00036, -0.00398, 0.98988, -0.00151, 0.08427],
             ...],
 "shape": [24, 20], "normalized": false,
 "units": {"position": "metres, end-effector position relative to the current end-effector pose",
           "rotation": "6-D rotation representation (unitless), relative to the current end-effector pose",
           "gripper": "metres (gripper width)"},
 "action_layout": {"right_arm": {"position_m": [0, 3], "rotation_6d": [3, 9], "gripper_width_m": [9, 10]},
                   "left_arm": {"position_m": [10, 13], "rotation_6d": [13, 19], "gripper_width_m": [19, 20]}},
 "chunk": {"horizon": 24, "action_dim": 20, "rate_hz": 30},
 "prompt_bucket": 416, "num_tokens": 409, "num_steps": 5, "seed": 0, "model": "rdt2-base-p150",
 "timing_ms": {"preprocess": 27.8, "inference": 67.8, "total": 95.8}}
  • actions has 24 rows (future steps at 30 Hz) and 20 columns: the right arm in columns 0 to 9 and the left arm in 10 to 19, each [x, y, z] in metres and a 6-D rotation, both relative to the current end-effector pose, then the gripper width in metres. These are the values of the official RDTInferencer.step; the official README additionally rescales the gripper columns for its robot (/ 0.088 * 0.1), which this server does not.
  • timing_ms is measured in the server: preprocess (image decode, request checks, tokenization, pixel layout), inference (upload, device programs, read-back, un-normalization) and total; they vary from request to request (Expected performance has the distribution).
  • Errors: 400 with the reason for a camera count other than 2, an image that does not decode or has another geometry, an empty instruction or one with special-token text, a prompt longer than 512 tokens (never truncated), a bucket that does not hold the prompt, a state that is not 20 finite numbers, num_steps other than 5, a body above 32 MB; 422 for an unknown field or a wrong type; 500 with the exception text for a device failure.
  • The server accepts connections once every prompt bucket is warm (tt serve returns then); before that, connections fail. A start-up failure ends the server, with the reason in its log.
  • The Python API (from tt_rdt2 import RDT2FM) returns the same chunks; see code/PYTHON.md. It needs a tt-metal build with the patches in patches/.

Example output

RDT2-FM on a Tenstorrent p150: the two wrist-camera images of the sample and the x, y, z and gripper columns of both arms of the returned 24-step chunk, against the official code in float32

The chunk of the response above, returned by this package, and the chunk of the official RDT2 code run in float32 on a CPU for the same request: position, and gripper width of both arms over the 24 steps. The images are sample 0 of the Apache-2.0 dataset robotics-diffusion-transformer/BimanualUR5eExample; attributions for media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the official RDT2 code (RDTInferencer.step of thu-ml/RDT2 at commit 0797b4c6, same weights) run in float32 on a CPU, with the same cameras, instruction, state and initial noise. The reference column is the official code run in bfloat16, its deployment precision, against the same float32 run:

eval this package official code in bfloat16 cases
Shipped sample (real frame, 409 tokens): largest difference in position / 6-D rotation / gripper width 0.63 mm / 0.0009 / 0.05 mm 3.86 mm / 0.0060 / 0.24 mm 1
Shipped sample: 1 - PCC of the normalized chunk 1.8e-7 5.0e-6 1
Synthetic image pairs x 2 instructions x 2 seeds + 1 real frame (416-token bucket): largest position / rotation / gripper difference; lowest PCC 1.34 mm / 0.0035 / 1.68 mm; 0.99989 4.86 mm / 0.0072 / 1.64 mm; 0.99992 (3 of the cases) 13
Zero and 10 x N(0, 1) states, 409-, 436- and 492-token prompts (416 / 448 / 512 buckets) 0.64 mm / 0.0010 / 0.49 mm; 0.999996 8.88 mm / 0.0081 / 0.61 mm; 0.99998 5
Further real frames and two synthetic image pairs, 416- to 499-token prompts, zero, N(0, 1) and 10 x N(0, 1) states 1.64 mm / 0.0040 / 0.76 mm; 0.999990 6.71 mm / 0.0128 / 0.70 mm; 0.99997 8
Other real frames (one brightened, one dimmed), a random image pair, constant and noise images, a Chinese / English instruction; 404- to 512-token prompts; zero to 10 x N(0, 1) states 1.26 mm / 0.0058 / 0.49 mm; 0.999991 5.72 mm / 0.0103 / 0.61 mm; 0.999986 10

Positions and gripper widths are relative values in metres; PCC is over the 24 x 20 normalized chunk. Each of the 26 cases with a bfloat16 reference stays within 3 times (position, rotation, gripper) and 10 times (1 - PCC) the larger of two bfloat16 references for that case (the official bfloat16 run, and the bfloat16 action expert on the official bfloat16 key/value caches); the largest ratios are 1.7 (gripper width on a 492-token prompt with a zero state), 1.5 (gripper width on a 416-token real frame) and 1.6 for 1 - PCC (a synthetic image pair). The other 10 cases (synthetic image pairs of the 13-case row, without a bfloat16 run) stay within PCC 0.999 and 10 mm / 0.02 / 3 mm. The server returns the same chunks: all 36 cases were also sent to it as PNG images, twice each, within a run of 500 requests. The real frames are from the public recording BimanualUR5eExample (Apache-2.0). The RDT2-FM card publishes no task-success figure for these weights; no closed-loop or on-robot run was made with this package.

speed (batch 1, warm) bucket 416 bucket 448 bucket 512
Device time per call, trace replay of the three programs, median 69.8 ms 71.5 ms 75.4 ms
Python model() call on the sample (pre-processing + device + un-normalization), median 76.8 ms – –
Served POST /predict, server timing_ms.total, median (p99) 78.9 ms (83.5) 79.4 ms (84.6) 81.0 ms (85.3)
Served POST /predict, client round trip on loopback, median (p99) 83.3 ms (91.0) 83.9 ms (90.3) 85.2 ms (92.7)

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (120 calls per setting) and 500 consecutive served requests over every bucket with varied images, instructions, seeds and states (126 / 155 / 219 per bucket), one at a time. Of the device time at 416 tokens, the vision program takes 27.2 ms, the language model 32.0 ms and the action expert 13.0 ms; the host pre-processing (image resize, patch layout, tokenization) takes about 5.5 ms of the call, and image decoding and the request checks about 6 ms more in the server. The served latency depends on the host CPU: these runs shared the host with other jobs (load average about 4); in a run of 600 requests at a load average of 4.5 to 6, the server time was 79 to 82 ms at the median and 94 to 110 ms at p99 (decoding and pre-processing slow down; the device inference median stayed at 67 to 70 ms).

Limitations

  • Not for controlling a robot or any other physical system without the user's own validation: no closed-loop, simulator or on-robot run was made with this package, and the actions are relative end-effector targets for the bimanual UR5e setup the checkpoint was trained for.
  • Inference with the RDT2-FM weights only. RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: RDT2-VQ supplies only the vision-language backbone here. Fine-tuned checkpoints and training were not tested.
  • Fixed configuration: exactly 2 cameras in the checkpoint's order (left_stereo, right_stereo) whose side-by-side concatenation resizes to 392x756 (two 384x384 images); 5 flow-matching steps; prompt buckets of 416, 448 and 512 tokens, so instructions of at most 112 tokens (longer ones are refused, never truncated). The official code also runs other camera counts, image sizes and step counts; this package does not.
  • Batch 1, requests served one at a time. One p150 only: device dispatch runs on the chip's Ethernet cores, so multi-chip meshes are not supported; other boards were not tested.
  • Precision: bfloat16 weights; vision-tower matrix products at HiFi2 fidelity, language model and action expert at HiFi4, residual streams in float32. Against the official float32 run the chunks differ by up to 1.6 mm in position, 0.006 in rotation and 1.7 mm in gripper width on the 36 cases above (the official bfloat16 run, on the 26 cases it was run for: up to 8.9 mm, 0.013 and 1.6 mm). Not compared with a GPU run of the official code.
  • The default noise seed is fixed (0), so identical requests return identical chunks; send seed to sample other chunks.
  • An experimental language-model engine selected by RDT2_ENGINE=prefix=mk hangs the device on repeated calls at the 512-token bucket; it is refused unless RDT2_EXPERIMENTAL_PREFIX_MK=1, and no serving profile sets it.

Licensing

Related packages

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/rdt2-base-p150/discussions β€” that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from β€” code/ in this repo is byte-identical to the model code inside the image:

component built from
tt-metal 44d66500520fda9f2c7060c0f6b41ec48f7ab37e (dirty tree β€” the image includes uncommitted changes)
code/ digest 42b342b1c244bf43 (sha256, first 16 hex digits)
built 2026-11-19T16:53:56+00:00 by tt-model 0.1.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Model tree for changh95/rdt2-base-p150

Collection including changh95/rdt2-base-p150

Paper for changh95/rdt2-base-p150