rdt2-base-p150 on Tenstorrent p150
Derived from robotics-diffusion-transformer/RDT2-FM (the flow-matching action expert, commit
f803808e) and the vision-language backbone of robotics-diffusion-transformer/RDT2-VQ (Qwen2.5-VL-7B, commitec7e7f3a), released by the RDT2 team with the paper arXiv:2602.03310 and the code thu-ml/RDT2, with unmodified weights. See the original model cards for training details.
RDT2-FM, a vision-language-action model for two-arm manipulation. The Qwen2.5-VL-7B backbone of RDT2-VQ reads two wrist-camera images and a language instruction; of it, the vision tower (0.68B parameters) and the language model up to layer 13 (3.58B including the embedding) run, and the key/value caches of language-model layers 0 to 13 condition a 0.49B-parameter RDT action expert (14 blocks, hidden size 1024) that turns Gaussian noise into an action chunk in 5 Euler flow-matching steps (about 4.6 TFLOP per call). In: the left_stereo and right_stereo cameras (384x384 each in the official setup, concatenated side by side and resized to 392x756), an instruction of up to 112 tokens and an optional 20-value proprio state. Out: 24 future steps at 30 Hz of 20 values for two UR5e arms (per arm: end-effector position and 6-D rotation relative to the current pose, gripper width), after the checkpoint's own action normalizer. The server includes the official RDT2 pre-processing (camera concatenation, the Qwen2.5-VL image resize and normalization, the chat-template prompt) and the action un-normalization. The vision tower and the action expert each run as one persistent device program, the language model as tt-nn operations with two fused device programs per layer (residual add with RMSNorm; head split with rotary embedding); each prompt length bucket (416, 448 or 512 tokens) is replayed from its own captured trace.
Runs on Tenstorrent p150 (mesh P150).
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
At a glance
| Hardware | p150 |
| License | apache-2.0 |
Intended use
Research inference with the RDT2-FM checkpoint: action chunks for a bimanual UR5e setup with wrist cameras from images and an instruction, for example to inspect the policy, to compare implementations or to run offline evaluation, one request at a time.
Prerequisites
- The Tenstorrent CLI,
tt:uv tool install tenstorrent(or use tt-model alone, see the Quickstart) - Docker
- Tenstorrent hardware: p150
- About 16 GB of free disk: 11 GB of weights in the Hugging Face cache and 4.1 GB for the container image (a
0.9 GB download).
tt servedownloads the two RDT2-VQ shards (10 GB); the server downloads the RDT2-FM action expert (1 GB), the action normalizer and the tokenizer file into the same cache on its first start - Network access to huggingface.co on the first start, or those three files fetched beforehand at the commits listed under Licensing
Quickstart
tt serve changh95/rdt2-base-p150
tt serve downloads the Docker image and the robotics-diffusion-transformer/RDT2-VQ at ec7e7f3aa4259224481c3cbd5816cc3f35354bec weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.
Without tt-cli: tt-model serve changh95/rdt2-base-p150.
Send the shipped sample (two camera images of a public bimanual UR5e recording) to /predict, on the port
tt serve reported:
S=https://huggingface.co/changh95/rdt2-base-p150/resolve/main/code/tt_rdt2/samples
curl -sLO $S/ur5e_bimanual_example_0_left_stereo.png
curl -sLO $S/ur5e_bimanual_example_0_right_stereo.png
printf '{"images":["%s","%s"],"instruction":"Pour water from the plastic bottle into the cup","seed":0}' \
"$(base64 -w0 ur5e_bimanual_example_0_left_stereo.png)" "$(base64 -w0 ur5e_bimanual_example_0_right_stereo.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
The smoke test of this repo (Python 3, standard library only) sends the same request, compares the chunk with the stored output of the official RDT2 code for it, and checks a repeat, the 512-token bucket and the refusals:
hf download changh95/rdt2-base-p150 --include "code/tt_rdt2/*" --local-dir rdt2-base-p150
python3 rdt2-base-p150/code/tt_rdt2/server/smoke_test.py --url http://localhost:20000
Capabilities
This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β its request and response shapes are the model's own. See the author's notes for the payload it expects.
POST /predict takes one observation and returns one action chunk. GET /health and GET /info report the
server, the device, the weights and the fixed configuration; GET /v1/models is a stub, not a chat API.
request field (POST /predict) |
meaning |
|---|---|
images |
exactly 2 base64 PNG or JPEG images in the checkpoint's order: left_stereo (left-arm wrist camera), right_stereo (right-arm wrist camera). Two 384x384 images, or any equal-height pair whose side-by-side concatenation resizes to the same 392x756 input |
instruction (or prompt) |
the task in natural language; required; at most 112 tokens; no special-token text such as <|im_end|> |
state (optional) |
20 floats, the raw proprio state; default zeros, as in the official RDT2-FM deployment |
seed (optional) |
integer seed of the initial noise, 0 to 2^63-1; default 0. The same request with the same seed returns the same chunk |
prompt_bucket (optional) |
416, 448 or 512 tokens; default the smallest that holds the prompt (instructions of up to 16, 48 or 112 tokens) |
num_steps (optional) |
must be 5 |
Response for the quickstart request (values rounded; arrays shortened):
{"actions": [[-0.00575, -0.00308, 0.00008, 0.99595, 0.00249, 0.00108, -0.00151, 0.98854, 0.00008, 0.0858,
0.0007, 0.00139, -0.00084, 1.00048, 0.00236, -0.00036, -0.00398, 0.98988, -0.00151, 0.08427],
...],
"shape": [24, 20], "normalized": false,
"units": {"position": "metres, end-effector position relative to the current end-effector pose",
"rotation": "6-D rotation representation (unitless), relative to the current end-effector pose",
"gripper": "metres (gripper width)"},
"action_layout": {"right_arm": {"position_m": [0, 3], "rotation_6d": [3, 9], "gripper_width_m": [9, 10]},
"left_arm": {"position_m": [10, 13], "rotation_6d": [13, 19], "gripper_width_m": [19, 20]}},
"chunk": {"horizon": 24, "action_dim": 20, "rate_hz": 30},
"prompt_bucket": 416, "num_tokens": 409, "num_steps": 5, "seed": 0, "model": "rdt2-base-p150",
"timing_ms": {"preprocess": 27.8, "inference": 67.8, "total": 95.8}}
actionshas 24 rows (future steps at 30 Hz) and 20 columns: the right arm in columns 0 to 9 and the left arm in 10 to 19, each[x, y, z]in metres and a 6-D rotation, both relative to the current end-effector pose, then the gripper width in metres. These are the values of the officialRDTInferencer.step; the official README additionally rescales the gripper columns for its robot (/ 0.088 * 0.1), which this server does not.timing_msis measured in the server:preprocess(image decode, request checks, tokenization, pixel layout),inference(upload, device programs, read-back, un-normalization) andtotal; they vary from request to request (Expected performance has the distribution).- Errors: 400 with the reason for a camera count other than 2, an image that does not decode or has another
geometry, an empty instruction or one with special-token text, a prompt longer than 512 tokens (never
truncated), a bucket that does not hold the prompt, a state that is not 20 finite numbers,
num_stepsother than 5, a body above 32 MB; 422 for an unknown field or a wrong type; 500 with the exception text for a device failure. - The server accepts connections once every prompt bucket is warm (
tt servereturns then); before that, connections fail. A start-up failure ends the server, with the reason in its log. - The Python API (
from tt_rdt2 import RDT2FM) returns the same chunks; see code/PYTHON.md. It needs a tt-metal build with the patches in patches/.
Example output
The chunk of the response above, returned by this package, and the chunk of the official RDT2 code run in float32
on a CPU for the same request: position, and gripper width of both arms over the 24 steps. The images are sample 0
of the Apache-2.0 dataset
robotics-diffusion-transformer/BimanualUR5eExample;
attributions for media/: media/ATTRIBUTION.md.
Expected performance
Agreement with the official RDT2 code (RDTInferencer.step of thu-ml/RDT2 at
commit 0797b4c6, same weights) run in float32 on a CPU, with the same cameras, instruction, state and initial
noise. The reference column is the official code run in bfloat16, its deployment precision, against the same
float32 run:
| eval | this package | official code in bfloat16 | cases |
|---|---|---|---|
| Shipped sample (real frame, 409 tokens): largest difference in position / 6-D rotation / gripper width | 0.63 mm / 0.0009 / 0.05 mm | 3.86 mm / 0.0060 / 0.24 mm | 1 |
| Shipped sample: 1 - PCC of the normalized chunk | 1.8e-7 | 5.0e-6 | 1 |
| Synthetic image pairs x 2 instructions x 2 seeds + 1 real frame (416-token bucket): largest position / rotation / gripper difference; lowest PCC | 1.34 mm / 0.0035 / 1.68 mm; 0.99989 | 4.86 mm / 0.0072 / 1.64 mm; 0.99992 (3 of the cases) | 13 |
| Zero and 10 x N(0, 1) states, 409-, 436- and 492-token prompts (416 / 448 / 512 buckets) | 0.64 mm / 0.0010 / 0.49 mm; 0.999996 | 8.88 mm / 0.0081 / 0.61 mm; 0.99998 | 5 |
| Further real frames and two synthetic image pairs, 416- to 499-token prompts, zero, N(0, 1) and 10 x N(0, 1) states | 1.64 mm / 0.0040 / 0.76 mm; 0.999990 | 6.71 mm / 0.0128 / 0.70 mm; 0.99997 | 8 |
| Other real frames (one brightened, one dimmed), a random image pair, constant and noise images, a Chinese / English instruction; 404- to 512-token prompts; zero to 10 x N(0, 1) states | 1.26 mm / 0.0058 / 0.49 mm; 0.999991 | 5.72 mm / 0.0103 / 0.61 mm; 0.999986 | 10 |
Positions and gripper widths are relative values in metres; PCC is over the 24 x 20 normalized chunk. Each of the 26 cases with a bfloat16 reference stays within 3 times (position, rotation, gripper) and 10 times (1 - PCC) the larger of two bfloat16 references for that case (the official bfloat16 run, and the bfloat16 action expert on the official bfloat16 key/value caches); the largest ratios are 1.7 (gripper width on a 492-token prompt with a zero state), 1.5 (gripper width on a 416-token real frame) and 1.6 for 1 - PCC (a synthetic image pair). The other 10 cases (synthetic image pairs of the 13-case row, without a bfloat16 run) stay within PCC 0.999 and 10 mm / 0.02 / 3 mm. The server returns the same chunks: all 36 cases were also sent to it as PNG images, twice each, within a run of 500 requests. The real frames are from the public recording BimanualUR5eExample (Apache-2.0). The RDT2-FM card publishes no task-success figure for these weights; no closed-loop or on-robot run was made with this package.
| speed (batch 1, warm) | bucket 416 | bucket 448 | bucket 512 |
|---|---|---|---|
| Device time per call, trace replay of the three programs, median | 69.8 ms | 71.5 ms | 75.4 ms |
Python model() call on the sample (pre-processing + device + un-normalization), median |
76.8 ms | β | β |
Served POST /predict, server timing_ms.total, median (p99) |
78.9 ms (83.5) | 79.4 ms (84.6) | 81.0 ms (85.3) |
Served POST /predict, client round trip on loopback, median (p99) |
83.3 ms (91.0) | 83.9 ms (90.3) | 85.2 ms (92.7) |
Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (120
calls per setting) and 500 consecutive served requests over every bucket with varied images, instructions, seeds
and states (126 / 155 / 219 per bucket), one at a time. Of the device time at 416 tokens, the vision program
takes 27.2 ms, the language model 32.0 ms and the action expert 13.0 ms; the host pre-processing (image resize,
patch layout, tokenization) takes about 5.5 ms of the call, and image decoding and the request checks about 6 ms
more in the server. The served latency depends on the host CPU: these runs shared the host with other jobs (load
average about 4); in a run of 600 requests at a load average of 4.5 to 6, the server time was 79 to 82 ms at the
median and 94 to 110 ms at p99 (decoding and pre-processing slow down; the device inference median stayed at 67
to 70 ms).
Limitations
- Not for controlling a robot or any other physical system without the user's own validation: no closed-loop, simulator or on-robot run was made with this package, and the actions are relative end-effector targets for the bimanual UR5e setup the checkpoint was trained for.
- Inference with the RDT2-FM weights only. RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: RDT2-VQ supplies only the vision-language backbone here. Fine-tuned checkpoints and training were not tested.
- Fixed configuration: exactly 2 cameras in the checkpoint's order (
left_stereo,right_stereo) whose side-by-side concatenation resizes to 392x756 (two 384x384 images); 5 flow-matching steps; prompt buckets of 416, 448 and 512 tokens, so instructions of at most 112 tokens (longer ones are refused, never truncated). The official code also runs other camera counts, image sizes and step counts; this package does not. - Batch 1, requests served one at a time. One p150 only: device dispatch runs on the chip's Ethernet cores, so multi-chip meshes are not supported; other boards were not tested.
- Precision: bfloat16 weights; vision-tower matrix products at HiFi2 fidelity, language model and action expert at HiFi4, residual streams in float32. Against the official float32 run the chunks differ by up to 1.6 mm in position, 0.006 in rotation and 1.7 mm in gripper width on the 36 cases above (the official bfloat16 run, on the 26 cases it was run for: up to 8.9 mm, 0.013 and 1.6 mm). Not compared with a GPU run of the official code.
- The default noise seed is fixed (0), so identical requests return identical chunks; send
seedto sample other chunks. - An experimental language-model engine selected by
RDT2_ENGINE=prefix=mkhangs the device on repeated calls at the 512-token bucket; it is refused unlessRDT2_EXPERIMENTAL_PREFIX_MK=1, and no serving profile sets it.
Licensing
- Weights, not redistributed here: robotics-diffusion-transformer/RDT2-FM
at commit
f803808e3f79c77bba8bb4f3326e83d04e60783a, robotics-diffusion-transformer/RDT2-VQ at commitec7e7f3aa4259224481c3cbd5816cc3f35354bec(2 of its 4 shards) and the action normalizer file of robotics-diffusion-transformer/RVQActionTokenizer at commita6a614ae049cc3b9fa6bd4f5fadaeee3375d4dec, all Apache-2.0; the tokenizer file of Qwen/Qwen2.5-VL-7B-Instruct at commitcc594898137f460bfe9f0759e9844b3ce807cfb5, Apache-2.0. Downloaded at those commits bytt serveand by the server into the Hugging Face cache. - Port and serving code (
code/) and the tt-metal patches in patches/: Apache-2.0. The pre- and post-processing re-implement the inference path of thu-ml/RDT2 at commit0797b4c65e588e088d41602685e00dc2bc95852f(Apache-2.0); no upstream code is included. - Sample (
code/tt_rdt2/samples/) andmedia/sample_actions.png: sample 0 of robotics-diffusion-transformer/BimanualUR5eExample at revision5550bc13, Apache-2.0; media/ATTRIBUTION.md.
Related packages
- changh95/pi05-base-p150: the pi-0.5 base vision-language-action model on p150.
- changh95/GR00T-N1.7-3B-p150, changh95/GR00T-N1.6-3B-p150 and changh95/GR00T-N1.5-3B-p150: NVIDIA Isaac GR00T N1.x policies on p150.
- changh95/xvla-base-p150: X-VLA base on p150.
- changh95/RLDX-1-PT-p150 and changh95/RLDX-1-PT-IMG-p150: RLDX-1 on p150.
Feedback
Questions or problems with this package: open a discussion at https://huggingface.co/changh95/rdt2-base-p150/discussions β that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.
Provenance
The exact sources the image was built from β code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | 44d66500520fda9f2c7060c0f6b41ec48f7ab37e (dirty tree β the image includes uncommitted changes) |
code/ digest |
42b342b1c244bf43 (sha256, first 16 hex digits) |
| built | 2026-11-19T16:53:56+00:00 by tt-model 0.1.0 |
Model tree for changh95/rdt2-base-p150
Base model
robotics-diffusion-transformer/rdt-1b