|
Download code/PYTHON.md from changh95/rdt2-base-p150: direct link, hf CLI and curl.
- Browser
- Download file 10.8 kB
-
https://huggingface.co/changh95/rdt2-base-p150/resolve/main/code/PYTHON.md
- Command line
-
hf download hf://changh95/rdt2-base-p150/code/PYTHON.md
-
curl -L -o PYTHON.md https://huggingface.co/changh95/rdt2-base-p150/resolve/main/code/PYTHON.md
10.8 kB
| # rdt2-base-p150: Python API | |
| RDT2-FM on one Tenstorrent Blackhole p150 (batch 1): two wrist-camera images, a language instruction and an optional | |
| 20-d proprio state in; a 24 x 20 action chunk (two UR5e arms, 24 steps at 30 Hz) out. The package is `tt_rdt2`; the | |
| HTTP server (`tt_rdt2.server.app`) calls exactly this API. | |
| ## Install | |
| The package needs a tt-metal build (`ttnn`) of commit `44d66500520fda9f2c7060c0f6b41ec48f7ab37e` with the two patches | |
| of this repo's `patches/` folder applied (`tt-metal-eth-dispatch.patch`: dispatch on the Ethernet cores, which frees the | |
| 12 x 10 Tensix grid; `tt-metal-reshape-rm-sys1419.patch`: a reshape guard). `ttnn` is not on PyPI. In that environment, | |
| from the repo root: | |
| ```bash | |
| pip install -e . # the API: numpy<2, pillow, tokenizers, safetensors, huggingface_hub | |
| pip install -e ".[server,test]" # + fastapi / uvicorn / pydantic and pytest / httpx | |
| ``` | |
| ## Quickstart | |
| ```python | |
| from tt_rdt2 import RDT2FM, load_sample | |
| sample = load_sample() # shipped sample: two 384x384 cameras, instruction, zero state | |
| with RDT2FM.from_pretrained() as model: # weights -> device -> every prompt bucket traced and warm | |
| out = model(sample["images"], sample["instruction"], state=sample["state"], seed=0) | |
| print(out.actions.shape, out.prompt_bucket, out.num_tokens, out.timing_ms) | |
| ``` | |
| `examples/quickstart.py` is the runnable version. | |
| ## `RDT2FM.from_pretrained` | |
| ```python | |
| RDT2FM.from_pretrained(model_id=None, revision=None, *, device_id=0, device=None, dispatch="eth", engine=None, | |
| precision="baseline", weights_dirs=None, local_files_only=False, warmup=True, | |
| allow_fallback=False, log_fn=None) | |
| ``` | |
| 1. **Weights first, no device.** Four pinned repositories are resolved to local directories: | |
| | role | repository @ revision | files read | | |
| | --- | --- | --- | | |
| | `vlm` | `robotics-diffusion-transformer/RDT2-VQ` @ `ec7e7f3aa4259224481c3cbd5816cc3f35354bec` | `config.json`, `model.safetensors.index.json`, `model-00001-of-00004.safetensors`, `model-00002-of-00004.safetensors` (vision tower, embedding, LM layers 0..13: 9.96 GB) | | |
| | `expert` | `robotics-diffusion-transformer/RDT2-FM` @ `f803808e3f79c77bba8bb4f3326e83d04e60783a` | `config.json`, `pytorch_model.bin` (975 MB, loaded with `torch.load(weights_only=True)`) | | |
| | `normalizer` | `robotics-diffusion-transformer/RVQActionTokenizer` @ `a6a614ae049cc3b9fa6bd4f5fadaeee3375d4dec` | `umi_normalizer_wo_downsample_indentity_rot.pt` (28 kB) | | |
| | `processor` | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5` | `tokenizer.json` (no weights) | | |
| Per role: `weights_dirs[role]` > `$RDT2_VLM_DIR` / `$RDT2_EXPERT_DIR` / `$RDT2_NORMALIZER_DIR` / | |
| `$RDT2_PROCESSOR_DIR` > (VLM only) a local directory given as `model_id` > `hf_hub_download(repo, file, | |
| revision=<pin>)` into the Hugging Face cache. A file already cached at a pinned commit is used without network | |
| access; on a network failure the cache is retried with `local_files_only=True`. `model_id` / `revision` replace | |
| the VLM repository (a fine-tuned RDT2-VQ with the same layout); the other three stay pinned. | |
| 2. **Device.** `dispatch="eth"` (default) opens the chip with dispatch on the Ethernet cores: a 12 x 10 compute grid. | |
| `dispatch="worker"` uses Tensix dispatch (11 x 10). The grid is read from the device. If ETH dispatch is not | |
| available, `allow_fallback=True` opens with WORKER dispatch and a warning; the default raises. `device=` takes a | |
| caller-owned device (opened with a trace region of at least 512 MB); `close()` then leaves it open. | |
| 3. **Programs.** VISION (Qwen2.5-VL ViT, 32 blocks + merger), PREFIX (embedding, image-token scatter, LM layers | |
| 0..12 + the K/V of layer 13) and EXPERT (state prologue + 5 Euler steps of the 14-block RDT expert) are loaded | |
| with all three prompt buckets (416 / 448 / 512 tokens) resident. `engine` selects per program the persistent | |
| megakernel (`mk`) or the ttnn op-by-op path (`ttnn`); the PREFIX also has `fused` (the ttnn graph with tuned | |
| matmul configurations and two fused `generic_op` programs per layer: residual add + RMSNorm, and head split + | |
| RoPE in fp32). The default is `vision=mk, prefix=fused, expert=mk` (`"prefix=ttnn"` etc. override it; `ttnn` is | |
| the fallback of each program; `RDT2_ENGINE` sets it from the environment). `prefix=mk` is **experimental**: it | |
| hangs the device on repeated trace replays at bucket 512, and it is refused unless `RDT2_EXPERIMENTAL_PREFIX_MK=1` | |
| is set in the environment (development only). | |
| 4. **Warm-up** (`warmup=True`): one Metal trace per bucket is captured (the first boot JIT-compiles the kernels), then | |
| one call per bucket runs; the first real call is as fast as later ones. `model.warmup()` is idempotent and returns | |
| the per-bucket warm call time in ms. | |
| `precision="baseline"` is the gated preset (ViT at HiFi2, LM and expert at HiFi4, bf16 weights, fp32 residual | |
| streams). The other presets of `tt_rdt2/tt/common.py` are development presets and were not gated with the default | |
| engine: `fast` fails the long-prompt gate; the bfp8-weight presets were not re-gated with `prefix=fused`; the | |
| presets with a bf16 language-model residual (`hifi4_bf16`, `hifi2_bf16`, `hifi2_lmmlp_bfp8`, `hifi2_bfp8`, | |
| `hifi2_bfp8_all`) and `baseline_kfused` are refused by `prefix=fused` and need `engine="prefix=ttnn"`. | |
| ## Calling the model | |
| ```python | |
| out = model(images, instruction, state=None, seed=None, prompt_bucket=None, num_steps=None) # -> ActionChunk | |
| ``` | |
| | argument | meaning | | |
| | --- | --- | | |
| | `images` | exactly 2 cameras in the checkpoint's order `[left_stereo, right_stereo]` (left-arm wrist camera, right-arm wrist camera), or a mapping `{"left_stereo": ..., "right_stereo": ...}`. Each: an `HxWx3` uint8 RGB array, a PIL image, PNG / JPEG bytes or a file path | | |
| | `instruction` (or `prompt=`) | the task in natural language; non-empty, no special-token text such as `<|im_end|>`, at most 112 tokens | | |
| | `state` | optional raw 20-d proprio state (not normalized); default zeros, which is what the official RDT2-FM deployment sends | | |
| | `seed` | optional integer in [0, 2**63 - 1]: the initial flow-matching noise is `torch.randn((1, 24, 20), generator=torch.Generator().manual_seed(seed))`; default 0. The same request with the same seed returns the same chunk, bit for bit | | |
| | `prompt_bucket` | optional 416 / 448 / 512; default the smallest bucket that holds the prompt. Pad keys are masked, so the bucket changes the chunk only within rounding | | |
| | `num_steps` | optional; must be 5 | | |
| The pre-processing is the official RDT2 inferencer's (`thu-ml/RDT2` `RDTInferencer.step`): the two cameras are | |
| concatenated side by side (left | right), resized to 392 x 756 with the uint8 bicubic-antialias kernel of the Qwen2.5-VL | |
| processor, CLIP-normalized and cut into 1512 patches (378 image tokens); the prompt is the Qwen chat template with one | |
| user turn (image, instruction) followed by `<|im_start|>assistant\n<|quad_start|>`, tokenized with the Qwen2.5-VL | |
| tokenizer. The prompt is 400 fixed tokens plus the instruction's: buckets 416 / 448 / 512 hold instructions of up to | |
| 16 / 48 / 112 tokens. Two 384 x 384 cameras are the official setup; any pair of equal-height images whose | |
| concatenation resizes to the same 392 x 756 grid (for example 2 x 380 x 380) is accepted and resized exactly as the | |
| official processor does. Other geometries are refused, never padded or cropped. | |
| `InputError` (a `ValueError`) is raised, before anything reaches the device, for: a camera count other than 2, | |
| unequal heights, an unsupported geometry, an image that does not decode, an empty instruction or one with | |
| special-token text, a prompt longer than the largest bucket (never truncated), a bucket that does not hold the prompt, | |
| a state that is not 20 finite numbers, a seed out of range, `num_steps != 5`. | |
| ## Output: `ActionChunk` | |
| | field | content | | |
| | --- | --- | | |
| | `actions` | (24, 20) float32: the official `RDTInferencer.step` output, un-normalized with the UMI normalizer. Columns 0..9 right arm, 10..19 left arm; per arm `[x, y, z]` end-effector position in metres, a 6-D rotation, the gripper width in metres. Positions and rotations are relative to the current end-effector pose; 24 steps at 30 Hz | | |
| | `actions_normalized` | (24, 20) float32: the expert's normalized output (`actions = (normalized - offset) / scale`) | | |
| | `prompt_bucket`, `num_tokens`, `seed`, `num_steps` | what ran | | |
| | `timing_ms` | `preprocess` (request checks, image decode, tokenization, pixel and id layout), `inference` (input upload, trace replay, read-back, un-normalization), `total` | | |
| `out.to_dict()` is the `/predict` JSON (`actions`, `shape`, `normalized: false`, `units`, `action_layout`, `chunk`, | |
| `prompt_bucket`, `num_tokens`, `num_steps`, `seed`, `timing_ms`). The official README rescales the gripper columns | |
| (`a[:, 9|19] / 0.088 * 0.1`) for its robot; this package returns `RDTInferencer.step`'s values without that step. | |
| ## Lifetime and threads | |
| `close()` releases the traces and closes the device it opened (idempotent; also at interpreter exit and on `with` | |
| exit). Calls are serialized by a lock (one chip, batch 1); one model per chip and process. `model.info` holds the | |
| weights (repository, revision, files, source per role), engines, device (dispatch, grid), buckets, fixed | |
| configuration and the warm-up times; it is the server's `/info`. | |
| ## Speed | |
| On one p150 (ETH dispatch, 12 x 10 grid; 8-vCPU host), the shipped sample, warm, default engines, median of 120: | |
| device time per call (trace replay) 69.8 ms at bucket 416, 71.5 ms at 448, 75.4 ms at 512 (VISION 27.2, PREFIX | |
| 32.0, EXPERT 13.0 ms at 416); host pre-processing 5.5 ms; `model()` on the sample 76.8 ms. `code/scripts/bench.py` | |
| measures these, the per-stage traces and the transfers. Served numbers are in SERVING.md. | |
| ## Tests | |
| ```bash | |
| TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_rdt2/tests/test_api_host.py code/tt_rdt2/tests/test_server_host.py | |
| python -m pytest -q -s code/tt_rdt2/tests/test_e2e_device.py # chip: serving path vs the stored reference | |
| python -m pytest -q -s code/tt_rdt2/tests/test_stress_device.py # chip: 500-request stress gate | |
| python -m pytest -q -s code/tt_rdt2/tests/test_tt_device.py # chip: per-program gates (needs the golden data) | |
| ``` | |
| Host tests need no device (missing weights or golden data skip). The device suite `test_tt_device.py` and the golden | |
| cases of the stress gate need the official golden dumps (`RDT2_GOLDEN_DIR`), which are not shipped. | |
| ## Not included | |
| RDT2-VQ's own action head (autoregressive action tokens decoded by the RVQ tokenizer) is not served: this package | |
| runs RDT2-FM, which uses RDT2-VQ only as the vision-language backbone (K/V of layers 0..13). | |