vla-jepa-base-p150 on Tenstorrent p150
Derived from lerobot/VLA-JEPA-Pretrain (commit
e946c3e5), the pretrained VLA-JEPA policy published by LeRobot, with unmodified weights. Method: VLA-JEPA (arXiv:2602.10098); reference inference code: LeRobot 0.7.0 (lerobot/policies/vla_jepa). See the original model card for training details.
VLA-JEPA, a 2.3B-parameter vision-language-action policy (as used for inference): a Qwen3-VL-2B vision encoder (24 ViT blocks with DeepStack features) and its 28-layer language model encode the camera images (two by default) and a task instruction; the pre-norm output of the last layer at 32 action-query tokens conditions a 16-layer DiT flow-matching action head, which turns Gaussian noise into a chunk of 7 future actions of 7 values each in 4 Euler steps by default. The checkpoint's V-JEPA2 world model is used only in training and is not run. Each camera is resized to 224 x 224 and upscaled to 256 x 256 by the Qwen3-VL processor (64 image tokens per camera). The package includes LeRobot's pre- and post-processing for this checkpoint: the camera resize, the prompt template, the Qwen3-VL tokenizer with VLA-JEPA's added tokens, and the post-processor (clip, gripper snap, un-normalisation with the checkpoint's statistics, gripper binarisation). Pre- and post-processing run on the host CPU; the vision encoder, the language model and the action head (all its Euler steps) run on the device as three persistent programs, replayed from Metal traces captured per prompt length (a new instruction's inputs are uploaded while the vision encoder runs). The number of cameras (1-4) and of flow-matching steps (1-16) are chosen when the server starts (LeRobot's code accepts other values of both); the prompt lengths follow the camera count.
Runs on Tenstorrent p150 (mesh P150).
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
At a glance
| Hardware | p150 |
| License | apache-2.0 |
Intended use
Research and evaluation of the pretrained VLA-JEPA policy: generating action chunks from two camera views and a language instruction, for example to inspect the pretrained policy, to compare implementations, or as a starting point for offline experiments, one request at a time.
Prerequisites
- The Tenstorrent CLI,
tt:uv tool install tenstorrent(or use tt-model alone, see the Quickstart) - Docker
- Tenstorrent hardware: p150
- About 11 GB of free disk: 6.2 GB for the weights in the Hugging Face cache and 4.1 GB for the container image
Quickstart
tt serve changh95/vla-jepa-base-p150
tt serve downloads the Docker image and the lerobot/VLA-JEPA-Pretrain at e946c3e5b538d760f4b4ff239d1b1c12090c041d weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.
Without tt-cli: tt-model serve changh95/vla-jepa-base-p150.
Send the sample shipped in this repo (one LIBERO-spatial frame: an agent-view and a wrist camera, and its task)
to /predict, on the port tt serve reported:
for f in image.png wrist_image.png sample.json; do
curl -sLO https://huggingface.co/changh95/vla-jepa-base-p150/resolve/main/code/tt_vla_jepa/samples/libero_spatial_ep0_f40/$f
done
python3 - <<'PY'
import base64, json
b64 = lambda f: base64.b64encode(open(f, "rb").read()).decode()
task = json.load(open("sample.json"))["task"]
json.dump({"images": [b64("image.png"), b64("wrist_image.png")], "instruction": task, "seed": 0},
open("req.json", "w"))
PY
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
Capabilities
This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β its request and response shapes are the model's own. See the author's notes for the payload it expects.
POST /predict takes the camera images and an instruction as JSON and returns one action chunk. GET /health
reports readiness; GET /info reports the weights, the device, the cameras and their order, the prompt buckets,
the steps, the chunk shape and the limits; GET /v1/models is a stub, not a chat API.
request field (POST /predict) |
meaning |
|---|---|
images |
base64 PNG or JPEG RGB images, exactly as many as the server's camera count (2 by default), in the checkpoint's camera order: exterior_1_left, then exterior_2_left (further cameras: extra_2, extra_3; /info lists the order). Any size from 16 to 4096 px per side; each is resized to 224 x 224 (squashed, not cropped) as in LeRobot |
instruction (or prompt) |
the task in natural language; up to 111 tokens with 2 cameras (/info lists the limit per prompt bucket). Without it the default instruction "Execute the robot action." is used; Qwen special tokens inside it are refused |
seed (optional) |
integer seed of the initial flow-matching noise, default 0; the same request with the same seed returns the same chunk |
prompt_bucket (optional) |
224, 256, 288 or 320 prompt rows with 2 cameras (/info lists the buckets of the camera count); default the smallest that holds the prompt. Pad rows are masked, but a larger bucket can change the chunk slightly (by up to 0.009 on the normalised actions in the cases checked, the size of the device's bf16 error) |
num_steps (optional) |
must equal the server's number of flow-matching steps (4 by default) |
state |
refused: this checkpoint has no robot-state input |
Response for the quickstart request (values rounded, 2 of the 7 rows shown):
{"model": "vla-jepa-base-p150", "checkpoint": "lerobot/VLA-JEPA-Pretrain",
"actions": [[-0.1453, 0.0857, 0.2496, -0.2340, 0.0662, 0.0392, -1.0],
[-0.1314, 0.0961, 0.2459, -0.2381, 0.0479, 0.0320, -1.0], "... 7 rows"],
"actions_normalized": [[-0.1453, 0.0857, 0.2496, -0.2340, 0.0662, 0.0392, 0.8652], "... 7 rows"],
"normalized": false, "chunk_shape": [7, 7], "num_steps": 4, "prompt_bucket": 256, "num_tokens": 225,
"seed": 0, "noise_source": "seed", "image_sizes": [[256, 256], [256, 256]],
"timing_ms": {"decode": 2.7, "preprocess": 11.1, "inference": 22.4, "postprocess": 0.2, "total": 33.8}}
actions: 7 future steps x 7 values in the checkpoint's action space, after its LeRobot post-processor. The checkpoint's statistics map values 0-5 to [-1, 1] unchanged; value 6 is the gripper, -1 when the model's normalised gripper output is at least 0.5 and +1 otherwise. The model card does not document the physical meaning of the values.actions_normalized: the model output before post-processing.timing_msis measured in the server:decode(base64 and image decoding),preprocess(includingdecode: resize, Qwen3-VL image processing, tokenisation),inference(upload, device run, readback),postprocessandtotal.- Errors: HTTP 400 with the reason for a wrong number of images, an image that is not a PNG or JPEG (a
multi-picture MPO file is refused) or is out of the size range, an instruction that does not fit the largest
prompt bucket or contains a special token, a
num_stepsother than the server's, astate, a seed outside [0, 2^63) or a prompt bucket too small for the prompt; 422 for an unknown field or a value of the wrong type; 503 while the server starts; 500 with the exception text on a device failure. - A Python API with the same outputs (
tt_vla_jepa.VLAJEPA) is described incode/PYTHON.md; it needs a tt-metal build with the two patches inpatches/.
Configuration
The server builds its device programs and traces for one configuration when it starts; only the prompt bucket
changes per request. To serve another configuration, change VLA_JEPA_NUM_CAMERAS / VLA_JEPA_NUM_STEPS in the
container's launch command (tt-model serve --print prints it; tt-model serve has no flag for the environment)
and start it again; GET /info reports the configuration, the camera order and the prompt buckets.
SERVING.md (section 4.1) has the recipe.
| item | environment variable | range | default | fixed when |
|---|---|---|---|---|
| cameras | VLA_JEPA_NUM_CAMERAS |
1 to 4 (cameras 3 and 4 are further image inputs; the checkpoint was trained with 2) | 2 | the server starts |
| flow-matching (Euler) steps | VLA_JEPA_NUM_STEPS |
1 to 16 | 4 | the server starts |
| prompt bucket | none (request field prompt_bucket) |
4 buckets, 32 rows apart: 160-256 rows with 1 camera, 224-320 with 2, 320-416 with 3, 384-480 with 4 | the smallest that holds the prompt | each request |
| action chunk, image size, batch | none | 7 x 7 actions; every camera resized to 224 x 224, then 256 x 256; batch 1 | always |
With 3 or 4 cameras the language model runs as a ttnn program instead of the persistent one, which is slower (see the speed table).
Example output
The shipped sample (agent-view and wrist camera of one LIBERO-spatial frame, task "pick up the black bowl next to the cookie box and place it on the plate", seed 0): the normalised action chunk this package returns (dashed), the fp32 CPU reference of the same weights (solid) and the output of LeRobot's own VLA-JEPA code in fp32 on the same request (dots). LIBERO data: Apache-2.0 per the lerobot/libero_spatial_image dataset card.
Expected performance
Agreement of the action chunk with LeRobot 0.7.0's own VLA-JEPA code run in fp32 on the CPU (same weights, images, instruction and initial noise; PCC and largest absolute difference over the 7 x 7 normalised actions). The last column gives the same comparison for LeRobot's default bf16 run where it was computed.
| eval | this package | LeRobot bf16 | requests |
|---|---|---|---|
| Shipped sample (LIBERO frame, seed 0): PCC / largest difference | 0.99997 / 0.0067 | β | 1 |
| Golden requests (gradient, noise and shapes images with 2 instructions each, and one real frame with its task; 2 seeds each): lowest / mean PCC | 0.99987 / 0.99997 | β | 14 |
| Prompt-length cases (every prompt bucket, full and padded, empty and digit-only instructions): lowest PCC | 0.99982 | β | 8 |
| Further real and synthetic camera frames (DROID-style, UR5e, flat, checkerboard, noise), new seeds and instructions: lowest / mean PCC | 0.99899 / 0.99979 | 0.99949 / 0.99980 | 6 |
| Conditioning (the 32 x 2048 features the action head reads), further frames: lowest PCC | 0.99782 | 0.99590 | 6 |
| Gripper decisions that differ from the fp32 output where its margin to the 0.5 threshold exceeds the error | 0 | β | 28 |
| All 22 golden and prompt-length requests served over HTTP among 500 mixed requests: lowest / mean PCC | 0.99982 / 0.99996 | β | 22 |
| Every configuration of 1-4 cameras x 1 / 2 / 4 / 8 / 16 steps against LeRobot's code run with the same configuration (3 real and synthetic cases each): lowest / median PCC | 0.99631 / 0.99997 | 0.99831 / 0.99992 | 60 |
The golden and prompt-length requests were generated with LeRobot's code at fixed noise; the fp32 CPU reference of this package reproduces them to PCC 0.9999999. The rows above the last are for the default configuration (2 cameras, 4 steps); the last row counts each case once per configuration. The model card of lerobot/VLA-JEPA-Pretrain publishes no evaluation results.
| speed (batch 1, warm) | input | median | p99 | throughput |
|---|---|---|---|---|
| Device time per inference (one trace replay of the 3 device programs, 10 ms between replays), prompt buckets 224 / 256 / 288 / 320 | shipped sample and its task (a shorter task for 224), 2 cameras, 4 steps | 21.6 / 21.6 / 22.0 / 22.4 ms | β | 44-46 /s |
| Python call with the images already decoded (pre-processing, upload, device, readback, post-processing), the instruction repeated | shipped sample (2 x 256 x 256), bucket 256 | 28.1 ms | 30.9 ms (p90) | 35 /s |
| Python call, a new instruction on every call | shipped sample cameras, bucket 256 | 30.9 ms | 31.6 ms (p90) | 32 /s |
Served POST /predict, server timing_ms.total, prompt buckets 224 / 256 / 288 / 320 |
shipped sample (2 x 256 x 256 PNG) and its task (a shorter task for 224) | 31.0 / 31.6 / 32.8 / 33.3 ms | 32.1 / 33.5 / 33.3 / 34.0 ms | β |
Served POST /predict, client round trip on loopback |
shipped sample, bucket 256 | 34.8 ms | 37.5 ms | β |
Served POST /predict, server timing_ms.total, the instruction alternating between two tasks |
shipped sample cameras, bucket 256 | 31.5 ms | 33.9 ms | β |
Served POST /predict, server timing_ms.total |
240 mixed requests: all buckets, PNG / JPEG cameras of 224 x 224 to 1280 x 720, 16 instructions | 36.4 ms | 74.9 ms | β |
Served POST /predict, server timing_ms.total, 1 camera and 8 steps (set when the server starts), prompt buckets 160 / 192 / 224 / 256 |
the sample's first camera and its task | 30.5 / 30.6 / 31.1 / 31.0 ms | 31.0 / 31.2 / 32.6 / 32.2 ms | β |
| Device time per inference, other configurations (10 ms between replays), 1 camera and 4 steps, prompt buckets 160 to 256 | the sample's first camera and its task | 21.0-21.5 ms | β | β |
| Device time per inference, 2 cameras and 1 step / 16 steps, prompt buckets 224 to 320 | shipped sample and its task (a shorter task for 224) | 19.4-20.3 / 30.4-31.2 ms | β | β |
| Device time per inference, 3 cameras and 4 steps (the language model as a ttnn program), prompt buckets 320 to 416 | the sample's cameras repeated, its task | 44.1-57.9 ms | β | β |
| Device time per inference, 4 cameras and 4 steps / 16 steps (the language model as a ttnn program), prompt buckets 384 to 480 | the sample's cameras repeated, its task | 56.8-71.3 / 65.7-80.2 ms | β | β |
Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host shared with other jobs:
code/scripts/bench.py --sample (100 trace replays per bucket after 3 warm ones, each started 10 ms after the
previous one ended, as between served requests; 33 Python calls per bucket and mode; host load average about 4-8)
and POST /predict requests to this container over loopback HTTP, one at a time (60 per bucket with the sample
after one warm request, 60 alternating, then 240 mixed requests; host load average about 5-6; at 1 camera
and 8 steps 60 per bucket, timing_ms.inference 24.9-25.5 ms). Replayed back to back, the chip lowers its
clock under the sustained load and a default replay takes 21.7 / 22.3 / 23.5 / 24.1 ms. The device part of a served
request (timing_ms.inference: upload, device run, readback) is 22.4-23.1 ms (median per bucket). A request
whose instruction is not the one last used in its bucket uploads the instruction's inputs while the vision
encoder runs, so alternating two instructions costs about the same as repeating one; a new instruction also
pays its tokenisation and prompt tables once (about 3 ms in the Python call). Image decoding and pre-processing run on the host CPU
and grow with the camera resolution and the host load: 8.9 ms for the two 256 x 256 PNGs of the sample, up
to 62 ms in the mixed requests.
Limitations
- Not for controlling a physical robot or any safety function: these are pretrained base weights, and this package was checked offline only (against LeRobot's code and a CPU reference), with no closed-loop, simulator or on-robot run.
- Not a measure of VLA-JEPA's task performance: the model card publishes no evaluation, and the rows under Expected performance compare implementations of the same network. The model card does not document what the 7 action values mean for a robot.
- The configuration is fixed when the server starts: 1-4 cameras (default 2, in the checkpoint's order
exterior_1_left,exterior_2_left) and 1-16 flow-matching steps (default 4), with 4 prompt buckets per camera count (2 cameras: 224 to 320 rows, instructions up to 111 tokens). A request with another number of cameras or steps is refused (HTTP 400), as are longer instructions, special tokens inside the instruction and a robot state (the checkpoint has no state input). The checkpoint was trained with 2 cameras; with 3 or 4 cameras the language model runs as a slower ttnn program. - Other configurations agree with LeRobot's code run with the same configuration to PCC 0.9963 or better on every case checked, but on some cases the error is more than 4 times LeRobot's own bf16 error (the acceptance criterion of this package): 4 of 60 case-configurations (1 camera at 8 and 16 steps on one case; 4 cameras at 8 and 16 steps on another, largest difference 0.041 on the normalised actions where LeRobot's bf16 run is at 0.004-0.006) and 1 of 12 independently made ones (4 cameras, 1 step). It depends on the case, not on a step count. The default configuration met it on every case.
- Batch 1; requests are served one at a time. One p150 only: the image runs device dispatch on the chip's
Ethernet cores, so multi-chip meshes are not supported. Only
lerobot/VLA-JEPA-Pretrainis served; no training or fine-tuning. - Precision: bfloat16 weights and activations with fp32 accumulation, residual streams and normalisation. Over 28 compared requests the action chunk agrees with the fp32 output to PCC 0.999 or better on all but one: on a real UR5e camera frame the agreement is 0.99899 (LeRobot's own bf16 run: 0.99949), and its gripper value differs by 0.047 from the fp32 output while that frame's margin to the 0.5 snap threshold is 0.036, so the binarised gripper of such near-threshold steps can differ from the fp32 result (it did not in the compared requests). On some real camera frames single visual tokens of the DeepStack features differ more from fp32 than in LeRobot's bf16 run, with no visible effect on the actions of the compared requests.
- The initial noise is seeded (seed 0 by default), so the same request always returns the same chunk; LeRobot
draws new noise on every call. Pass
seedfor another draw. - Cameras of any aspect ratio are squashed to 224 x 224, as LeRobot does for this checkpoint. Host CPU work (image decoding and pre-processing) is part of every request and grows with the camera resolution and with the load of the host CPU.
Licensing
- Weights: lerobot/VLA-JEPA-Pretrain at commit
e946c3e5b538d760f4b4ff239d1b1c12090c041d, Apache-2.0 per its model card. Downloaded bytt serveat that commit, not redistributed here. - Tokenizer (
code/tt_vla_jepa/assets/qwen3_vl_2b_instruct/tokenizer.json): byte-identical to the file of Qwen/Qwen3-VL-2B-Instruct at commit89644892, Apache-2.0 per its model card. - Port, server and scripts (
code/), which re-implement LeRobot's (Apache-2.0) VLA-JEPA pre- and post-processing and the Qwen3-VL image processing and chat template of Hugging Face transformers (Apache-2.0), and the tt-metal patches inpatches/: Apache-2.0. - Sample (
code/tt_vla_jepa/samples/libero_spatial_ep0_f40/) andmedia/demo_sample.png: one frame of lerobot/libero_spatial_image (commitd86c0b94; LIBERO benchmark, Liu et al. 2023), Apache-2.0 per its dataset card;reference.jsonholds the outputs of the CPU reference and of LeRobot's code for it.
Related packages
- changh95/pi05-base-p150: the pi0.5 base vision-language-action policy on p150.
- changh95/GR00T-N1.7-3B-p150, changh95/GR00T-N1.6-3B-p150 and changh95/GR00T-N1.5-3B-p150: NVIDIA Isaac GR00T N1.x policies on p150 (GR00T N1.7 also uses a Qwen3-VL vision tower).
- changh95/xvla-base-p150: X-VLA on p150.
- changh95/RLDX-1-PT-p150 and changh95/RLDX-1-PT-IMG-p150: RLDX-1 on p150.
Feedback
Questions or problems with this package: open a discussion at https://huggingface.co/changh95/vla-jepa-base-p150/discussions β that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.
Provenance
The exact sources the image was built from β code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | 44d66500520fda9f2c7060c0f6b41ec48f7ab37e (dirty tree β the image includes uncommitted changes) |
code/ digest |
683f8fd5c109552e (sha256, first 16 hex digits) |
| built | 2026-11-20T07:29:15+00:00 by tt-model 0.1.0 |
Model tree for changh95/vla-jepa-base-p150
Base model
lerobot/VLA-JEPA-Pretrain