moge-2-p150 on Tenstorrent p150

Derived from Ruicheng/moge-2-vitl-normal, the ViT-L checkpoint with the normal head of MoGe-2 by Microsoft Research (microsoft/MoGe), with unmodified weights. Paper: MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details. See the original model card and repository for training details.

MoGe-2 ViT-L with the normal head, 331M parameters: a DINOv2 ViT-L/14 encoder (309M parameters), a convolutional neck and four heads (point map, surface normals, validity mask, metric scale). One RGB image goes in and is placed on a fixed 1920x1080 canvas (1800 ViT tokens), letterboxed by default. Out come, at the input resolution, a metric point map in camera space, metric depth, unit surface normals, a validity mask and the camera intrinsics (focal length and field of view) estimated from the image. The network runs on the chip; image decoding, the canvas fit, the recovery of focal length and depth shift, the resize back to the input size and the output encoding run on the host CPU.

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware p150
License mit

Intended use

Monocular geometry from single RGB images: depth, 3D point maps, surface normals and the camera's field of view, for research, prototyping and offline analysis such as 3D scene previews and dataset annotation.

Prerequisites

  • The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
  • Docker
  • Tenstorrent hardware: p150
  • An x86-64 host CPU with AVX2 and FMA (x86-64-v3) for the host-side speeds under Expected performance; on other CPUs the server uses a slower host path

Quickstart

tt serve changh95/moge-2-p150

tt serve downloads the Docker image and the Ruicheng/moge-2-vitl-normal at cb0e8bbd6b1e243589717c78e750b1ba4c093acf weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/moge-2-p150.

Send the sample image of this repo to /predict, on the port tt serve reported. The response is JSON with the dense outputs as a base64 .npz (about 39 MB for this 1920x1080 image):

curl -sLO https://huggingface.co/changh95/moge-2-p150/resolve/main/media/source.png
printf '{"image":"%s"}' "$(base64 -w0 source.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' --data-binary @req.json -o resp.json
python3 -c "import base64, io, json, numpy as np; r = json.load(open('resp.json')); \
o = np.load(io.BytesIO(base64.b64decode(r['outputs']['npz']))); print(o['depth'].shape, r['fov_x_deg'])"

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β€” its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one base64 image and returns its geometry. GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

request field meaning
image base64 PNG or JPEG of one image, at least 8x8 pixels and at most 32 MB; a data:image/...;base64, prefix is accepted
output_format (optional) npz (default): the arrays as a base64 .npz in the JSON; npz_binary: the same .npz as the response body (application/x-npz), the other response fields as JSON in the X-MoGe-Meta header; png: depth as a 16-bit PNG, normals and mask as 8-bit PNGs; json: nested lists, only for images up to 512x512 pixels
fit (optional) pad (default): letterbox onto the 1920x1080 canvas, keeping the aspect ratio; stretch: resize to the canvas
fov_x (optional) the image's horizontal field of view in degrees, if known (between 1 and 179); then only the depth shift is estimated
apply_mask (optional) default true: invalid pixels get inf depth and points and a zero normal
force_projection (optional) default true: the points are re-projected from the depth and the intrinsics
include_depth_png (optional) default false: with npz or json, also return the 16-bit depth PNG

Response of the quickstart request above (media/source.png, 1920x1080), npz data and the format notes in encoding elided:

{"model": "MoGe-2 ViT-L (normal)", "weights": {"repo": "Ruicheng/moge-2-vitl-normal", "revision": "cb0e8bbd6b1e243589717c78e750b1ba4c093acf"},
 "height": 1080, "width": 1920,
 "canonical": {"width": 1920, "height": 1080, "fit": "pad", "region": [0, 0, 1920, 1080], "num_tokens": 1800, "token_grid": [32, 57]},
 "metric_scale": 10.332945823669434, "focal_rel_half_diag": 0.9335933327674866, "shift": -0.10347557067871094,
 "intrinsics": [[0.5355780124664307, 0.0, 0.5], [0.0, 0.9521387219429016, 0.5], [0.0, 0.0, 1.0]],
 "intrinsics_pixels": [[1028.3097839355469, 0.0, 960.0], [0.0, 1028.3098196983337, 540.0], [0.0, 0.0, 1.0]],
 "fov_x_deg": 86.06467404887222, "fov_y_deg": 55.410943192460806, "mask_coverage": 0.9054615162037037,
 "depth_m": {"min": 2.506209135055542, "median": 13.321823120117188, "max": 255.86788940429688},
 "apply_mask": true, "force_projection": true,
 "outputs": {"npz": "..."}, "encoding": {"output_format": "npz", "npz": "..."},
 "timing_ms": {"decode": 14.8, "preprocess": 0.1, "device": 32.2, "postprocess": 34.5, "encode": 34.0, "total": 115.6}}
  • outputs.npz decodes with np.load to arrays at the input resolution: points float32 [H, W, 3] in metres, camera space with OpenCV axes (x right, y down, z forward); depth float32 [H, W] in metres; normal float16 [H, W, 3], unit normals in camera space; mask uint8 [H, W], 1 = valid; intrinsics float32 3x3; metric_scale float32.
  • intrinsics is normalized (fx, cx in units of the image width, fy, cy in units of the height); intrinsics_pixels is the same matrix in pixels. metric_scale is already applied to points and depth. mask_coverage is the share of valid pixels and depth_m the minimum, median and maximum valid depth. canonical.region is where the image sits on the 1920x1080 canvas (x0, y0, x1, y1).
  • Pixels without valid geometry, such as the sky, have mask 0 and, with apply_mask: true, inf depth and points and a zero normal.
  • png returns outputs.depth_png16 (metres = value x encoding.depth_png_scale, 0 = invalid), outputs.normal_png (n = value / 255 x 2 - 1) and outputs.mask_png (255 = valid); points are not encoded, unproject the depth with the intrinsics. For a 1920x1080 image the npz JSON response is about 39 MB and the npz_binary body about 29 MB.
  • timing_ms gives the server-side stages: decode (base64 and image), preprocess (canvas fit), device (the model call: upload, chip, readback), postprocess (focal and shift recovery, resize to the input size), encode and total.
  • Errors: 400 for an image that cannot be decoded, is smaller than 8x8 or larger than 32 MB, json above 512x512 pixels, include_depth_png with npz_binary, or invalid request fields; 503 while the server starts.
  • A Python API with the same outputs (tt_moge.MoGeModel; its arrays equal /predict's npz output bit for bit) is described in code/PYTHON.md, with an example in code/examples/quickstart.py. It is installed in this package's image; elsewhere it needs a host ttnn build of the tt-metal commit in Provenance with patches/tt-metal-8b98410.local.patch applied.

Example output

input depth normals
Sample street scene, input MoGe-2 on a Tenstorrent p150: depth of the sample image MoGe-2 on a Tenstorrent p150: surface normals of the sample image

The depth and normals of media/source.png as code/examples/quickstart.py writes them with this package's image: depth drawn as inverse depth with the Spectral colormap (near is red, far is blue), normals as RGB (x to red, up to green, towards the camera to blue); invalid pixels (the sky) are black in the depth and gray in the normals.

Expected performance

Geometry accuracy against the ground truth of three public benchmarks, beside the figure the MoGe-2 paper publishes and the same model run in fp32 on the CPU with the same inputs (Rel % / Ξ΄1 %; lower Rel and higher Ξ΄1 are better):

eval this package fp32 CPU reference MoGe-2 paper images
NYUv2: metric point map 4.62 / 98.1 4.54 / 98.1 4.44 / 98.3 218 of 654
NYUv2: metric depth 7.79 / 96.4 7.76 / 96.4 7.33 / 96.1 218 of 654
NYUv2: metric depth, true FoV given 6.76 / 97.2 6.59 / 97.3 6.46 / 96.9 218 of 654
NYUv2: affine-invariant depth 2.98 / 98.5 2.98 / 98.5 2.89 / 98.6 218 of 654
KITTI: metric point map 6.93 / 94.9 7.00 / 94.7 7.44 / 94.4 163 of 652
KITTI: metric depth 15.0 / 76.4 15.2 / 76.0 18.1 / 62.9 163 of 652
KITTI: metric depth, true FoV given 8.88 / 93.3 8.61 / 93.4 8.64 / 93.7 163 of 652
KITTI: affine-invariant depth 3.84 / 98.0 3.83 / 98.0 3.75 / 98.1 163 of 652
iBims-1: metric point map 6.06 / 97.4 – 5.63 / 97.4 100 of 100
iBims-1: metric depth 13.3 / 85.0 – 13.6 / 83.0 100 of 100
iBims-1: metric depth, true FoV given 10.9 / 87.4 – 9.92 / 92.4 100 of 100
iBims-1: affine-invariant depth 2.66 / 98.6 – 2.36 / 98.8 100 of 100
Depth against the fp32 reference: AbsRel mean (max) / PCC mean (min) 0.43 % (1.58 %) / 0.99996 (0.99970) – – 381
Normals against the fp32 reference: mean angle (max of the per-image means) 0.37Β° (0.83Β°) – – 381
Metric scale against the fp32 reference, mean (range) +0.65 % (βˆ’0.56 to +3.1 %) – – 381
model.forward outputs against the fp32 forward on media/source.png: points / depth / normal / mask PCC 0.999953 / 0.999919 / 0.999958 / 0.999996 – – 1

Sets: MoGe's processed evaluation data (the Hugging Face dataset Ruicheng/monocular-geometry-evaluation): NYUv2 official test split (654 images, every 3rd used), KITTI Eigen test split (652, every 4th) and iBims-1 (100, all). Metrics: those of MoGe's own evaluation code. Rel is the mean relative error and Ξ΄1 the share of pixels within a factor 1.25 of the ground truth (for point maps: with a point error below 25 % of the point's distance), both per-image means; the metric point map is compared after the optimal translation, metric depth without any alignment, affine-invariant depth after a scale and shift fit. The field of view is estimated from the image except in the "true FoV given" rows, which pass the ground-truth horizontal FoV as fov_x. This package and the fp32 reference run the served pipeline (the image on the 1920x1080 canvas, 1800 tokens) through the Python API; the fp32 reference is the upstream MoGe-2 PyTorch model (microsoft/MoGe) with the same weights, run on NYUv2 and KITTI only. MoGe-2 paper: Table B.4, row "Ours", which is the ViT-L model without the normal head (Ruicheng/moge-2-vitl, 326M parameters) evaluated on the full sets at 3600 tokens on each image's own aspect ratio.

speed (batch 1, warm) input median p99 throughput
Device time per image, back-to-back trace replays 1920x1080 canvas, 1800 tokens 17.86 ms 17.96 ms 56.0 images/s
Python model.forward(x) (upload, device run, readback, host output pass) media/source.png as float32 45.6 ms 107.0 ms 21.9 images/s
Python model(array) call, returning a MoGeOutput media/source.png, decoded uint8 1920x1080 76.2 ms 184.9 ms 13.1 images/s
Python model(path) call, also decoding the PNG media/source.png 93.8 ms 240.5 ms 10.7 images/s
Served POST /predict (npz), server timing_ms.total media/source.png 164.7 ms 439.5 ms –
Served POST /predict (png), server timing_ms.total 800x600 resize of media/source.png 100.3 ms 252.6 ms –
Served POST /predict (npz), client round trip on loopback media/source.png 409.7 ms 445.9 ms 2.4 requests/s
Served POST /predict (npz_binary), client round trip on loopback media/source.png 229.5 ms 327.4 ms 4.4 requests/s
Served POST /predict (png), client round trip on loopback 800x600 resize of media/source.png 112.6 ms 130.6 ms 8.9 requests/s
Served POST /predict (npz), 2 concurrent clients media/source.png 458.0 ms 533.0 ms 2.45 to 3.40 requests/s

Measured with this package's image on one p150 in an 8-vCPU AMD EPYC-Rome host shared with other jobs (1-minute load average 11 to 31 during the runs), with code/scripts/bench_inproc.py (5 timings of 50 back-to-back replays for the device row, 200 calls after 20 warm-up calls for the others, 100 for model(path)) and code/scripts/bench_http.py (100 requests per case, one at a time on one keep-alive connection after 10 warm-up requests, then 2 clients with 100 requests each), in 3 runs (median of the runs' medians, highest p99; throughput 1000 / median); the client round-trip rows are one further run of the same requests, timed from sending the request to the parsed JSON response (the client_ms that bench_http.py reports also includes a hash of the outputs), and the 2-client throughput spans 4 runs. The host-side rows rise with the host's load (model(array) medians of 52 to 134 ms across the runs); the device row does not.

Limitations

  • Not for driving, robot navigation, obstacle avoidance or any other safety function, and not for measurements that people or machines rely on: with the field of view estimated from the image, the metric depth error (Rel) was 7.8 % on NYUv2, 15.0 % on KITTI and 13.3 % on iBims-1 above. Validate the metric scale on your own scenes before you use distances.
  • Not a MoGe-2 accuracy benchmark: this package serves the normal-head checkpoint on a fixed 1920x1080 canvas with 1800 tokens, while the paper's figures are for the checkpoint without the normal head at 3600 tokens on each image's own aspect ratio. The fixed canvas costs accuracy on some data: on 73 of the NYUv2 images above, the fp32 model's metric point map Rel is 4.68 % with this package's settings and 4.12 % with MoGe's own.
  • Not for training or fine-tuning: inference only, with the moge-2-vitl-normal weights. The other MoGe and MoGe-2 checkpoints are not packaged.
  • One image per request, batch 1, requests served one at a time. Every image is fitted onto the 1920x1080 canvas (letterboxed with fit: pad, distorted with fit: stretch), so very wide or tall images use fewer of the 1800 tokens; other canvases and token budgets were not tested. The EXIF orientation of JPEG files is not applied.
  • Accuracy was measured on NYUv2, KITTI and iBims-1 only; the other evaluation sets of the paper (ETH3D, GSO, Sintel, DDAD, DIODE, Spring, HAMMER) were not evaluated, and the surface normals were compared with the fp32 reference only, not with ground truth.
  • The estimated field of view can be far off: on the KITTI images above it was off by 10.0Β° on average (9.6Β° in fp32 on the CPU), and the metric depth error fell from 15.0 % to 8.88 % when the true FoV was given. Pass fov_x when you know it.
  • The chip computes in bfloat16, with 8-bit block floating point weights in parts of the network. Against the fp32 CPU reference, the depth differs by 0.43 % on average (1.58 % at most per image), the normals by 0.37Β° and the metric scale by +0.65 % on average (βˆ’0.56 to +3.1 %); the estimated horizontal FoV is 0.44Β° wider on average.
  • One p150 only (one chip): multi-chip meshes are not supported, and other boards were not tested. The Python API was tested inside this package's image only.
  • The latency of model() and of the served requests depends on the host CPU and rises with its load; the device time does not. On a CPU without AVX2 and FMA, the host part falls back to a slower torch path: forced onto that path here, model(array) took 642 ms instead of 52 to 134 ms.

Licensing

  • Weights: Ruicheng/moge-2-vitl-normal at commit cb0e8bbd6b1e243589717c78e750b1ba4c093acf, MIT (licence), downloaded by tt serve at that commit, not redistributed here.
  • Port and serving code (code/): Apache-2.0. code/tt_moge/reference/ carries upstream code under its own licences: microsoft/MoGe (MIT, Β© Microsoft Corporation) with its DINOv2 modules (Apache-2.0, Β© Meta Platforms, Inc. and affiliates) and utils3d (MIT); the licence texts and the file list are in licenses/. code/models/common/lightweightmodule.py is from tt-metal (Apache-2.0). code/tt_moge/host_ext/prebuilt/ is a compiled build of code/tt_moge/host_ext/moge_host.cpp, made by code/scripts/build_host_ext.sh.
  • patches/tt-metal-8b98410.local.patch: all the changes the image's tt-metal carries on top of the public commit in Provenance; patches/tt-metal-eth-dispatch.patch is a subset of it. tt-metal is Apache-2.0.
  • media/depth.png and media/normal.png are this package's outputs for media/source.png.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/moge-2-p150/discussions β€” that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from β€” code/ in this repo is byte-identical to the model code inside the image:

component built from
tt-metal 8b98410e730bb504fea43a88609756e34821d91d (dirty tree β€” the image includes uncommitted changes)
code/ digest ebc4f823a4c89591 (sha256, first 16 hex digits)
built 2026-11-12T14:54:16+00:00 by tt-model 0.1.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for changh95/moge-2-p150

Finetuned
(5)
this model

Collection including changh95/moge-2-p150

Paper for changh95/moge-2-p150