| --- |
| license: other |
| license_name: mit-and-apache-2.0 |
| license_link: https://huggingface.co/FixedSeed/lattice-vision/blob/main/README.md |
| library_name: onnx |
| tags: |
| - onnxruntime-web |
| - webgpu |
| - matting |
| - pose-estimation |
| --- |
| |
| # Lattice vision models |
|
|
| ONNX files that [Lattice](https://fixedseed.com/lattice) downloads on first use for its Matte and Body Pose nodes. They |
| run in onnxruntime-web on WebGPU, with every node on the GPU. The weights are the original authors', under their |
| licences; only the format changed (see below). |
|
|
| | File | Model | Authors | Licence | |
| | --- | --- | --- | --- | |
| | `birefnet-lite-matting-512-fp16.onnx`, `-768-`, `-1024-` | BiRefNet_lite-matting (epoch 110), one fixed square input size per file | Peng Zheng et al., [ZhengPeng7/BiRefNet](https://github.com/ZhengPeng7/BiRefNet) | MIT (`LICENSE-BiRefNet-MIT.txt`) | |
| | `rtmw-m-256x192-fp16.onnx` | RTMW-m (rtmw-dw-l-m_simcc-cocktail14, 256x192): 133 COCO-WholeBody points | OpenMMLab, [open-mmlab/mmpose](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmpose) | Apache-2.0 (`LICENSE-mmpose-Apache-2.0.txt`) | |
| | `rtmw-l-384x288-fp16.onnx` | RTMW-l (rtmw-dw-x-l_simcc-cocktail14, 384x288) | OpenMMLab | Apache-2.0 | |
| | `yolox-tiny-humanart.onnx` | YOLOX-tiny trained on Human-Art (person boxes, NMS included), OpenMMLab's end2end.onnx unchanged | OpenMMLab (YOLOX: Megvii) | Apache-2.0 | |
| |
| ## What was changed |
| |
| - **BiRefNet**: exported from the PyTorch weights at opset 17 with each torchvision `deform_conv2d` replaced by one |
| `GridSample` plus a `MatMul` (the same maths, within 5e-5 of torchvision's op; the whole model within 3e-5 on its |
| logits), so no int64 index arithmetic falls back to the CPU on WebGPU. Then onnxslim, fp16 weights with float32 inputs |
| and outputs, and the backbone's weights, which the export writes once per call (BiRefNet runs its backbone at full and |
| half scale), stored once. `scripts/export_birefnet_rt.py` and `scripts/dedupe.py` make them. Input `input_image` |
| (1, 3, S, S), RGB 0..1 normalized with ImageNet's mean and std; output `output_image`, logits (apply a sigmoid). |
| Accuracy on 100 P3M-500-NP portraits (MSE x 1e-3): 5.10 at 512, 3.60 at 768. |
| - **RTMW**: fp16 weights with float32 inputs and outputs (`scripts/convert_rtmw.py`); keypoints within 0.08 px of the |
| float32 model in its crop. Input (N, 3, H, W) RGB normalized with mean 123.675, 116.28, 103.53 and std 58.395, 57.12, |
| 57.375 (0..255 scale), a person's box grown 1.25 times at the model's aspect; outputs SimCC `simcc_x` (N, 133, 2W) and |
| `simcc_y` (N, 133, 2H): argmax / 2 is the point in the crop. |
| - **YOLOX-tiny**: unchanged. Input (1, 3, 416, 416) BGR 0..255, the picture scaled into the top left and grey (114) |
| around; outputs `dets` (1, K, 5: x1, y1, x2, y2, score) and `labels` (1, K; 0 = person). |
|
|
