Publish WiLoR MLX checkpoint

README.md

@@ -0,0 +1,119 @@
+---
+license: mit
+library_name: mlx
+tags:
+  - mlx
+  - hand-pose-estimation
+  - apple-silicon
+  - wilor
+  - mano
+datasets:
+  - hand-pose
+base_model: warmshao/WiLoR-mini
+---
+
+# WiLoR-MLX: Hand Pose Estimation on Apple Silicon
+
+MLX port of [WiLoR-mini](https://github.com/warmshao/WiLoR-mini) for native Apple Silicon inference. Complete pipeline: ViT-H/16 backbone + MANO hand model + RefineNet refinement.
+
+**Code:** [github.com/lyonsno/wilor-mlx](https://github.com/lyonsno/wilor-mlx)
+
+## Available Weights
+
+| Variant | File | Size | Precision | Notes |
+|---|---|---|---|---|
+| **float32** | `wilor-mlx.safetensors` | 2.4 GB | Full | Reference quality, recommended |
+| **int4** | `wilor-mlx-int4.safetensors` | 490 MB | 4-bit quantized | 5x smaller download, same speed |
+
+Both variants produce near-identical inference speed on Apple Silicon (see benchmarks below). Choose based on download size and precision needs.
+
+These weights contain only ViT backbone, RefineNet, and learned embedding parameters — no MANO data is bundled or rehosted. `WiLoR.from_pretrained()` handles MANO automatically by fetching upstream [WiLoR-mini](https://huggingface.co/warmshao/WiLoR-mini) assets and converting locally on your machine. The [MANO hand model](https://mano.is.tue.mpg.de/) is separately licensed by the Max Planck Institute.
+
+## Performance
+
+**Apple M4 Max, single-image (1×256×256×3), float32:**
+
+### Same-harness Perceptasia saved-frame route
+
+| Stage | Backend | p50 | p90 | p95 | p99 |
+|---|---|---|---|---|---|
+| model | **MLX (wilor-mlx)** | **37.0 ms** | **37.3 ms** | **37.4 ms** | **37.7 ms** |
+| model | PyTorch MPS | 48.8 ms | 49.6 ms | 49.8 ms | 54.2 ms |
+| full route | **MLX (wilor-mlx)** | **48.8 ms** | **49.5 ms** | **49.5 ms** | **49.6 ms** |
+| full route | PyTorch MPS | 60.1 ms | 60.8 ms | 61.1 ms | 62.1 ms |
+
+Same M4 Max machine, same saved-frame harness, 40 recent active Perceptasia frames. MLX is about 1.3x faster on the pose/reconstruction model stage and about 1.2x faster on the full saved-frame route. This replaces the older app-tail telemetry as the launch comparison denominator.
+
+### Isolated model benchmark
+
+| Backend | p50 | p90 | min | FPS |
+|---|---|---|---|---|
+| **MLX (wilor-mlx)** | **36 ms** | **36 ms** | **36 ms** | **28** |
+| PyTorch MPS (2.5.0) | 50 ms | 51 ms | 49 ms | 20 |
+
+1.4x faster in pure model compute. Same deterministic input, 100 iterations after 30 warmup, batched timing.
+
+### Quantization impact on speed
+
+| Variant | p50 | FPS | Notes |
+|---|---|---|---|
+| float32 | 36 ms | 28 | Reference |
+| float16 | 36 ms | 28 | Equal ALU throughput on M4 Max |
+| int4 | 37 ms | 27 | Dequant overhead ≈ bandwidth savings |
+
+On Apple Silicon, float16 and int4 do not improve latency for this model size (210 tokens × 1280 dim). The GPU is compute-overhead-bound, not bandwidth-bound. Int4's value is purely download size reduction (2.4 GB → 490 MB).
+
+## Numerical Accuracy
+
+Compared against PyTorch WiLoR-mini on identical float32 inputs:
+
+| Variant | pred_vertices max diff | pred_keypoints_3d max diff |
+|---|---|---|
+| float32 | 0.006 (sub-mm) | 0.006 (sub-mm) |
+| int4 | 0.044 (< 2mm) | 0.044 (< 2mm) |
+
+Both are within visual tolerance for real-time hand tracking.
+
+## Quick Start
+
+```python
+from wilor_mlx import WiLoR
+import mlx.core as mx
+
+# Everything downloads and caches automatically
+# First run requires torch for one-time MANO conversion; after that, torch is not used
+model = WiLoR.from_pretrained()
+
+# Inference
+image = mx.array(your_256x256_hand_crop)  # (1, 256, 256, 3) uint8
+result = model(image)
+mx.eval(result)
+
+keypoints = result['pred_keypoints_3d']  # (1, 21, 3)
+vertices = result['pred_vertices']        # (1, 778, 3)
+```
+
+See [github.com/lyonsno/wilor-mlx](https://github.com/lyonsno/wilor-mlx) for full documentation.
+
+## Architecture
+
+- **ViT-H/16 backbone:** 1280 embed dim, 32 layers, 16 heads, 210 tokens (192 patches + 18 learnable)
+- **MANO hand model:** 778 vertices, 16 joints, Linear Blend Skinning with kinematic chain
+- **RefineNet:** Multi-scale deconvolution + bilinear grid sampling + MANO parameter refinement
+- **Total parameters:** ~610M
+
+## Citation
+
+```bibtex
+@article{zhan2024wilor,
+  title={WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild},
+  author={Zhan, Rolandos Alexandros and others},
+  year={2024}
+}
+```
+
+## License
+
+The wilor-mlx code and these weight files are MIT licensed. The weights contain only ViT backbone, RefineNet, and learned embedding parameters — no MANO data is bundled or rehosted.
+
+The [MANO hand model](https://mano.is.tue.mpg.de/) is separately licensed by the Max Planck Institute. `WiLoR.from_pretrained()` fetches upstream [WiLoR-mini](https://huggingface.co/warmshao/WiLoR-mini) assets and converts MANO data locally on your machine. You can also supply your own MANO data via `mano_path=...`.

wilor-mlx-int4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e43b1ba75eeeee4f7d7dc7938d4c2103f4d66f19918f5121b1b57522d0d277e
+size 512479104

wilor-mlx.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b81d32af65df273e58126b21c7261c57648b7f06035569ff346d4980872c0ee1
+size 2557175539