Mirror of mlboydaisuke/TripoSplat-CoreAI

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+decode_fp32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+dinov3_fp16.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+dit_fp16.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+gs_fp16.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+octree_fp32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+vae_fp16.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,87 @@
+---
+license: mit
+pipeline_tag: image-to-3d
+tags:
+- coreai
+- core-ml
+- apple-silicon
+- gaussian-splatting
+- image-to-3d
+- 3d-generation
+base_model:
+- VAST-AI/TripoSplat
+---
+
+> **Mirror** of [`mlboydaisuke/TripoSplat-CoreAI`](https://huggingface.co/mlboydaisuke/TripoSplat-CoreAI) — the canonical repo ([CoreAI Model Zoo](https://github.com/john-rocky/coreai-model-zoo)). Updates land there first.
+
+
+# TripoSplat → Core AI (zoo's first 3D)
+
+[VAST-AI/TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) — **single image → 3D Gaussian
+splats** (`.ply`/`.splat`), MIT. The zoo's first 3D model: outputs drop straight into a Gaussian-splat
+viewer (e.g. Apple RealityKit on visionOS, or MetalSplatter on iOS/macOS).
+
+Pure-PyTorch pipeline (no diffusers/CUDA kernels): bg-removal → DINOv3 ViT-H encode + Flux2-VAE encode
+→ 20-step flow-matching DiT denoiser → octree probability sampler → Gaussian decoder → splats.
+
+**This repo holds the Core AI `.aimodel` bundles** (each is a directory). Conversion + runner scripts
+live in the [coreai-models-community](https://huggingface.co/mlboydaisuke) zoo (`conversion/triposplat/`).
+
+<!-- gen-cards:use-it begin id=triposplat (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
+## Use it
+
+▶️ **Run it (source)** — [`apps/TripoSplatMac`](https://github.com/john-rocky/coreai-model-zoo/tree/main/apps/TripoSplatMac),
+the zoo app that ships this model (single image → 3D Gaussian splats on Mac: 5 converted nets + host flow sampler and octree resampling; build & run steps in its README).
+
+<!-- gen-cards:use-it end -->
+
+## What runs on Core AI
+
+5 neural nets converted (each gated converted-vs-eager **cos = 1.000000**):
+
+| net | shape | bundle | dtype |
+|---|---|---|---|
+| DINOv3 ViT-H encoder | (1,3,1024,1024)→(1,4101,1280) | `dinov3_fp16.aimodel` | fp16 |
+| Flux2-VAE encoder | (1,3,1024,1024)→(1,4096,128) | `vae_fp16.aimodel` | fp16 |
+| DiT denoiser (one step) | latent(1,8192,16)+cam(1,1,5)+t+feat1(1,4101,1280)+feat2(1,4101,128)→latent,cam | `dit_fp16.aimodel` | fp16 |
+| Octree probability decoder | x(1,8192,3)+l(1,)+cond(1,8192,16)→logits(1,8192,8) | `octree_fp32.aimodel` | fp32 |
+| Decode (gs + build_gaussians + .ply activations, baked) | points(1,8192,3)+cond(1,8192,16)→(262144,14) | `decode_fp32.aimodel` | fp32 |
+
+The flow-matching sampler (`FlowEulerCfgSampler`) and the octree `sample_probs` systematic resampling
+stay **host-side** (data-dependent control flow). Scripts: `_conv_*.py` convert+gate each net;
+`_conv_fp16.py` makes the half-size fp16 bundles; `_conv_decode.py` bakes build_gaussians + the
+Gaussian `.ply`-activation math into one net so the runner just writes raw floats.
+
+## model.py patches (the reusable contribution — see the zoo's conversion guide)
+
+coreai-torch 0.4.0 needed six edits to VAST's `model.py`; all are general gotchas:
+
+1. **float-arg `aten.arange`** → `bad_optional_access` C++ abort. Use int-arg arange (DINOv3 RoPE).
+2. **fx `got multiple values for 'mod'`** — submodule called with `mod=` kwarg. Pass positionally.
+3. **No complex ops** — rewrote the DiT's complex RoPE (`torch.polar`/`view_as_complex`) as real
+   cos/sin math (`apply_rotary_emb`, `RePo3DRotaryEmbedding.forward`).
+4. **Constant-folded `sin/cos` of huge args is low-precision** (cos→0.5) — the DiT positional embed
+   computed from the fixed Sobol constant was folded wrong; precompute it into a `register_buffer`.
+5. **`F.normalize` drops the eps clamp** → near-zero vectors blow up ~1e13; rewrote `MultiHeadRMSNorm`
+   as explicit `x*rsqrt(mean(x²)+eps)`. (Emergent only at large seq len — gate by VISUAL/true-scale.)
+6. **`prog.optimize()` hangs** on the 24-block/12k-token DiT graph (>90 min) — skip it
+   (`convert(optimize=False)`), AOT `coreai-build` optimizes for the device anyway.
+
+Plus: int8 desaturates this model (per-net cos 0.9998 but colors collapse → use **fp16**, which is
+GPU-identical to fp32 — gate fp16 on GPU/visual, its CPU cos looks bad but that's a CPU-compute
+artifact). Octree decoder: int64 `l` (resolution) input → CoreAIError 3 at runtime, pass it as float32.
+
+## Running it
+
+- **Mac**: `_run_coreai.py` (or `app_backend.py --input <img>`) loads the bundles via coreai.runtime
+  (`SpecializationOptions.default()` = GPU; ~2 min/gen at 20 steps on Apple silicon, full quality).
+  End-to-end latent gate vs torch-DiT: **cos 0.999999**.
+- **Mac app / iPhone client**: `TripoSplatMac` (standalone) and `TripoSplatPhone`
+  (capture on iPhone → Mac server `server.py` → view splats in MetalSplatter / RealityKit).
+
+## On-device note
+
+Full on-device (iPhone) was **verified infeasible** with this model: DINOv3 ViT-H AOT `.aimodelc`
+is ~3.1 GB and the DiT's 12294-token full-attention score matrix alone is ~4.8 GB, both over the
+~3.3 GB iOS app memory budget (weight precision doesn't fix the attention working set). Needs
+flash-attention conversion / weight streaming. The Mac-link client is the shipped path.

decode_fp32.aimodel/main.hash

@@ -0,0 +1 @@
+C��{_���4NS�S*�evZ>�{���&B

decode_fp32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43f3e57b5ff3e37fca344e5303d0532a08e8bd65765a3ea77bc8e0d71c261742
+size 949402894

decode_fp32.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

dinov3_fp16.aimodel/main.hash

@@ -0,0 +1 @@
+inXP�ܩJ�\�������J��qT��6	�Q�

dinov3_fp16.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:696e58509edca94afe5cb5ac841ae413f0f58b4ae8fe107154a9b73609a751bd
+size 1681455018

dinov3_fp16.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

dit_fp16.aimodel/main.hash

@@ -0,0 +1 @@
+�q�^o8^YH�n�g��g�V�����\��G��

dit_fp16.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc710bb65e6f1e385e59488e6ea2678bbe67a656f4adf7dddc5c91b647b9e79c
+size 758927980

dit_fp16.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

gs_fp16.aimodel/main.hash

@@ -0,0 +1 @@
+&��贜�+%>|� o�*w�H�B����z

gs_fp16.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26ffaae8b49ca31d2b253e7c9f206ff1a42a778e1c4807ed42b0860fc6cb1a7a
+size 475102303

gs_fp16.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

octree_fp32.aimodel/main.hash

@@ -0,0 +1 @@
+e}ž��$v����4M��;��BC�V��a��

octree_fp32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:657dc29ebbb92476e2d6e6cf344d8aa53bcce291424319c056c0f09161eb15a6
+size 203236017

octree_fp32.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

vae_fp16.aimodel/main.hash

@@ -0,0 +1 @@
+�	K$ k�6,TA-�kh�y(@a�衉&N�I

vae_fp16.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc094b24206beb8f36042c54412dd66b68de7928401961a6e8a18916264ee649
+size 68919930

vae_fp16.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}