Create README.md

README.md

@@ -0,0 +1,478 @@
+
+
+
+
+⸻
+
+license: mit
+library_name: onnxruntime
+pipeline_tag: image-to-3d
+tags:
+
+* image-to-3d
+* 3d
+* gaussian-splatting
+* 3d-gaussian-splatting
+* webgpu
+* onnx
+* onnxruntime-web
+* browser
+* local-inference
+* triposplat
+* computer-vision
+    base_model:
+* VAST-AI/TripoSplat
+
+⸻
+
+TripoSplat WebGPU
+
+Browser-local WebGPU conversion of TripoSplat for single-image generation of 3D Gaussian splats.
+
+This repository contains ONNX model artifacts prepared for the experimental TripoSplat WebGPU runtime.
+
+The goal is to run the TripoSplat image-to-3D pipeline locally in a WebGPU-capable browser using ONNX Runtime WebGPU, without uploading the source image to a remote inference service.
+
+Alpha engineering release
+
+This is not yet a production-ready browser model. The complete five-stage pipeline executes and exports structurally valid Gaussian scenes on the recorded test system, but official end-to-end numerical and rendered-image parity has not yet been established. See Validation status and Limitations.
+
+Model description
+
+TripoSplat converts one 2D image into a variable-resolution 3D Gaussian representation.
+
+The original model was developed by TripoAI and released by VAST-AI Research. This repository does not train a new model or alter the intended learned task. It converts and packages the released TripoSplat pipeline for execution through ONNX Runtime WebGPU.
+
+Inputs
+
+A single RGB image containing an object or foreground subject.
+
+For the current WebGPU runtime:
+
+* Prepared images with an isolated foreground can be processed directly.
+* Arbitrary photographs require a caller-provided background-removal stage.
+* Browser-local BiRefNet background removal is planned but is not yet bundled and qualified.
+* Results are generally more predictable when the subject is clearly visible, centered, and not heavily occluded.
+
+Outputs
+
+A 3D Gaussian scene containing up to 262,144 Gaussians, exportable as:
+
+* .ply
+* .splat
+
+The scene contains Gaussian positions, appearance, opacity, orientation, and scale data suitable for compatible Gaussian-splat viewers and downstream tools.
+
+Intended uses
+
+This model repository is intended for:
+
+* Browser-local image-to-3D research
+* WebGPU portability experiments
+* ONNX Runtime WebGPU validation
+* Gaussian-splat generation prototypes
+* Interactive 3D creation tools
+* AR, VR, spatial-computing, game, and simulation experiments
+* Comparing browser inference against the official PyTorch implementation
+* Research into local and privacy-preserving generative inference
+
+It is currently best treated as an engineering and interoperability release, not a production inference endpoint.
+
+Out-of-scope uses
+
+This release should not currently be relied upon for:
+
+* Guaranteed numerical equivalence with official TripoSplat
+* Guaranteed visual equivalence with official TripoSplat
+* Production workloads requiring stable latency or memory bounds
+* Automated processing on unqualified low-memory devices
+* Safety-critical, medical, legal, forensic, or measurement applications
+* Accurate physical reconstruction from one photograph
+* Recovery of unseen geometry as factual ground truth
+* Deployment without reviewing browser, CDN, analytics, and application privacy behavior
+
+Generated geometry and appearance are model predictions. They should not be interpreted as a complete or physically verified reconstruction of the depicted object.
+
+Pipeline architecture
+
+The browser pipeline consists of five principal stages:
+
+1. DINOv3 image encoder
+    * Encodes the prepared source image into visual features.
+2. Flux VAE encoder
+    * Produces latent image conditioning.
+3. Diffusion Transformer
+    * Performs classifier-free-guided flow sampling.
+    * The browser runtime supports the official fast 4-step and quality 20-step schedules.
+4. Dynamic octree
+    * Predicts and refines the spatial distribution of Gaussian centers over eight octree levels.
+5. Gaussian feature decoder
+    * Converts the final learned features into Gaussian scene attributes.
+
+Sampling, octree traversal, scene construction, and export coordination are implemented in TypeScript around ONNX Runtime WebGPU sessions.
+
+Repository contents
+
+The model bundle uses an immutable manifest describing every required ONNX graph and external-data object.
+
+Current bundle characteristics:
+
+Property	Value
+Numerical format	FP32
+Total declared model size	6,465,182,402 bytes
+Number of manifest objects	10
+Maximum generated Gaussian count	262,144
+Browser execution backend	ONNX Runtime WebGPU
+WASM fallback used for reported tests	No
+
+Some ONNX graphs use external-data sidecars. The graph and sidecar files must retain the exact paths encoded in each ONNX graph.
+
+Do not rename or flatten external-data files without rewriting the corresponding ONNX graph.
+
+Using the model
+
+The model artifacts are designed for the companion runtime:
+
+https://github.com/yosun/TripoSplatWebGPU
+
+Clone and start the development application:
+
+git clone https://github.com/yosun/TripoSplatWebGPU.git
+cd TripoSplatWebGPU
+corepack enable
+pnpm install --frozen-lockfile
+pnpm dev
+
+Open:
+
+http://localhost:<port>/e2e-lab.html
+
+The end-to-end lab can:
+
+1. Load a prepared source image
+2. Execute the five-stage WebGPU pipeline
+3. Construct the Gaussian scene
+4. Export .ply and .splat
+5. Load the exported result into the browser viewer
+6. Produce a structured qualification report
+
+The laboratory pages are validation surfaces, not polished end-user interfaces.
+
+Hosting the model files
+
+Because the current model bundle is approximately 6.47 GB, production deployments should serve it from static object storage or a model CDN rather than proxying it through an application server.
+
+The host should support:
+
+* HTTPS
+* Cross-origin resource sharing
+* Byte-range requests
+* Long-lived caching
+* Immutable versioned object paths
+* Correct MIME types
+* Exact ONNX external-data locations
+* Sufficient file-size limits
+
+Appropriate deployment patterns include object storage combined with a CDN.
+
+Application serverless functions should not relay the model objects unless the platform is explicitly designed for multi-gigabyte static model delivery.
+
+Validation status
+
+Validation was performed against deterministic fixtures generated from the untouched official PyTorch implementation.
+
+Component-level validation
+
+Component	Current result
+TripoSplat preprocessing and DINOv3 encoder	Strict pass on one recorded fixture
+Flux VAE encoder	Pass on one recorded fixture
+One FP32 DiT invocation	Strict pass
+Four-step CFG/Euler loop	Qualification pass, strict diagnostic fail
+Twenty-step guided loop	Qualification fail and strict diagnostic fail
+Full eight-level octree trajectory	Pass
+Raw Gaussian feature decoder	Pass
+Packaged five-stage browser execution	Structural and viewer pass
+Official whole-scene parity	Not established
+Official render parity	Not established
+Repeated-generation memory qualification	Not established
+16 GB device qualification	Not established
+Microsoft Edge qualification	Not established
+
+A component pass means that the recorded fixture met the declared numerical thresholds for that specific test. It does not establish correctness for all images or devices.
+
+Recorded test environment
+
+The published browser measurements were collected on:
+
+Property	Test environment
+Device	Apple M3 Max Mac
+Unified memory	128 GB
+Operating system	macOS 26.3
+Browser	Chrome 150
+Runtime	ONNX Runtime WebGPU 1.27.0
+WASM fallback	Disabled
+
+These results must not be extrapolated to lower-memory systems.
+
+Recorded component measurements
+
+Stage	Recorded result
+DINOv3	8,702.1 ms cold model/session load; 4,262.4 ms inference
+Flux VAE	1,304.1 ms median inference
+One FP32 DiT invocation	12,584.6 ms cold model/session load; 11,844.6 ms inference
+Four-step sampling loop	102,268.8 ms sampling wall time across eight DiT calls
+Twenty-step sampling loop	676,669.1 ms sampling wall time across forty DiT calls
+Eight-level octree	1,327.4 ms model load; 5,713.7 ms primary trajectory inference
+Gaussian decoder	7,929.6 ms model load; 5,370.6 ms inference
+Packaged four-step generation	247,901.5 ms through generate() in the recorded qualification run
+Final packaged consumer run	210,158.7 ms through PLY and .splat export
+
+These measurements include different combinations of loading, temporary artifact preparation, pipeline execution, export, and viewer qualification. They are engineering measurements from one machine, not standardized product latency claims.
+
+Known limitations
+
+End-to-end parity remains open
+
+Individual stages can closely match their PyTorch fixtures while small discrepancies accumulate across iterative sampling and scene decoding.
+
+The current release does not claim:
+
+* Whole-scene numerical parity
+* Rendered-image parity
+* Equivalent image quality to the official implementation
+* Equivalent behavior across arbitrary inputs
+
+Twenty-step sampling is not qualified
+
+The official 20-step quality schedule completes all expected browser model calls, but its final latent state currently fails the declared qualification thresholds.
+
+Use the 20-step path for investigation, not as a verified parity path.
+
+Four-step sampling has a diagnostic discrepancy
+
+The 4-step schedule passes the current qualification envelope but fails a stricter diagnostic comparison. It is sufficient for structural end-to-end testing, not for claiming exact official parity.
+
+High memory requirements
+
+The current model bundle is FP32 and approximately 6.47 GB before accounting for:
+
+* WebGPU buffers
+* Worker allocations
+* Decoded model data
+* Intermediate tensors
+* Browser overhead
+* Export buffers
+* Viewer resources
+* Unified-memory duplication or driver behavior
+
+Browser heap counters do not expose all worker, GPU, driver, or unified-memory allocations. A reliable peak-memory figure is not yet available.
+
+Limited hardware qualification
+
+The current measured environment has 128 GB of unified memory.
+
+No claim is made yet for:
+
+* 16 GB Macs
+* Integrated GPUs with limited shared memory
+* Mobile browsers
+* Mobile GPUs
+* Intel Macs
+* Discrete Windows GPUs
+* Linux browser configurations
+* Safari WebGPU
+* Firefox WebGPU
+
+Background removal is not bundled
+
+The TripoSplat WebGPU package currently expects either:
+
+* A prepared foreground input, or
+* A caller-provided local removeBackground implementation
+
+Background-removal behavior can materially affect the generated scene.
+
+Single-view ambiguity
+
+A single image does not fully specify hidden surfaces, depth, scale, material, or topology. The model may invent or simplify unseen geometry.
+
+Input sensitivity
+
+Results may degrade for:
+
+* Multiple overlapping objects
+* Heavy occlusion
+* Transparent or reflective objects
+* Very thin structures
+* Motion blur
+* Extreme crops
+* Small subjects
+* Highly cluttered backgrounds
+* Ambiguous silhouettes
+* Unusual camera projections
+* Text-heavy or diagrammatic inputs
+
+Privacy
+
+The companion runtime is designed for browser-local inference.
+
+Under the intended architecture:
+
+* Source images are decoded in the browser.
+* Source images are not sent to a model inference server.
+* Model computation runs through local WebGPU execution.
+* Generated Gaussian data can remain local unless the surrounding application uploads it.
+
+Browser-local inference does not make every deployment private by default. Deployers must separately review:
+
+* Analytics
+* Error reporting
+* Authentication
+* CDN logs
+* Signed URLs
+* Browser extensions
+* Application uploads
+* Published result pages
+* Content Security Policy
+* Retention and deletion behavior
+
+Do not log source images or signed model URLs unintentionally.
+
+Security considerations
+
+Model files should be treated as immutable executable inputs to the ONNX runtime.
+
+Deployers should:
+
+* Verify artifact hashes against the manifest
+* Serve versioned immutable files
+* Restrict allowed model origins
+* Configure a suitable Content Security Policy
+* Avoid loading arbitrary user-supplied ONNX graphs
+* Validate exported scene sizes before allocating viewer resources
+* Keep ONNX Runtime and browser versions current
+* Avoid exposing private CDN credentials to browser code
+* Rate-limit any optional publishing or storage service
+
+Bias and representational limitations
+
+This conversion inherits the behavior and limitations of the original TripoSplat model and its training data.
+
+The WebGPU project did not train the underlying model and does not currently have sufficient information to provide a complete audit of:
+
+* Training-data composition
+* Geographic representation
+* Cultural representation
+* Object-category balance
+* Copyright status of every training example
+* Performance disparities across image domains
+
+Users should evaluate the model on their own intended data and document domain-specific failure patterns before deployment.
+
+Environmental considerations
+
+Browser-local execution transfers computation from centralized inference infrastructure to the user’s device.
+
+Potential advantages include:
+
+* No dedicated inference server for each generation
+* Local reuse of cached model artifacts
+* Reduced image-upload requirements
+
+Potential costs include:
+
+* A large initial model download
+* Significant local GPU and memory use
+* Long FP32 execution times
+* Repeated energy use on client hardware
+
+Quantized models, reduced precision, graph fusion, model partitioning, and artifact deduplication have not yet been fully qualified.
+
+Relationship to the original model
+
+This repository is an unofficial WebGPU conversion and engineering project.
+
+It is not the official TripoSplat repository and is not presented as an official TripoAI browser release.
+
+The official sources remain the reference for:
+
+* Model architecture
+* Learned weights
+* PyTorch numerical behavior
+* Intended generation schedules
+* Research claims
+* Original licensing
+* Scientific citation
+
+Official model:
+
+VAST-AI/TripoSplat
+
+Official implementation:
+
+https://github.com/VAST-AI-Research/TripoSplat
+
+WebGPU implementation:
+
+https://github.com/yosun/TripoSplatWebGPU
+
+Licensing
+
+The original TripoSplat code and released model weights are provided under the MIT License by their authors.
+
+The ONNX artifacts in this repository are conversions of those released weights and remain subject to the applicable upstream license and notices.
+
+The companion WebGPU source repository contains additional runtime and inherited software components. Review its THIRD_PARTY_NOTICES.md before redistributing the complete application or publishing a derived software package.
+
+This repository does not grant rights over user-provided input images or generated content that are not otherwise held by the user.
+
+Attribution
+
+Original model
+
+TripoSplat was developed by TripoAI and released through VAST-AI Research.
+
+WebGPU conversion and runtime
+
+The ONNX/WebGPU conversion, browser runtime, validation harnesses, packaging, and Gaussian export integration are maintained by Yosun Chang / AI3D.
+
+This work builds from the ml-sharp-web browser application chassis while keeping SHARP-specific inference and TripoSplat inference logically separate.
+
+Citation
+
+Please cite the original TripoSplat paper when using the model:
+
+@misc{yan2026generative3dgaussianslearned,
+  title         = {Generative 3D Gaussians with Learned Density Control},
+  author        = {Runjie Yan and Yan-Pei Cao and Peng Wang and Ding Liang and Yuan-Chen Guo},
+  year          = {2026},
+  eprint        = {2605.16355},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.GR},
+  url           = {https://arxiv.org/abs/2605.16355}
+}
+
+When discussing this specific browser conversion, also reference:
+
+Yosun Chang. TripoSplat WebGPU: an experimental browser-local
+ONNX Runtime WebGPU conversion of TripoSplat. 2026.
+https://github.com/yosun/TripoSplatWebGPU
+
+Development status
+
+This project is under active development.
+
+The principal remaining release gates are:
+
+* Resolve iterative sampling drift
+* Establish official whole-scene numerical parity
+* Establish rendered-image parity
+* Qualify repeated generations
+* Measure peak total memory
+* Test 16 GB hardware
+* Test additional browsers and GPU families
+* Bundle and validate local background removal
+* Qualify reduced-precision or compressed artifacts
+* Publish a stable package and immutable model revision
+
+For current evidence, reports, fixtures, and contribution instructions, see the companion source repository.