Add SHARP NDC ONNX FP32 (opset18) + external data

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sharp_ndc_opset18.onnx.data filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,6 @@
+# Colab / notebooks
+.ipynb_checkpoints/
+
+# If you ever use upload_large_folder, it creates a local cache directory
+cache/
+**/cache/huggingface/

LICENSE

@@ -0,0 +1,47 @@
+Copyright (C) 2025 Apple Inc. All Rights Reserved.
+
+Disclaimer: IMPORTANT:  This Apple software is supplied to you by Apple
+Inc. ("Apple") in consideration of your agreement to the following
+terms, and your use, installation, modification or redistribution of
+this Apple software constitutes acceptance of these terms.  If you do
+not agree with these terms, please do not use, install, modify or
+redistribute this Apple software.
+
+In consideration of your agreement to abide by the following terms, and
+subject to these terms, Apple grants you a personal, non-exclusive
+license, under Apple's copyrights in this original Apple software (the
+"Apple Software"), to use, reproduce, modify and redistribute the Apple
+Software, with or without modifications, in source and/or binary forms;
+provided that if you redistribute the Apple Software in its entirety and
+without modifications, you must retain this notice and the following
+text and disclaimers in all such redistributions of the Apple Software.
+Neither the name, trademarks, service marks or logos of Apple Inc. may
+be used to endorse or promote products derived from the Apple Software
+without specific prior written permission from Apple.  Except as
+expressly stated in this notice, no other rights or licenses, express or
+implied, are granted by Apple herein, including but not limited to any
+patent rights that may be infringed by your derivative works or by other
+works in which the Apple Software may be incorporated.
+
+The Apple Software is provided by Apple on an "AS IS" basis.  APPLE
+MAKES NO WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION
+THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS
+FOR A PARTICULAR PURPOSE, REGARDING THE APPLE SOFTWARE OR ITS USE AND
+OPERATION ALONE OR IN COMBINATION WITH YOUR PRODUCTS.
+
+IN NO EVENT SHALL APPLE BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL
+OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) ARISING IN ANY WAY OUT OF THE USE, REPRODUCTION,
+MODIFICATION AND/OR DISTRIBUTION OF THE APPLE SOFTWARE, HOWEVER CAUSED
+AND WHETHER UNDER THEORY OF CONTRACT, TORT (INCLUDING NEGLIGENCE),
+STRICT LIABILITY OR OTHERWISE, EVEN IF APPLE HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+
+-------------------------------------------------------------------------------
+SOFTWARE DISTRIBUTED IN THIS REPOSITORY:
+
+This software includes a number of subcomponents with separate
+copyright notices and license terms - please see the file ACKNOWLEDGEMENTS.
+-------------------------------------------------------------------------------

README.md

@@ -0,0 +1,74 @@
+---
+library_name: onnxruntime
+tags:
+- onnx
+- apple
+- sharp
+- view-synthesis
+- 3d-gaussian-splatting
+base_model: apple/Sharp
+---
+
+# SHARP (NDC) — ONNX FP32 (opset 18)
+
+This repository contains an **ONNX FP32** export of Apple's **SHARP** model ("Single-image view synthesis via 3D Gaussians"), exported from the official checkpoint **sharp_2572gikvuh.pt**.
+
+## What this ONNX contains (important)
+This ONNX exports **only the core predictor network**:
+
+`gaussians_ndc = predictor(image_resized_pt, disparity_factor)`
+
+It outputs **3D Gaussians in NDC space**.  
+It does **NOT** include the NDC→metric unprojection step because Apple's unprojection path uses SVD + CPU fp64 conversions, which is not a good fit for ONNX portability.
+
+If you want metric-space Gaussians and/or `.ply` export, do that step outside ONNX (e.g., reuse Apple’s `sharp.utils.gaussians.unproject_gaussians` in Python).
+
+## Files
+- `sharp_ndc_opset18.onnx` (graph)
+- `sharp_ndc_opset18.onnx.data` (external weights — required!)
+- `export_config.json`, `run_manifest.json`, `parity_metrics.json` for reproducibility
+
+## Input contract
+**Inputs**
+- `image_resized_pt`: float32 tensor of shape **[1, 3, 1536, 1536]**
+  - RGB
+  - normalized to **[0, 1]** (divide by 255)
+  - resized with **bilinear** and **align_corners=True**
+  - layout **NCHW**
+- `disparity_factor`: float32 tensor of shape **[1]**
+  - computed as: `disparity_factor = f_px / width_original`
+
+Where:
+- `width_original` is the input image width before resizing
+- `f_px` is focal length in pixels (Apple code defaults if EXIF is missing)
+
+**Outputs**
+- `mean_vectors`: [1, 1179648, 3]
+- `singular_values`: [1, 1179648, 3]
+- `quaternions`: [1, 1179648, 4]
+- `colors`: [1, 1179648, 3]
+- `opacities`: [1, 1179648]
+
+All outputs are float32.
+
+## Minimal ONNX Runtime example
+
+```python
+import numpy as np
+import onnxruntime as ort
+
+sess = ort.InferenceSession(
+    "sharp_ndc_opset18.onnx",
+    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
+)
+
+# Provide:
+# - image_resized_pt as np.float32 [1,3,1536,1536]
+# - disparity_factor as np.float32 [1]
+outputs = sess.run(
+    ["mean_vectors","singular_values","quaternions","colors","opacities"],
+    {"image_resized_pt": image_resized_pt, "disparity_factor": disparity_factor},
+)
+```
+
+

export_config.json

@@ -0,0 +1,24 @@
+{
+  "onnx_path": "/content/sharp_work/phase3/onnx/sharp_ndc_opset18.onnx",
+  "opset_version": 18,
+  "inputs": {
+    "image_resized_pt": [
+      1,
+      3,
+      1536,
+      1536
+    ],
+    "disparity_factor": [
+      1
+    ]
+  },
+  "outputs": [
+    "mean_vectors",
+    "singular_values",
+    "quaternions",
+    "colors",
+    "opacities"
+  ],
+  "external_data": true,
+  "dynamo": true
+}

inference_onnx.py

@@ -0,0 +1,23 @@
+import numpy as np
+from PIL import Image
+import onnxruntime as ort
+
+def preprocess(image_path: str, f_px: float) -> tuple[np.ndarray, np.ndarray]:
+    img = Image.open(image_path).convert("RGB")
+    w, h = img.size
+    x = np.asarray(img).astype(np.float32) / 255.0  # HWC [0,1]
+    x = np.transpose(x, (2, 0, 1))[None, ...]       # NCHW
+    # Resize to 1536x1536 with bilinear + align_corners=True:
+    # For a minimal example, rely on ORT/consumer to match training preprocessing.
+    # (For exact match, use the same resize code as Apple.)
+    disparity_factor = np.array([f_px / float(w)], dtype=np.float32)
+    return x.astype(np.float32), disparity_factor
+
+if __name__ == "__main__":
+    sess = ort.InferenceSession(
+        "sharp_ndc_opset18.onnx",
+        providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
+    )
+    print("Providers:", sess.get_providers())
+    # Example usage requires you to supply f_px (or choose an approximate default).
+    # image_resized_pt should be [1,3,1536,1536] — see README for exact contract.

parity_metrics.json

@@ -0,0 +1,72 @@
+{
+  "providers": [
+    "CUDAExecutionProvider",
+    "CPUExecutionProvider"
+  ],
+  "outputs": {
+    "mean_vectors": {
+      "shape": [
+        1,
+        1179648,
+        3
+      ],
+      "dtype": "float32",
+      "max_abs": 0.13911914825439453,
+      "mean_abs": 0.00018130234093405306,
+      "ref_max_abs": 6.372216701507568,
+      "nan_out": false,
+      "inf_out": false
+    },
+    "singular_values": {
+      "shape": [
+        1,
+        1179648,
+        3
+      ],
+      "dtype": "float32",
+      "max_abs": 0.035487107932567596,
+      "mean_abs": 4.6182478399714455e-05,
+      "ref_max_abs": 0.07875056564807892,
+      "nan_out": false,
+      "inf_out": false
+    },
+    "quaternions": {
+      "shape": [
+        1,
+        1179648,
+        4
+      ],
+      "dtype": "float32",
+      "max_abs": 13.2420015335083,
+      "mean_abs": 0.008916228078305721,
+      "ref_max_abs": 20.611085891723633,
+      "nan_out": false,
+      "inf_out": false
+    },
+    "colors": {
+      "shape": [
+        1,
+        1179648,
+        3
+      ],
+      "dtype": "float32",
+      "max_abs": 0.06479707360267639,
+      "mean_abs": 0.0002580047002993524,
+      "ref_max_abs": 0.9927035570144653,
+      "nan_out": false,
+      "inf_out": false
+    },
+    "opacities": {
+      "shape": [
+        1,
+        1179648
+      ],
+      "dtype": "float32",
+      "max_abs": 0.6220522522926331,
+      "mean_abs": 0.008647891692817211,
+      "ref_max_abs": 1.0,
+      "nan_out": false,
+      "inf_out": false
+    }
+  }
+}

requirements.txt

@@ -0,0 +1,3 @@
+numpy
+pillow
+onnxruntime-gpu

run_manifest.json

@@ -0,0 +1,10 @@
+{
+  "python": "3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0]",
+  "platform": "Linux-6.6.105+-x86_64-with-glibc2.35",
+  "torch": "2.9.0+cu126",
+  "torch_cuda": "12.6",
+  "cuda_available": true,
+  "gpu": "NVIDIA L4",
+  "checkpoint_path": "/content/sharp_work/weights/sharp_2572gikvuh.pt",
+  "checkpoint_sha256": "94211a75198c47f61fca7d739ba08a215418d8d398d48fddf023baccc24f073d"
+}

sha256sums.txt

@@ -0,0 +1,5 @@
+97c7c35a1e5ff1c1d0762556952cfeb5cc0ff3915cd0118b8c1c0359829b59de  sharp_ndc_opset18.onnx
+23caa148af9590d4880c47fb69ba1dbd56dde91eab50e2c3d9254ddc1d001604  sharp_ndc_opset18.onnx.data
+d5e13c449dd15f8538c26566bbbffa10172278b9d5f6e500a27c0c7c837f3d4f  export_config.json
+5613ce0464dfbd2bc0902e2f874a836ea800dc6e2ec1a8e9213d98360e15bab7  run_manifest.json
+438d4881b4ece849eaea46065aeaa4243a260648a5619b330f31c30b730fbcf4  parity_metrics.json

sharp_ndc_opset18.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97c7c35a1e5ff1c1d0762556952cfeb5cc0ff3915cd0118b8c1c0359829b59de
+size 7279677

sharp_ndc_opset18.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23caa148af9590d4880c47fb69ba1dbd56dde91eab50e2c3d9254ddc1d001604
+size 2616066048