README.md

---
license: apple-amlr
base_model: apple/Sharp
tags:
  - qnn
  - qualcomm
  - htp
  - dlc
  - hexagon
  - gaussian-splatting
  - 3d-reconstruction
  - single-image
  - on-device
  - android
  - sharp
library_name: ml-sharp
pipeline_tag: image-to-3d
---

# SHARP QNN

This is a **Qualcomm QNN (Qualcomm Neural Network)** conversion of [apple/Sharp](https://huggingface.co/apple/Sharp) for **on-device inference** on Android devices with Snapdragon processors. The models run on the **Hexagon Tensor Processor (HTP)** DSP via the QNN SDK, enabling fully offline, single-image 3D Gaussian Splatting reconstruction directly on a smartphone.

Models are available in multiple precision variants under `dlc/`:

| Directory | Precision | Description |
|-----------|-----------|-------------|
| `dlc/w8a16/` | W8A16 | Weights: UFIXED_POINT_8, Activations: UFIXED_POINT_16 |

Each variant contains 5 DLC files (all required):

| File | Model | Description |
|------|-------|-------------|
| `pe.dlc` | Patch Encoder | Encodes 35 image patches (384×384) |
| `ie.dlc` | Image Encoder | Encodes the full image (1536×1536) |
| `rest_a.dlc` | REST Segment A | Feature Fusion (6 inputs → 6 outputs) |
| `rest_b.dlc` | REST Segment B | Disparity Estimation |
| `rest_c.dlc` | REST Segment C | Gaussian Delta Prediction |

---

# Sharp Monocular View Synthesis in Less Than a Second

*Original README from [apple/Sharp](https://huggingface.co/apple/Sharp):*

[![Project Page](https://img.shields.io/badge/Project-Page-green)](https://apple.github.io/ml-sharp/)
[![arXiv](https://img.shields.io/badge/arXiv-2512.10685-b31b1b.svg)](https://arxiv.org/abs/2512.10685)

This software project accompanies the research paper: _Sharp Monocular View Synthesis in Less Than a Second_
by _Lars Mescheder, Wei Dong, Shiwei Li, Xuyang Bai, Marcel Santos, Peiyun Hu, Bruno Lecouat, Mingmin Zhen, Amaël Delaunoy,
Tian Fang, Yanghai Tsin, Stephan Richter and Vladlen Koltun_.

![](https://huggingface.co/kjcpc/ML-Sharp-QNN/resolve/main/data/teaser.jpg)

We present SHARP, an approach to photorealistic view synthesis from a single image. Given a single photograph, SHARP regresses the parameters of a 3D Gaussian representation of the depicted scene. This is done in less than a second on a standard GPU via a single feedforward pass through a neural network. The 3D Gaussian representation produced by SHARP can then be rendered in real time, yielding high-resolution photorealistic images for nearby views. The representation is metric, with absolute scale, supporting metric camera movements. Experimental results demonstrate that SHARP delivers robust zero-shot generalization across datasets. It sets a new state of the art on multiple datasets, reducing LPIPS by 25–34% and DISTS by 21–43% versus the best prior model, while lowering the synthesis time by three orders of magnitude.

## Getting started

Please, follow the steps in the [code repository](https://github.com/apple/ml-sharp) to set up your environment. Then you can download the checkpoint from the _Files and versions_ tab above, or use the `huggingface-hub` CLI:

```bash
pip install huggingface-hub
huggingface-cli download --include sharp_2572gikvuh.pt --local-dir . apple/Sharp
```

To run prediction:

```
sharp predict -i /path/to/input/images -o /path/to/output/gaussians -c sharp_2572gikvuh.pt
```

The results will be 3D gaussian splats (3DGS) in the output folder. The 3DGS `.ply` files are compatible to various public 3DGS renderers. We follow the OpenCV coordinate convention (x right, y down, z forward). The 3DGS scene center is roughly at (0, 0, +z). When dealing with 3rdparty renderers, please scale and rotate to re-center the scene accordingly.

### Rendering trajectories (CUDA GPU only)

Additionally you can render videos with a camera trajectory. While the gaussians prediction works for all CPU, CUDA, and MPS, rendering videos via the `--render` option currently requires a CUDA GPU. The gsplat renderer takes a while to initialize at the first launch.

```
sharp predict -i /path/to/input/images -o /path/to/output/gaussians --render -c sharp_2572gikvuh.pt

# Or from the intermediate gaussians:
sharp render -i /path/to/output/gaussians -o /path/to/output/renderings -c sharp_2572gikvuh.pt
```

## Evaluation

Please refer to the paper for both quantitative and qualitative evaluations.
Additionally, please check out this [qualitative examples page](https://apple.github.io/ml-sharp/) containing several video comparisons against related work.

## Citation

If you find our work useful, please cite the following paper:

```bibtex
@inproceedings{Sharp2025:arxiv,
  title      = {Sharp Monocular View Synthesis in Less Than a Second},
  author     = {Lars Mescheder and Wei Dong and Shiwei Li and Xuyang Bai and Marcel Santos and Peiyun Hu and Bruno Lecouat and Mingmin Zhen and Ama\"{e}l Delaunoy and Tian Fang and Yanghai Tsin and Stephan R. Richter and Vladlen Koltun},
  journal    = {arXiv preprint arXiv:2512.10685},
  year       = {2025},
  url        = {https://arxiv.org/abs/2512.10685},
}
```

## Acknowledgements

Our codebase is built using multiple opensource contributions, please see [ACKNOWLEDGEMENTS](ACKNOWLEDGEMENTS) for more details.

---

# QNN Conversion Details

## Model Information

| Property | Value |
|----------|-------|
| **Precision** | Multiple variants available (see `dlc/` directory) |
| **Quantization** | Per-tensor scale + offset (BW_SCALE_OFFSET) |
| **Format** | QNN DLC (`.dlc`) — device-side compile to HTP Context Binary required |
| **QNN SDK** | 2.48.0.260626 |
| **HTP Architecture** | V68 and above |
| **Input Layout** | NCHW float32, value range [0, 1] |
| **Output Layout** | NCHW float32 (auto-dequantized) |

## Model Architecture

The 5 models form a serial inference pipeline:

```
Input Image
    │
    ├──► PE (Patch Encoder)     ──► 35 patch features
    │
    ├──► IE (Image Encoder)     ──► global image features
    │
    ├──► Merge                  ──► 6 fused feature maps
    │
    ├──► REST_A (Feature Fusion)──► 6 edge tensors
    │
    ├──► REST_B (Disparity)     ──► disparity map
    │
    ├──► REST_C (Gaussian Delta)──► delta parameters
    │
    └──► Post Processing        ──► output.ply (3D Gaussian Splat)
```

## Input / Output Shapes

| Model   | Input Shape(s)        | Output Shape(s)          | Description            |
|---------|----------------------|--------------------------|------------------------|
| **PE**  | [1, 3, 384, 384]    | [1, 1024, 24, 24]       | Patch encoder (×35)    |
| **IE**  | [1, 3, 1536, 1536]  | [1, 1024, 96, 96]       | Image encoder          |
| **REST_A** | 6 feature tensors | 6 edge tensors          | Feature fusion         |
| **REST_B** | 3 feature tensors | [1, 1, 1536, 1536]      | Disparity estimation   |
| **REST_C** | 5 input tensors  | [1, 41, 192, 256]       | Gaussian delta         |

*注：REST_A/B/C 的输入来自前序模型的输出，tensor 名称由 DLC 元数据决定，在运行时按名称自动匹配。*

## Usage with the Android App

1. Choose a precision variant from the `dlc/` directory (e.g., `dlc/w8a16/`)
2. Download all 5 `.dlc` files from that variant
3. Place them in the app's model directory under `dlc/`:
   ```
   <modelRoot>/dlc/pe.dlc
   <modelRoot>/dlc/ie.dlc
   <modelRoot>/dlc/rest_a.dlc
   <modelRoot>/dlc/rest_b.dlc
   <modelRoot>/dlc/rest_c.dlc
   ```
3. The app will auto-detect and import them on the Models screen
4. On first use, each DLC will be compiled to a device-specific HTP Context Binary (`.bin`); subsequent runs use the compiled binary directly

Alternatively, you can import the models through the app's UI by selecting each `.dlc` file.

## Hardware Requirements

- **Snapdragon with HTP V68 or above** (Snapdragon 8cx Gen 3 / 8 Gen 1 and newer)
- **Android 12+**
- **~8 GB RAM** recommended (the pipeline peaks at ~2 GB during REST inference)
- The app uses QNN SDK libraries bundled in the APK; no additional setup is required

## Conversion Notes

- The original PyTorch checkpoint (`sharp_2572gikvuh.pt`) was converted to ONNX, then to QNN DLC (`.dlc`) using QNN SDK 2.48.0.260626.
- DLC files are cross-architecture — the app compiles them to device-specific HTP Context Binary (`.bin`) on first run.
- The app supports both importing pre-compiled `.bin` files and compiling `.dlc` files on-device; this repository provides the DLC format for maximum portability.
- Quantization: Multiple precision variants available. Quantization parameters (scale/offset) are embedded in the DLC and auto-detected at runtime.
- The non-neural-network parts of the pipeline (image preprocessing, patch splitting, feature merging, unprojection, PLY export) were ported from the original Python code to C++ and run on the CPU.
- All 5 models must be from the **same precision** variant; mixing different precision models will produce incorrect results.