kjcpc/ML-Sharp-QNN

SHARP QNN

This is a Qualcomm QNN (Qualcomm Neural Network) conversion of apple/Sharp for on-device inference on Android devices with Snapdragon processors. The models run on the Hexagon Tensor Processor (HTP) DSP via the QNN SDK, enabling fully offline, single-image 3D Gaussian Splatting reconstruction directly on a smartphone.

Models are available in multiple precision variants under dlc/:

Directory	Precision	Description
dlc/w8a16/	W8A16	Weights: UFIXED_POINT_8, Activations: UFIXED_POINT_16

Each variant contains 5 DLC files (all required):

File	Model	Description
pe.dlc	Patch Encoder	Encodes 35 image patches (384×384)
ie.dlc	Image Encoder	Encodes the full image (1536×1536)
rest_a.dlc	REST Segment A	Feature Fusion (6 inputs → 6 outputs)
rest_b.dlc	REST Segment B	Disparity Estimation
rest_c.dlc	REST Segment C	Gaussian Delta Prediction

---

Sharp Monocular View Synthesis in Less Than a Second

Original README from apple/Sharp:

This software project accompanies the research paper: Sharp Monocular View Synthesis in Less Than a Second by Lars Mescheder, Wei Dong, Shiwei Li, Xuyang Bai, Marcel Santos, Peiyun Hu, Bruno Lecouat, Mingmin Zhen, Amaël Delaunoy, Tian Fang, Yanghai Tsin, Stephan Richter and Vladlen Koltun.

We present SHARP, an approach to photorealistic view synthesis from a single image. Given a single photograph, SHARP regresses the parameters of a 3D Gaussian representation of the depicted scene. This is done in less than a second on a standard GPU via a single feedforward pass through a neural network. The 3D Gaussian representation produced by SHARP can then be rendered in real time, yielding high-resolution photorealistic images for nearby views. The representation is metric, with absolute scale, supporting metric camera movements. Experimental results demonstrate that SHARP delivers robust zero-shot generalization across datasets. It sets a new state of the art on multiple datasets, reducing LPIPS by 25–34% and DISTS by 21–43% versus the best prior model, while lowering the synthesis time by three orders of magnitude.

Getting started

Please, follow the steps in the code repository to set up your environment. Then you can download the checkpoint from the Files and versions tab above, or use the huggingface-hub CLI:

pip install huggingface-hub
huggingface-cli download --include sharp_2572gikvuh.pt --local-dir . apple/Sharp

To run prediction:

sharp predict -i /path/to/input/images -o /path/to/output/gaussians -c sharp_2572gikvuh.pt

The results will be 3D gaussian splats (3DGS) in the output folder. The 3DGS .ply files are compatible to various public 3DGS renderers. We follow the OpenCV coordinate convention (x right, y down, z forward). The 3DGS scene center is roughly at (0, 0, +z). When dealing with 3rdparty renderers, please scale and rotate to re-center the scene accordingly.

Rendering trajectories (CUDA GPU only)

Additionally you can render videos with a camera trajectory. While the gaussians prediction works for all CPU, CUDA, and MPS, rendering videos via the --render option currently requires a CUDA GPU. The gsplat renderer takes a while to initialize at the first launch.

sharp predict -i /path/to/input/images -o /path/to/output/gaussians --render -c sharp_2572gikvuh.pt

# Or from the intermediate gaussians:
sharp render -i /path/to/output/gaussians -o /path/to/output/renderings -c sharp_2572gikvuh.pt

Evaluation

Please refer to the paper for both quantitative and qualitative evaluations. Additionally, please check out this qualitative examples page containing several video comparisons against related work.

Citation

If you find our work useful, please cite the following paper:

@inproceedings{Sharp2025:arxiv,
  title      = {Sharp Monocular View Synthesis in Less Than a Second},
  author     = {Lars Mescheder and Wei Dong and Shiwei Li and Xuyang Bai and Marcel Santos and Peiyun Hu and Bruno Lecouat and Mingmin Zhen and Ama\"{e}l Delaunoy and Tian Fang and Yanghai Tsin and Stephan R. Richter and Vladlen Koltun},
  journal    = {arXiv preprint arXiv:2512.10685},
  year       = {2025},
  url        = {https://arxiv.org/abs/2512.10685},
}

Acknowledgements

Our codebase is built using multiple opensource contributions, please see ACKNOWLEDGEMENTS for more details.

---

QNN Conversion Details

Model Information

Property	Value
Precision	Multiple variants available (see dlc/ directory)
Quantization	Per-tensor scale + offset (BW_SCALE_OFFSET)
Format	QNN DLC (.dlc) — device-side compile to HTP Context Binary required
QNN SDK	2.48.0.260626
HTP Architecture	V68 and above
Input Layout	NCHW float32, value range [0, 1]
Output Layout	NCHW float32 (auto-dequantized)

Model Architecture

The 5 models form a serial inference pipeline:

Input Image
    │
    ├──► PE (Patch Encoder)     ──► 35 patch features
    │
    ├──► IE (Image Encoder)     ──► global image features
    │
    ├──► Merge                  ──► 6 fused feature maps
    │
    ├──► REST_A (Feature Fusion)──► 6 edge tensors
    │
    ├──► REST_B (Disparity)     ──► disparity map
    │
    ├──► REST_C (Gaussian Delta)──► delta parameters
    │
    └──► Post Processing        ──► output.ply (3D Gaussian Splat)

Input / Output Shapes

Model	Input Shape(s)	Output Shape(s)	Description
PE	[1, 3, 384, 384]	[1, 1024, 24, 24]	Patch encoder (×35)
IE	[1, 3, 1536, 1536]	[1, 1024, 96, 96]	Image encoder
REST_A	6 feature tensors	6 edge tensors	Feature fusion
REST_B	3 feature tensors	[1, 1, 1536, 1536]	Disparity estimation
REST_C	5 input tensors	[1, 41, 192, 256]	Gaussian delta

注：REST_A/B/C 的输入来自前序模型的输出，tensor 名称由 DLC 元数据决定，在运行时按名称自动匹配。

Usage with the Android App

1. Choose a precision variant from the dlc/ directory (e.g., dlc/w8a16/)
2. Download all 5 .dlc files from that variant
3. Place them in the app's model directory under dlc/:

   <modelRoot>/dlc/pe.dlc
   <modelRoot>/dlc/ie.dlc
   <modelRoot>/dlc/rest_a.dlc
   <modelRoot>/dlc/rest_b.dlc
   <modelRoot>/dlc/rest_c.dlc
   

4. The app will auto-detect and import them on the Models screen
5. On first use, each DLC will be compiled to a device-specific HTP Context Binary (.bin); subsequent runs use the compiled binary directly

Alternatively, you can import the models through the app's UI by selecting each .dlc file.

Hardware Requirements

• Snapdragon with HTP V68 or above (Snapdragon 8cx Gen 3 / 8 Gen 1 and newer)
• Android 12+
• ~8 GB RAM recommended (the pipeline peaks at ~2 GB during REST inference)
• The app uses QNN SDK libraries bundled in the APK; no additional setup is required

Conversion Notes

• The original PyTorch checkpoint (sharp_2572gikvuh.pt) was converted to ONNX, then to QNN DLC (.dlc) using QNN SDK 2.48.0.260626.
• DLC files are cross-architecture — the app compiles them to device-specific HTP Context Binary (.bin) on first run.
• The app supports both importing pre-compiled .bin files and compiling .dlc files on-device; this repository provides the DLC format for maximum portability.
• Quantization: Multiple precision variants available. Quantization parameters (scale/offset) are embedded in the DLC and auto-detected at runtime.
• The non-neural-network parts of the pipeline (image preprocessing, patch splitting, feature merging, unprojection, PLY export) were ported from the original Python code to C++ and run on the CPU.
• All 5 models must be from the same precision variant; mixing different precision models will produce incorrect results.