jdzhang0929/SPGen-VAE

SPGen: AutoEncoder & Normal Estimator

This repository contains two components of SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generation (SIGGRAPH Asia 2025).

[Paper] [Project Page] [Code]

Folder	Description
vae/	AutoEncoder for single-channel Spherical Projection (SP) depth maps (diffusers.AutoencoderKL, 1 input / output channel, 4 latent channels).
normal_estimator/	Point-cloud normal estimator (3D UNet adapted from Shape As Points) used to orient the points lifted from the SP maps before Poisson reconstruction.

Usage

git clone https://github.com/Evergreen0929/Spherical-Projection-Shape-Generation
cd Spherical-Projection-Shape-Generation
huggingface-cli download jdzhang0929/SPGen-VAE --local-dir weights/SPGen-VAE

from diffusers import AutoencoderKL
from spgen.models.normal_estimator import NormalEstimator
from spgen.utils import set_circular_padding

vae = AutoencoderKL.from_pretrained("weights/SPGen-VAE", subfolder="vae")
set_circular_padding(vae)  # SP maps wrap around along the azimuth, the AutoEncoder uses circular padding

normal_estimator = NormalEstimator.from_pretrained("weights/SPGen-VAE/normal_estimator")

With inference.py of the code repository, pass --vae_path weights/SPGen-VAE/vae --normal_estimator_path weights/SPGen-VAE/normal_estimator.

• AutoEncoder input: one SP depth map layer of resolution 256 x 512, normalized to [-1, 1]; every layer of an object is encoded independently into a 4 x 32 x 64 latent.
• Normal estimator input: (B, N, 3) points of a shape normalized to [-0.5, 0.5] (25,600 points in the paper); the output is a unit normal per point. It runs in fp32 and requires torch_scatter.

Training

• AutoEncoder: initialized from the SDXL AutoEncoder with the RGB input / output convolutions replaced by single-channel ones and circular padding, finetuned for 40k iterations (batch size 64, lr 1e-4, cosine annealing) with the geometry regularization of the paper: L = L1 + 0.1 L_edge + 5e-4 L_spec + 1e-8 KL.
• Normal estimator: trained from scratch for 40k iterations with an L1 loss on oriented point clouds sampled from watertight remeshes of the ground-truth meshes.
• Data: the Objaverse training split listed in representations/uids/train.txt and 299 additional Sketchfab models. The held-out evaluation objects (representations/uids/test.txt) and the GSO evaluation objects were not used for training.

Training scripts: train_vae.py and train_normal_estimator.py in the code repository.

License

MIT. The normal estimator architecture is adapted from Shape As Points (MIT) and pytorch-3dunet (MIT).

Citation

@inproceedings{zhang2025spgen,
  title={SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generation},
  author={Zhang, Jingdong and Chen, Weikai and Liu, Yuan and Wang, Jionghao and Yu, Zhengming and Shen, Zhuowen and Yang, Bo and Wang, Wenping and Li, Xin},
  booktitle={SIGGRAPH Asia 2025 Conference Papers},
  year={2025},
  publisher={ACM},
  doi={10.1145/3757377.3763959}
}