| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - novel-view-synthesis |
| - gaussian-splatting |
| - image-to-3d |
| - diffusion |
| pipeline_tag: image-to-3d |
| --- |
| |
| # NovelGS |
|
|
| Official model release for [NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model](https://arxiv.org/abs/2411.16779). |
|
|
| This repository contains the official pretrained NovelGS weights. |
|
|
| ## Files |
|
|
| - `model.safetensors`: 353 NovelGS tensors without optimizer, callbacks, trainer state, or frozen LPIPS/VGG state. |
| - `weights-manifest.json`: sizes, SHA-256 hashes, tensor names, dtypes, and shapes. |
| - `SHA256SUMS`: checksums for release artifacts. |
|
|
| The 353 NovelGS tensors in safetensors were compared tensor-by-tensor with the specified original research checkpoint: key sets, shapes, dtypes, and values are exactly equal. |
|
|
| ## Sanitization and provenance |
|
|
| The published safetensors file is derived from `step=00010000.ckpt` with source SHA-256 `53eeda416228259db8226e8d9e6809717a45f9f104a1f6fcbf98e16759649f2f`. The release conversion removes 36 frozen `lpips.*` tensors (58,870,528 bytes) and clears Lightning callback state containing private machine paths. `weights-manifest.json` records the source hash, exclusions, tensor metadata, and artifact hashes. |
|
|
| ## Architecture and input |
|
|
| NovelGS uses a 24-layer transformer of width 768 to predict pixel-aligned 3D Gaussians. The released 512-pixel checkpoint consumes four posed condition images and denoises one target view. The camera input consists of normalized camera-to-world matrices and normalized `[fx, fy, cx, cy]` intrinsics. |
|
|
| Use the code release at `https://github.com/moonsliu/NovelGS` for inference and training instructions. |
|
|
| ## Loading |
|
|
| ```python |
| from safetensors.torch import load_file |
| from omegaconf import OmegaConf |
| from src.utils.train_util import instantiate_from_config |
| |
| config = OmegaConf.load("configs/train_512.yaml") |
| model = instantiate_from_config(config.model) |
| incompatible = model.load_state_dict(load_file("model.safetensors"), strict=False) |
| assert not incompatible.unexpected_keys |
| assert all(key.startswith("lpips.") for key in incompatible.missing_keys) |
| model.eval().cuda() |
| ``` |
| The LPIPS/VGG loss network is not bundled; the training code obtains it from the pinned upstream dependency. |
|
|
| ## Training data |
|
|
| The model was trained on a filtered set of approximately 270K objects derived from Objaverse 1.0, rendered into 32 random 512×512 RGBA views per object. The original assets and rendered dataset are not distributed. Objaverse objects retain their individual licenses. |
|
|
| ## Intended use and limitations |
|
|
| Intended for research on sparse-view reconstruction, Gaussian Splatting, and novel-view synthesis. Results may be geometrically incomplete or inconsistent for transparent, reflective, thin, heavily occluded, or out-of-distribution objects. The model does not establish ownership or licensing of its input or output content. |
|
|
| The optional single-image workflow uses Zero123++ weights under CC-BY-NC 4.0 and is therefore non-commercial. The core CUDA rasterizer distributed with the code has its own research-only license. These restrictions are separate from the Apache-2.0 license of NovelGS-authored code and NovelGS weights. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{liu2024novelgs, |
| title={NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model}, |
| author={Liu, Jinpeng and Xu, Jiale and Cheng, Weihao and Gao, Yiming and Wang, Xintao and Shan, Ying and Tang, Yansong}, |
| journal={arXiv preprint arXiv:2411.16779}, |
| year={2024} |
| } |
| ``` |
|
|
