PRG-Fusion

Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Sipeng He · Jialei Chen · Zhen Fang · Dongchun Ren · Feng Zhao

Paper · Code

PRG-Fusion synthesizes photorealistic driving videos along novel trajectories by combining reconstruction evidence with generative video priors. It uses dense 3D Gaussian Splatting (3DGS) renderings for appearance control and sparse LiDAR projections for geometric control.

PRG-Fusion converts region-wise reconstruction degradation evidence into Preserve, Repair, and Generate (PRG) labels. These labels route information across the video diffusion model:

• Preserve reliable appearance from dense 3DGS renderings.
• Repair uncertain geometry using sparse LiDAR projections.
• Generate missing content with video priors.

The model follows a two-stage training strategy: geometric control is first learned from sparse LiDAR projections, followed by appearance control from dense 3DGS renderings.

Model Details

Model	PRG-Fusion
Base model	NVIDIA Cosmos-Transfer2.5-2B
Conditioning	Dense 3DGS renderings and sparse LiDAR projections
Checkpoint format	PyTorch Distributed Checkpoint (DCP)
Input	Text prompt and aligned target-trajectory control videos
Output	Novel-trajectory driving video
Evaluation domain	Waymo driving scenes

Download

Download the checkpoint directly into the layout expected by the PRG-Fusion code:

uv tool install -U huggingface_hub
hf download Post-pone/PRG-Fusion \
  --revision 19af7dc45c73a07bcee5f3a4a92bfc276bd99175 \
  --include '.metadata' '*.distcp' \
  --local-dir checkpoints/PRG-Fusion/model

The complete checkpoint is approximately 15.95 GB:

checkpoints/PRG-Fusion/
├── model/
│   ├── .metadata
│   ├── __0_0.distcp
│   └── __1_0.distcp

All three files are required. Keep the original filenames because .metadata references the two DCP shards.

Usage

Clone and install the PRG-Fusion code on a Linux NVIDIA GPU host:

git clone https://github.com/Pre-cici/PRG-Fusion.git
cd PRG-Fusion

uv venv --python 3.10
uv sync --extra cu130
uv run --extra cu130 python examples/check_environment.py

Use cu128 instead of cu130 on a CUDA 12.8 system. Follow the repository's checkpoint guide to download the remaining Cosmos tokenizer and text-conditioning components.

The demo inputs ship with the repository in assets/demo/:

assets/demo/
├── vis.mp4                  # Dense 3DGS renderings for appearance control
├── lidar.mp4                # Sparse colored LiDAR projections for geometric control
└── prg/                     # Lossless PRG region labels (preserve / repair / generate) for the dense gate

The demo configuration runs one 93-frame chunk at 960 × 640. To synthesize other trajectories, prepare an aligned 3DGS rendering, a sparse colored LiDAR projection, and PRG region labels in the same layout. Then run:

uv run --extra cu130 python examples/inference.py \
  --config configs/inference/demo.yaml

Generated videos and run metadata are written to outputs/debug/<timestamp>_demo/ by default.

Checkpoint Integrity

File	SHA-256
.metadata	93f8707b53b7f16bb84ddb6a9c7a231feb8f5c4892261ebdc2f4eab9895a13bd
__0_0.distcp	efbee0f62f9a0ad02efc2859483972d33161a018e4f37549ab664aa3a6db15c7
__1_0.distcp	d3846b51b9c4691083fa90d6cf3106add1aa6227584a9896e22e7873dcf1c4f6

Limitations

The model is designed for research on driving view synthesis. Generated videos may contain temporal, geometric, or appearance artifacts, especially under large viewpoint changes or in regions with limited reconstruction evidence. It is not intended for safety-critical driving decisions.

Acknowledgements

We thank the NVIDIA Cosmos team for making their work available. We also thank the DriveStudio and FreeVS teams for sharing their work.

License

PRG-Fusion is a derivative of NVIDIA Cosmos-Transfer2.5-2B and is released under the NVIDIA Open Model License. Built on NVIDIA Cosmos. See LICENSE.pdf and NOTICE for details.

Citation

@misc{he2026prgfusion,
  title         = {{PRG-Fusion}: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis},
  author        = {Sipeng He and Jialei Chen and Zhen Fang and Dongchun Ren and Feng Zhao},
  year          = {2026},
  eprint        = {2609.06948},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.06948}
}