README.md

---
license: mit
tags:
- 3d-gaussian-splatting
- novel-view-synthesis
- compression
- feed-forward
---

<p align="center">
  <h1 align="center">CodecSplat: Ultra-Compact Latent Coding for Feed-Forward 3D Gaussian Splatting</h1>
  <h3 align="center">
    <a href="https://arxiv.org/abs/2605.25563">Paper</a> |
    <a href="https://github.com/pengpeng-yu/CodecSplat">Code</a> |
    <a href="https://huggingface.co/pengpeng-yu/CodecSplat">Models</a>
  </h3>
</p>

CodecSplat is a latent coding framework for feed-forward 3D Gaussian splatting.
Instead of compressing the final irregular 3D Gaussian primitives, it
entropy-codes an intermediate 2D feature representation used for depth and
Gaussian prediction. This keeps the scene representation compact while
preserving fast feed-forward reconstruction.

## Installation

The code is developed with Python 3.10 and PyTorch 2.9.1. Please refer to the official
[PyTorch installation guide](https://pytorch.org/get-started/locally/) if your
CUDA version or platform differs from the example below.

```bash
conda create -y -n py310torch291 python=3.10
conda activate py310torch291

pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
  --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt
```

## Model Zoo

Pretrained weights are available on
[Hugging Face](https://huggingface.co/pengpeng-yu/CodecSplat):

| Checkpoint | Dataset | Input views | Resolution | Notes |
| --- | --- | ---: | --- | --- |
| [CodecSplat-Re10K-2view](https://huggingface.co/pengpeng-yu/CodecSplat/resolve/main/codecsplat-base-re10k-256x256-view2-e3a545db.pth) | RealEstate10K | 2 | 256x256 | Variable-rate, lmb=16-1024 |
| [CodecSplat-DL3DV-8view](https://huggingface.co/pengpeng-yu/CodecSplat/resolve/main/codecsplat-base-dl3dv-256x448-view8-075062d0.pth) | DL3DV | 8 | 256x448 | Variable-rate, lmb=16-1024 |

After downloading a checkpoint, place it under `pretrained/` or update the
`checkpointing.pretrained_model=...` entry in the corresponding test script.

## Datasets

CodecSplat is evaluated on 256x256 RealEstate10K and 256x448 DL3DV. The
datasets used by the provided scripts are available from:

- [RealEstate10K 360p torch format](https://huggingface.co/datasets/lhmd/re10k_torch)
- [DL3DV-ALL-480P](https://huggingface.co/datasets/DL3DV/DL3DV-ALL-480P)
- [DL3DV-Benchmark](https://huggingface.co/datasets/DL3DV/DL3DV-Benchmark)

The RealEstate10K release above is already in the expected torch-chunk format.
For DL3DV, convert the downloaded source data with:

```bash
mkdir -p datasets

# DL3DV benchmark/test split.
python src/scripts/convert_dl3dv.py \
  --split test \
  --input_dir /path/to/DL3DV-Benchmark \
  --output_dir datasets/DL3DV-Benchmark-480P-Processed \
  --img_subdir images_8

# DL3DV training split. Test scenes are excluded by index.
python src/scripts/convert_dl3dv.py \
  --split train \
  --input_dir /path/to/DL3DV-ALL-480P \
  --output_dir datasets/DL3DV-ALL-480P-Processed \
  --img_subdir images_8 \
  --test_index_path datasets/DL3DV-Benchmark-480P-Processed/test/index.json

# Use the DL3DV benchmark split as the test split for DL3DV evaluation.
ln -s ../DL3DV-Benchmark-480P-Processed/test datasets/DL3DV-ALL-480P-Processed/test
```

The resulting layout should look like:

```text
datasets
├── DL3DV-ALL-480P
│   ├── 1K
│   │   ├── 0a1b7c20...
│   │   │   ├── images_8  # 270x480
│   │   │   └── transforms.json
│   │   └── ...
├── DL3DV-Benchmark
│   ├── 0a1b7c20...
│   │   ├── images_8  # 270x480
│   │   └── transforms.json
│   └── ...
├── RE10K-Torch-360p
│   ├── train
│   │   ├── 000000.torch
│   │   ├── ...
│   │   └── index.json
│   └── test
│       ├── 000000.torch
│       ├── ...
│       └── index.json
├── DL3DV-ALL-480P-Processed
│   ├── train
│   │   ├── 000000.torch
│   │   ├── ...
│   │   └── index.json
│   └── test -> ../DL3DV-Benchmark-480P-Processed/test
└── DL3DV-Benchmark-480P-Processed
    └── test
        ├── 000000.torch
        ├── ...
        └── index.json
```

## Evaluation

The main codec evaluation entry point is:

```bash
# Run both datasets at the default rate points, lmb=16 and lmb=1024.
python scripts/test_codec_bitstream_rd.py all

# Run one dataset at selected rate points.
python scripts/test_codec_bitstream_rd.py dl3dv 16 128 1024
python scripts/test_codec_bitstream_rd.py re10k 16 128 1024
```

Equivalent single-dataset shell scripts are also provided:

```bash
# DL3DV 8-view 256x448 evaluation.
lmb=1024 bash scripts/test_dl3dv_view8_256x448.sh

# RealEstate10K 2-view 256x256 evaluation.
lmb=1024 bash scripts/test_re10k_view2_256x256.sh
```

Useful test flags:

- `test.save_image=true`: save rendered target views.
- `test.save_gt_image=true`: save ground-truth target views.
- `test.save_input_images=true`: save context/input views.
- `test.save_depth=true`: save predicted depths.
- `test.save_gaussian=true`: save reconstructed Gaussian primitives as PLY.
- `test.save_video=true`: save rendered target-view sequences as MP4 when the
  evaluation index contains consecutive frames.
- `model.encoder.codec_lmb_range=[L,L]`: evaluate a fixed rate point L.
- `model.encoder.codec_eval_use_bitstream=true`: run actual compress/decompress bitstream evaluation.

## Video Rendering

CodecSplat supports video rendering through `test.save_video=true`.

```bash
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=12 \
dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_100_ctx_12v_video.json \
dataset.roots=[datasets/DL3DV-Benchmark-480P-Processed] \
dataset.image_shape=[256,448] \
dataset.test_len=1 \
test.save_video=true \
test.compute_scores=false \
test.render_chunk_size=10 \
test.stabilize_camera=true \
model.encoder.codec_lmb_range=[1024.0,1024.0] \
model.encoder.codec_eval_use_bitstream=true \
checkpointing.pretrained_model=/path/to/dl3dv_codec_checkpoint.ckpt \
checkpointing.no_strict_load=true \
output_dir=outputs/codecsplat_video_smoke
```

## Training

Training is performed in two stages.

Stage 1 trains the feed-forward Gaussian reconstruction model without the learned
feature codec. Stage 2 enables the feature codec and trains the compression path.

```bash
# DL3DV 8-view, 256x448.
bash scripts/train_dl3dv_view8_256x448_stage1.sh
bash scripts/train_dl3dv_view8_256x448_stage2.sh

# RealEstate10K 2-view, 256x256.
bash scripts/train_re10k_view2_256x256_stage1.sh
bash scripts/train_re10k_view2_256x256_stage2.sh
```

The comments in the training scripts use notation such as `1 x 8 GPUs` or
`4 x 8 GPUs`. This means per-GPU batch size times the number of GPUs. In our
experiments, the 8-GPU setting refers to 8 NVIDIA RTX 5880 Ada 48GB GPUs. On
machines with fewer GPUs, gradient accumulation can be used to approximate the
same effective batch size. For example, replacing 8 GPUs with 4 GPUs can often
be approximated by doubling `trainer.accumulate_grad_batches`.

Most provided training commands use `trainer.precision=bf16-mixed`. Geometry,
depth probability, entropy-probability, and rasterization-sensitive operations
are kept in float32 internally where needed.

The provided stage-1 training scripts warm-start from DepthSplat Gaussian
splatting checkpoints. Download the
corresponding checkpoints before training:

```bash
mkdir -p pretrained

# RealEstate10K 2-view 256x256 warm start.
wget https://huggingface.co/haofeixu/depthsplat/resolve/main/depthsplat-gs-base-re10k-256x256-view2-ca7b6795.pth -P pretrained

# DL3DV 2-6 view 256x448 warm start.
wget https://huggingface.co/haofeixu/depthsplat/resolve/main/depthsplat-gs-base-dl3dv-256x448-randview2-6-02c7b19d.pth -P pretrained
```

Stage-2 training should be initialized from the corresponding stage-1
checkpoint. Update the `checkpointing.pretrained_model` paths in
`scripts/train_*_stage2.sh` to your stage-1 output.

## Camera Conventions

The camera intrinsic matrices are normalized, with the first row divided by the
image width and the second row divided by the image height.

The camera extrinsic matrices follow the OpenCV camera-to-world convention:
`+X` right, `+Y` down, and `+Z` pointing into the scene.

## Citation

If you find this project useful, please cite CodecSplat:

```bibtex
@article{yu2026codecsplat,
  title   = {CodecSplat: Ultra-Compact Latent Coding for Feed-Forward 3D Gaussian Splatting},
  author  = {Yu, Pengpeng and Jiang, Runqing and Zhang, Qi and Li, Dingquan and Wang, Jing and Guo, Yulan},
  journal = {arXiv preprint arXiv:2605.25563},
  year    = {2026}
}
```

## Acknowledgements

This project builds on several excellent open-source projects, including
[DepthSplat](https://github.com/cvg/depthsplat),
[ReSplat](https://github.com/cvg/resplat),
[gsplat](https://github.com/nerfstudio-project/gsplat),
[Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2), and
[lossy-vae](https://github.com/duanzhiihao/lossy-vae).

We also thank the dataset authors of
[RealEstate10K](https://google.github.io/realestate10k/) and
[DL3DV](https://github.com/DL3DV-10K/Dataset).
We are grateful to the authors and maintainers for making their work available.