README.md

---
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: image-to-3d
base_model: veichta/zipsplat
base_model_relation: finetune
tags:
  - applied-intuition
  - 3d-reconstruction
  - gaussian-splatting
  - novel-view-synthesis
  - streaming
  - feed-forward
  - arxiv:2610.08978
paper: 2610.08978
---

<p align="center">
  <img src="https://huggingface.co/AppliedIntuitionResearch/S2Tok/resolve/main/assets/icon.svg" width="120"/>
</p>

# S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens

<p align="center">
  <a href="https://arxiv.org/abs/2610.08978"><img src="https://img.shields.io/badge/arXiv-2610.08978-b31b1b.svg" alt="arXiv"></a>
  <a href="https://s2tok.github.io/"><img src="https://img.shields.io/badge/Project-Website-blue" alt="Project Page"></a>
  <a href="https://github.com/Applied-Intuition-Open-Source/S2Tok"><img src="https://img.shields.io/badge/GitHub-Code-black?logo=github" alt="Code"></a>
  <a href="https://creativecommons.org/licenses/by-nc/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey" alt="License"></a>
</p>

S2Tok reconstructs a scene as 3D Gaussians from an uncalibrated image stream, one frame at a time.
It keeps a persistent, size-adaptive set of spatial tokens as its scene state: each new frame
updates the existing tokens, and a learned admission module adds new tokens only for content that
is not yet represented. A hierarchical decoder turns the tokens into non-pixel-aligned Gaussians,
so the scene can be rendered from new views at any point of the stream without keeping past frames.

> ⚠️ **Research release.** This model is released for research purposes under the terms below. It is not a production system and does not include any Applied Intuition proprietary data, product code, or production checkpoints.

## Model Details

| | |
|---|---|
| **Developed by** | Applied Intuition — AI Research |
| **Model type** | Streaming feed-forward transformer (DA3-Giant ViT-G/14 backbone) with a persistent token memory, a learned admission module and a 3D Gaussian decoder |
| **Paper** | [S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens](https://arxiv.org/abs/2610.08978) |
| **Code** | https://github.com/Applied-Intuition-Open-Source/S2Tok |
| **Project page** | https://s2tok.github.io/ |
| **Initialized from** | [ZipSplat](https://huggingface.co/veichta/zipsplat) (CC BY-NC 4.0), itself initialized from [DA3-Giant](https://huggingface.co/depth-anything/DA3-GIANT) (CC BY-NC 4.0) |
| **Training data** | [DL3DV-10K](https://github.com/DL3DV-10K/Dataset) and [RealEstate10K](https://google.github.io/realestate10k/) |
| **License (weights)** | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — see [WEIGHTS_LICENSE.md](WEIGHTS_LICENSE.md) |
| **Contact** | GitHub Issues on the code repo |

## Checkpoints

| Checkpoint | Description | Size | Link |
|---|---|---|---|
| `s2tok_252.pt` | 1.72B params. 252x252 input, trained on 12–24 frame sequences from RealEstate10K and DL3DV-10K. | 6.87 GB | [s2tok_252.pt](s2tok_252.pt) |

The file holds `model` (inference weights) and `config` (architecture settings), and loads with
`torch.load(..., weights_only=True)`. SHA-256 checksums are in [SHA256SUMS](SHA256SUMS).

## Intended Use & Limitations

**Intended use:** research on streaming and feed-forward 3D reconstruction and novel-view
synthesis.

**Out of scope / limitations:**

- Frames are center-cropped and resized to 252x252. Gaussians are expressed in the camera frame of
  the first image, up to a global scale.
- The memory holds at most 4,000 tokens (128k Gaussians); once it is full, no new content is
  admitted.
- Trained on 12–24 frame sequences of static scenes and evaluated at 12 and 50 views; retention
  over much longer streams is not established.

**Commercial use is not permitted** under CC BY-NC 4.0. The weights are also subject to the terms
of their upstream model and datasets (see [License](#license)).

## How to Use

```python
from huggingface_hub import hf_hub_download
from s2tok import load_model

path = hf_hub_download("AppliedIntuitionResearch/S2Tok", "s2tok_252.pt")
model = load_model(path, device="cuda")

memory = model.init_memory()
for image in frames:  # [3, 252, 252] RGB tensors in [0, 1]
    out, memory = model.step(image.cuda(), memory)
out.gaussians.save_ply("scene.ply")  # Gaussians after the last frame (3DGS PLY)
```

Streaming demo with an interactive viewer:

```bash
python demo.py --model_path s2tok_252.pt --seq_path path/to/video.mp4 --output_dir outputs/my_scene
```

Full installation and usage instructions: see the [GitHub repository](https://github.com/Applied-Intuition-Open-Source/S2Tok).

## License

The weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/),
**non-commercial use only** ([WEIGHTS_LICENSE.md](WEIGHTS_LICENSE.md), full text in
[LICENSE](LICENSE)). The non-commercial restriction is also required by their upstream sources:

- fine-tuned from the released [ZipSplat](https://github.com/cvg/ZipSplat) checkpoint
  ([CC BY-NC 4.0](https://huggingface.co/veichta/zipsplat)), which is initialized from
  [DA3-Giant](https://huggingface.co/depth-anything/DA3-GIANT) (CC BY-NC 4.0); depth labels for
  training were produced with [DA3-Nested-Giant-Large](https://huggingface.co/depth-anything/DA3NESTED-GIANT-LARGE)
  (CC BY-NC 4.0);
- trained on [DL3DV-10K](https://github.com/DL3DV-10K/Dataset) (DL3DV-10K Terms of Use and
  CC BY-NC 4.0: non-commercial research and education only) and
  [RealEstate10K](https://google.github.io/realestate10k/) (Google; frames from YouTube videos).

Use of the weights must also comply with the terms of these models and datasets. No training
data is included in this repository.

## Citation

```bibtex
@article{li2026s2tok,
  title   = {S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens},
  author  = {Li, Fang and Yenphraphai, Jiraphon and Herau, Quentin and Meng, Depu and Hu, Yihan and Xu, Tianshuo and Ahuja, Narendra and Zhan, Wei},
  journal = {arXiv preprint arXiv:2610.08978},
  year    = {2026}
}
```

## Acknowledgments

This model builds on [ZipSplat](https://github.com/cvg/ZipSplat) (code Apache-2.0, weights
CC BY-NC 4.0), [Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3) and
[DINOv2](https://github.com/facebookresearch/dinov2) (Apache-2.0). It is trained on
[DL3DV-10K](https://github.com/DL3DV-10K/Dataset) and
[RealEstate10K](https://google.github.io/realestate10k/). We thank the authors for making their
work available.