Instructions to use pci-lab/worldflow3d with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use pci-lab/worldflow3d with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("pci-lab/worldflow3d", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Front3D-only card after Waymo split
Browse files
README.md
CHANGED
|
@@ -1,108 +1,50 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
| 3 |
tags:
|
| 4 |
-
- 3d-generation
|
| 5 |
-
- diffusion
|
| 6 |
- flow-matching
|
| 7 |
-
-
|
| 8 |
-
-
|
| 9 |
-
-
|
| 10 |
-
-
|
| 11 |
-
datasets:
|
| 12 |
-
- waymo-open-dataset
|
| 13 |
-
- 3D-FRONT
|
| 14 |
---
|
| 15 |
|
| 16 |
-
# WorldFlow3D
|
| 17 |
|
| 18 |
-
|
| 19 |
-
matching
|
| 20 |
-
|
| 21 |
-
|
| 22 |
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
|
|
|
| 26 |
|
| 27 |
-
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
|
| 30 |
-
*code* ships in the `worldflow3d` package, so the subfolders hold only weights + config:
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
front3d-
|
| 36 |
-
front3d-color/ source-flow + color refinement (unet in=4)
|
| 37 |
-
```
|
| 38 |
-
|
| 39 |
-
The four `unet/` folders are four distinct trained models, not copies.
|
| 40 |
|
| 41 |
## Usage
|
| 42 |
|
| 43 |
-
```bash
|
| 44 |
-
pip install worldflow3d # see the GitHub repo for source install
|
| 45 |
-
```
|
| 46 |
-
|
| 47 |
```python
|
| 48 |
-
import numpy as np
|
| 49 |
from worldflow3d import WorldFlow3DPipeline
|
| 50 |
-
from worldflow3d.pipeline.datatypes import LayoutContext
|
| 51 |
-
from worldflow3d.conditioning.waymo import load_waymo_map_json
|
| 52 |
-
from worldflow3d.recon import save_mesh
|
| 53 |
-
|
| 54 |
-
# Load the Waymo coarse -> refine cascade straight from this repo.
|
| 55 |
-
pipe = WorldFlow3DPipeline.from_hub(
|
| 56 |
-
"pci-lab/worldflow3d", stage="waymo-coarse", refinement_stages=["waymo-refine"],
|
| 57 |
-
device="cuda",
|
| 58 |
-
)
|
| 59 |
-
|
| 60 |
-
map_json = load_waymo_map_json("102062", map_dir="waymo_maps", data_root="waymo_scenes")
|
| 61 |
-
transform = np.load("waymo_scenes/coord_transforms/udf_voxel_0.4/segment-102062_training.npy",
|
| 62 |
-
allow_pickle=True).item()
|
| 63 |
-
ctx = LayoutContext("waymo", "102062", map_json, transform_data=transform)
|
| 64 |
-
|
| 65 |
-
result = pipe(layout_context=ctx, cfg_scale=1.5, sampling_steps=30,
|
| 66 |
-
refine=True, refine_sampling_steps=30, use_uniform_chunking=True,
|
| 67 |
-
smaller_map=True, fraction=0.12)
|
| 68 |
-
save_mesh(result.voxels.cpu(), result.voxel_size, "scene.ply")
|
| 69 |
-
```
|
| 70 |
-
|
| 71 |
-
Indoor (3D-FRONT), layout-conditioned coarse → color refinement:
|
| 72 |
|
| 73 |
-
```python
|
| 74 |
pipe = WorldFlow3DPipeline.from_hub(
|
| 75 |
-
"pci-lab/worldflow3d", stage="front3d-coarse",
|
| 76 |
-
device="cuda",
|
| 77 |
)
|
| 78 |
```
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
> does **not** work here: diffusers reads `model_index.json` only from a repo *root*,
|
| 83 |
-
> never a subfolder — so a subfolder/cascade repo is loaded via the package.
|
| 84 |
-
|
| 85 |
-
## Model details
|
| 86 |
-
|
| 87 |
-
| | |
|
| 88 |
-
|---|---|
|
| 89 |
-
| Architecture | 3D UNet, velocity-prediction (flow-matching) Euler sampler |
|
| 90 |
-
| Representation | Truncated UDF voxels (direct-diffusion; no VAE). Z is the vertical axis |
|
| 91 |
-
| Conditioning | Waymo vector maps (lanes, road edges/lines, crosswalks); 3D-FRONT layouts (walls/floors/doors/windows/furniture); optional tags (time/weather/location) |
|
| 92 |
-
| Generation | Chunked, coarse→refine cascade; sequential or simultaneous (feather-averaged) |
|
| 93 |
-
| Frameworks | `torch>=2.2`, `diffusers>=0.29` (tested on torch 2.2/diffusers 0.29.2 and torch 2.5/diffusers 0.38) |
|
| 94 |
-
|
| 95 |
-
## Training data & licensing
|
| 96 |
-
|
| 97 |
-
The weights are **derived from**:
|
| 98 |
-
- **Waymo Open Dataset** — <https://waymo.com/open/> — non-commercial research license;
|
| 99 |
-
redistribution of raw data is restricted. The `waymo-*` weights are for research use
|
| 100 |
-
consistent with the WOD terms.
|
| 101 |
-
- **3D-FRONT** — Alibaba; research license.
|
| 102 |
-
|
| 103 |
-
Code and weights are Apache-2.0, but **your use must also comply with the upstream
|
| 104 |
-
dataset licenses**. No raw dataset content is redistributed in this repo.
|
| 105 |
-
|
| 106 |
-
## Citation
|
| 107 |
-
|
| 108 |
-
If you use WorldFlow3D, please cite the project (see the GitHub repository).
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: 3d-front-license-nc
|
| 4 |
+
license_link: https://3dfront.github.io/
|
| 5 |
+
library_name: worldflow3d
|
| 6 |
tags:
|
| 7 |
+
- 3d-scene-generation
|
|
|
|
| 8 |
- flow-matching
|
| 9 |
+
- diffusers
|
| 10 |
+
- indoor
|
| 11 |
+
- front3d
|
| 12 |
+
- non-commercial
|
|
|
|
|
|
|
|
|
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# WorldFlow3D — Front3D (indoor) models
|
| 16 |
|
| 17 |
+
Layout-conditioned 3D indoor-scene generation (3D-FRONT), as a coarse → color
|
| 18 |
+
refinement flow-matching cascade. Use with the
|
| 19 |
+
[`worldflow3d`](https://github.com/princeton-computational-imaging/WorldFlow3D)
|
| 20 |
+
package (`pip install worldflow3d`).
|
| 21 |
|
| 22 |
+
> **License.** These models are trained on the **3D-FRONT** dataset and are
|
| 23 |
+
> released for **non-commercial / research use** under the 3D-FRONT dataset
|
| 24 |
+
> terms. The `worldflow3d` *code* is Apache-2.0, but that license does not grant
|
| 25 |
+
> rights to these 3D-FRONT-derived weights. See the 3D-FRONT dataset terms.
|
| 26 |
|
| 27 |
+
The **Waymo** (outdoor) models are trained on the Waymo Open Dataset and live in
|
| 28 |
+
a separate repo, [`pci-lab/worldflow3d-waymo`](https://huggingface.co/pci-lab/worldflow3d-waymo),
|
| 29 |
+
under the Waymo Dataset License (non-commercial).
|
| 30 |
|
| 31 |
+
## Cascade stages
|
|
|
|
| 32 |
|
| 33 |
+
| Subfolder | Role |
|
| 34 |
+
|-----------|------|
|
| 35 |
+
| `front3d-coarse` | coarse layout-conditioned generation (direct-diffusion) |
|
| 36 |
+
| `front3d-color` | source-flow + color refinement (UNet in=4, color mesh sidecar) |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
## Usage
|
| 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
```python
|
|
|
|
| 41 |
from worldflow3d import WorldFlow3DPipeline
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
|
|
|
| 43 |
pipe = WorldFlow3DPipeline.from_hub(
|
| 44 |
+
"pci-lab/worldflow3d", stage="front3d-coarse",
|
| 45 |
+
refinement_stages=["front3d-color"], device="cuda",
|
| 46 |
)
|
| 47 |
```
|
| 48 |
|
| 49 |
+
See the [GitHub repo](https://github.com/princeton-computational-imaging/WorldFlow3D)
|
| 50 |
+
for full docs and the `generate_indoor` CLI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|