ostjul commited on
Commit
e08927a
·
verified ·
1 Parent(s): 5f1f03a

Front3D-only card after Waymo split

Browse files
Files changed (1) hide show
  1. README.md +30 -88
README.md CHANGED
@@ -1,108 +1,50 @@
1
  ---
2
- license: apache-2.0
 
 
 
3
  tags:
4
- - 3d-generation
5
- - diffusion
6
  - flow-matching
7
- - voxel
8
- - scene-generation
9
- - autonomous-driving
10
- - indoor-scenes
11
- datasets:
12
- - waymo-open-dataset
13
- - 3D-FRONT
14
  ---
15
 
16
- # WorldFlow3D
17
 
18
- Hierarchical, map/layout-conditioned **3D scene generation** via chunked flow
19
- matching. This repo holds the released model **cascade** (coarse refinement) for
20
- both outdoor (Waymo, map-conditioned) and indoor (3D-FRONT, layout-conditioned)
21
- scenes, packaged in the [diffusers](https://github.com/huggingface/diffusers) style.
22
 
23
- - **Code:** <https://github.com/princeton-computational-imaging/WorldFlow3D>
24
- - **License:** Apache-2.0 (code & weights). **Training data** (Waymo Open Dataset,
25
- 3D-FRONT) carries its own terms see *Training data & licensing* below.
 
26
 
27
- ## Repository layout (one repo = the whole cascade)
 
 
28
 
29
- Each stage is a self-contained diffusers layout in its own **subfolder**. The model
30
- *code* ships in the `worldflow3d` package, so the subfolders hold only weights + config:
31
 
32
- ```
33
- waymo-coarse/ coarse outdoor geometry (map-conditioned), 0.4 m voxels
34
- waymo-refine/ source-flow refinement -> 0.2 m voxels
35
- front3d-coarse/ coarse indoor geometry
36
- front3d-color/ source-flow + color refinement (unet in=4)
37
- ```
38
-
39
- The four `unet/` folders are four distinct trained models, not copies.
40
 
41
  ## Usage
42
 
43
- ```bash
44
- pip install worldflow3d # see the GitHub repo for source install
45
- ```
46
-
47
  ```python
48
- import numpy as np
49
  from worldflow3d import WorldFlow3DPipeline
50
- from worldflow3d.pipeline.datatypes import LayoutContext
51
- from worldflow3d.conditioning.waymo import load_waymo_map_json
52
- from worldflow3d.recon import save_mesh
53
-
54
- # Load the Waymo coarse -> refine cascade straight from this repo.
55
- pipe = WorldFlow3DPipeline.from_hub(
56
- "pci-lab/worldflow3d", stage="waymo-coarse", refinement_stages=["waymo-refine"],
57
- device="cuda",
58
- )
59
-
60
- map_json = load_waymo_map_json("102062", map_dir="waymo_maps", data_root="waymo_scenes")
61
- transform = np.load("waymo_scenes/coord_transforms/udf_voxel_0.4/segment-102062_training.npy",
62
- allow_pickle=True).item()
63
- ctx = LayoutContext("waymo", "102062", map_json, transform_data=transform)
64
-
65
- result = pipe(layout_context=ctx, cfg_scale=1.5, sampling_steps=30,
66
- refine=True, refine_sampling_steps=30, use_uniform_chunking=True,
67
- smaller_map=True, fraction=0.12)
68
- save_mesh(result.voxels.cpu(), result.voxel_size, "scene.ply")
69
- ```
70
-
71
- Indoor (3D-FRONT), layout-conditioned coarse → color refinement:
72
 
73
- ```python
74
  pipe = WorldFlow3DPipeline.from_hub(
75
- "pci-lab/worldflow3d", stage="front3d-coarse", refinement_stages=["front3d-color"],
76
- device="cuda",
77
  )
78
  ```
79
 
80
- > **Loading note.** Use `WorldFlow3DPipeline.from_hub(...)` (this package). The bare
81
- > `diffusers.DiffusionPipeline.from_pretrained(repo, trust_remote_code=True)` one-liner
82
- > does **not** work here: diffusers reads `model_index.json` only from a repo *root*,
83
- > never a subfolder — so a subfolder/cascade repo is loaded via the package.
84
-
85
- ## Model details
86
-
87
- | | |
88
- |---|---|
89
- | Architecture | 3D UNet, velocity-prediction (flow-matching) Euler sampler |
90
- | Representation | Truncated UDF voxels (direct-diffusion; no VAE). Z is the vertical axis |
91
- | Conditioning | Waymo vector maps (lanes, road edges/lines, crosswalks); 3D-FRONT layouts (walls/floors/doors/windows/furniture); optional tags (time/weather/location) |
92
- | Generation | Chunked, coarse→refine cascade; sequential or simultaneous (feather-averaged) |
93
- | Frameworks | `torch>=2.2`, `diffusers>=0.29` (tested on torch 2.2/diffusers 0.29.2 and torch 2.5/diffusers 0.38) |
94
-
95
- ## Training data & licensing
96
-
97
- The weights are **derived from**:
98
- - **Waymo Open Dataset** — <https://waymo.com/open/> — non-commercial research license;
99
- redistribution of raw data is restricted. The `waymo-*` weights are for research use
100
- consistent with the WOD terms.
101
- - **3D-FRONT** — Alibaba; research license.
102
-
103
- Code and weights are Apache-2.0, but **your use must also comply with the upstream
104
- dataset licenses**. No raw dataset content is redistributed in this repo.
105
-
106
- ## Citation
107
-
108
- If you use WorldFlow3D, please cite the project (see the GitHub repository).
 
1
  ---
2
+ license: other
3
+ license_name: 3d-front-license-nc
4
+ license_link: https://3dfront.github.io/
5
+ library_name: worldflow3d
6
  tags:
7
+ - 3d-scene-generation
 
8
  - flow-matching
9
+ - diffusers
10
+ - indoor
11
+ - front3d
12
+ - non-commercial
 
 
 
13
  ---
14
 
15
+ # WorldFlow3D — Front3D (indoor) models
16
 
17
+ Layout-conditioned 3D indoor-scene generation (3D-FRONT), as a coarse → color
18
+ refinement flow-matching cascade. Use with the
19
+ [`worldflow3d`](https://github.com/princeton-computational-imaging/WorldFlow3D)
20
+ package (`pip install worldflow3d`).
21
 
22
+ > **License.** These models are trained on the **3D-FRONT** dataset and are
23
+ > released for **non-commercial / research use** under the 3D-FRONT dataset
24
+ > terms. The `worldflow3d` *code* is Apache-2.0, but that license does not grant
25
+ > rights to these 3D-FRONT-derived weights. See the 3D-FRONT dataset terms.
26
 
27
+ The **Waymo** (outdoor) models are trained on the Waymo Open Dataset and live in
28
+ a separate repo, [`pci-lab/worldflow3d-waymo`](https://huggingface.co/pci-lab/worldflow3d-waymo),
29
+ under the Waymo Dataset License (non-commercial).
30
 
31
+ ## Cascade stages
 
32
 
33
+ | Subfolder | Role |
34
+ |-----------|------|
35
+ | `front3d-coarse` | coarse layout-conditioned generation (direct-diffusion) |
36
+ | `front3d-color` | source-flow + color refinement (UNet in=4, color mesh sidecar) |
 
 
 
 
37
 
38
  ## Usage
39
 
 
 
 
 
40
  ```python
 
41
  from worldflow3d import WorldFlow3DPipeline
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
 
43
  pipe = WorldFlow3DPipeline.from_hub(
44
+ "pci-lab/worldflow3d", stage="front3d-coarse",
45
+ refinement_stages=["front3d-color"], device="cuda",
46
  )
47
  ```
48
 
49
+ See the [GitHub repo](https://github.com/princeton-computational-imaging/WorldFlow3D)
50
+ for full docs and the `generate_indoor` CLI.