| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: diffusers |
| pipeline_tag: image-to-3d |
| base_model: |
| - TencentARC/Pixal3D |
| tags: |
| - pixal3d |
| - modular-diffusers |
| - image-to-3d |
| --- |
| |
| # Pixal3D Modular Diffusers |
|
|
| <video controls loop muted playsinline src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/Pixal3D/Pixal3D_jester_seed42_bf16.mp4"></video> |
|
|
| *Generated with the code below (seed `42`, 12 steps, 1024 resolution).* |
|
|
| Modular Diffusers blocks for [TencentARC/Pixal3D](https://huggingface.co/TencentARC/Pixal3D): generate 3D geometry and materials from a single image. The model code and weights are in [OzzyGT/Pixal3D](https://huggingface.co/OzzyGT/Pixal3D). |
|
|
| Requires PyTorch, `diffusers==0.41.0`, `transformers==5.20.2`, `accelerate>=1.15`, `safetensors>=0.8`, `numpy` and `pillow`. |
|
|
| ## Licensing |
|
|
| The blocks are [Apache-2.0](LICENSE), with the camera projection and latent statistics adapted from [TencentARC/Pixal3D](https://github.com/TencentARC/Pixal3D) under [MIT](LICENSE-PIXAL3D). See [NOTICE](NOTICE). This repository contains no model weights. |
|
|
| ## Usage |
|
|
| The example uses `trust_remote_code=True`, which loads these blocks, and `trust_components_code=True`, which allows the model repository's custom classes. |
|
|
| > **Note:** both flags run Python code downloaded from the Hub: `trust_remote_code=True` runs the code of the blocks repository, and `trust_components_code=True` runs the code shipped in the `components_repo`. Only use them with repositories you trust, and review their code before passing different ones. |
| |
| ```python |
| import math |
| |
| import torch |
| from diffusers import ModularPipelineBlocks |
| from diffusers.utils import load_image |
|
|
|
|
| blocks = ModularPipelineBlocks.from_pretrained( |
| "OzzyGT/Pixal3D_Modular", |
| trust_remote_code=True, |
| components_repo="OzzyGT/Pixal3D", |
| trust_components_code=True, |
| ) |
| pipe = blocks.init_pipeline() |
| flow_names = ( |
| "sparse_structure_flow_model", |
| "shape_slat_flow_model_512", |
| "shape_slat_flow_model_1024", |
| "tex_slat_flow_model_1024", |
| ) |
| dtypes = {spec.name: torch.float32 for spec in blocks.expected_components} |
| dtypes.update({name: torch.bfloat16 for name in (*flow_names, "image_encoder")}) |
| pipe.load_components(dtype=dtypes) |
| pipe.to("cuda") |
| |
| image = load_image("https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/Pixal3D/Pixal3D_jester_input.png") |
| result = pipe( |
| image=image, |
| camera_angle_x=math.radians(33.85), |
| distance=1.643, |
| mesh_scale=1.0, |
| resolution=1024, |
| num_inference_steps=12, |
| generator=torch.Generator().manual_seed(42), |
| output=["geometry", "materials", "actual_resolution"], |
| ) |
| geometry = result["geometry"] |
| materials = result["materials"] |
| ``` |
| |
| The input is a prepared image: the subject on black (RGBA is composited on black), with no automatic background removal or crop. `camera_angle_x` and `distance` must match that image; the sample's camera was estimated with [MoGe-2](https://huggingface.co/Ruicheng/moge-2-vitl). `resolution` can be 1024 or 1536. |
|
|
| The output is sparse voxels, not a mesh: `geometry` holds the flexible-dual-grid cells for mesh extraction, `materials` the per-cell base color, metallic, roughness and alpha, and `actual_resolution` the grid resolution used (a 1536 request can fall back toward 1024). Coordinates are normalized and Y-up. |
|
|
