| --- |
| license: openrail++ |
| base_model: stepfun-ai/Step1X-3D |
| pipeline_tag: image-to-3d |
| tags: |
| - diffusers-3d |
| - step1x3d |
| - 3d |
| - image-to-3d |
| - texturing |
| --- |
| |
| # Step1X-3D Texture for diffusers-3d |
|
|
| [stepfun-ai/Step1X-3D](https://huggingface.co/stepfun-ai/Step1X-3D) converted into a [diffusers-3d](https://github.com/suvadityamuk/diffusers-3d) |
| pipeline: ordinary Diffusers and Transformers component folders plus the `object3d_model_index.json` sidecar that |
| the package's auto-loader validates before downloading anything. Nothing here requires remote code. |
|
|
| ## Install |
|
|
| ```bash |
| pip install git+https://github.com/suvadityamuk/diffusers-3d.git |
| pip install "git+https://github.com/suvadityamuk/diffusers-3d.git#subdirectory=packages/diffusers-3d" |
| ``` |
|
|
| `diffusers-3d` runs every network in plain PyTorch on CPU or GPU. Rendering Gaussian splats needs the optional |
| `gsplat` backend; the PBR-textured mesh for TRELLIS.2 needs CuMesh (see the package docs). |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from diffusers_3d import AutoPipelineFor3D, AutoPipelineForImageTo3D, ImageCondition |
| |
| shape = AutoPipelineForImageTo3D.from_pretrained("suvadityamuk/Step1X-3D-Geometry-1300m-diffusers-3d").to("cuda") |
| mesh = shape(ImageCondition(image=rgba), generator=torch.Generator("cuda").manual_seed(0)).objects[0] |
| |
| texture = AutoPipelineFor3D.from_pretrained("suvadityamuk/Step1X-3D-Texture-diffusers-3d", dtype=torch.float16).to("cuda") |
| textured = texture(mesh, ImageCondition(image=rgba), generator=torch.Generator("cuda").manual_seed(0)).objects[0] |
| ``` |
|
|
| `mesh` is any `MeshAsset`, for example a shape pipeline's output, and `rgba` a `(4, H, W)` tensor in `[0, 1]` whose alpha masks the object. The result is a UV-textured `MeshAsset`; export it with `TrimeshBackend().export_mesh(textured.to("cpu"), "textured.glb")` (needs the `portable` extra). The pipeline generates six orthographic views with SDXL, Step1X-3D's multi-view adapter, and a T2I-Adapter fed position and normal renders, keeps them consistent through a shared latent texture, and bakes a base-colour texture in plain PyTorch. `output.previews` holds the six views. |
|
|
| ## Components |
|
|
| | Folder | Class | |
| |---|---| |
| | `adapter` | `diffusers.T2IAdapter` | |
| | `mv_adapter` | `diffusers_3d.families.step1x3d.texture_attention.Step1X3DMultiViewAdapter` | |
| | `scheduler` | `diffusers.DDPMScheduler` | |
| | `text_encoder` | `transformers.CLIPTextModel` | |
| | `text_encoder_2` | `transformers.CLIPTextModelWithProjection` | |
| | `tokenizer` | `transformers.CLIPTokenizer` | |
| | `tokenizer_2` | `transformers.CLIPTokenizer` | |
| | `unet` | `diffusers.UNet2DConditionModel` | |
| | `vae` | `diffusers.AutoencoderKL` | |
|
|
| ## Provenance |
|
|
| Converted with `diffusers-3d-convert-step1x3d-texture` from `diffusers-3d 0.1.0.dev0`; the package's family README lists the pinned upstream |
| revisions and how the conversion maps each tensor. Parity tests against the pinned upstream code are part of the |
| package test suite. |
|
|
| ## License and attribution |
|
|
| This repository combines three sets of weights, each under its own license: |
|
|
| - the SDXL base UNet, text encoders, and tokenizers from |
| [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0), under |
| the CreativeML Open RAIL++-M License in `LICENSE-SDXL.md`, including its use-based restrictions, which apply to |
| everyone who uses these files; |
| - the VAE from [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) (MIT); |
| - Step1X-3D's T2I-Adapter and multi-view adapter weights from `Step1X-3D-Texture` (Apache-2.0, Copyright (c) |
| StepFun; see `LICENSE-APACHE-2.0` and `NOTICE`). |
|
|
| The texture synchronisation follows SyncMVD (MIT, `LICENSE-MIT`). Not affiliated with or endorsed by StepFun or |
| Stability AI. |
|
|
