suvadityamuk/idol-diffusers-3d

IDOL for diffusers-3d

speridlabs/idol converted into a diffusers-3d pipeline: ordinary Diffusers and Transformers component folders that the installed package loads. Nothing here requires remote code.

Install

pip install git+https://github.com/suvadityamuk/diffusers-3d.git
pip install "git+https://github.com/suvadityamuk/diffusers-3d.git#subdirectory=packages/diffusers-3d"

diffusers-3d runs every network in plain PyTorch on CPU or GPU. Rendering Gaussian splats needs the optional gsplat backend; the PBR-textured mesh for TRELLIS.2 needs CuMesh (see the package docs).

Usage

import torch
from diffusers_3d import AutoPipelineForImageTo3D, ImageCondition

pipeline = AutoPipelineForImageTo3D.from_pretrained("suvadityamuk/idol-diffusers-3d").to("cuda")
splat = pipeline(ImageCondition(image=rgba), mode="real", generator=torch.Generator("cuda").manual_seed(42)).objects[0]
camera = splat.metadata["camera"]  # predicted yaw, pitch, radius, and FOV of the input view
splat.save("object.ply")

DINOv3 and the FLUX.2 VAE are not included. Both are gated third-party weights, so from_pretrained loads image_encoder from facebook/dinov3-vith16plus-pretrain-lvd1689m (DINOv3 License) and vae from the vae/ folder of black-forest-labs/FLUX.2-dev (FLUX non-commercial license), each at a pinned revision. Accept both licenses and log in (hf auth login) first; without access, loading raises an error that says so. Inputs without usable alpha are cut out with BiRefNet using ZhengPeng7/BiRefNet weights (MIT), as upstream does (pipeline.disable_background_removal() skips it for inputs that already carry alpha).

mode="synthetic" (the default, for renders) crops 1.2 times the object; mode="real" (photos) crops 1.7 times. Defaults match upstream: 50 steps, shift 3.0, 262144 Gaussians, guidance_scale=4.0 (upstream's cfg_strength=3.0 in the Diffusers convention). Without dtype, each component loads in upstream's precision; pass dtype=torch.float32 on CPU.

Components

Folder	Class
decoder	diffusers_3d.families.triposplat.decoder.TripoSplatGaussianDecoder
image_encoder	transformers.DINOv3ViTModel (not included; loads from facebook/dinov3-vith16plus-pretrain-lvd1689m)
scheduler	diffusers.FlowMatchEulerDiscreteScheduler
transformer	diffusers_3d.families.triposplat.models.TripoSplatTransformerModel
vae	diffusers_3d.families.triposplat.models.TripoSplatImageVAE (not included; loads from black-forest-labs/FLUX.2-dev)

Provenance

Converted with diffusers-3d-convert-idol from diffusers-3d 0.1.0.dev0; the package's family README lists the pinned upstream revisions and how the conversion maps each tensor. Parity tests against the pinned upstream code are part of the package test suite.

License and attribution

IDOL weights and code: Apache-2.0, Copyright (c) Speridlabs. IDOL keeps the architecture of VAST-AI-Research TripoSplat (MIT, LICENSE-MIT). The DINOv3 and FLUX.2 VAE encoders are not redistributed; they load from their original repositories under their own licenses. See LICENSE-APACHE-2.0 and NOTICE. Not affiliated with or endorsed by Speridlabs, VAST, Meta, or Black Forest Labs.