suvadityamuk/sam-3d-objects-diffusers-3d

SAM 3D Objects for diffusers-3d

facebook/sam-3d-objects converted into a diffusers-3d pipeline: ordinary Diffusers and Transformers component folders plus the object3d_model_index.json sidecar that the package's auto-loader validates before downloading anything. Nothing here requires remote code.

Install

pip install git+https://github.com/suvadityamuk/diffusers-3d.git
pip install "git+https://github.com/suvadityamuk/diffusers-3d.git#subdirectory=packages/diffusers-3d"

diffusers-3d runs every network in plain PyTorch on CPU or GPU. Rendering Gaussian splats needs the optional gsplat backend; the PBR-textured mesh for TRELLIS.2 needs CuMesh (see the package docs).

Usage

import torch
from diffusers_3d import AutoPipelineForImageTo3D, ImageCondition

pipeline = AutoPipelineForImageTo3D.from_pretrained("suvadityamuk/sam-3d-objects-diffusers-3d").to("cuda")
output = pipeline(ImageCondition(image=rgb, mask=mask), formats=("gaussian", "mesh"))
gaussians, mesh = output.objects

rgb is a (3, H, W) tensor in [0, 1] and mask a (1, H, W) tensor selecting the object. The object comes back placed in the input camera's frame. Pass pointmap= (camera-space points per pixel) to use depth; without it the pointmap tokens are zero, as in the model's training dropout.

Components

Folder	Class
gaussian_decoder	diffusers_3d.families.trellis.decoders.TrellisSLatGaussianDecoder
image_encoder	transformers.Dinov2WithRegistersModel
mesh_decoder	diffusers_3d.families.trellis.decoders.TrellisSLatMeshDecoder
slat_conditioner	diffusers_3d.families.sam3d_objects.conditioner.Sam3DConditionEmbedder
slat_flow_model	diffusers_3d.families.trellis.models.TrellisSLatFlowModel
slat_scheduler	diffusers_3d.families.trellis.scheduler.TrellisFlowEulerScheduler
sparse_structure_conditioner	diffusers_3d.families.sam3d_objects.conditioner.Sam3DConditionEmbedder
sparse_structure_decoder	diffusers_3d.families.trellis.decoders.TrellisSparseStructureDecoder
sparse_structure_flow_model	diffusers_3d.families.sam3d_objects.models.Sam3DSparseStructureFlowModel
sparse_structure_scheduler	diffusers_3d.families.trellis.scheduler.TrellisFlowEulerScheduler

Provenance

Converted with diffusers-3d-convert-sam3d from diffusers-3d 0.1.0.dev0; the package's family README lists the pinned upstream revisions and how the conversion maps each tensor. Parity tests against the pinned upstream code are part of the package test suite.

License and attribution

SAM 3D Objects weights and code: SAM License, Copyright (c) Meta Platforms, Inc. and affiliates. This repository redistributes converted SAM Materials under that Agreement, a copy of which is in LICENSE-SAM; by using these files you agree to it. It is not an open-source license: it forbids military, warfare, nuclear, espionage, and weapons uses and use by parties subject to trade controls, and publications using the model must acknowledge it. Not affiliated with or endorsed by Meta.