File size: 3,474 Bytes
332cfec 4210613 332cfec 4210613 332cfec | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
license: other
license_name: sam-license
license_link: https://huggingface.co/suvadityamuk/sam-3d-objects-diffusers-3d/blob/main/LICENSE-SAM
base_model: facebook/sam-3d-objects
pipeline_tag: image-to-3d
tags:
- diffusers-3d
- sam3d-objects
- 3d
- image-to-3d
---
# SAM 3D Objects for diffusers-3d
[facebook/sam-3d-objects](https://huggingface.co/facebook/sam-3d-objects) converted into a [diffusers-3d](https://github.com/suvadityamuk/diffusers-3d)
pipeline: ordinary Diffusers and Transformers component folders plus the `object3d_model_index.json` sidecar that
the package's auto-loader validates before downloading anything. Nothing here requires remote code.
## Install
```bash
pip install git+https://github.com/suvadityamuk/diffusers-3d.git
pip install "git+https://github.com/suvadityamuk/diffusers-3d.git#subdirectory=packages/diffusers-3d"
```
`diffusers-3d` runs every network in plain PyTorch on CPU or GPU. Rendering Gaussian splats needs the optional
`gsplat` backend; the PBR-textured mesh for TRELLIS.2 needs CuMesh (see the package docs).
## Usage
```python
import torch
from diffusers_3d import AutoPipelineForImageTo3D, ImageCondition
pipeline = AutoPipelineForImageTo3D.from_pretrained("suvadityamuk/sam-3d-objects-diffusers-3d").to("cuda")
output = pipeline(ImageCondition(image=rgb, mask=mask), formats=("gaussian", "mesh"))
gaussians, mesh = output.objects
```
`rgb` is a `(3, H, W)` tensor in `[0, 1]` and `mask` a `(1, H, W)` tensor selecting the object. The object comes back placed in the input camera's frame. Pass `pointmap=` (camera-space points per pixel) to use depth; without it the pointmap tokens are zero, as in the model's training dropout.
## Components
| Folder | Class |
|---|---|
| `gaussian_decoder` | `diffusers_3d.families.trellis.decoders.TrellisSLatGaussianDecoder` |
| `image_encoder` | `transformers.Dinov2WithRegistersModel` |
| `mesh_decoder` | `diffusers_3d.families.trellis.decoders.TrellisSLatMeshDecoder` |
| `slat_conditioner` | `diffusers_3d.families.sam3d_objects.conditioner.Sam3DConditionEmbedder` |
| `slat_flow_model` | `diffusers_3d.families.trellis.models.TrellisSLatFlowModel` |
| `slat_scheduler` | `diffusers_3d.families.trellis.scheduler.TrellisFlowEulerScheduler` |
| `sparse_structure_conditioner` | `diffusers_3d.families.sam3d_objects.conditioner.Sam3DConditionEmbedder` |
| `sparse_structure_decoder` | `diffusers_3d.families.trellis.decoders.TrellisSparseStructureDecoder` |
| `sparse_structure_flow_model` | `diffusers_3d.families.sam3d_objects.models.Sam3DSparseStructureFlowModel` |
| `sparse_structure_scheduler` | `diffusers_3d.families.trellis.scheduler.TrellisFlowEulerScheduler` |
## Provenance
Converted with `diffusers-3d-convert-sam3d` from `diffusers-3d 0.1.0.dev0`; the package's family README lists the pinned upstream
revisions and how the conversion maps each tensor. Parity tests against the pinned upstream code are part of the
package test suite.
## License and attribution
SAM 3D Objects weights and code: SAM License, Copyright (c) Meta Platforms, Inc. and affiliates.
This repository redistributes converted SAM Materials under that Agreement, a copy of which is in `LICENSE-SAM`;
by using these files you agree to it. It is not an open-source license: it forbids military, warfare, nuclear,
espionage, and weapons uses and use by parties subject to trade controls, and publications using the model must
acknowledge it. Not affiliated with or endorsed by Meta.
|