README.md

---
library_name: mlx
pipeline_tag: image-to-3d
license: other
license_name: sam-license
license_link: https://github.com/facebookresearch/sam-3d-objects/blob/main/LICENSE
tags:
  - mlx
  - apple-silicon
  - sam-3d
  - sam-3d-objects
  - image-to-3d
  - 3d-reconstruction
  - gaussian-splatting
  - mesh
  - glb
  - safetensors
base_model:
  - facebook/sam-3d-objects
---

# SAM 3D Objects MLX for mlx-spatial

Run SAM 3D Objects on Apple Silicon through `mlx-spatial`, using an MLX-ready safetensors bundle instead of local PyTorch checkpoint conversion.

This bundle is for users who want masked single-image object reconstruction on a Mac: download the model, provide an image plus object mask, and generate SAM3D Gaussian or mesh artifacts with `mlx-spatial-sam3d`. No CUDA is required.

## Quick Start: Masked Image to 3D Object

Install `mlx-spatial`:

```bash
pip install mlx-spatial
```

Download this model bundle:

```bash
hf download appautomaton/sam-3d-objects-mlx \
  --local-dir weights/sam-3d-objects-mlx
```

Validate the local layout:

```bash
mlx-spatial-sam3d validate weights/sam-3d-objects-mlx
mlx-spatial-sam3d inspect weights/sam-3d-objects-mlx
```

Generate a Gaussian-splat PLY:

```bash
mlx-spatial-sam3d reconstruct weights/sam-3d-objects-mlx image.png \
  --mask mask.png \
  --output outputs/sam3d/object/gaussians.ply \
  --trace-output outputs/sam3d/object/trace.json
```

Generate a GLB mesh as well:

```bash
mlx-spatial-sam3d reconstruct weights/sam-3d-objects-mlx image.png \
  --mask mask.png \
  --output outputs/sam3d/object/gaussians.ply \
  --glb-output outputs/sam3d/object/object.glb \
  --trace-output outputs/sam3d/object/trace.json
```

The trace records quality diagnostics such as sparse-structure occupancy, geometry range, opacity, selected mask, and output paths.

## What This Model Bundle Provides

This Hugging Face repository contains the converted SAM 3D Objects checkpoint bundle expected by `mlx-spatial`:

```text
checkpoints/pipeline.yaml
checkpoints/ss_generator.safetensors
checkpoints/slat_generator.safetensors
checkpoints/ss_decoder.safetensors
checkpoints/slat_decoder_gs.safetensors
checkpoints/slat_decoder_gs_4.safetensors
checkpoints/slat_decoder_mesh.safetensors
checkpoints/conversion_metadata/
conversion_manifest.json
weight-audit-source-vs-mlx.json
```

It also includes the converted MoGe ViT-L pointmap dependency used by the default SAM3D preprocessing path:

```text
moge/model.safetensors
moge/conversion_metadata/model.yaml
```

The bundled MoGe checkpoint lets the normal `mlx-spatial-sam3d reconstruct` command run from one model repository. Advanced users can still pass a different MoGe root or provide an external pointmap.

## Best For

- Apple Silicon MLX inference experiments.
- Masked single-image object reconstruction.
- SAM3D Gaussian Splat PLY generation with `mlx-spatial`.
- SAM3D mesh or GLB export workflows.
- Researchers and developers who need SAM 3D Objects weights in safetensors format.

## Current Limitations

- This is an unofficial converted derivative bundle, not an official Meta or MoGe release.
- The upstream `facebook/sam-3d-objects` Hugging Face repository is gated. Users should have access to the upstream model and accept the upstream terms before using this conversion.
- Reconstruction requires an input image and a useful binary object mask.
- Standard 3D Gaussian viewers may use different coordinate conventions than SAM 3D Objects' native output convention.
- This is not an int8, 4-bit, or otherwise quantized model.
- CUDA is not required and is not used by `mlx-spatial` SAM3D inference.

## Conversion Fidelity

The converted checkpoint bundle was audited against the original SAM 3D Objects checkpoint files.

| Role | Tensors | Missing | Extra | Shape mismatches | Nonzero numeric diffs | Max abs diff |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `ss_generator` | 1,741 | 0 | 0 | 0 | 0 | 0.0 |
| `slat_generator` | 1,225 | 0 | 0 | 0 | 0 | 0.0 |
| `ss_decoder` | 74 | 0 | 0 | 0 | 0 | 0.0 |
| `slat_decoder_gs` | 101 | 0 | 0 | 0 | 0 | 0.0 |
| `slat_decoder_mesh` | 120 | 0 | 0 | 0 | 0 | 0.0 |
| `slat_decoder_gs_4` | 101 | 0 | 0 | 0 | 0 | 0.0 |

Total compared SAM3D tensors: 3,362.

Some decoder tensors are stored as `float32` in this safetensors bundle even when the source checkpoint tensor was `float16`. This is lossless for value preservation. The numeric audit compares values after `float32` materialization and found zero difference.

See `weight-audit-source-vs-mlx.json` for the audit summary.

## Conversion Details

This bundle was produced from the original SAM 3D Objects checkpoint layout with:

```bash
mlx-spatial-sam3d convert weights/sam-3d-objects \
  --output-root weights/sam-3d-objects-mlx \
  --moge-root weights/moge-vitl \
  --moge-output-root weights/sam-3d-objects-mlx/moge \
  --max-archive-gb 16
```

The conversion rewrites checkpoint references in `pipeline.yaml` from PyTorch checkpoint files to `.safetensors` files. It does not quantize the model or change the architecture.

## Project Links

- Runtime package: `mlx-spatial`
- `mlx-spatial` PyPI package: https://pypi.org/project/mlx-spatial/
- `mlx-spatial` source: https://github.com/appautomaton/mlx-spatial
- This model repo: https://huggingface.co/appautomaton/sam-3d-objects-mlx

## Upstream Source and License

This bundle is based on Meta's SAM 3D Objects release and includes a converted MoGe dependency:

- Upstream SAM 3D Objects model: https://huggingface.co/facebook/sam-3d-objects
- Upstream SAM 3D Objects code: https://github.com/facebookresearch/sam-3d-objects
- SAM License: https://github.com/facebookresearch/sam-3d-objects/blob/main/LICENSE
- MoGe dependency: included as converted `moge/model.safetensors`

The original SAM 3D Objects checkpoints and code are licensed by Meta under the SAM License. The included MoGe dependency follows its own upstream license and terms.

This repository is not an official Meta or MoGe release. Users are responsible for complying with the upstream SAM 3D Objects and MoGe license, access, and use requirements.

If you use this conversion, cite the original SAM 3D Objects work and link to the upstream model and code.