Hugging Face's logo

appautomaton
/
sam-3d-objects-mlx-8bit

Image-to-3D
MLX
Safetensors
SAM 3D Objects
apple-silicon
sam-3d
3d-reconstruction
gaussian-splatting
mesh
glb
8-bit precision
affine-quantization
mixed-precision
Model card Files
xet
 Community
sam-3d-objects-mlx-8bit / README.md
tamarher's picture
tamarher
Upload SAM3D MLX 8-bit affine runtime bundle
0a5844c verified
preview code
|
Raw
History Blame Contribute Delete
10.4 kB
metadata 
license: other
license_name: sam-license
license_link: https://github.com/facebookresearch/sam-3d-objects/blob/main/LICENSE
library_name: mlx
pipeline_tag: image-to-3d
base_model:
  - appautomaton/sam-3d-objects-mlx
base_model_relation: quantized
tags:
  - mlx
  - apple-silicon
  - safetensors
  - sam-3d
  - sam-3d-objects
  - image-to-3d
  - 3d-reconstruction
  - gaussian-splatting
  - mesh
  - glb
  - 8-bit
  - affine-quantization
  - mixed-precision

SAM 3D Objects MLX 8-bit Affine for mlx-spatial

App Automaton project GitHub — appautomaton/mlx-spatial mlx-spatial documentation mlx-spatial on PyPI

SAM 3D Objects MLX full-precision variant SAM 3D Objects MLX 8-bit variant

A self-contained SAM 3D Objects inference bundle with selective 8-bit affine weights, built for direct execution by mlx-spatial on Apple Silicon. Large transformer matrices run through MLX's packed quantized matrix multiplication, while convolutional and accuracy-sensitive boundary tensors remain in their original precision.

This is an unofficial quantized derivative. It is not a Meta or MoGe release.

Variants

Variant Precision Checkpoint size Model
MLX converted Source FP16/FP32 13.705 GB sam-3d-objects-mlx
MLX 8-bit Selective affine INT8 with retained FP16/FP32 4.625 GB This model

The 8-bit checkpoint payload is 66.3% smaller. Both variants expose the same 3,783 logical tensors with identical names, shapes, and declared source dtypes. The full-precision repository is not required to run this variant.

Compatibility

This format requires an mlx-spatial build that includes SAM3D affine checkpoint support. Until that support is available in a tagged PyPI release, install the current project revision:

pip install \
  "mlx-spatial @ git+https://github.com/appautomaton/mlx-spatial.git@main"

The runtime targets Apple Silicon and MLX 0.32.x. It does not use Torch, CUDA, or a dequantized full-precision checkpoint.

Use

Download the complete bundle:

hf download appautomaton/sam-3d-objects-mlx-8bit \
  --local-dir weights/sam-3d-objects-mlx-8bit

Validate and inspect it without loading all weights:

mlx-spatial-sam3d validate weights/sam-3d-objects-mlx-8bit
mlx-spatial-sam3d inspect weights/sam-3d-objects-mlx-8bit

Generate a Gaussian Splat PLY:

mlx-spatial-sam3d reconstruct \
  weights/sam-3d-objects-mlx-8bit \
  inputs/sam3d/object-rmbg.png \
  --mask inputs/sam3d/object-mask.png \
  --moge-root weights/sam-3d-objects-mlx-8bit/moge \
  --output outputs/sam3d/object-8bit/gaussians.ply \
  --trace-output outputs/sam3d/object-8bit/trace.json \
  --memory-profile balanced

Add --glb-output outputs/sam3d/object-8bit/object.glb to decode and export a mesh-backed GLB in the same run.

SAM3D requires a binary object mask aligned with the input image. A clean foreground extraction is strongly recommended: transparent background pixels should have alpha zero, and the supplied mask should cover the intended object without including the surrounding scene.

Bundle

The repository preserves the directory and checkpoint names used by the full-precision MLX bundle:

Checkpoint Logical tensors INT8 matrices Bytes
checkpoints/ss_generator.safetensors 1,741 556 2,197,959,476
checkpoints/ss_decoder.safetensors 74 0 147,592,136
checkpoints/slat_generator.safetensors 1,225 384 1,578,129,504
checkpoints/slat_decoder_gs.safetensors 101 48 97,148,339
checkpoints/slat_decoder_gs_4.safetensors 101 48 95,942,210
checkpoints/slat_decoder_mesh.safetensors 120 48 119,546,584
moge/model.safetensors 421 96 388,551,606
Total 3,783 1,180 4,624,869,855

The remaining YAML configuration, conversion metadata, audit, and license files are small and are included alongside the checkpoints.

Quantization

The quantization scheme is affine 8-bit with group size 64. Packed weights are stored as uint32 with FP32 scales and biases and are executed directly with mx.quantized_matmul.

The following block-internal two-dimensional weights are quantized:

  • DINO condition-encoder attention and MLP matrices;
  • the point-condition transformer block;
  • sparse-structure and structured-latent generator transformer blocks;
  • Gaussian and mesh decoder transformer torso blocks;
  • the bundled MoGe ViT backbone attention and MLP matrices.

The following tensors remain in their original precision:

  • all dense, sparse, and transposed convolutions;
  • the complete sparse-structure decoder;
  • patch embeddings, input mappings, condition projections, and latent mappings;
  • normalization parameters, biases, learned tokens, and positional tensors;
  • final Gaussian, mesh, geometry, pose, and MoGe output heads;
  • matrices whose input dimension is incompatible with group size 64.

Physical packed arrays use internal qweight, scale, and bias suffixes. The runtime reconstructs the original logical tensor names from metadata, so model code and configuration continue to use the same checkpoint contract. Format details are embedded under mlx_spatial.sam3d.quantization in each quantized safetensors file.

Reproducing the Bundle

Starting from the full-precision MLX conversion:

mlx-spatial-sam3d-quantize \
  weights/sam-3d-objects-mlx \
  weights/sam-3d-objects-mlx-8bit \
  --bits 8 \
  --group-size 64

The quantized output is a complete runtime root. Do not place copies of the full-precision checkpoints in the 8-bit repository.

Verification

  • Logical checkpoint inspection matched all 3,783 source tensor names, shapes, and dtypes exactly.
  • The quantization inventory contains 1,180 packed matrices across six quantized checkpoints; the convolutional sparse-structure decoder remains unchanged.
  • A sampled static weight audit measured median 8-bit reconstruction SQNR of approximately 45 dB across checkpoint groups, with a worst observed matrix at 41 dB.
  • The SAM3D test suite passed 163 tests, with one test deselected.
  • A background-removed 900 × 900 input with an aligned mask completed all nine inference stages with no blocker and produced 495,008 finite Gaussians.
  • The generated geometry passed the runtime's nominal axis-range check.

The end-to-end run establishes runtime compatibility and artifact health. It is not a formal claim of visual equivalence to the full-precision model, and the observed run is not presented as a general performance benchmark.

Limitations

  • Quantization can change sparse occupancy, geometry, appearance, and other generation details relative to the full-precision variant.
  • A useful object-aligned mask is required. Background leakage, broad masks, soft edges, reflections, and thin structures can reduce reconstruction quality.
  • Single-view reconstruction cannot determine unseen geometry with certainty.
  • The packed checkpoint format requires mlx-spatial; generic safetensors loaders will see the physical packed arrays rather than the logical weights.
  • This bundle supports inference, not training or fine-tuning.
  • Standard PLY viewers may not render Gaussian Splat fields correctly. Use a 3DGS-aware viewer for the Gaussian artifact, or request GLB export.

License and Attribution

This bundle is derived from Meta's SAM 3D Objects release and is distributed under the SAM License. Read the bundled LICENSE and the upstream SAM License before use. Redistribution of SAM Materials and derivative works remains subject to that agreement.

The bundle also contains a converted and quantized Ruicheng/moge-vitl checkpoint used for pointmap estimation. That checkpoint is published under Apache 2.0. Users are responsible for complying with both sets of terms.

This repository is not affiliated with or endorsed by Meta or the MoGe authors. Publications using these weights should acknowledge the original SAM 3D Objects and MoGe work.

Links