appautomaton/sam-3d-objects-mlx-8bit

SAM 3D Objects MLX 8-bit Affine for mlx-spatial

A self-contained SAM 3D Objects inference bundle with selective 8-bit affine weights, built for direct execution by mlx-spatial on Apple Silicon. Large transformer matrices run through MLX's packed quantized matrix multiplication, while convolutional and accuracy-sensitive boundary tensors remain in their original precision.

This is an unofficial quantized derivative. It is not a Meta or MoGe release.

Variants

Variant	Precision	Checkpoint size	Model
MLX converted	Source FP16/FP32	13.705 GB	sam-3d-objects-mlx
MLX 8-bit	Selective affine INT8 with retained FP16/FP32	4.625 GB	This model

The 8-bit checkpoint payload is 66.3% smaller. Both variants expose the same 3,783 logical tensors with identical names, shapes, and declared source dtypes. The full-precision repository is not required to run this variant.

Compatibility

This format requires an mlx-spatial build that includes SAM3D affine checkpoint support. Until that support is available in a tagged PyPI release, install the current project revision:

pip install \
  "mlx-spatial @ git+https://github.com/appautomaton/mlx-spatial.git@main"

The runtime targets Apple Silicon and MLX 0.32.x. It does not use Torch, CUDA, or a dequantized full-precision checkpoint.

Use

Download the complete bundle:

hf download appautomaton/sam-3d-objects-mlx-8bit \
  --local-dir weights/sam-3d-objects-mlx-8bit

Validate and inspect it without loading all weights:

mlx-spatial-sam3d validate weights/sam-3d-objects-mlx-8bit
mlx-spatial-sam3d inspect weights/sam-3d-objects-mlx-8bit

Generate a Gaussian Splat PLY:

mlx-spatial-sam3d reconstruct \
  weights/sam-3d-objects-mlx-8bit \
  inputs/sam3d/object-rmbg.png \
  --mask inputs/sam3d/object-mask.png \
  --moge-root weights/sam-3d-objects-mlx-8bit/moge \
  --output outputs/sam3d/object-8bit/gaussians.ply \
  --trace-output outputs/sam3d/object-8bit/trace.json \
  --memory-profile balanced

Add --glb-output outputs/sam3d/object-8bit/object.glb to decode and export a mesh-backed GLB in the same run.

SAM3D requires a binary object mask aligned with the input image. A clean foreground extraction is strongly recommended: transparent background pixels should have alpha zero, and the supplied mask should cover the intended object without including the surrounding scene.

Bundle

The repository preserves the directory and checkpoint names used by the full-precision MLX bundle:

Checkpoint	Logical tensors	INT8 matrices	Bytes
checkpoints/ss_generator.safetensors	1,741	556	2,197,959,476
checkpoints/ss_decoder.safetensors	74	0	147,592,136
checkpoints/slat_generator.safetensors	1,225	384	1,578,129,504
checkpoints/slat_decoder_gs.safetensors	101	48	97,148,339
checkpoints/slat_decoder_gs_4.safetensors	101	48	95,942,210
checkpoints/slat_decoder_mesh.safetensors	120	48	119,546,584
moge/model.safetensors	421	96	388,551,606
Total	3,783	1,180	4,624,869,855

The remaining YAML configuration, conversion metadata, audit, and license files are small and are included alongside the checkpoints.

Quantization

The quantization scheme is affine 8-bit with group size 64. Packed weights are stored as uint32 with FP32 scales and biases and are executed directly with mx.quantized_matmul.

The following block-internal two-dimensional weights are quantized:

• DINO condition-encoder attention and MLP matrices;
• the point-condition transformer block;
• sparse-structure and structured-latent generator transformer blocks;
• Gaussian and mesh decoder transformer torso blocks;
• the bundled MoGe ViT backbone attention and MLP matrices.

The following tensors remain in their original precision:

• all dense, sparse, and transposed convolutions;
• the complete sparse-structure decoder;
• patch embeddings, input mappings, condition projections, and latent mappings;
• normalization parameters, biases, learned tokens, and positional tensors;
• final Gaussian, mesh, geometry, pose, and MoGe output heads;
• matrices whose input dimension is incompatible with group size 64.

Physical packed arrays use internal qweight, scale, and bias suffixes. The runtime reconstructs the original logical tensor names from metadata, so model code and configuration continue to use the same checkpoint contract. Format details are embedded under mlx_spatial.sam3d.quantization in each quantized safetensors file.

Reproducing the Bundle

Starting from the full-precision MLX conversion:

mlx-spatial-sam3d-quantize \
  weights/sam-3d-objects-mlx \
  weights/sam-3d-objects-mlx-8bit \
  --bits 8 \
  --group-size 64

The quantized output is a complete runtime root. Do not place copies of the full-precision checkpoints in the 8-bit repository.

Verification

• Logical checkpoint inspection matched all 3,783 source tensor names, shapes, and dtypes exactly.
• The quantization inventory contains 1,180 packed matrices across six quantized checkpoints; the convolutional sparse-structure decoder remains unchanged.
• A sampled static weight audit measured median 8-bit reconstruction SQNR of approximately 45 dB across checkpoint groups, with a worst observed matrix at 41 dB.
• The SAM3D test suite passed 163 tests, with one test deselected.
• A background-removed 900 × 900 input with an aligned mask completed all nine inference stages with no blocker and produced 495,008 finite Gaussians.
• The generated geometry passed the runtime's nominal axis-range check.

The end-to-end run establishes runtime compatibility and artifact health. It is not a formal claim of visual equivalence to the full-precision model, and the observed run is not presented as a general performance benchmark.

Limitations

• Quantization can change sparse occupancy, geometry, appearance, and other generation details relative to the full-precision variant.
• A useful object-aligned mask is required. Background leakage, broad masks, soft edges, reflections, and thin structures can reduce reconstruction quality.
• Single-view reconstruction cannot determine unseen geometry with certainty.
• The packed checkpoint format requires mlx-spatial; generic safetensors loaders will see the physical packed arrays rather than the logical weights.
• This bundle supports inference, not training or fine-tuning.
• Standard PLY viewers may not render Gaussian Splat fields correctly. Use a 3DGS-aware viewer for the Gaussian artifact, or request GLB export.

License and Attribution

This bundle is derived from Meta's SAM 3D Objects release and is distributed under the SAM License. Read the bundled LICENSE and the upstream SAM License before use. Redistribution of SAM Materials and derivative works remains subject to that agreement.

The bundle also contains a converted and quantized Ruicheng/moge-vitl checkpoint used for pointmap estimation. That checkpoint is published under Apache 2.0. Users are responsible for complying with both sets of terms.

This repository is not affiliated with or endorsed by Meta or the MoGe authors. Publications using these weights should acknowledge the original SAM 3D Objects and MoGe work.

Links

• App Automaton
• AppAutomaton models on Hugging Face
• appautomaton/mlx-spatial — MLX-native 3D and spatial inference for Apple Silicon.
• mlx-spatial documentation
• mlx-spatial on PyPI
• SAM3D guide
• Full-precision MLX variant
• Upstream SAM 3D Objects model
• Upstream SAM 3D Objects source
• MoGe ViT-L checkpoint
• MoGe source
• MLX