Open4bits / Granite-4.0-Micro-MLX-3Bit

This repository provides the Granite-4.0 Micro model quantized to 3-bit in MLX format, published by Open4bits to enable efficient local inference with low memory usage and broad hardware compatibility.

The underlying Granite-4.0 model and architecture are developed and owned by their original authors. This repository contains only a 3-bit quantized MLX conversion of the original model weights.

The model is designed for lightweight, high-performance text generation and instruction-following tasks, making it suitable for local and resource-constrained environments.

Open4bits has started supporting MLX models to broaden compatibility with emerging quantization formats and efficient runtimes.

---

Model Overview

Granite-4.0 Micro is a compact variant of the Granite-4.0 architecture optimized for efficient inference and lower resource footprints. This release provides a 3-bit quantized checkpoint in MLX format, enabling fast inference on CPUs and supported accelerators with reduced memory demands.

---

Model Details

• Base Model: Granite-4.0
• Variant: Micro
• Quantization: 3-bit
• Format: MLX
• Task: Text generation, instruction following
• Weight tying: Preserved
• Compatibility: MLX-enabled inference engines and supported runtimes

This quantized format balances inference performance with lower resource requirements while preserving core architectural design.

---

Intended Use

This model is intended for:

• Local text generation and chat applications
• CPU-based or resource-efficient deployments
• Research, experimentation, and prototyping
• Offline or self-hosted AI systems

---

Limitations

• Reduced performance compared to full-precision variants
• Output quality depends on prompt engineering and inference settings
• Not fine-tuned for highly domain-specific tasks

---

License

This model follows the Apache licence 2.0 of the base Granite-4.0 model. Users must comply with the licensing conditions defined by the original creators.

---

Support

If you find this model useful, please consider supporting the project. Your support encourages Open4bits to continue releasing and maintaining efficient open models for the community.