| --- |
| license: apache-2.0 |
| --- |
| # Granite 4 GGUF (4-bit Quantized) |
|
|
| This repository hosts GGUF-format quantized versions of **IBM Granite 4** models at multiple parameter sizes. |
|
|
| The models provided here are intended for **local inference** and are suitable for use with **SciTools’ Understand and Onboard**, as well as other tools and runtimes that support the GGUF format (for example, llama.cpp-based applications). |
|
|
| --- |
|
|
| ## Model Details |
|
|
| - Base models: IBM Granite 4 |
| - Variants provided: Micro 3B |
| - Format: GGUF |
| - Quantization: 4-bit (model-specific, see table below) |
| - Intended use: Local inference, code understanding, general-purpose chat |
| - Languages: English (as supported by Granite 4) |
|
|
| --- |
|
|
|
|
| ## Quantization Process |
|
|
| - The **3B Granite 4 Micro** model is quantized using **Unsloth** tooling. |
| - No additional fine-tuning, rebalancing, or prompt modification was applied. |
| - Quantization parameters were not altered from their original sources. |
|
|
| These models are redistributed as-is to provide reproducible, efficient GGUF variants suitable for local workflows. |
|
|
| --- |
|
|
| ## What We Did Not Do |
|
|
| To be explicit: |
|
|
| - No additional fine-tuning |
| - No instruction rebalancing |
| - No safety, alignment, or prompt modifications |
| - No merging or model surgery |
|
|
| Any observed behavior is attributable to **Granite 4 and the applied quantization**, not downstream changes. |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| These models are suitable for: |
|
|
| - SciTools Understand and SciTools Onboard |
| - Local AI workflows |
| - Code comprehension and exploration |
| - Interactive chat and analysis |
| - Integration into developer tools that support GGUF |
|
|
| They are not intended for: |
|
|
| - Safety-critical or regulated decision-making |
| - Use cases requiring guaranteed factual accuracy |
| - Production deployment without independent evaluation |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - As 4-bit quantized models, some reduction in reasoning depth and precision is expected compared to full-precision checkpoints. |
| - Output quality varies between the 1B and 3B variants. |
| - Like all large language models, Granite 4 may produce incorrect or misleading outputs. |
|
|
| Evaluate carefully for your specific workload. |
|
|
| --- |
|
|
| ## License & Attribution |
|
|
| - Original models: IBM (Granite 4) |
| - Quantization: IBM and Unsloth |
| - Format: GGUF (llama.cpp ecosystem) |
|
|
| Please refer to the original Granite 4 license and usage terms. This repository redistributes quantized artifacts only and does not modify the underlying licensing conditions. |
|
|
| --- |
|
|
| ## Acknowledgements |
|
|
| Thanks to **IBM** for releasing the Granite 4 models and to **Unsloth** for providing efficient, reproducible quantization that enables practical local inference. |
|
|
