| --- |
| library_name: pytorch |
| license: other |
| tags: |
| - llm |
| - generative_ai |
| - android |
| pipeline_tag: text-generation |
|
|
| --- |
| |
|  |
|
|
| # Granite-4.0-Micro: Optimized for Qualcomm Devices |
|
|
| Granite 4.0 is a family of open language models from IBM designed for enterprise AI workloads including code generation, summarization, and retrieval-augmented generation. |
|
|
| This is based on the implementation of Granite-4.0-Micro found [here](https://huggingface.co/ibm-granite/granite-4.0-h-micro). |
| This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.57.2/src/qai_hub_models/models/granite_4_0_micro) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary). |
|
|
| Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device. |
|
|
| ## Deploying Granite-4.0-Micro on-device |
|
|
| Follow the [GenieX quickstart](https://geniex.aihub.qualcomm.com/en/get-started/quickstart) to install GenieX and deploy the model on a target device. |
|
|
| ## Getting Started |
| There are two ways to deploy this model on your device: |
|
|
| ### Option 1: Download Pre-Exported Models |
|
|
| Below are pre-exported model assets ready for deployment. |
|
|
| | Runtime | Precision | Chipset | SDK Versions | Download | |
| |---|---|---|---|---| |
| | GENIEX_LLAMACPP | q4_0 | Universal | | [Download](https://huggingface.co/unsloth/granite-4.0-h-micro-GGUF/resolve/main/granite-4.0-h-micro-Q4_0.gguf) |
|
|
| For more device-specific assets and performance metrics, visit **[Granite-4.0-Micro on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/granite_4_0_micro)**. |
|
|
|
|
| ### Option 2: Export with Custom Configurations |
|
|
| Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.57.2/src/qai_hub_models/models/granite_4_0_micro) Python library to compile and export the model with your own: |
| - Custom weights (e.g., fine-tuned checkpoints) |
| - Custom input shapes |
| - Target device and runtime configurations |
|
|
| This option is ideal if you need to customize the model beyond the default configuration provided here. |
|
|
| See our repository for [Granite-4.0-Micro on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.57.2/src/qai_hub_models/models/granite_4_0_micro) for usage instructions. |
|
|
| ## Model Details |
|
|
| **Model Type:** Model_use_case.text_generation |
| |
| **Model Stats:** |
| - Model architecture: Mamba-2 Hybrid architecture combining State Space Model (SSM) layers with Attention layers for efficient long-context processing. |
| - Supported languages: English |
| - TTFT: Time To First Token is the time it takes to generate the first response token. This is expressed as a range because it varies based on the length of the prompt. |
| - Response Rate: Rate of response generation after the first response token. |
| |
| ## Performance Summary |
| | Model | Runtime | Precision | Chipset | Context Length | Response Rate (tokens per second) | Time To First Token (range, seconds) |
| |---|---|---|---|---|---|--- |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Gen 5 Mobile | 512 | 25.719503 | 0.8499905 - 3.399962 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Gen 5 Mobile | 512 | 26.283972 | 0.8927765000000001 - 3.5711060000000003 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Gen 5 Mobile | 512 | 16.840913 | 0.23313175 - 0.932527 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Gen 5 Mobile | 4096 | 24.522214 | 1.1895051875 - 38.064166 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Gen 5 Mobile | 4096 | 22.047256 | 1.4127757187499999 - 45.208822999999995 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Gen 5 Mobile | 4096 | 15.315768 | 0.3090850625 - 9.890722 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Mobile | 512 | 25.130048 | 0.9481944999999999 - 3.7927779999999998 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Mobile | 512 | 26.288118 | 0.95749725 - 3.829989 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Mobile | 512 | 15.760148 | 0.30860325 - 1.234413 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Mobile | 4096 | 22.639083 | 1.1536551875 - 36.916966 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Mobile | 4096 | 23.049816 | 1.22555709375 - 39.217827 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® 8 Elite Mobile | 4096 | 14.515548 | 0.36028896875 - 11.529247 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® X Elite | 512 | 17.843282 | 0.69928525 - 2.797141 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® X Elite | 512 | 13.447186 | 0.6960275 - 2.78411 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® X Elite | 512 | 11.275263 | 0.34302550000000004 - 1.3721020000000002 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® X Elite | 4096 | 25.288631 | 0.51570025 - 16.502408 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® X Elite | 4096 | 24.661293 | 0.54267315625 - 17.365541 |
| | Granite-4.0-Micro | GENIEX_LLAMACPP | q4_0 | Snapdragon® X Elite | 4096 | 11.122384 | 0.37017084375000003 - 11.845467000000001 |
| |
| ## License |
| * The license for the original implementation of Granite-4.0-Micro can be found |
| [here](https://github.com/ibm-granite/granite-4.0-language-models/blob/main/LICENSE). |
| |
| ## References |
| * [Granite 4.0](https://www.ibm.com/granite/docs/models/granite) |
| * [Source Model Implementation](https://huggingface.co/ibm-granite/granite-4.0-h-micro) |
| |
| ## Community |
| * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI. |
| * For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com). |
| |
| ## Usage and Limitations |
| |
| This model may not be used for or in connection with any of the following applications: |
| |
| - Accessing essential private and public services and benefits; |
| - Administration of justice and democratic processes; |
| - Assessing or recognizing the emotional state of a person; |
| - Biometric and biometrics-based systems, including categorization of persons based on sensitive characteristics; |
| - Education and vocational training; |
| - Employment and workers management; |
| - Exploitation of the vulnerabilities of persons resulting in harmful behavior; |
| - General purpose social scoring; |
| - Law enforcement; |
| - Management and operation of critical infrastructure; |
| - Migration, asylum and border control management; |
| - Predictive policing; |
| - Real-time remote biometric identification in public spaces; |
| - Recommender systems of social media platforms; |
| - Scraping of facial images (from the internet or otherwise); and/or |
| - Subliminal manipulation |
| |
