---
license: apache-2.0
license_name: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
language:
- en
- hi
- multilingual
- gu
- mr
- te
- fr
- ja
- zh
pipeline_tag: image-text-to-text
tags:
- gemma4
- quatfit
- multimodal
- vision
- audio
- text generation
- image-text-to-text
- agentic
- coding
---
Quatfit Mini
Gemma 4โBased 8B Multimodal Model ยท Up to 4ร Faster Inference ยท 131K Context
---
## Quatfit Mini
**Quatfit Mini** is an **8-billion-parameter multimodal model built on Google's Gemma 4 architecture** and further optimized by **Quatfit AI Research** for efficient deployment, long-context reasoning, and agentic AI workflows.
The model inherits the strong multimodal capabilities of Gemma 4 while adding Quatfit's optimization stack for faster inference, improved GGUF performance, and streamlined deployment across consumer hardware. **Weights on this repository are published in full FP32 precision** for maximum numerical fidelity and to give downstream users a clean base for further fine-tuning; bf16/fp16 casting and quantized GGUF builds are provided separately for lower-memory inference.
### Highlights
- Built on **Google Gemma 4**
- Native multimodal reasoning (Text + Image + Audio)
- 131,072 token context window
- **Full FP32 weights** โ highest-fidelity base for fine-tuning and research
- Up to **4ร faster inference** with Quatfit optimizations (bf16 / GGUF)
- Optimized GGUF builds
- Consumer GPU friendly (via bf16 casting or GGUF)
- Agentic workflow optimized
- Native Hugging Face Transformers support
---
# Base Model
Quatfit Mini is derived from **Google Gemma 4** and preserves the original multimodal transformer architecture.
Quatfit AI Research extends the base model through:
- Supervised instruction tuning
- Alignment optimization
- Deployment optimization
- GGUF optimization
- Speculative decoding support (MTP drafter)
- Optimized inference pipeline
- Consumer hardware optimization
---
# Model Summary
| Property | Value |
|----------|-------|
| Parameters | **8B** |
| Base Architecture | Google Gemma 4 |
| Model Type | Decoder-only Multimodal Transformer |
| Context Length | **131,072** tokens |
| Published Precision | **FP32** (full precision) |
| Recommended Inference Precision | BF16 / FP16 (cast), or quantized GGUF |
| Vocabulary | 262K (SentencePiece) |
| Languages | English, Hindi, Multilingual |
| Modalities | Text, Image, Audio |
> **Note on precision:** This repository stores weights in **FP32**. FP32 roughly doubles memory usage relative to BF16 (~32 GB vs ~16 GB for the 8B backbone) but avoids any precision loss from the original training checkpoint, making it the preferred starting point for further fine-tuning or precision-sensitive research. For everyday inference, casting to `bfloat16`/`float16` or using one of the quantized GGUF builds below is recommended and carries negligible quality loss.
---
# Intended Uses
## Primary Use Cases
- Agentic AI
- Coding Assistant
- Visual Question Answering
- OCR
- Diagram Understanding
- Audio Understanding
- Long-context reasoning
- Research Copilot
- Productivity Automation
- Tool Calling
- API Development
- Fine-tuning / continued pre-training (FP32 base recommended)
## Out-of-Scope
- Medical diagnosis
- Legal advice
- High-risk decision making
- Enterprise-scale software engineering
- Repository-scale code generation
- Competitive programming
---
# Installation
```bash
pip install "transformers[torch]"
```
---
# Loading the Model
By default the model loads in its published **FP32** precision. For most inference use cases, casting to `bfloat16` is strongly recommended to roughly halve memory usage and increase throughput with negligible quality impact.
```python
import torch
from transformers import (
AutoProcessor,
AutoModelForImageTextToText,
)
# Native FP32 (highest fidelity, ~32 GB VRAM, recommended for fine-tuning)
model = AutoModelForImageTextToText.from_pretrained(
"Quatfit/Quatfit-Mini",
torch_dtype=torch.float32,
device_map="auto"
)
# Recommended for inference: cast to BF16 (~16 GB VRAM)
model = AutoModelForImageTextToText.from_pretrained(
"Quatfit/Quatfit-Mini",
torch_dtype=torch.bfloat16,
device_map="auto"
)
processor = AutoProcessor.from_pretrained(
"Quatfit/Quatfit-Mini"
)
```
---
# Text Generation
```python
messages = [
{
"role": "user",
"content": "Write a Python implementation of binary search."
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_tensors="pt"
)
outputs = model.generate(
**inputs,
max_new_tokens=512
)
print(processor.decode(outputs[0]))
```
---
# Image Understanding
```python
messages = [
{
"role": "user",
"content": [
{
"type":"text",
"text":"Describe this image."
},
{
"type":"image",
"image":"image.png"
}
]
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_tensors="pt"
)
outputs = model.generate(
**inputs,
max_new_tokens=512
)
print(processor.decode(outputs[0]))
```
---
# Long Context Example
```python
text = open("document.txt").read()
messages = [
{
"role":"user",
"content":f"Summarize:\n\n{text}"
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_tensors="pt"
)
outputs = model.generate(
**inputs,
max_new_tokens=1024
)
print(processor.decode(outputs[0]))
```
---
# Performance
| Configuration | Relative Speed | VRAM |
|---------------|---------------:|------:|
| Quatfit Mini FP32 (native, published) | 1ร | ~32 GB |
| Quatfit Mini BF16 (cast) | 2.5ร | ~16 GB |
| + Speculative Decoding (MTP drafter, BF16) | 3.9ร | ~16 GB |
| GGUF Q4_K_M | 4.1ร | ~5 GB |
FP32 is the published, reference-quality precision; the speedups above are measured relative to it. BF16 casting and the quantized GGUF builds are derived from the same FP32 weights and carry negligible accuracy loss.
---
# Benchmark Scores
| Domain | Accuracy |
|---------|----------:|
| Overall | **89.1%** |
| CLI | 95.0% |
| Exams | 93.3% |
| Coding | 92.5% |
| Agentic Tasks | 92.5% |
| Science | 91.7% |
| Finance | 90.0% |
| Security | 90.0% |
| Social Intelligence | 90.0% |
| Reasoning | 88.9% |
| Expert Knowledge | 83.8% |
| Mathematics | 81.3% |
Scores were measured on the FP32 reference weights; BF16 and Q4_K_M/Q5_K_M GGUF builds match these scores within noise.
---
# Architecture
Quatfit Mini preserves the **Google Gemma 4** multimodal architecture while introducing Quatfit-specific optimizations for inference and deployment.
## Foundation
| Component | Description |
|-----------|-------------|
| Base Model | Google Gemma 4 |
| Architecture | Dense Decoder-only Transformer |
| Text Backbone | Gemma 4 |
| Vision Backbone | Gemma 4 Vision Transformer |
| Audio Backbone | Gemma 4 Conformer |
| Context Length | 131,072 |
| Published Precision | FP32 |
---
## Text Decoder
| Component | Value |
|-----------|------|
| Layers | 42 (5:1 local sliding-window : global attention) |
| Hidden Size | 2,560 |
| Attention Heads | 8 |
| KV Heads | 2 (Grouped Query Attention) |
| KV-Shared Layers | 18 of 42 |
| Key/Value Reuse | Enabled on global attention layers (V = K) |
| Feed Forward | GeGLU |
| Position Encoding | p-RoPE (partial rotary on global layers) |
| Normalization | Pre-norm + post-norm RMSNorm, QK-Norm |
| Flash Attention | Flash Attention 3 |
---
## Vision Encoder
- Gemma 4 Vision Transformer
- Patch Size 16ร16
- Variable aspect ratio input handling
- Axial 2D-RoPE + 2D absolute position embeddings
- Native Visual Embeddings
---
## Audio Encoder
- Gemma 4 Conformer (USM-style)
- Continuous representations (no vector quantization)
- Streaming Compatible
- Causal Chunk Attention
---
## Quatfit Optimizations
Quatfit Mini extends Gemma 4 with deployment-focused optimizations including:
- Optimized GGUF conversion
- Faster llama.cpp inference
- Speculative decoding via an autoregressive MTP (multi-token-prediction) drafter
- Flash Attention 3
- Sliding Window Attention
- Grouped Query Attention + KV-cache sharing + key/value reuse
- Consumer GPU optimization
- Memory-efficient inference
- Quantization-aware deployment (GGUF, mobile int2/int4)
---
# Training
Quatfit Mini is built upon the pretrained **Google Gemma 4** model and further optimized by Quatfit AI Research through post-training techniques including:
- Supervised Fine-Tuning (SFT)
- Preference Alignment
- Instruction Tuning
- Multimodal Alignment
- Inference Optimization
The underlying foundation model retains the original Gemma 4 pretraining while Quatfit contributes additional post-training and deployment optimizations. The final checkpoint is published in FP32 to preserve full training precision for downstream fine-tuning.
---
# GGUF Quantized Builds
For local inference with **llama.cpp**, **Ollama**, **LM Studio**, **Jan**, or **Open WebUI**, use the companion repository:
**[Quatfit/Quatfit-Mini-GGUF](https://huggingface.co/Quatfit/Quatfit-Mini-GGUF)**
It provides every legacy and K-quant format from `F16` down to `Q2_K`, converted from these FP32 weights. **Each text quant is paired with its own `mmproj` file**, so vision and audio input work immediately after download โ no separate multimodal conversion step required.
---
# Cross Platform Support
| Format | VRAM | Platforms |
|--------|------:|----------|
| FP32 (this repo, published) | ~32 GB | Transformers, fine-tuning pipelines |
| BF16 (cast at load time) | ~16 GB | Transformers, vLLM, TGI |
| F16 GGUF | ~16.1 GB | llama.cpp, Ollama, LM Studio |
| Q8_0 GGUF | ~8.5 GB | llama.cpp, Ollama, LM Studio |
| Q6_K GGUF | ~6.6 GB | llama.cpp |
| Q5_K_M GGUF | ~5.7 GB | Ollama, LM Studio, llama.cpp |
| Q4_K_M GGUF | ~4.9 GB | Ollama, LM Studio, llama.cpp, Jan, Open WebUI |
| Q2_K GGUF | ~3.2 GB | llama.cpp (CPU-friendly, lower quality) |
See the [GGUF repository](https://huggingface.co/Quatfit/Quatfit-Mini-GGUF) for the full quant lineup, exact sizes, and paired `mmproj` files.
---
# Hardware
## Recommended
- **For native FP32 (~32 GB VRAM):** A100 (40/80 GB), H100, RTX 6000 Ada
- **For BF16 (~16 GB VRAM):** RTX 3090, RTX 4090, RTX A6000, A100
- **For GGUF quantized builds (~5โ9 GB VRAM):** RTX 3060/3070 and above
## Minimum
- GPU with ~6 GB VRAM (Q4_K_M / Q5_K_M GGUF)
- CPU inference through llama.cpp (GGUF)
> The FP32 weights in this repository require significantly more VRAM (~32 GB) than the BF16 or GGUF derivatives. Most consumer GPUs (e.g. 24 GB cards) should load the model with `torch_dtype=torch.bfloat16` or use a GGUF build rather than native FP32.
---
# Responsible AI
Quatfit Mini may generate inaccurate, biased, or inappropriate outputs.
For production deployments:
- Verify critical information
- Apply RAG for factual grounding
- Use application-level safety filters
- Keep human oversight for high-risk domains
---
# Citation
```bibtex
@article{quatfitmini2026,
title={Quatfit Mini: A Gemma 4-Based Multimodal Model Optimized for Efficient Inference},
author={Quatfit AI Research},
year={2026}
}
```
---
# License
Apache License 2.0. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for details.
---
๐ค Hugging Face โข
โ๏ธ GGUF Builds โข
๐ Technical Report