README.md

---
license: apache-2.0
license_name: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
language:
- en
- hi
- multilingual
- gu
- mr
- te
- fr
- ja
- zh
pipeline_tag: image-text-to-text
tags:
- gemma4
- quatfit
- multimodal
- vision
- audio
- text generation
- image-text-to-text
- agentic
- coding
---

<p align="center">
  <img src="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/banner.png" alt="Quatfit Mini Banner" width="100%">
</p>

<h1 align="center">Quatfit Mini</h1>
<h3 align="center">Gemma 4–Based 8B Multimodal Model · Up to 4× Faster Inference · 131K Context</h3>

<p align="center">
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Model%20Hub-FFD21E?style=flat-square" alt="Hugging Face"></a>
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini-GGUF"><img src="https://img.shields.io/badge/⚙️%20GGUF%20Builds-Quatfit--Mini--GGUF-4052F5?style=flat-square" alt="GGUF Builds"></a>
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf"><img src="https://img.shields.io/badge/📄%20Technical%20Report-PDF-8A2BE2?style=flat-square" alt="Technical Report"></a>
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini"><img src="https://img.shields.io/badge/⚡%204×%20Faster-FP32%20%7C%20GGUF-00C853?style=flat-square" alt="Performance"></a>
  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/📜%20License-Apache%202.0-E53935?style=flat-square" alt="License"></a>
</p>

---

## Quatfit Mini

**Quatfit Mini** is an **8-billion-parameter multimodal model built on Google's Gemma 4 architecture** and further optimized by **Quatfit AI Research** for efficient deployment, long-context reasoning, and agentic AI workflows.

The model inherits the strong multimodal capabilities of Gemma 4 while adding Quatfit's optimization stack for faster inference, improved GGUF performance, and streamlined deployment across consumer hardware. **Weights on this repository are published in full FP32 precision** for maximum numerical fidelity and to give downstream users a clean base for further fine-tuning; bf16/fp16 casting and quantized GGUF builds are provided separately for lower-memory inference.

### Highlights

- Built on **Google Gemma 4**
- Native multimodal reasoning (Text + Image + Audio)
- 131,072 token context window
- **Full FP32 weights** — highest-fidelity base for fine-tuning and research
- Up to **4× faster inference** with Quatfit optimizations (bf16 / GGUF)
- Optimized GGUF builds
- Consumer GPU friendly (via bf16 casting or GGUF)
- Agentic workflow optimized
- Native Hugging Face Transformers support

---

# Base Model

Quatfit Mini is derived from **Google Gemma 4** and preserves the original multimodal transformer architecture.

Quatfit AI Research extends the base model through:

- Supervised instruction tuning
- Alignment optimization
- Deployment optimization
- GGUF optimization
- Speculative decoding support (MTP drafter)
- Optimized inference pipeline
- Consumer hardware optimization

---

# Model Summary

| Property | Value |
|----------|-------|
| Parameters | **8B** |
| Base Architecture | Google Gemma 4 |
| Model Type | Decoder-only Multimodal Transformer |
| Context Length | **131,072** tokens |
| Published Precision | **FP32** (full precision) |
| Recommended Inference Precision | BF16 / FP16 (cast), or quantized GGUF |
| Vocabulary | 262K (SentencePiece) |
| Languages | English, Hindi, Multilingual |
| Modalities | Text, Image, Audio |

> **Note on precision:** This repository stores weights in **FP32**. FP32 roughly doubles memory usage relative to BF16 (~32 GB vs ~16 GB for the 8B backbone) but avoids any precision loss from the original training checkpoint, making it the preferred starting point for further fine-tuning or precision-sensitive research. For everyday inference, casting to `bfloat16`/`float16` or using one of the quantized GGUF builds below is recommended and carries negligible quality loss.

---

# Intended Uses

## Primary Use Cases

- Agentic AI
- Coding Assistant
- Visual Question Answering
- OCR
- Diagram Understanding
- Audio Understanding
- Long-context reasoning
- Research Copilot
- Productivity Automation
- Tool Calling
- API Development
- Fine-tuning / continued pre-training (FP32 base recommended)

## Out-of-Scope

- Medical diagnosis
- Legal advice
- High-risk decision making
- Enterprise-scale software engineering
- Repository-scale code generation
- Competitive programming

---

# Installation

```bash
pip install "transformers[torch]"
```

---

# Loading the Model

By default the model loads in its published **FP32** precision. For most inference use cases, casting to `bfloat16` is strongly recommended to roughly halve memory usage and increase throughput with negligible quality impact.

```python
import torch
from transformers import (
    AutoProcessor,
    AutoModelForImageTextToText,
)

# Native FP32 (highest fidelity, ~32 GB VRAM, recommended for fine-tuning)
model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini",
    torch_dtype=torch.float32,
    device_map="auto"
)

# Recommended for inference: cast to BF16 (~16 GB VRAM)
model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "Quatfit/Quatfit-Mini"
)
```

---

# Text Generation

```python
messages = [
    {
        "role": "user",
        "content": "Write a Python implementation of binary search."
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=512
)

print(processor.decode(outputs[0]))
```

---

# Image Understanding

```python
messages = [
    {
        "role": "user",
        "content": [
            {
                "type":"text",
                "text":"Describe this image."
            },
            {
                "type":"image",
                "image":"image.png"
            }
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=512
)

print(processor.decode(outputs[0]))
```

---

# Long Context Example

```python
text = open("document.txt").read()

messages = [
    {
        "role":"user",
        "content":f"Summarize:\n\n{text}"
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024
)

print(processor.decode(outputs[0]))
```

---

# Performance

| Configuration | Relative Speed | VRAM |
|---------------|---------------:|------:|
| Quatfit Mini FP32 (native, published) | 1× | ~32 GB |
| Quatfit Mini BF16 (cast) | 2.5× | ~16 GB |
| + Speculative Decoding (MTP drafter, BF16) | 3.9× | ~16 GB |
| GGUF Q4_K_M | 4.1× | ~5 GB |

FP32 is the published, reference-quality precision; the speedups above are measured relative to it. BF16 casting and the quantized GGUF builds are derived from the same FP32 weights and carry negligible accuracy loss.

---

# Benchmark Scores

| Domain | Accuracy |
|---------|----------:|
| Overall | **89.1%** |
| CLI | 95.0% |
| Exams | 93.3% |
| Coding | 92.5% |
| Agentic Tasks | 92.5% |
| Science | 91.7% |
| Finance | 90.0% |
| Security | 90.0% |
| Social Intelligence | 90.0% |
| Reasoning | 88.9% |
| Expert Knowledge | 83.8% |
| Mathematics | 81.3% |

Scores were measured on the FP32 reference weights; BF16 and Q4_K_M/Q5_K_M GGUF builds match these scores within noise.

---

# Architecture

Quatfit Mini preserves the **Google Gemma 4** multimodal architecture while introducing Quatfit-specific optimizations for inference and deployment.

## Foundation

| Component | Description |
|-----------|-------------|
| Base Model | Google Gemma 4 |
| Architecture | Dense Decoder-only Transformer |
| Text Backbone | Gemma 4 |
| Vision Backbone | Gemma 4 Vision Transformer |
| Audio Backbone | Gemma 4 Conformer |
| Context Length | 131,072 |
| Published Precision | FP32 |

---

## Text Decoder

| Component | Value |
|-----------|------|
| Layers | 42 (5:1 local sliding-window : global attention) |
| Hidden Size | 2,560 |
| Attention Heads | 8 |
| KV Heads | 2 (Grouped Query Attention) |
| KV-Shared Layers | 18 of 42 |
| Key/Value Reuse | Enabled on global attention layers (V = K) |
| Feed Forward | GeGLU |
| Position Encoding | p-RoPE (partial rotary on global layers) |
| Normalization | Pre-norm + post-norm RMSNorm, QK-Norm |
| Flash Attention | Flash Attention 3 |

---

## Vision Encoder

- Gemma 4 Vision Transformer
- Patch Size 16×16
- Variable aspect ratio input handling
- Axial 2D-RoPE + 2D absolute position embeddings
- Native Visual Embeddings

---

## Audio Encoder

- Gemma 4 Conformer (USM-style)
- Continuous representations (no vector quantization)
- Streaming Compatible
- Causal Chunk Attention

---

## Quatfit Optimizations

Quatfit Mini extends Gemma 4 with deployment-focused optimizations including:

- Optimized GGUF conversion
- Faster llama.cpp inference
- Speculative decoding via an autoregressive MTP (multi-token-prediction) drafter
- Flash Attention 3
- Sliding Window Attention
- Grouped Query Attention + KV-cache sharing + key/value reuse
- Consumer GPU optimization
- Memory-efficient inference
- Quantization-aware deployment (GGUF, mobile int2/int4)

---

# Training

Quatfit Mini is built upon the pretrained **Google Gemma 4** model and further optimized by Quatfit AI Research through post-training techniques including:

- Supervised Fine-Tuning (SFT)
- Preference Alignment
- Instruction Tuning
- Multimodal Alignment
- Inference Optimization

The underlying foundation model retains the original Gemma 4 pretraining while Quatfit contributes additional post-training and deployment optimizations. The final checkpoint is published in FP32 to preserve full training precision for downstream fine-tuning.

---

# GGUF Quantized Builds

For local inference with **llama.cpp**, **Ollama**, **LM Studio**, **Jan**, or **Open WebUI**, use the companion repository:

**[Quatfit/Quatfit-Mini-GGUF](https://huggingface.co/Quatfit/Quatfit-Mini-GGUF)**

It provides every legacy and K-quant format from `F16` down to `Q2_K`, converted from these FP32 weights. **Each text quant is paired with its own `mmproj` file**, so vision and audio input work immediately after download — no separate multimodal conversion step required.

---

# Cross Platform Support

| Format | VRAM | Platforms |
|--------|------:|----------|
| FP32 (this repo, published) | ~32 GB | Transformers, fine-tuning pipelines |
| BF16 (cast at load time) | ~16 GB | Transformers, vLLM, TGI |
| F16 GGUF | ~16.1 GB | llama.cpp, Ollama, LM Studio |
| Q8_0 GGUF | ~8.5 GB | llama.cpp, Ollama, LM Studio |
| Q6_K GGUF | ~6.6 GB | llama.cpp |
| Q5_K_M GGUF | ~5.7 GB | Ollama, LM Studio, llama.cpp |
| Q4_K_M GGUF | ~4.9 GB | Ollama, LM Studio, llama.cpp, Jan, Open WebUI |
| Q2_K GGUF | ~3.2 GB | llama.cpp (CPU-friendly, lower quality) |

See the [GGUF repository](https://huggingface.co/Quatfit/Quatfit-Mini-GGUF) for the full quant lineup, exact sizes, and paired `mmproj` files.

---

# Hardware

## Recommended

- **For native FP32 (~32 GB VRAM):** A100 (40/80 GB), H100, RTX 6000 Ada
- **For BF16 (~16 GB VRAM):** RTX 3090, RTX 4090, RTX A6000, A100
- **For GGUF quantized builds (~5–9 GB VRAM):** RTX 3060/3070 and above

## Minimum

- GPU with ~6 GB VRAM (Q4_K_M / Q5_K_M GGUF)
- CPU inference through llama.cpp (GGUF)

> The FP32 weights in this repository require significantly more VRAM (~32 GB) than the BF16 or GGUF derivatives. Most consumer GPUs (e.g. 24 GB cards) should load the model with `torch_dtype=torch.bfloat16` or use a GGUF build rather than native FP32.

---

# Responsible AI

Quatfit Mini may generate inaccurate, biased, or inappropriate outputs.

For production deployments:

- Verify critical information
- Apply RAG for factual grounding
- Use application-level safety filters
- Keep human oversight for high-risk domains

---

# Citation

```bibtex
@article{quatfitmini2026,
    title={Quatfit Mini: A Gemma 4-Based Multimodal Model Optimized for Efficient Inference},
    author={Quatfit AI Research},
    year={2026}
}
```

---

# License

Apache License 2.0. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for details.

---

<p align="center">
<a href="https://huggingface.co/Quatfit/Quatfit-Mini">🤗 Hugging Face</a> •
<a href="https://huggingface.co/Quatfit/Quatfit-Mini-GGUF">⚙️ GGUF Builds</a> •
<a href="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf">📄 Technical Report</a>
</p>