Quatfit Mini

Gemma 4–Based 8B Multimodal Model · Up to 4× Faster Inference · 131K Context

Quatfit Mini

Quatfit Mini is an 8-billion-parameter multimodal model built on Google's Gemma 4 architecture and further optimized by Quatfit AI Research for efficient deployment, long-context reasoning, and agentic AI workflows.

The model inherits the strong multimodal capabilities of Gemma 4 while adding Quatfit's optimization stack for faster inference, improved GGUF performance, and streamlined deployment across consumer hardware. Weights on this repository are published in full FP32 precision for maximum numerical fidelity and to give downstream users a clean base for further fine-tuning; bf16/fp16 casting and quantized GGUF builds are provided separately for lower-memory inference.

Highlights

Built on Google Gemma 4
Native multimodal reasoning (Text + Image + Audio)
131,072 token context window
Full FP32 weights — highest-fidelity base for fine-tuning and research
Up to 4× faster inference with Quatfit optimizations (bf16 / GGUF)
Optimized GGUF builds
Consumer GPU friendly (via bf16 casting or GGUF)
Agentic workflow optimized
Native Hugging Face Transformers support

Base Model

Quatfit Mini is derived from Google Gemma 4 and preserves the original multimodal transformer architecture.

Quatfit AI Research extends the base model through:

Supervised instruction tuning
Alignment optimization
Deployment optimization
GGUF optimization
Speculative decoding support (MTP drafter)
Optimized inference pipeline
Consumer hardware optimization

Model Summary

Property	Value
Parameters	8B
Base Architecture	Google Gemma 4
Model Type	Decoder-only Multimodal Transformer
Context Length	131,072 tokens
Published Precision	FP32 (full precision)
Recommended Inference Precision	BF16 / FP16 (cast), or quantized GGUF
Vocabulary	262K (SentencePiece)
Languages	English, Hindi, Multilingual
Modalities	Text, Image, Audio

Note on precision: This repository stores weights in FP32. FP32 roughly doubles memory usage relative to BF16 (~32 GB vs ~16 GB for the 8B backbone) but avoids any precision loss from the original training checkpoint, making it the preferred starting point for further fine-tuning or precision-sensitive research. For everyday inference, casting to bfloat16/float16 or using one of the quantized GGUF builds below is recommended and carries negligible quality loss.

Intended Uses

Primary Use Cases

Agentic AI
Coding Assistant
Visual Question Answering
OCR
Diagram Understanding
Audio Understanding
Long-context reasoning
Research Copilot
Productivity Automation
Tool Calling
API Development
Fine-tuning / continued pre-training (FP32 base recommended)

Out-of-Scope

Medical diagnosis
Legal advice
High-risk decision making
Enterprise-scale software engineering
Repository-scale code generation
Competitive programming

Installation

pip install "transformers[torch]"

Loading the Model

By default the model loads in its published FP32 precision. For most inference use cases, casting to bfloat16 is strongly recommended to roughly halve memory usage and increase throughput with negligible quality impact.

import torch
from transformers import (
    AutoProcessor,
    AutoModelForImageTextToText,
)

# Native FP32 (highest fidelity, ~32 GB VRAM, recommended for fine-tuning)
model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini",
    torch_dtype=torch.float32,
    device_map="auto"
)

# Recommended for inference: cast to BF16 (~16 GB VRAM)
model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "Quatfit/Quatfit-Mini"
)

Text Generation

messages = [
    {
        "role": "user",
        "content": "Write a Python implementation of binary search."
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=512
)

print(processor.decode(outputs[0]))

Image Understanding

messages = [
    {
        "role": "user",
        "content": [
            {
                "type":"text",
                "text":"Describe this image."
            },
            {
                "type":"image",
                "image":"image.png"
            }
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=512
)

print(processor.decode(outputs[0]))

Long Context Example

text = open("document.txt").read()

messages = [
    {
        "role":"user",
        "content":f"Summarize:\n\n{text}"
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024
)

print(processor.decode(outputs[0]))

Performance

Configuration	Relative Speed	VRAM
Quatfit Mini FP32 (native, published)	1×	~32 GB
Quatfit Mini BF16 (cast)	2.5×	~16 GB
+ Speculative Decoding (MTP drafter, BF16)	3.9×	~16 GB
GGUF Q4_K_M	4.1×	~5 GB

FP32 is the published, reference-quality precision; the speedups above are measured relative to it. BF16 casting and the quantized GGUF builds are derived from the same FP32 weights and carry negligible accuracy loss.

Benchmark Scores

Domain	Accuracy
Overall	89.1%
CLI	95.0%
Exams	93.3%
Coding	92.5%
Agentic Tasks	92.5%
Science	91.7%
Finance	90.0%
Security	90.0%
Social Intelligence	90.0%
Reasoning	88.9%
Expert Knowledge	83.8%
Mathematics	81.3%

Scores were measured on the FP32 reference weights; BF16 and Q4_K_M/Q5_K_M GGUF builds match these scores within noise.

Architecture

Quatfit Mini preserves the Google Gemma 4 multimodal architecture while introducing Quatfit-specific optimizations for inference and deployment.

Foundation

Component	Description
Base Model	Google Gemma 4
Architecture	Dense Decoder-only Transformer
Text Backbone	Gemma 4
Vision Backbone	Gemma 4 Vision Transformer
Audio Backbone	Gemma 4 Conformer
Context Length	131,072
Published Precision	FP32

Text Decoder

Component	Value
Layers	42 (5:1 local sliding-window : global attention)
Hidden Size	2,560
Attention Heads	8
KV Heads	2 (Grouped Query Attention)
KV-Shared Layers	18 of 42
Key/Value Reuse	Enabled on global attention layers (V = K)
Feed Forward	GeGLU
Position Encoding	p-RoPE (partial rotary on global layers)
Normalization	Pre-norm + post-norm RMSNorm, QK-Norm
Flash Attention	Flash Attention 3

Vision Encoder

Gemma 4 Vision Transformer
Patch Size 16×16
Variable aspect ratio input handling
Axial 2D-RoPE + 2D absolute position embeddings
Native Visual Embeddings

Audio Encoder

Gemma 4 Conformer (USM-style)
Continuous representations (no vector quantization)
Streaming Compatible
Causal Chunk Attention

Quatfit Optimizations

Quatfit Mini extends Gemma 4 with deployment-focused optimizations including:

Optimized GGUF conversion
Faster llama.cpp inference
Speculative decoding via an autoregressive MTP (multi-token-prediction) drafter
Flash Attention 3
Sliding Window Attention
Grouped Query Attention + KV-cache sharing + key/value reuse
Consumer GPU optimization
Memory-efficient inference
Quantization-aware deployment (GGUF, mobile int2/int4)

Training

Quatfit Mini is built upon the pretrained Google Gemma 4 model and further optimized by Quatfit AI Research through post-training techniques including:

Supervised Fine-Tuning (SFT)
Preference Alignment
Instruction Tuning
Multimodal Alignment
Inference Optimization

The underlying foundation model retains the original Gemma 4 pretraining while Quatfit contributes additional post-training and deployment optimizations. The final checkpoint is published in FP32 to preserve full training precision for downstream fine-tuning.

GGUF Quantized Builds

For local inference with llama.cpp, Ollama, LM Studio, Jan, or Open WebUI, use the companion repository:

Quatfit/Quatfit-Mini-GGUF

It provides every legacy and K-quant format from F16 down to Q2_K, converted from these FP32 weights. Each text quant is paired with its own mmproj file, so vision and audio input work immediately after download — no separate multimodal conversion step required.

Cross Platform Support

Format	VRAM	Platforms
FP32 (this repo, published)	~32 GB	Transformers, fine-tuning pipelines
BF16 (cast at load time)	~16 GB	Transformers, vLLM, TGI
F16 GGUF	~16.1 GB	llama.cpp, Ollama, LM Studio
Q8_0 GGUF	~8.5 GB	llama.cpp, Ollama, LM Studio
Q6_K GGUF	~6.6 GB	llama.cpp
Q5_K_M GGUF	~5.7 GB	Ollama, LM Studio, llama.cpp
Q4_K_M GGUF	~4.9 GB	Ollama, LM Studio, llama.cpp, Jan, Open WebUI
Q2_K GGUF	~3.2 GB	llama.cpp (CPU-friendly, lower quality)

See the GGUF repository for the full quant lineup, exact sizes, and paired mmproj files.

Hardware

Minimum

GPU with ~6 GB VRAM (Q4_K_M / Q5_K_M GGUF)
CPU inference through llama.cpp (GGUF)

The FP32 weights in this repository require significantly more VRAM (~32 GB) than the BF16 or GGUF derivatives. Most consumer GPUs (e.g. 24 GB cards) should load the model with torch_dtype=torch.bfloat16 or use a GGUF build rather than native FP32.

Responsible AI

Quatfit Mini may generate inaccurate, biased, or inappropriate outputs.

For production deployments:

Verify critical information
Apply RAG for factual grounding
Use application-level safety filters
Keep human oversight for high-risk domains

Citation

@article{quatfitmini2026,
    title={Quatfit Mini: A Gemma 4-Based Multimodal Model Optimized for Efficient Inference},
    author={Quatfit AI Research},
    year={2026}
}

License

Apache License 2.0. See LICENSE for details.

🤗 Hugging Face • ⚙️ GGUF Builds • 📄 Technical Report

Downloads last month: 874

Safetensors

Model size

8B params

Tensor type

F32

Model tree for Quatfit/Quatfit-Mini

Quantizations

1 model