Quatfit/Quatfit-Mini-FP8

Quatfit Mini — FP8

FP8-Quantized Build for High-Throughput GPU Serving · 131K Context

---

Overview

This repository provides an FP8-quantized build of Quatfit Mini, an 8B-parameter multimodal model built on the Gemma 4 architecture. It is intended for high-throughput, GPU-served inference on FP8-capable hardware (NVIDIA Hopper, Ada Lovelace, and Blackwell generations), where it delivers close to BF16 quality at roughly half the memory footprint and with hardware-accelerated FP8 tensor-core throughput.

Unlike the GGUF builds — which target CPU and consumer-GPU local inference via llama.cpp — this repository is designed for server-side deployment with runtimes such as vLLM, TensorRT-LLM, and SGLang.

For local inference on consumer GPUs or CPU, use Quatfit/Quatfit-Mini-GGUF instead. For fine-tuning or maximum numerical fidelity, use the base Quatfit/Quatfit-Mini (FP32) repository.

---

Quantization Recipe

Property	Value
Format	FP8 (OCP E4M3)
Scope	Text decoder linear layers (attention projections + GeGLU FFN)
Scaling	Per-channel static weight scales, per-tensor dynamic activation scales
Calibration	Small representative text/code/multilingual calibration set
Vision / audio encoders	Kept at BF16 (unquantized)
Embedding + LM head	Kept at BF16

Following the same reasoning applied to the GGUF mmproj files, the vision and audio encoders are more sensitive to aggressive quantization than the text backbone, so they are left in BF16 here rather than cast to FP8. Only the text decoder's linear layers — the overwhelming majority of parameters and compute — are quantized, which is where FP8's throughput and memory benefits matter most.

---

Model Summary

Property	Value
Parameters	8B
Base Architecture	Google Gemma 4
Model Type	Decoder-only Multimodal Transformer
Context Length	131,072 tokens
Weight Precision	FP8 (E4M3) — text decoder; BF16 — encoders, embeddings, LM head
On-Disk Size	~11.5 GB
Vocabulary	262K (SentencePiece)
Modalities	Text, Image, Audio

---

Why FP8

FP8 differs from the quantized GGUF builds in an important way: on supported hardware, FP8 isn't just a smaller file — it runs on native FP8 tensor cores, so it increases raw compute throughput in addition to cutting memory and bandwidth, rather than only reducing memory like a CPU-oriented integer quant. Relative to the BF16-cast version of Quatfit Mini on the same GPU:

Configuration	Relative Throughput	VRAM
Quatfit Mini BF16	1×	~16 GB
Quatfit Mini FP8 (this repo)	~1.6–1.9×	~9.5 GB
Quatfit Mini FP8 + speculative decoding (MTP drafter)	~3–3.5×	~11.5 GB

Exact gains depend on GPU generation, batch size, and sequence length; the ranges above reflect typical serving conditions rather than a specific benchmark run.

---

Hardware Requirements

FP8 tensor-core acceleration requires one of the following NVIDIA architectures:

• Hopper — H100, H200
• Ada Lovelace — L4, L40, L40S, RTX 4090, RTX 4080
• Blackwell — B100, B200, RTX 50-series

On GPUs without native FP8 support (e.g. Ampere: A100, A10, RTX 3090), most runtimes will either reject FP8 weights or fall back to upcasting them to BF16 at load time — memory savings are retained, but the throughput benefit of native FP8 compute is not. For Ampere-class or older hardware, casting the base FP32 model to BF16 directly, or using a GGUF build, is generally a better fit.

---

Usage

vLLM

vllm serve Quatfit/Quatfit-Mini-FP8 \
  --max-model-len 131072 \
  --quantization fp8

from vllm import LLM, SamplingParams

llm = LLM(model="Quatfit/Quatfit-Mini-FP8", quantization="fp8", max_model_len=131072)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

outputs = llm.generate(["Explain the difference between GQA and MHA."], sampling_params)
print(outputs[0].outputs[0].text)

TensorRT-LLM

trtllm-build \
  --checkpoint_dir Quatfit-Mini-FP8-checkpoint \
  --output_dir ./trt_engines/quatfit-mini-fp8 \
  --gemm_plugin fp8 \
  --max_input_len 131072

🤗 Transformers

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini-FP8",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("Quatfit/Quatfit-Mini-FP8")

messages = [{"role": "user", "content": "Write a Python implementation of binary search."}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(outputs[0]))

On GPUs without native FP8 tensor cores, Transformers will run this checkpoint by upcasting to BF16 automatically — you'll still get the ~9.5 GB memory footprint benefit, but not the FP8 compute speedup.

---

Prompt Format

Quatfit Mini uses the Gemma 4-style turn format with support for tool calling and an optional thinking mode:

<|turn>user
Your message here<turn|>
<|turn>model

To enable extended reasoning traces, prefix the conversation with a system turn containing <|think|>. The model will emit its reasoning inside <|channel>thought ... <channel|> before the final response.

---

Benchmark Scores

Domain	Accuracy
Overall	89.0%
CLI	95.0%
Exams	93.3%
Coding	92.3%
Agentic Tasks	92.5%
Science	91.4%
Finance	90.0%
Security	89.8%
Social Intelligence	90.0%
Reasoning	88.6%
Expert Knowledge	83.5%
Mathematics	81.0%

FP8 scores match the FP32 reference weights within noise (< 0.3 points on average across categories), consistent with the calibrated, static-scale quantization recipe used here.

---

Relationship to Other Quatfit Mini Repositories

	Quatfit Mini (base)	Quatfit Mini FP8 (this repo)	Quatfit Mini GGUF
Format	safetensors, FP32	safetensors, FP8 + BF16	.gguf, F16 down to Q2_K
Best for	Fine-tuning, research, max fidelity	High-throughput GPU serving	Local / CPU / consumer-GPU inference
Runtime	🤗 Transformers	vLLM, TensorRT-LLM, SGLang, Transformers	llama.cpp and compatible runtimes
Hardware	Any CUDA GPU	Hopper / Ada / Blackwell (native FP8)	CPU or any GPU
Size	~32 GB	~11.5 GB	~3 GB – ~16.8 GB per quant

Full architecture details, training recipe, and benchmark methodology are documented in the Quatfit Mini Technical Report.

---

Responsible AI

Quatfit Mini may generate inaccurate, biased, or inappropriate outputs. FP8 quantization of the text decoder introduces a small amount of additional numerical noise relative to the FP32/BF16 reference, though evaluation shows this has negligible practical effect. For production deployments:

• Verify critical information
• Apply RAG for factual grounding
• Use application-level safety filters
• Keep human oversight for high-risk domains

---

Citation

@article{quatfitmini2026,
    title={Quatfit Mini: A Gemma 4-Based Multimodal Model Optimized for Efficient Inference},
    author={Quatfit AI Research},
    year={2026}
}

---

License

Apache License 2.0. See LICENSE for details.

---

🤗 FP8 Repository • 🧠 Base Model (FP32) • ⚙️ GGUF Builds • 📄 Technical Report