Instructions to use stuchapin/Clairic-Flash-W4A16-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use stuchapin/Clairic-Flash-W4A16-AutoRound with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="stuchapin/Clairic-Flash-W4A16-AutoRound") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("stuchapin/Clairic-Flash-W4A16-AutoRound") model = AutoModelForMultimodalLM.from_pretrained("stuchapin/Clairic-Flash-W4A16-AutoRound", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use stuchapin/Clairic-Flash-W4A16-AutoRound with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "stuchapin/Clairic-Flash-W4A16-AutoRound" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "stuchapin/Clairic-Flash-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/stuchapin/Clairic-Flash-W4A16-AutoRound
- SGLang
How to use stuchapin/Clairic-Flash-W4A16-AutoRound with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "stuchapin/Clairic-Flash-W4A16-AutoRound" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "stuchapin/Clairic-Flash-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "stuchapin/Clairic-Flash-W4A16-AutoRound" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "stuchapin/Clairic-Flash-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use stuchapin/Clairic-Flash-W4A16-AutoRound with Docker Model Runner:
docker model run hf.co/stuchapin/Clairic-Flash-W4A16-AutoRound
Clairic Flash โ W4A16 (AutoRound)
A 4-bit weight-only quantization of Cloudflare/clairic-flash, Cloudflare's 9B System One extraction model (base: Qwen3.5-9B). It keeps Cloudflare's typed answer heads and runtime, so the request and response format is unchanged: you name typed fields and get values, confidence scores, null probabilities and source spans back.
11 GB on disk instead of 19.6 GB, 11.1 GiB of weights in GPU memory instead of 18.2 GiB, and it runs on NVIDIA Ampere cards (RTX 3090 tested), which the original runtime does not support as shipped (see Patches).
All credit for the model goes to Cloudflare. This repository only changes the precision of some weights and adds two small runtime patches.
What was quantized
| Part | Precision |
|---|---|
MLP (gate_proj, up_proj, down_proj) in all 32 decoder layers |
int4, group size 128, symmetric |
Full attention (q/k/v/o_proj) in the 8 full-attention layers |
int4, group size 128, symmetric |
| Gated DeltaNet linear attention (24 layers) | BF16 (kept: these projections are the most rounding-sensitive part of a Qwen3.5-style stack, and the runtime reads in_proj_qkv.weight directly) |
Embeddings, lm_head |
BF16 (the runtime reads lm_head.weight for its readout) |
| Vision encoder | BF16 |
Typed answer heads (clair_head.safetensors) |
BF16, copied unchanged |
mtp.* tensors |
BF16, carried over unchanged (model_extra_tensors.safetensors; not used by the runtime) |
- Tool: AutoRound 0.16.1, 100 iterations, 256 calibration
samples ร 2048 tokens (text only, not extraction-specific),
auto_round:auto_gptqpacking. - Text-only calibration through the full model; the vision tower is not quantized.
Results
A sanity check, not a benchmark: 13 extraction requests with 49 scored fields, run through the
model's own load_release_model + systemone on one RTX 3090. One request is the example from
Cloudflare's model card; the other 12 are synthetic (support ticket, lease, shipping notice, tax
computation, relative date, meeting email, bank statement, a request with no answer present,
estimate/forecast, two receipt images, one invoice image).
| BF16 (original weights) | W4A16 (this repo) | |
|---|---|---|
| Fields correct vs. ground truth | 46 / 49 | 47 / 49 |
| Same answer as BF16, on the fields BF16 got right | โ | 46 / 46 |
| Weights in GPU memory | 18.2 GiB | 11.1 GiB |
| Peak GPU memory during the test | 21.2 GiB | 14.2 GiB |
| All 13 requests, warm (RTX 3090) | 62.0 s | 32.6 s (1.9ร faster) |
Where the two builds differ:
- Two arithmetic fields are wrong at both precisions, with very low confidence (โค 0.18): a three-term total (subtotal + tax + shipping) and a due date computed as "issue date + net 30". In the same tests the model extracted the inputs (subtotal, tax rate, "30 days") correctly, so extract the parts and do the arithmetic in code.
- On an invoice image with no "From:" label, BF16 returned null and W4A16 returned the correct vendor.
- Estimates and forecasts shifted by about 1% (e.g. 50.96 โ 50.39).
Absolute latencies above are slower than Cloudflare's published numbers because of the fallback kernels on this setup (see Limitations); the BF16/W4A16 ratio is the useful part.
Usage
Python 3.11+; install the packages in requirements.txt plus auto-round for the int4 kernels:
pip install -r requirements.txt
pip install "auto-round==0.16.1"
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("stuchapin/Clairic-Flash-W4A16-AutoRound")
sys.path.insert(0, path)
from clairic_model import load_release_model, systemone
model, processor = load_release_model(path, device="cuda")
response = systemone(
model,
processor,
{
"state": (
"Invoice from Northwind Supply. Issued 2026-09-02. "
"Items: Pens; Paper. Total: $1080.00."
),
"fields": {
"vendor": {"type": "text", "instructions": "The invoice vendor."},
"total": {"type": "number", "decimals": 2},
"issued": {"type": "date"},
"items": {"type": "list", "items": "text",
"instructions": "The purchased item names, in order."},
},
},
)
print(response["answers"])
The full request and response format (field types, images, PDFs, estimates, forecasts, scores) is documented on the original model card.
Patches
clairic_runtime/math.py is modified from Cloudflare's original (marked at the top of the file).
Everything else in clairic_runtime/ and clairic_model.py is unchanged.
- Ampere support.
_matmulsends every contiguous BF16 GEMM to DeepGEMM, which raisesUnsupported architecturebelow compute capability 9.0, so the original runtime fails on RTX 30-series cards even with the BF16 weights. DeepGEMM is now used only on compute capability โฅ 9; older GPUs use the runtime's own Triton matmul fallback. Hopper and newer take the original path. - Quantized layers.
install()readgate_proj.weightandq_proj.weightonly to decide the dtype of its fused paths. Quantized linear layers have no.weight, so loading failed withAttributeError. A small_dtype_ref()helper falls back to the embedding weight's dtype.
Limitations
- First run is slow. AutoRound's
tritonv2_zpkernels compile on first use (about 11 minutes in our first run, 20โ135 s per request). Triton caches them on disk, so later runs are fast. - AutoRound picks its Triton backend here and suggests
gptqmodel>=2.0for a faster one; not tested. - Tested without
causal_conv1d, so linear attention used the PyTorch fallback in both builds. - On Ampere the runtime uses its Triton GEMM fallback instead of DeepGEMM; answers on Hopper hardware may differ slightly. Not tested on Hopper.
- 13 requests and 3 images is a small test. Validate on your own documents and set your own
confidence and
p_nullthresholds, as Cloudflare's card recommends.
License
Apache-2.0, the same as the original model. See LICENSE. Model weights, heads and runtime are
ยฉ Cloudflare, Inc. and the Qwen team, under that licence; this repository modifies weight
precision and clairic_runtime/math.py only.
- Downloads last month
- -