Instructions to use stuchapin/Clairic-W4A16-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use stuchapin/Clairic-W4A16-AutoRound with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="stuchapin/Clairic-W4A16-AutoRound") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("stuchapin/Clairic-W4A16-AutoRound") model = AutoModelForMultimodalLM.from_pretrained("stuchapin/Clairic-W4A16-AutoRound", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use stuchapin/Clairic-W4A16-AutoRound with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "stuchapin/Clairic-W4A16-AutoRound" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "stuchapin/Clairic-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/stuchapin/Clairic-W4A16-AutoRound
- SGLang
How to use stuchapin/Clairic-W4A16-AutoRound with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "stuchapin/Clairic-W4A16-AutoRound" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "stuchapin/Clairic-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "stuchapin/Clairic-W4A16-AutoRound" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "stuchapin/Clairic-W4A16-AutoRound", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use stuchapin/Clairic-W4A16-AutoRound with Docker Model Runner:
docker model run hf.co/stuchapin/Clairic-W4A16-AutoRound
Clairic (27B) โ W4A16 (AutoRound)
A 4-bit weight-only quantization of Cloudflare/clairic, Cloudflare's 27B System One extraction model (base: Qwen3.8-27B). It keeps Cloudflare's typed answer heads and runtime, so the request and response format is unchanged: you name typed fields and get values, confidence scores, null probabilities and source spans back.
27 GB on disk instead of 53 GB, and it runs split across two 24 GB NVIDIA Ampere cards (2ร RTX 3090 tested), which the original runtime does not support as shipped (see Patches).
For a single-card option, see the 9B sibling stuchapin/Clairic-Flash-W4A16-AutoRound (about 14 GiB peak).
All credit for the model goes to Cloudflare. This repository only changes the precision of some weights and adds small runtime patches.
Hardware
- About 25.3 GiB of weights and 30.7 GiB peak GPU memory in our test, so it does not fit one 24 GB card. Tested on two RTX 3090s: 16.4 GiB peak on GPU 0, 14.3 GiB on GPU 1.
- A single GPU with more than about 32 GB should also work (not tested).
What was quantized
| Part | Precision |
|---|---|
MLP (gate_proj, up_proj, down_proj) in all 64 decoder layers |
int4, group size 128, symmetric |
Full attention (q/k/v/o_proj) in the 16 full-attention layers |
int4, group size 128, symmetric |
| Gated DeltaNet linear attention (48 layers) | BF16 (the most rounding-sensitive part of a Qwen3.5-style stack; the runtime also reads in_proj_qkv.weight directly) |
Embeddings, lm_head |
BF16 (the runtime reads lm_head.weight for its readout) |
| Vision encoder | BF16 |
Typed answer heads (clair_head.safetensors) |
BF16, copied unchanged |
mtp.* tensors |
BF16, carried over unchanged (not used by the runtime) |
- 256 of 607 linear layers are int4.
- Tool: AutoRound 0.16.1, 100 iterations, 256 calibration
samples ร 2048 tokens (text only, not extraction-specific),
auto_round:auto_gptqpacking. About 75 minutes of tuning on one RTX 3090.
Results
A sanity check, not a benchmark: 13 extraction requests with 49 scored fields, run through the
model's own load_release_model + systemone split over two RTX 3090s. One request is the example
from Cloudflare's model card; the other 12 are synthetic (support ticket, lease, shipping notice,
tax computation, relative date, meeting email, bank statement, a request with no answer present,
estimate/forecast, two receipt images, one invoice image). Two passes; the second gave identical
answers to the first on all 13 requests.
No BF16 reference for the 27B: the 53 GB original does not fit 48 GB of VRAM, so the quant is compared against ground truth only. For comparison, the same test on Clairic Flash (9B):
| Clairic Flash BF16 | Clairic Flash W4A16 | Clairic 27B W4A16 (this repo) | |
|---|---|---|---|
| Fields correct vs. ground truth | 46 / 49 | 47 / 49 | 47 / 49 |
| GPUs | 1ร RTX 3090 | 1ร RTX 3090 | 2ร RTX 3090 |
| Peak GPU memory | 21.2 GiB | 14.2 GiB | 30.7 GiB (16.4 + 14.3) |
| All 13 requests, warm | 62.0 s | 32.6 s | 75.0 s |
The two fields the 27B gets wrong are the same two multi-step arithmetic fields that Flash gets wrong, both with low confidence:
subtotal + 8.5% tax + shipping(truth 272.40): returned 266.40, confidence 0.18. The single-step tax (20.40) was right at confidence 0.92.- due date as "issue date + net 30" (truth 2026-04-14): returned the issue date 2026-03-15, confidence 0.28.
Extract the parts and do the arithmetic in code. Unlike Flash BF16, the 27B also read the vendor of an invoice image with no "From:" label (confidence 0.999).
The 27B is slower than Flash here: it is three times the size and runs without CUDA graphs when split (see Patches). Absolute latencies are slower than Cloudflare's published numbers because of the fallback kernels on this setup (see Limitations).
Usage
Python 3.11+; install the packages in requirements.txt plus auto-round for the int4 kernels:
pip install -r requirements.txt
pip install "auto-round==0.16.1"
Split over two GPUs. Keep embeddings, vision, final norm and lm_head on GPU 0, where the runtime
places the typed heads; put the first 22 decoder layers on GPU 0 and the rest on GPU 1:
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("stuchapin/Clairic-W4A16-AutoRound")
sys.path.insert(0, path)
from clairic_model import load_release_model, systemone
n_layers, on_gpu0 = 64, 22
device_map = {
"model.visual": 0,
"model.language_model.embed_tokens": 0,
"model.language_model.rotary_emb": 0,
"model.language_model.norm": 0,
"lm_head": 0,
**{f"model.language_model.layers.{i}": 0 if i < on_gpu0 else 1 for i in range(n_layers)},
}
model, processor = load_release_model(path, device="cuda:0", device_map=device_map)
response = systemone(
model,
processor,
{
"state": (
"Invoice from Northwind Supply. Issued 2026-09-02. "
"Items: Pens; Paper. Total: $1080.00."
),
"fields": {
"vendor": {"type": "text", "instructions": "The invoice vendor."},
"total": {"type": "number", "decimals": 2},
"issued": {"type": "date"},
"items": {"type": "list", "items": "text",
"instructions": "The purchased item names, in order."},
},
},
)
print(response["answers"])
On a single GPU with enough memory, load_release_model(path, device="cuda") is enough.
The full request and response format (field types, images, PDFs, estimates, forecasts, scores) is documented on the original model card.
Patches
clairic_runtime/math.py and clairic_runtime/native.py are modified from Cloudflare's original
(marked at the top of each file). Everything else in clairic_runtime/ and clairic_model.py is
unchanged. On a single GPU the multi-GPU changes do nothing.
- Ampere support (
math.py)._matmulsends every contiguous BF16 GEMM to DeepGEMM, which raisesUnsupported architecturebelow compute capability 9.0. DeepGEMM is now used only on compute capability โฅ 9; older GPUs use the runtime's own Triton matmul fallback. - Quantized layers (
math.py).install()readgate_proj.weightandq_proj.weightonly for their dtype; quantized linear layers have no.weight. A_dtype_ref()helper falls back to the embedding weight's dtype. - Multi-GPU split (
math.py). The runtime replaces each layer'sforwardwith fused kernels, which also removes accelerate's hook that moves tensors between GPUs, so a split model failed withMismatched device index. When the model spans several GPUs, each replaced forward now moves its tensor inputs to that layer's GPU and makes it the current device first. - No CUDA graphs when split (
native.py). Per-layer CUDA-graph capture assumes one device and ran GPU 0 out of memory on a 2 ร 24 GB split. It is skipped when the model spans several GPUs; the decoder runs eagerly.
Limitations
- First run is slow. AutoRound's
tritonv2_zpkernels compile on first use. Triton caches them on disk, so later runs are fast. - AutoRound picks its Triton backend here and suggests
gptqmodel>=2.0for a faster one; not tested. - Tested without
causal_conv1d, so linear attention used the PyTorch fallback. - No BF16 comparison for the 27B (it does not fit the test machine); quality is measured against ground truth only.
- Not tested on Hopper or on a single large GPU. On Ampere the runtime uses its Triton GEMM fallback instead of DeepGEMM, so answers on Hopper may differ slightly.
- 13 requests and 3 images is a small test. Validate on your own documents and set your own
confidence and
p_nullthresholds, as Cloudflare's card recommends.
License
Apache-2.0, the same as the original model. See LICENSE. Model weights, heads and runtime are
ยฉ Cloudflare, Inc. and the Qwen team, under that licence; this repository modifies weight
precision and clairic_runtime/math.py and clairic_runtime/native.py only.
- Downloads last month
- -