Clairic (27B) โ€” W4A16 (AutoRound)

A 4-bit weight-only quantization of Cloudflare/clairic, Cloudflare's 27B System One extraction model (base: Qwen3.8-27B). It keeps Cloudflare's typed answer heads and runtime, so the request and response format is unchanged: you name typed fields and get values, confidence scores, null probabilities and source spans back.

27 GB on disk instead of 53 GB, and it runs split across two 24 GB NVIDIA Ampere cards (2ร— RTX 3090 tested), which the original runtime does not support as shipped (see Patches).

For a single-card option, see the 9B sibling stuchapin/Clairic-Flash-W4A16-AutoRound (about 14 GiB peak).

All credit for the model goes to Cloudflare. This repository only changes the precision of some weights and adds small runtime patches.

Hardware

  • About 25.3 GiB of weights and 30.7 GiB peak GPU memory in our test, so it does not fit one 24 GB card. Tested on two RTX 3090s: 16.4 GiB peak on GPU 0, 14.3 GiB on GPU 1.
  • A single GPU with more than about 32 GB should also work (not tested).

What was quantized

Part Precision
MLP (gate_proj, up_proj, down_proj) in all 64 decoder layers int4, group size 128, symmetric
Full attention (q/k/v/o_proj) in the 16 full-attention layers int4, group size 128, symmetric
Gated DeltaNet linear attention (48 layers) BF16 (the most rounding-sensitive part of a Qwen3.5-style stack; the runtime also reads in_proj_qkv.weight directly)
Embeddings, lm_head BF16 (the runtime reads lm_head.weight for its readout)
Vision encoder BF16
Typed answer heads (clair_head.safetensors) BF16, copied unchanged
mtp.* tensors BF16, carried over unchanged (not used by the runtime)
  • 256 of 607 linear layers are int4.
  • Tool: AutoRound 0.16.1, 100 iterations, 256 calibration samples ร— 2048 tokens (text only, not extraction-specific), auto_round:auto_gptq packing. About 75 minutes of tuning on one RTX 3090.

Results

A sanity check, not a benchmark: 13 extraction requests with 49 scored fields, run through the model's own load_release_model + systemone split over two RTX 3090s. One request is the example from Cloudflare's model card; the other 12 are synthetic (support ticket, lease, shipping notice, tax computation, relative date, meeting email, bank statement, a request with no answer present, estimate/forecast, two receipt images, one invoice image). Two passes; the second gave identical answers to the first on all 13 requests.

No BF16 reference for the 27B: the 53 GB original does not fit 48 GB of VRAM, so the quant is compared against ground truth only. For comparison, the same test on Clairic Flash (9B):

Clairic Flash BF16 Clairic Flash W4A16 Clairic 27B W4A16 (this repo)
Fields correct vs. ground truth 46 / 49 47 / 49 47 / 49
GPUs 1ร— RTX 3090 1ร— RTX 3090 2ร— RTX 3090
Peak GPU memory 21.2 GiB 14.2 GiB 30.7 GiB (16.4 + 14.3)
All 13 requests, warm 62.0 s 32.6 s 75.0 s

The two fields the 27B gets wrong are the same two multi-step arithmetic fields that Flash gets wrong, both with low confidence:

  • subtotal + 8.5% tax + shipping (truth 272.40): returned 266.40, confidence 0.18. The single-step tax (20.40) was right at confidence 0.92.
  • due date as "issue date + net 30" (truth 2026-04-14): returned the issue date 2026-03-15, confidence 0.28.

Extract the parts and do the arithmetic in code. Unlike Flash BF16, the 27B also read the vendor of an invoice image with no "From:" label (confidence 0.999).

The 27B is slower than Flash here: it is three times the size and runs without CUDA graphs when split (see Patches). Absolute latencies are slower than Cloudflare's published numbers because of the fallback kernels on this setup (see Limitations).

Usage

Python 3.11+; install the packages in requirements.txt plus auto-round for the int4 kernels:

pip install -r requirements.txt
pip install "auto-round==0.16.1"

Split over two GPUs. Keep embeddings, vision, final norm and lm_head on GPU 0, where the runtime places the typed heads; put the first 22 decoder layers on GPU 0 and the rest on GPU 1:

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("stuchapin/Clairic-W4A16-AutoRound")
sys.path.insert(0, path)
from clairic_model import load_release_model, systemone

n_layers, on_gpu0 = 64, 22
device_map = {
    "model.visual": 0,
    "model.language_model.embed_tokens": 0,
    "model.language_model.rotary_emb": 0,
    "model.language_model.norm": 0,
    "lm_head": 0,
    **{f"model.language_model.layers.{i}": 0 if i < on_gpu0 else 1 for i in range(n_layers)},
}
model, processor = load_release_model(path, device="cuda:0", device_map=device_map)

response = systemone(
    model,
    processor,
    {
        "state": (
            "Invoice from Northwind Supply. Issued 2026-09-02. "
            "Items: Pens; Paper. Total: $1080.00."
        ),
        "fields": {
            "vendor": {"type": "text", "instructions": "The invoice vendor."},
            "total": {"type": "number", "decimals": 2},
            "issued": {"type": "date"},
            "items": {"type": "list", "items": "text",
                      "instructions": "The purchased item names, in order."},
        },
    },
)
print(response["answers"])

On a single GPU with enough memory, load_release_model(path, device="cuda") is enough.

The full request and response format (field types, images, PDFs, estimates, forecasts, scores) is documented on the original model card.

Patches

clairic_runtime/math.py and clairic_runtime/native.py are modified from Cloudflare's original (marked at the top of each file). Everything else in clairic_runtime/ and clairic_model.py is unchanged. On a single GPU the multi-GPU changes do nothing.

  1. Ampere support (math.py). _matmul sends every contiguous BF16 GEMM to DeepGEMM, which raises Unsupported architecture below compute capability 9.0. DeepGEMM is now used only on compute capability โ‰ฅ 9; older GPUs use the runtime's own Triton matmul fallback.
  2. Quantized layers (math.py). install() read gate_proj.weight and q_proj.weight only for their dtype; quantized linear layers have no .weight. A _dtype_ref() helper falls back to the embedding weight's dtype.
  3. Multi-GPU split (math.py). The runtime replaces each layer's forward with fused kernels, which also removes accelerate's hook that moves tensors between GPUs, so a split model failed with Mismatched device index. When the model spans several GPUs, each replaced forward now moves its tensor inputs to that layer's GPU and makes it the current device first.
  4. No CUDA graphs when split (native.py). Per-layer CUDA-graph capture assumes one device and ran GPU 0 out of memory on a 2 ร— 24 GB split. It is skipped when the model spans several GPUs; the decoder runs eagerly.

Limitations

  • First run is slow. AutoRound's tritonv2_zp kernels compile on first use. Triton caches them on disk, so later runs are fast.
  • AutoRound picks its Triton backend here and suggests gptqmodel>=2.0 for a faster one; not tested.
  • Tested without causal_conv1d, so linear attention used the PyTorch fallback.
  • No BF16 comparison for the 27B (it does not fit the test machine); quality is measured against ground truth only.
  • Not tested on Hopper or on a single large GPU. On Ampere the runtime uses its Triton GEMM fallback instead of DeepGEMM, so answers on Hopper may differ slightly.
  • 13 requests and 3 images is a small test. Validate on your own documents and set your own confidence and p_null thresholds, as Cloudflare's card recommends.

License

Apache-2.0, the same as the original model. See LICENSE. Model weights, heads and runtime are ยฉ Cloudflare, Inc. and the Qwen team, under that licence; this repository modifies weight precision and clairic_runtime/math.py and clairic_runtime/native.py only.

Downloads last month
-
Safetensors
Model size
11B params
Tensor type
I32
ยท
BF16
ยท
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for stuchapin/Clairic-W4A16-AutoRound

Base model

Qwen/Qwen3.8-27B
Quantized
(4)
this model

Collection including stuchapin/Clairic-W4A16-AutoRound