Clairic Flash โ€” W4A16 (AutoRound)

A 4-bit weight-only quantization of Cloudflare/clairic-flash, Cloudflare's 9B System One extraction model (base: Qwen3.5-9B). It keeps Cloudflare's typed answer heads and runtime, so the request and response format is unchanged: you name typed fields and get values, confidence scores, null probabilities and source spans back.

11 GB on disk instead of 19.6 GB, 11.1 GiB of weights in GPU memory instead of 18.2 GiB, and it runs on NVIDIA Ampere cards (RTX 3090 tested), which the original runtime does not support as shipped (see Patches).

All credit for the model goes to Cloudflare. This repository only changes the precision of some weights and adds two small runtime patches.

What was quantized

Part Precision
MLP (gate_proj, up_proj, down_proj) in all 32 decoder layers int4, group size 128, symmetric
Full attention (q/k/v/o_proj) in the 8 full-attention layers int4, group size 128, symmetric
Gated DeltaNet linear attention (24 layers) BF16 (kept: these projections are the most rounding-sensitive part of a Qwen3.5-style stack, and the runtime reads in_proj_qkv.weight directly)
Embeddings, lm_head BF16 (the runtime reads lm_head.weight for its readout)
Vision encoder BF16
Typed answer heads (clair_head.safetensors) BF16, copied unchanged
mtp.* tensors BF16, carried over unchanged (model_extra_tensors.safetensors; not used by the runtime)
  • Tool: AutoRound 0.16.1, 100 iterations, 256 calibration samples ร— 2048 tokens (text only, not extraction-specific), auto_round:auto_gptq packing.
  • Text-only calibration through the full model; the vision tower is not quantized.

Results

A sanity check, not a benchmark: 13 extraction requests with 49 scored fields, run through the model's own load_release_model + systemone on one RTX 3090. One request is the example from Cloudflare's model card; the other 12 are synthetic (support ticket, lease, shipping notice, tax computation, relative date, meeting email, bank statement, a request with no answer present, estimate/forecast, two receipt images, one invoice image).

BF16 (original weights) W4A16 (this repo)
Fields correct vs. ground truth 46 / 49 47 / 49
Same answer as BF16, on the fields BF16 got right โ€” 46 / 46
Weights in GPU memory 18.2 GiB 11.1 GiB
Peak GPU memory during the test 21.2 GiB 14.2 GiB
All 13 requests, warm (RTX 3090) 62.0 s 32.6 s (1.9ร— faster)

Where the two builds differ:

  • Two arithmetic fields are wrong at both precisions, with very low confidence (โ‰ค 0.18): a three-term total (subtotal + tax + shipping) and a due date computed as "issue date + net 30". In the same tests the model extracted the inputs (subtotal, tax rate, "30 days") correctly, so extract the parts and do the arithmetic in code.
  • On an invoice image with no "From:" label, BF16 returned null and W4A16 returned the correct vendor.
  • Estimates and forecasts shifted by about 1% (e.g. 50.96 โ†’ 50.39).

Absolute latencies above are slower than Cloudflare's published numbers because of the fallback kernels on this setup (see Limitations); the BF16/W4A16 ratio is the useful part.

Usage

Python 3.11+; install the packages in requirements.txt plus auto-round for the int4 kernels:

pip install -r requirements.txt
pip install "auto-round==0.16.1"
import sys
from huggingface_hub import snapshot_download

path = snapshot_download("stuchapin/Clairic-Flash-W4A16-AutoRound")
sys.path.insert(0, path)
from clairic_model import load_release_model, systemone

model, processor = load_release_model(path, device="cuda")

response = systemone(
    model,
    processor,
    {
        "state": (
            "Invoice from Northwind Supply. Issued 2026-09-02. "
            "Items: Pens; Paper. Total: $1080.00."
        ),
        "fields": {
            "vendor": {"type": "text", "instructions": "The invoice vendor."},
            "total": {"type": "number", "decimals": 2},
            "issued": {"type": "date"},
            "items": {"type": "list", "items": "text",
                      "instructions": "The purchased item names, in order."},
        },
    },
)
print(response["answers"])

The full request and response format (field types, images, PDFs, estimates, forecasts, scores) is documented on the original model card.

Patches

clairic_runtime/math.py is modified from Cloudflare's original (marked at the top of the file). Everything else in clairic_runtime/ and clairic_model.py is unchanged.

  1. Ampere support. _matmul sends every contiguous BF16 GEMM to DeepGEMM, which raises Unsupported architecture below compute capability 9.0, so the original runtime fails on RTX 30-series cards even with the BF16 weights. DeepGEMM is now used only on compute capability โ‰ฅ 9; older GPUs use the runtime's own Triton matmul fallback. Hopper and newer take the original path.
  2. Quantized layers. install() read gate_proj.weight and q_proj.weight only to decide the dtype of its fused paths. Quantized linear layers have no .weight, so loading failed with AttributeError. A small _dtype_ref() helper falls back to the embedding weight's dtype.

Limitations

  • First run is slow. AutoRound's tritonv2_zp kernels compile on first use (about 11 minutes in our first run, 20โ€“135 s per request). Triton caches them on disk, so later runs are fast.
  • AutoRound picks its Triton backend here and suggests gptqmodel>=2.0 for a faster one; not tested.
  • Tested without causal_conv1d, so linear attention used the PyTorch fallback in both builds.
  • On Ampere the runtime uses its Triton GEMM fallback instead of DeepGEMM; answers on Hopper hardware may differ slightly. Not tested on Hopper.
  • 13 requests and 3 images is a small test. Validate on your own documents and set your own confidence and p_null thresholds, as Cloudflare's card recommends.

License

Apache-2.0, the same as the original model. See LICENSE. Model weights, heads and runtime are ยฉ Cloudflare, Inc. and the Qwen team, under that licence; this repository modifies weight precision and clairic_runtime/math.py only.

Downloads last month
-
Safetensors
Model size
5B params
Tensor type
F32
ยท
I32
ยท
BF16
ยท
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for stuchapin/Clairic-Flash-W4A16-AutoRound

Finetuned
Qwen/Qwen3.5-9B
Quantized
(4)
this model