README.md

---
license: apache-2.0
base_model: Cloudflare/clef-flash
base_model_relation: quantized
library_name: transformers
tags:
- clef
- nvfp4
- compressed-tensors
- llmcompressor
- quantized
---

# clef-flash-NVFP4

This is an NVFP4 (W4A4) quantization of [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash)
at revision `17f0b0ad`.

- **Quantized:** the 96 text MLP projections (`gate/up/down_proj`).
- **Kept in BF16:** the gated-deltanet and full-attention layers, `lm_head` (the joint head
  reads its rows), the vision tower and the joint head.
- **Size:** 12.1 GB, versus 19.1 GB for BF16 (decimal GB).

## Method

- **Tool:** llmcompressor 0.13.0 `oneshot` with `QuantizationModifier(scheme="NVFP4")`.
  Weights use FP4 with group size 16 and FP8 scales.
- **Calibration:** per-tensor input-activation global scales, taken from 248 records
  (3.4M tokens, up to 87K tokens long) built from legal, news, science, books, tools,
  routing and tax sources.
- **Decontamination:** the calibration data was checked against the evaluation sets.
- **Stack:** torch 2.13 (cu132), transformers 5.14.1, compressed-tensors 0.18.0,
  on an RTX PRO 6000 (SM120).

## Parity vs BF16

The held-out set has 256 records, one question each: LongBench v2 (128, 4 options),
banking77 (64, 77 options) and LEDGAR (64, 100 options). Inputs run from 1.7K to
237K tokens. Each model gets one forward pass per record through the joint head.

| | BF16 | **NVFP4 (this repo)** | NVFP4 incl. attention + deltanet (not released) |
|---|---|---|---|
| Accuracy | 62.1% | **63.7%** | 60.9% |
| Δ accuracy, 95% CI (paired bootstrap) | | **+1.6 pt [−1.2, +4.3]** | −1.2 pt [−5.1, +2.7] |
| Top-1 agreement with BF16 | | 0.926 [0.887, 0.952] | 0.883 [0.838, 0.917] |
| Agreement where BF16 margin ≥ 0.5 (n=156) | | **1.000** | 0.994 |
| Mean KL(BF16 ‖ quant) | | 0.020 | 0.048 |
| Size (GB) | 19.1 | 12.1 | 9.1 |

Release gates: the accuracy delta's 95% lower bound must be ≥ −2 pt, and agreement on
confident decisions must be ≥ 0.98. This model passes both. The variant that also
quantizes attention and deltanet fails the accuracy gate, losing mostly at 8K–32K
tokens (−6.1 pt).

Every disagreement with BF16 is on a question where BF16's own margin is below 0.5.

These gates were set after the first full report. Under the originally proposed gates
(overall agreement ≥ 0.95, every length band ≥ 0.90) this model fails at 0.926 overall
and 0.878 on the 8K–32K band. Those gates were dropped because band slices of n=26–53
are too small to gate on.

Per-band, per-cohort and per-margin breakdowns are in [`metrics/`](metrics/).

## Usage

The repo uses the same code as the base model (`joint_schema_model.py`). Load the
backbone decompressed: the joint head calls `model.language_model` directly, so the
packed weights have to be expanded at load time.

```python
import json, torch, joint_schema_model as jsm
from safetensors.torch import load_file
from transformers import Qwen3_5ForConditionalGeneration, CompressedTensorsConfig

d = "<local path to this repo>"
backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
    d, dtype=torch.bfloat16, device_map={"": "cuda:0"},
    quantization_config=CompressedTensorsConfig(run_compressed=False))
head = jsm.JointSchemaHead(**json.load(open(f"{d}/joint_head_config.json")))
head.load_state_dict(load_file(f"{d}/joint_head.safetensors"))
model = jsm.ClefModel(backbone, head.to("cuda:0", torch.bfloat16)).eval()
```

The metrics above were measured with this path, which simulates W4A4 numerically in
eager PyTorch. Native FP4 kernel serving (e.g. vLLM on Blackwell) has not been
evaluated.

## License

Apache-2.0, inherited from the base model. This is a quantized derivative of
Cloudflare/clef-flash. The weights were not otherwise modified.