Scripts

These scripts made the FP8 copy of clef-flash in this repository, and compared it with the BF16 model.

File	What it does
`quantize_fp8.py`	Checks Cloudflare's files against their published hashes, quantizes with llm-compressor (`FP8_BLOCK`), and checks that every tensor except the FP8 weights is bit-identical to Cloudflare's. It writes `MANIFEST.json`.
`parity.py`	Sends the same 59 requests (54 text in 5 languages, 5 Wikimedia Commons images) to two model servers, and writes a short report: how far each probability moved, and how often the answer changed.
`run_all.sh`	Runs everything end to end: quantize, serve BF16, ask, serve FP8, ask, report. It is written for our GPU host (AMD MI210, Docker, the vLLM 0.22.1 ROCm image, a web proxy), so expect to change its settings.

Make the FP8 weights yourself

Use the same versions we used: llm-compressor 0.13.0, transformers 5.12.1 and torch 2.10.0 (llm-compressor brings compressed-tensors 0.18.0). A CPU is enough, with about 35 GB of RAM. Hide the GPU, because llm-compressor uses a GPU when it finds one.

hf download Cloudflare/clef-flash --revision 41318bf2106d135c7a92da745fec4b255279b4de --local-dir clef-flash
pip install llmcompressor==0.13.0 transformers==5.12.1 torch==2.10.0
CUDA_VISIBLE_DEVICES= HIP_VISIBLE_DEVICES= python quantize_fp8.py --src clef-flash --dst clef-flash-fp8

The script stops if any input file is not Cloudflare's file at that revision. At the end, it prints OK only if the 200 expected weights are FP8 and every other tensor is bit-identical to the original.

To compare your result with ours, compare the files in your MANIFEST.json with the one in this repository. script_sha256 in our MANIFEST.json is the SHA-256 of quantize_fp8.py in this folder. With the same versions, two runs on our machine gave identical bytes. We have not compared runs across different machines.

One deviation from a default llm-compressor run. For calibration, llm-compressor converts Qwen's norm weights to 1 + w and back in BF16, which rounds them (llmcompressor/modeling/offset_norm.py). FP8_BLOCK without calibration data does not use the norms, so quantize_fp8.py skips that step, and the norms stay exactly Cloudflare's.

Compare two servers

python parity.py run --label bf16 --out bf16.json --url http://localhost:8080/v1/models/clef-flash:predict
python parity.py run --label fp8 --out fp8.json --url http://localhost:8080/v1/models/clef-flash:predict
python parity.py compare bf16.json fp8.json --report parity-report.md

run expects the request and response format of Cloudflare's systemone behind a KServe :predict endpoint, as our model server provides. Add your own requests with --requests file.jsonl, with one request body on each line. The report says PASS when no probability moves by more than 0.03 and at least 99% of the answers are the same. Change these limits with --max-change and --min-agreement.

License

Apache-2.0, the same as the rest of this repository.