File size: 3,070 Bytes
7850e45 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | # Scripts
These scripts made the FP8 copy of clef-flash in this repository, and compared it with the BF16 model.
| File | What it does |
|---|---|
| `quantize_fp8.py` | Checks Cloudflare's files against their published hashes, quantizes with llm-compressor (`FP8_BLOCK`), and checks that every tensor except the FP8 weights is bit-identical to Cloudflare's. It writes `MANIFEST.json`. |
| `parity.py` | Sends the same 59 requests (54 text in 5 languages, 5 Wikimedia Commons images) to two model servers, and writes a short report: how far each probability moved, and how often the answer changed. |
| `run_all.sh` | Runs everything end to end: quantize, serve BF16, ask, serve FP8, ask, report. It is written for our GPU host (AMD MI210, Docker, the vLLM 0.22.1 ROCm image, a web proxy), so expect to change its settings. |
## Make the FP8 weights yourself
Use the same versions we used: llm-compressor 0.13.0, transformers 5.12.1 and torch 2.10.0 (llm-compressor brings compressed-tensors 0.18.0). A CPU is enough, with about 35 GB of RAM. Hide the GPU, because llm-compressor uses a GPU when it finds one.
```bash
hf download Cloudflare/clef-flash --revision 41318bf2106d135c7a92da745fec4b255279b4de --local-dir clef-flash
pip install llmcompressor==0.13.0 transformers==5.12.1 torch==2.10.0
CUDA_VISIBLE_DEVICES= HIP_VISIBLE_DEVICES= python quantize_fp8.py --src clef-flash --dst clef-flash-fp8
```
The script stops if any input file is not Cloudflare's file at that revision. At the end, it prints `OK` only if the 200 expected weights are FP8 and every other tensor is bit-identical to the original.
To compare your result with ours, compare the `files` in your `MANIFEST.json` with the one in this repository. `script_sha256` in our `MANIFEST.json` is the SHA-256 of `quantize_fp8.py` in this folder. With the same versions, two runs on our machine gave identical bytes. We have not compared runs across different machines.
**One deviation from a default llm-compressor run.** For calibration, llm-compressor converts Qwen's norm weights to `1 + w` and back in BF16, which rounds them (`llmcompressor/modeling/offset_norm.py`). FP8_BLOCK without calibration data does not use the norms, so `quantize_fp8.py` skips that step, and the norms stay exactly Cloudflare's.
## Compare two servers
```bash
python parity.py run --label bf16 --out bf16.json --url http://localhost:8080/v1/models/clef-flash:predict
python parity.py run --label fp8 --out fp8.json --url http://localhost:8080/v1/models/clef-flash:predict
python parity.py compare bf16.json fp8.json --report parity-report.md
```
`run` expects the request and response format of Cloudflare's `systemone` behind a KServe `:predict` endpoint, as our model server provides. Add your own requests with `--requests file.jsonl`, with one request body on each line. The report says PASS when no probability moves by more than 0.03 and at least 99% of the answers are the same. Change these limits with `--max-change` and `--min-agreement`.
## License
Apache-2.0, the same as the rest of this repository.
|