| # Scripts |
|
|
| These scripts made the FP8 copy of clef-flash in this repository, and compared it with the BF16 model. |
|
|
| | File | What it does | |
| |---|---| |
| | `quantize_fp8.py` | Checks Cloudflare's files against their published hashes, quantizes with llm-compressor (`FP8_BLOCK`), and checks that every tensor except the FP8 weights is bit-identical to Cloudflare's. It writes `MANIFEST.json`. | |
| | `parity.py` | Sends the same 59 requests (54 text in 5 languages, 5 Wikimedia Commons images) to two model servers, and writes a short report: how far each probability moved, and how often the answer changed. | |
| | `run_all.sh` | Runs everything end to end: quantize, serve BF16, ask, serve FP8, ask, report. It is written for our GPU host (AMD MI210, Docker, the vLLM 0.22.1 ROCm image, a web proxy), so expect to change its settings. | |
|
|
| ## Make the FP8 weights yourself |
|
|
| Use the same versions we used: llm-compressor 0.13.0, transformers 5.12.1 and torch 2.10.0 (llm-compressor brings compressed-tensors 0.18.0). A CPU is enough, with about 35 GB of RAM. Hide the GPU, because llm-compressor uses a GPU when it finds one. |
|
|
| ```bash |
| hf download Cloudflare/clef-flash --revision 41318bf2106d135c7a92da745fec4b255279b4de --local-dir clef-flash |
| pip install llmcompressor==0.13.0 transformers==5.12.1 torch==2.10.0 |
| CUDA_VISIBLE_DEVICES= HIP_VISIBLE_DEVICES= python quantize_fp8.py --src clef-flash --dst clef-flash-fp8 |
| ``` |
|
|
| The script stops if any input file is not Cloudflare's file at that revision. At the end, it prints `OK` only if the 200 expected weights are FP8 and every other tensor is bit-identical to the original. |
|
|
| To compare your result with ours, compare the `files` in your `MANIFEST.json` with the one in this repository. `script_sha256` in our `MANIFEST.json` is the SHA-256 of `quantize_fp8.py` in this folder. With the same versions, two runs on our machine gave identical bytes. We have not compared runs across different machines. |
|
|
| **One deviation from a default llm-compressor run.** For calibration, llm-compressor converts Qwen's norm weights to `1 + w` and back in BF16, which rounds them (`llmcompressor/modeling/offset_norm.py`). FP8_BLOCK without calibration data does not use the norms, so `quantize_fp8.py` skips that step, and the norms stay exactly Cloudflare's. |
|
|
| ## Compare two servers |
|
|
| ```bash |
| python parity.py run --label bf16 --out bf16.json --url http://localhost:8080/v1/models/clef-flash:predict |
| python parity.py run --label fp8 --out fp8.json --url http://localhost:8080/v1/models/clef-flash:predict |
| python parity.py compare bf16.json fp8.json --report parity-report.md |
| ``` |
|
|
| `run` expects the request and response format of Cloudflare's `systemone` behind a KServe `:predict` endpoint, as our model server provides. Add your own requests with `--requests file.jsonl`, with one request body on each line. The report says PASS when no probability moves by more than 0.03 and at least 99% of the answers are the same. Change these limits with `--max-change` and `--min-agreement`. |
|
|
| ## License |
|
|
| Apache-2.0, the same as the rest of this repository. |
|
|
