---
license: apache-2.0
base_model: Cloudflare/clef-flash
base_model_relation: quantized
tags:
- clef
- decision-model
- systemone
- gguf
- bloomery
---

# Clef-Flash GGUFs on bloomery: which quant to pick

This page has no weights. It shows how close each of
[bartowski's Clef-Flash GGUFs](https://huggingface.co/bartowski/Cloudflare_clef-flash-GGUF) answers to the
official BF16 model, when run with [bloomery](https://github.com/midagedev/bloomery). Use it to pick a file for your
GPU.

This is an unofficial page. It is not made by or affiliated with Cloudflare. The model is
[Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash) (Apache-2.0).

## Why bloomery

Clef is a decision model. One prompt pass gives the backbone's hidden states, and a small "joint schema head" turns
them into a probability for every allowed option. A GGUF holds only the backbone. llama.cpp has no way to run the
head. bloomery runs both and serves the release's SystemOne API (`POST /v1/systemone`).

## Agreement with the official model

We ran 15 requests (8 English, 7 Korean; 31 questions) through Cloudflare's own Python code at BF16 and through
bloomery with each file. "Top equal" counts questions where both pick the same option. "Max |Δp|" is the largest
probability gap on the chosen option.

| File | Size | Top equal | Korean | Max \|Δp\| | Prompt, tok/s |
|---|---:|---:|---:|---:|---:|
| Q8_0 | 9.55 GB | 30/31 | 15/15 | 0.014 | 3,289 |
| Q6_K_L | 8.11 GB | 31/31 | 15/15 | 0.042 | 3,333 |
| Q6_K | 7.79 GB | 31/31 | 15/15 | 0.025 | 3,363 |
| Q6_K_S | 7.51 GB | 31/31 | 15/15 | 0.034 | 3,383 |
| Q5_K_M | 6.88 GB | 31/31 | 15/15 | 0.052 | 3,862 |
| Q5_K_S | 6.50 GB | 31/31 | 15/15 | 0.057 | 3,998 |
| Q4_K_L | 6.20 GB | 30/31 | 15/15 | 0.077 | 4,128 |
| Q4_K_M | 5.84 GB | 29/31 | 15/15 | 0.072 | 4,322 |
| Q4_K_S | 5.48 GB | 29/31 | 15/15 | 0.078 | 4,526 |
| Q3_K_L | 4.66 GB | 30/31 | 15/15 | 0.080 | 4,167 |
| Q3_K_M | 4.48 GB | 30/31 | 15/15 | 0.102 | 4,087 |
| Q3_K_S | 4.26 GB | 29/31 | 15/15 | 0.136 | 4,033 |

- Every miss is a question where the official model is itself almost a tie (0.474 against 0.471) or near 0.5
  (0.512). So the top choice is stable down to Q3_K_S. The probabilities drift as the file gets smaller.
- If you use the probabilities, not only the top choice, pick Q6_K or Q5_K_M.
- Prompt speed: one RTX A6000, a 4,067-token request, one request at a time. These are functional runs, not
  benchmark runs; the same file moves about 3–4 % between runs. The head runs on the CPU: a median of 22 ms per request, about 100 ms at 4,000 tokens.
- Not supported yet: the IQ files, Q2_K, Q4_0, Q4_1 and BF16. bloomery refuses them by name at load.

The full table, the requests and the scripts are in the bloomery repository:
[`tools/ref/clef/agreement.md`](https://github.com/midagedev/bloomery/blob/main/tools/ref/clef/agreement.md).

## Run it

You need a Linux x86-64 host with an NVIDIA RTX 30-series or A-series GPU (sm_86) and the bloomery toolchain
([build guide](https://github.com/midagedev/bloomery/blob/main/docs/BUILD.md#clef-flash)).

```sh
hf download bartowski/Cloudflare_clef-flash-GGUF Cloudflare_clef-flash-Q5_K_M.gguf --local-dir ~/models/clef-flash
hf download Cloudflare/clef-flash joint_head.safetensors joint_head_config.json --local-dir ~/models/clef-flash/hf

cargo oxide build --arch sm_86 -- -p bloomery-gpu-gates --features clef --release --bin bloomery_serve_clef
target/release/bloomery_serve_clef --model ~/models/clef-flash/Cloudflare_clef-flash-Q5_K_M.gguf \
  --head ~/models/clef-flash/hf/joint_head.safetensors --port 8091

curl -s http://127.0.0.1:8091/v1/systemone -d '{"model": "clef-flash",
  "state": "User: what is the weather in Seoul tomorrow? Tools available: web_search, calculator, calendar.",
  "questions": {"tool": {"type": "choice", "instructions": "Which tool should the agent call next?",
    "criteria": {"web_search": "Look something up online", "calculator": "Do arithmetic",
                 "calendar": "Read or write events", "none": "Answer directly"}}}}'
```

The request and response follow the release's SystemOne format. Each response also has `timings` (`prompt_n`,
`prompt_ms`, `head_ms`).

## Limits

- Text states only (no images or video).
- One request at a time.
- sm_86 GPUs only (RTX 3090, RTX A6000).

## Credits

Model: Cloudflare ([Clef-Flash](https://huggingface.co/Cloudflare/clef-flash), Apache-2.0). GGUFs: bartowski. Engine:
bloomery (MIT). AI assistants helped write bloomery and this page; every number above was measured.