| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen3.8-27B |
| - Qwen/Qwen3.5-9B |
| library_name: peft |
| tags: |
| - clef |
| - lora |
| - decision-model |
| --- |
| |
| # clef-lora |
|
|
| The LoRA adapters inside Cloudflare's Clef models, extracted from their merged weights. |
|
|
| | Path | Model | Source | Base | Adapter | |
| |---|---|---|---|---| |
| | `/` | Clef (27B) | `Cloudflare/clef@ed3eed33` | `Qwen/Qwen3.8-27B@1d4bf0f2` | 186 linears, layers 40–63 | |
| | `clef-flash/` | Clef-flash (9B) | `Cloudflare/clef-flash@8b2e5fd1` | `Qwen/Qwen3.5-9B@c2022362` | 248 linears, all 32 layers | |
|
|
| Both are PEFT LoRAs in bf16, `r=256`, `lora_alpha=256`. The Gated DeltaNet `in_proj_a`/`in_proj_b` modules use |
| their full width instead (48 for Clef, 32 for Clef-flash), set via `rank_pattern`/`alpha_pattern`. |
|
|
| ## Usage |
|
|
| The adapter replaces the backbone only. The decision head and encoding code come from the Cloudflare repo. |
|
|
| ```python |
| import json, sys, torch |
| from pathlib import Path |
| from huggingface_hub import snapshot_download |
| from peft import PeftModel |
| from safetensors.torch import load_file |
| from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration |
| |
| clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce", |
| allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"])) |
| sys.path.insert(0, str(clef)) |
| import joint_schema_model as jsm |
| |
| backbone = Qwen3_5ForConditionalGeneration.from_pretrained( |
| "Qwen/Qwen3.8-27B", revision="1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", dtype=torch.bfloat16, device_map={"": "cuda"}) |
| backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora") |
| head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text())) |
| head.load_state_dict(load_file(clef / "joint_head.safetensors")) |
| model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval() |
| processor = AutoProcessor.from_pretrained(clef) |
| |
| print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.", |
| "questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}})) |
| ``` |
|
|
| For Clef-flash, use `Cloudflare/clef-flash@8b2e5fd1`, `Qwen/Qwen3.5-9B@c2022362` and `subfolder="clef-flash"`. |
| Install `flash-linear-attention` and `causal-conv1d`; without them transformers uses a much slower path. |
|
|
| ## Extraction |
|
|
| - `delta = W_finetuned - W_base` per tensor in fp32, with the base cast to the fine-tune's dtype first. |
| - Only the linears listed above change. Embeddings, `lm_head`, norms and the vision tower match the base. |
| - Each delta is exactly rank 256 above the bf16 merge-rounding floor. An exact SVD gives `B = U√S`, `A = √S Vᵀ`. |
| - The rank-256 fit leaves 0.2–4% relative error per tensor, matching the expected rounding noise. |
| `extraction.json` lists it per module. |
|
|
| ## Verification |
|
|
| Base + adapter + the release's own head against the released model, on the same requests: |
|
|
| | Model | Test set | Answers changed | Max KL | |
| |---|---|---:|---:| |
| | Clef | Decision Index 0.2.1, 6,373 requests (30,950 questions) | 0.09% | 0.0035 | |
| | Clef | Vision, 11 benchmarks (2,400 rows) | 0.12% | 0.0024 | |
| | Clef-flash | 120 synthetic requests (231 questions) | 0.43% (1) | 0.0059 | |
|
|
| This is bf16 rounding noise: the same arithmetic as the merged weights, done in a different order. |
|
|
| ## On an FP8 base |
|
|
| The Clef adapter, unchanged, on `Qwen/Qwen3.8-27B-FP8` with FP8 weights and activations emulated in bf16 on H200. |
| Paired on 27,231 Decision Index requests (114,220 gold-scored questions): |
|
|
| - Accuracy: no significant change (net −0.012% of questions; mean −0.05 points per benchmark). |
| - Calibration: unchanged. |
| - 1.15% of answers flip, almost all near-ties, in both directions. |
|
|
| ## License |
|
|
| Apache-2.0, following the source models and their bases. |
|
|
