Blame for README.md

---
license: apache-2.0
base_model:
- Qwen/Qwen3.8-27B
- Qwen/Qwen3.5-9B
library_name: peft
tags:
- clef
- lora
- decision-model
---

# clef-lora

The LoRA adapters inside Cloudflare's Clef models, extracted from their merged weights.

| Path | Model | Source | Base | Adapter |
|---|---|---|---|---|
| `/` | Clef (27B) | `Cloudflare/clef@ed3eed33` | `Qwen/Qwen3.8-27B@1d4bf0f2` | 186 linears, layers 40–63 |
| `clef-flash/` | Clef-flash (9B) | `Cloudflare/clef-flash@8b2e5fd1` | `Qwen/Qwen3.5-9B@c2022362` | 248 linears, all 32 layers |

Both are PEFT LoRAs in bf16, `r=256`, `lora_alpha=256`. The Gated DeltaNet `in_proj_a`/`in_proj_b` modules use
their full width instead (48 for Clef, 32 for Clef-flash), set via `rank_pattern`/`alpha_pattern`.

## Usage

The adapter replaces the backbone only. The decision head and encoding code come from the Cloudflare repo.

```python
import json, sys, torch
from pathlib import Path
from huggingface_hub import snapshot_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce",
                              allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"]))
sys.path.insert(0, str(clef))
import joint_schema_model as jsm

backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.8-27B", revision="1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", dtype=torch.bfloat16, device_map={"": "cuda"})
backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora")
head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text()))
head.load_state_dict(load_file(clef / "joint_head.safetensors"))
model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval()
processor = AutoProcessor.from_pretrained(clef)

print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.",
                                       "questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}}))
```

For Clef-flash, use `Cloudflare/clef-flash@8b2e5fd1`, `Qwen/Qwen3.5-9B@c2022362` and `subfolder="clef-flash"`.
Install `flash-linear-attention` and `causal-conv1d`; without them transformers uses a much slower path.

## Extraction

- `delta = W_finetuned - W_base` per tensor in fp32, with the base cast to the fine-tune's dtype first.
- Only the linears listed above change. Embeddings, `lm_head`, norms and the vision tower match the base.
- Each delta is exactly rank 256 above the bf16 merge-rounding floor. An exact SVD gives `B = U√S`, `A = √S Vᵀ`.
- The rank-256 fit leaves 0.2–4% relative error per tensor, matching the expected rounding noise.
  `extraction.json` lists it per module.

## Verification

Base + adapter + the release's own head against the released model, on the same requests:

| Model | Test set | Answers changed | Max KL |
|---|---|---:|---:|
| Clef | Decision Index 0.2.1, 6,373 requests (30,950 questions) | 0.09% | 0.0035 |
| Clef | Vision, 11 benchmarks (2,400 rows) | 0.12% | 0.0024 |
| Clef-flash | 120 synthetic requests (231 questions) | 0.43% (1) | 0.0059 |

This is bf16 rounding noise: the same arithmetic as the merged weights, done in a different order.

## On an FP8 base

The Clef adapter, unchanged, on `Qwen/Qwen3.8-27B-FP8` with FP8 weights and activations emulated in bf16 on H200.
Paired on 27,231 Decision Index requests (114,220 gold-scored questions):

- Accuracy: no significant change (net −0.012% of questions; mean −0.05 points per benchmark).
- Calibration: unchanged.
- 1.15% of answers flip, almost all near-ties, in both directions.

## License

Apache-2.0, following the source models and their bases.