File size: 3,633 Bytes
e4ba6c4 0d511e7 0ea2a8e b887778 b2d1660 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
language:
- en
license: other
tags:
- security
- prompt-injection
- litert
---
# CLEF-Screen-V2
Redo fine-tune of the CLEF-Flash `JointSchemaHead` (121,762,820 parameters,
122 tensors) on frozen Cloudflare CLEF-Flash LiteRT backbone hidden states.
Only the head is trained; the backbone is unmodified. This repository root
is a complete CLEF LiteRT checkpoint directory with the fine-tuned head.
## Training
- Compute: DigitalOcean GPU droplet jev-retrain (AMD MI355X 288GB, gpu-mi355x1-288gb-spot), train+eval 90 min (plus 44 min backbone encode).
- Method: `ft_clef.py --cache-only (backbone hidden states pre-encoded with pinned PyTorch clef-flash)`, 3 epoch(s), lr 3e-05, batch 16,
pos_weight 1.0. Training details and hashes: `redo-provenance.json`;
full log: `redo-training-log.jsonl` when present.
- Head dtype in this repo: BF16 (deployment contract). Trained head was already BF16; no conversion.
## Evaluation status
Run clef_v04 (fixed-data retrain). Training set rebuilt 1:1 balanced: 16,322 rows (8,232 positive / 8,090 negative, SHA-256 e3916d4946cbe6e4787579ba73a82485f1e9ee186c241444586aeafa26e3bb87), replacing the 92%-positive frozen-v0.3 train split (94,864 pos / 8,026 neg); includes +86 mined real-world technique-catalog rows and +184 synthetic tool-spoof rows (labelled synthetic in provenance); pos_weight 1.0 (was 3.0). Frozen dev/test splits unchanged. Epoch 3 selected on dev AUC only (per-epoch dev AUCs 0.9635 / 0.9811 / 0.9840; untrained-head dev baseline 0.7516). Dev (n=21,977; 20,282 pos / 1,695 neg): AUC 0.9840, TPR@0.5 0.939, FPR@0.5 0.050. Frozen test (n=21,965; 20,271 pos / 1,694 neg), evaluated once at end of training: AUC 0.9833, TPR@0.5 0.940, FPR@0.5 0.054. Per-row scores ship in this repo (redo-dev-scores.jsonl, redo-test-scores.jsonl, redo-final-results.json) and reproduce these numbers exactly. Caveats: dev/test are ~92% positive so FPR rests on ~1,700 negatives; this run trains and evaluates the tool_output_filter task only (action_authorization_guard n=0); no adaptive-attack evaluation or independent replication yet. Uploaded on owner direction ahead of the Tuesday-v2 frozen-chain replication gate; see redo-provenance.json.
No frozen-test, adaptive-attack, or independent-replication result exists yet
for this checkpoint unless stated above. Treat all numbers as development
measurements, not deployment guarantees.
## Efficacy summary chart
Checked 2026-11-05 against the live run3 counts and the per-row score files. Q1 row 2 is stop replay at 0.99: 51/249 tasks finished, 7/249 hijacked, 35/37 normal tasks passed, adaptive 53/100. The judge normal rate is 141/148. No per-check timing is on the chart. CLEF-Screen caught 20098/20271 (99.1%). `build_replication_package.py` rerun matched the published AUCs. Two gaps stay labeled: CLEF-DualScreen has no local per-row test file, and the SecJev redo numbers are bench draws.
## Final model comparison
CLEF-Screen-V2 vs SecJev-4B-Spot2-V3 vs hosted TypeSafe ultrafast JEV (jev-1.14.0, scored live 2026-11-05). Tasks and test sets differ per row and are printed on the graph; all metrics recomputed from per-row scores.
## Rent vs own
Hosted TypeSafe ultrafast JEV (jev-1.14.0, zero-shot) vs own fine-tuned checkpoints. Threshold 0.5; different test sets per model as printed on the graphic; SecJev task is tool-call authorization.
|