Embed Rent-vs-Own infographic in model card

b2d1660 verified about 12 hours ago

3.63 kB

---
language:
- en
license: other
tags:
- security
- prompt-injection
- litert
---
# CLEF-Screen-V2

Redo fine-tune of the CLEF-Flash `JointSchemaHead` (121,762,820 parameters,
122 tensors) on frozen Cloudflare CLEF-Flash LiteRT backbone hidden states.
Only the head is trained; the backbone is unmodified. This repository root
is a complete CLEF LiteRT checkpoint directory with the fine-tuned head.

## Training

- Compute: DigitalOcean GPU droplet jev-retrain (AMD MI355X 288GB, gpu-mi355x1-288gb-spot), train+eval 90 min (plus 44 min backbone encode).
- Method: `ft_clef.py --cache-only (backbone hidden states pre-encoded with pinned PyTorch clef-flash)`, 3 epoch(s), lr 3e-05, batch 16,
  pos_weight 1.0. Training details and hashes: `redo-provenance.json`;
  full log: `redo-training-log.jsonl` when present.
- Head dtype in this repo: BF16 (deployment contract). Trained head was already BF16; no conversion.
## Evaluation status
Run clef_v04 (fixed-data retrain). Training set rebuilt 1:1 balanced: 16,322 rows (8,232 positive / 8,090 negative, SHA-256 e3916d4946cbe6e4787579ba73a82485f1e9ee186c241444586aeafa26e3bb87), replacing the 92%-positive frozen-v0.3 train split (94,864 pos / 8,026 neg); includes +86 mined real-world technique-catalog rows and +184 synthetic tool-spoof rows (labelled synthetic in provenance); pos_weight 1.0 (was 3.0). Frozen dev/test splits unchanged. Epoch 3 selected on dev AUC only (per-epoch dev AUCs 0.9635 / 0.9811 / 0.9840; untrained-head dev baseline 0.7516). Dev (n=21,977; 20,282 pos / 1,695 neg): AUC 0.9840, TPR@0.5 0.939, FPR@0.5 0.050. Frozen test (n=21,965; 20,271 pos / 1,694 neg), evaluated once at end of training: AUC 0.9833, TPR@0.5 0.940, FPR@0.5 0.054. Per-row scores ship in this repo (redo-dev-scores.jsonl, redo-test-scores.jsonl, redo-final-results.json) and reproduce these numbers exactly. Caveats: dev/test are ~92% positive so FPR rests on ~1,700 negatives; this run trains and evaluates the tool_output_filter task only (action_authorization_guard n=0); no adaptive-attack evaluation or independent replication yet. Uploaded on owner direction ahead of the Tuesday-v2 frozen-chain replication gate; see redo-provenance.json.
No frozen-test, adaptive-attack, or independent-replication result exists yet
for this checkpoint unless stated above. Treat all numbers as development
measurements, not deployment guarantees.
## Efficacy summary chart
![Can Small Security Models Stop Prompt Injection — And Is Fine-Tuning Worth It?](jev_human_questions_chart.png)
Checked 2026-11-05 against the live run3 counts and the per-row score files. Q1 row 2 is stop replay at 0.99: 51/249 tasks finished, 7/249 hijacked, 35/37 normal tasks passed, adaptive 53/100. The judge normal rate is 141/148. No per-check timing is on the chart. CLEF-Screen caught 20098/20271 (99.1%). `build_replication_package.py` rerun matched the published AUCs. Two gaps stay labeled: CLEF-DualScreen has no local per-row test file, and the SecJev redo numbers are bench draws.
## Final model comparison
![Final JEV Models — Efficacy and Cost](final_three_models.png)
CLEF-Screen-V2 vs SecJev-4B-Spot2-V3 vs hosted TypeSafe ultrafast JEV (jev-1.14.0, scored live 2026-11-05). Tasks and test sets differ per row and are printed on the graph; all metrics recomputed from per-row scores.
## Rent vs own
![RENT vs. OWN: Fine-Tuning Cuts False Positives by 62%](rent_vs_own_infographic.png)
Hosted TypeSafe ultrafast JEV (jev-1.14.0, zero-shot) vs own fine-tuned checkpoints. Threshold 0.5; different test sets per model as printed on the graphic; SecJev task is tool-call authorization.

	---
	language:
	- en
	license: other
	tags:
	- security
	- prompt-injection
	- litert
	---

	# CLEF-Screen-V2

	Redo fine-tune of the CLEF-Flash `JointSchemaHead` (121,762,820 parameters,
	122 tensors) on frozen Cloudflare CLEF-Flash LiteRT backbone hidden states.
	Only the head is trained; the backbone is unmodified. This repository root
	is a complete CLEF LiteRT checkpoint directory with the fine-tuned head.

	## Training

	- Compute: DigitalOcean GPU droplet jev-retrain (AMD MI355X 288GB, gpu-mi355x1-288gb-spot), train+eval 90 min (plus 44 min backbone encode).
	- Method: `ft_clef.py --cache-only (backbone hidden states pre-encoded with pinned PyTorch clef-flash)`, 3 epoch(s), lr 3e-05, batch 16,
	pos_weight 1.0. Training details and hashes: `redo-provenance.json`;
	full log: `redo-training-log.jsonl` when present.
	- Head dtype in this repo: BF16 (deployment contract). Trained head was already BF16; no conversion.

	## Evaluation status

	Run clef_v04 (fixed-data retrain). Training set rebuilt 1:1 balanced: 16,322 rows (8,232 positive / 8,090 negative, SHA-256 e3916d4946cbe6e4787579ba73a82485f1e9ee186c241444586aeafa26e3bb87), replacing the 92%-positive frozen-v0.3 train split (94,864 pos / 8,026 neg); includes +86 mined real-world technique-catalog rows and +184 synthetic tool-spoof rows (labelled synthetic in provenance); pos_weight 1.0 (was 3.0). Frozen dev/test splits unchanged. Epoch 3 selected on dev AUC only (per-epoch dev AUCs 0.9635 / 0.9811 / 0.9840; untrained-head dev baseline 0.7516). Dev (n=21,977; 20,282 pos / 1,695 neg): AUC 0.9840, TPR@0.5 0.939, FPR@0.5 0.050. Frozen test (n=21,965; 20,271 pos / 1,694 neg), evaluated once at end of training: AUC 0.9833, TPR@0.5 0.940, FPR@0.5 0.054. Per-row scores ship in this repo (redo-dev-scores.jsonl, redo-test-scores.jsonl, redo-final-results.json) and reproduce these numbers exactly. Caveats: dev/test are ~92% positive so FPR rests on ~1,700 negatives; this run trains and evaluates the tool_output_filter task only (action_authorization_guard n=0); no adaptive-attack evaluation or independent replication yet. Uploaded on owner direction ahead of the Tuesday-v2 frozen-chain replication gate; see redo-provenance.json.

	No frozen-test, adaptive-attack, or independent-replication result exists yet
	for this checkpoint unless stated above. Treat all numbers as development
	measurements, not deployment guarantees.

	## Efficacy summary chart

	![Can Small Security Models Stop Prompt Injection — And Is Fine-Tuning Worth It?](jev_human_questions_chart.png)

	Checked 2026-11-05 against the live run3 counts and the per-row score files. Q1 row 2 is stop replay at 0.99: 51/249 tasks finished, 7/249 hijacked, 35/37 normal tasks passed, adaptive 53/100. The judge normal rate is 141/148. No per-check timing is on the chart. CLEF-Screen caught 20098/20271 (99.1%). `build_replication_package.py` rerun matched the published AUCs. Two gaps stay labeled: CLEF-DualScreen has no local per-row test file, and the SecJev redo numbers are bench draws.

	## Final model comparison

	![Final JEV Models — Efficacy and Cost](final_three_models.png)

	CLEF-Screen-V2 vs SecJev-4B-Spot2-V3 vs hosted TypeSafe ultrafast JEV (jev-1.14.0, scored live 2026-11-05). Tasks and test sets differ per row and are printed on the graph; all metrics recomputed from per-row scores.

	## Rent vs own

	![RENT vs. OWN: Fine-Tuning Cuts False Positives by 62%](rent_vs_own_infographic.png)

	Hosted TypeSafe ultrafast JEV (jev-1.14.0, zero-shot) vs own fine-tuned checkpoints. Threshold 0.5; different test sets per model as printed on the graphic; SecJev task is tool-call authorization.