language:
- en
license: other
license_name: secjev-use-and-distribution-agreement-v1.0
license_link: LICENSE
base_model: Qwen/Qwen3.5-4B-Base
tags:
- security
- decision-model
- lora
- prompt-injection
SecJev-4B-Spot2-V3
SecJev-4B action-authorization guard (Spot 2: outgoing tool calls), full-data redo fine-tune, seed 20261102, epoch 3 of 3 (dev-selected: last and best epoch). Repository root is a complete Kev checkpoint directory. The Qwen3.5-4B-Base backbone is not re-hosted; bootstrap.py fetches the pinned backbone and Kev source.
Training
- Compute: DigitalOcean GPU droplet jev-retrain (AMD Instinct MI355X, ROCm), finished 2026-11-04 20:41 UTC.
- Method: ft_action.py, LoRA + head (33,776,128 trainable params), 3 epochs, batch 16 questions, lr 3e-5, bf16 weights, seed 20261102.
- Data: frozen Spot-2 v0.2 full splits (train 6,413 / dev 1,482).
- Training log: redo-training-log.jsonl. Hashes: redo-provenance.json.
Evaluation (read this before citing any number)
Dev (1,482 rows, only 85 positives) is near-saturated and does NOT predict held-out-domain test performance:
| split | AUC | at dev-calibrated threshold (dev FPR<=1%) | at 0.5 |
|---|---|---|---|
| Dev (independent local rescore) | 0.9971 | TPR 0.953 / FPR 0.010 | - |
| Full test (7,192 rows, droplet scores) | 0.8661 | TPR 0.776 / FPR 0.221 | TPR 0.544 / FPR 0.072 |
| Test banking+slack subset (3,044 rows) | 0.8236 | TPR 0.691 / FPR 0.230 | TPR 0.425 / FPR 0.076 |
The dev-calibrated threshold (0.0021) transfers badly across domains (test FPR 0.23). Balanced-accuracy 95% CI on test: [0.714, 0.751]. Per-suite and full analysis in redo-frozen-analysis.json; per-row scores in redo-dev-scores-local-rescore.jsonl and redo-test-scores-banking-slack.jsonl. Epoch selection used dev only (epoch AUCs 0.9954 / 0.9966 / 0.9967). A seed-20261101 sibling run finished at dev AUC 0.9926 and is not published; with 85 dev positives, seed differences of this size are within noise. No independent replication under the Tuesday-v2 frozen chain exists yet.