Add full 7,192-row test scores and analysis; card now leads with full test (AUC 0.8661)
4596d1f verified | language: | |
| - en | |
| license: other | |
| license_name: secjev-use-and-distribution-agreement-v1.0 | |
| license_link: LICENSE | |
| base_model: Qwen/Qwen3.5-4B-Base | |
| tags: | |
| - security | |
| - decision-model | |
| - lora | |
| - prompt-injection | |
| # SecJev-4B-Spot2-V3 | |
| SecJev-4B action-authorization guard (Spot 2: outgoing tool calls), | |
| full-data redo fine-tune, seed 20261102, epoch 3 of 3 (dev-selected: | |
| last and best epoch). Repository root is a complete Kev checkpoint | |
| directory. The Qwen3.5-4B-Base backbone is not re-hosted; bootstrap.py | |
| fetches the pinned backbone and Kev source. | |
| ## Training | |
| - Compute: DigitalOcean GPU droplet jev-retrain (AMD Instinct MI355X, | |
| ROCm), finished 2026-11-04 20:41 UTC. | |
| - Method: ft_action.py, LoRA + head (33,776,128 trainable params), | |
| 3 epochs, batch 16 questions, lr 3e-5, bf16 weights, seed 20261102. | |
| - Data: frozen Spot-2 v0.2 full splits (train 6,413 / dev 1,482). | |
| - Training log: redo-training-log.jsonl. Hashes: redo-provenance.json. | |
| ## Evaluation (read this before citing any number) | |
| Dev (1,482 rows, only 85 positives) is near-saturated and does NOT | |
| predict held-out-domain test performance: | |
| | split | AUC | at dev-calibrated threshold (dev FPR<=1%) | at 0.5 | | |
| |---|---|---|---| | |
| | Dev (independent local rescore) | 0.9971 | TPR 0.953 / FPR 0.010 | - | | |
| | Full test (7,192 rows, droplet scores) | 0.8661 | TPR 0.776 / FPR 0.221 | TPR 0.544 / FPR 0.072 | | |
| | Test banking+slack subset (3,044 rows) | 0.8236 | TPR 0.691 / FPR 0.230 | TPR 0.425 / FPR 0.076 | | |
| The dev-calibrated threshold (0.0021) transfers badly across domains | |
| (test FPR 0.23). Balanced-accuracy 95% CI on test: [0.714, 0.751]. | |
| Per-suite and full analysis in redo-frozen-analysis.json; per-row | |
| scores in redo-dev-scores-local-rescore.jsonl and | |
| redo-test-scores-banking-slack.jsonl. Epoch selection used dev only | |
| (epoch AUCs 0.9954 / 0.9966 / 0.9967). A seed-20261101 sibling run | |
| finished at dev AUC 0.9926 and is not published; with 85 dev | |
| positives, seed differences of this size are within noise. No | |
| independent replication under the Tuesday-v2 frozen chain exists yet. | |