cti-ner-roberta-large (clean LLM-gold retrain, v2)

roberta-large fine-tuned as a BIO token classifier for cyber threat intelligence NER on CISA advisories: 16 entity types (attack technique id/name, CVE/CWE identifiers, file hash/name/path, IP address, email address, malware family, threat actor group, hacking tool, tactic, TLP classification, YARA rule name, affected software product).

This version replaces the earlier checkpoint trained on dictionary-silver labels. Training gold is LLM-proposed (Claude Sonnet 5, exhaustive extraction under per-label span-format validators) with the evaluation split human-curated. See the paper in the repository below for the methodology and the circularity analysis that motivated the switch.

Evaluation (measured on this checkpoint)

Document-level span scoring on the held-out 8-advisory test split (strict = exact span + label), verified independently with the repository harness (ner.finetune_roberta --eval-only):

Metric	Strict	Relaxed
Micro F1	0.63	0.69
Macro F1	0.71	0.75

The paper's committed result for this architecture (data/gold_v2/results/trained-roberta-large.json) is 0.648 strict micro from an earlier training run; this checkpoint is a re-run with identical data and hyperparameters, and the 1.4-point difference is seed/hardware variance on an 8-document test set. Either way the architecture ordering holds: SecureBERT 0.71 $>$ RoBERTa-base 0.659 $>$ RoBERTa-large $>$ GLiNER 0.560 on the identical gold and harness, i.e., domain-adaptive pretraining beats raw parameter count on this task.

Training details

• Data: 66 training advisories (2,799 chunks), 8 val, 8 test; document-level split committed in the repository (data/split_slugs.json).
• 8 epochs, lr 2e-5, seed 0, wordpiece-budget sliding windows; best epoch by validation strict F1. Trained on Colab CUDA via notebooks/RoBERTa_Large_CTI_Colab.ipynb (corpus embedded in the notebook).

Usage

Inference should replicate the sliding-window word-tagging procedure in ner/finetune_roberta.py (see notebooks/hf_models_demo.ipynb in the repository for a worked example); a naive single-pass tokenizer call truncates long, hash-heavy advisory text.

Links

• Code, gold data, paper: https://github.com/dandye/nlp_capstone
• Companion models: pzryathzsdhc/cti-ner-securebert, pzryathzsdhc/cti-ner-gliner