Arcspan — Cybersecurity NER via Fine-tuned OpenAI Privacy Filter

Arcspan fine-tunes OpenAI's Privacy Filter — a 1.5B-parameter sparse MoE bidirectional token classifier — for cybersecurity entity extraction from threat intelligence reports.

The key insight: the Privacy Filter's label space is fully replaceable. By swapping the output head and fine-tuning on cybersecurity NER data, we get a model that extracts IOCs, malware names, CVEs, threat actors, and affected systems — at 50M active parameters (11× fewer than the CyNER baseline at 560M dense params).

TL;DR: Same task as CyNER, fraction of the compute, fully local/offline.

---

Why this exists

SOC teams and threat intelligence platforms need fast, local IOC extraction. Existing tools like CyNER use large dense transformers — expensive to run, can't be deployed on-prem without significant hardware, and can't process sensitive reports that shouldn't leave the network.

The Privacy Filter's sparse MoE architecture activates only 50M of its 1.5B parameters per token, making it fast enough for real workloads while maintaining strong accuracy on the entity types that matter most.

---

Model Architecture

Property	Value
Base model	openai/privacy-filter
Total parameters	1.5B
Active parameters (per token)	~50M (top-4 MoE routing)
Experts	128 total, 4 active per token
Layers	8 transformer blocks
Hidden size	640
Attention	Grouped-query (14 Q heads, 2 KV heads), RoPE
Context window	128K tokens (effective: 257-token banded window)
Decoding	Constrained Viterbi (BIOES transition enforcement)
Precision	bfloat16
License	Apache 2.0

---

Label Space

5-class cybersecurity NER schema using BIOES tagging:

Label	What it covers
Malware	Malware families, ransomware, trojans, backdoors, botnets, campaigns
Indicator	IOCs — IPs, domains, URLs, file hashes, file paths, registry keys, email addresses
System	Operating systems, software, platforms, infrastructure components
Organization	Threat actors, APT groups, vendors, affected organizations
Vulnerability	CVEs, exploit names, vulnerability descriptions

---

Benchmarks

R8 Checkpoint — Exact-Match Micro F1 (paper-comparable)

Evaluated with strict exact span boundary matching (seqeval-style). These are the honest, paper-comparable numbers — not the relaxed containment scores.

Benchmark	Micro F1	Precision	Recall	Notes
APTNER (independent)	0.498	0.668	0.397	APT report style, independent held-out set
CyNER	0.405	0.454	0.365	Original CyNER test set

Per-Class Breakdown — APTNER (Exact Match)

Class	F1	Precision	Recall
Malware	0.707	0.793	0.637
Indicator	0.667	0.661	0.673
Vulnerability	0.500	0.429	0.600
Organization	0.326	0.500	0.242
System	0.160	0.615	0.092

Per-Class Breakdown — CyNER (Exact Match)

Class	F1	Precision	Recall
Malware	0.577	0.585	0.570
Indicator	0.250	0.518	0.165
System	0.399	0.412	0.387
Organization	0.316	0.288	0.351
Vulnerability	0.375	0.500	0.300

Containment Span F1 (OPF native eval, relaxed boundary matching)

Test Set	Span F1	Precision	Recall
Enriched (primary)	0.550	0.513	0.591
APTNER	0.550	0.687	0.459
CyNER	0.468	0.512	0.431
SecureBERT2	0.451	0.513	0.403

Note on scoring: Containment F1 is ~5 points higher than exact-match F1 due to relaxed boundary matching. Exact-match is the paper-comparable metric.

What R8 tells us

• Strong on Malware + Indicator — F1 0.58–0.71, these are the most common IOC types
• Weak on Organization + System recall — the model detects entities but misses many in APT-report style prose; this is a training data representation gap, not an architecture limit
• Boundary precision is solid — APTNER detection F1 is 0.80, meaning the model finds entities; exact-match drops because boundary offsets aren't always perfect
• No data leakage — R8 was trained with strict deduplication; all held-out sets have zero exact or prefix-80 overlap with training data

---

Checkpoints

checkpoints/r8_5class/epoch_4 ← Main checkpoint

Full multi-source fine-tune. Best checkpoint from a 7-epoch training run (early stopped at patience=3).

Parameter	Value
Training examples	28,675
Best epoch	4
Val loss	0.1089
Optimizer	AdamW, focal loss γ=2
Learning rate	5e-5 (cosine), LLRD 0.9
Batch size	4 (grad accum 2)
O-downsampling	0.7

checkpoints/cyner_v1_sanity

Sanity-check fine-tune on 50 examples to validate the training pipeline end-to-end. Not intended for production use.

Parameter	Value
Training examples	50
Best epoch	1
Best val loss	0.687
Val token accuracy	90.1%

---

Training Data (R8)

28,675 examples aggregated from 20+ sources, deduplicated and leakage-cleaned:

Source	Records
CyNER v2	4,563
CyberNER STIX	3,723
DNRTI	2,834
APTNER	2,584
APT reports (LLM-annotated)	2,263
NVD CVEs v2	1,995
MITRE ATT&CK v2	1,485
Synthetic IOC v2	1,292
CyberNER	1,204
CyNER original	717
Defanged augmentation	652
ExploitDB	500
NVD CVE	338
+ 9 more sources	~345

Leakage audit: Zero exact-match and zero prefix-80 overlap between training data and any held-out benchmark (APTNER, CyNER, SecureBERT2, enriched test, validation).

---

Usage

# Install the base framework
pip install -e vendor/privacy-filter

# Run inference with the R8 checkpoint
opf --checkpoint checkpoints/r8_5class/epoch_4 --device cpu \
  "APT29 deployed Cobalt Strike beacon via CVE-2021-44228 against Microsoft Exchange servers."

# Evaluate on a JSONL test file
opf eval data/processed/cyner_test.jsonl \
  --checkpoint checkpoints/r8_5class/epoch_4 \
  --device cpu

# Fine-tune further with a custom dataset
opf train my_train.jsonl \
  --val my_val.jsonl \
  --output-dir ./my_checkpoint

Input format (JSONL):

{"text": "Emotet was distributed via malicious Word documents exploiting CVE-2017-11882.", "spans": []}

Output format:

{
  "text": "Emotet was distributed via malicious Word documents exploiting CVE-2017-11882.",
  "spans": [
    {"start": 0, "end": 6, "label": "Malware", "text": "Emotet"},
    {"start": 62, "end": 77, "label": "Vulnerability", "text": "CVE-2017-11882"}
  ]
}

---

Repository Structure

arcspan/
├── checkpoints/
│   ├── r8_5class/epoch_4/      ← Main checkpoint (model.safetensors + config.json)
│   └── cyner_v1_sanity/        ← Sanity-check checkpoint
├── data/
│   ├── processed/              ← Training/eval JSONL splits (all benchmarks)
│   └── label_spaces/           ← Label space JSON configs
├── scripts/                    ← Training, eval, and data processing scripts
├── src/arcspan/                ← Core Python package
├── results/                    ← Benchmark results and audit reports
├── research/
│   ├── decisions/              ← Architecture Decision Records (ADRs)
│   ├── notes/progress/         ← Research progress notes
│   └── paper/                  ← Paper draft and outline
└── vendor/privacy-filter/      ← OpenAI Privacy Filter (vendored)

---

Roadmap

• R9 training — Stricter leakage-clean multi-source dataset (24,518 examples), targeting improved Organization + System recall
• O-logit bias decoding — Zero-cost inference trick to trade precision for recall on weak classes
• Research paper — "Sparse MoE vs. dense transformer for cybersecurity NER: an efficiency comparison"
• Future verticals — Energy/power systems, clinical NER (same architecture, different label space)

---

Citation

If you use Arcspan in your work, please cite the base model:

@misc{openai2025privacyfilter,
  title={Privacy Filter},
  author={OpenAI},
  year={2025},
  url={https://huggingface.co/openai/privacy-filter}
}

---

License

Apache 2.0 — same as the base OpenAI Privacy Filter.