cybersec-mpnet-finetuned

A cybersecurity-domain sentence embedding model fine-tuned from sentence-transformers/all-mpnet-base-v2 on 2,179 CTI-Bench training pairs from two sources:

• CTI-ATE — real threat intelligence report excerpts mapped to MITRE ATT&CK technique descriptions (from the STIX bundle)
• CTI-MCQ — 2,500 cybersecurity MCQ questions covering ATT&CK, CWE, and CAPEC; 500 held out as evaluation set (before training, to prevent data leakage), 2,000 used for training

Built as part of an Agentic Cybersecurity Threat Intelligence & Incident Response Advisor capstone project — a LangGraph-based pipeline that ingests raw security logs, enriches IoCs via threat feeds (AbuseIPDB, VirusTotal, ThreatFox), maps to MITRE ATT&CK, and generates grounded incident response playbooks.

Requirements

sentence-transformers>=5.5.1

This model was trained with sentence-transformers 5.5.1. Older versions will fail to load due to internal module path changes introduced in 5.5.x. If you see a ModuleNotFoundError: No module named 'sentence_transformers.base' error, upgrade:

pip install "sentence-transformers>=5.5.1"

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sindsub/cybersec-mpnet-finetuned")

# Encode cybersecurity text
embeddings = model.encode([
    "T1059.001 - PowerShell command execution via encoded payload",
    "Adversary used scheduled task for persistence after initial access",
    "CVE-2024-1234: Remote code execution in Apache HTTP Server",
])

# Semantic similarity
similarity = model.similarity(embeddings[0], embeddings[1])
print(similarity)  # Higher for semantically related CTI text

Intended Use

• RAG retrieval over MITRE ATT&CK, CISA KEV, NVD CVE, and NIST SP 800-61 knowledge bases
• Semantic search across threat intelligence documents
• RAGAS-style faithfulness evaluation — cosine similarity between LLM-generated playbook steps and their cited source passages
• IoC and technique clustering in threat correlation pipelines

Training Details

Parameter	Value
Base model	sentence-transformers/all-mpnet-base-v2
Training set	CTI-Bench — 2,179 QA pairs
Domains	MITRE ATT&CK, CVE/CWE, CAPEC, threat intelligence
Loss function	MultipleNegativesRankingLoss (MNRL)
Epochs	4
Batch size	16
Training loss	0.41 → 0.08
Hardware	CPU-only

Evaluation Results

Self-retrieval proxy evaluation across four ChromaDB collections (25 chunks sampled per collection, query = chunk's own text):

Collection	Hit@1	Hit@3	Hit@5
MITRE ATT&CK techniques	100%	100%	100%
NVD CVE records	0%	100%	100%
CISA KEV entries	0%	100%	100%
NIST SP 800-61 procedures	0%	0%	100%
Overall	25%	75%	100%

Hit@5 = 1.000 — exceeds the ≥ 0.90 target. Runtime: 100 queries, CPU-only.

RAGAS-style faithfulness (cosine similarity between generated playbook step embeddings and cited source embeddings): 0.548 — a significant improvement over the 0.143 Jaccard score of the base model on the same data.

Model Card

• Developed by: Sindhu S
• Model type: Sentence Transformer (bi-encoder)
• Language: English
• License: Apache 2.0
• Parameters: 109M (inherited from all-mpnet-base-v2)
• Size: ~418 MB (safetensors)

Limitations

• Fine-tuned on CTI-Bench which is primarily MITRE ATT&CK / CVE focused — may not generalise as well to other security domains (e.g., physical security, fraud)
• Training was CPU-only with a small batch size; GPU training with larger batches and more epochs may improve quality further
• Hit@1 is low (25%) for collections with dense, similar entries — a reranker is recommended for precision-critical applications