dsuyu1/FedDAPT-security-v1

FedDAPT Security v1

A domain-adapted security LLM trained using federated learning across simulated multi-tenant security environments. Built on Mistral-7B with QLoRA adapters.

What this model does

Specializes in cybersecurity tasks including incident summarization, alert triage, and threat intelligence analysis. Trained without centralizing any organization's private security data.

Results

Method	ROUGE-L
Zero-shot Mistral-7B	0.367
Centralized DAPT	0.330
FedDAPT (this model)	0.707

FedDAPT achieved 2.1x improvement over centralized training on incident summarization.

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "dsuyu1/FedDAPT-security-v1")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

prompt = """### Instruction:
Summarize the following security incident in one sentence.

### Input:
Incident timeline: initial_access: phishing with macro attachment -> execution: PowerShell encoded command -> c2: Cobalt Strike HTTPS beacon -> lateral: SMB + PsExec lateral movement -> impact: ransomware across endpoints.

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Training Details

• Framework: FedDAPT (Federated Domain-Adaptive Pre-Training)
• Base model: Mistral-7B-v0.1
• Adapter: LoRA (r=16, alpha=32, targets: q_proj, v_proj)
• Quantization: QLoRA 4-bit NF4
• Aggregation: FedAvg with FedProx (mu=0.01)
• Clients: 3 (endpoint-focused, network-focused, cloud/CTI-focused)
• Rounds: 20
• Corpus: Curated from MITRE ATT&CK, SigmaHQ, NVD, CISA KEV, MITRE CAR (44,754 raw -> 4,265 curated)
• Curation: NVIDIA NeMo Curator (dedup, PII redaction, quality filtering)

Limitations

• Trained on public proxy data, not real operational telemetry
• Triage accuracy (35%) has room for improvement
• Requires instruction format (### Instruction / ### Input / ### Response)
• 7B parameter model requires GPU for inference

Citation

Villarreal, D. "Smarter SecOps: Leveraging Private, Federated Transfer Learning" BSides RGV 2026.