🛡️ MoE Sovereign Security Expert 4B (moe-expert-security-4b)

Vulnerability Classification, High-Recall Secret Scanning & STRIDE Threat Modeling

---

📌 Executive Summary & Architectural Role

moe-expert-security-4b is a specialized 4-billion parameter Small Language Model (SLM) distilled from DeepSeek-V3 and Mistral-Large-2407 on the LUMI-G Supercomputer (8× AMD Instinct™ MI250X 128GB GPUs).

Within the MoE Sovereign compound AI system, it functions as the Cybersecurity, Static Vulnerability Analysis & Hardening Expert. It is optimized for high-recall secret scanning, accurate Common Weakness Enumeration (CWE) classification, STRIDE threat surface modeling, and the synthesis of production hardening manifests (AppArmor profiles, Seccomp filters, Kubernetes NetworkPolicies).

---

🎯 Functional Scope & Capabilities

1. Static Application Security Analysis (SAST): Identifies memory safety flaws, injection vectors (CWE-89, CWE-78), broken access controls (CWE-862), and SSRF vulnerabilities.
2. High-Recall Secret & Token Scanning: Detects embedded private keys, high-entropy tokens, and credentials across complex multi-file codebases.
3. STRIDE Threat Modeling: Formulates systematic threat vectors across trust boundaries, microservice architectures, and CI/CD pipelines.
4. Hardening Manifest Synthesis: Generates concrete Linux kernel security policies (Seccomp, AppArmor) and container isolation manifests.

---

🎯 Training Objectives & Intended Behavioral Specialization

Capability	Base Stock Qwen 3.5 4B	moe-expert-security-4b (Distilled)
Vulnerability Precision	High rate of false alarms on benign code patterns	Deterministic CWE Classification with verifiable exploitation vectors
Secret Detection	Misses obfuscated or fragmented credentials	High-Entropy Token & Key Detection with regex and entropy validation
Hardening Directives	Generic recommendations ("use HTTPS", "sanitize input")	Production Hardening Manifests (Seccomp JSON, SELinux, CSP headers)
Threat Modeling	Ad-hoc lists of general security risks	Structured STRIDE Matrix mapped directly to system trust boundaries

---

📊 Empirical Evaluation (Held-Out Benchmark Suite)

ℹ️ Evaluation Status: Evaluated on held-out validation splits ($N=1,000$, zero training contamination). Full cross-architecture ablation suites across Compound AI vs. Monolithic LLMs are undergoing active execution in the Sovereign Scientific Benchmark Suite v1.

Evaluated on a held-out benchmark suite of 1,000 cybersecurity and vulnerability audit tasks (derived from CVE corpora and synthetic vulnerability benchmarks) with zero training overlap:

Evaluation Metric	Base Stock Qwen 3.5 4B	moe-expert-security-4b (Distilled)	Delta ($\Delta$)
CWE-1000 Classification Accuracy	63.4 %	94.7 %	+31.3 %
Secret Scanning Recall (High-Entropy / Keys)	71.2 %	98.6 %	+27.4 %
Secret Scanning Precision	65.8 %	95.1 %	+29.3 %
False Positive Rate on Benign Code Patterns	22.4 %	3.8 %	-18.6 %
STRIDE Threat Coverage Completeness	57.0 %	92.3 %	+35.3 %
Valid Hardening Policy Syntax (Seccomp/AppArmor)	52.6 %	96.4 %	+43.8 %

Note: Evaluated at temperature=0.05 across 3 independent seeds. Precision/Recall evaluated on a balanced dataset of 500 vulnerable/secret-containing snippets and 500 benign snippets.

---

🏋️ Training Setup & Distillation Methodology

+-----------------------------------------------------------------------------------+
|                            LUMI-G DISTILLATION PIPELINE                           |
|                                                                                   |
|  [ Teachers: DeepSeek-V3 + Mistral-Large-2407 ]                                   |
|                       |                                                           |
|                       v  (CWE Benchmark Verification + Exploit Validation)        |
|  [ SFT Dataset: 32,800 High-Assurance Security Trajectories ]                     |
|                       |                                                           |
|                       v  (DeepSpeed ZeRO-2, ROCm 7.0, PyTorch 2.6, 8x MI250X)     |
|  [ Student: Qwen3.5-4B Hybrid Linear Attention + Mamba Base ]                     |
|                       |                                                           |
|                       v  (LoRA r=16, alpha=32, target_modules: q/k/v/o/gate/up/down)|
|  [ Output: final_adapter -> CPU-BF16 Merge -> GGUF Q4_K_M & Q8_0 ]                |
+-----------------------------------------------------------------------------------+

Hyperparameters:

• Compute Cluster: LUMI-G (8× AMD Instinct MI250X 128GB GPUs, Slurm Job #21189561)
• Base Architecture: Qwen3.5-4B (Hybrid Linear Attention + Mamba in BF16)
• Dataset Size: 32,800 validated security audit trajectories
• Epochs: 3.0
• Effective Batch Size: 128 (Micro-batch 4 × 8 GPUs × Gradient Accumulation 4)
• Learning Rate: $1.5 \times 10^{-5}$ with Cosine Decay and Warmup
• LoRA Configuration: $r=16$, $\alpha=32$, Dropout $0.05$, Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training Loss (Final): 0.0078
• Token Accuracy (Final): 99.83 %

---

⚠️ Known Limitations & Failure Modes

1. Novel Zero-Day Logic Flaws: The model excels at recognized CWE patterns and structural vulnerabilities, but novel protocol-level zero-days require human security audit.
2. Dynamic Runtime Exploitation: As a static analysis SLM, it models vulnerability likelihood; dynamic runtime behavior should be confirmed with fuzzing / DAST toolchains.
3. Obfuscated Malware Analysis: Heavily packed, polymorphic binary payloads should be routed to dedicated sandbox analysis tools via MCP.

---

💻 Quickstart Guide (Ollama & Llama.cpp)

1. Ollama Modelfile

FROM ./moe-expert-security-4b-Q4_K_M.gguf
PARAMETER num_ctx 262144
PARAMETER temperature 0.05
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""

2. Python Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "h3rb3rn/moe-expert-security-4b"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = "<|im_start|>user\nAnalyze this C++ memory buffer management snippet for potential CWE-122 heap-based buffer overflow vulnerabilities.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.05)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📑 Citation

@misc{moe_sovereign_2026_security4b,
  author = {Horn, Philipp and MoE Sovereign Core AI Team},
  title = {MoE Sovereign Security Expert 4B: Vulnerability Classification & Threat Modeling SLM},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/h3rb3rn/moe-expert-security-4b}},
  note = {Trained on the EuroHPC LUMI-G Supercomputer}
}