xpertsystems/cyb005-baseline-classifier

CYB005 Baseline Classifier

Threat-actor capability-tier classifier trained on the CYB005 synthetic ransomware campaign sample. Predicts which of 4 actor tiers (lone_actor / organised_syndicate / raas_affiliate / nation_state_nexus) is behind an observed ransomware campaign from per-timestep telemetry.

Baseline reference, not for production use. This model demonstrates that the CYB005 sample dataset is learnable end-to-end and gives prospective buyers a working starting point for threat-attribution research. It is not a production threat-intelligence system, attribution engine, or incident-response tool. See Limitations.

Model overview

Property	Value
Task	4-class actor_capability_tier classification
Training data	xpertsystems/cyb005-sample (37,489 timesteps across 500 ransomware campaigns)
Models	XGBoost + PyTorch MLP
Input features	63 (after one-hot encoding)
Split	Group-aware by campaign_id (disjoint train/val/test campaigns)
Validation	Single seed (artifact) + multi-seed aggregate across 10 seeds
License	CC-BY-NC-4.0 (matches dataset)
Status	Reference baseline

Why this task — and why CYB005 ships it where CYB002/3/4 could not

This is the first XpertSystems baseline that targets the dataset's stated headline use case. The CYB005 README's first suggested use case is "ransomware classifier models (4-tier actor attribution)", and that is exactly what this baseline ships.

In CYB002 (kill-chain), CYB003 (malware family), and CYB004 (actor tier), the sample datasets had only ~100 groups (events / samples / campaigns), which limits group-aware test folds to ~15 unseen groups and 1.5–2 groups per class. Each baseline had to pivot to a phase-prediction subtask that was learnable at sample size.

CYB005's sample is intentionally 5× larger — 500 campaigns — because the README explicitly notes that "benchmarks are conditional on small actor-tier subsets". The larger sample makes a held-out test fold of 75 disjoint campaigns possible, with each of the four tiers represented by 11–30 unseen test campaigns. Tier attribution becomes genuinely learnable, and that's what we publish.

Two model artifacts are published. They are designed to be used together — disagreement is a useful triage signal:

• model_xgb.json — gradient-boosted trees, primary recommendation
• model_mlp.safetensors — PyTorch MLP in SafeTensors format

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb005-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import (
    transform_single, load_meta, INT_TO_LABEL, build_segment_lookup
)

meta = load_meta(paths["feature_meta.json"])
xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])
seg_lookup = build_segment_lookup("path/to/victim_topology.csv")

# Predict (see inference_example.ipynb for the full pattern)
seg_aggs = seg_lookup.get(my_record["target_segment_id"], {})
X = transform_single(my_record, meta, segment_aggregates=seg_aggs)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB005, 37,489 per-timestep telemetry rows from 500 ransomware campaigns (75 timesteps per campaign):

Tier	Campaigns	Timestep rows	Train share
organised_syndicate	200	14,998	40.0%
raas_affiliate	150	11,250	30.0%
lone_actor	75	5,625	15.0%
nation_state_nexus	75	5,616	15.0%

Group-aware split

A single campaign generates 75 highly-correlated timesteps. Random row-level splitting would put timesteps from the same campaign in both train and test, inflating metrics in a way that does not generalize to new campaigns.

This release uses GroupShuffleSplit by campaign_id (nested, 70/15/15):

Fold	Campaigns	Timesteps
Train	350	26,242
Validation	75	5,624
Test	75	5,623

All test campaigns are completely unseen during training. Class imbalance is addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical feature recipe. 63 features survive after encoding, drawn from:

• Per-timestep numeric (15): timestep, files_encrypted_cumulative, encryption_throughput_mbps, endpoints_compromised, lateral_move_count, credential_harvest_count, c2_bytes_exfiltrated, defender_alert_score, blast_radius_pct, living_off_land_score, attribution_risk_score, data_exfiltrated_gb, wiper_flag, double_extortion_flag, ir_activated
• Per-timestep categorical (2, one-hot): attack_phase, detection_outcome
• Victim segment (10 numeric, 3 categorical one-hot): EDR coverage, network segmentation quality, patch posture, IR latency, endpoint count, AD domain complexity, SOC maturity score, backup recovery probability, backup recovery time, SIEM cadence; segment_type, soc_maturity_tier, backup_maturity_tier
• Engineered (6): c2_intensity_score, escalation_velocity, is_destructive, dwell_efficiency, is_post_detonation, lotl_intensity_bin
• Ordinal (1): segment_id_hash (segment ID hashed to integer)

Leakage audit

Three columns were audited as potential tier oracles. None were dropped for this task:

Feature	Cross-tier ranges (mean)	Verdict
attribution_risk_score	lone 0.016 / nation_state 0.017 / organised 0.026 / raas 0.025	Overlapping; NOT an oracle. Keep.
living_off_land_score	lone 0.05 / nation_state 0.20 / organised 0.16 / raas 0.13	Mild correlation with massive overlap (std 0.08–0.25). Real observable. Keep.
attack_phase	Phase-purity vs tier is ~uniform	No oracle relationship. Keep.

detection_outcome contains a recovery_in_progress value that is 1:1 identical to the attack_phase value of the same name (purity 0.89 vs phase), but this only matters for phase prediction, not tier prediction. The column is kept as a feature for tier work.

The honest result of dropping the two candidate-leakage columns (attribution_risk_score + living_off_land_score) is a 2pp accuracy reduction — confirming they provide modest legitimate signal, not oracle leakage. They are kept in the published pipeline.

Evaluation

Test-set metrics, seed 42 (n = 5,623 timesteps from 75 disjoint campaigns)

XGBoost (the published model_xgb.json artifact)

Metric	Value
Macro ROC-AUC (OvR)	0.8736
Accuracy	0.6898
Macro-F1	0.6751
Weighted-F1	0.6939

MLP (the published model_mlp.safetensors artifact)

Metric	Value
Macro ROC-AUC (OvR)	0.8072
Accuracy	0.5118
Macro-F1	0.5121
Weighted-F1	0.5160

The MLP underperforms XGBoost on this task (a common pattern on tabular data with limited training scale). Both are published so users can pick the right tool, and disagreement between them is a useful triage signal.

Multi-seed robustness (XGBoost, 10 seeds)

Stable performance across seeds — all 10 seeds yield all 4 tiers in the test fold:

Metric	Mean	Std	Min	Max
Accuracy	0.603	0.040	0.533	0.690
Macro-F1	0.593	0.047	0.509	0.675
Macro ROC-AUC OvR	0.853	0.031	0.796	0.891

Full per-seed results in multi_seed_results.json.

Seed 42 happens to be a stronger-than-average seed (acc 0.69 vs mean 0.60). The published artifact uses seed 42 because it produces clean ROC-AUC computation; the multi-seed aggregate ROC-AUC of 0.853 ± 0.031 is the honest performance estimate.

Per-class F1 (seed 42)

Tier	Class share	XGBoost F1	MLP F1
organised_syndicate	40%	0.739	0.520
nation_state_nexus	15%	0.686	0.602
raas_affiliate	30%	0.646	0.499
lone_actor	15%	0.630	0.428

The model performs evenly across all four classes — no single tier collapses. The strongest performance on minority nation_state_nexus (F1 0.69 despite only 15% prevalence) suggests the model picks up on nation-state-specific behaviours (high LotL score, wiper deployment, sustained C2 dwell) reliably. The hardest tier is lone_actor, the behaviourally most variable class.

Ablation: which feature groups matter

Configuration	Accuracy	Macro-F1	ROC-AUC	Δ accuracy
Full feature set (published)	0.6898	0.6751	0.8736	—
No behavioural features	0.5673	0.5214	0.8107	−0.1225
No topology features	0.6146	0.6302	0.8707	−0.0752
No timestep	0.6717	0.6417	0.8673	−0.0181
No engineered features	0.6882	0.6563	0.8747	−0.0016

Four findings:

1. Behavioural features carry the most tier signal (drops 12 pp accuracy, 15 pp macro-F1 when removed). This is the most important finding: tier prediction is genuinely behaviour-driven, not a topology-lookup shortcut. Sustained C2 intensity, lateral-move velocity, wiper deployment, and LotL technique use jointly discriminate tiers.
2. Topology contributes ~7 pp accuracy. Defender posture (SOC maturity, backup tier, EDR coverage) provides useful conditioning context — actors target environments differently by tier.
3. timestep matters much less than for phase prediction (drops only ~2 pp). This is expected and good: phase prediction depends on knowing where in the lifecycle you are; tier prediction depends on how the actor operates, which is more invariant to timestep.
4. Engineered features barely contribute on their own — the trees recover most of the c2_intensity, escalation_velocity, etc. signal directly from the raw features. They remain in the pipeline as a documented baseline-feature reference.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 4 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 63 → 128 → 64 → 4, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters (learning rate, batch size, n_estimators, early-stopping patience, weight decay, class-weighting strategy) are held internally by XpertSystems and are not part of this release.

Limitations

This is a baseline reference, not a production threat-attribution system.

1. Adjacent-tier confusion is honest. The hardest discriminations are lone_actor ↔ nation_state_nexus (both small minorities, sometimes behaviourally similar in early-phase recon) and raas_affiliate ↔ organised_syndicate (operationally similar in mid-campaign). Confusion-matrix-aware downstream logic (e.g. flagging disagreement between XGBoost and MLP for analyst review) is recommended.

2. MLP weaker than XGBoost. The MLP lags ~18 pp accuracy behind XGBoost. This is a common pattern on tabular data when training set sizes don't justify deep-model parameter counts. Both are published; the recommendation is XGBoost as the primary predictor and the MLP for disagreement-as-triage signal.

3. Synthetic-vs-real transfer. The dataset is synthetic and calibrated to ransomware threat-intelligence benchmark targets (Mandiant M-Trends, CrowdStrike GTR, Coveware Quarterly, Sophos State of Ransomware, IBM CODB, Verizon DBIR, CISA #StopRansomware, Chainalysis). Real ransomware telemetry has different noise characteristics, adversary adaptation, and instrumentation gaps. Do not assume metrics transfer.

4. Adversarial robustness not evaluated. The dataset is not adversarially generated; the model has not been red-teamed against tier-spoofing campaigns (a real attacker may deliberately mimic another tier's TTPs to evade attribution).

5. Per-tier sample sizes are still modest. lone_actor and nation_state_nexus have only 75 training campaigns each. The full ~5,500-campaign CYB005 product (with ~825 per minority tier) would tighten the per-class confidence intervals materially.

Notes on dataset schema

The CYB005 sample dataset README describes some fields differently from the actual schema. The model was trained on the actual schema; this note helps buyers reconcile what they read with what they receive.

What the README says	What the data actually contains
"7 attack phases" (initial_access, persistence, privilege_escalation, lateral_movement, data_exfiltration, encryption_deployment, ransom_demand)	8 attack phases: initial_access, internal_recon, privilege_escalation, lateral_movement, exfiltration_staging, encryption_detonation, ransom_negotiation, recovery_in_progress. (No persistence phase as a distinct value; recovery_in_progress is the dominant phase at 35% of rows because campaigns run beyond detonation.)
Backup tiers include cloud_replicated, immutable_object_lock	Backup tiers in the actual data use offsite_unverified, offsite_verified_immutable for those concepts
Summary has campaign_outcome, dwell_time_pre_detonation_hrs	Neither field exists. Use total_dwell_time_hrs and campaign_success_flag / detection_phase instead
Per-timestep includes endpoints_compromised, lateral_pivots, edr_alerted, siem_correlated, lotl_technique_used, vss_deletion_attempted, wiper_component_deployed, dwell_hours, c2_beacon_active, backup_maturity_tier	Actual per-timestep columns: endpoints_compromised ✓, lateral_move_count (not pivots), no edr_alerted/siem_correlated/vss_deletion_attempted/dwell_hours/c2_beacon_active; defender_alert_score and attribution_risk_score exist instead; backup_maturity_tier is only on per-campaign victim_topology, not per-timestep

None of these discrepancies affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

• Evaluating fit of the CYB005 dataset for your threat-attribution or ransomware-research work
• Baseline reference for new model architectures (especially sequence models, which should beat this baseline by leveraging temporal context across the 75-step campaign)
• Teaching and demo for multi-class tabular classification on cybersecurity telemetry
• Feature engineering reference for ransomware campaign attribution

Out-of-scope use

• Production threat-actor attribution on real ransomware campaigns
• Incident-response decision-making on real systems
• Adversarial-evasion evaluation (dataset not adversarially generated)
• Any operational security or law-enforcement decision

Reproducibility

Outputs above were produced with seed = 42 (published artifact), group-aware nested GroupShuffleSplit (70/15/15 by campaign_id), on the published sample (xpertsystems/cyb005-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits.

The training script itself is private to XpertSystems.

Files in this repo

File	Purpose
model_xgb.json	XGBoost weights (seed 42)
model_mlp.safetensors	PyTorch MLP weights (seed 42)
feature_engineering.py	Feature pipeline (load → join topology → engineer → encode)
feature_meta.json	Feature column order + categorical levels
feature_scaler.json	MLP input mean/std (XGBoost ignores)
validation_results.json	Per-class metrics, confusion matrix, architecture
ablation_results.json	Per-feature-group ablation
multi_seed_results.json	XGBoost metrics across 10 seeds with aggregate statistics
inference_example.ipynb	End-to-end inference demo notebook
README.md	This file

Contact and full product

The full CYB005 dataset contains ~358,000 rows across four files, with calibrated benchmark validation against 12 metrics drawn from authoritative ransomware threat-intelligence sources (Mandiant M-Trends, CrowdStrike GTR, Coveware Quarterly Ransomware Report, Sophos State of Ransomware, IBM CODB, Verizon DBIR, CISA #StopRansomware, Chainalysis). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 pradeep@xpertsystems.ai
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb005-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
  • https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)
  • https://huggingface.co/xpertsystems/cyb004-baseline-classifier (phishing campaign phase)

Citation

@misc{xpertsystems_cyb005_baseline_2026,
  title  = {CYB005 Baseline Classifier: XGBoost and MLP for Ransomware Actor-Tier Attribution},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb005-baseline-classifier},
  note   = {Baseline reference model trained on xpertsystems/cyb005-sample}
}