File size: 2,089 Bytes
aad0df8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
tags:
- cybersecurity
- industrial-control-systems
- bert
- from-scratch
- synthetic-data
---
# Protocol Guardian
Protocol Guardian is a tiny BERT-compatible classifier trained from scratch to rank
industrial-control commands as routine or hazardous. It uses SNIP's 512-token
byte-level BPE vocabulary but does not inherit language-model weights.
The project also trains a transfer-learning variant from Google's two-layer,
128-hidden-dimension BERT miniature hosted on the Hugging Face Hub.
The final test contains command templates never present in training or validation.
This measures phrasing transfer within a synthetic domain rather than memorization of
identical strings.
## Reproduce
```powershell
uv run python projects/protocol-guardian/generate_data.py
uv run python projects/protocol-guardian/train.py
uv run python projects/protocol-guardian/train_pretrained.py
uv run python projects/protocol-guardian/train_embedding.py
```
This is a research classifier, not a substitute for deterministic safety controls,
authorization checks, or an engineering review.
## Verified results
The benchmark evolved through three increasingly strict template regimes. The final
split contains 1,200 examples from templates absent from both training and semantic
validation.
| Variant | Held-out accuracy | Precision | Recall | F1 |
| --- | ---: | ---: | ---: | ---: |
| 118K BERT from scratch, first corpus | 65.92% | 97.51% | 32.67% | 0.4894 |
| 118K BERT from scratch, broader corpus | 40.75% | 42.34% | 51.17% | 0.4634 |
| Google two-layer BERT transfer | 50.00% | 0.00% | 0.00% | 0.0000 |
| MiniLM embeddings + linear head | **88.58%** | **100.00%** | **77.17%** | **0.8711** |
The winning head has only 385 fitted parameters over 384-dimensional normalized
embeddings from `sentence-transformers/all-MiniLM-L6-v2`. Its test confusion matrix
was `[[600, 0], [137, 463]]`. The failed transformer variants are retained to document
how misleading same-template validation can be.
|