---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
tags:
- cybersecurity
- industrial-control-systems
- bert
- from-scratch
- synthetic-data
---

# Protocol Guardian

Protocol Guardian is a tiny BERT-compatible classifier trained from scratch to rank
industrial-control commands as routine or hazardous. It uses SNIP's 512-token
byte-level BPE vocabulary but does not inherit language-model weights.

The project also trains a transfer-learning variant from Google's two-layer,
128-hidden-dimension BERT miniature hosted on the Hugging Face Hub.

The final test contains command templates never present in training or validation.
This measures phrasing transfer within a synthetic domain rather than memorization of
identical strings.

## Reproduce

```powershell
uv run python projects/protocol-guardian/generate_data.py
uv run python projects/protocol-guardian/train.py
uv run python projects/protocol-guardian/train_pretrained.py
uv run python projects/protocol-guardian/train_embedding.py
```

This is a research classifier, not a substitute for deterministic safety controls,
authorization checks, or an engineering review.

## Verified results

The benchmark evolved through three increasingly strict template regimes. The final
split contains 1,200 examples from templates absent from both training and semantic
validation.

| Variant | Held-out accuracy | Precision | Recall | F1 |
| --- | ---: | ---: | ---: | ---: |
| 118K BERT from scratch, first corpus | 65.92% | 97.51% | 32.67% | 0.4894 |
| 118K BERT from scratch, broader corpus | 40.75% | 42.34% | 51.17% | 0.4634 |
| Google two-layer BERT transfer | 50.00% | 0.00% | 0.00% | 0.0000 |
| MiniLM embeddings + linear head | **88.58%** | **100.00%** | **77.17%** | **0.8711** |

The winning head has only 385 fitted parameters over 384-dimensional normalized
embeddings from `sentence-transformers/all-MiniLM-L6-v2`. Its test confusion matrix
was `[[600, 0], [137, 463]]`. The failed transformer variants are retained to document
how misleading same-template validation can be.