| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-classification |
| tags: |
| - cybersecurity |
| - industrial-control-systems |
| - bert |
| - from-scratch |
| - synthetic-data |
| --- |
| |
| # Protocol Guardian |
|
|
| Protocol Guardian is a tiny BERT-compatible classifier trained from scratch to rank |
| industrial-control commands as routine or hazardous. It uses SNIP's 512-token |
| byte-level BPE vocabulary but does not inherit language-model weights. |
|
|
| The project also trains a transfer-learning variant from Google's two-layer, |
| 128-hidden-dimension BERT miniature hosted on the Hugging Face Hub. |
|
|
| The final test contains command templates never present in training or validation. |
| This measures phrasing transfer within a synthetic domain rather than memorization of |
| identical strings. |
|
|
| ## Reproduce |
|
|
| ```powershell |
| uv run python projects/protocol-guardian/generate_data.py |
| uv run python projects/protocol-guardian/train.py |
| uv run python projects/protocol-guardian/train_pretrained.py |
| uv run python projects/protocol-guardian/train_embedding.py |
| ``` |
|
|
| This is a research classifier, not a substitute for deterministic safety controls, |
| authorization checks, or an engineering review. |
|
|
| ## Verified results |
|
|
| The benchmark evolved through three increasingly strict template regimes. The final |
| split contains 1,200 examples from templates absent from both training and semantic |
| validation. |
|
|
| | Variant | Held-out accuracy | Precision | Recall | F1 | |
| | --- | ---: | ---: | ---: | ---: | |
| | 118K BERT from scratch, first corpus | 65.92% | 97.51% | 32.67% | 0.4894 | |
| | 118K BERT from scratch, broader corpus | 40.75% | 42.34% | 51.17% | 0.4634 | |
| | Google two-layer BERT transfer | 50.00% | 0.00% | 0.00% | 0.0000 | |
| | MiniLM embeddings + linear head | **88.58%** | **100.00%** | **77.17%** | **0.8711** | |
|
|
| The winning head has only 385 fitted parameters over 384-dimensional normalized |
| embeddings from `sentence-transformers/all-MiniLM-L6-v2`. Its test confusion matrix |
| was `[[600, 0], [137, 463]]`. The failed transformer variants are retained to document |
| how misleading same-template validation can be. |
|
|
