Protocol Guardian
Protocol Guardian is a tiny BERT-compatible classifier trained from scratch to rank
industrial-control commands as routine or hazardous. It uses SNIP's 512-token
byte-level BPE vocabulary but does not inherit language-model weights.
The project also trains a transfer-learning variant from Google's two-layer,
128-hidden-dimension BERT miniature hosted on the Hugging Face Hub.
The final test contains command templates never present in training or validation.
This measures phrasing transfer within a synthetic domain rather than memorization of
identical strings.
Reproduce
1uv run python projects/protocol-guardian/generate_data.py
2uv run python projects/protocol-guardian/train.py
3uv run python projects/protocol-guardian/train_pretrained.py
4uv run python projects/protocol-guardian/train_embedding.py
This is a research classifier, not a substitute for deterministic safety controls,
authorization checks, or an engineering review.
Verified results
The benchmark evolved through three increasingly strict template regimes. The final
split contains 1,200 examples from templates absent from both training and semantic
validation.
| Variant | Held-out accuracy | Precision | Recall | F1 |
|---|
| 118K BERT from scratch, first corpus | 65.92% | 97.51% | 32.67% | 0.4894 |
| 118K BERT from scratch, broader corpus | 40.75% | 42.34% | 51.17% | 0.4634 |
| Google two-layer BERT transfer | 50.00% | 0.00% | 0.00% | 0.0000 |
| MiniLM embeddings + linear head | 88.58% | 100.00% | 77.17% | 0.8711 |
The winning head has only 385 fitted parameters over 384-dimensional normalized
embeddings from sentence-transformers/all-MiniLM-L6-v2. Its test confusion matrix
was [[600, 0], [137, 463]]. The failed transformer variants are retained to document
how misleading same-template validation can be.