cti-ner-securebert (clean LLM-gold retrain, v2)
SecureBERT (ehsanaghaei/SecureBERT, RoBERTa architecture, domain-pretrained
on ~2M security documents) fine-tuned as a BIO token classifier for cyber
threat intelligence NER on CISA advisories: 16 entity types (attack technique
id/name, CVE/CWE identifiers, file hash/name/path, IP address, email address,
malware family, threat actor group, hacking tool, tactic, TLP classification,
YARA rule name, affected software product).
This version replaces the earlier checkpoint trained on dictionary-silver
labels. Training gold is LLM-proposed (Claude Sonnet 5, exhaustive
extraction under per-label span-format validators) with the evaluation split
human-curated. See the paper in the repository below for the methodology and
the circularity analysis that motivated the switch.
Evaluation (measured on this checkpoint)
Document-level span scoring on the held-out 8-advisory test split
(strict = exact span + label):
| Metric | Strict | Relaxed |
|---|
| Micro F1 | 0.71 | 0.76 |
| Macro F1 | 0.74 | 0.77 |
Best of the four fine-tuned architectures studied (RoBERTa-base 0.659,
RoBERTa-large 0.648, GLiNER 0.560 on the identical gold and harness).
Per-label strict F1 highlights: attack technique id 0.99, cwe identifier
1.00, tlp classification 0.97, email address 0.98, cve identifier 0.95,
ip address 0.91; hardest labels remain threat actor group 0.32 and hacking
tool 0.48.
Training details
- Data: 66 training advisories (2,799 chunks), 8 val, 8 test; document-level
split committed in the repository (
data/gold_v2/split_full.json).
- 8 epochs, lr 2e-5, batch size 16, seed 0, wordpiece-budget sliding
windows; best epoch by validation strict F1 (epoch 7, val 0.664).
- Trained on Apple Silicon (MPS). Reproduces the paper's CUDA-trained
result (0.712 strict micro) within seed/hardware variance; the paper's
committed numbers come from
data/gold_v2/results/trained-securebert.json.
Usage
Inference should replicate the sliding-window word-tagging procedure in
ner/finetune_roberta.py (see notebooks/hf_models_demo.ipynb in the
repository for a worked example); a naive single-pass tokenizer call
truncates long, hash-heavy advisory text.
Links
- Code, gold data, paper: https://github.com/dandye/nlp_capstone
- Companion models:
pzryathzsdhc/cti-ner-roberta-large,
pzryathzsdhc/cti-ner-gliner
