Fathom turns raw sandbox reports (CAPE, Joe Sandbox, etc.) into high-quality ATT&CK-mapped malware analysis. It outperforms general-purpose models on cybersecurity tasks while remaining fully open-source and runnable on a single AMD MI300X / A100 80GB.
Model Overview
Base: Mixtral-8x7B-Instruct-v0.1 (full bf16, no quantization)
Hardware: AMD MI300X (205.8 GB VRAM) — full bf16 training
Key Innovation: Evidence extraction layer + structured behavioral prompts → 9× improvement in real ATT&CK mapping
Designed for:
Malware analysts & threat hunters
SOC / DFIR teams
CAPE / sandbox report enrichment
Automated ATT&CK technique extraction
Benchmark Results
All results use the real Fathom pipeline ([INST] chat template + 8192 context + structured evidence from CAPE extraction layer v3). Greedy decoding, bf16.
1. General Cybersecurity Knowledge (vs. Closed & Open Models)
This is the original Run-7 breakdown. The n=12 run below re-scores samples 12/15/16 as part
of its validation gate; small per-sample differences (e.g. sample 15) vs. this table reflect
run-to-run generation variance, not a data conflict — both runs used the identical pipeline,
prompt, and generation config.
n=12 expansion (preliminary): the same pipeline and generation config, run on 12 real
CAPE-detonated samples across 5 malware families (Emotet, Formbook, Dridex, Conti, Agent
Tesla). Exact F1 0.408, Parent F1 0.514 — lower than the n=3 headline above, as expected:
those three were unusually clean samples, and a broader sample set is a more credible signal.
Ground truth for 9 of the 12 samples is auto-derived from sandbox behavior and not yet
hand-verified, so this number is not yet final. Full per-sample results, raw predictions, and
logs: journal-artifacts/evaluation/
in the source repo.
Rigorous ground-truth F1 (23 test cases): Exact = 0.184, Parent = 0.344 (synthetic); real CAPE = 0.841 after pipeline fixes
5. Key Discovery: Mal-API-2019 Analysis
We evaluated Fathom on the public Mal-API-2019 dataset (Catak & Yazı, arXiv:1905.01999) — 7,107 API call sequences from Cuckoo Sandbox.
Variant
Accuracy
Macro F1
Raw API sequences
12.6%
0.030
Filtered behavioral groups
10.9%
0.052
Insight:
Raw API sequences alone are insufficient for reliable family classification. The dataset contains heavy loader noise and families share nearly identical behavioral APIs. Ground-truth labels come from static AV signatures, not behavioral semantics.
“ In contrast, Fathom’s full evidence extraction pipeline achieves 0.841 Parent F1 on real CAPEv2 reports. This demonstrates that structured behavioral evidence + multi-source context (not raw API text) is the critical enabler for production-grade malware analysis.”
CyberMetric questions overlap slightly with the training corpus (measured, disclosed, and
shown not to inflate scores — see the paper). To settle the question independently, Fathom was
also evaluated on two benchmarks that share no data with CyberMetric, MMLU, or the training
corpus (verified with an 8-gram shingle-overlap check, <0.3% overlap on every subset scored):
SECURE (Bhusal et al., 2024) — ATT&CK technique and
CWE weakness mapping, the same task Fathom is built for, with a published base Mixtral-8x7B
baseline for direct comparison:
Subset
Fathom unified-v2
Published base Mixtral-8x7B
Delta
MAET (ATT&CK technique MCQ, n=1072)
87.78%
80.9%
+6.9pp
CWET (CWE weakness MCQ, n=965)
87.88%
83.4%
+4.5pp
SECURE's other two MCQ subsets, KCV and VOOD (CVE-severity boolean prediction — a different
task from Fathom's ATT&CK/malware-analysis focus), were scored but excluded from this
comparison: the official task grounds each question in an embedded CVE-JSON record long
enough to exceed this serving deployment's batched-token limit, which broke that grounding.
Scores without it aren't a fair comparison to the published baseline, so they're reported
separately in the raw results rather than here.
CyberSOCEval (Meta CyberSecEval 4) — multi-select
reasoning over real sandbox detonation logs and CTI reports, no published Mixtral baseline
exists yet:
Subset
Exact-set accuracy
Mean Jaccard overlap
Malware-analysis (n=609)
10.18%
0.415
Threat-intel reasoning (n=588)
27.04%
0.555
For context, the paper's conservative 20-case synthetic-suite lower bound is Parent F1
0.344 — the independent-benchmark numbers above sit alongside that as additional,
non-overlapping evidence, not a replacement for it.
1@misc{fathom2026,
2 title={Fathom: Expert Cybersecurity Analysis with Mixtral LoRA Adapters},
3 author={Abdul Hadi and Muhammad Haseeb and Muhammad Ammar and Sana Aurangzeb},
4 year={2026},
5 howpublished={\url{https://huggingface.co/umer07/fathom-mixtral}},
6}
