Gemma4Defense-2B — Model Card

Model Information

Intended Use

Intended Use Cases

• CWE classification — mapping vulnerability descriptions (CVEs, advisories) to MITRE CWE categories
• Cyber threat intelligence Q&A — answering structured questions about cybersecurity concepts, attacks, controls
• Defensive analysis assistants — supporting human analysts who triage CVEs, prioritize patches, or document threat-actor behavior
• Cybersecurity benchmarking — as a reference for compact-model performance on CTI-Bench RCM/MCQ subsets

Downstream Use

• Security operations center (SOC) ticket triage tools that suggest a likely CWE for an incoming CVE
• Vulnerability management dashboards that pre-classify CVE feeds before human review
• Educational tutoring assistants for cybersecurity coursework grounded in CTI-Bench-style content
• Internal cyber knowledge bases / chat assistants for security teams

Out-of-Scope Use

1. Generating harmful content — the model must not be used to produce exploit code, weaponized proof-of-concept payloads, attacker tradecraft, or instructions that materially aid offensive operations.
2. Critical security decisions without human oversight — the model should not auto-execute remediation, blocklist updates, account lockouts, or any action whose reversal carries cost; outputs are advisory and require qualified human review.
3. Legal or medical advice — the model is trained on cybersecurity domain content and is not appropriate for legal, medical, or other regulated-advice contexts.
4. Non-security use cases — general chat, code generation, summarization, translation, or other domains outside its specialization will produce lower-quality output than purpose-built models.
5. Violation of laws or regulations — including but not limited to unauthorized vulnerability scanning, illegal data access, or misuse contrary to applicable cybersecurity statutes (CFAA, GDPR, etc.).

Hardware Requirements

• "Per-token effective" parameters reflect Gemma-4's Per-Layer Embedding architecture: ~2.3 B parameters activate per token, but the full ~5 B weight matrix must be resident in VRAM during inference. The compute cost at inference scales with the per-token effective count.
• Compute (FLOPs / token) is approximately proportional to the per-token effective parameter count at fixed context length, so per-token inference cost is roughly 0.29× that of an 8 B model.
• Quantized variants (int8, int4) further reduce VRAM by ~½ and ~¼ respectively. The released checkpoint is bf16 only; community quantization is not validated by the authors of this release.

How to Get Started with the Model

Training and Evaluation

Training Data

• CTI-RCM 2021 (decontaminated) — CVE → CWE classification examples drawn from MITRE/NVD public records dated 2021. Items appearing in the CTI-Bench evaluation splits were explicitly removed prior to training. (~6,776 records)
• CVE / CTI synthetic Q&A — defensive-analyst-style cyber question–answer pairs grounded in CVE descriptions, designed to teach domain reasoning while preserving terse-answer formats. (~5,776 records)

Methodology

• Direct SFT, not knowledge distillation. Knowledge-distillation variants from a larger 20B teacher model (CyberPal-2.0-20B) were evaluated during recipe development. At the corpus sizes tested (≤ 15K supervised records), direct SFT on the curated corpus outperformed distillation on the headline benchmarks. The released model is direct SFT only.
• Decontaminated training data. An earlier internal iteration showed ~72% test-set overlap when trained on undeduplicated CTI corpora, producing inflated CTI-RCM scores that did not generalize. The released model trains exclusively on the 2021 cohort with CTI-Bench overlap items removed.
• Instruction-tuned base, not pre-trained base. Direct SFT on the IT checkpoint preserves the existing format priors (terse-answer multiple-choice convention) better than SFT on the pre-trained base; comparable runs on base checkpoints showed substantial CTI-MCQ format-binding decay (~−14 to −38 pp in the worst case) at the same corpus scale.
• Multi-trial benchmarking. All headline numbers are means of 5 independent trials with random sampling seeds at temperature 0.3; standard deviations are reported alongside.
• Cross-substrate validation. The identical training corpus and hyperparameters were independently applied to a separate 4B instruction-tuned base in a different model family; the two runs converge to within 0.9 points on CTI-RCM, providing a built-in robustness check that the result is recipe-driven rather than substrate-specific.

Training Setup

Evaluation

Headline result

Pre / post fine-tune comparison

Comparison to other cybersecurity-relevant models we evaluated

Key highlights

• Beats Foundation-Sec-Instruct-8B on CTI-MCQ by +10.5 points at approximately one-quarter the parameter count.
• Stays within ~1 point of Foundation-Sec-Instruct-8B on CTI-RCM under the same evaluation protocol.
• The identical recipe applied to a separate 4B instruction-tuned base in a different model family reproduces the CTI-RCM result within 0.9 points — a built-in robustness check that the result is recipe-driven, not substrate-specific.
• Independent reproduction of CyberPal-2.0-20B at the Foundation-Sec protocol confirms its CTI-MCQ accuracy within 2 points of its paper claim.

Limitations

1. Domain-specific knowledge limitations. The model is trained on cybersecurity domain text and is not a general assistant. Tasks outside this domain will produce lower-quality output than purpose-built general models.
2. Time-anchored training data. The CTI-RCM training cohort is drawn from 2021 records. Vulnerability classes that emerged or rose in prevalence after 2021 (e.g., AI/ML-specific weaknesses, recent supply-chain CWEs) are under-represented in training and will be classified less accurately.
3. English-only. All training and evaluation data are in English; multilingual cyber tasks will degrade.
4. CTI-RCM gap. Foundation-Sec-Instruct-8B remains slightly stronger on CTI-RCM under this protocol (-1.0 point gap, within multi-trial measurement noise but still real). Production deployments where CWE classification is the primary metric should benchmark both models on their specific input distribution.
5. No safety RLHF. The model is supervised-fine-tuned only; the training data emphasizes defensive-analyst framing but no formal reinforcement-learning safety alignment was applied.
6. Multimodal architecture inherited. Gemma-4 ships as a multimodal base with vision and audio towers. This release contains only the text-language-model weights extracted post-merge; downstream tooling that expects the multimodal config should consume the published Gemma4ForCausalLM config (already declared in the repo).

Recommendations

1. Always have qualified security professionals review model outputs before implementation for any operational use case (patch prioritization, ticket routing, blocklisting).
2. Use this model as an assistive tool rather than a replacement for expert human judgment, especially for novel vulnerability classes outside the 2021 training cohort.
3. Validate on your own input distribution before deployment. Public CTI-Bench performance does not perfectly transfer to internal advisory feeds, vendor-proprietary CWE taxonomies, or non-English content.
4. Monitor for drift. As new CVE / CWE patterns emerge, periodically re-evaluate; consider supplementing with retrieval over a current vulnerability knowledge base for time-sensitive queries.
5. Apply standard prompt-injection mitigations when wrapping the model in agentic workflows that accept external content (advisory feeds, scraped pages); domain-SFT does not confer prompt-injection resistance.

Citation