CyberSecQwen-4B — Model Card
🏆
AMD Developer Hackathon submission. Full project writeup, demo video, and judging context at
lablab.ai/ai-hackathons/amd-developer/athena19/cybersecqwen-4b-cti-specialist-fine-tuned-on-amd.
Model Information
CyberSecQwen-4B is a 4B-parameter language model specialized for defensive cybersecurity tasks, fine-tuned from
Qwen3-4B-Instruct-2507. It is purpose-built for two evaluation skills measured by
CTI-Bench: mapping CVE descriptions to their CWE category (CTI-RCM) and answering cyber threat intelligence multiple-choice questions (CTI-MCQ).
Under the evaluation protocol of
Foundation-Sec-8B (arXiv:2504.21039), CyberSecQwen-4B retains
97.3% of Foundation-Sec-Instruct-8B's CTI-RCM accuracy while exceeding its CTI-MCQ by
+8.7 points, at half the parameter count.
The full training, merge, and evaluation pipeline runs end-to-end on a single AMD Instinct MI300X 192GB instance using ROCm + vLLM + FlashAttention-2. A companion model trained with the same recipe on Gemma-4-E2B-it —
Gemma4Defense-2B — converges to the same CTI-RCM accuracy within 0.9 points (0.6754 vs 0.6664), demonstrating that the result is recipe-driven rather than substrate-specific.
| |
|---|
| Base model | Qwen/Qwen3-4B-Instruct-2507 |
| Parameters | 4.0B total (3.6B non-embedding) |
| Architecture | Qwen3 (RoPE, GQA 32:8, head_dim=128, 36 layers) |
| Context length | 32,768 native |
| Adapter | LoRA r=64, alpha=64, dropout=0.05 |
| Precision | bfloat16 |
| Languages | English |
| License | Apache 2.0 |
Intended Use
Intended Use Cases
CyberSecQwen-4B is intended for security practitioners, researchers, and engineers working on:
- CWE classification — mapping vulnerability descriptions (CVEs, advisories) to MITRE CWE categories
- Cyber threat intelligence Q&A — answering structured questions about cybersecurity concepts, attacks, controls
- Defensive analysis assistants — supporting human analysts who triage CVEs, prioritize patches, or document threat-actor behavior
- Cybersecurity benchmarking on AMD hardware — as a reference fine-tune for the AMD MI300X stack and a comparator for compact-model performance on CTI-Bench
Downstream Use
The model can be used as a building block in:
- Security operations center (SOC) ticket triage tools that suggest a likely CWE for an incoming CVE
- Vulnerability management dashboards that pre-classify CVE feeds before human review
- Internal cyber knowledge bases / chat assistants for security teams
- Reference deployments demonstrating CTI workloads on AMD MI300X via vLLM ROCm
Out-of-Scope Use
The following uses are out-of-scope and are neither recommended nor intended use cases:
- Generating harmful content — the model must not be used to produce exploit code, weaponized proof-of-concept payloads, attacker tradecraft, or instructions that materially aid offensive operations.
- Critical security decisions without human oversight — the model should not auto-execute remediation, blocklist updates, account lockouts, or any action whose reversal carries cost; outputs are advisory and require qualified human review.
- Legal or medical advice — the model is trained on cybersecurity domain content and is not appropriate for legal, medical, or other regulated-advice contexts.
- Non-security use cases — general chat, code generation, summarization, translation, or other domains outside its specialization will produce lower-quality output than purpose-built models.
- Violation of laws or regulations — including but not limited to unauthorized vulnerability scanning, illegal data access, or misuse contrary to applicable cybersecurity statutes (CFAA, GDPR, etc.).
Hardware Requirements
The numbers below are first-principles estimates from the bf16 weight footprint plus typical KV-cache overhead at the trained 4096-token context. They are not measured throughput numbers; for production deployment, profile against your specific traffic pattern.
| Specification | CyberSecQwen-4B | Foundation-Sec-Instruct-8B (reference) |
|---|
| Parameters (total / non-embedding) | 4.0 B / 3.6 B | 8 B |
| bf16 weight file on disk | ~8.0 GB | ~16 GB |
| Inference VRAM, weights only (bf16) | ~8 GB | ~16 GB |
| Inference VRAM, weights + 4 K KV cache (bf16) | ~9–10 GB | ~17–18 GB |
| Single-GPU class (bf16, headroom for batch ≥ 1) | Fits on any 12 GB+ consumer card | Typically requires a 24 GB+ datacenter card |
| AMD Instinct MI300X 192 GB (validated) | Fits trivially with very large batch / long context | Fits trivially |
Notes:
- Compute (FLOPs / token) is approximately proportional to the parameter count at fixed context length, so per-token inference cost is roughly 0.50× that of an 8 B model.
- Quantized variants (int8, int4) further reduce VRAM by ~½ and ~¼ respectively. The released checkpoint is bf16 only; community quantization is not validated by the authors of this release.
- This model has been validated end-to-end on AMD Instinct MI300X via vLLM ROCm + FlashAttention-2; consult the "How to Get Started" section below for the exact serving command on AMD hardware.
How to Get Started with the Model
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model_id = "athena129/CyberSecQwen-4B"
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(
7 model_id,
8 torch_dtype=torch.bfloat16,
9 device_map="auto",
10)
11
12cve = ("A deserialization vulnerability in the destruct() function of Laravel "
13 "v8.5.9 allows attackers to execute arbitrary commands.")
14
15messages = [{
16 "role": "user",
17 "content": (
18 "Analyze the following CVE description and map it to the appropriate CWE. "
19 "Provide a brief justification for your choice. "
20 "Ensure the last line of your response contains only the CWE ID.\n\n"
21 f"CVE Description: {cve}"
22 ),
23}]
24prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
25inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
26output = model.generate(**inputs, max_new_tokens=256, temperature=0.3, do_sample=True)
27print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Serving via vLLM on AMD MI300X
1docker run --rm --network=host --device=/dev/kfd --device=/dev/dri \
2 -e VLLM_ROCM_USE_AITER=1 -e TORCH_BLAS_PREFER_HIPBLASLT=1 \
3 vllm/vllm-openai-rocm:latest \
4 --model athena129/CyberSecQwen-4B \
5 --served-model-name cybersecqwen-4b \
6 --attention-backend TRITON_ATTN \
7 --dtype bfloat16 \
8 --max-model-len 4096 \
9 --gpu-memory-utilization 0.9
Training and Evaluation
Training Data
The model was trained on a combined cybersecurity corpus of approximately 14,776 supervised records:
- CTI-RCM 2021 (decontaminated) — CVE → CWE classification examples drawn from MITRE/NVD public records dated 2021. Items appearing in the CTI-Bench evaluation splits were explicitly removed prior to training. (~6,776 records)
- CVE / CTI synthetic Q&A — defensive-analyst-style cyber question–answer pairs grounded in CVE descriptions. (~8,000 records)
Decontamination matters here: an earlier internal version of this work showed roughly 72% test-set overlap when trained on undeduplicated CTI corpora, producing inflated CTI-RCM scores that did not generalize. The released model trains exclusively on the 2021 cohort with overlap items removed.
Methodology
This model uses direct supervised fine-tuning (SFT) of an instruction-tuned base via LoRA. The training recipe was selected through a controlled-experiment series across multiple trained variants spanning two model families and several corpus compositions, with multi-trial benchmark validation locking the released hyperparameters.
Key methodological choices that informed the released recipe:
- Direct SFT, not knowledge distillation. Knowledge-distillation variants from a larger 20B teacher model (CyberPal-2.0-20B) were evaluated during recipe development. At the corpus sizes tested (≤ 15K supervised records), direct SFT on the curated corpus outperformed distillation on the headline benchmarks. The released model is direct SFT only.
- Decontaminated training data. An earlier internal iteration showed ~72% test-set overlap when trained on undeduplicated CTI corpora, producing inflated CTI-RCM scores that did not generalize. The released model trains exclusively on the 2021 cohort with CTI-Bench overlap items removed.
- Instruction-tuned base, not pre-trained base. Direct SFT on the IT checkpoint preserves the existing format priors (terse-answer multiple-choice convention) better than SFT on the pre-trained base; comparable runs on base checkpoints (Qwen3-4B-Base + identical recipe) showed substantial CTI-MCQ format-binding decay at the same corpus scale.
- Recipe portability across substrates was an explicit design goal. The same corpus + hyperparameters were applied independently to Gemma-4-E2B-it (Gemma4Defense-2B). Both models converge to within 0.9 points on CTI-RCM, providing a built-in robustness check that the result is recipe-driven rather than substrate-specific.
- Multi-trial benchmarking. All headline numbers are means of 5 independent trials with random sampling seeds at temperature 0.3; standard deviations are reported alongside.
- AMD MI300X end-to-end pipeline. Training, adapter merging, and evaluation all run on a single AMD Instinct MI300X 192 GB instance via PyTorch + ROCm + Hugging Face transformers + PEFT + TRL inside the official vLLM ROCm Docker image. FlashAttention-2 is enabled in training for forward-and-backward passes; vLLM serves with TRITON_ATTN backend for inference.
Training Setup
| Hyperparameter | Value |
|---|
| Adapter | LoRA, r=64, alpha=64, dropout=0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Learning rate | 5e-5 |
| Schedule | cosine, warmup_ratio=0.05 |
| Weight decay | 0.01 |
| Per-device batch size | 2 |
| Gradient accumulation | 8 (effective batch = 16) |
| Epochs | 10 |
| Max sequence length | 4096 |
| Precision | bfloat16 |
| Attention implementation | flash_attention_2 |
| Random seed | 42 |
The base model was Qwen3-4B-Instruct-2507, an instruction-tuned variant with Apache 2.0 licensing. Training was performed end-to-end on a single AMD Instinct MI300X 192GB instance via the AMD Developer Cloud, using PyTorch + ROCm 7 + Hugging Face transformers, peft, and trl 0.29.1 inside the official vllm/vllm-openai-rocm Docker image.
FlashAttention-2 is enabled because Qwen3-4B's attention head dimension (128) fits within the gfx942 shared-memory budget on AMD MI300X — the same FA2 approach is not viable on Gemma-4 due to its 512 head_dim on global-attention layers, which is why the companion Gemma4Defense-2B trains with sdpa instead.
Evaluation
Evaluated under the
Foundation-Sec-8B protocol (arXiv:2504.21039 §B.3-B.4): zero-shot for instruction-tuned models, 5-shot for pretrained base models, dataset's own
Prompt column as the user message, no system prompt, temperature 0.3, max-tokens 512, concurrency 32. Reported numbers are the mean of
5 independent trials with random sampling seeds; standard deviations are reported alongside.
Headline result
| Benchmark | Metric | CyberSecQwen-4B | Foundation-Sec-Instruct-8B | Δ |
|---|
| CTI-MCQ (2,500 items) | strict_acc, 5-trial mean ± std | 0.5868 ± 0.0029 | 0.4996 | +8.7 pp |
| CTI-RCM (1,000 items) | strict_acc, 5-trial mean ± std | 0.6664 ± 0.0023 | 0.6850 | -1.9 pp |
Parseable rates were 100% on CTI-RCM and 98.1% on CTI-MCQ — the model produces well-formed outputs in the expected response convention.
Pre / post fine-tune comparison
The improvement attributable to this fine-tune over its starting checkpoint:
| Stage | CTI-RCM | CTI-MCQ |
|---|
| Qwen3-4B-Instruct-2507 (raw, instruction-tuned base) | 0.519 | 0.473 |
| CyberSecQwen-4B (this fine-tune) | 0.6664 | 0.5868 |
| Lift | +15.1 pp | +12.0 pp |
Qwen3-4B-Instruct-2507's raw CTI-MCQ score (0.473) is substantially lower than its corresponding base model's score (0.667) under the chat-template evaluation — the same instruction-tuning-collapses-MCQ effect we observe for Foundation-Sec-Instruct (-15.6 pp vs Foundation-Sec base). This fine-tune recovers and exceeds the IT starting point on both subsets, restoring most of the MCQ format binding the instruction tuning eroded while delivering a substantial CTI-RCM lift.
Comparison to other cybersecurity-relevant models we evaluated
All numbers below were measured by us under the protocol above (with the noted shot count), not quoted from third-party papers. CyberPal-2.0-20B numbers reflect a single-trial run at our protocol — its own paper reports 0.874 / 0.757 using a different prompt template (Figure 11 of arXiv:2510.14113); the +2pp MCQ match validated our harness, while the RCM gap likely reflects the template difference.
| Model | Size | CTI-RCM | CTI-MCQ | Notes |
|---|
| Foundation-Sec-8B (base) | 8B | 0.745 | 0.655 | 5-shot pretrained reference |
| Foundation-Sec-Instruct-8B | 8B | 0.685 | 0.500 | 0-shot, our TARGET |
| CyberPal-2.0-20B (cyber-pal-security/CyberOss-2.0-20B) | 20B | 0.728* | 0.738* | independently verified at our protocol |
| CyberSecQwen-4B (this model) | 4B | 0.6664 ± 0.0023 | 0.5868 ± 0.0029 | 5-trial mean ± std |
| Gemma4Defense-2B (companion) | 2.3B | 0.6754 ± 0.0035 | 0.6042 ± 0.0090 | same recipe, different substrate |
| Qwen3-4B-Instruct-2507 (raw) | 4B | 0.519 | 0.473 | 0-shot, our base |
| Qwen3-4B-Base (raw) | 4B | 0.517 | 0.667 | 5-shot |
| Gemma-4-E4B-it (raw) | 5.1B effective | 0.618 | 0.666 | 0-shot |
| Gemma-4-E4B-base (raw) | 5.1B effective | 0.588 | 0.666 | 5-shot |
* Single-trial values from our independent reproduction.
Key highlights
- Beats Foundation-Sec-Instruct-8B on CTI-MCQ by +8.7 points at half the parameter count.
- Stays within ~2 points of Foundation-Sec-Instruct-8B on CTI-RCM under the same evaluation protocol.
- Cross-substrate companion (Gemma4Defense-2B) reproduces the CTI-RCM result within 0.9 points using the same recipe on a different model family.
- Independent reproduction of CyberPal-2.0-20B at the Foundation-Sec protocol confirms its CTI-MCQ accuracy within 2 points of its paper claim.
- Trained, merged, and evaluated end-to-end on a single AMD MI300X 192GB instance with FlashAttention-2 enabled.
Limitations
-
Domain-specific knowledge limitations. The model is trained on cybersecurity domain text and is not a general assistant. Tasks outside this domain will produce lower-quality output than purpose-built general models.
-
Time-anchored training data. The CTI-RCM training cohort is drawn from 2021 records. Vulnerability classes that emerged or rose in prevalence after 2021 (e.g., AI/ML-specific weaknesses, recent supply-chain CWEs) are under-represented in training and will be classified less accurately.
-
English-only. All training and evaluation data are in English; multilingual cyber tasks will degrade.
-
CTI-RCM gap. Foundation-Sec-Instruct-8B remains stronger on CTI-RCM under this protocol (-1.9 point gap). Production deployments where CWE classification is the primary metric should benchmark both models on their specific input distribution.
-
No safety RLHF. The model is supervised-fine-tuned only; the training data emphasizes defensive-analyst framing but no formal reinforcement-learning safety alignment was applied.
-
Chat template note. The repository ships with a minimal training-aligned chat_template.jinja matching the format used during SFT (Qwen <|im_start|> / <|im_end|> user-and-assistant turns, no thinking-mode block). Inference via tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) produces correctly-formatted prompts; downstream tooling that injects system prompts or thinking-mode toggles outside this template may degrade output quality.
Recommendations
- Always have qualified security professionals review model outputs before implementation for any operational use case (patch prioritization, ticket routing, blocklisting).
- Use this model as an assistive tool rather than a replacement for expert human judgment, especially for novel vulnerability classes outside the 2021 training cohort.
- Validate on your own input distribution before deployment. Public CTI-Bench performance does not perfectly transfer to internal advisory feeds, vendor-proprietary CWE taxonomies, or non-English content.
- Monitor for drift. As new CVE / CWE patterns emerge, periodically re-evaluate; consider supplementing with retrieval over a current vulnerability knowledge base for time-sensitive queries.
- Apply standard prompt-injection mitigations when wrapping the model in agentic workflows that accept external content (advisory feeds, scraped pages); domain-SFT does not confer prompt-injection resistance.
Companion Model
Gemma4Defense-2B is a sister release fine-tuned with the same training corpus and hyperparameters, on the Gemma-4-E2B-it base. The two models converge to within 0.9 points on CTI-RCM (0.6664 Qwen vs 0.6754 Gemma, 5-trial mean) — the same recipe produces equivalent task performance across two distinct model families. The Gemma variant is licensed under the Gemma Terms of Use; CyberSecQwen-4B (Apache 2.0) is appropriate for use cases where Gemma terms are not a fit.
Citation
If you use this model, please cite:
1@misc{cybersecqwen2026,
2 title = {CyberSecQwen-4B: A Compact CTI Specialist Fine-Tuned from Qwen3-4B-Instruct-2507 on AMD MI300X},
3 author = {Mulia, Samuel},
4 year = {2026},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/athena129/CyberSecQwen-4B}
7}
The evaluation protocol is from:
1@article{foundation-sec-8b,
2 title = {Foundation-Sec-8B: A Cybersecurity-Specialized Language Model},
3 author = {Cisco Foundation AI},
4 journal = {arXiv preprint arXiv:2504.21039},
5 year = {2025},
6 url = {https://arxiv.org/abs/2504.21039}
7}
The benchmark is from:
1@misc{cti-bench,
2 title = {CTI-Bench: A Benchmark Suite for Cybersecurity LLMs},
3 author = {Alam, Md Tanvirul and Bhusal, Dipkamal and Park, Youngja and Rastogi, Nidhi},
4 year = {2024},
5 url = {https://github.com/xashru/cti-bench}
6}
