RedSage-Qwen3-8B-Base is a cybersecurity-specialized Large Language Model (LLM) developed by RISys-Lab. It represents the second stage of the RedSage pre-training pipeline.
This model builds upon RedSage-Qwen3-8B-CFW by undergoing Targeted Pre-Training on high-quality, curated cybersecurity resources (RedSage-Seed and RedSage-Dump). While the previous stage focused on breadth using web data, this stage focuses on depth, technical standards, and verified skills.
Fine-tuning: Serving as a high-quality foundation for downstream cybersecurity tasks (e.g., incident response, malware analysis).
Research: Investigating the impact of curated versus web-scale data in domain adaptation.
Completion: Code completion and technical writing in cybersecurity contexts.
Note: As a base model, this checkpoint has not been instruction-tuned (SFT) or aligned (DPO). It behaves like a completion engine. For a chat-ready assistant, please see RISys-Lab/RedSage-Qwen3-8B-DPO.
Training Lineage
RedSage employs a multi-stage training pipeline. This model represents the output of Stage 2.
This model was trained on approximately 850 million tokens of curated data, split into two collections:
RedSage-Seed (~150M Tokens): A highly curated collection of 28,637 samples converted to structured Markdown.
Knowledge: General concepts and Frameworks (MITRE ATT&CK, CAPEC, CWE, OWASP).
Skills: Offensive security resources including write-ups, hacking techniques, and payload examples.
Tools: Manuals and cheat sheets for CLI tools and Kali Linux.
RedSage-Dump (~700M Tokens): A larger aggregation of 459K technical documents.
Sources: Computer education portals, cybersecurity news, RFC entries, NIST publications, and the National Vulnerability Database (NVD).
Performance
RedSage-8B-Base achieves state-of-the-art performance among 8B models, showing significant improvements over the general-purpose Qwen3-8B-Base. It achieves the highest mean score on external benchmarks among all 8B base models tested.
RedSage-Bench (0-shot Accuracy)
Category
Qwen3-8B-Base
RedSage-8B-Base
Macro Average
84.24
85.05
Knowledge (General)
83.08
83.12
Knowledge (Frameworks)
81.94
84.94
Skill (Offensive)
88.23
88.72
Tools (CLI)
85.08
85.44
Tools (Kali)
78.86
79.36
External Cybersecurity Benchmarks (5-shot)
Benchmark
Qwen3-8B-Base
RedSage-8B-Base
Mean
80.81
84.56
CTI-Bench (MCQ)
68.80
71.04
CTI-Bench (RCM)
63.50
78.40
CyberMetric (500)
92.00
92.60
MMLU (Security)
83.00
87.00
SecBench (En)
82.84
81.76
SecEva (MCQ)
75.60
75.83
SECURE (CWET)
92.70
93.22
SECURE (KCV)
75.05
87.20
SECURE (MEAT)
93.81
94.00
Training Procedure
The model was trained using the Axolotl framework.
Learning Rate: 2.5e-6 (constant with linear warmup)
Optimizer: AdamW
Epochs: 1
Usage
python
1from transformers import AutoTokenizer, AutoModelForCausalLM
23model_id ="RISys-Lab/RedSage-Qwen3-8B-Base"45tokenizer = AutoTokenizer.from_pretrained(model_id)6model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")78text ="The primary difference between a firewall and an IDS is"9inputs = tokenizer(text, return_tensors="pt").to("cuda")1011outputs = model.generate(**inputs, max_new_tokens=50)12print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
If you use this model or dataset, please cite our paper:
@inproceedings{suryanto2026redsage,
title={RedSage: A Cybersecurity Generalist {LLM}},
author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=W4FAenIrQ2}
}
