You can deploy this model and start earning money today!

RISys-Lab/
RedSage-Qwen3-8B-CFW

Name: RedSage-Qwen3-8B-CFW
Availability: InStock

generated_from_trainer

cybersecurity

continual-pretraining

text-generation-inference

endpoints_compatible

Views

No views yet

RedSage-Qwen3-8B-CFW

Model Summary

RedSage-Qwen3-8B-CFW is a cybersecurity-specialized Large Language Model (LLM) developed by [RISys-Lab]. It is the result of Continued Pre-training (CPT) on the CyberFineWeb corpus.

This model serves as the foundational stage of the RedSage pipeline. It takes the general-purpose Qwen3-8B-Base and adapts it to the cybersecurity domain using ~11.7 billion tokens of filtered, high-quality cybersecurity web data. To maintain general reasoning capabilities, it utilizes a data replay strategy with educational content.

Paper: RedSage: A Cybersecurity Generalist LLM (arXiv)
Repository: GitHub
Base Model: Qwen/Qwen3-8B-Base
Variant: CFW (CyberFineWeb Continued Pre-training)

Intended Use

This model is a base model intended for:

Further fine-tuning on downstream cybersecurity tasks.
Research into domain adaptation and continual pre-training dynamics.
Cybersecurity text completion and generation.

Note: As a base model, this checkpoint has not been instruction-tuned (SFT) or aligned (DPO). It behaves like a completion engine. For a chat-ready assistant, please see RISys-Lab/RedSage-Qwen3-8B-DPO.

Training Lineage

RedSage employs a multi-stage training pipeline. This model represents the output of Stage 1.

Stage 1: Continual Pre-Training (CPT) -> RedSage-Qwen3-8B-CFW (Current Model)
- Data: CyberFineWeb (11.8B tokens)
Stage 2: Targeted Pre-Training -> RedSage-Qwen3-8B-Base
Stage 3: Supervised Fine-Tuning (SFT) -> RedSage-Qwen3-8B-Ins
Stage 4: Direct Preference Optimization (DPO) -> RedSage-Qwen3-8B-DPO

Training Data: CyberFineWeb

This model was trained on CyberFineWeb, a large-scale cybersecurity corpus constructed by filtering the FineWeb dataset (2013–2024).

Filtering: A ModernBERT-base classifier was trained on the Cybersecurity Topic Classification dataset to identify cybersecurity content within Common Crawl.
Dataset Size: The filtering process yielded ~125M documents (~89.8B tokens). We select the latest subset of ~11.7B tokens for this training stage.
General Knowledge Replay: To prevent catastrophic forgetting, we mixed the cybersecurity data with a 30% replay ratio of FineWeb-Edu samples.

Performance

RedSage-8B-CFW demonstrates improved performance over the general-purpose Qwen3-8B-Base on cybersecurity benchmarks while maintaining general capabilities.

RedSage-Bench (0-shot Accuracy)

Category	Qwen3-8B-Base	RedSage-8B-CFW
Macro Average	84.24	84.86
Knowledge (Gen)	83.08	83.62
Knowledge (Frameworks)	81.94	83.30
Skill (Offensive)	88.23	88.81
Tools (CLI)	85.08	85.30
Tools (Kali)	78.86	79.32

External Cybersecurity Benchmarks (5-shot)

Benchmark	Qwen3-8B-Base	RedSage-8B-CFW
Mean	80.81	82.66
CTI-Bench (MCQ)	68.80	68.40
CTI-Bench (RCM)	63.50	67.60
CyberMetric (500)	92.00	93.80
MMLU (Security)	83.00	86.00
SecBench (En)	82.84	83.62
SecEva (MCQ)	75.60	76.10
SECURE (CWET)	92.70	93.33
SECURE (KCV)	75.05	81.34
SECURE (MEAT)	93.81	93.72

Training Procedure

The model was trained using the Axolotl framework

Learning Rate: 2.5e-6 (constant with linear warmup)
Optimizer: AdamW
Epochs: 1

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "RISys-Lab/RedSage-Qwen3-8B-CFW"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

text = "The primary difference between a firewall and an IDS is"
inputs = tokenizer(text, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model or dataset, please cite our paper:

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}