RedSage-Qwen3-8B-CFW

Model Summary

RedSage-Qwen3-8B-CFW is a cybersecurity-specialized Large Language Model (LLM) developed by [RISys-Lab]. It is the result of Continued Pre-training (CPT) on the CyberFineWeb corpus.

This model serves as the foundational stage of the RedSage pipeline. It takes the general-purpose Qwen3-8B-Base and adapts it to the cybersecurity domain using ~11.7 billion tokens of filtered, high-quality cybersecurity web data. To maintain general reasoning capabilities, it utilizes a data replay strategy with educational content.

• Paper: RedSage: A Cybersecurity Generalist LLM
• Repository: GitHub
• Base Model: Qwen/Qwen3-8B-Base
• Variant: CFW (CyberFineWeb Continued Pre-training)

Intended Use

This model is a base model intended for:

1. Further fine-tuning on downstream cybersecurity tasks.
2. Research into domain adaptation and continual pre-training dynamics.
3. Cybersecurity text completion and generation.

Note: As a base model, this checkpoint has not been instruction-tuned (SFT) or aligned (DPO). It behaves like a completion engine. For a chat-ready assistant, please see RISys-Lab/RedSage-Qwen3-8B-DPO.

Training Lineage

RedSage employs a multi-stage training pipeline. This model represents the output of Stage 1.

1. Stage 1: Continual Pre-Training (CPT) -> RedSage-Qwen3-8B-CFW (Current Model)
   • Data: CyberFineWeb (11.8B tokens)
2. Stage 2: Targeted Pre-Training -> RedSage-Qwen3-8B-Base
3. Stage 3: Supervised Fine-Tuning (SFT) -> RedSage-Qwen3-8B-Ins
4. Stage 4: Direct Preference Optimization (DPO) -> RedSage-Qwen3-8B-DPO

Training Data: CyberFineWeb

This model was trained on CyberFineWeb, a large-scale cybersecurity corpus constructed by filtering the FineWeb dataset (2013–2024).

1. Filtering: A ModernBERT-base classifier was trained on the Cybersecurity Topic Classification dataset to identify cybersecurity content within Common Crawl.
2. Dataset Size: The filtering process yielded ~125M documents (~89.8B tokens). We select the latest subset of ~11.7B tokens for this training stage.
3. General Knowledge Replay: To prevent catastrophic forgetting, we mixed the cybersecurity data with a 30% replay ratio of FineWeb-Edu samples.

Performance

RedSage-8B-CFW demonstrates improved performance over the general-purpose Qwen3-8B-Base on cybersecurity benchmarks while maintaining general capabilities.

RedSage-Bench (0-shot Accuracy)

Category	Qwen3-8B-Base	RedSage-8B-CFW
Macro Average	84.24	84.86
Knowledge (Gen)	83.08	83.62
Knowledge (Frameworks)	81.94	83.30
Skill (Offensive)	88.23	88.81
Tools (CLI)	85.08	85.30
Tools (Kali)	78.86	79.32

External Cybersecurity Benchmarks (5-shot)

Benchmark	Qwen3-8B-Base	RedSage-8B-CFW
Mean	80.81	82.66
CTI-Bench (MCQ)	68.80	68.40
CTI-Bench (RCM)	63.50	67.60
CyberMetric (500)	92.00	93.80
MMLU (Security)	83.00	86.00
SecBench (En)	82.84	83.62
SecEva (MCQ)	75.60	76.10
SECURE (CWET)	92.70	93.33
SECURE (KCV)	75.05	81.34
SECURE (MEAT)	93.81	93.72

Training Procedure

The model was trained using the Axolotl framework

• Learning Rate: 2.5e-6 (constant with linear warmup)
• Optimizer: AdamW
• Epochs: 1

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "RISys-Lab/RedSage-Qwen3-8B-CFW"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

text = "The primary difference between a firewall and an IDS is"
inputs = tokenizer(text, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model or dataset, please cite our paper:

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}