---
license: apache-2.0
base_model: Qwen/Qwen3-Coder-Next
tags:
- cybersecurity
- expert-upcycling
- self-distillation
- mixture-of-experts
- qwen3-next
- knowledge-injection
language: [en]
pipeline_tag: text-generation
library_name: transformers
---

# Qwen3-Coder-Next-Cyber

Cybersecurity domain specialization of **[Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next)** (the **instruct** model), built by **Expert Upcycling + freeze** and knowledge-injected via **self-distillation**.

The 80B instruct backbone is **frozen** (instruction-following and coding ability preserved by construction); cybersecurity capability lives in **32 added experts per layer** (512 → 544) that are the only trained part.

> Research artifact. Cyber-knowledge measured on CyberMetric-500 (**92.6%**, see Evaluation); behavioral evaluation via downstream adapters.

## What it is

| | |
|---|---|
| **Base** | Qwen3-Coder-Next (instruct, 80B total / ~3B active, hybrid Gated DeltaNet + Attention + MoE, 512 experts top-10 + 1 shared, Apache-2.0) |
| **Experts** | 512 → **544** (k=32 cyber experts added per layer, 48 layers) |
| **Trainable** | only {new experts 512-543, router, shared expert} = **~5.04B** of ~85B; backbone frozen |
| **Total params** | ~84.5B |

## How it was built (full pipeline)

This model is the result of a multi-stage pipeline. Each stage only ever trained the added cyber experts + router + shared expert; the original backbone stayed frozen throughout.

1. **Expert upcycling (surgery)** — 32 new experts per layer were added (512 → 544), cloned from high-utility experts (utility = gradient-norm squared, greedy per-layer allocation), with the router extended and top-k unchanged. This creates fresh capacity without altering the base.

2. **Continued pre-training of the new experts (~200M tokens)** — the 32 new experts were first trained on a curated cybersecurity corpus (MITRE ATT&CK, CWE/OWASP, HackTricks, CVE, compute-foundations) with the 80B backbone frozen. An anti-leak load-balancing term kept the new experts alive (`frac_new ≈ 0.146` of routed mass at the end, vs 0.059 uniform → alive and used), final CPT loss 0.68. Framework: Megatron-core via ms-swift (`megatron pt`), 4×A100-80GB.

3. **Backbone correction (base → instruct)** — the initial CPT was done on the *base* backbone. Because the whole design freezes the backbone, the trained experts could be **transplanted** onto the *instruct* backbone (Qwen3-Coder-Next) as a warm-start — keeping the instruct model's instruction-following intact while inheriting the experts' cyber specialization.

4. **Self-distillation (knowledge injection)** — the model reads a domain passage in-context (teacher, open-book) and produces an answer; the closed-book student (same weights) learns that answer. Only the 32 experts + router + shared were trained (~43k self-distilled QA, 1350 steps). This makes the stored knowledge *extractable* closed-book.

Why this recipe rather than a large full-model CPT: naive SFT on new facts is unreliable and increases hallucination (Gekhman 2024); self-distillation from the model's own open-book reading is in-distribution and data-efficient (arXiv 2412.14964); and keeping the backbone frozen means the base coding/general ability cannot be eroded (knowledge lives in the experts, behavior in later LoRA adapters).

## Knowledge domains (self-distillation sources)

MITRE ATT&CK techniques, OWASP (WSTG/Top-10), HackTricks, PayloadsAllTheThings, Atomic Red Team, adversary-emulation, OSCP/PEN-200, threat-intelligence, and detection tooling (osquery/Falco/Suricata). ~10.8k concepts, ~43k question-answer pairs.

## Evaluation

**CyberMetric-500** (500 cybersecurity multiple-choice questions, first-token letter scoring):

| Model | Accuracy |
|---|---|
| **Qwen3-Coder-Next-Cyber** | **92.60%** (463/500) |

Strong absolute cybersecurity knowledge. This measures *knowledge*; task behavior (red/blue-team reasoning, verdict / MITRE / action) is added by downstream adapters — see below.

## Downstream

- **Behavioral SFT adapter**: `Qwen3-Coder-Next-Cyber-SFT-lora` — LoRA teaching red-team / blue-team behavior with `<think>` reasoning (CyberMetric-500 preserved at 92.8%). Load base + adapter; use `repetition_penalty ≈ 1.15`.
- Next stage: GRPO / RLVR with verifiable rewards.

## Intended use & limitations

Domain-adapted assistant base for cybersecurity (offensive and defensive knowledge is symmetric). Intended as the shared base for downstream behavioral adapters (red-team / blue-team LoRA SFT + RL). Research artifact.

## Responsible use

Encodes offensive-security knowledge for **authorized** testing, defense, and education. Do not use against systems you do not own or lack permission to test.

## License

Apache-2.0 (inherited from Qwen3-Coder-Next).

## Lineage & engineering notes

- Base: Qwen3-Coder-Next (instruct, Qwen team).
- Upcycling method: *Expert Upcycling*, arXiv:2604.19835.
- Full CPT + self-distillation engineering notes (corpus tiers, Megatron setup, freeze regex, anti-leak term): repository `kalithos-cybersec` (`recipes/cpt/` and `recipes/knowledge-injection/`).