Qwen3-Coder-Next-Cyber

Cybersecurity domain specialization of Qwen/Qwen3-Coder-Next (the instruct model), built by Expert Upcycling + freeze and knowledge-injected via self-distillation.

The 80B instruct backbone is frozen (instruction-following and coding ability preserved by construction); cybersecurity capability lives in 32 added experts per layer (512 to 544) that are the only trained part.

Research artifact. Downstream evaluation (cyber benchmarks) pending.

What it is

Base	Qwen3-Coder-Next (instruct, 80B total / ~3B active, hybrid Gated DeltaNet + Attention + MoE, 512 experts top-10 + 1 shared, Apache-2.0)
Experts	512 to 544 (k=32 cyber experts added per layer, 48 layers)
Trainable	only {new experts 512-543, router, shared expert} = ~5.04B of ~85B; backbone frozen
Total params	~84.5B

How it was built (full pipeline)

This model is the result of a multi-stage pipeline. Each stage only ever trained the added cyber experts + router + shared expert; the original backbone stayed frozen throughout.

1. Expert upcycling (surgery) — 32 new experts per layer were added (512 to 544), cloned from high-utility experts (utility = gradient-norm squared, greedy per-layer allocation), with the router extended and top-k unchanged. This creates fresh capacity without altering the base.

2. Continued pre-training of the new experts (~200M tokens) — the 32 new experts were first trained on a curated cybersecurity corpus (MITRE ATT&CK, CWE/OWASP, HackTricks, CVE, compute-foundations) with the 80B backbone frozen. An anti-leak load-balancing term kept the new experts alive (avoiding capacity death). This gave the experts an initial cyber specialization.

3. Backbone correction (base to instruct) — the initial CPT was done on the base backbone. Because the whole design freezes the backbone, the trained experts could be transplanted onto the instruct backbone (Qwen3-Coder-Next) as a warm-start — keeping the instruct model's instruction-following intact while inheriting the experts' cyber specialization.

4. Self-distillation (knowledge injection) — the model reads a domain passage in-context (teacher, open-book) and produces an answer; the closed-book student (same weights) learns that answer. Only the 32 experts + router + shared were trained (~43k self-distilled QA, 1350 steps). This makes the stored knowledge extractable closed-book.

Why this recipe rather than a large full-model CPT: naive SFT on new facts is unreliable and increases hallucination (Gekhman 2024); self-distillation from the model's own open-book reading is in-distribution and data-efficient (arXiv 2412.14964); and keeping the backbone frozen means the base coding/general ability cannot be eroded (knowledge lives in the experts, behavior in later LoRA adapters).

Knowledge domains (self-distillation sources)

MITRE ATT&CK techniques, OWASP (WSTG/Top-10), HackTricks, PayloadsAllTheThings, Atomic Red Team, adversary-emulation, OSCP/PEN-200, threat-intelligence, and detection tooling (osquery/Falco/Suricata). ~10.8k concepts, ~43k question-answer pairs.

Intended use & limitations

Domain-adapted assistant base for cybersecurity (offensive and defensive knowledge is symmetric). Intended as the shared base for downstream behavioral adapters (red-team / blue-team LoRA SFT + RL). Research artifact; quantitative cyber-benchmark gains are not yet measured.

Responsible use

Encodes offensive-security knowledge for authorized testing, defense, and education. Do not use against systems you do not own or lack permission to test.

License

Apache-2.0 (inherited from Qwen3-Coder-Next).