bebrws/security-8b

k3-sec-8b (v8)

An 8B cybersecurity agent (offense + defense) fine-tuned from Qwen/Qwen3-8B on 2197 verified agentic + Q&A traces, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6 (rounds 1–3). Trained to operate an autonomous security harness — plan, run bash, read observations, write files, finish — not just answer security questions.

Pipeline per iteration: failure analysis on eval traces → parameterized, decontaminated seed factory → teacher best-of-3 rejection-sampled agentic transcripts (judged + artifact-checked) → full-FT SFT → multi-run attack/defend eval → next round.

Version 8 highlights

• Data: 2197 unique rows (2103 agentic + 94 Q&A). Round 8 was a 136-trace DNS top-up (75 decode + 75 detector-contract, $8). Round 7 was the broad-coverage round: 985 traces across ALL ten eval skills (flaky six weighted 100-120, solid four kept warm at 60-80, 80 generalization). Round 6 added 564 GLM-5.2 traces targeting the five v3 tasks that never passed, generated from 580 parameterized seeds with decontamination-by-construction (every eval-graded string is blacklisted and asserted absent). Best-of-3 rejection with a glm-4.7-flash judge (kept 97%). Note: v3's advertised 157 rows contained only 135 unique after legacy merge duplicates; v4 is a genuine 5.2× data increase.
• Training: full FT bf16, 2 epochs, lr 1e-5 cosine, eff. batch 32, seq 8192, adamw_8bit, 5.37M tokens, ~42 min on 1× A100-80GB. train_loss 0.616 · token-acc 87.2% (v7: 0.627 / 86.6%, v6: 0.838 / 83.1%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN rope_scaling for 131072-token serving.
• Eval (fixed 10-task synthetic attack/defend lab, agentic harness, 3 runs): mean 8.0/10 with ZERO variance (8, 8, 8) -- attack side 5/5 in all three runs (atk-dns fixed by the top-up). Defend side: bruteforce/webshell(2/3)/harden solid, def-detect-dns 1/3, def-ioc regressed to 0/3 (round-9 target).
• MMLU spot check (60 questions, temp 0, same harness): v8 0.533 vs v7 0.550 vs base Qwen3-8B 0.550 -- general capability statistically indistinguishable from base. NO capability collapse from the agentic diet.

Per-task pass rates (P across runs)

Task	v3 (4 runs)	v4 (3 runs)
atk-sqli	0/4	0/3
atk-hash	4/4	3/3
atk-re	3/4	2/3
atk-dns	0/4	1/3
atk-jwt	0/4	3/3
def-bruteforce	3/4	3/3
def-webshell	2/4	2/3
def-harden	4/4	3/3
def-detect-dns	0/4	1/3
def-ioc	0/4	2/3

Run-to-run variance is significant at temperature 0.7; single-run scores are not meaningful for this suite. Known v4 gap: atk-sqli — the model prefers to start the staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts (trace-verified behavioral prior, targeted in round 5).

Usage

vLLM, short-task/eval serving (disable static YaRN):

python3 -m vllm.entrypoints.openai.api_server \
  --model bebrws/k3-sec-8b --revision v7cti \
  --port 8000 --hf-overrides '{"rope_scaling":null}' --max-model-len 32768

Long-context serving: omit --hf-overrides and set --max-model-len 131072. Recommended sampling for agentic loops (non-thinking): temperature=0.7 top_p=0.8 top_k=20 min_p=0, chat_template_kwargs.enable_thinking=false, per-step completion cap ~4096 tokens.

Tool / function calling

Supported. The chat template accepts a tools argument (OpenAI-style JSON function schemas) and renders them into the system turn inside <tools></tools>. The model emits calls as:

<tool_call>
{"name": "<function-name>", "arguments": {<args-json-object>}}
</tool_call>

Multiple calls may be emitted in a single assistant turn. Return each result as a message with role: "tool"; the template renders it as <tool_response>…</tool_response>, and consecutive tool messages are merged into one user turn.

messages = [{"role": "user", "content": "Scan 10.0.0.5 for open ports"}]
tools = [{
    "type": "function",
    "function": {
        "name": "exec_shell_command",
        "description": "Run a shell command and return its output",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string"}},
            "required": ["command"],
        },
    },
}]

text = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, tokenize=False
)

vLLM serving with native tool-call parsing:

python3 -m vllm.entrypoints.openai.api_server \
  --model bebrws/k3-sec-8b \
  --enable-auto-tool-choice --tool-call-parser hermes

llama.cpp requires --jinja for the embedded template (and therefore tool calls) to be used.

Intended use & limitations

Defensive/offensive security research artifact, evaluated on a small synthetic lab. Not for: real intrusion activity, exploit weaponization, unsupervised security decisions, or non-security tasks. Outputs require qualified human review. Attack-side competence is deliberately scoped to CTF/lab-grade tasks.

Version history (8B lineage)

Version	Data	Eval mean	Notes
v1	135 traces	4/10 single	first 8B run
v2	149 traces	6/10 single	failure-targeted r2
v3	157 (135 unique)	4.0/10 (4 runs)	parser-fixed harness baseline
v4	699	6.67/10 (3 runs)	GLM-5.2 scale-up, jwt fixed
v5	891	6.67/10 (3 runs: 4,8,8)	sqli breakthrough, ioc fixed; harden regressed (newline stripping)
v6	1076	7.0/10 (3 runs: 7,8,6)	harden fixed, webshell solid
v7	2061	8.33/10 (3 runs: 8,7,10)	gate passed; sqli 3/3
v8	2197	8.0/10 (3 runs: 8,8,8)	attack 5/5 x3; MMLU == base; ioc regressed

Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2 and Kimi teachers and filtered by automated judging.

My main question

Did k3-sec-8b iterations beat their base model?

Answer: Yes — by v6, clearly. But early iterations were worse than base.

The k3-sec-8b line starts training from Qwen/Qwen3-8B (per docs/training-history.md). All numbers below are on the project's fixed 10-task agentic lab (5 attack + 5 defend, identical sampling conditions).

Iteration	Attack	Defend	Combined	vs base
Qwen3-8B base (3 runs, 2026-07-31)	12/15	12/15	24/30 (9, 8, 7 per run)	—
k3-sec-8b-v1	2/5	2/5	4/10	below base
k3-sec-8b-v2	3/5	3/5	6/10	below base
k3-sec-8b-v3 (4-run baseline)	—	—	mean 4.0/10	below base
k3-sec-8b-v6 (3 runs, 2026-07-31)	15/15	12/15	27/30 (9, 9, 9 per run)	+3 overall

Details

• v6 vs base (head-to-head, 3 runs each): v6 wins 27/30 vs 24/30.
  • Attack: v6 is a perfect 15/15 (all 5 attack tasks, all 3 runs); base is 12/15 (atk-dns failed all 3 runs).
  • Defense: tied 12/15 both (def-detect-dns fails for both; base also drops def-webshell/def-ioc once each).
  • Consistency: v6 scores 9/10 on every run; the base declines 9 → 8 → 7 across runs.
• The training took several iterations to pay off. v1 (4/10), v2 (6/10), and v3 (mean 4.0/10 across 4 runs) all scored below the base — early SFT rounds initially hurt the strong base model before later rounds (agentic file-writing data, failure-targeted rounds, GLM-5.2 bulk traces) pushed v6 above it.
• Context: the Qwen3-8B base is itself unusually strong on this lab (24/30) — stronger than Foundation-Sec-8B-Instruct (8/30) and RedSage-Qwen3-8B-taught (16/30) measured on the same benchmark. Beating it at all is a meaningful bar.

Sources

• data/eval_cmp_base_r{1,2,3}.json — Qwen3-8B base runs
• data/eval_cmp_student_r{1,2,3}.json — k3-sec-8b v6 runs
• docs/training-history.md — v1–v3 iteration evals (data/eval_8b*.json)

Aside

Also interesting: base Qwen3-8B is itself very strong on this lab (24/30 = 80%) — stronger than FSec-Instruct (8/30) and stronger than RedSage-taught (16/30)! That's a notable context point for the report: the k3-sec-8b v6 is the strongest model evaluated on this lab so far.

External comparison: k3-sec-8b v7cti vs Foundation-Sec-1.1-8B-Instruct (Q8_0 GGUF)

Comparison note: Foundation-Sec-1.1-8B-Instruct (Cisco Foundation AI, Aug 2025) appears to be the closest cutting-edge cybersecurity-specialized instruct model to compare against — same 8B class, instruction-tuned, security-domain. Both models were evaluated in their Q8_0 GGUF format (the most similar quantized format available for each), served via vLLM on identical RTX 4090 hardware with identical sampling. Full report: RunPod evaluation, 2026-08-02.

Results (3 runs × 10 tasks = 30 trials per model)

Benchmark	k3-sec-8b v7cti Q8_0 GGUF	Foundation-Sec-1.1-8B-Instruct Q8_0 GGUF
Agentic lab — ATTACK	9/15 (60%)	5/15 (33%)
Agentic lab — DEFEND	6/15 (40%)	3/15 (20%)
Agentic lab — TOTAL	15/30 (50%)	8/30 (27%)
Knowledge battery (45 MCQ)	43/45 (96%)	45/45 (100%)
Per-run consistency	5/10 · 5/10 · 5/10	3/10 · 4/10 · 1/10

Per-task pass rates (passes / 3 runs)

Task	k3-sec-8b Q8_0	FSec-1.1 Q8_0	Winner
atk-sqli	3/3	2/3	k3-sec-8b
atk-hash	2/3	2/3	tie
atk-re	2/3	1/3	k3-sec-8b
atk-dns	0/3	0/3	neither
atk-jwt	2/3	0/3	k3-sec-8b
def-bruteforce	2/3	0/3	k3-sec-8b
def-webshell	3/3	0/3	k3-sec-8b
def-harden	0/3	3/3	FSec-1.1
def-detect-dns	1/3	0/3	k3-sec-8b
def-ioc	0/3	0/3	neither

k3-sec-8b wins or ties 8 of 10 tasks. Its standout is def-webshell (3/3 vs 0/3) — log analysis and firewall-rule writing requiring multi-step shell-tool operation. FSec-1.1's only decisive win is def-harden (3/3 vs 0/3) — single-shot SSH config editing where instruction-following suffices.

Key takeaways

• k3-sec-8b is the more capable agentic model (nearly 2× the operational score), consistent with its training on agentic tool-use traces. It scores a stable 5/10 every run; FSec-1.1 is volatile (1–4/10).
• FSec-1.1 has slightly stronger factual knowledge (perfect 45/45 vs 43/45 on the MCQ battery), consistent with its 5.1B-token cybersecurity CPT. But that knowledge doesn't translate to agentic capability on this harness.
• Neither model solves atk-dns or def-ioc — the hardest tasks on this lab.
• The Q8_0 GGUF format costs k3-sec-8b ~1 knowledge-quiz point vs bf16 (43 vs 44) but does not materially degrade agentic performance.