README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-8B
tags: [cybersecurity, agentic, red-team, blue-team, distillation, glm-5.2, kimi-k3, function-calling, tool-calling, tool-use]
---

# k3-sec-8b (v8)

An 8B cybersecurity agent (offense + defense) fine-tuned from `Qwen/Qwen3-8B` on
**2197 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
(rounds 1–3). Trained to operate an autonomous security harness — plan, run bash, read
observations, write files, finish — not just answer security questions.

Pipeline per iteration: failure analysis on eval traces → parameterized, decontaminated
seed factory → teacher best-of-3 rejection-sampled agentic transcripts (judged +
artifact-checked) → full-FT SFT → multi-run attack/defend eval → next round.

## Version 8 highlights

- **Data**: 2197 unique rows (2103 agentic + 94 Q&A). Round 8 was a 136-trace DNS
  top-up (75 decode + 75 detector-contract, $8). Round 7 was the broad-coverage
  round: 985 traces across ALL ten eval skills (flaky six weighted 100-120, solid four
  kept warm at 60-80, 80 generalization). Round 6 added
  564 GLM-5.2 traces targeting the five v3 tasks that never passed, generated from 580
  parameterized seeds with decontamination-by-construction (every eval-graded string is
  blacklisted and asserted absent). Best-of-3 rejection with a glm-4.7-flash judge
  (kept 97%). Note: v3's advertised 157 rows contained only 135 unique after legacy
  merge duplicates; v4 is a genuine 5.2× data increase.
- **Training**: full FT bf16, 2 epochs, lr 1e-5 cosine, eff. batch 32, seq 8192,
  adamw_8bit, 5.37M tokens, ~42 min on 1× A100-80GB. **train_loss 0.616 · token-acc
  87.2%** (v7: 0.627 / 86.6%, v6: 0.838 / 83.1%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN `rope_scaling` for 131072-token
  serving.
- **Eval** (fixed 10-task synthetic attack/defend lab, agentic harness, 3 runs):
  **mean 8.0/10 with ZERO variance (8, 8, 8) -- attack side 5/5 in all three runs**
  (atk-dns fixed by the top-up). Defend side: bruteforce/webshell(2/3)/harden solid,
  def-detect-dns 1/3, def-ioc regressed to 0/3 (round-9 target).
- **MMLU spot check** (60 questions, temp 0, same harness): v8 0.533 vs v7 0.550 vs
  **base Qwen3-8B 0.550** -- general capability statistically indistinguishable from
  base. NO capability collapse from the agentic diet.

### Per-task pass rates (P across runs)

| Task | v3 (4 runs) | v4 (3 runs) |
|---|---|---|
| atk-sqli | 0/4 | 0/3 |
| atk-hash | 4/4 | 3/3 |
| atk-re | 3/4 | 2/3 |
| atk-dns | 0/4 | 1/3 |
| atk-jwt | 0/4 | **3/3** |
| def-bruteforce | 3/4 | 3/3 |
| def-webshell | 2/4 | 2/3 |
| def-harden | 4/4 | 3/3 |
| def-detect-dns | 0/4 | 1/3 |
| def-ioc | 0/4 | 2/3 |

Run-to-run variance is significant at temperature 0.7; single-run scores are not
meaningful for this suite. Known v4 gap: `atk-sqli` — the model prefers to start the
staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
(trace-verified behavioral prior, targeted in round 5).

## Usage

vLLM, short-task/eval serving (disable static YaRN):

```bash
python3 -m vllm.entrypoints.openai.api_server \
  --model bebrws/k3-sec-8b --revision v7cti \
  --port 8000 --hf-overrides '{"rope_scaling":null}' --max-model-len 32768
```

Long-context serving: omit `--hf-overrides` and set `--max-model-len 131072`.
Recommended sampling for agentic loops (non-thinking): `temperature=0.7 top_p=0.8
top_k=20 min_p=0`, `chat_template_kwargs.enable_thinking=false`, per-step completion
cap ~4096 tokens.

## Tool / function calling

Supported. The chat template accepts a `tools` argument (OpenAI-style JSON function
schemas) and renders them into the system turn inside `<tools></tools>`. The model emits
calls as:

```
<tool_call>
{"name": "<function-name>", "arguments": {<args-json-object>}}
</tool_call>
```

Multiple calls may be emitted in a single assistant turn. Return each result as a message
with `role: "tool"`; the template renders it as `<tool_response>…</tool_response>`, and
consecutive tool messages are merged into one user turn.

```python
messages = [{"role": "user", "content": "Scan 10.0.0.5 for open ports"}]
tools = [{
    "type": "function",
    "function": {
        "name": "exec_shell_command",
        "description": "Run a shell command and return its output",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string"}},
            "required": ["command"],
        },
    },
}]

text = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, tokenize=False
)
```

vLLM serving with native tool-call parsing:

```bash
python3 -m vllm.entrypoints.openai.api_server \
  --model bebrws/k3-sec-8b \
  --enable-auto-tool-choice --tool-call-parser hermes
```

`llama.cpp` requires `--jinja` for the embedded template (and therefore tool calls) to be
used.

## Intended use & limitations

Defensive/offensive **security research artifact**, evaluated on a small synthetic lab.
Not for: real intrusion activity, exploit weaponization, unsupervised security
decisions, or non-security tasks. Outputs require qualified human review. Attack-side
competence is deliberately scoped to CTF/lab-grade tasks.

## Version history (8B lineage)

| Version | Data | Eval mean | Notes |
|---|---|---|---|
| v1 | 135 traces | 4/10 single | first 8B run |
| v2 | 149 traces | 6/10 single | failure-targeted r2 |
| v3 | 157 (135 unique) | 4.0/10 (4 runs) | parser-fixed harness baseline |
| v4 | 699 | 6.67/10 (3 runs) | GLM-5.2 scale-up, jwt fixed |
| v5 | 891 | 6.67/10 (3 runs: 4,8,8) | sqli breakthrough, ioc fixed; harden regressed (newline stripping) |
| v6 | 1076 | 7.0/10 (3 runs: 7,8,6) | harden fixed, webshell solid |
| v7 | 2061 | 8.33/10 (3 runs: 8,7,10) | gate passed; sqli 3/3 |
| **v8** | **2197** | **8.0/10 (3 runs: 8,8,8)** | attack 5/5 x3; MMLU == base; ioc regressed |

Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2
and Kimi teachers and filtered by automated judging.


## My main question

# Did k3-sec-8b iterations beat their base model?

**Answer: Yes — by v6, clearly. But early iterations were worse than base.**

The k3-sec-8b line starts training from **`Qwen/Qwen3-8B`** (per `docs/training-history.md`).
All numbers below are on the project's fixed 10-task agentic lab (5 attack + 5 defend,
identical sampling conditions).

| Iteration | Attack | Defend | Combined | vs base |
|---|---:|---:|---:|---|
| **Qwen3-8B base** (3 runs, 2026-07-31) | 12/15 | 12/15 | **24/30** (9, 8, 7 per run) | — |
| k3-sec-8b-v1 | 2/5 | 2/5 | 4/10 | below base |
| k3-sec-8b-v2 | 3/5 | 3/5 | 6/10 | below base |
| k3-sec-8b-v3 (4-run baseline) | — | — | mean 4.0/10 | below base |
| **k3-sec-8b-v6** (3 runs, 2026-07-31) | **15/15** | 12/15 | **27/30** (9, 9, 9 per run) | **+3 overall** |

## Details

- **v6 vs base (head-to-head, 3 runs each):** v6 wins 27/30 vs 24/30.
  - Attack: v6 is a **perfect 15/15** (all 5 attack tasks, all 3 runs); base is 12/15
    (atk-dns failed all 3 runs).
  - Defense: tied 12/15 both (def-detect-dns fails for both; base also drops def-webshell/def-ioc once each).
  - Consistency: v6 scores 9/10 on *every* run; the base declines 9 → 8 → 7 across runs.
- **The training took several iterations to pay off.** v1 (4/10), v2 (6/10), and v3
  (mean 4.0/10 across 4 runs) all scored *below* the base — early SFT rounds initially
  hurt the strong base model before later rounds (agentic file-writing data,
  failure-targeted rounds, GLM-5.2 bulk traces) pushed v6 above it.
- **Context:** the Qwen3-8B base is itself unusually strong on this lab (24/30) —
  stronger than Foundation-Sec-8B-Instruct (8/30) and RedSage-Qwen3-8B-taught (16/30)
  measured on the same benchmark. Beating it at all is a meaningful bar.

## Sources

- `data/eval_cmp_base_r{1,2,3}.json` — Qwen3-8B base runs
- `data/eval_cmp_student_r{1,2,3}.json` — k3-sec-8b v6 runs
- `docs/training-history.md` — v1–v3 iteration evals (`data/eval_8b*.json`)

## Aside

Also interesting: base Qwen3-8B is itself very strong on this lab (24/30 = 80%) — stronger than FSec-Instruct (8/30) and stronger than RedSage-taught (16/30)! That's a notable context point for the report: the k3-sec-8b v6 is the strongest model evaluated on this lab so far.


## External comparison: k3-sec-8b v7cti vs Foundation-Sec-1.1-8B-Instruct (Q8_0 GGUF)

> **Comparison note:** Foundation-Sec-1.1-8B-Instruct (Cisco Foundation AI, Aug 2025)
> appears to be the closest cutting-edge cybersecurity-specialized instruct model to
> compare against — same 8B class, instruction-tuned, security-domain. Both models
> were evaluated in their **Q8_0 GGUF** format (the most similar quantized format
> available for each), served via vLLM on identical RTX 4090 hardware with identical
> sampling. Full report: [RunPod evaluation, 2026-08-02](https://huggingface.co/bebrws/k3-sec-8b).

### Results (3 runs × 10 tasks = 30 trials per model)

| Benchmark | k3-sec-8b v7cti Q8_0 GGUF | Foundation-Sec-1.1-8B-Instruct Q8_0 GGUF |
|---|---:|---:|
| **Agentic lab — ATTACK** | **9/15 (60%)** | 5/15 (33%) |
| **Agentic lab — DEFEND** | **6/15 (40%)** | 3/15 (20%) |
| **Agentic lab — TOTAL** | **15/30 (50%)** | 8/30 (27%) |
| Knowledge battery (45 MCQ) | 43/45 (96%) | **45/45 (100%)** |
| Per-run consistency | 5/10 · 5/10 · 5/10 | 3/10 · 4/10 · 1/10 |

### Per-task pass rates (passes / 3 runs)

| Task | k3-sec-8b Q8_0 | FSec-1.1 Q8_0 | Winner |
|---|---:|---:|---|
| atk-sqli | **3/3** | 2/3 | k3-sec-8b |
| atk-hash | 2/3 | 2/3 | tie |
| atk-re | **2/3** | 1/3 | k3-sec-8b |
| atk-dns | 0/3 | 0/3 | neither |
| atk-jwt | **2/3** | 0/3 | k3-sec-8b |
| def-bruteforce | **2/3** | 0/3 | k3-sec-8b |
| def-webshell | **3/3** | 0/3 | k3-sec-8b |
| def-harden | 0/3 | **3/3** | FSec-1.1 |
| def-detect-dns | **1/3** | 0/3 | k3-sec-8b |
| def-ioc | 0/3 | 0/3 | neither |

k3-sec-8b wins or ties 8 of 10 tasks. Its standout is **def-webshell (3/3 vs 0/3)** —
log analysis and firewall-rule writing requiring multi-step shell-tool operation. FSec-1.1's
only decisive win is **def-harden (3/3 vs 0/3)** — single-shot SSH config editing where
instruction-following suffices.

### Key takeaways

- **k3-sec-8b is the more capable agentic model** (nearly 2× the operational score),
  consistent with its training on agentic tool-use traces. It scores a stable 5/10 every
  run; FSec-1.1 is volatile (1–4/10).
- **FSec-1.1 has slightly stronger factual knowledge** (perfect 45/45 vs 43/45 on the MCQ
  battery), consistent with its 5.1B-token cybersecurity CPT. But that knowledge doesn't
  translate to agentic capability on this harness.
- **Neither model solves atk-dns or def-ioc** — the hardest tasks on this lab.
- The Q8_0 GGUF format costs k3-sec-8b ~1 knowledge-quiz point vs bf16 (43 vs 44) but does
  not materially degrade agentic performance.