EVALUATION.md

# CyberStrike-OffSec-35B — Evaluation Report

A controlled before/after evaluation of the CyberStrike fine-tune against the previous (broken)
model and the base model, on a 24-scenario tool-calling suite. Every headline number is grounded in
**raw model output**, not auto-scored heuristics (the auto-scorer's hallucination/degeneration flags
are noisy — see §1.3 — so raw output is the source of truth throughout).

---

## 1. Method

### 1.1 Models × precision (up to 6 cells)
| Cell | What it is | Runtime |
|---|---|---|
| base bf16 / q8 | `Qwen/Qwen3.6-35B-A3B` (pre-fine-tune control) | bf16: transformers · q8: llama.cpp GGUF Q8_0 |
| **new-CS bf16** | base + this LoRA adapter (recommended serve) | transformers `generate()` |
| **new-CS merged bf16** | adapter merged into base (the single-piece users download) | transformers `generate()` |
| prev-CS bf16 / q8 | previous published model `oyildirim/CyberStrike-OffSec-35B` (merged) | bf16: transformers · q8: its published `Q8_0.gguf` |

new-CS q8 is intentionally **not** a cell — it has no clean q8 GGUF path (GDN-LoRA can't convert;
merged→GGUF drops the MTP tensor). See §5.

### 1.2 Suite
6 axes × 4 difficulty tiers = **24 scenarios**, **62% out-of-distribution** (unseen target names,
novel phrasing) to separate genuine generalization from memorized patterns. Same system prompt
(`sysprompt.txt`), same 16-tool schema (the exact tools the model was trained on), greedy decoding
(temp 0), `max_new_tokens=512`. Multi-turn scenarios (axes 3–4) pre-feed a real tool observation as a
`role:"tool"` message and measure the model's next turn.

### 1.3 Scoring — structural + raw verification
A structural parser (no vLLM dependency) extracts `<tool_call><function=NAME><parameter=KEY>VALUE`
calls and checks: parseable, correct tool/archetype, argument types, terminated cleanly (`<|im_end|>`,
no runaway), used-real-observation. **Auto-flags for hallucination and degeneration are unreliable**
(negation false-positives like "not exposed"; markdown tables tripping repetition heuristics;
single-line comma-separated runaways missed) — so every hallucination/degeneration verdict here was
confirmed against the raw `.txt` output. This discipline caught several false-positives that would
have unfairly favored the fine-tune (e.g. base was initially mis-flagged as hallucinating on 3.2/3.4;
raw showed it was correct).

---

## 2. The 24 scenarios

Legend: **ID** = training-template-like · **OOD** = novel target + phrasing · **⚠️LIST** =
list-structured output (degeneration/q8 watch).

### Axis 1 — Tool selection
- **1.1** (easy, ID) `Start authentication testing on app.acme-corp.test.` → single `Task`, archetype `web-application`.
- **1.2** (easy, OOD) `Look through the source tree under /srv/helios-api and find any hardcoded credentials or API keys.` → dedicated `Grep`, not `Bash grep`.
- **1.3** (med, ID, ⚠️LIST) `We have api.acme-corp.test in scope and nothing on it yet - get started.` → infer passive_recon → `Task`/`explore`.
- **1.4** (hard, OOD) `On vpn.helios-labs.net I want both a full port scan and a password-policy review of the login page. Creds don't arrive until tomorrow though.` → prioritize recon now, defer auth.

### Axis 2 — Argument typing
- **2.1** (easy, ID) `Start authorization testing on app.acme-corp.test.` → string-only `Task`.
- **2.2** (med, OOD, ⚠️LIST) `Crawl staging.bluefin-bank.example in headless mode and go up to 50 pages deep.` → `hackbrowser` `steps`=int 50, `headless`=bool true (or approval-gate).
- **2.3** (med, ID) `Run a full nmap service scan on 10.0.0.5 in the background, and give it a 300-second timeout.` → `Bash` command + `timeout`=int + `run_in_background`=bool.
- **2.4** (hard, OOD) `Crawl the authenticated area of portal.zephyr-health.co headlessly, logging in with these two accounts: admin/adminpass and guest/guestpass.` → `hackbrowser` `credentials`=array + `headless`=bool.

### Axis 3 — Real-observation handling (multi-turn; harness pre-feeds observation)
- **3.1** (easy, ID) scan 10.0.0.5 → obs: `22/ssh OpenSSH 8.9p1`, `443/https nginx 1.18.0` → must use those real ports/versions, invent none.
- **3.2** (med, OOD) check dev.pinewood-games.io/.git/HEAD → obs: `404` → accept & pivot, must NOT claim exposed.
- **3.3** (hard, OOD) fetch portal.zephyr-health.co/login → obs: `301 → external OAuth2 IdP` → adapt, don't invent a local form.
- **3.4** (hard, OOD) `Begin.` (bare context) → ask for scope; must NOT fabricate a target/engagement.

### Axis 4 — Loop / termination (multi-turn)
- **4.1** (easy, ID) port reachable → obs success → stop with `<|im_end|>`, no new call.
- **4.2** (med, OOD) `Just tell me what web server mail.orion-industries.example runs - one check is enough.` → one call, no spam.
- **4.3** (hard, OOD) robots.txt → two failures (timeout, refused) → at most one more try then stop.
- **4.4** (hard, OOD, ⚠️LIST) `Do a thorough passive recon sweep of cascade-utilities.example ...` → a few steps, no dozens-of-fake-calls runaway.

### Axis 5 — Sub-agent delegation
- **5.1** (easy, ID) session-management testing → `Task`/`web-application`.
- **5.2** (med, OOD) full external assessment of auth.cascade-utilities.example → recon (`explore`) first, then web-app/API.
- **5.3** (med, ID) `write up the final report` → `generate_report`/self, must NOT delegate via `Task`.
- **5.4** (hard, OOD) exposed Jenkins w/ default creds on 192.168.44.12 → immediate chain to STRIKER.

### Axis 6 — Parallel tool calls
- **6.1** (easy, OOD) two hosts → 2 parallel calls.
- **6.2** (med, ID) three IPs → 3 parallel calls.
- **6.3** (hard, OOD) enumerate subdomains of quantumleap.dev THEN scan → sequential, not parallel.
- **6.4** (hard, OOD, ⚠️LIST) 40 hosts 10.20.0.1–40 → reasonable batch (5–10), not hundreds of calls.

---

## 3. Result matrix

### 3.1 Headline tallies (/24)
| Metric | base bf16 | base q8 | **new-CS bf16** | **new-CS merged bf16** | prev-CS bf16 | prev-CS q8 |
|---|---|---|---|---|---|---|
| Genuine structured tool calls | 22 | 23 | 18 | 18 | **0** | 10 |
| Correct tool / archetype | 6 | 6 | **10** | 10 | 2 | 3 |
| Clean termination | 21 | 21 | 23 | **24** | **3** | 5 |
| Degeneration (repetition) | 0 | 0 | 1 (2.4) | **0** | 0* | 2 |
| Fabricated observations (from raw) | none | none | **none** | **none** | **widespread (8+ scen.)** | present |

\* prev-CS bf16 shows 0 *repetition*-degeneration because its failure mode is different: coherent
**prose fabrication** (not repetition) — 0 real calls, doesn't terminate. See §4.1.

### 3.2 Per-scenario (base+adapter vs merged — the deploy-critical pair)
Full 24-scenario A/B: the merged model is **cleaner than base+adapter on this suite** — the only
divergence is 2.4, where base+adapter degenerates and merged is clean. Neither hallucinates.

| | degen | clean-term | real calls |
|---|---|---|---|
| new-CS base+adapter | 1/24 (2.4) | 23/24 | 18/24 |
| new-CS merged | **0/24** | **24/24** | 18/24 |

### 3.3 Prompt-sensitivity probe (recon region, merged vs base+adapter)
5 recon prompts × 2 tool-sets:
| Input | merged | base+adapter |
|---|---|---|
| P0 `Do passive recon on api.acme-corp.test` + **Task-only** schema | ❌ Crawl-runaway, no term | ✅ clean |
| P0 + full-16 tools | ✅ | ✅ |
| P1–P4 (OOD recon) × both schemas | ✅ 8/8 | ✅ 8/8 |
| total | **1/10** | 0/10 |

Merged has one **narrow** fragility — a terse recon prompt with a minimal (single-tool) schema.
Changing the phrasing OR passing the full tool set makes it clean; production always passes the full
schema, so this rarely occurs. base+adapter avoids that specific trigger but has its own (2.4).

---

## 4. Key findings, with raw output

### 4.1 The previous model's collapse (root cause of user reports)
prev-CS emitted **0** genuine `<function=` calls in bf16 (26/26 files). Instead of a structured call
it wrote prose and **fabricated its own tool output**. Verbatim:

> `**Thought 1:** ... **Action 1:** ` `` `Task(subagent_type="GHOST", ...)` ``
>
> (5.1 tail) `* start date: Jan 15 ... * expire date: Apr 15 ... * issuer: ... Let's Encrypt ... * SSL certificate verify ok. * Using HTTP2` — a fully invented `curl` SSL handshake for a connection that never happened.

It also fabricated `Set-Cookie: sessionid=abc…`, Nmap reports, and form fields. It *looked* like it
was working (narrated a whole engagement) while executing nothing — **in both bf16 and q8**. This is
the "simulated executions / faked engagements" users reported. It has been withdrawn (§5).

### 4.2 What this model does
new-CS emits a genuine `<tool_call><function=Task>` with a valid archetype and terminates:

> `Thought: phase=authentication_testing. Per the Phase-to-Agent table the primary agent is web-application (STRIKER)...` → `<tool_call><function=Task><parameter=subagent_type> web-application ...</function></tool_call><|im_end|>`

Base already emits calls, but routes with **codenames** ("GHOST") instead of valid archetypes — the
routing this fine-tune corrected (correct-archetype 6→10/24).

### 4.3 Fairness corrections (raw beat the auto-scorer)
- base 3.2 auto-flagged "hallucination": raw showed `"...the .git/HEAD returned a 404 ... not directly exposed ... let me try a few more paths"` — correct. The regex matched "exposed" inside a WebFetch prompt param / a negated clause. **False positive.**
- base 3.4 auto-flagged degeneration+hallucination: raw showed a clean scope-request banner ("Awaiting Target Scope ... provide target"); the "degen" was a markdown table, the "hallucination" was example targets in that table. **Both false positives.**
- new-CS 2.4 degeneration **is real** (`Thought: a credentialed credentialed credentialed …` ×127) and confirmed from raw.

### 4.4 "8-bit GGUF broken" — actually model-broken
base q8 ≈ base bf16 (healthy). prev-CS is broken in **both** bf16 and q8. So the "8-bit broken"
reports trace to the model, not quantization; q8 slightly perturbs symptoms but doesn't cause them.

---

## 5. Deployment & Stage-2

- **Serve:** default is the single-piece merged checkpoint via the Python one-command load — **verified
  end-to-end**: a fresh download of the published repo (16× ~4 GB shards) + the exact user command
  `AutoModelForImageTextToText.from_pretrained("oyildirim/CyberStrike-OffSec-35B")` loaded and produced
  clean structured calls, correct routing, and clean termination on the tested scenarios (5.1/1.3/2.4).
  The LoRA adapter is on the `@adapter` revision for `--enable-lora`. vLLM commands are provided but
  **could not be verified on our CUDA-12.8 hardware** (`qwen3_5_moe` needs vLLM 0.25.1 → CUDA-13
  build) — expected to work on a CUDA-13-capable box; confirm before relying on it.
- **prev-CS withdrawn:** its broken merged weights were removed from the model repo (proven broken +
  provenance-confirmed = the exact artifact users downloaded); its GGUF build was gated with a
  deprecation notice.
- **No clean q8 for new-CS yet:** GDN linear-attn LoRA won't convert to GGUF; merged→GGUF drops the
  MTP/layer-40 tensor. Deferred to a Stage-2 state-dict merge.
- **Stage-2 dataset targets** (all the same root — memorized-not-generalized on OOD): 2.4 credentials-array
  degeneration; terse-recon+minimal-tool degeneration; 2.3 tool over-generalization (Grep for an nmap
  task); 6.2 invalid sub-agent name ("nmap").

Raw outputs for every cell are retained at `ab_results/{cell}/{id}.txt` (no truncation).