EVALUATION.md

CyberStrike-OffSec-35B — Evaluation Report

A controlled before/after evaluation of the CyberStrike fine-tune against the previous (broken) model and the base model, on a 24-scenario tool-calling suite. Every headline number is grounded in raw model output, not auto-scored heuristics (the auto-scorer's hallucination/degeneration flags are noisy — see §1.3 — so raw output is the source of truth throughout).

---

1. Method

1.1 Models × precision (up to 6 cells)

Cell	What it is	Runtime
base bf16 / q8	Qwen/Qwen3.6-35B-A3B (pre-fine-tune control)	bf16: transformers · q8: llama.cpp GGUF Q8_0
new-CS bf16	base + this LoRA adapter (recommended serve)	transformers generate()
new-CS merged bf16	adapter merged into base (the single-piece users download)	transformers generate()
prev-CS bf16 / q8	previous published model oyildirim/CyberStrike-OffSec-35B (merged)	bf16: transformers · q8: its published Q8_0.gguf

new-CS q8 is intentionally not a cell — it has no clean q8 GGUF path (GDN-LoRA can't convert; merged→GGUF drops the MTP tensor). See §5.

1.2 Suite

6 axes × 4 difficulty tiers = 24 scenarios, 62% out-of-distribution (unseen target names, novel phrasing) to separate genuine generalization from memorized patterns. Same system prompt (sysprompt.txt), same 16-tool schema (the exact tools the model was trained on), greedy decoding (temp 0), max_new_tokens=512. Multi-turn scenarios (axes 3–4) pre-feed a real tool observation as a role:"tool" message and measure the model's next turn.

1.3 Scoring — structural + raw verification

A structural parser (no vLLM dependency) extracts <tool_call><function=NAME><parameter=KEY>VALUE calls and checks: parseable, correct tool/archetype, argument types, terminated cleanly (<|im_end|>, no runaway), used-real-observation. Auto-flags for hallucination and degeneration are unreliable (negation false-positives like "not exposed"; markdown tables tripping repetition heuristics; single-line comma-separated runaways missed) — so every hallucination/degeneration verdict here was confirmed against the raw .txt output. This discipline caught several false-positives that would have unfairly favored the fine-tune (e.g. base was initially mis-flagged as hallucinating on 3.2/3.4; raw showed it was correct).

---

2. The 24 scenarios

Legend: ID = training-template-like · OOD = novel target + phrasing · ⚠️LIST = list-structured output (degeneration/q8 watch).

Axis 1 — Tool selection

• 1.1 (easy, ID) Start authentication testing on app.acme-corp.test. → single Task, archetype web-application.
• 1.2 (easy, OOD) Look through the source tree under /srv/helios-api and find any hardcoded credentials or API keys. → dedicated Grep, not Bash grep.
• 1.3 (med, ID, ⚠️LIST) We have api.acme-corp.test in scope and nothing on it yet - get started. → infer passive_recon → Task/explore.
• 1.4 (hard, OOD) On vpn.helios-labs.net I want both a full port scan and a password-policy review of the login page. Creds don't arrive until tomorrow though. → prioritize recon now, defer auth.

Axis 2 — Argument typing

• 2.1 (easy, ID) Start authorization testing on app.acme-corp.test. → string-only Task.
• 2.2 (med, OOD, ⚠️LIST) Crawl staging.bluefin-bank.example in headless mode and go up to 50 pages deep. → hackbrowser steps=int 50, headless=bool true (or approval-gate).
• 2.3 (med, ID) Run a full nmap service scan on 10.0.0.5 in the background, and give it a 300-second timeout. → Bash command + timeout=int + run_in_background=bool.
• 2.4 (hard, OOD) Crawl the authenticated area of portal.zephyr-health.co headlessly, logging in with these two accounts: admin/adminpass and guest/guestpass. → hackbrowser credentials=array + headless=bool.

Axis 3 — Real-observation handling (multi-turn; harness pre-feeds observation)

• 3.1 (easy, ID) scan 10.0.0.5 → obs: 22/ssh OpenSSH 8.9p1, 443/https nginx 1.18.0 → must use those real ports/versions, invent none.
• 3.2 (med, OOD) check dev.pinewood-games.io/.git/HEAD → obs: 404 → accept & pivot, must NOT claim exposed.
• 3.3 (hard, OOD) fetch portal.zephyr-health.co/login → obs: 301 → external OAuth2 IdP → adapt, don't invent a local form.
• 3.4 (hard, OOD) Begin. (bare context) → ask for scope; must NOT fabricate a target/engagement.

Axis 4 — Loop / termination (multi-turn)

• 4.1 (easy, ID) port reachable → obs success → stop with <|im_end|>, no new call.
• 4.2 (med, OOD) Just tell me what web server mail.orion-industries.example runs - one check is enough. → one call, no spam.
• 4.3 (hard, OOD) robots.txt → two failures (timeout, refused) → at most one more try then stop.
• 4.4 (hard, OOD, ⚠️LIST) Do a thorough passive recon sweep of cascade-utilities.example ... → a few steps, no dozens-of-fake-calls runaway.

Axis 5 — Sub-agent delegation

• 5.1 (easy, ID) session-management testing → Task/web-application.
• 5.2 (med, OOD) full external assessment of auth.cascade-utilities.example → recon (explore) first, then web-app/API.
• 5.3 (med, ID) write up the final report → generate_report/self, must NOT delegate via Task.
• 5.4 (hard, OOD) exposed Jenkins w/ default creds on 192.168.44.12 → immediate chain to STRIKER.

Axis 6 — Parallel tool calls

• 6.1 (easy, OOD) two hosts → 2 parallel calls.
• 6.2 (med, ID) three IPs → 3 parallel calls.
• 6.3 (hard, OOD) enumerate subdomains of quantumleap.dev THEN scan → sequential, not parallel.
• 6.4 (hard, OOD, ⚠️LIST) 40 hosts 10.20.0.1–40 → reasonable batch (5–10), not hundreds of calls.

---

3. Result matrix

3.1 Headline tallies (/24)

Metric	base bf16	base q8	new-CS bf16	new-CS merged bf16	prev-CS bf16	prev-CS q8
Genuine structured tool calls	22	23	18	18	0	10
Correct tool / archetype	6	6	10	10	2	3
Clean termination	21	21	23	24	3	5
Degeneration (repetition)	0	0	1 (2.4)	0	0*	2
Fabricated observations (from raw)	none	none	none	none	widespread (8+ scen.)	present

* prev-CS bf16 shows 0 repetition-degeneration because its failure mode is different: coherent prose fabrication (not repetition) — 0 real calls, doesn't terminate. See §4.1.

3.2 Per-scenario (base+adapter vs merged — the deploy-critical pair)

Full 24-scenario A/B: the merged model is cleaner than base+adapter on this suite — the only divergence is 2.4, where base+adapter degenerates and merged is clean. Neither hallucinates.

	degen	clean-term	real calls
new-CS base+adapter	1/24 (2.4)	23/24	18/24
new-CS merged	0/24	24/24	18/24

3.3 Prompt-sensitivity probe (recon region, merged vs base+adapter)

5 recon prompts × 2 tool-sets:

Input	merged	base+adapter
P0 Do passive recon on api.acme-corp.test + Task-only schema	❌ Crawl-runaway, no term	✅ clean
P0 + full-16 tools	✅	✅
P1–P4 (OOD recon) × both schemas	✅ 8/8	✅ 8/8
total	1/10	0/10

Merged has one narrow fragility — a terse recon prompt with a minimal (single-tool) schema. Changing the phrasing OR passing the full tool set makes it clean; production always passes the full schema, so this rarely occurs. base+adapter avoids that specific trigger but has its own (2.4).

---

4. Key findings, with raw output

4.1 The previous model's collapse (root cause of user reports)

prev-CS emitted 0 genuine <function= calls in bf16 (26/26 files). Instead of a structured call it wrote prose and fabricated its own tool output. Verbatim:

**Thought 1:** ... **Action 1:** `Task(subagent_type="GHOST", ...)`

(5.1 tail) * start date: Jan 15 ... * expire date: Apr 15 ... * issuer: ... Let's Encrypt ... * SSL certificate verify ok. * Using HTTP2 — a fully invented curl SSL handshake for a connection that never happened.

It also fabricated Set-Cookie: sessionid=abc…, Nmap reports, and form fields. It looked like it was working (narrated a whole engagement) while executing nothing — in both bf16 and q8. This is the "simulated executions / faked engagements" users reported. It has been withdrawn (§5).

4.2 What this model does

new-CS emits a genuine <tool_call><function=Task> with a valid archetype and terminates:

Thought: phase=authentication_testing. Per the Phase-to-Agent table the primary agent is web-application (STRIKER)... → <tool_call><function=Task><parameter=subagent_type> web-application ...</function></tool_call><|im_end|>

Base already emits calls, but routes with codenames ("GHOST") instead of valid archetypes — the routing this fine-tune corrected (correct-archetype 6→10/24).

4.3 Fairness corrections (raw beat the auto-scorer)

• base 3.2 auto-flagged "hallucination": raw showed "...the .git/HEAD returned a 404 ... not directly exposed ... let me try a few more paths" — correct. The regex matched "exposed" inside a WebFetch prompt param / a negated clause. False positive.
• base 3.4 auto-flagged degeneration+hallucination: raw showed a clean scope-request banner ("Awaiting Target Scope ... provide target"); the "degen" was a markdown table, the "hallucination" was example targets in that table. Both false positives.
• new-CS 2.4 degeneration is real (Thought: a credentialed credentialed credentialed … ×127) and confirmed from raw.

4.4 "8-bit GGUF broken" — actually model-broken

base q8 ≈ base bf16 (healthy). prev-CS is broken in both bf16 and q8. So the "8-bit broken" reports trace to the model, not quantization; q8 slightly perturbs symptoms but doesn't cause them.

---

5. Deployment & Stage-2

• Serve: default is the single-piece merged checkpoint via the Python one-command load — verified end-to-end: a fresh download of the published repo (16× ~4 GB shards) + the exact user command AutoModelForImageTextToText.from_pretrained("oyildirim/CyberStrike-OffSec-35B") loaded and produced clean structured calls, correct routing, and clean termination on the tested scenarios (5.1/1.3/2.4). The LoRA adapter is on the @adapter revision for --enable-lora. vLLM commands are provided but could not be verified on our CUDA-12.8 hardware (qwen3_5_moe needs vLLM 0.25.1 → CUDA-13 build) — expected to work on a CUDA-13-capable box; confirm before relying on it.
• prev-CS withdrawn: its broken merged weights were removed from the model repo (proven broken + provenance-confirmed = the exact artifact users downloaded); its GGUF build was gated with a deprecation notice.
• No clean q8 for new-CS yet: GDN linear-attn LoRA won't convert to GGUF; merged→GGUF drops the MTP/layer-40 tensor. Deferred to a Stage-2 state-dict merge.
• Stage-2 dataset targets (all the same root — memorized-not-generalized on OOD): 2.4 credentials-array degeneration; terse-recon+minimal-tool degeneration; 2.3 tool over-generalization (Grep for an nmap task); 6.2 invalid sub-agent name ("nmap").

Raw outputs for every cell are retained at ab_results/{cell}/{id}.txt (no truncation).