| # CyberStrike-OffSec-35B — Evaluation Report |
|
|
| A controlled before/after evaluation of the CyberStrike fine-tune against the previous (broken) |
| model and the base model, on a 24-scenario tool-calling suite. Every headline number is grounded in |
| **raw model output**, not auto-scored heuristics (the auto-scorer's hallucination/degeneration flags |
| are noisy — see §1.3 — so raw output is the source of truth throughout). |
|
|
| --- |
|
|
| ## 1. Method |
|
|
| ### 1.1 Models × precision (up to 6 cells) |
| | Cell | What it is | Runtime | |
| |---|---|---| |
| | base bf16 / q8 | `Qwen/Qwen3.6-35B-A3B` (pre-fine-tune control) | bf16: transformers · q8: llama.cpp GGUF Q8_0 | |
| | **new-CS bf16** | base + this LoRA adapter (recommended serve) | transformers `generate()` | |
| | **new-CS merged bf16** | adapter merged into base (the single-piece users download) | transformers `generate()` | |
| | prev-CS bf16 / q8 | previous published model `oyildirim/CyberStrike-OffSec-35B` (merged) | bf16: transformers · q8: its published `Q8_0.gguf` | |
|
|
| new-CS q8 is intentionally **not** a cell — it has no clean q8 GGUF path (GDN-LoRA can't convert; |
| merged→GGUF drops the MTP tensor). See §5. |
|
|
| ### 1.2 Suite |
| 6 axes × 4 difficulty tiers = **24 scenarios**, **62% out-of-distribution** (unseen target names, |
| novel phrasing) to separate genuine generalization from memorized patterns. Same system prompt |
| (`sysprompt.txt`), same 16-tool schema (the exact tools the model was trained on), greedy decoding |
| (temp 0), `max_new_tokens=512`. Multi-turn scenarios (axes 3–4) pre-feed a real tool observation as a |
| `role:"tool"` message and measure the model's next turn. |
|
|
| ### 1.3 Scoring — structural + raw verification |
| A structural parser (no vLLM dependency) extracts `<tool_call><function=NAME><parameter=KEY>VALUE` |
| calls and checks: parseable, correct tool/archetype, argument types, terminated cleanly (`<|im_end|>`, |
| no runaway), used-real-observation. **Auto-flags for hallucination and degeneration are unreliable** |
| (negation false-positives like "not exposed"; markdown tables tripping repetition heuristics; |
| single-line comma-separated runaways missed) — so every hallucination/degeneration verdict here was |
| confirmed against the raw `.txt` output. This discipline caught several false-positives that would |
| have unfairly favored the fine-tune (e.g. base was initially mis-flagged as hallucinating on 3.2/3.4; |
| raw showed it was correct). |
|
|
| --- |
|
|
| ## 2. The 24 scenarios |
|
|
| Legend: **ID** = training-template-like · **OOD** = novel target + phrasing · **⚠️LIST** = |
| list-structured output (degeneration/q8 watch). |
|
|
| ### Axis 1 — Tool selection |
| - **1.1** (easy, ID) `Start authentication testing on app.acme-corp.test.` → single `Task`, archetype `web-application`. |
| - **1.2** (easy, OOD) `Look through the source tree under /srv/helios-api and find any hardcoded credentials or API keys.` → dedicated `Grep`, not `Bash grep`. |
| - **1.3** (med, ID, ⚠️LIST) `We have api.acme-corp.test in scope and nothing on it yet - get started.` → infer passive_recon → `Task`/`explore`. |
| - **1.4** (hard, OOD) `On vpn.helios-labs.net I want both a full port scan and a password-policy review of the login page. Creds don't arrive until tomorrow though.` → prioritize recon now, defer auth. |
| |
| ### Axis 2 — Argument typing |
| - **2.1** (easy, ID) `Start authorization testing on app.acme-corp.test.` → string-only `Task`. |
| - **2.2** (med, OOD, ⚠️LIST) `Crawl staging.bluefin-bank.example in headless mode and go up to 50 pages deep.` → `hackbrowser` `steps`=int 50, `headless`=bool true (or approval-gate). |
| - **2.3** (med, ID) `Run a full nmap service scan on 10.0.0.5 in the background, and give it a 300-second timeout.` → `Bash` command + `timeout`=int + `run_in_background`=bool. |
| - **2.4** (hard, OOD) `Crawl the authenticated area of portal.zephyr-health.co headlessly, logging in with these two accounts: admin/adminpass and guest/guestpass.` → `hackbrowser` `credentials`=array + `headless`=bool. |
| |
| ### Axis 3 — Real-observation handling (multi-turn; harness pre-feeds observation) |
| - **3.1** (easy, ID) scan 10.0.0.5 → obs: `22/ssh OpenSSH 8.9p1`, `443/https nginx 1.18.0` → must use those real ports/versions, invent none. |
| - **3.2** (med, OOD) check dev.pinewood-games.io/.git/HEAD → obs: `404` → accept & pivot, must NOT claim exposed. |
| - **3.3** (hard, OOD) fetch portal.zephyr-health.co/login → obs: `301 → external OAuth2 IdP` → adapt, don't invent a local form. |
| - **3.4** (hard, OOD) `Begin.` (bare context) → ask for scope; must NOT fabricate a target/engagement. |
| |
| ### Axis 4 — Loop / termination (multi-turn) |
| - **4.1** (easy, ID) port reachable → obs success → stop with `<|im_end|>`, no new call. |
| - **4.2** (med, OOD) `Just tell me what web server mail.orion-industries.example runs - one check is enough.` → one call, no spam. |
| - **4.3** (hard, OOD) robots.txt → two failures (timeout, refused) → at most one more try then stop. |
| - **4.4** (hard, OOD, ⚠️LIST) `Do a thorough passive recon sweep of cascade-utilities.example ...` → a few steps, no dozens-of-fake-calls runaway. |
|
|
| ### Axis 5 — Sub-agent delegation |
| - **5.1** (easy, ID) session-management testing → `Task`/`web-application`. |
| - **5.2** (med, OOD) full external assessment of auth.cascade-utilities.example → recon (`explore`) first, then web-app/API. |
| - **5.3** (med, ID) `write up the final report` → `generate_report`/self, must NOT delegate via `Task`. |
| - **5.4** (hard, OOD) exposed Jenkins w/ default creds on 192.168.44.12 → immediate chain to STRIKER. |
|
|
| ### Axis 6 — Parallel tool calls |
| - **6.1** (easy, OOD) two hosts → 2 parallel calls. |
| - **6.2** (med, ID) three IPs → 3 parallel calls. |
| - **6.3** (hard, OOD) enumerate subdomains of quantumleap.dev THEN scan → sequential, not parallel. |
| - **6.4** (hard, OOD, ⚠️LIST) 40 hosts 10.20.0.1–40 → reasonable batch (5–10), not hundreds of calls. |
|
|
| --- |
|
|
| ## 3. Result matrix |
|
|
| ### 3.1 Headline tallies (/24) |
| | Metric | base bf16 | base q8 | **new-CS bf16** | **new-CS merged bf16** | prev-CS bf16 | prev-CS q8 | |
| |---|---|---|---|---|---|---| |
| | Genuine structured tool calls | 22 | 23 | 18 | 18 | **0** | 10 | |
| | Correct tool / archetype | 6 | 6 | **10** | 10 | 2 | 3 | |
| | Clean termination | 21 | 21 | 23 | **24** | **3** | 5 | |
| | Degeneration (repetition) | 0 | 0 | 1 (2.4) | **0** | 0* | 2 | |
| | Fabricated observations (from raw) | none | none | **none** | **none** | **widespread (8+ scen.)** | present | |
|
|
| \* prev-CS bf16 shows 0 *repetition*-degeneration because its failure mode is different: coherent |
| **prose fabrication** (not repetition) — 0 real calls, doesn't terminate. See §4.1. |
|
|
| ### 3.2 Per-scenario (base+adapter vs merged — the deploy-critical pair) |
| Full 24-scenario A/B: the merged model is **cleaner than base+adapter on this suite** — the only |
| divergence is 2.4, where base+adapter degenerates and merged is clean. Neither hallucinates. |
|
|
| | | degen | clean-term | real calls | |
| |---|---|---|---| |
| | new-CS base+adapter | 1/24 (2.4) | 23/24 | 18/24 | |
| | new-CS merged | **0/24** | **24/24** | 18/24 | |
|
|
| ### 3.3 Prompt-sensitivity probe (recon region, merged vs base+adapter) |
| 5 recon prompts × 2 tool-sets: |
| | Input | merged | base+adapter | |
| |---|---|---| |
| | P0 `Do passive recon on api.acme-corp.test` + **Task-only** schema | ❌ Crawl-runaway, no term | ✅ clean | |
| | P0 + full-16 tools | ✅ | ✅ | |
| | P1–P4 (OOD recon) × both schemas | ✅ 8/8 | ✅ 8/8 | |
| | total | **1/10** | 0/10 | |
|
|
| Merged has one **narrow** fragility — a terse recon prompt with a minimal (single-tool) schema. |
| Changing the phrasing OR passing the full tool set makes it clean; production always passes the full |
| schema, so this rarely occurs. base+adapter avoids that specific trigger but has its own (2.4). |
|
|
| --- |
|
|
| ## 4. Key findings, with raw output |
|
|
| ### 4.1 The previous model's collapse (root cause of user reports) |
| prev-CS emitted **0** genuine `<function=` calls in bf16 (26/26 files). Instead of a structured call |
| it wrote prose and **fabricated its own tool output**. Verbatim: |
|
|
| > `**Thought 1:** ... **Action 1:** ` `` `Task(subagent_type="GHOST", ...)` `` |
| > |
| > (5.1 tail) `* start date: Jan 15 ... * expire date: Apr 15 ... * issuer: ... Let's Encrypt ... * SSL certificate verify ok. * Using HTTP2` — a fully invented `curl` SSL handshake for a connection that never happened. |
| |
| It also fabricated `Set-Cookie: sessionid=abc…`, Nmap reports, and form fields. It *looked* like it |
| was working (narrated a whole engagement) while executing nothing — **in both bf16 and q8**. This is |
| the "simulated executions / faked engagements" users reported. It has been withdrawn (§5). |
| |
| ### 4.2 What this model does |
| new-CS emits a genuine `<tool_call><function=Task>` with a valid archetype and terminates: |
| |
| > `Thought: phase=authentication_testing. Per the Phase-to-Agent table the primary agent is web-application (STRIKER)...` → `<tool_call><function=Task><parameter=subagent_type> web-application ...</function></tool_call><|im_end|>` |
| |
| Base already emits calls, but routes with **codenames** ("GHOST") instead of valid archetypes — the |
| routing this fine-tune corrected (correct-archetype 6→10/24). |
| |
| ### 4.3 Fairness corrections (raw beat the auto-scorer) |
| - base 3.2 auto-flagged "hallucination": raw showed `"...the .git/HEAD returned a 404 ... not directly exposed ... let me try a few more paths"` — correct. The regex matched "exposed" inside a WebFetch prompt param / a negated clause. **False positive.** |
| - base 3.4 auto-flagged degeneration+hallucination: raw showed a clean scope-request banner ("Awaiting Target Scope ... provide target"); the "degen" was a markdown table, the "hallucination" was example targets in that table. **Both false positives.** |
| - new-CS 2.4 degeneration **is real** (`Thought: a credentialed credentialed credentialed …` ×127) and confirmed from raw. |
| |
| ### 4.4 "8-bit GGUF broken" — actually model-broken |
| base q8 ≈ base bf16 (healthy). prev-CS is broken in **both** bf16 and q8. So the "8-bit broken" |
| reports trace to the model, not quantization; q8 slightly perturbs symptoms but doesn't cause them. |
| |
| --- |
| |
| ## 5. Deployment & Stage-2 |
| |
| - **Serve:** default is the single-piece merged checkpoint via the Python one-command load — **verified |
| end-to-end**: a fresh download of the published repo (16× ~4 GB shards) + the exact user command |
| `AutoModelForImageTextToText.from_pretrained("oyildirim/CyberStrike-OffSec-35B")` loaded and produced |
| clean structured calls, correct routing, and clean termination on the tested scenarios (5.1/1.3/2.4). |
| The LoRA adapter is on the `@adapter` revision for `--enable-lora`. vLLM commands are provided but |
| **could not be verified on our CUDA-12.8 hardware** (`qwen3_5_moe` needs vLLM 0.25.1 → CUDA-13 |
| build) — expected to work on a CUDA-13-capable box; confirm before relying on it. |
| - **prev-CS withdrawn:** its broken merged weights were removed from the model repo (proven broken + |
| provenance-confirmed = the exact artifact users downloaded); its GGUF build was gated with a |
| deprecation notice. |
| - **No clean q8 for new-CS yet:** GDN linear-attn LoRA won't convert to GGUF; merged→GGUF drops the |
| MTP/layer-40 tensor. Deferred to a Stage-2 state-dict merge. |
| - **Stage-2 dataset targets** (all the same root — memorized-not-generalized on OOD): 2.4 credentials-array |
| degeneration; terse-recon+minimal-tool degeneration; 2.3 tool over-generalization (Grep for an nmap |
| task); 6.2 invalid sub-agent name ("nmap"). |
|
|
| Raw outputs for every cell are retained at `ab_results/{cell}/{id}.txt` (no truncation). |
|
|
