code/docs/PROJECT_GOAL.md

Project Goal: CyberGym-Targeted Fine-Tune of Qwen3.6-27B

Status: Planning / pre-implementation Owner: Adhik Joshi · Context: college research project (authorized security research / education) Last updated: 2026-06-27 Repo: infosec (greenfield)

---

0. TL;DR

Fine-tune the open-weight Qwen3.6-27B (reasoning + agentic, Apache-2.0) to raise its score on CyberGym — UC Berkeley's agentic vulnerability-reproduction benchmark — using a curated mix of offensive + defensive cybersecurity data, with <think> reasoning traces, trained as LoRA (then merged).

Confirmed decisions (2026-06-27):

1. CyberGym is the PRIMARY objective (success rate ↑), not knowledge benchmarks.
2. Data must be offensive + defensive.
3. LoRA for the trial; merge into the base checkpoint if it works (merge_and_unload).
4. <think> is required — the model must reason through tasks.

The hard truth this creates (read this): CyberGym is won by an agentic loop — explore a C/C++ repo over ~100 tool-calling steps and write a PoC input file that crashes the pre-patch build but not the patched one. Dedicated 32B coding-agent models still score ≤2%; frontier models top out ~18–22%. Supervised fine-tuning on Q&A — even great offensive data — has a low ceiling here. To actually move CyberGym we need a staged program:

SFT (LoRA) on code-vuln + reasoning + agentic-trajectory data → DPO on secure/insecure code → RL on the CyberGym task itself (verified pass/fail reward). The RL stage is the real lever; SFT makes the model coherent and reason-y enough for RL to work.

This document plans all stages, but Stage 1 (LoRA SFT) is the trial. Expect SFT to deliver modest CyberGym gains + large gains on code-vuln/knowledge evals; budget for the RL stage to get a big CyberGym jump.

---

1. Objective & Success Criteria

Primary: increase CyberGym success rate over the base Qwen3.6-27B, without regressing tool-calling / agentic ability.

Metric	Tier	Baseline	Target
CyberGym success rate (level1 sample, OpenHands + vLLM)	PRIMARY	base (expect low single digits)	> base; aim +2–5+ pts via SFT, more via RL
CyberGym 10-task subset — valid tool-calls emitted	Guardrail	base	no tool-call regression
Vulnerability-detection accuracy (C/C++)	Secondary	base	+5–15 pts
Cyber-knowledge MCQ (CyberMetric/SecEval/CTI-Bench)	Secondary	base	+5–15 pts
IFEval / MMLU-general	Guardrail	base	≥ base − 2 pts (no forgetting)

Definition of done (Stage 1): a merged LoRA checkpoint + eval report (CyberGym + code-vuln + guardrails + loss curves) showing net improvement and intact agentic ability, plus a reproducible pipeline in this repo. Stretch (Stage 2+): DPO + RL for a larger CyberGym jump.

---

2. Base Model — Qwen3.6-27B (verified)

• Real & live, released 2026-04-24, ~5.6M downloads, Apache-2.0 (clean commercial/research use, not gated).
• ~27.8B params, weights 55.6 GB bf16. Dense hybrid attention: 64 layers = 48 Gated-DeltaNet (linear) + 16 full-attention GQA layers. Hidden 5120, FFN 17408.
• Reasoning model with a ChatML <think> channel ON by default (enable_thinking=false injects an empty think block). Matches your <think> requirement natively.
• Multimodal (vision tower) — freeze it for text/code data.
• Untied embeddings, vocab 248,320; no BOS, EOS = <|im_end|>. Native context 256K.
• MTP head (speculative decoding) — freeze/ignore in SFT.
• Framework risk (Phase-0 gate): custom hybrid arch (model_type ~`qwen3_5) needs **very recent transformers (≥4.57.1)** + FLA/causal-conv1d` kernels. Older Axolotl/Unsloth/PEFT may not support it.
• Fallback if blocked: Qwen3-32B (dense, standard attention, text-only, well-supported) — also has a thinking mode. Memory math scales ×32/27.

---

3. Datasets

3.A The CyberGym-aligned set (NEW — the focus going forward)

Verified via HF API (rows/schema/license). Ranked by relevance to the agentic vuln-reproduction task.

Tier	Dataset	Rows	Schema	Why it matters for CyberGym	License
1 · C/C++ vuln code	athrv/megavul-vulnerability-detection	353,873	instruction/input/output + cwe_ids, cve_id, severity, cvss, repo, func, is_vulnerable	Real C/C++ functions w/ CVE/CWE labels — core memory-safety understanding	⚠️ none stated
1 · C++ vuln code	smart-code-analyzer-team/cpp-vulnerability-dataset	69,299	raw_snippet, security_context, data_flow_graph, ast_metadata…	C++ snippets w/ security context + data-flow — pattern recognition	⚠️ none stated
1 · CVE patches	DetectVul/CVEFixes (+ hitoshura25/cvefixes)	5,730	lines, raw_lines, label, type	Pre/post-patch diff lines — mirrors CyberGym's patch differential	⚠️ none stated
1 · SARD/Juliet C	lynchorange/SARD_Vulnerability_Detection_C	(n/a)	C test cases	NIST-style labeled vulnerable C	⚠️ verify (SARD is public-domain)
2 · Reasoning <think>	SecCoderX/...Reasoning_Vulnerability_Detection_SFT_Cold_Start	37,113	instruction, output	Vuln-detection with reasoning — feeds the <think> channel	⚠️ none stated
2 · Reasoning <think>	Leopo1d/OpenVul_Rejection_Sampling...Reasoning...SFT (+ GT companion)	6,276	key, prompt, completion	Reject-sampled reasoning traces — exactly our <think> method, pre-built	Apache-2.0
3 · Agentic loop (top)	nvidia/Nemotron-Terminal-Corpus	~366,000	conversations (terminal command→observe)	Closest public analogue to CyberGym's run-command-observe loop	CC-BY-4.0
3 · Agentic loop	Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k (bash-only), SWE-bench/SWE-smith-trajectories (76k), nebius/SWE-agent-trajectories (80k)	66–80k	messages/trajectory + patches + eval logs	Teach the multi-turn tool loop (Python-centric → use for loop mechanics, not domain)	MIT / CC-BY-4.0
3 · Reasoning+tools	interstellarninja/tool-use-multiturn-reasoning	14,600	conversations, tools (5–13 turns, interleaved <think>)	Call→reason→observe rhythm with <think>	Apache-2.0
3 · RL warm-start	nvidia/Nemotron-RL-Agentic-SWE-Pivot-v1	50,000	tool-call traj + pass_rate	Verifiable-reward format for Stage-3 RL	CC-BY-4.0
3 · Format blueprint	SecureAI-SE/fuzz-agent-trajectories	10	think→act(tool)→observe, offensive fuzzing	Exact offensive-agent format match — seed/template only, not trainable volume	CC-BY-4.0
2 · Reasoning <think> (cyber)	theelderemo/pentesting-explanations	5,900	MCQ + explanation + <think>	Offensive pentest reasoning with <think>	Apache-2.0
3 · Tool-output skill	tegridydev/infosec-tool-output	776	SAST findings → next-action	Teaches the "read security-tool output" sub-skill	Apache-2.0
4 · DPO	CyberNative/Code_Vulnerability_Security_DPO	4,656	lang, vulnerability, system, question, chosen, rejected	Ready-made secure-vs-vulnerable preference pairs for the DPO stage	Apache-2.0
5 · CVE knowledge	AlicanKiraz0/All-CVE-Records-Training-Dataset	297,441	System/User/Assistant	Broad CVE breadth (defensive context/recall)	Apache-2.0
5 · Threat-intel (def.)	reloading0101/threat-intelligence-dataset	~9,200	CTI Q&A fact-checked vs MITRE/CISA/EPSS	High-provenance defensive breadth	CC-BY-4.0
6 · RL env	AdithyaSK/repo2rlenv-cve-patches	(n/a)	CVE patches → RL env	CVE patches packaged as RL environments — Stage-3 fuel	Apache-2.0

Dedup flags: Trendyol/All-CVE-Chat-MultiTurn-1999-2025-Dataset is byte-identical to AlicanKiraz0/All-CVE-Records-... (same 297,441 rows / same size) → keep one. (Same pattern as the Soban/Trendyol mirror from the original list.) License flags: several Tier-1 sets have no stated license. For a college research project this is usually defensible (academic use of public, derived-from-open-source data), but record provenance and prefer the licensed equivalents where possible (OpenVul Apache-2.0, CVEFixes upstream is CC, SARD is NIST public-domain). Gated/non-commercial: WhitzardAgent/CyberSecurity-1M & -100B are the best-curated recent cyber corpora (exploitation chains, Sigma/YARA, CoT) but are gated behind an institutional-email approval (a personal Gmail will be rejected) and licensed academic/non-commercial — usable for a college project only if access is granted; do not assume availability.

⚠️ Structural gap (the key dataset finding): No public dataset contains end-to-end CyberGym-style C/C++ "PoC → crash → iterate" agent trajectories. The only exact-format match (SecureAI-SE/fuzz-agent-trajectories) is 10 rows — a blueprint, not training volume. Consequence for the plan: the agentic-loop skill must come from (a) transferring loop mechanics from the generic terminal/SWE trajectory sets above, (b) synthesizing our own trajectories (run an agent harness over training-split CyberGym/ARVO tasks, keep successful runs as SFT data), and/or (c) Stage-3 RL, which learns the loop directly from the environment. This is why RL (not SFT) is the real lever for the primary metric.

3.B The original 6 (now supplementary)

Still useful as defensive breadth + instruction-following, but secondary to 3.A for the CyberGym goal:

• ✅ AlicanKiraz0/Fenrir-v2.1 (99,870, EN, Apache-2.0), Trendyol/Cybersecurity-IT (53,202, EN, Apache-2.0), hcnote/High-Quality (270,271, ZH+EN, MIT, has offensive/exploit content + code → useful here).
• ◐ jobby32/QA-with-negatives (17,263) — question→positive SFT only.
• ⛔ Soban1234/... (mirror of Trendyol — drop). ⛔ clouditera/security-paper (raw bilingual book text, unspecified license — exclude; optional CPT only after legal review).

3.D Platform / language breadth (NodeJS · Python · PHP · Apache · Linux · web)

CyberGym's core is C/C++ memory-safety bugs (OSS-Fuzz/ARVO), and Apache httpd, nginx, and the Linux kernel are themselves C — so the C/C++ sets above already cover the "systems software" targets. NodeJS / Python / PHP are different vuln classes (injection, deserialization, SSRF, path traversal) — valuable for breadth and useful if we add web targets, but secondary to C/C++ for the CyberGym score. Honest framing: prioritize the first group for CyberGym; add the second for a well-rounded offensive+defensive model.

Group	Dataset	Rows	Schema	Coverage	License
Multi-language (best for your ask)	hitoshura25/crossvul	9,313	cwe_id, language, vulnerable_code, fixed_code, source	PHP/JS/Python/C/C++/Java/Ruby/Go… vuln+fix pairs (CrossVul)	Apache-2.0
C/C++ (large, real-world)	bstee615/diversevul	330,492	func, target, cwe, project, commit	C/C++ across many OSS projects (incl. Apache/Linux-type)	⚠️ none
C/C++ (highest quality)	colin/PrimeVul	233,941	func, target, cwe, project, commit	Rigorously de-noised C/C++ vuln detection	MIT
Linux/systems (C)	mjbommar/linux-cve-dossiers	2,556	cve_id, package, scope, completeness…	Linux CVE dossiers (kernel/distro packages)	CC-BY-4.0
Web / OWASP (knowledge)	shahrukh95/OWASP-and-NVD-question-answer-dataset	22,958	Question, Answer, split	OWASP + NVD web-vuln Q&A	⚠️ none
PHP (offensive/defensive)	nbuser32/PHP-Webshell-Dataset	30,270	instruction/input/output	PHP webshell detection/analysis	MIT
Web injection (NodeJS/PHP/Python apps)	PurpleAILAB/chatML_SQL_injection_dataset (+ XSS sets)	~2,000	conversations (ChatML)	SQLi/XSS exploitation in chat format	⚠️ none
Python	maddyrucos/code_vulnerability_python	848	func, target, project	Small Python vuln set	MIT

Notes: nasirhm/reposvul (repo-level, multi-language — adds whole-repo context like CyberGym) and Humanlearning/CyberSecurity_OWASP-sft-dataset looked relevant but their HF viewers returned empty — verify at source before relying on them. Several web sets are tiny/no-license → use as flavor, not backbone; record provenance.

3.C Corpus shape (planning)

• Code/vuln + reasoning core (3.A tiers 1–2): ~470K rows.
• Agentic trajectories (3.A tier 3): ~80K+ (heavy per-example token count — these are long).
• DPO pairs: ~5K. CVE/defensive breadth: ~300K (deduped) + original-6 supplementary.
• After dedup + decontamination, plan for ~400–600K SFT examples, token count dominated by the long agentic trajectories. Cap any single source at ≤30–40% of tokens.

---

4. Benchmark — CyberGym (verified) + the eval suite

CyberGym (sunblaze-ucb, arXiv 2506.02548, ICLR 2026): given a pre-patch C/C++ codebase + a vuln description, an agent must produce a PoC input file that triggers a sanitizer crash on the pre-patch build and not on the patched build (differential validation). 1,507 tasks / 188 projects; levels 0–3 (more info → easier); a 10-task curated subset ships for quick tests. Harness is agentic + Docker, uses LiteLLM → supports any OpenAI-compatible endpoint (our vLLM). Use binary-only mode (~130 GB). Baselines: GPT-5 ~22%, Sonnet-4 ~18%, open 32B coding agents ≤2%.

Self-hosted eval path (confirmed): serve the merged checkpoint with vLLM (--served-model-name my-qwen, tool-calling enabled) → run OpenHands with --model openai/my-qwen --llm.base_url http://HOST:PORT/v1, LLM_API_KEY=EMPTY, --difficulty level1.

Eval suite we'll report:

• PRIMARY: CyberGym — 10-task smoke (tool-call sanity) → ~100–200 task level1 sample (real number) → full 1,507 only if budget allows.
• Secondary: held-out C/C++ vulnerability-detection accuracy (from MegaVul/CVEFixes test splits); cyber-knowledge MCQ (CyberMetric, SecEval, CTI-Bench).
• Guardrails: IFEval (instruction following), MMLU-general (forgetting).
• Decontaminate all training data against every eval set (13-gram + fuzzy + embedding); freeze test splits.

---

5. Training Strategy (staged)

Stage 0 — Infra & go/no-go

Confirm Qwen3.6-27B loads + LoRA-trains + serves under vLLM with tool-calling on current transformers. If blocked, fall back to Qwen3-32B. Set up storage (~236 GB CyberGym / ~130 GB binary server), W&B, vLLM.

Stage 1 — LoRA SFT (the trial; "benchmark after each epoch")

Goal: make the model fluent in C/C++ vuln reasoning + the agentic tool-use format, with <think> traces.

• Data: 3.A tiers 1–3 (code-vuln + reasoning + agentic trajectories) + DPO-source positives + defensive breadth, all normalized to ChatML with <think> reasoning.
• <think> policy: keep reasoning ON. Use the pre-built reasoning datasets (SecCoderX, OpenVul) directly; for code-vuln sets lacking traces, synthesize <think> via rejection sampling (generate N reasonings with a strong model, keep those that reach the correct label/patch). Apply the format uniformly — inconsistent <think> is the #1 way to damage a Qwen reasoning model.
• Method: LoRA r=64, α=128, rsLoRA on, all linear targets, dropout 0.05, freeze vision tower + MTP head.
• Hyperparameters: lr 2e-4 cosine, warmup 0.03–0.05, 2–3 epochs, effective batch ~128, seq 8192 (agentic trajectories are long), packing on, gradient checkpointing on, loss-masked on assistant tokens, paged AdamW-8bit.
• Pilot first: a QLoRA smoke-test on a subset (1× 80GB, ~$20–60) to validate the pipeline before the real LoRA run.
• Per-epoch eval: at each epoch checkpoint run the fast eval (vuln-detection slice + IFEval + CyberGym 10-task). Select the checkpoint by best eval + non-rising val loss, never lowest train loss.

Stage 2 — DPO (optional, cheap)

Use CyberNative/Code_Vulnerability_Security_DPO (chosen=secure, rejected=vulnerable) to sharpen secure/insecure discrimination. LoRA-DPO, lr ~5e-6, 1 epoch. Small, fast, low risk.

Stage 3 — RL on the CyberGym task — "the real lever" (detailed)

Why RL fits CyberGym perfectly. CyberGym hands us a verifiable, machine-checkable, hard-to-game reward: the submitted PoC either crashes the pre-patch build and not the patched build, or it doesn't. That is the ideal setup for RLVR (RL with Verifiable Rewards) — the same paradigm behind reasoning models (e.g. DeepSeek-R1). No reward model to train; the environment is the judge. The catch: this is agentic, long-horizon, sparse-reward RL (a success needs ~100 correct tool-calling steps, reward only at the end), which is much harder than single-turn math RLVR. So we climb a ladder from cheap to powerful.

The ladder (do them in order):

1. Rejection-sampling Fine-Tuning / Expert Iteration (STaR / RFT) — START HERE, highest ROI.

   • Run the SFT model on training-split tasks, many high-temperature rollouts each.
   • Keep only trajectories that produced a validated PoC (reward = 1).
   • SFT on those winning trajectories (with their <think> reasoning).
   • Repeat — as the model improves it solves more tasks → generates more training data → improves further.
   • This is "RL" via filtered behavior cloning: no PPO/GRPO infrastructure, far cheaper and more stable, and it directly closes the "no public C/C++ agent-trajectory data" gap by manufacturing our own. For a college budget this alone is likely the biggest single win.

2. GRPO (Group Relative Policy Optimization) — the on-policy push.

   • Modern policy-gradient for verifiable rewards: sample a group of rollouts per task, advantage = reward normalized within the group, no value/critic model (cheaper than PPO). Best applied after RFT has warmed the model up.
   • Alternatives: PPO (needs a critic, heavier), RLOO, or offline trajectory-DPO (chosen = solved run, rejected = failed run) as a lightweight stepping stone.

3. Curriculum + reward shaping (to fight sparsity).

   • Curriculum: start at level3 (easiest — includes the patch + crash stack trace), then level2 → level1 → level0; or start from CyberGym's 5 "solvable" tasks. Get signal flowing before tackling hard tasks.
   • Shaped reward (dense, but outcome-dominant):
     • • locate the vulnerable function/file,
     • • produce any sanitizer crash on the pre-patch build (even the wrong bug),
     • + full reward for the differential success (crash pre-patch, clean post-patch — the true objective),
     • • small format reward (valid tool-call syntax, <think> present, valid PoC file written),
     • − small per-step/length penalty for efficiency.
   • Keep the differential-success term dominant so the model can't farm the shaping terms.

4. Anti-reward-hacking. The differential check already blocks the cheapest exploit (triggering an unrelated crash). Also: keep eval tasks completely out of RL training; watch for the agent crashing/escaping the harness itself; cap the tool/step budget.

Environment & data.

• The CyberGym Docker harness is the RL environment — rollouts POST candidate PoCs to the validation server → binary reward. Use binary-only mode (~130 GB) for rollout speed.
• Mandatory train/eval split: RL-train on a held-in subset of the 1,507 tasks (+ external verified tasks: ARVO / OSS-Fuzz reproducible bugs, AdithyaSK/repo2rlenv-cve-patches), evaluate on a disjoint held-out subset. Never train on eval tasks. More distinct environments → better generalization.

Frameworks.

• verl (ByteDance) — leading scalable RLVR/GRPO framework; supports multi-turn agentic rollouts + tool calling; pairs with vLLM. Top pick for the full RL run.
• OpenRLHF — PPO/GRPO/RLOO at scale. SkyRL / RAGEN / Agent-R1 / verifiers — purpose-built for multi-turn agentic RL (the hard part is rollouts that interleave model ↔ environment). TRL GRPOTrainer — simplest, good for RFT / single-turn proxies.
• The RFT phase needs no RL framework — just vLLM (rollouts) + the CyberGym harness (scoring) + a normal SFT trainer.

Compute reality (why this is the costly phase). Agentic RL is rollout-dominated: each episode = up to ~100 LLM calls + Docker execution, and GRPO needs a group (8–16) of episodes per task per step → thousands of ~100-step rollouts. You need a vLLM inference cluster (generation) + many parallel Docker workers (environment) + training GPUs (updates); throughput of inference/env, not the gradient step, is the bottleneck. Budget $1–5K+ and it can balloon — which is exactly why RFT (cheap, off-policy) should capture most of the gain first.

Recommended RL path (college budget): SFT (Stage 1) → RFT/Expert-Iteration on the training split (prioritize this) → if budget remains GRPO with curriculum + shaped reward via verl → evaluate on the held-out split throughout.

Realistic expectation. Frontier models sit at ~18–22% on CyberGym; dedicated 32B coding agents score ≤2% without this program. A 27B model after SFT + RFT (+ optional GRPO) can credibly aim for mid-single digits → low double digits at level1 — a strong, publishable result for an open 27B in a college research setting.

LoRA → merge

After Stage 1 (and DPO), peft.merge_and_unload() folds adapters into base weights → standalone checkpoint, zero inference overhead. bf16 LoRA merges cleanly; QLoRA merge is lossy (merge into dequantized bf16, then re-quantize for serving if needed). Re-benchmark the merged model to confirm parity before continuing to RL.

---

6. GPU, Time & Cost

470K SFT rows incl. long agentic trajectories → plan **600M–1B tokens/epoch** (trajectories dominate). 8× H100 ≈ $16–24/hr; 1–2× H100 ≈ $2–3/GPU-hr.

Stage	Method	Hardware	Wallclock	Compute $
Pilot	QLoRA, subset	1× H100 80GB	~6–12 hr	~$20–60
1. LoRA SFT (2–3 ep)	LoRA r=64, bf16 base	2× H100 80GB (or 8× FSDP)	2–4 days on 2× (½ day on 8×)	~$200–500
2. DPO	LoRA-DPO	1–2× H100	~6–12 hr	~$30–80
3. RL (CyberGym reward)	GRPO + Docker rollouts	4–8× H100 + Docker host	days–weeks	$1–5K+ (rollout-dominated)
CyberGym eval	vLLM + OpenHands	1× H100 + Docker	~1–2 days/sample	~$50–200

VRAM: LoRA bf16 base 55.6 GB + adapters + long-seq activations → 1× 80GB (tight, seq≤4096) or 2× 80GB (comfortable, seq 8192). Full FT (if ever) ~450–520 GB → 8× 80GB ZeRO-3.

Budget: Stage 1 trial ~$300–700 all-in (data prep + pilot + run + eval iterations). Add $1–5K+ only if you commit to Stage-3 RL. Storage ~370 GB.

---

7. Loss Curve & Monitoring

• SFT loss: starts ~1.2–1.6 (post-warmup), converges ~0.6–0.9. Below ~0.4 = memorization. Agentic-trajectory tokens may sit a bit higher.
• Healthy: sharp early drop → smooth decline → flatten; val tracks train with small stable gap.
• Overfitting: val plateaus/rises while train falls; sawtooth at epoch boundaries = memorizing → stop at 2–3 epochs.
• Track (W&B): train/val loss, grad norm, LR, tokens/s, per-epoch eval (vuln-detection + IFEval + CyberGym-10). For RL: reward curve, KL to ref, pass-rate on held-in tasks.
• Selection rule: best eval score + non-rising val loss, never lowest train loss (guards against agentic/tool-call regression).

---

8. Risks & Mitigations

Risk	Likelihood	Mitigation
SFT barely moves CyberGym (it's agentic/RL-shaped)	High	Set expectations; Stage-1 win = vuln-detection/knowledge + agentic coherence; Stage-3 RL for the CyberGym jump
Framework can't train the hybrid arch	High	Phase-0 go/no-go; fall back to Qwen3-32B
SFT regresses tool-calling/reasoning	Medium	LoRA (low forgetting); uniform <think>; blend agentic trajectories; per-epoch guardrail evals
Missing-license Tier-1 data	Medium	Record provenance; prefer licensed equivalents; academic-use framing; legal review before any release
Benchmark contamination	Medium	13-gram + fuzzy + embedding decontam; frozen test splits
RL harness cost/complexity	Medium	Gate Stage 3 on Stage-1 success; start with a small task subset
<think> inconsistency damages reasoning	Medium	Single uniform policy; use pre-built reasoning sets + rejection-sampled traces

---

9. Open Decisions (most now resolved)

✅ Primary = CyberGym · ✅ offensive+defensive · ✅ LoRA→merge · ✅ <think> required. Remaining:

1. Commit to Stage-3 RL now (changes budget to $1–5K+), or decide after Stage 1? (Recommend: decide after Stage 1.)
2. Base model: lock Qwen3.6-27B, pre-authorize Qwen3-32B fallback if Phase-0 blocks? (Recommend: yes.)
3. Reasoning-trace synthesis: budget for rejection-sampling <think> traces over code-vuln sets (needs a strong teacher model + inference $)? (Recommend: yes — it's core to the <think> goal.)

---

10. Phased Milestones

Phase	Output	Gate
0 Infra & go/no-go	Qwen3.6-27B trains+serves (or fallback)	Architecture trainable?
1 Baseline	Base scores on CyberGym + code-vuln + guardrails	Reproducible; must finish before QLoRA/LoRA training
2 Data engineering	Normalized+deduped+decontam mix w/ <think> traces + data card	Decontam verified
3 Pilot (QLoRA)	Loss curve + sane gens + 1 eval cycle	No tool-call regression
4 LoRA SFT + per-epoch eval	Checkpoints + curves + eval table	Hit Stage-1 criteria
5 Merge + benchmark	Merged checkpoint, CyberGym level1 sample vs base	Net improvement, no regression
6 (Opt) DPO	Sharpened secure/insecure model	Cheap win
7 (Opt) RL on CyberGym	v2 with real CyberGym jump	Cost/benefit justified

---

11. Proposed Repo Structure

infosec/
├── docs/PROJECT_GOAL.md
├── data/{download,processed,decontam,think_synthesis}/   # fetch, normalize, decontaminate, reasoning-trace synthesis
├── training/{recipes,configs,scripts,tests}/             # Vast.ai + Qwen recipe, data manifest/adapters, offline tests
├── eval/{cybergym,vuln_detection,knowledge,judge}/
├── serve/                                                # vLLM OpenAI-compatible + tool-calling
└── reports/                                              # baseline, per-epoch, merged, RL

12. Current Repo Artifacts

The implementation layer lives under training/:

• training/README.md — entrypoint and quick-start commands.
• training/recipes/pretraining_cybergym_baseline.md — mandatory base-model CyberGym benchmark before training.
• training/recipes/stage1_qwen36_27b_lora_sft.md — Stage 0 load gate, QLoRA pilot, and main LoRA SFT recipe.
• training/configs/*.yaml — pilot, Stage 1, and data-mix contracts.
• training/scripts/vastai_*.sh — Vast.ai search/create/bootstrap scripts.
• training/scripts/phase0_qwen_smoke.py — config/tokenizer/full-load smoke gate for Qwen/Qwen3.6-27B.
• training/scripts/run_sft.py and training/scripts/merge_lora.py — runnable PEFT training and merge entrypoints.

Data-engineering layer (implemented + offline-tested, 2026-06-27):

• training/configs/datasets.yaml — per-source download + adapter manifest; all schemas verified against the live HF datasets-server.
• training/scripts/hf_download.py — schema probe (stdlib) + split download (datasets), with gated-source detection.
• training/scripts/sft_adapters.py — per-source schema adapters; notably detection_func_target for PrimeVul/DiverseVul (raw func+target, which the generic normalizer silently dropped) and vuln_fix_pair for CrossVul.
• training/scripts/build_sft_dataset.py — adapt → route <think> (ready vs synthesize) → dedup → source caps → data card.
• training/scripts/synthesize_think.py — rejection-sampling <think> backfill (STaR-style) against a teacher endpoint, with an offline --mock.
• training/scripts/normalize_sft_jsonl.py / split_jsonl.py — ad-hoc converter + deterministic split.
• training/scripts/decontaminate.py — 13-gram + fuzzy (optional embedding) contamination gate; emits the decontam report the Stage-1 gate now requires.
• training/scripts/build_eval_sets.py — held-out C/C++ vuln-detection set (from reserved test splits) + disjoint knowledge-MCQ set.
• training/scripts/eval_endpoint.py — reusable scorer for any OpenAI-compatible endpoint; same command for the base baseline and every checkpoint (vuln-detection accuracy/precision/recall + MCQ accuracy).
• training/recipes/data_preparation.md — the full manifest-driven workflow.
• training/tests/test_data_pipeline.py — offline end-to-end test (54 checks: adapters → build → synthesize → re-build → split → eval-sets → endpoint-eval → decontaminate).

Data findings to fold into §3: (a) MegaVul/PrimeVul/DiverseVul ship test splits → reserve for the held-out vuln-detection eval, never train on them; (b) the two Tier-2 reasoning sets SecCoderX/...Cold_Start and Leopo1d/OpenVul... return HTTP 401 (gated) anonymously — they need huggingface-cli login + access approval before use, so the <think>-core relies more on synthesized traces until then. Still missing: decontamination against frozen eval splits, and a real teacher endpoint for synthesis.

Appendix — Source Pointers

• Model: huggingface.co/Qwen/Qwen3.6-27B · Benchmark: github.com/sunblaze-ucb/cybergym, arXiv 2506.02548, HF sunblaze-ucb/cybergym
• Datasets: see §3 (HF ids). Methods: QLoRA (2305.14314), "LoRA Learns Less, Forgets Less" (2405.09673), GRPO/DPO refs.