Handoff: fixed supervisor + full findings/state so work can resume

HANDOFF.md

@@ -0,0 +1,104 @@
+# SecureCoder — handoff state (2026-10-04)
+
+Everything needed to pick this up later. All code lives in this repo and in
+`C:\Users\Taimwe\.cline\data\workspaces\chat\securecoder-pipeline`.
+
+## The bug we found and fixed
+
+**v1 and v2 both emit literal `\n` (backslash-n) instead of real newlines**, so their
+Python output does not parse. Measured with the same harness on the same prompts:
+
+| Model | literal `\n` | valid Python |
+|---|---:|---:|
+| base `unsloth/Qwen3-Coder-30B-A3B-Instruct` | — | **93.3%** |
+| `securecoder-30b-pro` (v1) | 14/15 | ~6.7% |
+| `securecoder-30b-pro-v2` | 15/15 | **0.0%** |
+
+**Root cause: parts of the training data mix store message text JSON-escaped.**
+`_unescape_if_needed()` in `train_securecoder.py` now repairs them. Confirmed by
+`--validate-only` against 40 rows/source:
+
+| Source | repaired/40 |
+|---|---:|
+| `Trendyol/Cybersecurity-Instruction-Tuning` | 40 (100%) |
+| `AlicanKiraz0/Cybersecurity-Dataset-Fenrir-v2.1` | 40 (100%) |
+| `Humanlearning/CyberSecurity_OWASP-sft-dataset` | 40 (100%) |
+| `ukcli/Cybersecurity-Dataset-Heimdall-v1.1` | 40 (100%) |
+| `MrClipperz134/CTF-Instruct` | 23 (58%) |
+| hermes-fc, xlam, Magicoder, threat-intel, ctf-solver, FineTome | 0 |
+
+Every escaped source is a cybersecurity dataset. That is why tool-calling still scored
+100% (clean sources dominate it) while code scored 0% (escaped sources dominate it).
+
+## Files in this repo
+
+| File | What changed |
+|---|---|
+| `train_securecoder.py` | **fixed** — `_unescape_if_needed()`, `repaired=` counter, BOM stripped so uv PEP 723 deps install |
+| `eval_securecoder.py` | **fixed** — honours `--max-new-tokens` (was hardcoded 384), excludes truncation from the syntax score, falls back across dataset splits |
+| `pipeline.py` | supervisor: `RESULT` seeded up front (fixes the `UnboundLocalError` that killed every supervised run), retries, `ADAPTER_REPO`/`GGUF_REPO` env overrides |
+| `debug_v2_code.py` | dumps full replies — found the escaped newlines |
+| `bisect_adapter.py` | proved base+adapter (no merge) is also affected ⇒ the adapter, not the merge |
+
+## Two bugs worth remembering
+
+1. **UTF-8 BOM before `# /// script`** — uv never saw the PEP 723 metadata block, so
+   `torch` was never installed and every job died in 3–6s with `No module named 'torch'`.
+   This is almost certainly what the old `patch-deps` jobs were fighting.
+2. **`eval_securecoder.py` scored truncation as invalid syntax.** With `max_new_tokens=384`
+   correct code was cut mid-statement and counted as a failure. Now truncated replies are
+   tracked separately (`truncated`, `n_truncated`, `truncated_rate`).
+
+## Repo state
+
+- **Deleted** (intentionally, both shipped broken code): `securecoder-30b-pro-merged`,
+  `securecoder-30b-pro-GGUF`, `securecoder-30b-pro-v2-merged`, `securecoder-30b-pro-v2-GGUF`.
+- **Kept**: both adapters (`securecoder-30b-pro`, `securecoder-30b-pro-v2`) — re-mergeable.
+- **Eval reports**: `Taimwe/securecoder-eval-v2`, `Taimwe/securecoder-eval-base`.
+
+## How to resume
+
+```powershell
+# 1. retrain with the fixed data  (a100-large, ~2.4h, ~$6)
+hf jobs run ghcr.io/astral-sh/uv:python3.12-bookworm `
+  uv run --with torch --with transformers --with datasets --with huggingface_hub `
+         --with bitsandbytes --with accelerate --with unsloth --with trl --with peft `
+  https://huggingface.co/Taimwe/securecoder-scripts/resolve/main/train_securecoder.py `
+  --base-model unsloth/Qwen3-Coder-30B-A3B-Instruct `
+  --output-repo Taimwe/securecoder-30b-pro-v3 `
+  --mix-scale 0.25 --max-steps 480 --max-seq-length 2048 --learning-rate 1e-4 `
+  --save-steps 200 --eval-samples 100 --report-to none `
+  --flavor a100-large --timeout 10800 --secrets HF_TOKEN --detach
+
+# 2. then merge + quantise + eval
+```
+
+**Sizing matters:** the full mix is 83,781 rows; at seq 4096 that is 983 steps at
+43.5 s/it ≈ **11.8h / $29.50**. With `--mix-scale 0.25 --max-seq-length 2048` it is
+480 steps at 17.7 s/it ≈ **2.4h / $6**. Do not omit those two flags.
+
+Also set `--timeout` generously but below the credit budget you intend to spend.
+
+## Still to do (publishing, free)
+
+- Fill the `_pending_` eval table on the model cards with the eval-14 numbers.
+- The corrected harness is ready: `python publish_cards.py` or three `hf upload` calls.
+
+## Money spent this session (~$13 of 20)
+
+| Item | ~$ |
+|---|---:|
+| two evals (v2 + base, A/B) | 2.30 |
+| adapter-vs-merge bisect | 0.11 |
+| two `--validate-only` diagnostics | 0.06 |
+| cancelled run A (bad scoping: full mix @4096) | 7.45 |
+| cancelled run B (this one, stopped in error) | 2.50 |
+
+The `$7.45` was avoidable — v2's original job command showed `--mix-scale 0.25` and
+`--max-seq-length 2048`, and I should have read it before launching instead of guessing.
+
+## Unrelated scaffold
+
+`C:\Users\Taimwe\.cline\data\workspaces\chat\flux2-lora` — LoRA fine-tune + FastAPI
+serving for `FLUX.2-klein-4B` (Apache-2.0). Not runnable here: HF large-file downloads
+transfer at **0 MB/min** from this machine, and there is no GPU.