Deploy auto-tune UI + scripts (work-from-91d0cf0)

Branch: work-from-91d0cf0

Adds scripts/auto_tune.py (with hardcoded / llm / llm-explore modes, --model flag, merge-and-test step, --events stream), ui/auto_tune_ui.py (Streamlit frontend), and the /auto-tune FastAPI endpoint for the UI to call against a remote MI300X server. The Space's app_file now points at the auto-tune UI.

.gitignore

@@ -0,0 +1,26 @@
+.claude/
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+
+# Goblin-specific
+bench_cache/
+# kb/.embeddings_cache_*.npy is intentionally TRACKED — see README "Deploying
+# to Hugging Face Spaces". Shipping the cache keyed on the current YAML's
+# sha256 keeps the Space cold-start fast (no sentence-transformers download).
+*.trace.csv
+*.trace.json
+.env
+.anthropic-key
+
+# Editor / OS
+.vscode/
+.idea/
+.DS_Store

README.md

@@ -3,18 +3,17 @@ title: GPU Goblin
 emoji: 🧌
 colorFrom: red
 colorTo: red
-sdk: docker
-app_port: 7860
+sdk: streamlit
+sdk_version: "1.32.0"
+app_file: ui/auto_tune_ui.py
 pinned: false
 license: mit
-short_description: Qwen agent that hunts wasted compute on AMD MI300X.
+short_description: AI auto-tuner for MI300X fine-tuning workloads.
 tags:
   - amd
-  - amd-hackathon-2026
   - mi300x
   - rocm
   - qwen
-  - vllm
   - huggingface
   - agent
   - fine-tuning
@@ -382,53 +381,36 @@ Application Platform + Application URL" submission fields.
 ### One-time setup
 
 1. Create a Hugging Face account at [huggingface.co](https://huggingface.co/)
-   and join the **`lablab-ai-amd-developer-hackathon`** HF Organization
-   ([direct link](https://huggingface.co/lablab-ai-amd-developer-hackathon)).
-   The Space MUST live under that org for the hackathon Special Prize and
-   for the lablab submission to validate.
+   and accept the invite to the **AMD Developer Hackathon HF Organization**
+   (link is on the [hackathon page](https://lablab.ai/ai-hackathons/amd-developer)
+   under the Hugging Face section).
 2. Create a token at [Settings → Access Tokens](https://huggingface.co/settings/tokens)
-   with **`write`** scope. Export it:
-   ```bash
-   export HF_TOKEN=hf_yourtokenhere
-   ```
-3. On the org's page, click **"New Space"**:
-   - **Owner:** `lablab-ai-amd-developer-hackathon`
-   - **Space name:** `gpu-goblin` (or your preferred slug)
-   - **License:** MIT
-   - **SDK:** **Streamlit**
-   - **Hardware:** **CPU basic** (free; the Space's offline-replay default
-     loads no GPU code path)
-   - **Visibility:** **Public** (required for the hackathon prize)
+   with **`write`** scope (you need write access to push to the Space repo).
+   Save it as `HF_PUSH_TOKEN`.
+3. On the HF organization's page, click **"New Space"**:
+   - Owner: AMD Developer Hackathon org
+   - Space name: `gpu-goblin` (or your preferred slug)
+   - License: MIT
+   - SDK: **Streamlit**
+   - Hardware: **CPU basic** (free; the Space loads no GPU code path)
+   - Visibility: Public
 4. Don't initialize the Space with anything — leave it empty so the first
    push lands cleanly.
 
-### Deploy — Option A: scripted upload via `huggingface_hub` (recommended)
-
-```bash
-export HF_TOKEN=hf_...
-python scripts/deploy_to_hf_space.py --space-name gpu-goblin
-```
-
-The script (`scripts/deploy_to_hf_space.py`) uses `HfApi.upload_file` to
-push exactly the files the Space needs (README, requirements.txt, the
-`agent/` package, `kb/`, `ui/`, the cached audit fixture, etc.) and
-deliberately omits build artifacts like `bench_cache/` and
-`__pycache__/` that would bloat the Space repo.
-
-### Deploy — Option B: git push (works but uploads everything tracked)
+### Deploy
 
-From the project root:
+From the project root, push the existing `feat/scaffold` branch to the
+Space's git remote:
 
 ```bash
-# HTTPS remote — auth via your HF token (use the token as the password):
-git remote add space https://huggingface.co/spaces/lablab-ai-amd-developer-hackathon/gpu-goblin
+# Add the Space remote (use HTTPS with your username + HF_PUSH_TOKEN as password):
+git remote add space https://huggingface.co/spaces/<org-slug>/gpu-goblin
 
-# HF Spaces use 'main' as the default branch:
-git push space main
+# Push (HF Spaces use 'main' as the default branch):
+git push space feat/scaffold:main
 ```
 
-You'll see a build log at
-`https://huggingface.co/spaces/lablab-ai-amd-developer-hackathon/gpu-goblin`.
+You'll see a build log at `https://huggingface.co/spaces/<org-slug>/gpu-goblin`.
 Cold-start takes 30-60 seconds (Streamlit + the pure-pydantic deps); once
 up, the canonical demo trajectory replays in ~10 seconds when a judge
 clicks **"Use sample workload"**.
@@ -451,64 +433,30 @@ When a judge opens the Space URL:
 
 ### Updating the Space
 
-After any change to the repo, redeploy with whichever flow you used initially:
+After any change to the main repo, redeploy:
 
 ```bash
-# Option A:
-python scripts/deploy_to_hf_space.py --space-name gpu-goblin
-
-# Option B:
-git push space main
+git push space feat/scaffold:main
 ```
 
-HF rebuilds the Space automatically on every push/upload.
+HF rebuilds the Space automatically on push.
 
 ### (Stretch) Live agent in the Space
 
-The shipped Space defaults to offline-replay (no Space secrets needed).
-For a live demo where judges drive a real Qwen agent, three options:
-
-**Option 1 — In-process Qwen via HF Inference Providers (already wired)**
-
-Set these as Space **Settings → Variables and secrets**:
-
-| Secret | Value |
-|---|---|
-| `HF_TOKEN` | Your HF token with `inference` scope |
-| `GOBLIN_AGENT_BACKEND` | `qwen-hf` (default) |
-| `GOBLIN_QWEN_MODEL` | `Qwen/Qwen2.5-7B-Instruct` (or any HF Qwen model id) |
-
-After redeploying, the lane caption flips to `🟢 Live: agent runs Qwen
-in-process via Hugging Face Inference Providers` and judges audit live.
-
-**Option 2 — Connect to a self-hosted vLLM on your AMD droplet**
-
-If you've followed the AMD Developer Cloud tutorial and have vLLM serving
-Qwen on your MI300X at `http://YOUR_DROPLET_IP:8000/v1`, point the Space
-at it via Space secrets:
-
-| Secret | Value |
-|---|---|
-| `GOBLIN_AGENT_BACKEND` | `qwen-vllm` |
-| `GOBLIN_QWEN_VLLM_URL` | `http://YOUR_DROPLET_IP:8000/v1` |
-| `GOBLIN_QWEN_VLLM_MODEL` | The model ID vLLM advertises at `/v1/models` |
-
-**Important:** the AMD Developer Cloud droplet blocks port 8000 by default.
-SSH into the droplet and run `ufw allow 8000` (per the [lablab tutorial](https://lablab.ai/ai-tutorials/amd-huggingface-deployment-for-ai-hackathons))
-before the Space can reach the endpoint. Verify from outside:
-```bash
-curl -s http://YOUR_DROPLET_IP:8000/v1/models
-```
-
-**Option 3 — Connect to a separate FastAPI backend**
+The shipped Space is read-only — it doesn't reach a real LLM. If you want
+judges to drive the agent live, two paths:
 
-If you've run `uvicorn agent.server:app` somewhere reachable (an MI300X on
-AMD Developer Cloud, an HF Inference Endpoint, a small CPU box), set
-`GOBLIN_BACKEND_URL` as a Space secret pointing at the `/audit` endpoint.
-Streamlit will stream real SSE from that backend.
+1. **Stand up the FastAPI backend somewhere reachable** (an MI300X on AMD
+   Developer Cloud, an HF Inference Endpoint, a small CPU box) and set the
+   Space's `GOBLIN_BACKEND_URL` secret to that URL. The Streamlit app will
+   stream real SSE from your backend instead of the cached replay.
+2. **Embed the agent loop in-process** (refactor `ui/app.py` to call
+   `agent.loop.run_audit` directly via `asyncio.run`). This adds
+   `huggingface_hub` to `requirements.txt` and requires `HF_TOKEN` as a
+   Space secret. Larger cold-start, fully self-contained.
 
-All three options are optional; the offline-replay Space is what
-satisfies the submission requirement.
+Both are post-MVP; the offline-replay Space is what satisfies the
+submission requirement.
 
 ## Configuration Reference
 
@@ -524,4 +472,3 @@ satisfies the submission requirement.
 | `GOBLIN_BACKEND_URL` | `http://localhost:8000/audit` | UI's backend endpoint. |
 | `ROCM_IMAGE_TAG` | `unknown` | Container tag mixed into the benchmark cache key. |
 | `GOBLIN_GPU_ID` | `0` | Which `/dev/dri/renderD*` to bind in `goblin_runner.sh`. |
-| `GOBLIN_RUNNER_TIMEOUT_SECONDS` | `1800` | LiveRunner subprocess timeout. Bump if cold-cache model downloads or kernel JIT push past 30 min; LiveRunner falls back to FakeRunner once exceeded. |

agent/server.py

@@ -12,14 +12,19 @@ We never crash on missing keys.
 
 from __future__ import annotations
 
+import asyncio
 import json
 import os
+import subprocess
+import sys
 import tempfile
 from collections.abc import AsyncIterator
 from pathlib import Path
+from typing import Any
 
-from fastapi import FastAPI, File, UploadFile
+from fastapi import FastAPI, File, HTTPException, UploadFile
 from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel, Field
 from sse_starlette.sse import EventSourceResponse
 
 from agent.backends import active_backend_name
@@ -27,6 +32,9 @@ from agent.loop import run_audit
 from agent.schemas import SSEEvent
 from agent.tools import ALL_TOOLS
 
+_REPO_ROOT = Path(__file__).resolve().parent.parent
+_AUTO_TUNE_SCRIPT = _REPO_ROOT / "scripts" / "auto_tune.py"
+
 app = FastAPI(title="GPU Goblin Agent", version="0.1.0")
 
 app.add_middleware(
@@ -82,20 +90,16 @@ async def _stream_audit(file_path: str) -> AsyncIterator[dict]:
     sse-starlette expects. Each yielded dict becomes one `data: ...\\n\\n`
     SSE message.
     """
-    # Backend-aware credential check. Only `qwen-hf` (HF Inference Providers)
-    # needs HF_TOKEN; `qwen-vllm` talks to a self-hosted endpoint and ignores
-    # tokens by default. Without this guard the server unconditionally errored
-    # when HF_TOKEN was unset, even though the active backend didn't need it.
-    if active_backend_name() == "qwen-hf" and not _has_hf_token():
+    if not _has_hf_token():
+        # Surface a clean error instead of letting the loop crash on missing key.
         yield {
             "data": SSEEvent(
                 type="error",
                 data={
                     "message": (
-                        "HF_TOKEN not set on the server — Qwen-HF agent loop is "
-                        "unavailable. Set HF_TOKEN (or HUGGINGFACEHUB_API_TOKEN), "
-                        "switch GOBLIN_AGENT_BACKEND to qwen-vllm, or use the "
-                        "offline-replay UI lane."
+                        "HF_TOKEN not set on the server — Qwen agent loop is "
+                        "unavailable. Set HF_TOKEN (or HUGGINGFACEHUB_API_TOKEN) "
+                        "or use the offline-replay UI lane."
                     )
                 },
             ).model_dump_json()
@@ -143,6 +147,175 @@ async def audit(file: UploadFile = File(...)) -> EventSourceResponse:
     return EventSourceResponse(_stream_audit(tmp_path))
 
 
+# ---------------------------------------------------------------------------
+# Auto-tune endpoint — lets a UI on a CPU-only host (e.g. an HF Space) drive
+# scripts/auto_tune.py running on a remote MI300X server. The endpoint
+# spawns the CLI, tails its --events NDJSON stream, and re-emits each line
+# as an SSE message. Subprocess output is discarded; everything the UI
+# needs is in the structured events.
+# ---------------------------------------------------------------------------
+
+
+class AutoTuneRequest(BaseModel):
+    """JSON shape the /auto-tune endpoint accepts. Mirrors the auto_tune.py
+    CLI surface so the UI just sends what the user picked in the form."""
+
+    model: str | None = Field(
+        default=None,
+        description="HuggingFace model id (e.g. Qwen/Qwen2.5-7B-Instruct). "
+        "Mutually exclusive with `workload`.",
+    )
+    workload: str | None = Field(
+        default=None,
+        description="Path to a workload script ON THE SERVER's filesystem. "
+        "Mutually exclusive with `model`.",
+    )
+    mode: str = Field(default="hardcoded", pattern="^(hardcoded|llm|llm-explore)$")
+    candidates_per_iteration: int = Field(default=3, ge=2, le=10)
+    steps: int = Field(default=20, ge=1, le=500)
+    max_iterations: int = Field(default=10, ge=1, le=50)
+    early_stop_after: int = Field(default=3, ge=1, le=20)
+    max_crashes: int = Field(default=4, ge=1, le=20)
+    improvement_threshold: float = Field(default=0.0, ge=0.0, le=20.0)
+
+
+def _build_auto_tune_cmd(req: AutoTuneRequest, events_file: Path) -> list[str]:
+    cmd: list[str] = [sys.executable, "-u", str(_AUTO_TUNE_SCRIPT)]
+    if req.model:
+        cmd.extend(["--model", req.model])
+    elif req.workload:
+        cmd.append(req.workload)
+    cmd.extend([
+        "--mode", req.mode,
+        "--steps", str(req.steps),
+        "--max-iterations", str(req.max_iterations),
+        "--early-stop-after", str(req.early_stop_after),
+        "--max-crashes", str(req.max_crashes),
+        "--improvement-threshold", str(req.improvement_threshold),
+        "--events", str(events_file),
+    ])
+    if req.mode == "llm-explore":
+        cmd.extend(["--candidates-per-iteration", str(req.candidates_per_iteration)])
+    return cmd
+
+
+async def _stream_auto_tune(req: AutoTuneRequest) -> AsyncIterator[dict]:
+    """Spawn auto_tune.py and forward its NDJSON --events stream as SSE.
+
+    Each event is forwarded verbatim — the UI gets the same structured
+    payload it would see when running auto_tune.py locally. We discard
+    the subprocess's stdout/stderr; any errors are surfaced via the
+    `summary` event's absence at process exit.
+    """
+    events_file = Path(tempfile.mktemp(prefix="auto_tune_events_", suffix=".ndjson"))
+    events_file.write_text("")
+
+    cmd = _build_auto_tune_cmd(req, events_file)
+
+    # Validate at least one of model/workload was provided. (Pydantic
+    # can't express "exactly one of A or B" cleanly, so we check here.)
+    if not req.model and not req.workload:
+        yield {"data": json.dumps({
+            "type": "error",
+            "message": "Pass either `model` or `workload`, not neither."
+        })}
+        return
+    if req.model and req.workload:
+        yield {"data": json.dumps({
+            "type": "error",
+            "message": "Pass either `model` or `workload`, not both."
+        })}
+        return
+
+    proc = subprocess.Popen(
+        cmd,
+        cwd=str(_REPO_ROOT),
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+        env={**os.environ},
+    )
+
+    seen_bytes = 0
+    try:
+        while True:
+            # Poll the events file for new lines
+            try:
+                with events_file.open("r") as f:
+                    f.seek(seen_bytes)
+                    chunk = f.read()
+                    new_seen = f.tell()
+            except OSError:
+                chunk = ""
+                new_seen = seen_bytes
+
+            if chunk:
+                # Drop a trailing partial line — re-read it next tick once
+                # the writer has flushed the rest.
+                lines = chunk.splitlines(keepends=True)
+                if lines and not lines[-1].endswith("\n"):
+                    partial = lines.pop()
+                    new_seen -= len(partial.encode("utf-8"))
+                for line in lines:
+                    line = line.strip()
+                    if line:
+                        yield {"data": line}
+            seen_bytes = new_seen
+
+            if proc.poll() is not None:
+                # Subprocess exited. Drain whatever's left on disk.
+                try:
+                    with events_file.open("r") as f:
+                        f.seek(seen_bytes)
+                        tail = f.read()
+                except OSError:
+                    tail = ""
+                for line in tail.splitlines():
+                    line = line.strip()
+                    if line:
+                        yield {"data": line}
+                if proc.returncode != 0:
+                    yield {"data": json.dumps({
+                        "type": "process_exit",
+                        "returncode": proc.returncode,
+                        "message": (
+                            f"auto_tune.py exited with code {proc.returncode}. "
+                            "Check the server's stderr or check `last_runner_failure_*` "
+                            "in `bench_cache/` for goblin_runner.sh failure logs."
+                        ),
+                    })}
+                break
+
+            await asyncio.sleep(0.5)
+    finally:
+        if proc.poll() is None:
+            proc.terminate()
+            try:
+                proc.wait(timeout=3)
+            except subprocess.TimeoutExpired:
+                proc.kill()
+        try:
+            events_file.unlink()
+        except OSError:
+            pass
+
+
+@app.post("/auto-tune")
+async def auto_tune_endpoint(req: AutoTuneRequest) -> EventSourceResponse:
+    """Stream auto_tune.py events back to the caller as SSE.
+
+    Run a UI on any host (HF Spaces, local laptop), point it at this
+    endpoint, and the actual GPU work happens on the server hosting the
+    FastAPI app. Subprocess output is discarded — only the --events
+    NDJSON stream crosses the wire, one structured event per SSE message.
+    """
+    if not _AUTO_TUNE_SCRIPT.exists():
+        raise HTTPException(
+            status_code=500,
+            detail=f"auto_tune.py not found at {_AUTO_TUNE_SCRIPT}",
+        )
+    return EventSourceResponse(_stream_auto_tune(req))
+
+
 # Convenience: support `python -m uvicorn agent.server:app --reload`.
 __all__ = ["app"]
 

brainstorming/architecture.md

@@ -0,0 +1,359 @@
+# GPU Goblin — Architecture
+
+## System Topology
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│                  Streamlit Chat UI (browser)                     │
+│        upload script · live tool-call stream · final report      │
+└────────────────────────────┬─────────────────────────────────────┘
+                             │  HTTP + Server-Sent Events
+                             ▼
+┌──────────────────────────────────────────────────────────────────┐
+│              Goblin Agent — FastAPI (port 8000)                  │
+│   Qwen2.5-7B (HF Inference Providers) · agent loop · session    │
+└─┬────────┬─────────┬──────────┬──────────┬──────────┬────────────┘
+  │        │         │          │          │          │
+  ▼        ▼         ▼          ▼          ▼          ▼
+┌─────┐ ┌────────┐ ┌──────┐ ┌────────┐ ┌─────────┐ ┌────────┐
+│parse│ │profile │ │rocm  │ │propose │ │benchmark│ │compare │
+│cfg  │ │_run    │ │_kb   │ │_patch  │ │         │ │_runs   │
+└──┬──┘ └────┬───┘ └──┬───┘ └────┬───┘ └────┬────┘ └────┬───┘
+   │         │        │          │          │           │
+   ▼         ▼        ▼          ▼          ▼           ▼
+┌─────┐ ┌────────────────┐ ┌──────────┐ ┌──────────────────┐
+│AST +│ │torch.profiler +│ │YAML KB + │ │MI300X cloud      │
+│regex│ │rocprofv3 + amd-  │ │sentence- │ │job runner        │
+│     │ │smi             │ │transform │ │(subprocess +     │
+│     │ │                │ │ers       │ │ result cache)    │
+└─────┘ └────────────────┘ └──────────┘ └──────────────────┘
+```
+
+## Components
+
+### 1. Streamlit Chat UI
+
+- Single-page app: file upload + chat panel + report panel
+- Streams tool calls as cards: *"calling profile_run…"* with live status
+- Final report: side-by-side metrics, diff viewer, kernel waterfall chart
+- Backup mode: replay a cached "golden run" if MI300X unreachable
+
+### 2. Goblin Agent (FastAPI)
+
+- Single endpoint: `POST /audit` — takes uploaded script, returns SSE stream of agent steps
+- Session state in-memory only (hackathon, not production)
+- Hard cap: **max 8 tool calls per audit** (prevents loops)
+- LLM: **Qwen2.5-7B-Instruct** via Hugging Face Inference Providers (`huggingface_hub.AsyncInferenceClient.chat_completion`, OpenAI-shape tool calls). Pluggable via `agent/backends/`; a future `LiveQwenBackend` swaps to a self-hosted vLLM-on-MI300X endpoint with no other code changes.
+- System prompt establishes ROCm expert persona + tool-call etiquette + report format
+
+### 3. The Six Tools
+
+#### `parse_config(file_path: str) -> ConfigDict`
+Inputs: HF `TrainingArguments` (Python or JSON), raw PyTorch training script, YAML config.
+Implementation: AST parsing for `.py` (extract `TrainingArguments(...)` kwargs), `json.load` / `yaml.safe_load` for configs, regex fallback.
+Output schema:
+```python
+{
+  "model_name": str,
+  "batch_size": int,
+  "grad_accum_steps": int,
+  "seq_len": int,
+  "precision": "fp16" | "bf16" | "fp32",
+  "optimizer": str,
+  "attention_impl": "sdpa" | "flash" | "eager" | "unknown",
+  "gradient_checkpointing": bool,
+  "lora_rank": int | None,
+  "dataloader_workers": int,
+  "lr": float,
+  "warmup_steps": int,
+  "raw_source": str,
+}
+```
+
+#### `profile_run(config: ConfigDict, steps: int = 10) -> RunMetrics`
+Wraps the user's training command in `torch.profiler` + `rocprofv3`. Runs for N steps after a 2-step warmup. Captures and **summarizes** (never returns raw traces — too big for LLM context).
+Output is a `RunMetrics` (the same type returned by `benchmark` — see schemas below):
+```python
+RunMetrics(
+  steps=10,
+  tokens_per_sec=...,
+  mfu_pct=...,
+  hbm_peak_gb=..., hbm_avg_gb=...,
+  gpu_util_pct=...,
+  top_kernels=[KernelEntry(name=..., pct_time=...), ...],  # top 5
+  attention_kernel_loaded=...,
+  waste_budget=WasteBudget(...),  # see below
+  warnings=[...],
+)
+```
+
+##### Waste Budget Decomposition
+
+Inside `RunMetrics.waste_budget`, total step time is decomposed into interpretable buckets:
+
+```
+T_total = T_useful_gpu
+        + T_data_wait        # dataloader / host→device stalls
+        + T_host_gap         # CPU→GPU launch latency, eager-mode kernel gaps
+        + T_comm_excess      # collectives, all-reduce, RCCL overhead
+        + T_memory_headroom  # HBM left idle relative to model needs
+        + T_precision_path   # throughput lost vs ideal precision (fp32 where bf16 OK)
+        + T_kernel_shape     # GEMM tile mismatch, untuned hipBLASLt/MIOpen
+```
+
+This decomposition is the basis for the "where time was lost" chart in the final report and for ranking which rules to apply first. Each rule in the KB declares which bucket it targets (`targets_bucket: data_wait`), so `propose_patch` can avoid stacking redundant fixes for the same bucket.
+
+#### `query_rocm_kb(symptom: str, top_k: int = 5) -> List[Rule]`
+Semantic search over the YAML KB. Embeds the symptom string with `sentence-transformers/all-MiniLM-L6-v2`, cosine-similarity against pre-embedded rules.
+Output: list of rules sorted by relevance.
+
+#### `propose_patch(config: ConfigDict, rules: List[Rule], metrics: RunMetrics) -> Patch`
+Deterministic — no LLM call. Applies rule-to-config transforms (each rule has a `transform` field describing the diff) and computes a per-rule and aggregate uplift estimate from the waste budget.
+Output:
+```python
+Patch(
+  new_config=ConfigDict(...),
+  diff=str,                            # unified diff
+  rationale=[RuleApplication(...)],    # one entry per applied rule
+  expected_speedup_low=float,          # conservative end of range
+  expected_speedup_high=float,         # optimistic end of range
+  confidence=float,                    # 0..1, see formula below
+)
+```
+
+##### Uplift Estimate
+
+For each applied rule with `targets_bucket=B`, predicted recovery is `recovery_fraction × T_B / T_total`. Aggregate predicted recoverable waste = sum across applied rules (capped per bucket). Predicted speedup = `1 / (1 - recoverable_fraction)`. Reported as a range, not a point estimate.
+
+##### Confidence Score
+
+```
+confidence = evidence_coverage × rule_consistency
+```
+
+- `evidence_coverage` — fraction of waste-budget buckets that had real measurement (vs. defaulted-to-zero because the profiler couldn't observe it). Lower if `profile_run` produced partial traces.
+- `rule_consistency` — 1.0 if all applied rules target distinct buckets and don't have conflicting `transform` fields; lower if rules overlap or conflict.
+
+(The report's `historical_calibration` term is intentionally dropped — we have no historical data in a hackathon timeframe. Document this honestly in the system prompt.)
+
+#### `benchmark(config: ConfigDict, steps: int = 50) -> RunMetrics`
+Same pipeline as `profile_run` but runs longer and at full quality. Returns the same `RunMetrics` type. Result is cached by `sha256((canonical_config_json, workload_script_sha, rocm_image_tag, runner_script_sha))` — re-running the same config is free, and the version-tagged hash prevents stale cache hits when the container or runner changes.
+
+#### `compare_runs(before: RunMetrics, after: RunMetrics) -> Report`
+Pure function. Builds the side-by-side report dict the UI renders. Includes the side-by-side waste-budget bar chart that visually shows which buckets shrank.
+
+### 4. ROCm Knowledge Base
+
+Single file: `kb/rocm_rules.yaml`. ~25 rules at MVP. Schema:
+
+```yaml
+- id: precision.bf16_over_fp16_on_mi300x
+  category: precision
+  symptom: "fp16 used on MI300X"
+  detect:
+    config.precision: fp16
+  fix:
+    config.precision: bf16
+  expected_impact: "Same throughput, +numerical stability. Reduces NaN risk."
+  rocm_version_min: "6.0"
+  citation: "AMD ROCm Best Practices Guide §3.2"
+```
+
+Categories:
+- `precision` (bf16 over fp16 on CDNA3 matrix cores; FP8 for stretch inference scenarios)
+- `attention` (flash-attn ROCm fork via Optimum-AMD, PyTorch SDPA, packed sequences)
+- `memory` (batch-size sweetspot for 192 GB HBM3, gradient checkpoint thresholds, activation offloading)
+- `kernels` (hipBLASLt hint logging + offline tuning files; MIOpen `MIOPEN_FIND_*` autotune)
+- `env_vars` (`HSA_FORCE_FINE_GRAIN_PCIE`, `MIOPEN_FIND_MODE`, `NCCL_MIN_NCHANNELS=112`, NUMA auto-balancing disable)
+- `optimizer` (8-bit Adam on ROCm — **warn that bitsandbytes is not officially supported on ROCm**; recommend Optimum-AMD-validated alternatives or CPU-offload optimizers)
+- `data` (`num_workers`, `pin_memory=True`, `prefetch_factor`, `persistent_workers=True`, IterableDataset sharding correctness)
+- `compile` (`torch.compile` when graph-break evidence is low; `torch_compile=True` in `TrainingArguments`)
+- `collectives` (RCCL channel count, one-process-per-GPU vs one-process-many-GPUs)
+- `topology` (tensor parallelism within a single XGMI island; prefer TP over EP on single-node)
+
+Rules are hand-curated on Day 1 from ROCm docs + AMD blog posts. **This is the moat.** No LLM-generated rules. Each rule entry includes a `citation` field linking back to a ROCm doc page or AMD blog post — every recommendation in the final report carries this citation.
+
+**Footgun guardrail — `ROCPROFSYS_*` are NOT tuning vars.** ROCm Systems Profiler env vars (`ROCPROFSYS_MODE`, `ROCPROFSYS_USE_SAMPLING`, etc.) configure how the *profiler* observes a run; they do not affect how work is dispatched to the GPU. The KB **excludes** all `ROCPROFSYS_*` vars from optimization rules. Additionally, `parse_config` flags any user config that sets `ROCPROFSYS_*` as if it were a perf knob and surfaces a warning in the report ("These configure the profiler, not the workload — they will not change throughput").
+
+**Workload-validity disclaimer.** Every recommendation is valid for the specific tuple `(workload script, model, GPU=MI300X, ROCm version, framework version, batch/seq pattern)` observed during profiling. The system prompt instructs the agent to state this explicitly when uncertainty is high, and the final report includes a footer line: *"Recommendations validated against MI300X with ROCm <version> and PyTorch <version>. Re-run audit if you change model, hardware, or framework version."*
+
+### 5. Profiling Pipeline
+
+A wrapper script `goblin_runner.sh` invokes the user's training command inside the ROCm container, with:
+
+```bash
+# isolate to a single MI300X to keep concurrent benchmark runs sane
+export ROCR_VISIBLE_DEVICES=${GOBLIN_GPU_ID:-0}
+
+rocprofv3 --hsa-trace --kernel-trace -o trace.csv -- \
+  python -m torch.profiler.scheduler ... \
+  python <user_script> --max_steps=10
+```
+
+Post-processing (`profile_parser.py`):
+- parse `trace.csv` for kernel breakdown
+- parse `torch.profiler` JSON for tokens/sec + MFU
+- parse `amd-smi` polling output for HBM + util
+- merge into single `RunMetrics` with populated `WasteBudget`
+
+### 6. Benchmark Result Cache
+
+`bench_cache/` directory. Key: `sha256((canonical_config_json, workload_script_sha, rocm_image_tag, runner_script_sha))`. Value: full `RunMetrics` plus the unhashed key tuple for debuggability. Lets us re-run the demo without burning cloud time. The Streamlit UI exposes a `--no-cache` toggle for the Day-3 dry-run pass that confirms cached results haven't gone stale.
+
+### 7. Synthetic Corpus (`workloads/synthetic/`)
+
+A directory of pre-generated, deliberately misconfigured fine-tuning runs. Each entry is `{config.py, expected_findings.yaml, cached_metrics.json}`. Generated on Day 1 by ROCm Lead by perturbing the canonical Qwen2.5-7B LoRA workload along single dimensions: `num_workers=0`, FP32 instead of BF16, naive attention, `torch.compile=False`, untuned hipBLASLt, etc.
+
+Two purposes:
+1. **Backend Lead can develop on a laptop** — test the agent loop, KB queries, and patch generation against `cached_metrics.json` without touching the GPU.
+2. **Demo lane 1 (offline replay)** — judges can audit any synthetic scenario in <30 seconds without live MI300X. If cloud is unreachable on demo day, this is the safety net.
+
+Each scenario's `expected_findings.yaml` lists which rules *should* fire — gives us a regression test for KB recall.
+
+### 8. Demo Lanes
+
+The system runs in one of two modes:
+
+| Lane | Source of metrics | Use |
+|---|---|---|
+| **Offline replay** | Synthetic corpus + cached `RunMetrics` JSON | Dev, regression testing, demo backup if MI300X unreachable |
+| **Live MI300X** | `goblin_runner.sh` → real GPU | Canonical demo, "before/after with real numbers" pitch moment |
+
+The agent loop is identical in both lanes — only the `RunnerProtocol` implementation differs. This is the seam introduced to address the audit's testability finding.
+
+### 9. Secrets & Privacy
+
+Uploaded scripts and logs may contain API tokens, dataset paths, internal URLs. Before any artefact is persisted to disk or sent to the LLM:
+
+1. **Regex redaction pass** in `parse_config` for common patterns: `Bearer [A-Za-z0-9._-]+`, `sk-[A-Za-z0-9]{20,}`, `hf_[A-Za-z0-9]{20,}`, `/home/<user>/`, `s3://`, `wss?://`.
+2. **Ephemeral storage** — `bench_cache/` is gitignored; uploaded scripts are deleted at session end.
+3. **No model weights or training data required** — we audit configs and metrics only. The system prompt explicitly tells the agent it cannot ask for weights or datasets.
+
+This is a one-screen redaction pass, not a full PII pipeline. Sufficient for hackathon; would need expansion for production.
+
+## Data Flow — One Audit
+
+```
+[user uploads script]  ──► [redaction pass]
+        │
+        ▼
+parse_config ──► ConfigDict ────────────────────────────┐
+        │                                                │
+        ▼                                                │
+profile_run ──► RunMetrics + WasteBudget ──► query_rocm_kb │
+        │                                       │        │
+        │                                       ▼        │
+        │                              List[Rule] ──────►│
+        │                                                ▼
+        │                                       propose_patch (uplift + confidence)
+        │                                                │
+        │                                                ▼
+        │                                       Patch (new ConfigDict + diff)
+        │                                                │
+        ▼                                                ▼
+benchmark(original) ◄──── cache ─────► benchmark(patched)
+        │                                                │
+        └────────► compare_runs ◄────────────────────────┘
+                          │
+                          ▼
+                       Report ──► UI
+                       (waste-budget bar chart + metrics + diff)
+```
+
+## Tech Stack (frozen)
+
+| Layer | Choice | Why |
+|---|---|---|
+| Agent LLM | Qwen2.5-7B-Instruct via HF Inference Providers | Tool calling via OpenAI-compatible API; routes through Together / Fireworks-AI / Nebius (auto). Stretch: self-host on MI300X via vLLM. |
+| Backend | Python 3.11 + FastAPI + anthropic SDK | Standard, async-friendly |
+| Frontend | Streamlit | Ships in hours, not days |
+| Container | `rocm/pytorch:rocm6.1_ubuntu22.04_py3.10_pytorch_2.3` | Official AMD image |
+| Profiling | torch.profiler + rocprofv3 + amd-smi | Native ROCm tools |
+| KB index | sentence-transformers + numpy cosine | No vector DB needed for 25 rules |
+| Workload | Qwen2.5-7B + LoRA + alpaca-cleaned | Canonical, reproducible |
+| Hardware | MI300X cloud (single GPU, 192 GB HBM) | Hackathon constraint |
+
+## Repo Layout
+
+```
+amd-hackathon/
+├── brainstorming/          # this folder
+├── kb/
+│   └── rocm_rules.yaml     # the moat
+├── agent/
+│   ├── server.py           # FastAPI app
+│   ├── loop.py             # agent loop driver
+│   ├── prompts.py          # system prompt
+│   └── tools/
+│       ├── parse_config.py
+│       ├── profile_run.py
+│       ├── query_rocm_kb.py
+│       ├── propose_patch.py
+│       ├── benchmark.py
+│       └── compare_runs.py
+├── runner/
+│   ├── goblin_runner.sh    # rocprofv3 wrapper
+│   └── profile_parser.py
+├── ui/
+│   └── app.py              # Streamlit
+├── workloads/
+│   ├── train_qwen_lora.py    # canonical demo workload
+│   └── synthetic/            # pre-generated misconfigured runs + cached metrics
+│       ├── 01_no_workers/
+│       ├── 02_fp32_default/
+│       ├── 03_naive_attention/
+│       └── ...
+├── bench_cache/            # gitignored
+└── docker/
+    └── Dockerfile          # rocm/pytorch + our deps
+```
+
+## Agent Loop — Pseudocode
+
+The loop is provider-agnostic. It talks to a `Backend` (see `agent/backends/`); today's only concrete is `QwenHFBackend`.
+
+```python
+def run_audit(user_script_path: str) -> SSEStream:
+    backend = make_backend(system_prompt=SYSTEM_PROMPT)  # QwenHFBackend
+    backend.add_user_message(f"Audit this fine-tuning workload: {user_script_path}")
+
+    for step in range(MAX_STEPS):  # MAX_STEPS = 8
+        turn = await backend.next_turn(tool_schemas())
+        for text in turn.text_blocks:
+            yield {"type": "thought", "text": text}
+        for tc in turn.tool_calls:
+            yield {"type": "tool_call", "name": tc.name, "input": tc.input}
+            result = call_tool(tc.name, **tc.input)
+            yield {"type": "tool_result", "name": tc.name, "result": result}
+            backend.add_tool_result(tc.id, tc.name, result.content, is_error=not result.ok)
+        if turn.stop_reason == "end_turn":
+            break
+
+    yield {"type": "final_report", "report": extract_final_report(tool_results)}
+```
+
+The Backend protocol (`agent/backends/base.py`) is a 3-method contract:
+`add_user_message`, `next_turn`, `add_tool_result`. `QwenHFBackend` translates
+between this neutral shape and OpenAI-compatible chat-completion calls
+through HF Inference Providers.
+
+## Boundaries & Interfaces
+
+- **Agent ↔ Tools:** Each tool is a pure function with typed input/output. No shared state. Tools never call other tools — only the agent orchestrates.
+- **Tools ↔ MI300X:** Only `profile_run` and `benchmark` touch the GPU. Both go through `goblin_runner.sh` for consistency.
+- **Backend ↔ UI:** SSE stream of typed events (`thought`, `tool_call`, `tool_result`, `final_report`). UI just renders events; no agent logic on frontend.
+- **KB ↔ Code:** YAML is the single source of truth. Code consumes; never generates rules at runtime.
+
+## What's Deliberately Excluded (YAGNI)
+
+- ❌ Multi-GPU / distributed training analysis (single MI300X is enough; collective rules in KB still apply when user uploads multi-GPU configs, we just don't benchmark them)
+- ❌ Pretraining workloads (fine-tuning only)
+- ❌ vLLM / inference serving as a primary mode (Day-4 stretch only; vLLM-specific rules NOT in MVP KB)
+- ❌ Live job watching with intervention (offline audit only — much simpler)
+- ❌ User accounts / persistence / DB (in-memory session, hackathon)
+- ❌ Vector DB (25 rules + numpy is fine)
+- ❌ Polars/DuckDB for run storage (Pydantic + JSON is enough at this scale)
+- ❌ ML model for uplift prediction (deterministic waste-budget calculus is enough)
+- ❌ Auto-applying patches to the user's repo (we output a diff, user applies)
+- ❌ Sliders/what-if interactive panel (chat layer answers what-if conversationally; sliders are stretch only)
+- ❌ Cost calculator beyond a static "$ per training run" line (stretch goal only)

brainstorming/goals.md

@@ -0,0 +1,233 @@
+# GPU Goblin — Goals & Implementation Plan
+
+## North Star
+
+Win the AMD hackathon (**Track 1: AI Agents & Agentic Workflows**) by demonstrating a **real, reproducible 2×+ throughput improvement on a Qwen2.5-7B LoRA fine-tune on MI300X**, driven end-to-end by a Qwen-powered tool-using agent. Hits the Qwen Technology Partner challenge with end-to-end Qwen-on-AMD; deploys as a Hugging Face Space within the hackathon HF Organization.
+
+Everything else is in service of that demo.
+
+## Success Criteria
+
+| # | Criterion | Measurable As |
+|---|---|---|
+| 1 | Real MI300X speedup | ≥ 2.0× tokens/sec, before vs. after, on the canonical workload |
+| 2 | Agent is actually agentic | ≥ 4 tool calls per audit, visible in UI |
+| 3 | AMD differentiation | 100% of recommendations cite a ROCm-specific KB rule |
+| 4 | Uplift estimate honesty | Predicted speedup range (low–high) brackets the measured speedup on ≥ 8 of 10 synthetic corpus scenarios |
+| 5 | Offline parse coverage | ≥ 90% of synthetic corpus parsed without manual edits |
+| 6 | End-to-end reliability | Demo runs cleanly 5× in a row without manual intervention |
+| 7 | Pitch quality | 3-min demo + 1-min architecture + 1-min impact, rehearsed |
+| 8 | Backup safety | Offline-replay lane works fully when MI300X disconnected |
+
+## Mapping to Judging Criteria
+
+| Hackathon criterion | Where we score |
+|---|---|
+| **Application of Technology** | Tool-using agent loop on MI300X; rocprofv3 + torch.profiler integration; deterministic uplift via waste-budget calculus; ROCm-specific KB |
+| **Presentation** | Live before/after benchmark on real MI300X; waste-budget bar chart; chat-based "why?" interaction; 90-sec backup video |
+| **Business Value** | Each audit reports `$ saved per training run` and `time saved per epoch`, framed against $1.99/hr public MI300X pricing reference |
+| **Originality** | "AI for AI builders" — meta-positioning. Real benchmarks, not just LLM advice. Waste-budget decomposition is novel framing |
+
+## Team Roles (3 people × 4 days)
+
+### ROCm / ML Lead
+- Day 1: MI300X cloud env, ROCm container, baseline workload (Qwen2.5-7B LoRA on alpaca)
+- Day 1-2: Hand-curate 20-25 KB rules from ROCm docs + AMD blog
+- Day 2: `profile_run` + `benchmark` tools (rocprofv3 wrapper, parser)
+- Day 3: Validate end-to-end speedup on canonical demo, generate cached results
+- **Owns:** anything that touches the GPU
+
+### Agent / Backend Lead
+- Day 1: FastAPI skeleton, tool schemas, Qwen-via-HF tool-use plumbing
+- Day 1-2: `parse_config`, `propose_patch`, `query_rocm_kb`, `compare_runs` tools
+- Day 2: Full agent loop, system prompt, SSE streaming
+- Day 3: Hardening, error handling, max-steps cap, fallback behaviors
+- **Owns:** agent reasoning quality + tool wiring
+
+### Frontend / Demo Lead
+- Day 1: Streamlit skeleton, file upload, message log
+- Day 2: Tool-call cards (live status), report renderer, diff viewer
+- Day 3: Demo polish, charts, golden-run replay mode
+- Day 4: Pitch deck, 90-second backup video, dry runs
+- **Owns:** what the judges actually see
+
+Roles overlap on integration days — pair-program when blocked.
+
+## Day-by-Day Plan
+
+### Day 1 (May 5–6) — Foundations
+
+**ROCm Lead**
+- [ ] Provision MI300X cloud instance via AMD Developer Cloud ($100 credits), SSH access, persistent storage
+- [ ] Pull `rocm/pytorch:rocm6.1_*` image, verify GPU visible inside container
+- [ ] Run baseline `train_qwen_lora.py` (batch=4, fp16, naive attention, alpaca, 100 steps)
+- [ ] Capture baseline tokens/sec, MFU, HBM peak — *this is our "before"*
+- [ ] Generate **synthetic corpus** — 5-8 misconfigured variants of the canonical workload (FP32, num_workers=0, naive attention, etc.) with cached `RunMetrics` JSON for each
+- [ ] Start drafting KB rules (target 10 rules by EOD), each tagged with `targets_bucket` matching the waste-budget decomposition
+
+**Backend Lead**
+- [ ] Repo scaffold per architecture.md layout
+- [ ] `pip install fastapi anthropic sentence-transformers pyyaml pydantic`
+- [ ] Define `agent/schemas.py` with `RunMetrics`, `WasteBudget`, `ConfigDict`, `Patch`, `Rule`, `Report` as pydantic models — **Day-1 priority** (blocks all tools)
+- [ ] Define `RunnerProtocol` interface; build `FakeRunner` that loads cached metrics from `workloads/synthetic/` (lets backend dev without MI300X)
+- [ ] FastAPI `POST /audit` skeleton with SSE
+- [ ] `parse_config` tool — handle HF `TrainingArguments` first; include regex redaction pass for tokens/paths
+- [ ] Qwen tool-use hello-world (one tool, one round-trip via HF Inference Providers)
+
+**Frontend Lead**
+- [ ] Streamlit skeleton with file upload + chat panel
+- [ ] Hardcoded "fake audit" — render canned tool calls + report
+- [ ] Pick chart library (Altair recommended for Streamlit)
+
+**Day 1 Exit Criteria**
+- Baseline benchmark numbers in hand
+- Synthetic corpus has ≥ 3 cached scenarios (Backend Lead can now dev without GPU)
+- Schemas (`RunMetrics`, `WasteBudget`, etc.) frozen
+- `RunnerProtocol` + `FakeRunner` working end-to-end
+- Backend can call Qwen with one tool via HF Inference Providers
+- UI renders a fake audit
+- 10 KB rules drafted
+
+### Day 2 (May 6–7) — Core Build
+
+**ROCm Lead**
+- [ ] Finish KB to 20-25 rules; hand-tag categories + `targets_bucket`; pre-embed with sentence-transformers
+- [ ] Include the high-impact MI300X rules: BF16-over-FP16, AITER-flash-attn-via-Optimum-AMD, `NCCL_MIN_NCHANNELS=112`, NUMA disable, one-process-per-GPU, hipBLASLt hint logging, MIOpen `MIOPEN_FIND_*`, **bitsandbytes-not-supported-on-ROCm warning**, `num_workers`/`pin_memory`/`prefetch_factor`/`persistent_workers`
+- [ ] `profile_run` tool: rocprofv3 wrapper + torch.profiler + amd-smi → `RunMetrics` with `WasteBudget`
+- [ ] `benchmark` tool: same pipeline, longer run, with version-tagged cache
+- [ ] Validate on baseline workload: profile output makes sense, MFU is plausible, waste budget sums to ~T_total
+
+**Backend Lead**
+- [ ] All 6 tools wired and individually tested with fixtures from synthetic corpus
+- [ ] Full agent loop with max-steps cap, SSE event types finalized, error envelope per tool (`{ok, result, error}`)
+- [ ] System prompt iterated against test workloads — include MI300X hardware specs (304 CUs, 192 GB HBM3, ~5.3 TB/s, FP8 native) so the agent reasons quantitatively
+- [ ] `propose_patch` deterministic transformer + uplift estimator (waste-budget × bucket recovery) + confidence formula
+
+**Frontend Lead**
+- [ ] Live tool-call cards consuming real SSE stream
+- [ ] Final report layout: side-by-side metrics + diff + kernel chart + **waste-budget bar chart** (where time was lost, before vs after)
+- [ ] Lane toggle: `Live MI300X` vs `Offline replay (synthetic corpus)` — judges can pick either
+- [ ] First end-to-end run through the actual backend
+
+**Day 2 Exit Criteria**
+- Real audit runs end-to-end on a toy workload
+- Profile + benchmark return real MI300X numbers
+- KB has 20+ rules, embeddings pre-computed
+- UI streams real agent activity
+
+### Day 3 (May 7–8) — Demo Day Prep
+
+**All hands**
+- [ ] Run canonical demo (Qwen2.5-7B LoRA) end-to-end → confirm ≥ 2× speedup
+- [ ] Cache the demo benchmark results — don't burn cloud time on every rehearsal
+- [ ] Build "golden run" replay mode (read cached SSE events, replay timing)
+- [ ] Validate uplift accuracy on synthetic corpus: predicted range should bracket measured speedup on ≥ 8 of 10 scenarios
+- [ ] Polish system prompt for demo-friendly narration ("I'll start by…")
+- [ ] Tighten error handling — agent should never panic in front of judges; tool failures degrade gracefully with `{ok:false}` envelopes
+- [ ] Run with `--no-cache` once to verify cached results aren't masking real bugs
+- [ ] 5× clean dry runs, fix anything flaky
+
+**Day 3 Exit Criteria**
+- Canonical demo: cleanly runs in ≤ 4 minutes, ≥ 2× speedup, no manual fixes
+- Cached results enable offline replay
+- Offline-replay lane works fully when MI300X is disconnected (proven by unplugging cloud)
+- Golden-run video recorded as backup
+
+### Day 4 (May 8–9) — Pitch & Stretch
+
+**Frontend Lead**
+- [ ] Pitch deck (5 slides): problem, agent loop diagram, demo (live), KB rules sample, impact + ask
+- [ ] Cover image for the submission listing
+- [ ] Final 90-second backup video
+- [ ] Submission form filled (title, short + long description, tags), repo public, README crisp
+- [ ] **Build-in-Public bonus track:** at least one public update post (X/LinkedIn/Discord) showing the agent's first audit; one ROCm/Optimum-AMD feedback note based on what was rough during build
+
+**ROCm + Backend Leads (in parallel, optional stretch)**
+- [ ] vLLM inference workload as second demo (only if rock-solid on Day 3)
+- [ ] Cost calculator: `$ saved per training run` line, anchored on $1.99/hr public MI300X reference
+- [ ] What-if slider panel for batch / precision / attention (chat already does this conversationally, sliders are visual icing)
+- [ ] Stretch dream: self-host Qwen on MI300X via vLLM (replacing the HF-Inference-Providers path) — closes the loop entirely on AMD silicon. Mention in pitch even if not running live.
+
+**Day 4 Exit Criteria**
+- Submission complete by deadline
+- 5+ rehearsed dry runs of the pitch
+- Cover image, video, slides, repo all linked from submission form
+- Backup video on USB stick, in cloud, on phone
+
+## Scope Discipline (YAGNI)
+
+If we're behind schedule, **cut in this order**:
+
+1. ❌ All stretch goals (vLLM, cost calc, self-hosted agent)
+2. ❌ Live tool-call UI animations — static cards work
+3. ❌ Some KB categories (keep precision/attention/memory; drop env_vars/collectives)
+4. ❌ Multi-file script parsing — single-file only
+5. ❌ Dynamic batch-size search — hardcode the recommended batch for demo workload
+
+**Never cut:**
+- Real MI300X benchmark (that's the entire pitch)
+- Tool-using agent loop (that's the track fit)
+- ROCm-specific KB citations (that's the differentiation)
+
+## Risk Register
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| MI300X cloud quota burns out | Medium | High | Cache every benchmark, dev with 10-step traces, full benchmarks only for canonical demo |
+| `rocprofv3` flaky / version mismatch | Medium | High | Always run `torch.profiler` in parallel as fallback. Pre-record golden trace. Use `rocprofv3` not deprecated `rocprof`/`rocprofv2` |
+| MI300X cloud unreachable on demo day | Low | Critical | **Offline-replay lane** (synthetic corpus + cached metrics) provides full demo without cloud — judges can't tell the difference for the agent-reasoning portion |
+| Uploaded scripts contain user secrets / tokens | Medium | Medium | Regex redaction pass in `parse_config` before any persistence or LLM call; ephemeral storage; explicit "we never store weights or datasets" line in UI |
+| Agent loops infinitely | Low | Medium | Hard cap of 8 tool calls, fall back to "best effort" report after cap |
+| Recommendations are generic, not ROCm-specific | Medium | Critical | Hand-curate KB on Day 1 *before* wiring agent — KB is the moat |
+| Demo crashes during pitch | Low | Critical | Pre-recorded video backup. Golden-run replay mode. USB + cloud + phone copies |
+| Qwen2.5-7B doesn't fit a 12-batch on MI300X | Low | Medium | Have a fallback config in hand (batch=8 + grad_accum=2) |
+| LoRA on alpaca too easy — speedup looks staged | Low | High | Measure on a non-trivial seq_len (1024+), include MFU not just tokens/sec, show kernel breakdown to prove it's real |
+| HF Inference Provider rate limit / outage during demo | Low | Medium | Offline-replay UI lane plays cached_audit.json without any backend; pre-cache a full recorded session; have a backup HF token; `provider="auto"` already routes around individual provider outages |
+| Team member unavailable (illness, etc.) | Low | High | Pair on critical path (agent loop, KB) so no single point of failure |
+
+## Definition of Done — MVP
+
+GPU Goblin is "done enough to ship" when **all** of these are true:
+
+- ✅ A judge can upload `train_qwen_lora.py` (we provide it) and get a real audit
+- ✅ Agent makes ≥ 4 visible tool calls
+- ✅ Final report shows ≥ 2× tokens/sec, real numbers from MI300X
+- ✅ Every recommendation in the report cites a ROCm KB rule by ID
+- ✅ User can ask follow-up questions in chat ("why bf16?") and get cited answers
+- ✅ Full audit completes in ≤ 4 minutes
+- ✅ 5 consecutive dry runs succeed without manual intervention
+- ✅ Backup video and cached replay both work
+
+## Stretch Definition of Done
+
+- 🎯 Second canonical workload (vLLM inference) audited end-to-end
+- 🎯 Cost calculator: "you save $X per training run, $Y per epoch"
+- 🎯 Agent backed by self-hosted Qwen via vLLM on the same MI300X (the ultimate AMD story — replaces today's HF Inference Providers path with on-cluster serving)
+
+## Compute Budget — AMD Developer Cloud Credits
+
+Eligible participants get **$100 in AMD Developer Cloud credits**. Public reference price for MI300X is around $1.99/hr (single VM) or $3.39/hr (8× bare metal). Plan accordingly:
+
+| Activity | GPU-hours | Notes |
+|---|---|---|
+| Day-1 baseline + synthetic corpus generation | 4–6 | One-shot, results cached |
+| Day-2 KB validation runs | 2–3 | Sanity-check the rules fire on synthetic scenarios |
+| Day-3 canonical demo dry runs (cached) | 2–4 | Cache hits after the first run |
+| Day-3 `--no-cache` cold validation | 1 | Confirms nothing's stale |
+| Day-4 final dry runs + record video | 2–3 | Lock the demo |
+| **Total estimate** | **~12–17 hrs** | Well within $100 even at bare-metal rates |
+
+Backend Lead spends zero MI300X time after Day 1 — develops against synthetic corpus + `FakeRunner`.
+
+## Submission Checklist
+
+- [ ] Public GitHub repo with clear README + architecture diagram + setup instructions
+- [ ] 90-second demo video (live agent run, real MI300X numbers)
+- [ ] Pitch deck (PDF or slides URL)
+- [ ] Cover image (project listing visual)
+- [ ] Architecture diagram (PNG)
+- [ ] Sample audit report PDF (one canonical run, before/after, including waste-budget chart)
+- [ ] Short description (1-2 sentences) + long description (paragraph) + technology/category tags
+- [ ] At least one Build-in-Public post + one ROCm/Optimum-AMD feedback note (bonus track)
+- [ ] Hackathon submission form filled
+- [ ] Team member credits + contact

brainstorming/idea.md

@@ -0,0 +1,105 @@
+# GPU Goblin — Idea
+
+> **An AI agent that hunts wasted compute on AMD MI300X.**
+> Upload a fine-tuning config; the agent profiles a real run, diagnoses inefficiency, recommends ROCm-specific fixes, then re-runs and proves the speedup with hard numbers.
+
+---
+
+## The Problem
+
+Most teams fine-tuning LLMs on AMD hardware leave **2–3× of throughput on the floor** without realizing it. The waste hides in plain sight:
+
+- **HBM under-fed** — batch sizes copied from NVIDIA tutorials don't exploit MI300X's 192 GB.
+- **Wrong precision** — `fp16` is the reflexive default; `bf16` matches throughput on CDNA3 with better numerics.
+- **Naive attention** — teams ship without flash-attn ROCm fork or PyTorch SDPA enabled.
+- **Generic kernels** — hipBLASLt tuning, MIOpen autotune, RCCL collective tweaks all left at defaults.
+- **Wrong libraries** — `bitsandbytes` is the reflexive choice for 8-bit Adam, but it's **not officially supported on ROCm**. Optimum-AMD validates Flash Attention 2 + GPTQ/AWQ paths instead.
+- **Distributed footguns** — one-process-driving-many-GPUs serializes kernel launches on ROCm; NUMA auto-balancing causes variance; `NCCL_MIN_NCHANNELS` left at default.
+- **Knobs nobody knows about** — `HSA_FORCE_FINE_GRAIN_PCIE`, `MIOPEN_FIND_MODE`, hipBLASLt cache paths.
+
+Engineers don't fix this not because they're lazy — it's because the knowledge is **scattered across ROCm docs, AMD blog posts, GitHub issues, and Discord threads**. Nobody has time to read it all before launching a run.
+
+## The Solution — GPU Goblin
+
+A tool-using agent that does what an experienced AMD performance engineer would do, in one minute:
+
+1. **Read** the user's fine-tuning script or HF `TrainingArguments`.
+2. **Profile** a 10-step warm run on MI300X (torch.profiler + `rocprofv3` + amd-smi).
+3. **Diagnose** against a curated knowledge base of ROCm-specific optimizations.
+4. **Patch** the config — concrete diff, not vague advice.
+5. **Benchmark** the patched config on the same MI300X.
+6. **Report** before/after side-by-side with real numbers and citations.
+
+The user keeps interacting in chat: *"why bf16?"*, *"what if I can't change the optimizer?"*, *"how much $ does this save per epoch?"*
+
+## Why This Wins
+
+**Most teams build user-facing AI. We build AI for AI builders.** That alone is judge-bait.
+
+But the deeper hook is that GPU Goblin is **provably useful** in a way most hackathon projects aren't. We don't have to convince judges the recommendations are good — we *show* them, on the same MI300X, with the same model, in the same demo. Tokens/sec: `142 → 318`. End of debate.
+
+### Track Fit (Track 1: AI Agents & Agentic Workflows)
+
+- **Primary track:** AI Agents & Agentic Workflows. Real tool-using loop. The agent observes (profile), hypothesizes (KB query), tests (benchmark), refines (patch). Every step visible in chat. Not a one-shot LLM call dressed up as an agent.
+- **Fine-tuning is what we *audit*, not the track we enter.** Canonical workload is **Qwen2.5-7B-Instruct + LoRA on MI300X**. Recommendations are *fine-tuning specific* — batch sizing for LoRA, gradient checkpointing thresholds, optimizer choice (8-bit Adam on ROCm), seq-length packing.
+
+### Qwen Technology Partner Challenge
+
+GPU Goblin satisfies the Qwen partner challenge two ways:
+
+1. **Qwen as the agent brain.** The audit loop runs on `Qwen/Qwen2.5-7B-Instruct` served via Hugging Face Inference Providers. Every tool call, every recommendation, every chat answer is generated by Qwen.
+2. **Qwen as the audit target.** The canonical demo workload is a Qwen2.5-7B-Instruct LoRA fine-tune. The agent audits *the same model family it runs on*.
+
+Result: end-to-end Qwen on AMD silicon. Directly answers the judging criterion *"How effectively the chosen model(s) are integrated into the solution."*
+
+### Hugging Face Integration
+
+Hugging Face is the named Technology Partner — both **model hub** and **deployment layer**:
+
+- **Model hub:** Qwen models pulled from HF Hub at audit time + at agent-spin-up time.
+- **Optimum-AMD:** several KB rules cite Optimum-AMD validated paths (Flash Attention 2 on MI300, GPTQ/AWQ ROCm path).
+- **Inference Providers:** the agent's Qwen calls go through HF's Inference Providers router (`provider="auto"` selects Together / Fireworks-AI / Nebius based on availability).
+- **Deployment as HF Space:** the Streamlit UI ships as a Space within the AMD Developer Hackathon HF Organization, satisfying the "Demo Application Platform" + "Application URL" submission fields. The Space runs in offline-replay mode by default so it works without any live backend.
+
+### AMD Differentiation
+
+Every single recommendation cites a **ROCm-specific rule**, not generic PyTorch advice. The knowledge base is the moat:
+
+- ROCm env-var tuning (`HSA_*`, `MIOPEN_*`, `NCCL_MIN_NCHANNELS=112`)
+- hipBLASLt hint logging + offline tuning files; MIOpen `MIOPEN_FIND_*` autotune
+- Flash-attn ROCm fork (validated on MI300 via Optimum-AMD)
+- RCCL topology + tensor-parallel placement within a single XGMI island
+- bitsandbytes-on-ROCm gotchas (not officially supported — recommend alternatives)
+- bf16 vs fp16 on CDNA3 matrix cores
+- One-process-per-GPU vs one-process-many-GPUs (ROCm serializes launches in the latter)
+- NUMA auto-balancing disable for stable benchmarks
+- MI300X-specific: 304 CUs, 192 GB HBM3, ~5.3 TB/s peak bandwidth, native FP8
+
+This is the kind of insight you only get from someone who has actually shipped on MI300X. We bottle it.
+
+## Demo Narrative (3 minutes)
+
+**Setup (15s):** "Most teams waste 50%+ of their MI300X. Watch GPU Goblin find that waste live."
+
+**Live demo (2 min):**
+
+1. Drop in `train_qwen_lora.py` — batch=4, fp16, naive attention. (Looks normal.)
+2. Agent: *"Parsing config…"* → extracts hyperparams.
+3. Agent: *"Running 10-step profile on MI300X…"* → shows real metrics: HBM 38%, MFU 24%.
+4. Agent: *"Querying ROCm playbook…"* → 4 issues found, each with citation.
+5. Agent: *"Generating optimized config…"* → diff appears: bf16, batch=12, flash-attn ROCm, hipBLASLt env, packed sequences.
+6. Agent: *"Benchmarking new config — 50 steps on MI300X…"* → live progress.
+7. Final report: **142 → 318 tokens/sec (2.24×). MFU 24% → 51%. $X saved per epoch.**
+
+**Why it works (45s):** Show the agent loop. Show the KB. Land the line: *"the agent runs Qwen on AMD silicon, the audit target is Qwen on AMD silicon, the optimizations are AMD-specific. End-to-end AMD."*
+
+## What This Is *Not*
+
+- Not a generic PyTorch profiler GUI (`rocprofv3` + tensorboard already exist)
+- Not a chatbot wrapping a static FAQ (the agent runs real benchmarks)
+- Not a multi-cloud cost tool (focused on MI300X compute waste, not infra cost)
+- Not a fine-tuning trainer itself (we *audit* others' runs)
+
+## One-Line Pitch
+
+> **GPU Goblin is the AMD performance engineer you wish was on your team — except it costs five minutes and audits any fine-tuning run on MI300X.**

pyproject.toml

@@ -0,0 +1,43 @@
+[project]
+name = "gpu-goblin"
+version = "0.1.0"
+description = "An AI agent that hunts wasted compute on AMD MI300X"
+requires-python = ">=3.10"
+dependencies = [
+  "fastapi>=0.110",
+  "uvicorn[standard]>=0.27",
+  "python-multipart>=0.0.9",
+  "huggingface_hub>=0.28",
+  "pydantic>=2.6",
+  "pyyaml>=6.0",
+  "sentence-transformers>=2.7",
+  "numpy>=1.26",
+  "sse-starlette>=2.0",
+  "streamlit>=1.32",
+  "altair>=5.2",
+  "pandas>=2.2",
+  "requests>=2.31",
+  "openai>=1.30",
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0",
+  "pytest-asyncio>=0.23",
+  "ruff>=0.4",
+]
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+include = ["agent*", "runner*", "ui*"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP"]
+ignore = ["E501"]

runner/goblin_runner.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+# goblin_runner.sh — wrap the user's training command with rocprofv3 + amd-smi.
+#
+# Architecture: brainstorming/architecture.md §5 (Profiling Pipeline).
+#
+# Inputs (env vars):
+#   USER_SCRIPT     Path to the workload python file. Required.
+#   OUT_DIR         Directory to write trace.csv / torch_profile.json /
+#                   amd_smi.csv. Required. Created if missing.
+#   STEPS           --max_steps argument forwarded to the user script. Default 10.
+#   GOBLIN_GPU_ID   ROCR_VISIBLE_DEVICES value. Default 0.
+#
+# Outputs (in $OUT_DIR):
+#   trace.csv          rocprofv3 kernel trace
+#   torch_profile.json torch.profiler chrome trace (the user script writes this)
+#   amd_smi.csv        amd-smi telemetry sampled at 200 ms
+#   stdout.log         user-script stdout
+#   stderr.log         user-script stderr
+#
+# Failure mode: any non-zero rocprofv3 exit short-circuits the script. On
+# failure we dump the captured stdout/stderr logs to THIS script's own stderr
+# so `subprocess.run(capture_output=True)` in LiveRunner sees the real error
+# (not just an empty `[]` tail). LiveRunner then archives the whole OUT_DIR
+# under bench_cache/last_runner_failure_<ts>/ so you can inspect after-the-fact.
+
+set -uo pipefail
+
+: "${USER_SCRIPT:?USER_SCRIPT env var is required}"
+: "${OUT_DIR:?OUT_DIR env var is required}"
+STEPS="${STEPS:-10}"
+
+mkdir -p "$OUT_DIR"
+
+# Pin to a single MI300X so concurrent benchmark runs don't fight.
+export ROCR_VISIBLE_DEVICES="${GOBLIN_GPU_ID:-0}"
+
+# Background HBM/power telemetry. Different amd-smi versions ship slightly
+# different flag names (--interval / --watch / no flag at all in older
+# builds), so try a few variants and gracefully degrade. Telemetry is
+# optional — profile_parser tolerates a missing amd_smi.csv.
+start_amd_smi() {
+    if ! command -v amd-smi >/dev/null 2>&1; then
+        echo "amd-smi not on PATH; skipping telemetry sidecar" \
+            > "$OUT_DIR/amd_smi.err"
+        return 1
+    fi
+    # amd-smi telemetry surface drifted hard across rocm versions. We try
+    # subcommands in this order, newest-first; first one that survives 0.5s
+    # is kept:
+    #
+    #   1. `amd-smi metric --watch <s> --mem-usage --usage --csv` (ROCm 7.x).
+    #      Different code path from `monitor`, so it dodges the known
+    #      `AttributeError: 'Namespace' object has no attribute 'violation'`
+    #      crash that some ROCm 7.x point releases ship inside the `monitor`
+    #      subcommand. Produces VRAM_USED_MB / GFX_ACTIVITY / MEM_ACTIVITY
+    #      columns that profile_parser._pick_column already recognises.
+    #
+    #   2. `amd-smi monitor --watch <s> ...` (ROCm 6.x and the 7.x builds
+    #      where `monitor` actually works). Kept as a fallback for older
+    #      installs that may not have the `metric --watch` form.
+    #
+    #   3. `amd-smi monitor --interval <s>` (pre-6.0) and finally bare
+    #      `amd-smi monitor` (very old / implicit-all).
+    local variants=(
+        # ROCm 7.x: prefer the metric subcommand — bypasses the monitor bug.
+        "amd-smi metric --watch 1 --mem-usage --usage --csv"
+        "amd-smi metric -w 1 -m -u --csv"
+        # ROCm 7.x monitor (works on builds without the violation-attribute bug)
+        "amd-smi monitor --watch 1 --power-usage --gfx --mem --vram-usage --csv"
+        "amd-smi monitor -w 1 -p -u -m -v --csv"
+        # ROCm 6.x intermediate forms
+        "amd-smi monitor --watch 1 --csv"
+        "amd-smi monitor --watch 1"
+        # Older / fallback
+        "amd-smi monitor --interval 1 --csv"
+        "amd-smi monitor --interval 1"
+        "amd-smi monitor --csv"
+        "amd-smi monitor"
+    )
+    for cmd in "${variants[@]}"; do
+        # shellcheck disable=SC2086
+        $cmd > "$OUT_DIR/amd_smi.csv" 2> "$OUT_DIR/amd_smi.err" &
+        local pid=$!
+        sleep 0.5
+        if kill -0 "$pid" 2>/dev/null; then
+            AMD_SMI_PID=$pid
+            return 0
+        fi
+    done
+    # All variants failed. Telemetry is optional; mark it as deliberately
+    # skipped and let the main run proceed — profile_parser tolerates a
+    # missing/empty amd_smi.csv.
+    echo "amd-smi monitor: no compatible flag set in this build; telemetry skipped" \
+        > "$OUT_DIR/amd_smi.skipped"
+    rm -f "$OUT_DIR/amd_smi.csv"
+    return 1
+}
+
+AMD_SMI_PID=
+start_amd_smi || true   # never block the main run on telemetry
+
+cleanup() {
+    if [[ -n "${AMD_SMI_PID:-}" ]] && kill -0 "$AMD_SMI_PID" 2>/dev/null; then
+        kill "$AMD_SMI_PID" 2>/dev/null || true
+        wait "$AMD_SMI_PID" 2>/dev/null || true
+    fi
+}
+trap cleanup EXIT
+
+# Dump the captured logs to *our* stderr on a non-zero rocprofv3 exit so the
+# Python subprocess that spawned us actually sees the real error message.
+# Without this the redirected stdout.log / stderr.log live inside the tempdir
+# only and LiveRunner's stderr-tail check sees nothing.
+dump_failure_logs() {
+    local code=$?
+    if [[ $code -ne 0 ]]; then
+        {
+            echo "=== goblin_runner.sh failed with exit code $code ==="
+            echo "=== USER_SCRIPT: $USER_SCRIPT ==="
+            echo "=== OUT_DIR:     $OUT_DIR ==="
+            echo "=== ROCR_VISIBLE_DEVICES: ${ROCR_VISIBLE_DEVICES:-unset} ==="
+            echo
+            echo "=== last 50 lines of $OUT_DIR/stdout.log ==="
+            tail -n 50 "$OUT_DIR/stdout.log" 2>/dev/null || echo "(stdout.log missing)"
+            echo
+            echo "=== last 50 lines of $OUT_DIR/stderr.log ==="
+            tail -n 50 "$OUT_DIR/stderr.log" 2>/dev/null || echo "(stderr.log missing)"
+            echo
+            echo "=== last 20 lines of $OUT_DIR/amd_smi.err ==="
+            tail -n 20 "$OUT_DIR/amd_smi.err" 2>/dev/null || echo "(amd_smi.err missing)"
+        } 1>&2
+    fi
+    return $code
+}
+
+# rocprofv3 collects HSA + kernel traces. The user script is responsible for
+# writing torch_profile.json (the agent injects torch.profiler around the
+# training loop in Phase 3). --output-format csv keeps parsing simple.
+set +e
+rocprofv3 \
+    --hsa-trace --kernel-trace \
+    --output-directory "$OUT_DIR" \
+    --output-file trace \
+    --output-format csv \
+    -- \
+    python "$USER_SCRIPT" \
+        --max_steps="$STEPS" \
+        --torch_profile_out="$OUT_DIR/torch_profile.json" \
+        > "$OUT_DIR/stdout.log" 2> "$OUT_DIR/stderr.log"
+ROCPROF_EXIT=$?
+set -e
+
+if [[ $ROCPROF_EXIT -ne 0 ]]; then
+    (exit $ROCPROF_EXIT) || dump_failure_logs
+    exit $ROCPROF_EXIT
+fi
+
+# rocprofv3 may write trace_kernel_trace.csv etc. — normalize to trace.csv so
+# profile_parser has one stable filename to look for.
+if [[ ! -f "$OUT_DIR/trace.csv" ]]; then
+    for candidate in "$OUT_DIR"/trace*kernel*.csv "$OUT_DIR"/*kernel_trace.csv; do
+        if [[ -f "$candidate" ]]; then
+            cp "$candidate" "$OUT_DIR/trace.csv"
+            break
+        fi
+    done
+fi
+
+exit 0

runner/profile_parser.py

@@ -0,0 +1,693 @@
+"""profile_parser — turn goblin_runner.sh artefacts into RunMetrics.
+
+Reads three files written by goblin_runner.sh (architecture.md §5):
+
+    <out_dir>/trace.csv          rocprofv3 kernel trace
+    <out_dir>/torch_profile.json torch.profiler chrome trace
+    <out_dir>/amd_smi.csv        amd-smi telemetry, ~200 ms cadence
+
+and produces one `RunMetrics` with a populated `WasteBudget`.
+
+Each waste-budget bucket is computed with a deliberately simple, documented
+heuristic — these are best-effort signals for the agent, NOT measured
+ground-truth. See the docstring on `_waste_budget()` for the per-bucket
+formula and its known failure modes.
+
+This module is import-tolerant on machines without the rocprofv3 stack —
+it only reads files. Missing or unparseable files degrade individual
+metrics to zero and append a warning to RunMetrics.warnings rather than
+raising. LiveRunner ultimately decides what to do with parse failures.
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+import logging
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from agent.schemas import KernelEntry, RunMetrics, WasteBudget, WorkloadConfig
+
+_LOG = logging.getLogger(__name__)
+
+
+# MI300X has 192 GB HBM3. We treat sustained <70% utilisation as headroom.
+_HBM_TOTAL_GB = 192.0
+_HBM_HEALTHY_TARGET = 0.70
+
+# Kernel-name patterns used by the heuristics.
+#
+# We use `(?<![A-Za-z0-9])` / `(?![A-Za-z0-9])` instead of `\b` so that
+# underscores act as token separators — kernel names like
+# `rccl_AllReduce` and `hipBLASLt_generic_gemm` should match.
+_BOUND_L = r"(?<![A-Za-z0-9])"
+_BOUND_R = r"(?![A-Za-z0-9])"
+_RCCL_PATTERN = re.compile(
+    _BOUND_L + r"(rccl|nccl|all[_-]?reduce|broadcast|reduce[_-]?scatter)" + _BOUND_R, re.I
+)
+_GEMM_PATTERN = re.compile(_BOUND_L + r"(gemm|matmul|hgemm|sgemm|hipblaslt)" + _BOUND_R, re.I)
+_GENERIC_GEMM_PATTERN = re.compile(
+    _BOUND_L + r"(generic|fallback|naive|reference)" + _BOUND_R, re.I
+)
+_FP16_PATTERN = re.compile(_BOUND_L + r"(fp16|half|f16)" + _BOUND_R, re.I)
+_BF16_PATTERN = re.compile(_BOUND_L + r"(bf16|bfloat16)" + _BOUND_R, re.I)
+
+
+# ---------------------------------------------------------------------------
+# Public entry point
+# ---------------------------------------------------------------------------
+
+
+def parse(
+    out_dir: Path,
+    config: WorkloadConfig | None = None,
+    steps: int = 10,
+) -> RunMetrics:
+    """Build a `RunMetrics` from goblin_runner.sh artefacts in `out_dir`.
+
+    `config` is used by some heuristics (e.g. precision_path skips the
+    estimate when the config is already bf16). It can be None at the cost
+    of a more conservative waste-budget estimate.
+
+    Always returns a RunMetrics object — individual metrics degrade to
+    zero with a warning rather than raising, because LiveRunner ultimately
+    decides whether parse failures should trigger fallback.
+    """
+    warnings: list[str] = []
+
+    kernels = _read_kernels(out_dir / "trace.csv", warnings)
+    torch_summary = _read_torch_profile(out_dir / "torch_profile.json", warnings)
+    smi = _read_amd_smi(out_dir / "amd_smi.csv", warnings)
+
+    top_kernels = _top_kernels(kernels, top_n=5)
+    gpu_util_pct = smi.gpu_util_pct if smi.gpu_util_pct is not None else _gpu_util_from_kernels(
+        kernels, torch_summary
+    )
+    # Write the resolved gpu_util back into smi so _waste_budget sees the
+    # same number RunMetrics reports. Without this, a build where amd-smi's
+    # `usage` column returns N/A leaves smi.gpu_util_pct=None — and
+    # _waste_budget reads it as 0%, which forces host_gap to consume the
+    # full step time and collapses kernel_shape / precision_path to zero
+    # (both multiplied by gpu_util).
+    smi.gpu_util_pct = gpu_util_pct
+
+    waste_budget = _waste_budget(
+        kernels=kernels,
+        torch_summary=torch_summary,
+        smi=smi,
+        config=config,
+    )
+
+    attention_kernel_loaded = _detect_attention_kernel(kernels)
+
+    return RunMetrics(
+        steps=steps,
+        tokens_per_sec=torch_summary.tokens_per_sec or 0.0,
+        mfu_pct=torch_summary.mfu_pct or 0.0,
+        hbm_peak_gb=smi.hbm_peak_gb or 0.0,
+        hbm_avg_gb=smi.hbm_avg_gb or 0.0,
+        gpu_util_pct=gpu_util_pct,
+        top_kernels=top_kernels,
+        attention_kernel_loaded=attention_kernel_loaded,
+        waste_budget=waste_budget,
+        warnings=warnings,
+        rocm_version=smi.rocm_version or "unknown",
+        pytorch_version=torch_summary.pytorch_version or "unknown",
+        runner_kind="live",
+    )
+
+
+# ---------------------------------------------------------------------------
+# Internal data carriers
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class _Kernel:
+    name: str
+    duration_ns: int
+    """Kernel duration in nanoseconds."""
+
+    is_collective: bool = False
+    is_gemm: bool = False
+    is_generic_gemm: bool = False
+
+
+@dataclass
+class _TorchSummary:
+    tokens_per_sec: float | None = None
+    mfu_pct: float | None = None
+    pytorch_version: str | None = None
+    step_time_seconds: float | None = None
+    """Average wall-clock seconds per training step (used by waste budget)."""
+
+    host_busy_fraction: float | None = None
+    """Fraction of step time the host (CPU) spent doing non-launch work.
+    Heuristic: `cpu_self_time / step_time` reported by torch.profiler."""
+
+
+@dataclass
+class _SmiSummary:
+    hbm_peak_gb: float | None = None
+    hbm_avg_gb: float | None = None
+    gpu_util_pct: float | None = None
+    rocm_version: str | None = None
+
+
+# ---------------------------------------------------------------------------
+# trace.csv (rocprofv3) → list[_Kernel]
+# ---------------------------------------------------------------------------
+
+
+def _read_kernels(path: Path, warnings: list[str]) -> list[_Kernel]:
+    """Parse rocprofv3 kernel-trace CSV into a list of `_Kernel` records.
+
+    rocprofv3 column names vary slightly by version. We look up by header and
+    accept the common aliases; if the file is missing or unparseable we
+    return an empty list and append a warning so the caller can decide.
+    """
+    if not path.exists():
+        warnings.append(f"profile_parser: kernel trace not found at {path}")
+        return []
+
+    try:
+        with path.open(newline="") as f:
+            reader = csv.DictReader(f)
+            if reader.fieldnames is None:
+                warnings.append(f"profile_parser: empty kernel trace at {path}")
+                return []
+            name_col = _pick_column(reader.fieldnames, ["KernelName", "Kernel_Name", "kernel_name", "Name"])
+            start_col = _pick_column(reader.fieldnames, ["BeginNs", "start_ns", "BeginNS", "Start"])
+            end_col = _pick_column(reader.fieldnames, ["EndNs", "end_ns", "EndNS", "End"])
+            duration_col = _pick_column(reader.fieldnames, ["DurationNs", "duration_ns", "Duration"])
+
+            kernels: list[_Kernel] = []
+            for row in reader:
+                name = (row.get(name_col) or "").strip() if name_col else ""
+                if not name:
+                    continue
+                duration = _row_duration_ns(row, duration_col, start_col, end_col)
+                if duration <= 0:
+                    continue
+                kernels.append(
+                    _Kernel(
+                        name=name,
+                        duration_ns=duration,
+                        is_collective=bool(_RCCL_PATTERN.search(name)),
+                        is_gemm=bool(_GEMM_PATTERN.search(name)),
+                        is_generic_gemm=bool(
+                            _GEMM_PATTERN.search(name) and _GENERIC_GEMM_PATTERN.search(name)
+                        ),
+                    )
+                )
+            return kernels
+    except (OSError, csv.Error) as exc:
+        warnings.append(f"profile_parser: failed to read kernel trace ({exc})")
+        return []
+
+
+def _pick_column(fieldnames: list[str], candidates: list[str]) -> str | None:
+    """Pick the first matching column name, with three fallback tiers:
+        1. Exact match.
+        2. Case-insensitive exact match.
+        3. Substring match (case-insensitive) — every token of any candidate
+           must appear somewhere in the field name. Tolerates the column-name
+           drift between rocprofv3 / amd-smi versions (e.g. `VRAM_USED` vs
+           `vram_used_mb` vs `VRAM USED MB`).
+    """
+    for c in candidates:
+        if c in fieldnames:
+            return c
+    lower = {f.lower(): f for f in fieldnames}
+    for c in candidates:
+        if c.lower() in lower:
+            return lower[c.lower()]
+    # Substring tier: split each candidate on _/space, require all tokens
+    # appear in the (lowercased) field name. Avoids matching too eagerly
+    # by requiring every token of the candidate.
+    for c in candidates:
+        tokens = [t for t in c.lower().replace("_", " ").split() if t]
+        if not tokens:
+            continue
+        for fname in fieldnames:
+            fl = fname.lower()
+            if all(t in fl for t in tokens):
+                return fname
+    return None
+
+
+def _row_duration_ns(
+    row: dict[str, str], duration_col: str | None, start_col: str | None, end_col: str | None
+) -> int:
+    if duration_col and row.get(duration_col):
+        try:
+            return int(float(row[duration_col]))
+        except ValueError:
+            return 0
+    if start_col and end_col and row.get(start_col) and row.get(end_col):
+        try:
+            return int(float(row[end_col])) - int(float(row[start_col]))
+        except ValueError:
+            return 0
+    return 0
+
+
+def _top_kernels(kernels: list[_Kernel], top_n: int) -> list[KernelEntry]:
+    if not kernels:
+        return []
+    total = sum(k.duration_ns for k in kernels) or 1
+    by_name: dict[str, int] = {}
+    for k in kernels:
+        by_name[k.name] = by_name.get(k.name, 0) + k.duration_ns
+    ranked = sorted(by_name.items(), key=lambda kv: kv[1], reverse=True)[:top_n]
+    return [KernelEntry(name=name, pct_time=ns / total * 100.0) for name, ns in ranked]
+
+
+def _detect_attention_kernel(kernels: list[_Kernel]) -> str:
+    for k in kernels:
+        n = k.name.lower()
+        if "flash_attn" in n and "rocm" in n:
+            return "flash_rocm"
+        if "flash" in n and "attn" in n:
+            return "flash"
+        if "scaled_dot_product_attention" in n:
+            return "sdpa"
+    if kernels:
+        return "eager"  # nothing flash-shaped; default conservative label
+    return "unknown"
+
+
+def _gpu_util_from_kernels(kernels: list[_Kernel], torch_summary: _TorchSummary) -> float:
+    """Fallback GPU util when amd-smi is missing.
+
+    util ≈ sum(kernel duration) / total wall-clock step time.
+    """
+    if not kernels or torch_summary.step_time_seconds in (None, 0):
+        return 0.0
+    total_kernel_ns = sum(k.duration_ns for k in kernels)
+    wall_ns = torch_summary.step_time_seconds * 1e9
+    if wall_ns <= 0:
+        return 0.0
+    return min(100.0, total_kernel_ns / wall_ns * 100.0)
+
+
+# ---------------------------------------------------------------------------
+# torch_profile.json (torch.profiler chrome trace) → _TorchSummary
+# ---------------------------------------------------------------------------
+
+
+def _read_torch_profile(path: Path, warnings: list[str]) -> _TorchSummary:
+    """Pull tokens/sec, MFU, and step timing from a torch.profiler artefact.
+
+    The user script (workloads/train_qwen_lora.py in Phase 3) is
+    responsible for embedding `tokens_per_sec`, `mfu_pct`, `pytorch_version`
+    and `step_time_seconds` in the trace as `metadata` events. If those are
+    missing, we estimate `step_time_seconds` from the total trace duration.
+    """
+    summary = _TorchSummary()
+    if not path.exists():
+        warnings.append(f"profile_parser: torch profile not found at {path}")
+        return summary
+    try:
+        data = json.loads(path.read_text())
+    except (OSError, json.JSONDecodeError) as exc:
+        warnings.append(f"profile_parser: failed to read torch profile ({exc})")
+        return summary
+
+    # torch.profiler chrome trace is `{"traceEvents": [...], "metadata": {...}}`
+    metadata = data.get("metadata") if isinstance(data, dict) else None
+    if isinstance(metadata, dict):
+        summary.tokens_per_sec = _coerce_float(metadata.get("tokens_per_sec"))
+        summary.mfu_pct = _coerce_float(metadata.get("mfu_pct"))
+        summary.pytorch_version = metadata.get("pytorch_version") or metadata.get("torch_version")
+        summary.step_time_seconds = _coerce_float(metadata.get("step_time_seconds"))
+        summary.host_busy_fraction = _coerce_float(metadata.get("host_busy_fraction"))
+
+    events = data.get("traceEvents") if isinstance(data, dict) else None
+    if isinstance(events, list):
+        if summary.step_time_seconds is None:
+            summary.step_time_seconds = _step_time_from_events(events)
+        if summary.host_busy_fraction is None:
+            summary.host_busy_fraction = _host_busy_from_events(events)
+
+    return summary
+
+
+def _coerce_float(v: Any) -> float | None:
+    if v is None:
+        return None
+    try:
+        return float(v)
+    except (TypeError, ValueError):
+        return None
+
+
+def _step_time_from_events(events: list[dict]) -> float | None:
+    """Estimate per-step wall-clock seconds from chrome-trace duration events.
+
+    Looks for `name == "ProfilerStep#*"` complete events; falls back to the
+    overall trace span if those aren't present.
+    """
+    durations: list[float] = []
+    overall_start: float | None = None
+    overall_end: float | None = None
+    for ev in events:
+        if not isinstance(ev, dict):
+            continue
+        name = ev.get("name", "")
+        ts = ev.get("ts")
+        dur = ev.get("dur")
+        if isinstance(name, str) and name.startswith("ProfilerStep") and isinstance(dur, (int, float)):
+            durations.append(float(dur) / 1e6)  # us → seconds
+        if isinstance(ts, (int, float)) and isinstance(dur, (int, float)):
+            start = float(ts)
+            end = start + float(dur)
+            overall_start = start if overall_start is None else min(overall_start, start)
+            overall_end = end if overall_end is None else max(overall_end, end)
+    if durations:
+        return sum(durations) / len(durations)
+    if overall_start is not None and overall_end is not None and overall_end > overall_start:
+        return (overall_end - overall_start) / 1e6
+    return None
+
+
+def _host_busy_from_events(events: list[dict]) -> float | None:
+    """Heuristic: cpu_op event time / total event span.
+
+    Used by the data_wait waste bucket to disambiguate "GPU idle because the
+    host is busy preparing the next batch" from "GPU idle because nothing
+    is running anywhere".
+    """
+    cpu_op_us = 0.0
+    span_min: float | None = None
+    span_max: float | None = None
+    for ev in events:
+        if not isinstance(ev, dict):
+            continue
+        cat = ev.get("cat", "")
+        ts = ev.get("ts")
+        dur = ev.get("dur")
+        if not (isinstance(ts, (int, float)) and isinstance(dur, (int, float))):
+            continue
+        ts_f = float(ts)
+        dur_f = float(dur)
+        end = ts_f + dur_f
+        span_min = ts_f if span_min is None else min(span_min, ts_f)
+        span_max = end if span_max is None else max(span_max, end)
+        if isinstance(cat, str) and "cpu_op" in cat.lower():
+            cpu_op_us += dur_f
+    if span_min is None or span_max is None or span_max <= span_min:
+        return None
+    span = span_max - span_min
+    if span <= 0:
+        return None
+    return min(1.0, cpu_op_us / span)
+
+
+# ---------------------------------------------------------------------------
+# amd_smi.csv → _SmiSummary
+# ---------------------------------------------------------------------------
+
+
+def _read_amd_smi(path: Path, warnings: list[str]) -> _SmiSummary:
+    """Aggregate amd-smi polling output into HBM peak/avg + GPU util."""
+    summary = _SmiSummary()
+    if not path.exists():
+        warnings.append(f"profile_parser: amd-smi telemetry not found at {path}")
+        return summary
+
+    try:
+        raw = path.read_text()
+    except OSError as exc:
+        warnings.append(f"profile_parser: failed to read amd-smi telemetry ({exc})")
+        return summary
+
+    csv_text = _strip_amd_smi_preamble(raw)
+    if csv_text is None:
+        warnings.append(f"profile_parser: no parseable header in amd-smi telemetry at {path}")
+        return summary
+
+    try:
+        import io as _io
+
+        reader = csv.DictReader(_io.StringIO(csv_text))
+        if reader.fieldnames is None:
+            warnings.append(f"profile_parser: empty amd-smi telemetry at {path}")
+            return summary
+        hbm_col = _pick_column(
+            reader.fieldnames,
+            [
+                # `amd-smi metric --mem-usage --csv` (ROCm 7.x) emits
+                # "used_vram" in MB. Older `amd-smi monitor --vram-usage`
+                # emits "VRAM_USED" / "vram_used_mb" depending on minor
+                # version.
+                "used_vram",
+                "USED_VRAM",
+                "VRAM_USED_MB",
+                "vram_used_mb",
+                "VRAM_USED",
+                "VRAM_USED_GB",
+                "vram_used",
+                "VRAM Used",
+                # Older rocm 6.x naming
+                "MEM_USED",
+                "mem_used",
+            ],
+        )
+        util_col = _pick_column(
+            reader.fieldnames,
+            [
+                # `amd-smi metric --usage --csv` (ROCm 7.x) emits a single
+                # consolidated "usage" column. Some builds report N/A here
+                # — _coerce_float drops those silently and we fall back to
+                # the kernel-trace gpu_util estimate downstream.
+                "usage",
+                "USAGE",
+                # `amd-smi monitor --gfx` (ROCm 7.x) → "gfx_util"
+                "GFX_UTIL",
+                "gfx_util",
+                "GFX_UTILIZATION",
+                "gfx_utilization",
+                # rocm 6.x and older
+                "GFX_ACTIVITY",
+                "gfx_activity",
+                "GPU_USE",
+                "GFX %",
+                "Util",
+            ],
+        )
+        rocm_col = _pick_column(reader.fieldnames, ["ROCM_VERSION", "rocm_version"])
+
+        hbm_samples: list[float] = []
+        util_samples: list[float] = []
+        for row in reader:
+            if hbm_col:
+                hbm_gb = _hbm_to_gb(row.get(hbm_col), hbm_col)
+                if hbm_gb is not None:
+                    hbm_samples.append(hbm_gb)
+            if util_col:
+                util = _coerce_float(row.get(util_col))
+                if util is not None:
+                    util_samples.append(min(100.0, util))
+            if rocm_col and summary.rocm_version is None:
+                val = (row.get(rocm_col) or "").strip()
+                if val:
+                    summary.rocm_version = val
+        if hbm_samples:
+            summary.hbm_peak_gb = max(hbm_samples)
+            summary.hbm_avg_gb = sum(hbm_samples) / len(hbm_samples)
+        if util_samples:
+            summary.gpu_util_pct = sum(util_samples) / len(util_samples)
+        return summary
+    except csv.Error as exc:
+        warnings.append(f"profile_parser: failed to parse amd-smi telemetry ({exc})")
+        return summary
+
+
+def _strip_amd_smi_preamble(raw: str) -> str | None:
+    """Drop everything before the first real CSV header and dedupe repeated
+    header lines — both noise produced by `amd-smi <subcmd> --watch`.
+
+    --watch prints a "'CTRL' + 'C' to stop watching output:" banner once at
+    the top, then re-emits the CSV header on every iteration. csv.DictReader
+    naively reads the banner as fieldnames and treats every subsequent
+    header as a misshapen data row. Pre-strip both before handing it off.
+
+    Returns a CSV string ready for DictReader, or None if no header line
+    is recognisable.
+    """
+    lines = raw.splitlines()
+    header_idx: int | None = None
+    for i, line in enumerate(lines):
+        if "," not in line:
+            continue
+        lower = line.lower()
+        # Recognised tokens come from the columns amd-smi metric / monitor
+        # actually emit. Match conservatively — banners or other noise can
+        # contain commas too.
+        if any(tok in lower for tok in ("vram", "gfx_", "timestamp,", "gpu_use", "gpu,")):
+            header_idx = i
+            break
+    if header_idx is None:
+        return None
+
+    header = lines[header_idx]
+    data = [
+        line
+        for line in lines[header_idx + 1 :]
+        if line.strip() and line != header
+    ]
+    return header + "\n" + "\n".join(data) + ("\n" if data else "")
+
+
+def _hbm_to_gb(raw: str | None, column_name: str | None = None) -> float | None:
+    """amd-smi sometimes reports VRAM in MB, sometimes in GB.
+
+    First check the column name — `*vram*` / `*_mb` columns are MB-typed
+    in every amd-smi build we've seen; `*_gb` is GB. Without a column-name
+    hint, fall back to a value heuristic. The old "v > 1024 ⇒ MB" heuristic
+    misclassified small idle samples (e.g. 285 MB at GPU idle) as GB and
+    inflated the peak across the run, which then forced memory_headroom to
+    zero downstream.
+    """
+    if not raw:
+        return None
+    try:
+        v = float(str(raw).strip().replace("MB", "").replace("GB", "").replace(",", ""))
+    except ValueError:
+        return None
+    if column_name:
+        lower = column_name.lower()
+        if "_mb" in lower or lower.endswith("mb"):
+            return v / 1024.0
+        if "_gb" in lower or lower.endswith("gb"):
+            return v
+        if "vram" in lower or "mem" in lower:
+            # amd-smi metric / monitor default to MB for the bare
+            # `used_vram` / `mem_used` columns on every ROCm 6.x+ build.
+            return v / 1024.0
+    # No column hint — fall back to value heuristic.
+    if v > 1024.0:
+        return v / 1024.0
+    return v
+
+
+# ---------------------------------------------------------------------------
+# Waste-budget heuristics
+# ---------------------------------------------------------------------------
+
+
+def _waste_budget(
+    *,
+    kernels: list[_Kernel],
+    torch_summary: _TorchSummary,
+    smi: _SmiSummary,
+    config: WorkloadConfig | None,
+) -> WasteBudget:
+    """Decompose step time into the seven WasteBudget buckets (architecture.md §3).
+
+    These are HEURISTICS, not measurements. Each bucket is in seconds-per-step
+    so they can be summed against `step_time_seconds`. If we can't observe a
+    bucket we set it to 0 — `evidence_coverage` in `propose_patch` then
+    discounts confidence accordingly.
+
+    Per-bucket logic:
+
+      data_wait
+        Fraction of step time where GPU util was below 30% AND the host was
+        busy (host_busy_fraction > 0.5). Maps to dataloader / H2D copy stalls.
+
+      precision_path
+        If the user is already on bf16/fp8 we skip this bucket (recovery is 0).
+        Otherwise we estimate from kernel names: time spent in fp16-tagged
+        GEMMs is the recoverable surface; bf16-tagged kernels are not.
+
+      kernel_shape
+        Fraction of total GEMM kernel time spent on kernels whose names
+        match `generic|fallback|naive|reference` — i.e. hipBLASLt/MIOpen
+        couldn't pick a tuned tile size and fell back to a slow path.
+
+      host_gap
+        Time the GPU was idle while the host was NOT busy either — pure
+        launch latency / eager-mode kernel gaps. We approximate as
+        `(1 - gpu_util) * (1 - host_busy)` × step_time.
+
+      comm_excess
+        Sum of all collective-kernel duration (anything matching the rccl
+        pattern). Treated as 100% recoverable elsewhere — the rule decides
+        recovery_fraction.
+
+      memory_headroom
+        `(192 - hbm_peak) / 192 × small_constant`. Only counts the headroom
+        that exceeded a healthy 70% target — running at 60% of HBM costs us
+        roughly 0.07 of step time worth of optimisation surface.
+
+      useful_gpu
+        Whatever step time is left. Will be roughly the busy GPU time minus
+        comm_excess and the kernel-shape penalty.
+    """
+    step_t = torch_summary.step_time_seconds or 0.0
+    gpu_util = (smi.gpu_util_pct or 0.0) / 100.0  # 0..1
+    host_busy = torch_summary.host_busy_fraction or 0.0
+    if step_t <= 0:
+        return WasteBudget()
+
+    # data_wait: GPU under-utilised AND host busy → dataloader bottleneck.
+    if gpu_util < 0.30 and host_busy > 0.5:
+        data_wait = step_t * (1.0 - gpu_util) * host_busy
+    else:
+        data_wait = 0.0
+
+    # host_gap: GPU idle while host idle too → launch latency / kernel gaps.
+    host_gap = step_t * (1.0 - gpu_util) * (1.0 - host_busy)
+
+    # comm_excess: total time in collective kernels (in seconds).
+    comm_excess_ns = sum(k.duration_ns for k in kernels if k.is_collective)
+    comm_excess = comm_excess_ns / 1e9
+
+    # kernel_shape: fraction of GEMM time spent on un-tuned/generic kernels.
+    gemm_total_ns = sum(k.duration_ns for k in kernels if k.is_gemm) or 1
+    generic_gemm_ns = sum(k.duration_ns for k in kernels if k.is_generic_gemm)
+    kernel_shape = (generic_gemm_ns / gemm_total_ns) * step_t * gpu_util  # cap at "real" GPU time
+
+    # precision_path: only meaningful if config is fp16/fp32. Estimate from
+    # kernel name tags. If config is bf16+ we leave as 0 — already optimal.
+    precision_path = 0.0
+    if config is None or config.precision in {"fp16", "fp32"}:
+        fp16_ns = sum(k.duration_ns for k in kernels if _FP16_PATTERN.search(k.name))
+        bf16_ns = sum(k.duration_ns for k in kernels if _BF16_PATTERN.search(k.name))
+        denom = fp16_ns + bf16_ns
+        if denom > 0:
+            # Fraction of compute still on fp16. On MI300X bf16 is faster +
+            # more numerically stable, so this fraction is the recoverable
+            # precision_path surface.
+            precision_path = (fp16_ns / denom) * step_t * gpu_util * 0.10
+
+    # memory_headroom: headroom past the 70% healthy target × small constant.
+    memory_headroom = 0.0
+    hbm_peak = smi.hbm_peak_gb
+    if hbm_peak is not None and hbm_peak > 0:
+        utilisation = hbm_peak / _HBM_TOTAL_GB
+        if utilisation < _HBM_HEALTHY_TARGET:
+            slack = (_HBM_HEALTHY_TARGET - utilisation) / _HBM_HEALTHY_TARGET
+            # Small constant: HBM slack is real but only enables a fraction of
+            # potential gain (you still need a larger batch to use it). 0.05
+            # of step time per "unit of slack" is a conservative anchor.
+            memory_headroom = slack * step_t * 0.05
+
+    # useful_gpu: everything else. Clamp to >= 0.
+    spent = data_wait + host_gap + comm_excess + kernel_shape + precision_path + memory_headroom
+    useful_gpu = max(0.0, step_t - spent)
+
+    return WasteBudget(
+        useful_gpu=useful_gpu,
+        data_wait=data_wait,
+        host_gap=host_gap,
+        comm_excess=comm_excess,
+        memory_headroom=memory_headroom,
+        precision_path=precision_path,
+        kernel_shape=kernel_shape,
+    )

runner/protocol.py

@@ -222,41 +222,6 @@ def _archive_failure(out_dir: Path, proc: subprocess.CompletedProcess) -> Path:
     return dest
 
 
-def _default_runner_timeout_seconds() -> int:
-    """Resolve the LiveRunner subprocess timeout from env, with safe defaults.
-
-    Reads ``GOBLIN_RUNNER_TIMEOUT_SECONDS`` and validates it as a positive
-    int. Anything missing, empty, non-numeric, or non-positive falls back to
-    1800 seconds (30 minutes).
-
-    Why this helper exists: live MI300X audits sometimes overshoot the old
-    600s default — model download on a cold cache, ROCm kernel JIT on the
-    first step, or torch silently running on CPU after a botched pip
-    install. Operators need a knob to extend the budget without editing
-    code; this turns it into a single env var.
-    """
-    raw = os.environ.get("GOBLIN_RUNNER_TIMEOUT_SECONDS", "").strip()
-    if not raw:
-        return 1800
-    try:
-        val = int(raw)
-    except ValueError:
-        _LOG.warning(
-            "LiveRunner: GOBLIN_RUNNER_TIMEOUT_SECONDS=%r is not an int; "
-            "falling back to 1800s.",
-            raw,
-        )
-        return 1800
-    if val <= 0:
-        _LOG.warning(
-            "LiveRunner: GOBLIN_RUNNER_TIMEOUT_SECONDS=%d is not positive; "
-            "falling back to 1800s.",
-            val,
-        )
-        return 1800
-    return val
-
-
 class LiveRunner:
     """Real-MI300X path: shells out to goblin_runner.sh and parses artefacts.
 
@@ -271,26 +236,17 @@ class LiveRunner:
         self,
         runner_script: Path | str = _DEFAULT_RUNNER_SCRIPT,
         user_script: Path | str = _DEFAULT_USER_SCRIPT,
-        timeout_seconds: int | None = None,
+        timeout_seconds: int = 600,
         fake_fallback: FakeRunner | None = None,
     ) -> None:
-        # `timeout_seconds=None` (the default) means "consult
-        # GOBLIN_RUNNER_TIMEOUT_SECONDS, then fall back to 1800s (30 min)".
-        # Explicit ints from callers/tests still win.
-        #
-        # 30 min is generous on a healthy MI300X — a 50-step benchmark of
-        # Qwen2.5-7B LoRA at bs=1/seq=512 finishes in 2–5 min when torch is
-        # correctly using ROCm. The extra headroom absorbs cold model
-        # downloads, kernel JIT on first run, and slow CPU-fallback hiccups
-        # if torch ends up CPU-only. If your workload needs longer, bump
-        # the env var:
-        #     export GOBLIN_RUNNER_TIMEOUT_SECONDS=3600
+        # Default 600s (10 min). Profile runs (10 steps) finish in seconds
+        # on a healthy MI300X; benchmarks (50 steps) in a couple of minutes.
+        # 30 minutes was a leftover from a workload that wasn't honoring
+        # --max_steps and silently trained for hours. With max_steps wired
+        # correctly, 600s is generous.
         self.runner_script = Path(runner_script)
         self.user_script = Path(user_script)
-        self.timeout_seconds = (
-            timeout_seconds if timeout_seconds is not None
-            else _default_runner_timeout_seconds()
-        )
+        self.timeout_seconds = timeout_seconds
         self._fake = fake_fallback or FakeRunner()
 
     # ------------------------------------------------------------------

scripts/auto_tune.py

@@ -0,0 +1,1918 @@
+#!/usr/bin/env python
+"""Iterative auto-tuner for AMD MI300X / ROCm 7.0 workloads.
+
+Three modes, picked with `--mode`:
+
+  hardcoded (default)
+    Walks through a curated list of MI300X-specific tuning changes one
+    at a time. Deterministic, no LLM required — experiments are
+    derived from the rules in kb/rocm_rules.yaml.
+
+  llm
+    On each iteration, asks the LLM backend (qwen-hf via HF_TOKEN, or
+    qwen-vllm via GOBLIN_QWEN_VLLM_URL) for ONE next experiment given
+    the live waste_budget, history, and KB rules. Greedy coordinate
+    descent — accept changes that beat the current best by the
+    improvement threshold, otherwise revert.
+
+  llm-explore
+    On each iteration, asks the LLM for K candidate experiments at
+    once (--candidates-per-iteration, default 3). Benchmarks all K,
+    picks the one with the highest tokens/sec, and accepts only if it
+    beats the current best. Higher GPU cost (~Kx benchmarks per
+    iteration) but better at finding interaction effects that greedy
+    one-at-a-time can miss.
+
+After each change, runs a real benchmark via goblin_runner.sh and keeps
+the change only if tokens/sec improved meaningfully (>1% by default —
+the threshold cuts measurement noise). Stops when N consecutive
+experiments produce no improvement, or when the source of experiments
+is exhausted.
+
+Usage:
+    # hardcoded mode (default):
+    python scripts/auto_tune.py workloads/train_qwen_lora.py --steps 20
+
+    # LLM-driven greedy mode:
+    python scripts/auto_tune.py workloads/train_qwen_lora.py \\
+        --mode llm --steps 20
+
+    # LLM-driven multi-candidate exploration:
+    python scripts/auto_tune.py workloads/train_qwen_lora.py \\
+        --mode llm-explore --candidates-per-iteration 3 --steps 20
+
+Output:
+  - A row-by-row log of each experiment attempted, accepted or rejected
+  - A final summary with cumulative speedup
+  - A pointer to a temp file containing the best workload script for
+    diff-against-baseline inspection
+
+Extending hardcoded mode: add an Experiment to EXPERIMENTS. The
+substitutions field is a list of (regex_pattern, replacement) tuples
+applied with re.subn against the workload source. env_vars are exported
+into the goblin_runner.sh subprocess and persist on every accepted
+iteration.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+from dataclasses import dataclass, field
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+GOBLIN_RUNNER = REPO_ROOT / "runner" / "goblin_runner.sh"
+sys.path.insert(0, str(REPO_ROOT))
+
+# Optional structured-events output. When `--events FILE` is passed, the
+# script appends one JSON object per line at key milestones (baseline,
+# iteration start, candidate done, iteration done, summary). Used by the
+# Streamlit UI to render progress live; CLI users typically don't need it.
+_EVENTS_PATH: Path | None = None
+
+
+def _emit(event: dict) -> None:
+    """Append one NDJSON event to the events file if one was configured."""
+    if _EVENTS_PATH is None:
+        return
+    try:
+        with _EVENTS_PATH.open("a") as f:
+            f.write(json.dumps(event, default=str) + "\n")
+            f.flush()  # so a UI tailing the file sees events promptly
+    except OSError:
+        pass  # never crash the run on an event-write failure
+
+# Default workload template — used when the user passes --model instead
+# of an explicit workload path. We just substitute MODEL_ID and reuse all
+# the other defaults (fp16, batch=4, eager attention, LoRA r=16, …).
+_DEFAULT_WORKLOAD_TEMPLATE = REPO_ROOT / "workloads" / "train_qwen_lora.py"
+
+
+def _generate_workload_from_model(model_id: str, dest: Path) -> Path:
+    """Build a baseline workload by substituting MODEL_ID into the demo
+    template (`workloads/train_qwen_lora.py`). Writes to `dest`, returns
+    the path.
+
+    Caveats:
+    - Uses the demo's LoRA target_modules (`q_proj`, `v_proj`) which work
+      for the major decoder-only LLM families (Qwen, Llama, Mistral,
+      Gemma). MoE / GPT-2-style architectures will need a custom workload.
+    - The template overwrites HF_TOKEN with a redactable fake. Public
+      models load fine; gated models (Llama, etc.) need the user to edit
+      the generated workload or use a custom one.
+    """
+    if not _DEFAULT_WORKLOAD_TEMPLATE.exists():
+        raise SystemExit(
+            f"--model needs the template at {_DEFAULT_WORKLOAD_TEMPLATE}, but it's missing"
+        )
+    template_src = _DEFAULT_WORKLOAD_TEMPLATE.read_text()
+    new_src, n = re.subn(
+        r'MODEL_ID = "[^"]*"',
+        f'MODEL_ID = "{model_id}"',
+        template_src,
+    )
+    if n == 0:
+        raise SystemExit(
+            f"Couldn't find `MODEL_ID = \"...\"` in {_DEFAULT_WORKLOAD_TEMPLATE} "
+            "to substitute. Has the template format changed?"
+        )
+    dest.write_text(new_src)
+    return dest
+
+
+# POSIX env var name: leading letter or underscore, then alnum/underscore.
+# subprocess.run() raises ValueError if any key in the env dict violates
+# this. We validate up-front rather than letting the subprocess crash.
+_VALID_ENV_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+
+
+def _sanitize_env_vars(envs: dict, context: str = "") -> dict[str, str]:
+    """Clean an env_vars dict from the LLM:
+      1. Strip dotted prefixes (`env_vars.X` → `X`) the LLM mimics from the
+         KB transform notation.
+      2. Drop any key that still isn't a valid POSIX env var name. Warns
+         instead of crashing — the LLM occasionally embeds shell syntax
+         (e.g. `'NUMACTL_INTERLEAVE=1'` as a key) which would make
+         subprocess.run raise ValueError.
+    """
+    cleaned: dict[str, str] = {}
+    for k, v in envs.items():
+        key = str(k)
+        if "." in key:
+            stripped = key.rsplit(".", 1)[-1]
+            tag = f" [{context}]" if context else ""
+            print(f"  [warn]{tag} dotted env key {key!r}; using {stripped!r}")
+            key = stripped
+        if not _VALID_ENV_NAME.match(key):
+            tag = f" [{context}]" if context else ""
+            print(
+                f"  [warn]{tag} dropping invalid env var name {key!r} "
+                "(must match [A-Za-z_][A-Za-z0-9_]*)"
+            )
+            continue
+        cleaned[key] = str(v)
+    return cleaned
+
+
+@dataclass
+class Experiment:
+    name: str
+    description: str
+    rationale: str
+    substitutions: list[tuple[str, str]] = field(default_factory=list)
+    env_vars: dict[str, str] = field(default_factory=dict)
+
+
+# Curated for ROCm 7.0 + MI300X (CDNA3, 192 GB HBM3). Ordered roughly by
+# typical impact on Qwen-shaped LoRA fine-tuning workloads. Each
+# experiment stacks on top of any previously accepted ones.
+EXPERIMENTS: list[Experiment] = [
+    Experiment(
+        name="bf16_over_fp16",
+        description="Switch precision from fp16 to bf16",
+        rationale=(
+            "MI300X (CDNA3) prefers bf16: same throughput, larger numeric "
+            "range, no loss-scaler needed. fp16 underutilizes the matrix "
+            "engine on this arch."
+        ),
+        substitutions=[
+            (r"torch_dtype=torch\.float16", "torch_dtype=torch.bfloat16"),
+            (r"\bfp16=True\b", "bf16=True"),
+        ],
+    ),
+    Experiment(
+        name="batch_size_8",
+        description="Increase per_device_train_batch_size 4 → 8",
+        rationale="MI300X has 192 GB HBM; batch=4 leaves it on the floor.",
+        substitutions=[
+            (r"per_device_train_batch_size=4\b", "per_device_train_batch_size=8"),
+        ],
+    ),
+    Experiment(
+        name="batch_size_16",
+        description="Further increase per_device_train_batch_size to 16",
+        rationale="If batch=8 fit and improved, try doubling again.",
+        substitutions=[
+            (r"per_device_train_batch_size=\d+", "per_device_train_batch_size=16"),
+        ],
+    ),
+    Experiment(
+        name="batch_size_32",
+        description="Push per_device_train_batch_size to 32",
+        rationale=(
+            "MI300X has 192 GB HBM3 — batch 16 typically peaks ~130 GB. "
+            "If 16 fit, 32 likely fits too and reduces step overhead per "
+            "token. Reverts cleanly via OOM-as-crash if not."
+        ),
+        substitutions=[
+            (r"per_device_train_batch_size=\d+", "per_device_train_batch_size=32"),
+        ],
+    ),
+    Experiment(
+        name="sdpa_attention",
+        description="Switch attention from eager to SDPA",
+        rationale=(
+            "Eager attention is the slowest path. SDPA dispatches to the "
+            "best available kernel (flash on ROCm 7.x where supported, "
+            "memory-efficient elsewhere)."
+        ),
+        substitutions=[
+            (r'attn_implementation="eager"', 'attn_implementation="sdpa"'),
+        ],
+    ),
+    Experiment(
+        name="dataloader_workers_4",
+        description="Bump dataloader_num_workers 0 → 4",
+        rationale=(
+            "0 workers means the GPU sits idle while the host loads the "
+            "next batch. 4 is a safe value across most CPU configs."
+        ),
+        substitutions=[
+            (r"dataloader_num_workers=0", "dataloader_num_workers=4"),
+            (r"num_workers=0", "num_workers=4"),
+        ],
+    ),
+    Experiment(
+        name="pin_memory",
+        description="Enable dataloader_pin_memory",
+        rationale=(
+            "Pinned host buffers make H2D copies async and overlap with "
+            "the GPU. Worth it once you have >0 dataloader workers."
+        ),
+        substitutions=[
+            (r"dataloader_pin_memory=False", "dataloader_pin_memory=True"),
+            (r"\bpin_memory=False\b", "pin_memory=True"),
+        ],
+    ),
+    Experiment(
+        name="env_hipblaslt",
+        description="Set TORCH_BLAS_PREFER_HIPBLASLT=1",
+        rationale=(
+            "hipBLASLt is significantly faster than rocBLAS for the GEMM "
+            "shapes Qwen produces (LoRA-projected attention)."
+        ),
+        env_vars={"TORCH_BLAS_PREFER_HIPBLASLT": "1"},
+    ),
+    Experiment(
+        name="env_tunable_op",
+        description="Set PYTORCH_TUNABLEOP_ENABLED=1",
+        rationale=(
+            "Enables runtime kernel auto-tuning. Pays a first-run "
+            "warmup cost in exchange for a steady-state win on every "
+            "subsequent step."
+        ),
+        env_vars={"PYTORCH_TUNABLEOP_ENABLED": "1"},
+    ),
+    Experiment(
+        name="env_miopen_find",
+        description="Set MIOPEN_FIND_MODE=3",
+        rationale=(
+            "MIOpen FAST mode picks already-tuned kernels without on-the-"
+            "fly search. Reduces per-step variance."
+        ),
+        env_vars={"MIOPEN_FIND_MODE": "3"},
+    ),
+]
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def apply_substitutions(source: str, subs: list[tuple[str, str]]) -> str | None:
+    """Apply each (pattern, replacement) in order. Returns the new source,
+    or None if any pattern matched zero times (already applied or N/A for
+    this workload)."""
+    out = source
+    for pattern, replacement in subs:
+        new, n = re.subn(pattern, replacement, out)
+        if n == 0:
+            return None
+        out = new
+    return out
+
+
+def benchmark(
+    workload_path: Path,
+    steps: int,
+    env_overrides: dict[str, str],
+    timeout: int = 600,
+) -> dict | None:
+    """Run goblin_runner.sh on the workload, return parsed RunMetrics dict
+    or None on failure."""
+    with tempfile.TemporaryDirectory(prefix="auto_tune_run_") as out_dir_str:
+        out_dir = Path(out_dir_str)
+        env = os.environ.copy()
+        env["USER_SCRIPT"] = str(workload_path)
+        env["OUT_DIR"] = str(out_dir)
+        env["STEPS"] = str(steps)
+        # Candidate workload lives in /tmp, so its self-bootstrap line
+        # `sys.path.insert(0, dirname(dirname(__file__)))` resolves to /tmp
+        # — which has no `workloads/` package. Inject the real repo root via
+        # PYTHONPATH so `from workloads._runtime import ...` succeeds.
+        existing_pp = env.get("PYTHONPATH", "")
+        env["PYTHONPATH"] = (
+            str(REPO_ROOT) + (os.pathsep + existing_pp if existing_pp else "")
+        )
+        env.update(env_overrides)
+
+        try:
+            proc = subprocess.run(
+                [str(GOBLIN_RUNNER)],
+                env=env,
+                capture_output=True,
+                text=True,
+                timeout=timeout,
+            )
+        except subprocess.TimeoutExpired:
+            print(f"  TIMEOUT after {timeout}s")
+            return None
+        except ValueError as exc:
+            # subprocess.run validates env var names and raises ValueError
+            # for malformed keys (e.g. names containing '=' or spaces). The
+            # LLM has occasionally emitted those; we sanitize earlier but
+            # this is the last-resort backstop so a single bad candidate
+            # doesn't crash the whole tuning run.
+            print(f"  REJECTED — illegal env var name(s): {exc}")
+            print(f"  env keys offered: {list(env_overrides.keys())}")
+            return None
+        except OSError as exc:
+            print(f"  REJECTED — could not spawn goblin_runner.sh: {exc}")
+            return None
+
+        if proc.returncode != 0:
+            print(f"  goblin_runner.sh failed (exit {proc.returncode})")
+            tail = (proc.stderr or "").strip().splitlines()[-8:]
+            for line in tail:
+                print(f"    | {line}")
+            return None
+
+        try:
+            from runner import profile_parser
+
+            metrics = profile_parser.parse(out_dir, steps=steps)
+            return metrics.model_dump()
+        except Exception as exc:  # parser is defensive but be safe
+            print(f"  profile_parser raised: {type(exc).__name__}: {exc}")
+            return None
+
+
+def _delta_pct(new: float, baseline: float) -> float:
+    if baseline <= 0:
+        return 0.0
+    return (new - baseline) / baseline * 100.0
+
+
+# ---------------------------------------------------------------------------
+# LLM-driven experiment generator
+# ---------------------------------------------------------------------------
+
+
+_LLM_SYSTEM_PROMPT = """\
+You are an expert at tuning AMD MI300X (ROCm 7.0, CDNA3 arch, 192 GB
+HBM3) training workloads. The user is iteratively benchmarking changes
+to a transformers/peft fine-tuning script. On each turn you suggest ONE
+specific parameter change to try next, targeting the largest non-useful
+waste bucket in the most recent benchmark.
+
+Your output MUST be a single JSON object with this exact shape (no
+prose, no markdown fences, just the object):
+
+{
+  "name": "short_snake_case_name",
+  "rationale": "1-3 sentences on why this change addresses the worst waste bucket",
+  "substitutions": [["regex_pattern", "replacement"]],
+  "env_vars": {"VAR_NAME": "value"}
+}
+
+CRITICAL output rules — read carefully:
+
+1. env_vars keys are LITERAL POSIX shell environment variable names.
+   They MUST match the regex [A-Za-z_][A-Za-z0-9_]* — letters, digits,
+   underscores only, starting with a letter or underscore.
+   - NEVER prefix them with "env_vars." or any other dotted path.
+   - NEVER include "=" or shell syntax in the key — env var names are
+     identifiers, NOT assignments and NOT commands.
+   - If you want to invoke a command-line tool like `numactl` or
+     `taskset`, that CANNOT be expressed as an env_var. Don't try.
+     Either propose a `substitutions` change to the script, or skip.
+   Wrong:  {"env_vars.MIOPEN_FIND_MODE": "3"}
+   Wrong:  {"NUMACTL_INTERLEAVE=1": "numactl --interleave=all"}
+   Wrong:  {"export FOO": "bar"}
+   Right:  {"MIOPEN_FIND_MODE": "3"}
+   Right:  {"TORCH_BLAS_PREFER_HIPBLASLT": "1"}
+
+2. substitutions are (regex_pattern, replacement) pairs applied with
+   re.subn against the current workload source. Patterns must match at
+   least one occurrence in the source — if zero matches, the experiment
+   is auto-skipped (counted as no improvement).
+
+3. When the previous change for a parameter improved tokens/sec, push
+   that parameter further in the same direction next time. E.g. if
+   batch_size 4 → 8 won, try 8 → 16. If 16 won and HBM is still under
+   ~150 GB, try 32. Don't be timid — MI300X has 192 GB HBM3.
+
+4. Don't repeat any (name OR substitution OR env_var combo) from
+   history. If a change was rejected, don't propose the same numerical
+   value again — try a different one.
+
+5. If you cannot think of a productive next change, output:
+     {"name": "STOP", "rationale": "<why>", "substitutions": [], "env_vars": {}}
+
+CONCRETE OUTPUT EXAMPLES — match this shape exactly:
+
+Switch fp16 → bf16 (precision_path bucket):
+  {"name": "bf16_over_fp16",
+   "rationale": "MI300X CDNA3 matrix cores prefer bf16: same throughput, larger numeric range, no loss-scaler.",
+   "substitutions": [["fp16=True", "bf16=True"], ["torch_dtype=torch\\\\.float16", "torch_dtype=torch.bfloat16"]],
+   "env_vars": {}}
+
+Increase batch size to 16 (memory_headroom bucket):
+  {"name": "batch_size_16",
+   "rationale": "Current HBM peak is well under 192 GB; bigger batch saturates the GPU.",
+   "substitutions": [["per_device_train_batch_size=\\\\d+", "per_device_train_batch_size=16"]],
+   "env_vars": {}}
+
+Switch attention to SDPA (kernel_shape bucket):
+  {"name": "sdpa_attention",
+   "rationale": "Eager attention is the slowest path; SDPA dispatches to a tuned kernel.",
+   "substitutions": [["attn_implementation=\\"eager\\"", "attn_implementation=\\"sdpa\\""]],
+   "env_vars": {}}
+
+Bump dataloader workers (data_wait bucket):
+  {"name": "dataloader_workers_4",
+   "rationale": "0 workers starves the GPU between batches.",
+   "substitutions": [["dataloader_num_workers=0", "dataloader_num_workers=4"]],
+   "env_vars": {}}
+
+Set MIOpen FAST mode (kernel_shape bucket, env-only):
+  {"name": "miopen_find_fast",
+   "rationale": "FAST mode picks already-tuned kernels without on-the-fly search.",
+   "substitutions": [],
+   "env_vars": {"MIOPEN_FIND_MODE": "3"}}
+
+Prefer hipBLASLt (kernel_shape bucket, env-only):
+  {"name": "prefer_hipblaslt",
+   "rationale": "hipBLASLt is faster than rocBLAS for Qwen GEMM shapes on MI300X.",
+   "substitutions": [],
+   "env_vars": {"TORCH_BLAS_PREFER_HIPBLASLT": "1"}}
+"""
+
+
+_LLM_USER_TEMPLATE = """\
+Hardware facts (use these — do not contradict):
+- AMD MI300X, CDNA3 architecture, 192 GB HBM3
+- bf16 throughput on CDNA3 ≈ same as fp16, > fp32 (matrix engine is fp16/bf16/fp8 native)
+- fp32 is the SLOWEST option on this arch — never suggest it as an improvement
+
+Known incompatibilities for THIS workload (peft + LoRA on transformers Trainer):
+{incompatibilities}
+
+KB rules (one-liner per rule, for grounding):
+{kb_summary}
+
+Current accepted workload state — these are the literal values in the
+script after every change accepted so far. The next change you propose
+should mutate one of these (or set an env var). DO NOT propose a value
+that's already present here.
+{tunables}
+
+Latest benchmark (this is the result of the most recent ACCEPTED state):
+- tokens_per_sec: {tps:.1f}
+- mfu_pct:        {mfu:.2f}   (% of MI300X dense bf16 peak; healthy LoRA ranges 30-50%)
+- gpu_util_pct:   {util:.1f}
+- hbm_peak_gb:    {hbm:.2f}
+- waste_budget (seconds/step):
+{waste_lines}
+
+Sorted recoverable waste (largest first — go after these):
+{recoverable_sorted}
+
+History of changes already tried this run (newest first; outcomes are
+"accepted" / "rejected" / "crashed" / "skipped"):
+{history_lines}
+
+If the latest entry is "crashed", the change you propose next must be
+STRUCTURALLY different (different parameter, not just a different value
+of the same one).
+
+Suggest ONE next change targeting the largest recoverable bucket. JSON only.
+"""
+
+
+# Workload-specific incompatibilities the LLM otherwise wastes iterations on.
+# Keep this list short and concrete — it goes into every prompt.
+_KNOWN_INCOMPATIBILITIES = [
+    "gradient_checkpointing=True requires `model.enable_input_require_grads()`"
+    " before peft wrapping for LoRA models. Setting it via a single substitution"
+    " WILL CRASH the workload. Don't propose it.",
+    "bitsandbytes-based optimizers (`adamw_8bit`, `paged_adamw_8bit`) and"
+    " `load_in_8bit=True` are NOT supported on ROCm 7.x. Don't propose them.",
+    "torch_compile=True with peft/LoRA on ROCm 7.x triggers compile-time"
+    " errors with the current PyTorch nightly (2.9.x). Don't propose it"
+    " unless you have specific evidence it works on this version.",
+    "flash_attention_2 may not be installed (try `attn_implementation=\"sdpa\"`"
+    " before `\"flash_attention_2\"`).",
+    "persistent_workers=True requires num_workers > 0. PyTorch raises"
+    " `ValueError: persistent_workers option needs num_workers > 0` if you"
+    " enable it while num_workers=0. If the current workload has"
+    " dataloader_num_workers=0, do NOT propose persistent_workers=True"
+    " alone — pair it with `dataloader_num_workers=4` (or higher) in the"
+    " SAME experiment via two substitutions, or wait until a previous"
+    " experiment has bumped num_workers above 0.",
+    "dataloader_prefetch_factor only works when num_workers > 0 (same"
+    " constraint as persistent_workers). Same rule: bump num_workers in"
+    " the same experiment, or skip.",
+]
+
+
+def _kb_summary(rules_yaml_path: Path, max_chars: int = 6000) -> str:
+    """Return a compact one-line-per-rule summary of kb/rocm_rules.yaml.
+
+    Notably we DO NOT show the raw `transform` field — earlier versions
+    did and the LLM ended up copying its dotted-path notation literally
+    (`env_vars.MIOPEN_FIND_MODE` as the env var name, not as a dict
+    accessor). The system prompt's CONCRETE EXAMPLES section is the
+    canonical source of truth for output shape; this summary just
+    grounds the LLM's reasoning in the catalog of known issues.
+    """
+    if not rules_yaml_path.exists():
+        return "(KB rules file not found)"
+    try:
+        import yaml
+
+        rules = yaml.safe_load(rules_yaml_path.read_text()) or []
+    except Exception as exc:
+        return f"(failed to parse KB: {exc})"
+
+    lines = []
+    for r in rules:
+        if not isinstance(r, dict):
+            continue
+        rid = r.get("id", "?")
+        bucket = r.get("targets_bucket", "?")
+        sym = (r.get("symptom") or "").strip().replace("\n", " ")
+        if len(sym) > 110:
+            sym = sym[:107] + "..."
+        lines.append(f"- {rid:55s} [{bucket}]  {sym}")
+    text = "\n".join(lines)
+    if len(text) > max_chars:
+        text = text[:max_chars] + "\n... (truncated)"
+    return text
+
+
+# Map of (substring-in-source) → (parameter description, example regex
+# pattern, example replacement template). Each entry is a hint shown to
+# the LLM so it has a concrete target to point its substitutions at —
+# instead of guessing what the workload's literal config text looks like.
+_TUNABLE_HINTS: list[tuple[str, str, str, str]] = [
+    # (token to detect, description, regex_for_substitution, replacement_template)
+    ("torch_dtype=torch.float16",
+     "model precision (matches `torch_dtype=torch.float16`)",
+     r"torch_dtype=torch\.float16",
+     "torch_dtype=torch.bfloat16"),
+    ("torch_dtype=torch.bfloat16",
+     "model precision (already bf16)",
+     r"torch_dtype=torch\.bfloat16",
+     "torch_dtype=torch.float16"),
+    ("fp16=True",
+     "TrainingArguments fp16 (matches `fp16=True`)",
+     r"\bfp16=True\b",
+     "bf16=True"),
+    ("bf16=True",
+     "TrainingArguments bf16 (already bf16)",
+     r"\bbf16=True\b",
+     "fp16=True"),
+    ("attn_implementation=\"eager\"",
+     "attention impl (matches `attn_implementation=\"eager\"`)",
+     r'attn_implementation="eager"',
+     'attn_implementation="sdpa"'),
+    ("attn_implementation=\"sdpa\"",
+     "attention impl (currently sdpa; could try flash_attention_2)",
+     r'attn_implementation="sdpa"',
+     'attn_implementation="flash_attention_2"'),
+    ("per_device_train_batch_size=",
+     "per-device batch size (matches `per_device_train_batch_size=<N>`)",
+     r"per_device_train_batch_size=\d+",
+     "per_device_train_batch_size=<NEW_VALUE>"),
+    ("dataloader_num_workers=",
+     "dataloader workers (matches `dataloader_num_workers=<N>`)",
+     r"dataloader_num_workers=\d+",
+     "dataloader_num_workers=<NEW_VALUE>"),
+    ("dataloader_pin_memory=",
+     "dataloader pin_memory (matches `dataloader_pin_memory=<bool>`)",
+     r"dataloader_pin_memory=(True|False)",
+     "dataloader_pin_memory=True"),
+    ("gradient_checkpointing=",
+     "gradient checkpointing toggle",
+     r"gradient_checkpointing=(True|False)",
+     "gradient_checkpointing=True"),
+    ("torch_compile=",
+     "torch.compile toggle (use cautiously on ROCm 7.x)",
+     r"torch_compile=(True|False)",
+     "torch_compile=True"),
+    ("optim=\"adamw_torch\"",
+     "optimizer choice (currently adamw_torch)",
+     r'optim="adamw_torch"',
+     'optim="adamw_torch_fused"'),
+]
+
+
+def _tunables_summary(source: str) -> str:
+    """Detect which tunable parameters are present in the workload source
+    and surface their current literal values + ready-to-use regex patterns
+    so the LLM has concrete substitution targets.
+
+    Skips comment lines when reporting the "current" value — many workloads
+    document expected findings in a top-of-file comment block, and we want
+    the LLM to see the live config line, not the doc string.
+    """
+    lines: list[str] = []
+    source_lines = source.splitlines()
+    for token, desc, pattern, replacement in _TUNABLE_HINTS:
+        live_line: str | None = None
+        for raw in source_lines:
+            stripped = raw.lstrip()
+            if stripped.startswith("#"):
+                continue
+            if token in raw:
+                live_line = raw.strip()
+                break
+        if live_line is None:
+            continue
+        lines.append(
+            f"  • {desc}\n"
+            f"    current: {live_line}\n"
+            f"    pattern: {pattern!r}    replacement template: {replacement!r}"
+        )
+    if not lines:
+        return "  (no recognized tunables — substitutions will need to match other text)"
+    return "\n".join(lines)
+
+
+def _recoverable_sorted(waste: dict) -> str:
+    """List the non-useful_gpu buckets sorted by size, so the LLM can
+    explicitly target the biggest one first."""
+    if not waste:
+        return "  (no waste_budget available)"
+    items = [
+        (name, value)
+        for name, value in waste.items()
+        if name != "useful_gpu" and isinstance(value, (int, float))
+    ]
+    items.sort(key=lambda kv: kv[1], reverse=True)
+    if not items:
+        return "  (no recoverable buckets)"
+    return "\n".join(f"  {i + 1}. {name:18s} = {value:.4f}" for i, (name, value) in enumerate(items))
+
+
+def _config_snippet(source: str, max_lines: int = 80) -> str:
+    """Return the lines around `TrainingArguments(` and `from_pretrained(` so
+    the LLM sees the actual config it's modifying without us shipping the
+    whole script. Gives ~max_lines of context.
+    """
+    lines = source.splitlines()
+    keep: list[tuple[int, str]] = []
+    for i, line in enumerate(lines):
+        lower = line.lower()
+        if any(
+            tok in lower
+            for tok in (
+                "trainingarguments(",
+                "from_pretrained(",
+                "loraconfig(",
+                "dataloader(",
+                "torch_dtype",
+                "attn_implementation",
+                "fp16=",
+                "bf16=",
+                "per_device_train_batch_size",
+                "dataloader_num_workers",
+                "dataloader_pin_memory",
+                "gradient_checkpointing",
+                "torch_compile",
+                "optim=",
+            )
+        ):
+            keep.append((i, line))
+    if not keep:
+        return source[:2000]
+    # Coalesce nearby line indices into windows for readability
+    windows: list[list[str]] = []
+    last_idx = -10
+    cur: list[str] = []
+    for i, line in keep:
+        if i - last_idx > 3:
+            if cur:
+                windows.append(cur)
+            cur = []
+        cur.append(f"{i + 1:4d}: {line}")
+        last_idx = i
+    if cur:
+        windows.append(cur)
+    out = "\n\n".join("\n".join(w) for w in windows)
+    if out.count("\n") > max_lines:
+        out_lines = out.splitlines()[:max_lines]
+        out = "\n".join(out_lines) + "\n... (truncated)"
+    return out
+
+
+def _format_history(history: list[dict]) -> str:
+    if not history:
+        return "(none yet — this is the first iteration)"
+    lines = []
+    for h in reversed(history[-12:]):  # last 12 newest-first
+        outcome = h.get("outcome", "?")
+        delta = h.get("delta_pct")
+        delta_s = f"{delta:+.2f}%" if delta is not None else "n/a"
+        subs = h.get("substitutions") or []
+        envs = h.get("env_vars") or {}
+        change_repr = f"subs={subs} env={envs}"
+        lines.append(f"- {h['name']:25s} {outcome:9s} Δ {delta_s:8s}  {change_repr}")
+    return "\n".join(lines)
+
+
+def _format_waste(waste: dict) -> str:
+    keys = (
+        "useful_gpu",
+        "data_wait",
+        "host_gap",
+        "comm_excess",
+        "memory_headroom",
+        "precision_path",
+        "kernel_shape",
+    )
+    return "\n".join(f"    {k:18s} = {waste.get(k, 0.0):.4f}" for k in keys)
+
+
+def _build_llm_backend(system_prompt: str = _LLM_SYSTEM_PROMPT, max_tokens: int = 1024):
+    """Construct the same backend the agent loop uses. Surfaces a clear
+    message if neither HF_TOKEN nor a vLLM URL is configured."""
+    has_hf = bool(os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACEHUB_API_TOKEN"))
+    has_vllm = bool(os.environ.get("GOBLIN_QWEN_VLLM_URL"))
+    backend_kind = os.environ.get("GOBLIN_AGENT_BACKEND", "qwen-hf").lower()
+    if backend_kind in ("qwen-hf", "qwen", "hf", "") and not has_hf:
+        raise SystemExit(
+            "LLM mode requires HF_TOKEN (qwen-hf backend) or "
+            "GOBLIN_AGENT_BACKEND=qwen-vllm + GOBLIN_QWEN_VLLM_URL."
+        )
+    if backend_kind in ("qwen-vllm", "qwen_vllm", "vllm", "local") and not has_vllm:
+        raise SystemExit(
+            "LLM mode with qwen-vllm backend requires GOBLIN_QWEN_VLLM_URL."
+        )
+    from agent.backends import make_backend
+
+    return make_backend(system_prompt=system_prompt, max_tokens=max_tokens)
+
+
+async def _ask_llm_for_experiment(
+    backend,
+    *,
+    kb_summary: str,
+    source: str,
+    metrics: dict,
+    history: list[dict],
+) -> Experiment | None:
+    """One LLM turn → one Experiment (or None for STOP / parse failure)."""
+    waste = metrics.get("waste_budget") or {}
+    prompt = _LLM_USER_TEMPLATE.format(
+        incompatibilities="\n".join(f"- {line}" for line in _KNOWN_INCOMPATIBILITIES),
+        kb_summary=kb_summary,
+        tunables=_tunables_summary(source),
+        tps=metrics.get("tokens_per_sec", 0.0),
+        mfu=metrics.get("mfu_pct", 0.0),
+        util=metrics.get("gpu_util_pct", 0.0),
+        hbm=metrics.get("hbm_peak_gb", 0.0),
+        waste_lines=_format_waste(waste),
+        recoverable_sorted=_recoverable_sorted(waste),
+        history_lines=_format_history(history),
+    )
+    backend.add_user_message(prompt)
+    turn = await backend.next_turn(tool_schemas=[])
+    raw = " ".join(turn.text_blocks).strip()
+
+    obj = _extract_json_object(raw)
+    if obj is None:
+        print(f"  LLM response was not parseable JSON. Raw: {raw[:300]!r}")
+        return None
+
+    name = (obj.get("name") or "").strip()
+    if not name or name.upper() == "STOP":
+        print(f"  LLM signaled STOP: {obj.get('rationale', '(no rationale)')}")
+        return None
+
+    subs_raw = obj.get("substitutions") or []
+    envs = obj.get("env_vars") or {}
+    if not subs_raw and not envs:
+        print(f"  LLM returned an empty experiment ({name}); skipping")
+        return None
+
+    subs: list[tuple[str, str]] = []
+    for entry in subs_raw:
+        if isinstance(entry, list) and len(entry) == 2:
+            subs.append((str(entry[0]), str(entry[1])))
+        elif isinstance(entry, dict) and "pattern" in entry and "replacement" in entry:
+            subs.append((str(entry["pattern"]), str(entry["replacement"])))
+
+    cleaned_envs = _sanitize_env_vars(envs, context=name)
+    if not subs and not cleaned_envs:
+        # Everything got dropped during sanitization (bad env names + no
+        # valid substitutions). Treat as a no-op rather than benchmarking
+        # an unchanged workload.
+        print(f"  LLM experiment {name!r} had nothing valid after sanitization; skipping")
+        return None
+
+    return Experiment(
+        name=name,
+        description=obj.get("description") or name,
+        rationale=str(obj.get("rationale") or ""),
+        substitutions=subs,
+        env_vars=cleaned_envs,
+    )
+
+
+# ---------------------------------------------------------------------------
+# llm-explore mode: ask for K candidates per iteration
+# ---------------------------------------------------------------------------
+
+
+_LLM_EXPLORE_SYSTEM_PROMPT = """\
+You are an expert at tuning AMD MI300X (ROCm 7.0, CDNA3 arch, 192 GB
+HBM3) training workloads. The user is running a multi-candidate
+exploration: on every iteration you suggest K STRUCTURALLY-DIFFERENT
+candidate changes, the user benchmarks all of them, and the best one
+is accepted (if it beats the current best by the threshold).
+
+Your output MUST be a JSON ARRAY of K objects, no prose, no markdown
+fences, just the array:
+
+[
+  {"name": "...", "rationale": "...", "substitutions": [["regex", "repl"]], "env_vars": {"VAR": "value"}},
+  {"name": "...", "rationale": "...", "substitutions": [["regex", "repl"]], "env_vars": {"VAR": "value"}},
+  {"name": "...", "rationale": "...", "substitutions": [["regex", "repl"]], "env_vars": {"VAR": "value"}}
+]
+
+CRITICAL output rules:
+
+1. Each candidate must target a DIFFERENT waste bucket or parameter
+   category than the others. Diversity beats redundancy — don't propose
+   three batch-size bumps; propose one batch bump, one env var, one
+   precision/attention/dataloader change.
+
+2. env_vars keys are LITERAL POSIX shell environment variable names —
+   they MUST match the regex [A-Za-z_][A-Za-z0-9_]*. NEVER prefix them
+   with "env_vars." or any other dotted path. NEVER include "=" or
+   shell syntax in the key. If you want to invoke a CLI tool like
+   `numactl`, that's NOT an env var — skip the candidate entirely.
+   Wrong:  {"env_vars.MIOPEN_FIND_MODE": "3"}
+   Wrong:  {"NUMACTL_INTERLEAVE=1": "numactl --interleave=all"}
+   Right:  {"MIOPEN_FIND_MODE": "3"}
+
+3. substitutions are (regex_pattern, replacement) pairs applied with
+   re.subn. Patterns must match at least one occurrence — if zero
+   matches, that candidate is skipped.
+
+4. NEVER propose a (substitutions, env_vars) combination that already
+   appears in history with outcome rejected/crashed. Diversify within
+   the array AND across the run.
+
+5. If you genuinely cannot find K productive candidates, output fewer
+   (e.g. 2 if K=3). The user will benchmark whatever you provide. If
+   you have zero productive candidates, output:
+     [{"name": "STOP", "rationale": "<why>", "substitutions": [], "env_vars": {}}]
+
+CONCRETE OUTPUT EXAMPLES (for K=3):
+
+[
+  {"name": "bf16_over_fp16",
+   "rationale": "Largest recoverable bucket is precision_path; CDNA3 prefers bf16.",
+   "substitutions": [["fp16=True", "bf16=True"], ["torch_dtype=torch\\\\.float16", "torch_dtype=torch.bfloat16"]],
+   "env_vars": {}},
+  {"name": "batch_size_16",
+   "rationale": "HBM peak well under 192 GB; bigger batch saturates the GPU.",
+   "substitutions": [["per_device_train_batch_size=\\\\d+", "per_device_train_batch_size=16"]],
+   "env_vars": {}},
+  {"name": "prefer_hipblaslt",
+   "rationale": "hipBLASLt outperforms rocBLAS on Qwen GEMM shapes.",
+   "substitutions": [],
+   "env_vars": {"TORCH_BLAS_PREFER_HIPBLASLT": "1"}}
+]
+"""
+
+
+_LLM_EXPLORE_USER_TEMPLATE = """\
+Hardware facts (use these — do not contradict):
+- AMD MI300X, CDNA3 architecture, 192 GB HBM3
+- bf16 throughput on CDNA3 ≈ same as fp16, > fp32 (matrix engine is fp16/bf16/fp8 native)
+- fp32 is the SLOWEST option on this arch — never suggest it as an improvement
+
+Known incompatibilities for THIS workload (peft + LoRA on transformers Trainer):
+{incompatibilities}
+
+KB rules (one-liner per rule, for grounding):
+{kb_summary}
+
+Current accepted workload state — the literal values in the script
+after every change accepted so far. Each candidate you propose should
+mutate one of these (or set an env var). DO NOT propose a value that's
+already present here.
+{tunables}
+
+Latest benchmark (this is the result of the most recent ACCEPTED state):
+- tokens_per_sec: {tps:.1f}
+- mfu_pct:        {mfu:.2f}   (% of MI300X dense bf16 peak; healthy LoRA ranges 30-50%)
+- gpu_util_pct:   {util:.1f}
+- hbm_peak_gb:    {hbm:.2f}
+- waste_budget (seconds/step):
+{waste_lines}
+
+Sorted recoverable waste (largest first — go after these):
+{recoverable_sorted}
+
+Previously rejected (full fingerprint — DO NOT repropose any of these):
+{rejected_fingerprints}
+
+History of changes already tried this run (newest first; outcomes are
+"accepted" / "rejected" / "crashed" / "skipped"):
+{history_lines}
+
+Suggest {num_candidates} STRUCTURALLY-DIFFERENT candidate changes.
+Each must target a different waste bucket or parameter category. JSON
+array only.
+"""
+
+
+async def _ask_llm_for_experiments(
+    backend,
+    *,
+    kb_summary: str,
+    source: str,
+    metrics: dict,
+    history: list[dict],
+    num_candidates: int,
+) -> list[Experiment]:
+    """One LLM turn → up to `num_candidates` Experiments.
+
+    Returns an empty list on parse failure or STOP signal.
+    """
+    waste = metrics.get("waste_budget") or {}
+    prompt = _LLM_EXPLORE_USER_TEMPLATE.format(
+        num_candidates=num_candidates,
+        incompatibilities="\n".join(f"- {line}" for line in _KNOWN_INCOMPATIBILITIES),
+        kb_summary=kb_summary,
+        tunables=_tunables_summary(source),
+        tps=metrics.get("tokens_per_sec", 0.0),
+        mfu=metrics.get("mfu_pct", 0.0),
+        util=metrics.get("gpu_util_pct", 0.0),
+        hbm=metrics.get("hbm_peak_gb", 0.0),
+        waste_lines=_format_waste(waste),
+        recoverable_sorted=_recoverable_sorted(waste),
+        rejected_fingerprints=_format_rejected_fingerprints(history),
+        history_lines=_format_history(history),
+    )
+    backend.add_user_message(prompt)
+    turn = await backend.next_turn(tool_schemas=[])
+    raw = " ".join(turn.text_blocks).strip()
+
+    arr = _extract_json_array(raw)
+    if not arr:
+        print(f"  LLM response was not parseable JSON array. Raw: {raw[:300]!r}")
+        return []
+
+    experiments: list[Experiment] = []
+    for obj in arr:
+        if not isinstance(obj, dict):
+            continue
+        name = (obj.get("name") or "").strip()
+        if not name:
+            continue
+        if name.upper() == "STOP":
+            print(f"  LLM signaled STOP: {obj.get('rationale', '(no rationale)')}")
+            return []
+        subs_raw = obj.get("substitutions") or []
+        envs_raw = obj.get("env_vars") or {}
+        if not subs_raw and not envs_raw:
+            continue
+        subs = []
+        for entry in subs_raw:
+            if isinstance(entry, list) and len(entry) == 2:
+                subs.append((str(entry[0]), str(entry[1])))
+            elif isinstance(entry, dict) and "pattern" in entry and "replacement" in entry:
+                subs.append((str(entry["pattern"]), str(entry["replacement"])))
+        cleaned_envs = _sanitize_env_vars(envs_raw, context=name)
+        if not subs and not cleaned_envs:
+            print(f"  candidate {name!r} had nothing valid after sanitization; dropping")
+            continue
+        experiments.append(
+            Experiment(
+                name=name,
+                description=obj.get("description") or name,
+                rationale=str(obj.get("rationale") or ""),
+                substitutions=subs,
+                env_vars=cleaned_envs,
+            )
+        )
+    return experiments
+
+
+def _extract_json_array(text: str) -> list | None:
+    """Pull the first JSON array out of an LLM response, tolerating
+    markdown fences and leading prose. Returns None if nothing parseable."""
+    if not text:
+        return None
+    fence_match = re.search(r"```(?:json)?\s*(\[.*?\])\s*```", text, re.DOTALL)
+    if fence_match:
+        try:
+            obj = json.loads(fence_match.group(1))
+            if isinstance(obj, list):
+                return obj
+        except json.JSONDecodeError:
+            pass
+    depth = 0
+    start = -1
+    for i, ch in enumerate(text):
+        if ch == "[":
+            if depth == 0:
+                start = i
+            depth += 1
+        elif ch == "]":
+            depth -= 1
+            if depth == 0 and start >= 0:
+                blob = text[start : i + 1]
+                try:
+                    obj = json.loads(blob)
+                    if isinstance(obj, list):
+                        return obj
+                except json.JSONDecodeError:
+                    start = -1
+                    continue
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Dedup + history utilities (used by all LLM modes)
+# ---------------------------------------------------------------------------
+
+
+def _experiment_fingerprint(exp: Experiment) -> tuple:
+    """Hashable identity for an experiment — substitutions + env_vars,
+    NOT name (the LLM tends to give the same change different names)."""
+    subs = tuple(sorted(tuple(s) for s in exp.substitutions))
+    envs = tuple(sorted(exp.env_vars.items()))
+    return (subs, envs)
+
+
+def _build_merged_experiment(
+    exps: list[Experiment], base_source: str
+) -> tuple[Experiment | None, str]:
+    """Try to combine 2+ experiments into one. The merged experiment
+    applies all of their substitutions in sequence and unions their
+    env_vars. Returns (merged, "") on success, (None, reason) when the
+    merge is structurally unsafe — caller should fall back to using just
+    the individual winner.
+
+    Conflict detection:
+      - A later substitution's pattern must still match after earlier
+        substitutions have been applied (zero matches → conflict, e.g.
+        cand A rewrote `fp16=True` and cand B was also targeting it).
+      - Env var keys with conflicting values (same name, different value)
+        → conflict.
+      - Bad regex anywhere → conflict.
+    """
+    if len(exps) < 2:
+        return None, "need at least 2 experiments"
+
+    merged_subs: list[tuple[str, str]] = []
+    merged_envs: dict[str, str] = {}
+    test_source = base_source
+
+    for exp in exps:
+        for pattern, replacement in exp.substitutions:
+            try:
+                new_source, n = re.subn(pattern, replacement, test_source)
+            except re.error as e:
+                return None, f"bad regex in '{exp.name}': {e}"
+            if n == 0:
+                return None, (
+                    f"'{exp.name}' substitution {pattern!r} no longer matches "
+                    "after prior merges (likely overwrites an earlier change)"
+                )
+            test_source = new_source
+            merged_subs.append((pattern, replacement))
+        for k, v in exp.env_vars.items():
+            if k in merged_envs and merged_envs[k] != v:
+                return None, (
+                    f"env var conflict on {k!r}: {merged_envs[k]!r} vs {v!r}"
+                )
+            merged_envs[k] = v
+
+    short_names = "+".join(e.name[:14] for e in exps)
+    full_names = " + ".join(e.name for e in exps)
+    return (
+        Experiment(
+            name=f"merge[{short_names}]"[:60],
+            description=f"Merged: {full_names}",
+            rationale=(
+                f"Combined {len(exps)} candidates that each had positive delta "
+                "against the current best this iteration. Tests the compound "
+                "effect; falls back to the individual winner if it doesn't help."
+            ),
+            substitutions=merged_subs,
+            env_vars=merged_envs,
+        ),
+        "",
+    )
+
+
+def _is_duplicate_of_history(exp: Experiment, history: list[dict]) -> dict | None:
+    """If `exp` matches a prior history entry by fingerprint, return that
+    entry. Otherwise None."""
+    fp = _experiment_fingerprint(exp)
+    for h in history:
+        h_subs = tuple(
+            sorted(
+                (str(s[0]), str(s[1]))
+                for s in (h.get("substitutions") or [])
+                if isinstance(s, (list, tuple)) and len(s) == 2
+            )
+        )
+        h_envs = tuple(sorted((h.get("env_vars") or {}).items()))
+        if fp == (h_subs, h_envs):
+            return h
+    return None
+
+
+def _format_rejected_fingerprints(history: list[dict]) -> str:
+    """Compact list of every (substitutions, env_vars) the LLM has already
+    tried with outcome rejected/crashed/skipped — so it can't propose them
+    again under a different name."""
+    seen: set[tuple] = set()
+    lines: list[str] = []
+    for h in history:
+        outcome = h.get("outcome", "")
+        if outcome not in ("rejected", "crashed", "skipped"):
+            continue
+        subs = tuple(
+            sorted(
+                (str(s[0]), str(s[1]))
+                for s in (h.get("substitutions") or [])
+                if isinstance(s, (list, tuple)) and len(s) == 2
+            )
+        )
+        envs = tuple(sorted((h.get("env_vars") or {}).items()))
+        fp = (subs, envs)
+        if fp in seen:
+            continue
+        seen.add(fp)
+        lines.append(f"  - {outcome:9s} subs={list(subs)} env={dict(envs)}")
+    if not lines:
+        return "  (none yet)"
+    return "\n".join(lines)
+
+
+def _print_waste(metrics: dict, prefix: str = "  waste:       ") -> None:
+    """Print a one-line summary of waste_budget — useful is highlighted
+    first, then non-zero recoverable buckets sorted by size."""
+    wb = metrics.get("waste_budget") or {}
+    if not wb:
+        return
+    parts = [f"useful_gpu={wb.get('useful_gpu', 0.0):.3f}"]
+    others = [(k, v) for k, v in wb.items() if k != "useful_gpu" and isinstance(v, (int, float)) and v > 0]
+    others.sort(key=lambda kv: kv[1], reverse=True)
+    parts.extend(f"{k}={v:.3f}" for k, v in others)
+    print(prefix + ", ".join(parts))
+
+
+# ---------------------------------------------------------------------------
+# JSON object extractor (used by single-experiment llm mode)
+# ---------------------------------------------------------------------------
+
+
+def _extract_json_object(text: str) -> dict | None:
+    """Pull the first JSON object out of an LLM response, tolerating
+    markdown fences / leading prose."""
+    if not text:
+        return None
+    # strip ```json ... ``` fences if present
+    fence_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
+    if fence_match:
+        try:
+            return json.loads(fence_match.group(1))
+        except json.JSONDecodeError:
+            pass
+    # otherwise grab the first balanced { ... }
+    depth = 0
+    start = -1
+    for i, ch in enumerate(text):
+        if ch == "{":
+            if depth == 0:
+                start = i
+            depth += 1
+        elif ch == "}":
+            depth -= 1
+            if depth == 0 and start >= 0:
+                blob = text[start : i + 1]
+                try:
+                    return json.loads(blob)
+                except json.JSONDecodeError:
+                    start = -1
+                    continue
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    p.add_argument(
+        "workload",
+        type=Path,
+        nargs="?",
+        default=None,
+        help=(
+            "Path to a workload script (omit if using --model). When given, "
+            "the script is used as-is for the baseline benchmark."
+        ),
+    )
+    p.add_argument(
+        "--model",
+        type=str,
+        default=None,
+        help=(
+            "HuggingFace model id (e.g. Qwen/Qwen2.5-7B-Instruct, "
+            "meta-llama/Llama-3.2-3B). Generates a baseline workload from "
+            "workloads/train_qwen_lora.py with this MODEL_ID substituted in. "
+            "Use this OR a workload path, not both. For gated models, "
+            "ensure HF_TOKEN is set in your shell."
+        ),
+    )
+    p.add_argument(
+        "--mode",
+        choices=("hardcoded", "llm", "llm-explore"),
+        default="hardcoded",
+        help=(
+            "hardcoded (default): walk through the priority-ordered EXPERIMENTS list. "
+            "llm: ask the LLM for ONE next experiment per iteration (greedy). "
+            "llm-explore: ask for K candidates per iteration, benchmark all, keep "
+            "the best (slower but better at finding interaction effects)."
+        ),
+    )
+    p.add_argument(
+        "--candidates-per-iteration",
+        type=int,
+        default=3,
+        help="Only used when --mode llm-explore. Default 3.",
+    )
+    p.add_argument("--steps", type=int, default=20, help="Steps per benchmark")
+    p.add_argument(
+        "--max-iterations",
+        type=int,
+        default=0,
+        help=(
+            "Cap on experiments to try. Default: len(EXPERIMENTS) for hardcoded mode, "
+            "10 for llm mode."
+        ),
+    )
+    p.add_argument(
+        "--early-stop-after",
+        type=int,
+        default=3,
+        help=(
+            "Stop after N consecutive non-improvements. Crashes do NOT count "
+            "toward this — crashes mean the change was structurally bad, not "
+            "that we've exhausted ideas."
+        ),
+    )
+    p.add_argument(
+        "--max-crashes",
+        type=int,
+        default=4,
+        help=(
+            "Stop after N total subprocess crashes (separate from "
+            "--early-stop-after). Default 4 leaves room for the LLM to try "
+            "structurally different changes after a bad one."
+        ),
+    )
+    p.add_argument(
+        "--improvement-threshold",
+        type=float,
+        default=0.0,
+        help=(
+            "Min %% improvement over current best to accept. Default 0.0 "
+            "(any positive delta wins). Bump to 1.0 if your benchmarks are "
+            "noisy and you want to ignore sub-1%% deltas."
+        ),
+    )
+    p.add_argument(
+        "--events",
+        type=Path,
+        default=None,
+        help=(
+            "Optional NDJSON event stream output. If set, the script appends "
+            "one JSON event per line at baseline / iter / candidate / summary "
+            "milestones. Used by the Streamlit UI; CLI users don't need this."
+        ),
+    )
+    args = p.parse_args()
+    if args.events is not None:
+        global _EVENTS_PATH
+        _EVENTS_PATH = args.events
+        try:
+            args.events.write_text("")  # truncate any prior contents
+        except OSError as exc:
+            sys.stderr.write(f"--events: cannot open {args.events} for writing ({exc})\n")
+            return 1
+    if args.max_iterations <= 0:
+        if args.mode == "hardcoded":
+            args.max_iterations = len(EXPERIMENTS)
+        elif args.mode == "llm-explore":
+            args.max_iterations = 5  # K candidates per iter so 5 iters = 5K benchmarks
+        else:
+            args.max_iterations = 10
+
+    # Validate that exactly one workload source was provided
+    if args.workload is None and args.model is None:
+        sys.stderr.write(
+            "Pass either a workload path or --model MODEL_ID. "
+            "Examples:\n"
+            "  python scripts/auto_tune.py workloads/train_qwen_lora.py\n"
+            "  python scripts/auto_tune.py --model Qwen/Qwen2.5-7B-Instruct\n"
+        )
+        return 1
+    if args.workload is not None and args.model is not None:
+        sys.stderr.write(
+            "Pass EITHER a workload path OR --model, not both.\n"
+        )
+        return 1
+    if not GOBLIN_RUNNER.exists():
+        sys.stderr.write(f"goblin_runner.sh not found at {GOBLIN_RUNNER}\n")
+        return 1
+
+    workspace = Path(tempfile.mkdtemp(prefix="auto_tune_workloads_"))
+
+    if args.workload is not None:
+        workload = args.workload.resolve()
+        if not workload.exists():
+            sys.stderr.write(f"workload not found: {workload}\n")
+            return 1
+        workload_label = str(workload)
+    else:
+        # Generate baseline workload from --model
+        generated = workspace / "_generated_baseline.py"
+        workload = _generate_workload_from_model(args.model, generated)
+        workload_label = f"(generated from --model {args.model})\n                     "
+        workload_label += f"   {workload}\n                     "
+        workload_label += f"   template: {_DEFAULT_WORKLOAD_TEMPLATE}"
+
+    _emit({
+        "type": "started",
+        "mode": args.mode,
+        "workload": str(workload),
+        "model": args.model,
+        "steps": args.steps,
+        "max_iterations": args.max_iterations,
+        "early_stop_after": args.early_stop_after,
+        "max_crashes": args.max_crashes,
+        "improvement_threshold": args.improvement_threshold,
+        "candidates_per_iteration": (
+            args.candidates_per_iteration if args.mode == "llm-explore" else 1
+        ),
+        "workspace": str(workspace),
+    })
+    print(f"Auto-tune workspace: {workspace}")
+    print(f"Mode:                {args.mode}")
+    print(f"Workload:            {workload_label}")
+    print(f"Steps per benchmark: {args.steps}")
+    print(f"Max iterations:      {args.max_iterations}")
+    print(f"Early stop after:    {args.early_stop_after} non-improvements")
+    print(f"Max crashes:         {args.max_crashes} total")
+    print(f"Accept threshold:    {args.improvement_threshold:.1f}%\n")
+
+    # LLM mode setup happens before the baseline so we fail fast on missing
+    # credentials rather than after burning a baseline benchmark. Each LLM
+    # mode gets its own system prompt — the explore mode needs a much
+    # larger token budget to emit K JSON objects.
+    llm_backend = None
+    kb_summary = ""
+    if args.mode == "llm":
+        llm_backend = _build_llm_backend(_LLM_SYSTEM_PROMPT, max_tokens=1024)
+        kb_summary = _kb_summary(REPO_ROOT / "kb" / "rocm_rules.yaml")
+        print("LLM backend ready (single-candidate). KB summary loaded.\n")
+    elif args.mode == "llm-explore":
+        llm_backend = _build_llm_backend(_LLM_EXPLORE_SYSTEM_PROMPT, max_tokens=2048)
+        kb_summary = _kb_summary(REPO_ROOT / "kb" / "rocm_rules.yaml")
+        print(
+            f"LLM backend ready (multi-candidate, K={args.candidates_per_iteration}). "
+            "KB summary loaded.\n"
+        )
+
+    baseline_source = workload.read_text()
+    baseline_path = workspace / "00_baseline.py"
+    baseline_path.write_text(baseline_source)
+
+    print("=" * 60)
+    print("Baseline benchmark")
+    print("=" * 60)
+    baseline = benchmark(baseline_path, args.steps, {})
+    if baseline is None:
+        sys.stderr.write("Baseline benchmark failed; cannot continue.\n")
+        return 1
+
+    baseline_tps = baseline["tokens_per_sec"]
+    print(f"  tokens/sec:    {baseline_tps:.1f}")
+    print(f"  mfu_pct:       {baseline.get('mfu_pct', 0.0):.2f}")
+    print(f"  hbm_peak_gb:   {baseline['hbm_peak_gb']:.2f}")
+    print(f"  gpu_util_pct:  {baseline['gpu_util_pct']:.1f}")
+    print(
+        "  waste_budget:  "
+        + ", ".join(f"{k}={v:.3f}" for k, v in baseline["waste_budget"].items() if v > 0)
+    )
+    _emit({"type": "baseline", "metrics": baseline})
+
+    best_source = baseline_source
+    best_tps = baseline_tps
+    best_env: dict[str, str] = {}
+    last_metrics = baseline
+    accepted: list[tuple[str, float, float]] = []  # (name, tps, delta_pct)
+    rejected: list[tuple[str, str]] = []  # (name, reason)
+    history: list[dict] = []  # for LLM context
+    consecutive_no_improvement = 0
+    total_crashes = 0
+    file_counter = 0  # monotonically increases across all candidates
+
+    for i in range(args.max_iterations):
+        # ---- Get candidates list (1 for hardcoded/llm, K for llm-explore) ----
+        if args.mode == "hardcoded":
+            if i >= len(EXPERIMENTS):
+                print("\nReached end of EXPERIMENTS list.")
+                break
+            candidates = [EXPERIMENTS[i]]
+        elif args.mode == "llm":
+            print(f"\n[asking LLM for next experiment, iteration {i + 1}...]")
+            try:
+                exp = asyncio.run(
+                    _ask_llm_for_experiment(
+                        llm_backend,
+                        kb_summary=kb_summary,
+                        source=best_source,
+                        metrics=last_metrics,
+                        history=history,
+                    )
+                )
+            except Exception as exc:
+                print(f"  LLM call failed: {type(exc).__name__}: {exc}")
+                exp = None
+            if exp is None:
+                print("LLM produced no experiment — stopping.")
+                break
+            candidates = [exp]
+        else:  # llm-explore
+            K = args.candidates_per_iteration
+            print(f"\n[asking LLM for {K} candidates, iteration {i + 1}...]")
+            try:
+                candidates = asyncio.run(
+                    _ask_llm_for_experiments(
+                        llm_backend,
+                        kb_summary=kb_summary,
+                        source=best_source,
+                        metrics=last_metrics,
+                        history=history,
+                        num_candidates=K,
+                    )
+                )
+            except Exception as exc:
+                print(f"  LLM call failed: {type(exc).__name__}: {exc}")
+                candidates = []
+            if not candidates:
+                print("LLM produced no candidates — stopping.")
+                break
+            print(f"  LLM proposed {len(candidates)} candidate(s): "
+                  + ", ".join(c.name for c in candidates))
+
+        print()
+        print("=" * 60)
+        n_label = f" ({len(candidates)} candidates)" if len(candidates) > 1 else ""
+        print(f"Iteration {i + 1}{n_label}")
+        print("=" * 60)
+        _emit({
+            "type": "iter_start",
+            "iteration": i + 1,
+            "candidates": [
+                {
+                    "name": c.name,
+                    "rationale": c.rationale,
+                    "substitutions": c.substitutions,
+                    "env_vars": c.env_vars,
+                }
+                for c in candidates
+            ],
+        })
+
+        # ---- Evaluate each candidate against the CURRENT best ----
+        # Crucial for llm-explore: every candidate is benchmarked against
+        # the same best_source / best_env baseline, so the comparison is
+        # apples-to-apples. State updates only happen after the iteration's
+        # winner is chosen.
+        eval_results: list[dict] = []  # candidates that produced metrics
+        seen_this_iter: set[tuple] = set()  # within-batch dedup
+        crashed_this_iter = False
+        max_crashes_hit = False
+
+        for j, exp in enumerate(candidates):
+            cand_label = f"  Candidate {j + 1}/{len(candidates)}" if len(candidates) > 1 else "  Candidate"
+            print(f"\n{cand_label}: {exp.name}")
+            print(f"    description: {exp.description}")
+            print(f"    rationale:   {exp.rationale}")
+
+            # Helper to emit a per-candidate event with the consistent shape
+            # the UI expects. Called at every terminus below.
+            def _cand_event(outcome: str, metrics: dict | None = None,
+                            delta_vs_best: float | None = None,
+                            reason: str = "") -> None:
+                _emit({
+                    "type": "candidate",
+                    "iteration": i + 1,
+                    "candidate_index": j + 1,
+                    "n_candidates": len(candidates),
+                    "name": exp.name,
+                    "rationale": exp.rationale,
+                    "substitutions": exp.substitutions,
+                    "env_vars": exp.env_vars,
+                    "outcome": outcome,
+                    "metrics": metrics,
+                    "delta_vs_best": delta_vs_best,
+                    "reason": reason,
+                })
+
+            # Dedup: against prior iterations' history
+            dup = _is_duplicate_of_history(exp, history)
+            if dup is not None:
+                print(f"    SKIPPED — already tried as '{dup.get('name', '?')}' "
+                      f"(outcome '{dup.get('outcome', '?')}')")
+                history.append({
+                    "name": exp.name, "outcome": "skipped",
+                    "delta_pct": None,
+                    "substitutions": exp.substitutions, "env_vars": exp.env_vars,
+                })
+                _cand_event("skipped", reason=f"duplicate of '{dup.get('name', '?')}'")
+                continue
+
+            # Dedup: within the current batch (llm-explore can collide)
+            fp = _experiment_fingerprint(exp)
+            if fp in seen_this_iter:
+                print("    SKIPPED — duplicate of an earlier candidate in this iteration")
+                history.append({
+                    "name": exp.name, "outcome": "skipped",
+                    "delta_pct": None,
+                    "substitutions": exp.substitutions, "env_vars": exp.env_vars,
+                })
+                _cand_event("skipped", reason="duplicate of an earlier candidate this iteration")
+                continue
+            seen_this_iter.add(fp)
+
+            # Apply substitutions
+            if exp.substitutions:
+                try:
+                    candidate_source = apply_substitutions(best_source, exp.substitutions)
+                except re.error as exc:
+                    print(f"    SKIPPED — invalid regex from LLM: {exc}")
+                    rejected.append((exp.name, f"bad regex: {exc}"))
+                    history.append({
+                        "name": exp.name, "outcome": "rejected",
+                        "delta_pct": None,
+                        "substitutions": exp.substitutions, "env_vars": exp.env_vars,
+                    })
+                    _cand_event("rejected", reason=f"bad regex: {exc}")
+                    continue
+                if candidate_source is None:
+                    print("    SKIPPED — substitution patterns didn't match")
+                    rejected.append((exp.name, "patterns didn't match"))
+                    history.append({
+                        "name": exp.name, "outcome": "skipped",
+                        "delta_pct": None,
+                        "substitutions": exp.substitutions, "env_vars": exp.env_vars,
+                    })
+                    _cand_event("skipped", reason="substitution patterns didn't match")
+                    continue
+            else:
+                candidate_source = best_source
+
+            file_counter += 1
+            safe_name = re.sub(r"[^A-Za-z0-9_]+", "_", exp.name)[:40] or "exp"
+            candidate_path = workspace / f"{file_counter:03d}_iter{i + 1:02d}_{safe_name}.py"
+            candidate_path.write_text(candidate_source)
+
+            candidate_env = {**best_env, **exp.env_vars}
+            if exp.env_vars:
+                print(f"    env vars:    {exp.env_vars}")
+
+            m = benchmark(candidate_path, args.steps, candidate_env)
+            if m is None:
+                rejected.append((exp.name, "benchmark crashed"))
+                history.append({
+                    "name": exp.name, "outcome": "crashed",
+                    "delta_pct": None,
+                    "substitutions": exp.substitutions, "env_vars": exp.env_vars,
+                })
+                total_crashes += 1
+                crashed_this_iter = True
+                print(
+                    f"    CRASHED — counted toward max-crashes "
+                    f"({total_crashes}/{args.max_crashes})"
+                )
+                _cand_event("crashed", reason="benchmark subprocess failed")
+                if total_crashes >= args.max_crashes:
+                    max_crashes_hit = True
+                    break
+                continue
+
+            tps = m["tokens_per_sec"]
+            delta_vs_best = _delta_pct(tps, best_tps)
+            print(f"    tokens/sec:  {tps:.1f}  (Δ {delta_vs_best:+.2f}% vs current best)")
+            print(f"    mfu_pct:     {m.get('mfu_pct', 0.0):.2f}")
+            print(f"    hbm_peak_gb: {m['hbm_peak_gb']:.2f}")
+            print(f"    gpu_util_pct:{m['gpu_util_pct']:.1f}")
+            _print_waste(m, prefix="    waste:       ")
+
+            # Emit "evaluated" — outcome (accepted/rejected) is decided
+            # later when the iteration's winner is picked across all
+            # candidates. For UI display purposes the per-candidate metrics
+            # are already useful.
+            _cand_event("evaluated", metrics=m, delta_vs_best=delta_vs_best)
+
+            eval_results.append({
+                "exp": exp,
+                "candidate_source": candidate_source,
+                "candidate_env": candidate_env,
+                "metrics": m,
+                "delta_vs_best": delta_vs_best,
+            })
+
+        if max_crashes_hit:
+            print(
+                f"\nReached max-crashes ({args.max_crashes}) — stopping to "
+                "avoid burning more GPU on structurally bad changes."
+            )
+            break
+
+        # ---- Pick the iteration's winner from eval_results ----
+        if not eval_results:
+            # Every candidate was skipped or crashed
+            if crashed_this_iter:
+                print("\n  All candidates crashed or were skipped this iteration.")
+            else:
+                print("\n  All candidates were skipped this iteration.")
+            consecutive_no_improvement += 1
+        else:
+            winner = max(eval_results, key=lambda r: r["metrics"]["tokens_per_sec"])
+            winner_delta = winner["delta_vs_best"]
+
+            # ---- Optional merge step (llm-explore only) ----
+            # If 2+ candidates this iteration each beat the baseline, try
+            # combining them into one experiment and benchmark the merge.
+            # The merge replaces `winner` only if it strictly exceeds the
+            # individual winner's tokens/sec.
+            if args.mode == "llm-explore":
+                positives = [r for r in eval_results if r["delta_vs_best"] > 0]
+                if len(positives) >= 2:
+                    merged_exp, merge_reason = _build_merged_experiment(
+                        [r["exp"] for r in positives], best_source
+                    )
+                    if merged_exp is None:
+                        print(f"\n  MERGE SKIPPED — {merge_reason}")
+                        _emit({
+                            "type": "merge_attempt",
+                            "iteration": i + 1,
+                            "outcome": "skipped",
+                            "reason": merge_reason,
+                            "candidate_names": [r["exp"].name for r in positives],
+                        })
+                    else:
+                        print(
+                            f"\n  Merging {len(positives)} positive candidates: "
+                            f"{merged_exp.description}"
+                        )
+                        # Apply substitutions to get the merged source
+                        merged_source = best_source
+                        for pattern, replacement in merged_exp.substitutions:
+                            merged_source = re.sub(pattern, replacement, merged_source)
+                        merged_env = {**best_env, **merged_exp.env_vars}
+
+                        file_counter += 1
+                        merged_path = workspace / f"{file_counter:03d}_iter{i + 1:02d}_merge.py"
+                        merged_path.write_text(merged_source)
+                        if merged_exp.env_vars:
+                            print(f"    env vars: {merged_exp.env_vars}")
+
+                        m = benchmark(merged_path, args.steps, merged_env)
+                        if m is None:
+                            total_crashes += 1
+                            crashed_this_iter = True
+                            print(
+                                f"    MERGE CRASHED — counted toward max-crashes "
+                                f"({total_crashes}/{args.max_crashes})"
+                            )
+                            _emit({
+                                "type": "merge_attempt",
+                                "iteration": i + 1,
+                                "outcome": "crashed",
+                                "candidate_names": [r["exp"].name for r in positives],
+                                "merged_name": merged_exp.name,
+                            })
+                            if total_crashes >= args.max_crashes:
+                                max_crashes_hit = True
+                        else:
+                            tps = m["tokens_per_sec"]
+                            delta_vs_best = _delta_pct(tps, best_tps)
+                            print(
+                                f"    Merged tokens/sec: {tps:.1f}  "
+                                f"(Δ {delta_vs_best:+.2f}% vs baseline)"
+                            )
+                            print(f"    mfu_pct:           {m.get('mfu_pct', 0.0):.2f}")
+                            print(f"    hbm_peak_gb:       {m['hbm_peak_gb']:.2f}")
+
+                            individual_best_tps = winner["metrics"]["tokens_per_sec"]
+                            if tps > individual_best_tps:
+                                print(
+                                    f"    MERGE WINS — exceeds individual winner "
+                                    f"'{winner['exp'].name}' "
+                                    f"({tps:.1f} > {individual_best_tps:.1f})"
+                                )
+                                _emit({
+                                    "type": "merge_attempt",
+                                    "iteration": i + 1,
+                                    "outcome": "wins",
+                                    "candidate_names": [r["exp"].name for r in positives],
+                                    "merged_name": merged_exp.name,
+                                    "metrics": m,
+                                    "delta_vs_best": delta_vs_best,
+                                    "individual_best_name": winner["exp"].name,
+                                    "individual_best_tps": individual_best_tps,
+                                })
+                                # Promote merged to be the new winner
+                                winner = {
+                                    "exp": merged_exp,
+                                    "candidate_source": merged_source,
+                                    "candidate_env": merged_env,
+                                    "metrics": m,
+                                    "delta_vs_best": delta_vs_best,
+                                }
+                                winner_delta = delta_vs_best
+                            else:
+                                print(
+                                    f"    Merge didn't beat individual winner; "
+                                    f"keeping '{winner['exp'].name}'"
+                                )
+                                _emit({
+                                    "type": "merge_attempt",
+                                    "iteration": i + 1,
+                                    "outcome": "lost",
+                                    "candidate_names": [r["exp"].name for r in positives],
+                                    "merged_name": merged_exp.name,
+                                    "metrics": m,
+                                    "delta_vs_best": delta_vs_best,
+                                    "individual_best_name": winner["exp"].name,
+                                    "individual_best_tps": individual_best_tps,
+                                })
+
+            if winner_delta >= args.improvement_threshold:
+                print(
+                    f"\n  ACCEPTED — '{winner['exp'].name}' wins "
+                    f"(Δ {winner_delta:+.2f}% vs current best)"
+                )
+                best_source = winner["candidate_source"]
+                best_tps = winner["metrics"]["tokens_per_sec"]
+                best_env = winner["candidate_env"]
+                last_metrics = winner["metrics"]
+                accepted.append((winner["exp"].name, best_tps, winner_delta))
+                history.append({
+                    "name": winner["exp"].name, "outcome": "accepted",
+                    "delta_pct": winner_delta,
+                    "substitutions": winner["exp"].substitutions,
+                    "env_vars": winner["exp"].env_vars,
+                })
+                # Other candidates of this iteration get marked rejected
+                for r in eval_results:
+                    if r is winner:
+                        continue
+                    rejected.append((r["exp"].name, f"{r['delta_vs_best']:+.2f}%"))
+                    history.append({
+                        "name": r["exp"].name, "outcome": "rejected",
+                        "delta_pct": r["delta_vs_best"],
+                        "substitutions": r["exp"].substitutions,
+                        "env_vars": r["exp"].env_vars,
+                    })
+                consecutive_no_improvement = 0
+                _emit({
+                    "type": "iter_done",
+                    "iteration": i + 1,
+                    "outcome": "accepted",
+                    "winner_name": winner["exp"].name,
+                    "winner_delta": winner_delta,
+                    "best_tps": best_tps,
+                    "best_metrics": winner["metrics"],
+                    "best_env_vars": best_env,
+                })
+            else:
+                print(
+                    f"\n  ALL REJECTED — best candidate '{winner['exp'].name}' "
+                    f"only Δ {winner_delta:+.2f}% (threshold {args.improvement_threshold:.1f}%)"
+                )
+                for r in eval_results:
+                    rejected.append((r["exp"].name, f"{r['delta_vs_best']:+.2f}%"))
+                    history.append({
+                        "name": r["exp"].name, "outcome": "rejected",
+                        "delta_pct": r["delta_vs_best"],
+                        "substitutions": r["exp"].substitutions,
+                        "env_vars": r["exp"].env_vars,
+                    })
+                # Update last_metrics with the winner anyway so the LLM sees
+                # the latest waste_budget on the next turn.
+                if args.mode in ("llm", "llm-explore"):
+                    last_metrics = winner["metrics"]
+                consecutive_no_improvement += 1
+                _emit({
+                    "type": "iter_done",
+                    "iteration": i + 1,
+                    "outcome": "all_rejected",
+                    "winner_name": winner["exp"].name,
+                    "winner_delta": winner_delta,
+                    "best_tps": best_tps,
+                })
+
+        if consecutive_no_improvement >= args.early_stop_after:
+            print(
+                f"\nNo improvement for {args.early_stop_after} consecutive iterations — early stopping."
+            )
+            break
+
+    # Save best
+    best_path = workspace / "best.py"
+    best_path.write_text(best_source)
+
+    # Summary
+    print()
+    print("=" * 60)
+    print("AUTO-TUNE SUMMARY")
+    print("=" * 60)
+    print(f"Baseline tokens/sec: {baseline_tps:.1f}")
+    print(
+        f"Best tokens/sec:     {best_tps:.1f}  "
+        f"({_delta_pct(best_tps, baseline_tps):+.2f}% vs baseline)"
+    )
+    print()
+    print(f"Accepted ({len(accepted)}):")
+    for name, tps, delta in accepted:
+        print(f"  + {name:25s}  {tps:8.1f} tok/s  (Δ {delta:+.2f}%)")
+    print()
+    print(f"Rejected ({len(rejected)}):")
+    for name, reason in rejected:
+        print(f"  - {name:25s}  {reason}")
+    print()
+
+    if best_env:
+        print("Required env vars for best config:")
+        for k, v in best_env.items():
+            print(f"  export {k}={v}")
+        print()
+
+    print(f"Best workload script:  {best_path}")
+    print(f"Diff vs baseline:      diff {workload} {best_path}")
+
+    _emit({
+        "type": "summary",
+        "baseline_metrics": baseline,
+        "best_metrics": last_metrics,
+        "baseline_tps": baseline_tps,
+        "best_tps": best_tps,
+        "improvement_pct": _delta_pct(best_tps, baseline_tps),
+        "accepted": [
+            {"name": name, "tps": tps, "delta_pct": delta}
+            for name, tps, delta in accepted
+        ],
+        "rejected": [
+            {"name": name, "reason": reason}
+            for name, reason in rejected
+        ],
+        "best_env_vars": best_env,
+        "best_workload_path": str(best_path),
+        "baseline_workload_path": str(workload),
+    })
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/fixtures/sample_train.json

@@ -0,0 +1,27 @@
+{
+  "model_name_or_path": "Qwen/Qwen2.5-7B-Instruct",
+  "per_device_train_batch_size": 8,
+  "gradient_accumulation_steps": 4,
+  "max_seq_length": 4096,
+  "learning_rate": 0.0003,
+  "warmup_steps": 200,
+  "bf16": true,
+  "optim": "adamw_torch_fused",
+  "gradient_checkpointing": true,
+  "torch_compile": true,
+  "dataloader_num_workers": 4,
+  "dataloader_pin_memory": true,
+  "dataloader_prefetch_factor": 4,
+  "dataloader_persistent_workers": true,
+  "attn_implementation": "flash",
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "logging_steps": 25,
+  "output_dir": "./out",
+  "hub_token": "hf_jsonsamplehfabcdefghijklmnopqrs",
+  "checkpoint_uri": "s3://team-bucket/runs/qwen-lora-001/",
+  "env_vars": {
+    "HSA_FORCE_FINE_GRAIN_PCIE": "1",
+    "NCCL_MIN_NCHANNELS": "112"
+  }
+}

tests/fixtures/sample_train.py

@@ -0,0 +1,101 @@
+"""Realistic-looking HF Trainer fine-tuning script with secrets sprinkled in.
+
+Used as a fixture for parse_config — exercises every code path we care about:
+TrainingArguments kwargs, DataLoader kwargs, torch.compile, gradient
+checkpointing, os.environ assignments, LoRA config, and from_pretrained.
+"""
+
+import os
+
+import torch
+from torch.utils.data import DataLoader
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    Trainer,
+    TrainingArguments,
+)
+from peft import LoraConfig, get_peft_model
+from datasets import load_dataset
+
+# Secrets we expect parse_config to redact before storing raw_source.
+HF_TOKEN = "hf_abcdefghijklmnopqrstuvwxyz123456"
+OPENAI_KEY = "sk-abcdefghijklmnopqrstuvwxyz1234567890"
+GH_TOKEN = "gho_abcdefghijklmnopqrstuvwxyz123456"
+AUTH_HEADER = "Authorization: Bearer eyJhbGciOi.JIUzI1NiJ9.signature123"
+DATA_ROOT = "/home/researcher/datasets/alpaca"
+S3_BUCKET = "s3://my-team/checkpoints/qwen-lora/"
+WS_LOG = "wss://logs.internal.example.com/stream"
+
+# Environment variables the agent should capture into env_vars.
+os.environ["HSA_FORCE_FINE_GRAIN_PCIE"] = "1"
+os.environ["MIOPEN_FIND_MODE"] = "3"
+os.environ["NCCL_MIN_NCHANNELS"] = "112"
+
+MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    attn_implementation="eager",
+    token=HF_TOKEN,
+)
+
+# LoRA — rank should land in WorkloadConfig.lora_rank.
+lora_config = LoraConfig(
+    r=16,
+    lora_alpha=32,
+    target_modules=["q_proj", "v_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, lora_config)
+
+# Should set gradient_checkpointing=True via the explicit enable() call.
+model.gradient_checkpointing_enable()
+
+# Should flip torch_compile=True.
+model = torch.compile(model, mode="reduce-overhead")
+
+dataset = load_dataset("yahma/alpaca-cleaned", split="train")
+
+train_loader = DataLoader(
+    dataset,
+    batch_size=4,
+    num_workers=0,
+    pin_memory=False,
+    prefetch_factor=2,
+    persistent_workers=False,
+)
+
+training_args = TrainingArguments(
+    output_dir="./out",
+    per_device_train_batch_size=4,
+    gradient_accumulation_steps=8,
+    num_train_epochs=3,
+    learning_rate=2e-4,
+    warmup_steps=100,
+    fp16=True,
+    optim="adamw_torch",
+    logging_steps=10,
+    save_steps=500,
+    dataloader_num_workers=0,
+    dataloader_pin_memory=False,
+    gradient_checkpointing=True,
+    torch_compile=False,
+    report_to="none",
+    push_to_hub=False,
+    hub_token=HF_TOKEN,
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+)
+
+if __name__ == "__main__":
+    trainer.train()

tests/fixtures/sample_train.yaml

@@ -0,0 +1,26 @@
+# YAML training config — same schema as JSON, exercised separately so we know
+# yaml.safe_load + dict extraction agree with the JSON path.
+model_name_or_path: Qwen/Qwen2.5-7B-Instruct
+per_device_train_batch_size: 2
+gradient_accumulation_steps: 16
+max_seq_length: 8192
+learning_rate: 1.0e-4
+warmup_steps: 50
+bf16: true
+optim: adamw_torch
+gradient_checkpointing: true
+torch_compile: false
+dataloader_num_workers: 8
+dataloader_pin_memory: true
+dataloader_prefetch_factor: 2
+dataloader_persistent_workers: true
+attn_implementation: sdpa
+num_train_epochs: 1
+output_dir: ./out
+# Secrets that should be scrubbed:
+hub_token: "hf_yamlsamplehfabcdefghijklmnopqrs1"
+auth_header: "Bearer eyJ.payload.signaturetoken"
+data_path: "/home/teamuser/datasets/alpaca-cleaned"
+env_vars:
+  HSA_FORCE_FINE_GRAIN_PCIE: "1"
+  MIOPEN_FIND_MODE: "3"

tests/test_compare_runs_normalize.py

@@ -0,0 +1,155 @@
+"""Tests for compare_runs's flat-Patch recovery path.
+
+Live AMD-GPU lesson: Qwen models routinely forward a flat WorkloadConfig (or
+even just the changed-fields subset) as the ``patch=`` argument to
+compare_runs, instead of the full Patch envelope. ``_normalize_patch`` is
+the safety net — it must:
+
+  1. Pass real Patch dicts through unchanged.
+  2. Detect any flat-config shape (full WorkloadConfig, just dataloader
+     fields, just env_vars, etc.) — NOT just dicts with model_name.
+  3. Recover by substituting the cached propose_patch result when one
+     exists, or wrapping the flat config minimally as a last resort.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from agent.tools import compare_runs as cr_mod
+from agent.tools import propose_patch as pp_mod
+
+
+# ---------------------------------------------------------------------------
+# _looks_like_flat_config detection
+# ---------------------------------------------------------------------------
+
+
+class TestFlatConfigDetection:
+    def test_real_patch_is_not_flat(self) -> None:
+        real = {
+            "new_config": {"model_name": "x"},
+            "diff": "(no changes)",
+            "rationale": [],
+            "expected_speedup_low": 1.0,
+            "expected_speedup_high": 1.0,
+            "confidence": 0.0,
+        }
+        assert cr_mod._looks_like_flat_config(real) is False
+
+    def test_full_workload_config_is_flat(self) -> None:
+        flat = {
+            "model_name": "Qwen/Qwen2.5-7B-Instruct",
+            "precision": "bf16",
+            "attention_impl": "flash_rocm",
+            "batch_size": 12,
+        }
+        assert cr_mod._looks_like_flat_config(flat) is True
+
+    def test_dataloader_only_diff_is_flat(self) -> None:
+        # The exact failure mode from the live MI300X audit: model only
+        # passed the *changed* dataloader fields, no model_name in sight.
+        flat = {
+            "dataloader_persistent_workers": True,
+            "dataloader_pin_memory": True,
+            "dataloader_workers": 8,
+        }
+        assert cr_mod._looks_like_flat_config(flat) is True
+
+    def test_env_vars_only_diff_is_flat(self) -> None:
+        flat = {"env_vars": {"NCCL_MIN_NCHANNELS": "112"}}
+        assert cr_mod._looks_like_flat_config(flat) is True
+
+    def test_precision_only_diff_is_flat(self) -> None:
+        flat = {"precision": "bf16"}
+        assert cr_mod._looks_like_flat_config(flat) is True
+
+    def test_unrelated_dict_not_flat(self) -> None:
+        # Garbage dict with no WorkloadConfig fields → don't claim it's flat.
+        assert cr_mod._looks_like_flat_config({"foo": 1, "bar": 2}) is False
+
+    def test_non_dict_not_flat(self) -> None:
+        assert cr_mod._looks_like_flat_config(None) is False  # type: ignore[arg-type]
+        assert cr_mod._looks_like_flat_config("a string") is False  # type: ignore[arg-type]
+        assert cr_mod._looks_like_flat_config([1, 2, 3]) is False  # type: ignore[arg-type]
+
+
+# ---------------------------------------------------------------------------
+# _normalize_patch + cached-patch recovery
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def cached_patch(monkeypatch):
+    """Plant a fake `latest_patch()` so the recovery path picks it up."""
+    fake = {
+        "new_config": {"model_name": "Qwen/Qwen2.5-7B-Instruct", "precision": "bf16"},
+        "diff": "- precision: fp16\n+ precision: bf16",
+        "rationale": [
+            {
+                "rule_id": "precision.bf16_over_fp16_on_mi300x",
+                "rationale": "r",
+                "citation": "c",
+                "targets_bucket": "precision_path",
+                "estimated_recovery_seconds": 0.09,
+            }
+        ],
+        "expected_speedup_low": 1.05,
+        "expected_speedup_high": 1.30,
+        "confidence": 0.85,
+    }
+    monkeypatch.setattr(pp_mod, "_LAST_PATCH", fake)
+    yield fake
+
+
+class TestNormalizePatch:
+    def test_real_patch_passes_through(self) -> None:
+        real = {
+            "new_config": {"model_name": "x"},
+            "diff": "...",
+            "rationale": [],
+            "expected_speedup_low": 1.0,
+            "expected_speedup_high": 1.0,
+            "confidence": 0.0,
+        }
+        out, notes = cr_mod._normalize_patch(real)
+        assert out is real
+        assert notes == []
+
+    def test_dataloader_only_diff_recovers_via_cached(self, cached_patch) -> None:
+        # The exact live-AMD-GPU failure: model forwarded only the changed
+        # dataloader fields. Old code's narrow sentinel set (model_name etc.)
+        # would miss this. New behavior: detected, cached patch substituted.
+        flat = {
+            "dataloader_persistent_workers": True,
+            "dataloader_pin_memory": True,
+            "dataloader_workers": 8,
+        }
+        out, notes = cr_mod._normalize_patch(flat)
+        assert out is cached_patch  # full fidelity restored
+        assert any("substituted the cached" in n for n in notes)
+
+    def test_flat_config_falls_back_to_minimal_wrap_when_no_cache(
+        self, monkeypatch
+    ) -> None:
+        # No cached patch — must still produce a Patch-shape dict so
+        # compare_runs doesn't crash on Pydantic validation.
+        monkeypatch.setattr(pp_mod, "_LAST_PATCH", None)
+        flat = {"precision": "bf16"}
+        out, notes = cr_mod._normalize_patch(flat)
+        assert "new_config" in out
+        assert "diff" in out
+        assert out["expected_speedup_low"] == 1.0
+        assert out["confidence"] == 0.0
+        assert any("synthesized a minimal Patch" in n for n in notes)
+
+    def test_non_flat_garbage_passes_through_for_pydantic_to_reject(
+        self, monkeypatch
+    ) -> None:
+        # If it's neither a real Patch nor a recognizable flat config, let
+        # pydantic produce the clear ValidationError — don't silently mangle.
+        monkeypatch.setattr(pp_mod, "_LAST_PATCH", None)
+        garbage = {"foo": 1, "bar": [2]}
+        out, notes = cr_mod._normalize_patch(garbage)
+        assert out is garbage
+        assert notes == []

tests/test_loop.py

@@ -0,0 +1,322 @@
+"""Tests for the agent loop driver.
+
+We never hit a real LLM. The loop talks to a `Backend` (see
+`agent/backends/`); each test injects a `FakeBackend` whose `next_turn`
+returns a queued sequence of scripted `AgentTurn` objects. Tools are
+stubbed so we can drive specific control-flow paths.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+
+from agent import loop as loop_module
+from agent.backends.base import AgentTurn, Backend, ToolCall
+from agent.schemas import SSEEvent, ToolResult
+
+
+# ---------------------------------------------------------------------------
+# Fake backend
+# ---------------------------------------------------------------------------
+
+
+class FakeBackend(Backend):
+    """A scripted Backend for testing the loop in isolation.
+
+    Each test queues a list of `AgentTurn`s; calling `next_turn` pops the
+    next one. We also record every tool result the loop hands back so tests
+    can assert that error / id / content were threaded through correctly.
+    """
+
+    name = "fake"
+
+    def __init__(
+        self,
+        scripted_turns: list[AgentTurn] | None = None,
+        next_turn_raises: BaseException | None = None,
+    ) -> None:
+        self._scripted = list(scripted_turns or [])
+        self._raise_on_next = next_turn_raises
+        self.user_messages: list[str] = []
+        self.tool_results: list[dict[str, Any]] = []
+        self.turn_count = 0
+
+    def add_user_message(self, content: str) -> None:
+        self.user_messages.append(content)
+
+    def add_tool_result(
+        self,
+        tool_call_id: str,
+        name: str,
+        content: str,
+        is_error: bool,
+    ) -> None:
+        self.tool_results.append(
+            {
+                "id": tool_call_id,
+                "name": name,
+                "content": content,
+                "is_error": is_error,
+            }
+        )
+
+    async def next_turn(self, tool_schemas: list[dict[str, Any]]) -> AgentTurn:
+        self.turn_count += 1
+        if self._raise_on_next is not None:
+            exc = self._raise_on_next
+            self._raise_on_next = None
+            raise exc
+        if not self._scripted:
+            raise AssertionError(
+                "FakeBackend exhausted — loop made more turns than expected"
+            )
+        return self._scripted.pop(0)
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _install_backend(monkeypatch, backend: Backend) -> Backend:
+    """Replace `make_backend` so the loop sees our fake."""
+    monkeypatch.setattr(loop_module, "make_backend", lambda **_kwargs: backend)
+    return backend
+
+
+def _install_make_backend_raises(monkeypatch, exc: BaseException) -> None:
+    def boom(**_kwargs: Any) -> Backend:
+        raise exc
+
+    monkeypatch.setattr(loop_module, "make_backend", boom)
+
+
+def _install_fake_tools(
+    monkeypatch, tool_responses: dict[str, ToolResult]
+) -> list[str]:
+    """Replace `tools_module.call` and `tool_schemas`. Returns a list
+    that records the order tools were invoked.
+    """
+    invoked: list[str] = []
+
+    def fake_call(name: str, **_kwargs: Any) -> ToolResult:
+        invoked.append(name)
+        return tool_responses.get(
+            name, ToolResult(ok=False, error=f"no fake registered for {name}")
+        )
+
+    monkeypatch.setattr(loop_module.tools_module, "call", fake_call)
+    monkeypatch.setattr(loop_module.tools_module, "tool_schemas", lambda: [])
+    return invoked
+
+
+async def _collect(stream) -> list[SSEEvent]:
+    out: list[SSEEvent] = []
+    async for event in stream:
+        out.append(event)
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Tests
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_emits_thought_then_tool_call_then_tool_result(monkeypatch) -> None:
+    backend = FakeBackend(
+        scripted_turns=[
+            AgentTurn(
+                text_blocks=["I'll start by parsing the config."],
+                tool_calls=[
+                    ToolCall(id="tu_1", name="parse_config", input={"file_path": "/x.py"})
+                ],
+                stop_reason="tool_use",
+            ),
+            AgentTurn(text_blocks=["Done."], tool_calls=[], stop_reason="end_turn"),
+        ]
+    )
+    _install_backend(monkeypatch, backend)
+    invoked = _install_fake_tools(
+        monkeypatch,
+        {"parse_config": ToolResult(ok=True, result={"model_name": "x"})},
+    )
+
+    events = await _collect(loop_module.run_audit("/x.py"))
+    types = [e.type for e in events]
+
+    assert types[0] == "thought"
+    assert types[1] == "tool_call"
+    assert types[2] == "tool_result"
+    # No compare_runs ⇒ final event is the "no final report" error.
+    assert types[-1] == "error"
+    assert "without producing a final report" in events[-1].data["message"]
+    assert invoked == ["parse_config"]
+
+    # tool_call carries id/name/input; tool_result mirrors that plus ok/result/error.
+    assert events[1].data == {
+        "id": "tu_1",
+        "name": "parse_config",
+        "input": {"file_path": "/x.py"},
+    }
+    assert events[2].data["ok"] is True
+    assert events[2].data["result"] == {"model_name": "x"}
+    assert events[2].data["error"] is None
+
+    # The user message and tool result were threaded into the backend.
+    assert backend.user_messages == ["Audit this fine-tuning workload: /x.py"]
+    assert backend.tool_results == [
+        {
+            "id": "tu_1",
+            "name": "parse_config",
+            "content": '{"model_name": "x"}',
+            "is_error": False,
+        }
+    ]
+
+
+@pytest.mark.asyncio
+async def test_final_report_extracted_from_compare_runs(monkeypatch) -> None:
+    fake_report = {"workload_name": "test", "speedup_actual": 2.0}
+    backend = FakeBackend(
+        scripted_turns=[
+            AgentTurn(
+                text_blocks=["Wrapping up."],
+                tool_calls=[
+                    ToolCall(
+                        id="tu_compare",
+                        name="compare_runs",
+                        input={
+                            "workload_name": "t",
+                            "before": {},
+                            "after": {},
+                            "patch": {},
+                        },
+                    )
+                ],
+                stop_reason="end_turn",
+            ),
+        ]
+    )
+    _install_backend(monkeypatch, backend)
+    _install_fake_tools(
+        monkeypatch, {"compare_runs": ToolResult(ok=True, result=fake_report)}
+    )
+
+    events = await _collect(loop_module.run_audit("/x.py"))
+
+    assert events[-1].type == "final_report"
+    assert events[-1].data["report"] == fake_report
+
+
+@pytest.mark.asyncio
+async def test_tool_error_passes_through_does_not_crash(monkeypatch) -> None:
+    backend = FakeBackend(
+        scripted_turns=[
+            AgentTurn(
+                text_blocks=["Trying parse."],
+                tool_calls=[
+                    ToolCall(id="tu_1", name="parse_config", input={"file_path": "/bogus"})
+                ],
+                stop_reason="tool_use",
+            ),
+            AgentTurn(text_blocks=["Giving up."], tool_calls=[], stop_reason="end_turn"),
+        ]
+    )
+    _install_backend(monkeypatch, backend)
+    _install_fake_tools(
+        monkeypatch,
+        {"parse_config": ToolResult(ok=False, error="file not found")},
+    )
+
+    events = await _collect(loop_module.run_audit("/bogus"))
+    tool_result_events = [e for e in events if e.type == "tool_result"]
+    assert len(tool_result_events) == 1
+    assert tool_result_events[0].data["ok"] is False
+    assert tool_result_events[0].data["error"] == "file not found"
+    # The loop kept iterating rather than bailing.
+    assert events[-1].type == "error"  # no compare_runs ⇒ "no final report"
+
+    # Backend received an is_error=True tool result with the error message.
+    assert backend.tool_results[-1]["is_error"] is True
+    assert backend.tool_results[-1]["content"] == "file not found"
+
+
+@pytest.mark.asyncio
+async def test_backend_construction_failure_yields_error_event(monkeypatch) -> None:
+    _install_make_backend_raises(
+        monkeypatch, RuntimeError("HF_TOKEN is not set; Qwen backend cannot run.")
+    )
+    events = await _collect(loop_module.run_audit("/x.py"))
+    assert len(events) == 1
+    assert events[0].type == "error"
+    assert "HF_TOKEN" in events[0].data["message"]
+
+
+@pytest.mark.asyncio
+async def test_mid_loop_exception_is_caught(monkeypatch) -> None:
+    backend = FakeBackend(next_turn_raises=RuntimeError("boom"))
+    _install_backend(monkeypatch, backend)
+    monkeypatch.setattr(loop_module.tools_module, "tool_schemas", lambda: [])
+
+    events = await _collect(loop_module.run_audit("/x.py"))
+    assert events[-1].type == "error"
+    assert "boom" in events[-1].data["message"]
+
+
+@pytest.mark.asyncio
+async def test_loop_caps_at_max_steps(monkeypatch) -> None:
+    """Even if the model never says end_turn, we bail after MAX_STEPS."""
+    backend = FakeBackend(
+        scripted_turns=[
+            AgentTurn(
+                text_blocks=[f"step {i}"],
+                tool_calls=[
+                    ToolCall(id=f"tu_{i}", name="parse_config", input={"file_path": "/x.py"})
+                ],
+                stop_reason="tool_use",
+            )
+            for i in range(loop_module.MAX_STEPS + 2)  # extra so we'd overrun
+        ]
+    )
+    _install_backend(monkeypatch, backend)
+    _install_fake_tools(monkeypatch, {"parse_config": ToolResult(ok=True, result={})})
+
+    events = await _collect(loop_module.run_audit("/x.py"))
+    # Backend's next_turn was called exactly MAX_STEPS times.
+    assert backend.turn_count == loop_module.MAX_STEPS
+    # Last event is the "no final report" error (not a crash).
+    assert events[-1].type == "error"
+
+
+@pytest.mark.asyncio
+async def test_tool_call_id_is_threaded_to_backend(monkeypatch) -> None:
+    """The loop must hand the tool_call id back to the backend so the next
+    turn's request can correlate the tool_result with the originating call.
+    """
+    backend = FakeBackend(
+        scripted_turns=[
+            AgentTurn(
+                text_blocks=["parse"],
+                tool_calls=[
+                    ToolCall(id="tu_abc", name="parse_config", input={"file_path": "/x"})
+                ],
+                stop_reason="tool_use",
+            ),
+            AgentTurn(text_blocks=["done"], tool_calls=[], stop_reason="end_turn"),
+        ]
+    )
+    _install_backend(monkeypatch, backend)
+    _install_fake_tools(
+        monkeypatch, {"parse_config": ToolResult(ok=True, result={"a": 1})}
+    )
+
+    await _collect(loop_module.run_audit("/x"))
+
+    # Backend got exactly one tool_result with id=tu_abc.
+    assert len(backend.tool_results) == 1
+    assert backend.tool_results[0]["id"] == "tu_abc"
+    assert backend.tool_results[0]["name"] == "parse_config"
+    assert backend.tool_results[0]["is_error"] is False

tests/test_misnested_args.py

@@ -0,0 +1,95 @@
+"""Tests for the defensive misnested-arg extraction in benchmark + profile_run.
+
+Live AMD-GPU lesson: Qwen2.5-7B (and probably others) occasionally JSON-nests
+``steps`` / ``cache`` *inside* the ``config`` dict instead of at the top level
+alongside it. WorkloadConfig strict-validates extras, so without this defense
+the call errors out and a tool slot is wasted. The well-tuned scenario run
+on 2026-05-07 burned two of the eight available slots on this exact mistake;
+fixing it costs nothing and saves the audit.
+"""
+
+from __future__ import annotations
+
+import shutil
+
+from agent.tools import call
+
+
+def _baseline_config() -> dict:
+    return {
+        "model_name": "Qwen/Qwen2.5-7B-Instruct",
+        "batch_size": 4,
+        "precision": "fp16",
+        "attention_impl": "eager",
+        "dataloader_workers": 0,
+    }
+
+
+class TestBenchmarkMisnestedArgs:
+    def setup_method(self) -> None:
+        # Each test starts with an empty cache so cache-hit doesn't mask the
+        # behavior under test.
+        shutil.rmtree("bench_cache", ignore_errors=True)
+
+    def test_steps_nested_in_config_is_extracted(self) -> None:
+        """Old behavior: ``WorkloadConfig`` validation explodes with
+        'Extra inputs are not permitted [steps]'. New behavior: defensive
+        extraction pulls ``steps`` back to the top-level arg, call succeeds.
+        """
+        cfg = {**_baseline_config(), "steps": 25}
+        result = call("benchmark", config=cfg)
+        assert result.ok, result.error
+        assert result.result["steps"] == 25
+
+    def test_cache_nested_in_config_is_extracted(self) -> None:
+        cfg = {**_baseline_config(), "cache": False}
+        result = call("benchmark", config=cfg)
+        assert result.ok, result.error
+
+    def test_force_rerun_nested_in_config_is_extracted(self) -> None:
+        cfg = {**_baseline_config(), "force_rerun": True}
+        result = call("benchmark", config=cfg)
+        assert result.ok, result.error
+
+    def test_explicit_top_level_wins_over_nested(self) -> None:
+        """If caller passes BOTH (config has steps + top-level steps), the
+        explicit non-default top-level wins. Defensive code is for the
+        accident case, not for letting nesting silently override."""
+        cfg = {**_baseline_config(), "steps": 25}
+        result = call("benchmark", config=cfg, steps=37)
+        assert result.ok, result.error
+        assert result.result["steps"] == 37
+
+    def test_all_three_nested_at_once(self) -> None:
+        """The exact failure mode from the live run: model nested three
+        runtime args inside config. All three should get pulled out.
+        """
+        cfg = {
+            **_baseline_config(),
+            "steps": 30,
+            "cache": False,
+            "force_rerun": True,
+        }
+        result = call("benchmark", config=cfg)
+        assert result.ok, result.error
+        assert result.result["steps"] == 30
+
+
+class TestProfileRunMisnestedArgs:
+    def test_steps_nested_in_config_is_extracted(self) -> None:
+        cfg = {**_baseline_config(), "steps": 7}
+        result = call("profile_run", config=cfg)
+        assert result.ok, result.error
+        assert result.result["steps"] == 7
+
+    def test_explicit_top_level_wins(self) -> None:
+        cfg = {**_baseline_config(), "steps": 7}
+        result = call("profile_run", config=cfg, steps=15)
+        assert result.ok, result.error
+        assert result.result["steps"] == 15
+
+    def test_clean_config_unaffected(self) -> None:
+        """Sanity: when nothing is misnested, behavior is unchanged."""
+        result = call("profile_run", config=_baseline_config())
+        assert result.ok, result.error
+        assert result.result["steps"] == 10  # default

tests/test_parse_config.py

@@ -0,0 +1,377 @@
+"""Tests for ``agent.tools.parse_config._parse_config``.
+
+Covers all three input shapes (Python AST, JSON, YAML), redaction of every
+secret pattern we ship, error paths for missing/malformed inputs, and the
+WorkloadConfig field mapping for HF TrainingArguments + DataLoader kwargs.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from agent.schemas import WorkloadConfig
+from agent.tools.parse_config import PARSE_CONFIG, _parse_config, _parse_config_full
+
+FIXTURES = Path(__file__).parent / "fixtures"
+
+
+# ---------------------------------------------------------------------------
+# Python script path
+# ---------------------------------------------------------------------------
+
+
+class TestPythonScript:
+    def test_returns_ok(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        assert result.ok, result.error
+        assert result.result is not None
+
+    def test_extracts_training_arguments_kwargs(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        cfg = result.result
+        assert cfg["batch_size"] == 4
+        assert cfg["grad_accum_steps"] == 8
+        assert cfg["lr"] == pytest.approx(2e-4)
+        assert cfg["warmup_steps"] == 100
+        assert cfg["optimizer"] == "adamw_torch"
+        # fp16=True in TrainingArguments → precision should resolve to fp16.
+        assert cfg["precision"] == "fp16"
+
+    def test_dataloader_kwargs_captured(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        cfg = result.result
+        assert cfg["dataloader_workers"] == 0
+        assert cfg["dataloader_pin_memory"] is False
+        assert cfg["dataloader_prefetch_factor"] == 2
+        assert cfg["dataloader_persistent_workers"] is False
+
+    def test_torch_compile_call_flips_flag(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        # The script calls torch.compile(model, ...), even though the
+        # TrainingArguments has torch_compile=False — the explicit call wins.
+        assert result.result["torch_compile"] is True
+
+    def test_gradient_checkpointing_enable_call(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        assert result.result["gradient_checkpointing"] is True
+
+    def test_env_vars_captured(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        env = result.result["env_vars"]
+        assert env["HSA_FORCE_FINE_GRAIN_PCIE"] == "1"
+        assert env["MIOPEN_FIND_MODE"] == "3"
+        assert env["NCCL_MIN_NCHANNELS"] == "112"
+
+    def test_lora_rank_extracted(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        assert result.result["lora_rank"] == 16
+
+    def test_attention_impl_from_from_pretrained(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        assert result.result["attention_impl"] == "eager"
+
+    def test_model_name_resolved(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        assert result.result["model_name"] == "Qwen/Qwen2.5-7B-Instruct"
+
+
+# ---------------------------------------------------------------------------
+# JSON path
+# ---------------------------------------------------------------------------
+
+
+class TestJsonConfig:
+    def test_returns_ok(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.json"))
+        assert result.ok, result.error
+
+    def test_field_mapping(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.json")).result
+        assert cfg["model_name"] == "Qwen/Qwen2.5-7B-Instruct"
+        assert cfg["batch_size"] == 8
+        assert cfg["grad_accum_steps"] == 4
+        assert cfg["seq_len"] == 4096
+        assert cfg["precision"] == "bf16"
+        assert cfg["optimizer"] == "adamw_torch_fused"
+        assert cfg["torch_compile"] is True
+        assert cfg["gradient_checkpointing"] is True
+        assert cfg["dataloader_workers"] == 4
+        assert cfg["dataloader_pin_memory"] is True
+        assert cfg["dataloader_persistent_workers"] is True
+        assert cfg["attention_impl"] == "flash"
+
+    def test_env_vars_dict(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.json")).result
+        assert cfg["env_vars"]["HSA_FORCE_FINE_GRAIN_PCIE"] == "1"
+
+    def test_extras_collects_unmapped_fields(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.json")).result
+        # num_train_epochs has no slot in WorkloadConfig — must land in extras.
+        assert cfg["extras"]["num_train_epochs"] == 3
+        assert cfg["extras"]["save_steps"] == 500
+
+
+# ---------------------------------------------------------------------------
+# YAML path
+# ---------------------------------------------------------------------------
+
+
+class TestYamlConfig:
+    def test_returns_ok(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.yaml"))
+        assert result.ok, result.error
+
+    def test_field_mapping(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.yaml")).result
+        assert cfg["batch_size"] == 2
+        assert cfg["grad_accum_steps"] == 16
+        assert cfg["seq_len"] == 8192
+        assert cfg["precision"] == "bf16"
+        assert cfg["torch_compile"] is False
+        assert cfg["gradient_checkpointing"] is True
+        assert cfg["attention_impl"] == "sdpa"
+        assert cfg["dataloader_workers"] == 8
+        assert cfg["dataloader_persistent_workers"] is True
+
+
+# ---------------------------------------------------------------------------
+# Redaction
+# ---------------------------------------------------------------------------
+
+
+class TestRedaction:
+    def test_python_script_redactions(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.py")).result
+        labels = set(cfg["redactions"])
+        # Every secret pattern in sample_train.py should fire.
+        assert "hf_token" in labels
+        assert "openai_key" in labels
+        assert "github_token" in labels
+        assert "bearer_token" in labels
+        assert "home_path" in labels
+        assert "s3_uri" in labels
+        assert "ws_uri" in labels
+
+    def test_raw_source_is_scrubbed(self) -> None:
+        # `raw_source` is intentionally stripped from the tool result envelope
+        # (keeps the LLM conversation small) — use the `_full` helper to read
+        # it. The redaction labels list still proves which patterns fired.
+        cfg = _parse_config_full(str(FIXTURES / "sample_train.py"))
+        assert isinstance(cfg, WorkloadConfig)
+        raw = cfg.raw_source
+        assert "hf_abcdefghijklmnopqrstuvwxyz123456" not in raw
+        assert "sk-abcdefghijklmnopqrstuvwxyz1234567890" not in raw
+        assert "gho_abcdefghijklmnopqrstuvwxyz123456" not in raw
+        assert "/home/researcher/datasets/alpaca" not in raw
+        assert "s3://my-team/checkpoints/qwen-lora/" not in raw
+        assert "wss://logs.internal.example.com/stream" not in raw
+        assert "<REDACTED:hf_token>" in raw
+        assert "<REDACTED:openai_key>" in raw
+
+    def test_raw_source_excluded_from_tool_result(self) -> None:
+        # The tool result MUST NOT carry raw_source — it bloated the audit
+        # conversation past 8K on Qwen2.5-7B during the live AMD GPU run.
+        cfg = _parse_config(str(FIXTURES / "sample_train.py")).result
+        assert "raw_source" not in cfg
+
+    def test_json_redactions(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.json")).result
+        labels = set(cfg["redactions"])
+        assert "hf_token" in labels
+        assert "s3_uri" in labels
+        # raw_source is no longer in the result; verify scrubbing via the
+        # full-config helper.
+        full = _parse_config_full(str(FIXTURES / "sample_train.json"))
+        assert isinstance(full, WorkloadConfig)
+        assert "hf_jsonsamplehfabcdefghijklmnopqrs" not in full.raw_source
+
+    def test_extras_values_are_scrubbed(self) -> None:
+        # Secret-shaped values that landed in extras must also be redacted —
+        # otherwise the leak just moves from raw_source into extras.
+        cfg = _parse_config(str(FIXTURES / "sample_train.json")).result
+        extras = cfg["extras"]
+        assert "hf_jsonsamplehfabcdefghijklmnopqrs" not in extras.get("hub_token", "")
+        assert extras.get("hub_token", "").startswith("<REDACTED:")
+        assert extras.get("checkpoint_uri", "").startswith("<REDACTED:")
+
+    def test_yaml_redactions(self) -> None:
+        cfg = _parse_config(str(FIXTURES / "sample_train.yaml")).result
+        labels = set(cfg["redactions"])
+        assert "hf_token" in labels
+        assert "bearer_token" in labels
+        assert "home_path" in labels
+
+
+# ---------------------------------------------------------------------------
+# Failure modes
+# ---------------------------------------------------------------------------
+
+
+class TestErrors:
+    def test_missing_file(self) -> None:
+        result = _parse_config("/tmp/definitely-does-not-exist-xyz.py")
+        assert result.ok is False
+        assert "not found" in (result.error or "").lower()
+
+    def test_unsupported_extension(self, tmp_path: Path) -> None:
+        bad = tmp_path / "config.toml"
+        bad.write_text("model_name = 'foo'\n")
+        result = _parse_config(str(bad))
+        assert result.ok is False
+        assert "unsupported" in (result.error or "").lower()
+
+    def test_malformed_python(self, tmp_path: Path) -> None:
+        bad = tmp_path / "broken.py"
+        bad.write_text("def oops(:\n  pass\n")
+        result = _parse_config(str(bad))
+        assert result.ok is False
+        assert "parse error" in (result.error or "").lower()
+
+    def test_malformed_json(self, tmp_path: Path) -> None:
+        bad = tmp_path / "broken.json"
+        bad.write_text("{not really json")
+        result = _parse_config(str(bad))
+        assert result.ok is False
+        assert "json" in (result.error or "").lower()
+
+    def test_json_top_level_must_be_dict(self, tmp_path: Path) -> None:
+        bad = tmp_path / "list.json"
+        bad.write_text(json.dumps([{"foo": 1}]))
+        result = _parse_config(str(bad))
+        assert result.ok is False
+
+    def test_yaml_top_level_must_be_mapping(self, tmp_path: Path) -> None:
+        bad = tmp_path / "scalar.yaml"
+        bad.write_text("- 1\n- 2\n")
+        result = _parse_config(str(bad))
+        assert result.ok is False
+
+
+# ---------------------------------------------------------------------------
+# Schema invariants
+# ---------------------------------------------------------------------------
+
+
+class TestSchema:
+    def test_result_round_trips_through_workload_config(self) -> None:
+        result = _parse_config(str(FIXTURES / "sample_train.py"))
+        # Must be reconstructible — guards against extras-vs-fields collisions.
+        cfg = WorkloadConfig(**result.result)
+        assert cfg.model_name == "Qwen/Qwen2.5-7B-Instruct"
+
+    def test_defaults_when_field_absent(self, tmp_path: Path) -> None:
+        # Minimal config — only model_name. Everything else should fall back to schema defaults.
+        path = tmp_path / "tiny.json"
+        path.write_text(json.dumps({"model_name": "test/tiny"}))
+        result = _parse_config(str(path))
+        assert result.ok
+        cfg = result.result
+        assert cfg["batch_size"] == 1
+        assert cfg["precision"] == "fp16"
+        assert cfg["optimizer"] == "adamw_torch"
+        assert cfg["gradient_checkpointing"] is False
+        assert cfg["redactions"] == []
+
+    def test_tool_definition_unchanged_in_shape(self) -> None:
+        # The Tool definition should still expose name/description/input_schema/fn.
+        assert PARSE_CONFIG.name == "parse_config"
+        assert PARSE_CONFIG.fn is _parse_config
+        assert "file_path" in PARSE_CONFIG.input_schema["properties"]
+
+
+# ---------------------------------------------------------------------------
+# Regression: canonical + scenario workloads must parse with all the right
+# audit-relevant fields. These are what the live agent actually sees, so a
+# regression here directly degrades audit quality (the agent reasons over
+# HF defaults instead of the script's settings).
+# ---------------------------------------------------------------------------
+
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+
+
+class TestCanonicalWorkload:
+    def test_canonical_workload_extracts_full_config(self) -> None:
+        """The canonical demo workload must yield batch_size=4, lr=2e-4, etc.
+        — not HF defaults. Catches the `**dict_var` splat regression where
+        every TrainingArguments kwarg disappears.
+        """
+        result = _parse_config(str(REPO_ROOT / "workloads" / "train_qwen_lora.py"))
+        assert result.ok, result.error
+        cfg = result.result
+        assert cfg["model_name"] == "Qwen/Qwen2.5-7B-Instruct"
+        assert cfg["batch_size"] == 4, (
+            "expected batch_size=4 from per_device_train_batch_size; "
+            "did `**_ta_kwargs` splat hide the kwargs?"
+        )
+        assert cfg["grad_accum_steps"] == 8
+        assert cfg["lr"] == 2e-4
+        assert cfg["warmup_steps"] == 100
+        assert cfg["precision"] == "fp16"
+        assert cfg["attention_impl"] == "eager"
+        assert cfg["dataloader_workers"] == 0
+        assert cfg["dataloader_pin_memory"] is False
+        assert cfg["lora_rank"] == 16
+        assert cfg["torch_compile"] is False
+        assert cfg["env_vars"]["HSA_FORCE_FINE_GRAIN_PCIE"] == "1"
+
+
+class TestSplatKwargsResolution:
+    """`_ta = dict(k=v); Foo(**_ta)` must resolve back through the dict
+    constant. Defensive — the canonical workload no longer uses this
+    pattern, but third-party scripts often do.
+    """
+
+    def test_dict_function_call_splat(self, tmp_path) -> None:
+        src = """
+from transformers import TrainingArguments
+
+_ta = dict(
+    per_device_train_batch_size=8,
+    gradient_accumulation_steps=2,
+    fp16=True,
+    optim=\"adamw_torch_fused\",
+)
+training_args = TrainingArguments(output_dir=\"./out\", **_ta)
+"""
+        p = tmp_path / "splat.py"
+        p.write_text(src)
+        cfg = _parse_config(str(p)).result
+        assert cfg["batch_size"] == 8
+        assert cfg["grad_accum_steps"] == 2
+        assert cfg["precision"] == "fp16"
+        assert cfg["optimizer"] == "adamw_torch_fused"
+
+    def test_dict_literal_splat(self, tmp_path) -> None:
+        src = """
+from transformers import TrainingArguments
+
+_ta = {
+    "per_device_train_batch_size": 16,
+    "bf16": True,
+}
+training_args = TrainingArguments(output_dir=\"./out\", **_ta)
+"""
+        p = tmp_path / "splat_literal.py"
+        p.write_text(src)
+        cfg = _parse_config(str(p)).result
+        assert cfg["batch_size"] == 16
+        assert cfg["precision"] == "bf16"
+
+    def test_explicit_kwarg_overrides_splat(self, tmp_path) -> None:
+        src = """
+from transformers import TrainingArguments
+
+_ta = dict(per_device_train_batch_size=8)
+training_args = TrainingArguments(per_device_train_batch_size=32, **_ta)
+"""
+        p = tmp_path / "splat_override.py"
+        p.write_text(src)
+        cfg = _parse_config(str(p)).result
+        # Explicit kwarg wins over splat (both occur in the kwargs list,
+        # explicit comes first in the AST → setdefault keeps it).
+        assert cfg["batch_size"] == 32
+

tests/test_query_rocm_kb.py

@@ -0,0 +1,348 @@
+"""Tests for ``agent.tools.query_rocm_kb``.
+
+Coverage:
+  * The shipped ``kb/rocm_rules.yaml`` is loadable, validates against
+    :class:`Rule`, and contains every required-by-spec rule id.
+  * Each rule's ``targets_bucket`` is a valid :class:`WasteBucket` and its
+    ``category`` is a valid :class:`RuleCategory` (caught for free by
+    pydantic, but assert here for a clearer failure when the YAML drifts).
+  * Semantic search returns the bf16 rule first for an "fp16 on MI300X"
+    query and the flash-attn / sdpa rules first for an attention query.
+  * Bad inputs (empty symptom, ``top_k <= 0``, ``top_k`` larger than the
+    rule count) are handled gracefully via ``ToolResult.ok=False`` or
+    clamped to the rule count.
+  * Embeddings cache: the cache file is created on first import, and a
+    second ``_embed_rules`` call with the same YAML bytes hits the cache
+    without re-encoding.
+  * The frozen ``Tool`` definition retains its ``name``, ``description``,
+    and ``input_schema`` shape — the agent registry depends on these.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import get_args
+
+import pytest
+import yaml
+
+from agent.schemas import Rule, RuleCategory, ToolResult, WasteBucket
+from agent.tools.query_rocm_kb import (
+    _KB_YAML,
+    _RULES,
+    QUERY_ROCM_KB,
+    _cache_path,
+    _embed_rules,
+    _load_rules,
+    _query_rocm_kb,
+)
+
+
+REQUIRED_RULE_IDS = {
+    "precision.bf16_over_fp16_on_mi300x",
+    "attention.flash_rocm_over_eager",
+    "attention.sdpa_over_eager",
+    "memory.batch_too_small_for_192gb",
+    "memory.gradient_checkpointing_for_long_seq",
+    "data.dataloader_workers_zero",
+    "data.pin_memory_false",
+    "data.prefetch_factor_default",
+    "data.persistent_workers_false",
+    "compile.torch_compile_off",
+    "env.nccl_min_nchannels",
+    "env.numa_auto_balancing_disable",
+    "env.hsa_force_fine_grain_pcie",
+    "kernels.hipblaslt_hint_logging",
+    "kernels.miopen_find_mode_2",
+    "optimizer.bitsandbytes_not_supported_warning",
+    "collectives.one_process_per_gpu",
+    "topology.tp_within_xgmi_island",
+}
+
+
+# ---------------------------------------------------------------------------
+# YAML KB invariants
+# ---------------------------------------------------------------------------
+
+
+class TestYamlKB:
+    def test_kb_yaml_exists(self) -> None:
+        assert _KB_YAML.exists(), f"Missing {_KB_YAML}"
+
+    def test_kb_loads_and_validates(self) -> None:
+        rules, _raw = _load_rules(_KB_YAML)
+        # Every rule pydantic-validated.
+        for r in rules:
+            assert isinstance(r, Rule)
+        # Spec calls for 20-25 rules; allow a small wiggle room either way.
+        assert 18 <= len(rules) <= 30, f"Expected 18-30 rules, got {len(rules)}"
+
+    def test_required_rule_ids_present(self) -> None:
+        ids = {r.id for r in _RULES}
+        missing = REQUIRED_RULE_IDS - ids
+        assert not missing, f"Required rule ids missing from KB: {sorted(missing)}"
+
+    def test_rule_ids_unique(self) -> None:
+        ids = [r.id for r in _RULES]
+        dupes = {i for i in ids if ids.count(i) > 1}
+        assert not dupes, f"Duplicate rule ids: {sorted(dupes)}"
+
+    def test_categories_are_valid(self) -> None:
+        valid = set(get_args(RuleCategory))
+        for r in _RULES:
+            assert r.category in valid, f"{r.id}: invalid category {r.category!r}"
+
+    def test_targets_bucket_valid(self) -> None:
+        valid = set(get_args(WasteBucket))
+        for r in _RULES:
+            assert r.targets_bucket in valid, (
+                f"{r.id}: invalid targets_bucket {r.targets_bucket!r}"
+            )
+
+    def test_recovery_fraction_in_range(self) -> None:
+        for r in _RULES:
+            assert 0.0 <= r.expected_recovery_fraction <= 1.0, (
+                f"{r.id}: expected_recovery_fraction out of [0,1] "
+                f"({r.expected_recovery_fraction})"
+            )
+
+    def test_citations_non_empty(self) -> None:
+        for r in _RULES:
+            assert r.citation and r.citation.strip(), f"{r.id}: empty citation"
+
+    def test_bitsandbytes_is_warning_only(self) -> None:
+        rule = next(r for r in _RULES if r.id == "optimizer.bitsandbytes_not_supported_warning")
+        # Warning rule has empty transform — propose_patch must not auto-fix.
+        assert rule.transform == {}
+
+    def test_categories_cover_spec(self) -> None:
+        # Architecture §4 lists 10 categories. We expect rules in at least
+        # the high-impact ones the spec calls out as required.
+        cats = {r.category for r in _RULES}
+        for required in (
+            "precision",
+            "attention",
+            "memory",
+            "data",
+            "compile",
+            "env_vars",
+            "kernels",
+            "optimizer",
+            "collectives",
+            "topology",
+        ):
+            assert required in cats, f"No rule for category {required!r}"
+
+
+# ---------------------------------------------------------------------------
+# Skip-and-warn behaviour for invalid entries
+# ---------------------------------------------------------------------------
+
+
+class TestLoadRulesResilience:
+    def test_invalid_entry_is_skipped_with_warning(self, tmp_path: Path) -> None:
+        bad = tmp_path / "rules.yaml"
+        bad.write_text(
+            yaml.safe_dump(
+                [
+                    {
+                        "id": "good.rule",
+                        "category": "precision",
+                        "targets_bucket": "precision_path",
+                        "symptom": "fp16 used",
+                        "expected_impact": "switch to bf16",
+                        "citation": "ROCm guide",
+                    },
+                    {"id": "bad.rule", "category": "not_a_real_category"},
+                ]
+            )
+        )
+        with pytest.warns(UserWarning):
+            rules, _raw = _load_rules(bad)
+        assert [r.id for r in rules] == ["good.rule"]
+
+    def test_top_level_must_be_list(self, tmp_path: Path) -> None:
+        bad = tmp_path / "rules.yaml"
+        bad.write_text("not_a_list: 1\n")
+        with pytest.raises(ValueError, match="top-level"):
+            _load_rules(bad)
+
+
+# ---------------------------------------------------------------------------
+# Semantic search behaviour
+# ---------------------------------------------------------------------------
+
+
+class TestQuery:
+    def test_returns_ok_for_real_query(self) -> None:
+        result = _query_rocm_kb("fp16 used on MI300X with eager attention", top_k=5)
+        assert isinstance(result, ToolResult)
+        assert result.ok, result.error
+        rules = result.result["rules"]
+        assert 1 <= len(rules) <= 5
+
+    def test_fp16_query_returns_bf16_rule_in_top_results(self) -> None:
+        result = _query_rocm_kb("fp16 used on MI300X / CDNA3", top_k=3)
+        ids = [r["id"] for r in result.result["rules"]]
+        assert "precision.bf16_over_fp16_on_mi300x" in ids
+
+    def test_eager_attention_query_returns_attention_rules(self) -> None:
+        result = _query_rocm_kb(
+            "eager attention with no flash kernel loaded on MI300X", top_k=3
+        )
+        ids = [r["id"] for r in result.result["rules"]]
+        attention_ids = {
+            "attention.flash_rocm_over_eager",
+            "attention.sdpa_over_eager",
+        }
+        assert attention_ids & set(ids), (
+            f"Expected at least one of {attention_ids} in top 3, got {ids}"
+        )
+
+    def test_dataloader_query_returns_data_rules(self) -> None:
+        result = _query_rocm_kb(
+            "DataLoader num_workers is zero, GPU starves waiting for batches",
+            top_k=3,
+        )
+        ids = [r["id"] for r in result.result["rules"]]
+        data_ids = {
+            "data.dataloader_workers_zero",
+            "data.pin_memory_false",
+            "data.prefetch_factor_default",
+            "data.persistent_workers_false",
+        }
+        assert data_ids & set(ids), (
+            f"Expected at least one data.* rule in top 3, got {ids}"
+        )
+
+    def test_top_k_bounds_returned(self) -> None:
+        result = _query_rocm_kb("anything", top_k=2)
+        assert len(result.result["rules"]) == 2
+
+    def test_top_k_clamped_to_rule_count(self) -> None:
+        # Asking for more than we have should not crash; we should get every
+        # rule back, ordered by score.
+        result = _query_rocm_kb("anything", top_k=100)
+        assert len(result.result["rules"]) == len(_RULES)
+
+    def test_results_sorted_by_descending_score(self) -> None:
+        # If two queries with different focus return different top rules,
+        # ordering is real — top_1 differs by query.
+        a = _query_rocm_kb("fp16 numerical instability", top_k=1).result["rules"][0]
+        b = _query_rocm_kb("eager attention slow on long sequences", top_k=1).result[
+            "rules"
+        ][0]
+        assert a["id"] != b["id"], (
+            "Top rule should depend on query, but both queries returned "
+            f"{a['id']} — semantic search is degenerate."
+        )
+
+    def test_rule_payload_is_lite_shape(self) -> None:
+        # The LLM-facing rule payload is intentionally trimmed from the full
+        # Rule schema — only id / symptom / transform / expected_impact /
+        # citation make it through. This shrinks the audit conversation enough
+        # to fit Qwen2.5-7B's 8K window. Full Rule lookup happens server-side
+        # in propose_patch via the loaded KB.
+        result = _query_rocm_kb("any query", top_k=1)
+        payload = result.result["rules"][0]
+        assert set(payload.keys()) == {
+            "id",
+            "symptom",
+            "transform",
+            "expected_impact",
+            "citation",
+        }
+        # And the id resolves against the loaded KB so propose_patch can
+        # reconstruct the full Rule.
+        from agent.tools.query_rocm_kb import _RULES
+
+        kb_ids = {r.id for r in _RULES}
+        assert payload["id"] in kb_ids
+
+
+# ---------------------------------------------------------------------------
+# Failure modes
+# ---------------------------------------------------------------------------
+
+
+class TestErrors:
+    def test_empty_symptom(self) -> None:
+        result = _query_rocm_kb("", top_k=3)
+        assert result.ok is False
+        assert "symptom" in (result.error or "")
+
+    def test_whitespace_only_symptom(self) -> None:
+        result = _query_rocm_kb("   \t\n  ", top_k=3)
+        assert result.ok is False
+
+    def test_top_k_zero(self) -> None:
+        result = _query_rocm_kb("fp16", top_k=0)
+        assert result.ok is False
+        assert "top_k" in (result.error or "")
+
+    def test_top_k_negative(self) -> None:
+        result = _query_rocm_kb("fp16", top_k=-3)
+        assert result.ok is False
+
+
+# ---------------------------------------------------------------------------
+# Embeddings cache
+# ---------------------------------------------------------------------------
+
+
+class TestEmbeddingsCache:
+    def test_cache_file_was_created_on_import(self) -> None:
+        raw = _KB_YAML.read_bytes()
+        cache = _cache_path(raw)
+        assert cache.exists(), (
+            f"Expected embeddings cache at {cache}; cache write failed silently."
+        )
+
+    def test_second_embed_call_uses_cache_without_recoding(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        rules, raw = _load_rules(_KB_YAML)
+        # Sentinel: replace the lazy model getter so any encode() call would
+        # blow up. If the cache is hit, the model is never consulted.
+        from agent.tools import query_rocm_kb as kb_module
+
+        def explode() -> None:
+            raise AssertionError(
+                "_get_model() called on a cache-hit path; cache is not being used."
+            )
+
+        monkeypatch.setattr(kb_module, "_get_model", explode)
+        embeddings = _embed_rules(rules, raw)
+        assert embeddings.shape[0] == len(rules)
+        assert embeddings.dtype.kind == "f"
+
+
+# ---------------------------------------------------------------------------
+# Tool registry shape — the agent loop depends on these fields being stable.
+# ---------------------------------------------------------------------------
+
+
+class TestToolDefinition:
+    def test_name_is_query_rocm_kb(self) -> None:
+        assert QUERY_ROCM_KB.name == "query_rocm_kb"
+
+    def test_description_unchanged_keywords(self) -> None:
+        # Must still describe the search-by-symptom semantics; the system
+        # prompt references this language.
+        desc = QUERY_ROCM_KB.description
+        assert "ROCm" in desc and "symptom" in desc
+
+    def test_input_schema_shape(self) -> None:
+        schema = QUERY_ROCM_KB.input_schema
+        assert schema["type"] == "object"
+        # Both single-query (`symptom`) and batched (`symptoms`) shapes are
+        # advertised — either works at runtime, neither is strictly required
+        # in the schema because the impl validates "at least one" itself.
+        assert "symptom" in schema["properties"]
+        assert "symptoms" in schema["properties"]
+        assert "top_k" in schema["properties"]
+        assert schema["properties"]["symptoms"]["type"] == "array"
+        assert schema["properties"]["symptoms"]["items"] == {"type": "string"}
+
+    def test_fn_is_module_query(self) -> None:
+        assert QUERY_ROCM_KB.fn is _query_rocm_kb

tests/test_qwen_vllm_backend.py

@@ -0,0 +1,280 @@
+"""Tests for QwenVLLMBackend.
+
+Mocks the openai AsyncClient at the chat.completions level so no real
+network call ever happens. Verifies the OpenAI-shape conversation
+threading, tool-call extraction, and finish_reason mapping.
+"""
+
+from __future__ import annotations
+
+import os
+from types import SimpleNamespace
+from typing import Any
+
+import pytest
+
+from agent.backends import active_backend_name, make_backend
+from agent.backends.base import AgentTurn
+from agent.backends.qwen_vllm import (
+    QwenVLLMBackend,
+    _normalize_finish_reason,
+    _to_openai_tools,
+)
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+class _FakeChatCompletions:
+    """Stands in for ``client.chat.completions``. Records every call and
+    returns scripted responses one-by-one."""
+
+    def __init__(self, responses: list[Any]) -> None:
+        self.responses = list(responses)
+        self.calls: list[dict[str, Any]] = []
+
+    async def create(self, **kwargs: Any) -> Any:
+        self.calls.append(kwargs)
+        if not self.responses:
+            raise AssertionError(
+                "FakeChatCompletions exhausted — backend made more calls than expected"
+            )
+        return self.responses.pop(0)
+
+
+class _FakeClient:
+    def __init__(self, responses: list[Any]) -> None:
+        self.chat = SimpleNamespace(completions=_FakeChatCompletions(responses))
+
+
+def _scripted_response(
+    *,
+    content: str | None = None,
+    tool_calls: list[dict[str, Any]] | None = None,
+    finish_reason: str = "stop",
+) -> Any:
+    """Build the SimpleNamespace shape the openai SDK returns from
+    chat.completions.create(...).
+    """
+    tcs = []
+    if tool_calls:
+        for tc in tool_calls:
+            tcs.append(
+                SimpleNamespace(
+                    id=tc["id"],
+                    function=SimpleNamespace(
+                        name=tc["name"],
+                        arguments=tc.get("arguments", "{}"),
+                    ),
+                )
+            )
+    msg = SimpleNamespace(content=content, tool_calls=tcs or None)
+    choice = SimpleNamespace(message=msg, finish_reason=finish_reason)
+    return SimpleNamespace(choices=[choice])
+
+
+def _backend_with(responses: list[Any]) -> QwenVLLMBackend:
+    """Construct a QwenVLLMBackend with a scripted client."""
+    backend = QwenVLLMBackend.__new__(QwenVLLMBackend)
+    backend._system = "you are a test agent"
+    backend._model = "Qwen/Qwen2.5-7B-Instruct"
+    backend._base_url = "http://fake-vllm:8000/v1"
+    backend._api_key = "EMPTY"
+    backend._max_tokens = 1024
+    backend._client = _FakeClient(responses)
+    backend._conversation = [{"role": "system", "content": backend._system}]
+    return backend
+
+
+# ---------------------------------------------------------------------------
+# Tool-schema translation
+# ---------------------------------------------------------------------------
+
+
+def test_to_openai_tools_translates_neutral_shape() -> None:
+    neutral = [
+        {
+            "name": "parse_config",
+            "description": "Parse the file.",
+            "input_schema": {"type": "object", "properties": {"file_path": {"type": "string"}}},
+        },
+    ]
+    out = _to_openai_tools(neutral)
+    assert out == [
+        {
+            "type": "function",
+            "function": {
+                "name": "parse_config",
+                "description": "Parse the file.",
+                "parameters": {
+                    "type": "object",
+                    "properties": {"file_path": {"type": "string"}},
+                },
+            },
+        }
+    ]
+
+
+def test_to_openai_tools_handles_missing_input_schema() -> None:
+    out = _to_openai_tools([{"name": "x", "description": "y"}])
+    assert out[0]["function"]["parameters"] == {"type": "object", "properties": {}}
+
+
+def test_finish_reason_normalization() -> None:
+    assert _normalize_finish_reason("stop") == "end_turn"
+    assert _normalize_finish_reason("tool_calls") == "tool_use"
+    assert _normalize_finish_reason("length") == "max_tokens"
+    assert _normalize_finish_reason(None) == "other"
+    assert _normalize_finish_reason("weird") == "weird"
+
+
+# ---------------------------------------------------------------------------
+# next_turn behavior
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_next_turn_emits_text_block_and_end_turn() -> None:
+    backend = _backend_with([_scripted_response(content="hello there", finish_reason="stop")])
+    backend.add_user_message("audit /tmp/x.py")
+    turn = await backend.next_turn(tool_schemas=[])
+    assert isinstance(turn, AgentTurn)
+    assert turn.text_blocks == ["hello there"]
+    assert turn.tool_calls == []
+    assert turn.stop_reason == "end_turn"
+
+
+@pytest.mark.asyncio
+async def test_next_turn_emits_tool_calls_with_parsed_args() -> None:
+    backend = _backend_with(
+        [
+            _scripted_response(
+                content="calling parse_config",
+                tool_calls=[
+                    {
+                        "id": "tc-1",
+                        "name": "parse_config",
+                        "arguments": '{"file_path": "/tmp/x.py"}',
+                    }
+                ],
+                finish_reason="tool_calls",
+            )
+        ]
+    )
+    backend.add_user_message("audit /tmp/x.py")
+    turn = await backend.next_turn(tool_schemas=[])
+    assert turn.stop_reason == "tool_use"
+    assert len(turn.tool_calls) == 1
+    tc = turn.tool_calls[0]
+    assert tc.id == "tc-1"
+    assert tc.name == "parse_config"
+    assert tc.input == {"file_path": "/tmp/x.py"}
+
+
+@pytest.mark.asyncio
+async def test_next_turn_handles_malformed_tool_arguments() -> None:
+    """vLLM occasionally emits unparseable JSON in arguments — don't crash."""
+    backend = _backend_with(
+        [
+            _scripted_response(
+                tool_calls=[
+                    {"id": "tc-1", "name": "parse_config", "arguments": "{not-json"}
+                ],
+                finish_reason="tool_calls",
+            )
+        ]
+    )
+    backend.add_user_message("x")
+    turn = await backend.next_turn(tool_schemas=[])
+    # We get the call but with empty args rather than raising.
+    assert turn.tool_calls[0].input == {}
+
+
+@pytest.mark.asyncio
+async def test_tool_result_is_threaded_into_next_request() -> None:
+    backend = _backend_with(
+        [
+            _scripted_response(
+                tool_calls=[
+                    {"id": "tc-1", "name": "parse_config", "arguments": "{}"}
+                ],
+                finish_reason="tool_calls",
+            ),
+            _scripted_response(content="done", finish_reason="stop"),
+        ]
+    )
+    backend.add_user_message("audit")
+    await backend.next_turn(tool_schemas=[])
+    backend.add_tool_result("tc-1", "parse_config", '{"ok": true}', is_error=False)
+    await backend.next_turn(tool_schemas=[])
+
+    # The second create() call should include role="tool" referencing tc-1.
+    second_call = backend._client.chat.completions.calls[1]
+    msgs = second_call["messages"]
+    tool_msgs = [m for m in msgs if m["role"] == "tool"]
+    assert len(tool_msgs) == 1
+    assert tool_msgs[0]["tool_call_id"] == "tc-1"
+    assert tool_msgs[0]["content"] == '{"ok": true}'
+
+
+@pytest.mark.asyncio
+async def test_is_error_prefix_added_to_failed_tool_results() -> None:
+    backend = _backend_with([_scripted_response(content="adapting", finish_reason="stop")])
+    backend.add_tool_result("tc-1", "parse_config", "file not found", is_error=True)
+    await backend.next_turn(tool_schemas=[])
+    msgs = backend._client.chat.completions.calls[0]["messages"]
+    tool_msg = next(m for m in msgs if m["role"] == "tool")
+    assert tool_msg["content"].startswith("ERROR:")
+
+
+# ---------------------------------------------------------------------------
+# Factory selection via env var
+# ---------------------------------------------------------------------------
+
+
+def test_make_backend_picks_vllm_when_env_var_set(monkeypatch) -> None:
+    monkeypatch.setenv("GOBLIN_AGENT_BACKEND", "qwen-vllm")
+    monkeypatch.setenv("GOBLIN_QWEN_VLLM_URL", "http://test:8000/v1")
+    assert active_backend_name() == "qwen-vllm"
+    backend = make_backend(system_prompt="x")
+    assert isinstance(backend, QwenVLLMBackend)
+    assert backend._base_url == "http://test:8000/v1"
+
+
+def test_active_backend_name_aliases(monkeypatch) -> None:
+    for alias in ("qwen-vllm", "qwen_vllm", "vllm", "local", "QWEN-VLLM", "Vllm"):
+        monkeypatch.setenv("GOBLIN_AGENT_BACKEND", alias)
+        assert active_backend_name() == "qwen-vllm", alias
+    for alias in ("qwen-hf", "qwen", "hf", ""):
+        monkeypatch.setenv("GOBLIN_AGENT_BACKEND", alias)
+        assert active_backend_name() == "qwen-hf", alias
+
+
+def test_make_backend_default_is_hf(monkeypatch) -> None:
+    monkeypatch.delenv("GOBLIN_AGENT_BACKEND", raising=False)
+    monkeypatch.setenv("HF_TOKEN", "fake-test-token")
+    assert active_backend_name() == "qwen-hf"
+
+
+# ---------------------------------------------------------------------------
+# Construction-time error: openai SDK missing
+# ---------------------------------------------------------------------------
+
+
+def test_constructor_raises_when_openai_not_installed(monkeypatch) -> None:
+    """If openai isn't on PYTHONPATH, the backend's _build_client raises a
+    clear RuntimeError instead of an opaque ImportError."""
+    import sys
+
+    saved = sys.modules.pop("openai", None)
+    sys.modules["openai"] = None  # block re-import
+    try:
+        with pytest.raises(RuntimeError, match="openai"):
+            QwenVLLMBackend(system_prompt="x")
+    finally:
+        if saved is not None:
+            sys.modules["openai"] = saved
+        else:
+            sys.modules.pop("openai", None)

tests/test_runner.py

@@ -0,0 +1,311 @@
+"""Tests for runner/protocol.py and runner/profile_parser.py.
+
+Two laptop-only invariants:
+  1. FakeRunner still works exactly as before (the Phase 1 contract).
+  2. LiveRunner gracefully falls back to FakeRunner whenever GPU/profiler
+     tools are missing — this dev box has no AMD GPU, so every test here
+     should exercise the fallback path.
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+from unittest import mock
+
+import pytest
+
+from agent.schemas import RunMetrics, WorkloadConfig
+from runner import profile_parser
+from runner.protocol import FakeRunner, LiveRunner, _default_runner, gpu_available
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _baseline_config() -> WorkloadConfig:
+    return WorkloadConfig(
+        model_name="Qwen/Qwen2.5-7B-Instruct",
+        precision="fp16",
+        batch_size=4,
+        attention_impl="eager",
+        dataloader_workers=0,
+    )
+
+
+# ---------------------------------------------------------------------------
+# FakeRunner — unchanged contract
+# ---------------------------------------------------------------------------
+
+
+class TestFakeRunner:
+    def test_matches_baseline_scenario(self):
+        runner = FakeRunner()
+        metrics = runner.run(_baseline_config(), steps=10)
+        assert isinstance(metrics, RunMetrics)
+        assert metrics.runner_kind == "fake"
+        assert metrics.steps == 10
+        # 01_baseline_bad fixture
+        assert metrics.tokens_per_sec == pytest.approx(142.0)
+
+    def test_steps_override_takes_precedence(self):
+        runner = FakeRunner()
+        metrics = runner.run(_baseline_config(), steps=99)
+        assert metrics.steps == 99
+
+    def test_default_metrics_when_no_match(self):
+        runner = FakeRunner()
+        # An unknown model_name forces the no-match path.
+        cfg = _baseline_config().model_copy(update={"model_name": "unknown/model"})
+        metrics = runner.run(cfg, steps=7)
+        assert metrics.runner_kind == "fake"
+        assert metrics.steps == 7
+        assert any("FakeRunner" in w for w in metrics.warnings)
+
+    def test_corpus_dir_missing_returns_default(self, tmp_path):
+        runner = FakeRunner(corpus_dir=tmp_path / "nope")
+        metrics = runner.run(_baseline_config(), steps=10)
+        assert metrics.runner_kind == "fake"
+
+
+# ---------------------------------------------------------------------------
+# gpu_available — pure detection
+# ---------------------------------------------------------------------------
+
+
+class TestGpuAvailable:
+    def test_no_rocprofv3(self):
+        with mock.patch("runner.protocol.shutil.which", return_value=None):
+            ok, reason = gpu_available()
+        assert ok is False
+        assert reason and "rocprofv3" in reason
+
+    def test_no_amd_smi(self):
+        def which(name):
+            return "/usr/bin/rocprofv3" if name == "rocprofv3" else None
+
+        with mock.patch("runner.protocol.shutil.which", side_effect=which):
+            ok, reason = gpu_available()
+        assert ok is False
+        assert reason and "amd-smi" in reason
+
+    def test_no_render_device(self):
+        with mock.patch(
+            "runner.protocol.shutil.which",
+            side_effect=lambda name: f"/usr/bin/{name}",
+        ), mock.patch("runner.protocol._has_render_device", return_value=False):
+            ok, reason = gpu_available()
+        assert ok is False
+        assert reason and "renderD" in reason
+
+    def test_all_present(self):
+        with mock.patch(
+            "runner.protocol.shutil.which",
+            side_effect=lambda name: f"/usr/bin/{name}",
+        ), mock.patch("runner.protocol._has_render_device", return_value=True):
+            ok, reason = gpu_available()
+        assert ok is True
+        assert reason is None
+
+
+# ---------------------------------------------------------------------------
+# LiveRunner — must fall back on this no-GPU dev machine
+# ---------------------------------------------------------------------------
+
+
+class TestLiveRunnerFallback:
+    def test_falls_back_when_gpu_unavailable(self):
+        runner = LiveRunner()
+        metrics = runner.run(_baseline_config(), steps=10)
+        # On a laptop, gpu_available() returns False → FakeRunner path.
+        assert metrics.runner_kind == "fake"
+        # The warning must be the FIRST entry (LiveRunner prepends it).
+        assert metrics.warnings, "LiveRunner must surface a fallback warning"
+        assert "LiveRunner" in metrics.warnings[0]
+
+    def test_falls_back_when_runner_script_missing(self, tmp_path):
+        with mock.patch("runner.protocol.gpu_available", return_value=(True, None)):
+            runner = LiveRunner(runner_script=tmp_path / "does_not_exist.sh")
+            metrics = runner.run(_baseline_config(), steps=10)
+        assert metrics.runner_kind == "fake"
+        assert any("runner script not found" in w for w in metrics.warnings)
+
+    def test_falls_back_when_runner_script_not_executable(self, tmp_path):
+        script = tmp_path / "goblin_runner.sh"
+        script.write_text("#!/bin/sh\nexit 0\n")
+        # Deliberately don't chmod +x
+        with mock.patch("runner.protocol.gpu_available", return_value=(True, None)):
+            runner = LiveRunner(runner_script=script)
+            metrics = runner.run(_baseline_config(), steps=10)
+        assert metrics.runner_kind == "fake"
+        assert any("not executable" in w for w in metrics.warnings)
+
+    def test_falls_back_when_subprocess_returns_nonzero(self, tmp_path):
+        script = tmp_path / "goblin_runner.sh"
+        script.write_text("#!/usr/bin/env bash\nexit 7\n")
+        script.chmod(0o755)
+        with mock.patch("runner.protocol.gpu_available", return_value=(True, None)):
+            runner = LiveRunner(runner_script=script)
+            metrics = runner.run(_baseline_config(), steps=10)
+        assert metrics.runner_kind == "fake"
+        assert any("exited with code 7" in w for w in metrics.warnings)
+
+
+# ---------------------------------------------------------------------------
+# _default_runner — module-level factory
+# ---------------------------------------------------------------------------
+
+
+def test_default_runner_is_live_runner():
+    runner = _default_runner()
+    assert isinstance(runner, LiveRunner)
+
+
+# ---------------------------------------------------------------------------
+# profile_parser — graceful degradation when artefacts missing
+# ---------------------------------------------------------------------------
+
+
+class TestProfileParser:
+    def test_empty_dir_returns_zero_metrics_with_warnings(self, tmp_path):
+        metrics = profile_parser.parse(tmp_path, config=_baseline_config(), steps=10)
+        assert metrics.tokens_per_sec == 0.0
+        assert metrics.mfu_pct == 0.0
+        assert metrics.gpu_util_pct == 0.0
+        assert len(metrics.warnings) >= 3  # one warning per missing artefact
+
+    def test_parses_synthetic_artefacts(self, tmp_path):
+        # Minimal rocprofv3-shaped CSV
+        trace = tmp_path / "trace.csv"
+        with trace.open("w", newline="") as f:
+            w = csv.writer(f)
+            w.writerow(["KernelName", "DurationNs"])
+            w.writerow(["aten::matmul (fp16)", 5_000_000])
+            w.writerow(["aten::scaled_dot_product_attention", 3_000_000])
+            w.writerow(["rccl_AllReduce", 1_000_000])
+            w.writerow(["hipBLASLt_generic_gemm", 2_000_000])
+
+        # Minimal torch.profiler chrome trace with embedded metadata
+        torch_profile = {
+            "metadata": {
+                "tokens_per_sec": 142.0,
+                "mfu_pct": 24.0,
+                "pytorch_version": "2.3.0+rocm6.1",
+                "step_time_seconds": 0.5,
+                "host_busy_fraction": 0.6,
+            },
+            "traceEvents": [],
+        }
+        (tmp_path / "torch_profile.json").write_text(json.dumps(torch_profile))
+
+        # Minimal amd-smi telemetry
+        smi = tmp_path / "amd_smi.csv"
+        with smi.open("w", newline="") as f:
+            w = csv.writer(f)
+            w.writerow(["VRAM_USED_GB", "GFX_ACTIVITY"])
+            w.writerow(["72.0", "20.0"])  # < 30% util → triggers data_wait
+            w.writerow(["75.0", "22.0"])
+
+        metrics = profile_parser.parse(tmp_path, config=_baseline_config(), steps=10)
+        assert metrics.tokens_per_sec == pytest.approx(142.0)
+        assert metrics.mfu_pct == pytest.approx(24.0)
+        assert metrics.hbm_peak_gb == pytest.approx(75.0)
+        assert metrics.hbm_avg_gb == pytest.approx(73.5)
+        # comm_excess detected (rccl kernel, 1 ms)
+        assert metrics.waste_budget.comm_excess == pytest.approx(0.001)
+        # data_wait triggered (gpu util < 30, host_busy > 0.5)
+        assert metrics.waste_budget.data_wait > 0.0
+        # precision_path triggered (config.precision='fp16' AND fp16 kernels present)
+        assert metrics.waste_budget.precision_path > 0.0
+        # kernel_shape: generic GEMM detected
+        assert metrics.waste_budget.kernel_shape > 0.0
+        # memory_headroom: 75 GB used << 70% × 192 GB = 134.4 GB → slack
+        assert metrics.waste_budget.memory_headroom > 0.0
+
+    def test_bf16_config_skips_precision_path(self, tmp_path):
+        # Even with fp16-tagged kernels, a bf16 config means precision_path = 0
+        # because the user is already on the optimal precision.
+        trace = tmp_path / "trace.csv"
+        with trace.open("w", newline="") as f:
+            w = csv.writer(f)
+            w.writerow(["KernelName", "DurationNs"])
+            w.writerow(["matmul_fp16_kernel", 5_000_000])
+        torch_profile = {
+            "metadata": {
+                "tokens_per_sec": 318.0,
+                "mfu_pct": 51.0,
+                "step_time_seconds": 0.3,
+                "host_busy_fraction": 0.2,
+            },
+            "traceEvents": [],
+        }
+        (tmp_path / "torch_profile.json").write_text(json.dumps(torch_profile))
+        smi = tmp_path / "amd_smi.csv"
+        smi.write_text("VRAM_USED_GB,GFX_ACTIVITY\n168.0,86.0\n")
+
+        bf16_config = _baseline_config().model_copy(update={"precision": "bf16"})
+        metrics = profile_parser.parse(tmp_path, config=bf16_config, steps=50)
+        assert metrics.waste_budget.precision_path == 0.0
+
+
+# ---------------------------------------------------------------------------
+# Caching — exercise the benchmark tool's cache layer
+# ---------------------------------------------------------------------------
+
+
+class TestBenchmarkCache:
+    """The benchmark tool writes to the real bench_cache/ directory; isolate it."""
+
+    @pytest.fixture(autouse=True)
+    def _isolate_cache(self, tmp_path, monkeypatch):
+        monkeypatch.setattr("agent.tools.benchmark._CACHE_DIR", tmp_path / "bench_cache")
+        # Force ROCM_IMAGE_TAG to a known value so the key is reproducible.
+        monkeypatch.setenv("ROCM_IMAGE_TAG", "test-tag")
+        yield
+
+    def test_cache_hit_on_second_call(self):
+        from agent.tools.benchmark import _benchmark
+
+        cfg = _baseline_config().model_dump()
+        r1 = _benchmark(cfg, steps=50)
+        assert r1.ok
+        # Second call should HIT the cache and warn about it.
+        r2 = _benchmark(cfg, steps=50)
+        assert r2.ok
+        assert any("cache hit" in w for w in r2.result["warnings"])
+
+    def test_force_rerun_bypasses_cache(self):
+        from agent.tools.benchmark import _benchmark
+
+        cfg = _baseline_config().model_dump()
+        _benchmark(cfg, steps=50)
+        r2 = _benchmark(cfg, steps=50, force_rerun=True)
+        assert r2.ok
+        assert not any("cache hit" in w for w in r2.result["warnings"])
+
+    def test_different_steps_invalidate_cache(self):
+        from agent.tools.benchmark import _benchmark
+
+        cfg = _baseline_config().model_dump()
+        _benchmark(cfg, steps=50)
+        r2 = _benchmark(cfg, steps=100)
+        # Same config, different steps → different cache key → cold call.
+        assert not any("cache hit" in w for w in r2.result["warnings"])
+
+    def test_runner_script_change_invalidates_cache(self, tmp_path, monkeypatch):
+        from agent.tools.benchmark import _benchmark
+
+        cfg = _baseline_config().model_dump()
+        _benchmark(cfg, steps=50)
+
+        # Pretend the runner script changed by swapping the path the cache
+        # key reads from. (Simulates "container/runner version bump".)
+        fake_script = tmp_path / "different_runner.sh"
+        fake_script.write_text("# different content\n")
+        monkeypatch.setattr("agent.tools.benchmark._RUNNER_SCRIPT", fake_script)
+
+        r2 = _benchmark(cfg, steps=50)
+        assert not any("cache hit" in w for w in r2.result["warnings"])

ui/auto_tune_ui.py

@@ -0,0 +1,799 @@
+"""GPU Goblin auto-tune UI — Streamlit frontend for scripts/auto_tune.py.
+
+Run:
+    streamlit run ui/auto_tune_ui.py
+
+The UI:
+  1. Form: pick model OR workload path, mode, steps, etc.
+  2. Run: launches scripts/auto_tune.py as a subprocess with --events FILE
+  3. Live progress: tails the events file, renders iteration cards as they
+     arrive, updates the best-tokens/sec metric on every accepted change
+  4. Final report: improvement vs baseline, accepted vs rejected
+     experiments, waste-budget reduction chart, and a copyable diff.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+from typing import Any
+
+# Streamlit only puts ui/ on sys.path, so add the repo root for any helper
+# imports that might land in shared modules later.
+_REPO_ROOT = Path(__file__).resolve().parent.parent
+if str(_REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(_REPO_ROOT))
+
+import altair as alt
+import pandas as pd
+import requests
+import streamlit as st
+
+REPO_ROOT = _REPO_ROOT
+AUTO_TUNE_SCRIPT = REPO_ROOT / "scripts" / "auto_tune.py"
+
+WASTE_BUCKETS = [
+    "useful_gpu",
+    "data_wait",
+    "host_gap",
+    "comm_excess",
+    "memory_headroom",
+    "precision_path",
+    "kernel_shape",
+]
+
+# ---------------------------------------------------------------------------
+# Page setup
+# ---------------------------------------------------------------------------
+
+st.set_page_config(
+    page_title="GPU Goblin — Auto-Tune",
+    page_icon="🔧",
+    layout="wide",
+)
+
+st.title("🔧 GPU Goblin — Auto-Tune")
+st.caption(
+    "Iteratively find the fastest configuration for a fine-tuning workload "
+    "on AMD MI300X. Pick a model, hit Run, watch tokens/sec climb."
+)
+
+
+# ---------------------------------------------------------------------------
+# Form: inputs
+# ---------------------------------------------------------------------------
+
+with st.sidebar:
+    st.header("Run configuration")
+
+    # ---- Backend mode (Local subprocess vs remote GPU server) ----
+    default_backend = os.environ.get("GOBLIN_AUTO_TUNE_URL", "")
+    backend_mode = st.radio(
+        "Backend",
+        options=("Local subprocess", "Remote GPU server"),
+        index=1 if default_backend else 0,
+        help=(
+            "Local: launches scripts/auto_tune.py here (this host needs an MI300X). "
+            "Remote: POSTs to /auto-tune on a FastAPI server you've stood up on "
+            "the GPU host. Use Remote when running the UI on HF Spaces."
+        ),
+    )
+    backend_url = ""
+    if backend_mode == "Remote GPU server":
+        backend_url = st.text_input(
+            "Backend URL",
+            value=default_backend or "http://localhost:8000",
+            help="Base URL of the FastAPI server (no trailing slash). "
+            "/auto-tune is appended automatically.",
+        )
+
+    st.divider()
+    workload_source = st.radio(
+        "Workload source",
+        options=("Model id", "Custom workload script"),
+        index=0,
+        help=(
+            "Model id: auto-generates a baseline workload from the demo "
+            "template (Qwen-style LoRA fine-tune). Custom: point at any "
+            "Python training script."
+        ),
+    )
+
+    model_id = ""
+    workload_path = ""
+    if workload_source == "Model id":
+        model_id = st.text_input(
+            "Model id",
+            value="Qwen/Qwen2.5-7B-Instruct",
+            help=(
+                "Any HuggingFace causal-LM model id. For gated models "
+                "(Llama, etc.) ensure HF_TOKEN is set in the environment."
+            ),
+        )
+    else:
+        workload_path = st.text_input(
+            "Path to workload script (relative to repo root)",
+            value="workloads/train_qwen_lora.py",
+        )
+
+    st.divider()
+    st.subheader("Tuning strategy")
+
+    mode = st.selectbox(
+        "Mode",
+        options=("hardcoded", "llm", "llm-explore"),
+        index=0,
+        help=(
+            "hardcoded: priority-ordered playbook (no API key). "
+            "llm: LLM picks one experiment per iteration (greedy). "
+            "llm-explore: LLM proposes K candidates per iteration; best wins."
+        ),
+    )
+    candidates_per_iteration = 3
+    if mode == "llm-explore":
+        candidates_per_iteration = st.slider(
+            "Candidates per iteration (K)",
+            min_value=2,
+            max_value=6,
+            value=3,
+            help="Each iteration runs K benchmarks. Higher = broader search, more GPU time.",
+        )
+
+    steps = st.slider(
+        "Steps per benchmark",
+        min_value=10,
+        max_value=100,
+        value=20,
+        step=5,
+        help="More steps = lower variance but longer benchmarks.",
+    )
+
+    max_iterations = st.slider(
+        "Max iterations",
+        min_value=1,
+        max_value=20,
+        value=10,
+        help="Cap on tuning iterations. Default 10 for llm modes, 5 for llm-explore.",
+    )
+
+    early_stop_after = st.slider(
+        "Early stop after N non-improvements",
+        min_value=1,
+        max_value=10,
+        value=3,
+    )
+
+    max_crashes = st.slider(
+        "Max total crashes",
+        min_value=1,
+        max_value=10,
+        value=4,
+    )
+
+    improvement_threshold = st.number_input(
+        "Improvement threshold (%)",
+        min_value=0.0,
+        max_value=10.0,
+        value=0.0,
+        step=0.1,
+        help="Min % gain to accept. 0.0 = any positive delta wins.",
+    )
+
+    st.divider()
+    run_pressed = st.button("🚀 Run auto-tune", type="primary", use_container_width=True)
+
+
+# ---------------------------------------------------------------------------
+# Render helpers
+# ---------------------------------------------------------------------------
+
+
+def _format_tps(v: float) -> str:
+    return f"{v:,.0f}" if v else "—"
+
+
+def _format_pct(v: float | None, suffix: str = "%") -> str:
+    if v is None:
+        return "—"
+    return f"{v:+.2f}{suffix}"
+
+
+def _waste_chart(baseline: dict, current: dict | None) -> alt.Chart:
+    """Two-row stacked bar: baseline vs current waste_budget breakdown."""
+    rows = []
+    sources = [("baseline", baseline)]
+    if current is not None and current is not baseline:
+        sources.append(("current best", current))
+    for label, m in sources:
+        wb = (m or {}).get("waste_budget") or {}
+        for bucket in WASTE_BUCKETS:
+            v = float(wb.get(bucket, 0.0))
+            if v <= 0:
+                continue
+            rows.append({"run": label, "bucket": bucket, "seconds": v})
+    if not rows:
+        return None
+    df = pd.DataFrame(rows)
+    return (
+        alt.Chart(df)
+        .mark_bar()
+        .encode(
+            x=alt.X("seconds:Q", title="seconds / step"),
+            y=alt.Y("run:N", title=None, sort=["baseline", "current best"]),
+            color=alt.Color(
+                "bucket:N",
+                scale=alt.Scale(scheme="tableau10"),
+                sort=WASTE_BUCKETS,
+            ),
+            order=alt.Order("bucket:N", sort="ascending"),
+            tooltip=["run", "bucket", alt.Tooltip("seconds:Q", format=".3f")],
+        )
+        .properties(height=120)
+    )
+
+
+def _render_metrics_row(baseline: dict | None, current: dict | None) -> None:
+    """Top-of-page metrics: tokens/sec, mfu_pct, hbm, with deltas vs baseline."""
+    cols = st.columns(4)
+    if baseline is None:
+        for c, label in zip(cols, ["tokens/sec", "mfu_pct", "hbm_peak (GB)", "iterations"]):
+            c.metric(label, "—")
+        return
+
+    cur = current if current is not None else baseline
+    base_tps = float(baseline.get("tokens_per_sec") or 0)
+    cur_tps = float(cur.get("tokens_per_sec") or 0)
+    base_mfu = float(baseline.get("mfu_pct") or 0)
+    cur_mfu = float(cur.get("mfu_pct") or 0)
+    base_hbm = float(baseline.get("hbm_peak_gb") or 0)
+    cur_hbm = float(cur.get("hbm_peak_gb") or 0)
+
+    cols[0].metric(
+        "tokens/sec",
+        _format_tps(cur_tps),
+        delta=f"{cur_tps - base_tps:+,.0f}" if base_tps else None,
+    )
+    cols[1].metric(
+        "MFU %",
+        f"{cur_mfu:.2f}",
+        delta=f"{cur_mfu - base_mfu:+.2f} pts" if base_mfu else None,
+    )
+    cols[2].metric(
+        "HBM peak (GB)",
+        f"{cur_hbm:.1f}",
+        delta=f"{cur_hbm - base_hbm:+.1f}" if base_hbm else None,
+    )
+
+
+def _render_candidate_card(event: dict) -> None:
+    """One candidate's outcome inside an iteration."""
+    name = event.get("name", "?")
+    rationale = event.get("rationale", "")
+    outcome = event.get("outcome", "?")
+    metrics = event.get("metrics") or {}
+    delta = event.get("delta_vs_best")
+    reason = event.get("reason", "")
+
+    icon = {
+        "evaluated": "📊",
+        "skipped": "⏭️",
+        "rejected": "❌",
+        "crashed": "💥",
+    }.get(outcome, "•")
+
+    title = f"{icon}  **{name}**"
+    if outcome == "evaluated" and delta is not None:
+        title += f"  —  Δ {delta:+.2f}%"
+    elif outcome != "evaluated":
+        title += f"  —  *{outcome}*"
+
+    with st.container(border=True):
+        st.markdown(title)
+        if rationale:
+            st.caption(rationale)
+        if metrics:
+            sub = st.columns(4)
+            sub[0].metric("tokens/sec", f"{metrics.get('tokens_per_sec', 0):,.0f}")
+            sub[1].metric("MFU %", f"{metrics.get('mfu_pct', 0):.2f}")
+            sub[2].metric("HBM peak GB", f"{metrics.get('hbm_peak_gb', 0):.1f}")
+            sub[3].metric("Δ vs best", f"{delta:+.2f}%" if delta is not None else "—")
+            wb = metrics.get("waste_budget") or {}
+            non_zero = {k: v for k, v in wb.items() if k != "useful_gpu" and v > 0}
+            if non_zero:
+                wb_text = ", ".join(
+                    f"`{k}={v:.3f}`"
+                    for k, v in sorted(non_zero.items(), key=lambda kv: kv[1], reverse=True)
+                )
+                st.caption(f"Waste: useful_gpu=`{wb.get('useful_gpu', 0):.3f}`, {wb_text}")
+        if reason and outcome != "evaluated":
+            st.caption(f"Reason: {reason}")
+
+
+# ---------------------------------------------------------------------------
+# Run
+# ---------------------------------------------------------------------------
+
+
+def _build_command(events_file: Path) -> list[str]:
+    cmd: list[str] = [sys.executable, "-u", str(AUTO_TUNE_SCRIPT)]
+    if workload_source == "Model id":
+        if not model_id.strip():
+            raise ValueError("Model id is required.")
+        cmd.extend(["--model", model_id.strip()])
+    else:
+        wp = workload_path.strip()
+        if not wp:
+            raise ValueError("Workload path is required.")
+        full = (REPO_ROOT / wp).resolve() if not Path(wp).is_absolute() else Path(wp)
+        if not full.exists():
+            raise ValueError(f"Workload not found: {full}")
+        cmd.append(str(full))
+    cmd.extend([
+        "--mode", mode,
+        "--steps", str(steps),
+        "--max-iterations", str(max_iterations),
+        "--early-stop-after", str(early_stop_after),
+        "--max-crashes", str(max_crashes),
+        "--improvement-threshold", str(improvement_threshold),
+        "--events", str(events_file),
+    ])
+    if mode == "llm-explore":
+        cmd.extend(["--candidates-per-iteration", str(candidates_per_iteration)])
+    return cmd
+
+
+def _build_request_body() -> dict:
+    """Same config as _build_command, but as a JSON body for POST /auto-tune."""
+    body: dict[str, Any] = {
+        "mode": mode,
+        "steps": steps,
+        "max_iterations": max_iterations,
+        "early_stop_after": early_stop_after,
+        "max_crashes": max_crashes,
+        "improvement_threshold": float(improvement_threshold),
+    }
+    if mode == "llm-explore":
+        body["candidates_per_iteration"] = candidates_per_iteration
+    if workload_source == "Model id":
+        if not model_id.strip():
+            raise ValueError("Model id is required.")
+        body["model"] = model_id.strip()
+    else:
+        if not workload_path.strip():
+            raise ValueError("Workload path is required.")
+        body["workload"] = workload_path.strip()
+    return body
+
+
+def _read_events(path: Path, seen: int) -> tuple[list[dict], int]:
+    """Read events from byte position `seen` onward; return (new_events, new_seen)."""
+    if not path.exists():
+        return [], seen
+    try:
+        with path.open("r") as f:
+            f.seek(seen)
+            chunk = f.read()
+            new_seen = f.tell()
+    except OSError:
+        return [], seen
+    if not chunk:
+        return [], new_seen
+    out: list[dict] = []
+    # The last line may be partial if the writer is mid-flush; drop it
+    # and back the pointer up so we re-read it next tick.
+    pieces = chunk.splitlines(keepends=True)
+    if pieces and not pieces[-1].endswith("\n"):
+        partial = pieces.pop()
+        new_seen -= len(partial.encode("utf-8"))
+    for line in pieces:
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            out.append(json.loads(line))
+        except json.JSONDecodeError:
+            continue
+    return out, new_seen
+
+
+# ---------------------------------------------------------------------------
+# Initial state — empty page
+# ---------------------------------------------------------------------------
+
+if not run_pressed:
+    st.info(
+        "👈 Configure inputs in the sidebar and press **Run auto-tune**. "
+        "The script will profile your workload, try MI300X-specific tuning "
+        "changes, and report what worked."
+    )
+    _render_metrics_row(None, None)
+    st.subheader("How it works")
+    st.markdown(
+        """
+        1. **Baseline benchmark** — runs your workload as-is (or as generated
+           from `--model`) on the GPU and measures tokens/sec, MFU, HBM peak,
+           and a per-bucket waste budget.
+        2. **Iterative tuning** — applies one MI300X-specific change at a
+           time (e.g. `bf16`, `batch_size=16`, `TORCH_BLAS_PREFER_HIPBLASLT=1`)
+           and re-benchmarks. Keeps changes that beat the current best.
+        3. **Live progress** — every accepted/rejected/crashed candidate
+           shows up here as it happens.
+        4. **Final report** — overall improvement, which experiments won,
+           how much wastage was recovered, and a diff against the baseline
+           workload.
+        """
+    )
+    st.stop()
+
+
+# ---------------------------------------------------------------------------
+# Run path: launch subprocess + tail events
+# ---------------------------------------------------------------------------
+
+st.subheader("Live run")
+header = st.empty()
+
+# Validate inputs early so the spinner doesn't show before a clear error
+try:
+    if backend_mode == "Local subprocess":
+        events_file = Path(tempfile.NamedTemporaryFile(
+            prefix="auto_tune_events_", suffix=".ndjson", delete=False
+        ).name)
+        cmd = _build_command(events_file)
+        with header.container():
+            st.code(" ".join(cmd), language="bash")
+            st.caption(f"Events stream: `{events_file}`")
+    else:
+        if not backend_url.strip():
+            raise ValueError("Backend URL is required for Remote GPU server mode.")
+        body = _build_request_body()
+        events_file = None
+        cmd = None
+        with header.container():
+            st.code(
+                f"POST {backend_url.rstrip('/')}/auto-tune\n"
+                + json.dumps(body, indent=2),
+                language="bash",
+            )
+            st.caption(
+                "Events stream: SSE from remote server. "
+                "All GPU work happens on that host."
+            )
+except ValueError as exc:
+    st.error(str(exc))
+    st.stop()
+
+baseline_metrics: dict | None = None
+best_metrics: dict | None = None  # most recent accepted iteration's metrics
+final_summary: dict | None = None
+iter_idx_to_container: dict[int, Any] = {}
+
+metrics_row = st.empty()
+with metrics_row.container():
+    _render_metrics_row(None, None)
+
+progress_bar = st.progress(0, text="Starting auto_tune…")
+
+iters_section = st.container()
+iters_section.subheader("Iterations")
+
+stdout_buffer: list[str] = []
+expected_iters = max(1, max_iterations)
+return_code: int | None = None
+
+
+def _handle_event(event: dict) -> None:
+    """Render one event into the live UI. Mutates module-level state for
+    baseline/best_metrics/final_summary so the summary block below has
+    the data it needs after the run."""
+    global baseline_metrics, best_metrics, final_summary  # noqa: PLW0603
+    etype = event.get("type")
+    if etype == "started":
+        st.session_state["started_event"] = event
+    elif etype == "baseline":
+        baseline_metrics = event["metrics"]
+        best_metrics = baseline_metrics
+        with metrics_row.container():
+            _render_metrics_row(baseline_metrics, best_metrics)
+        with iters_section:
+            with st.container(border=True):
+                st.markdown("**Baseline**")
+                sub = st.columns(4)
+                sub[0].metric("tokens/sec", f"{baseline_metrics.get('tokens_per_sec', 0):,.0f}")
+                sub[1].metric("MFU %", f"{baseline_metrics.get('mfu_pct', 0):.2f}")
+                sub[2].metric("HBM peak GB", f"{baseline_metrics.get('hbm_peak_gb', 0):.1f}")
+                sub[3].metric("GPU util %", f"{baseline_metrics.get('gpu_util_pct', 0):.1f}")
+                wb = baseline_metrics.get("waste_budget") or {}
+                non_zero = {k: v for k, v in wb.items() if k != "useful_gpu" and v > 0}
+                if non_zero:
+                    wb_str = ", ".join(
+                        f"`{k}={v:.3f}`"
+                        for k, v in sorted(non_zero.items(), key=lambda kv: kv[1], reverse=True)
+                    )
+                    st.caption(f"Recoverable waste: {wb_str}")
+    elif etype == "iter_start":
+        i = event["iteration"]
+        with iters_section:
+            container = st.container(border=True)
+            iter_idx_to_container[i] = container
+            n_cand = len(event.get("candidates") or [])
+            cand_summary = ", ".join(
+                c.get("name", "?") for c in event.get("candidates") or []
+            )
+            container.markdown(
+                f"**Iteration {i}**  ·  {n_cand} candidate{'s' if n_cand != 1 else ''}: {cand_summary}"
+            )
+        progress_bar.progress(
+            min(0.99, (i - 1) / expected_iters),
+            text=f"Iteration {i} of up to {expected_iters} — proposed: {cand_summary}",
+        )
+    elif etype == "candidate":
+        i = event["iteration"]
+        container = iter_idx_to_container.get(i)
+        if container is not None:
+            with container:
+                _render_candidate_card(event)
+        progress_bar.progress(
+            min(0.99, (i - 1) / expected_iters + 0.05),
+            text=f"Iter {i} · candidate {event.get('candidate_index')}/"
+                 f"{event.get('n_candidates')}: {event.get('name')} "
+                 f"({event.get('outcome')})",
+        )
+    elif etype == "merge_attempt":
+        i = event["iteration"]
+        container = iter_idx_to_container.get(i)
+        if container is not None:
+            with container:
+                outcome = event.get("outcome", "?")
+                names = ", ".join(event.get("candidate_names") or [])
+                if outcome == "wins":
+                    delta = event.get("delta_vs_best", 0)
+                    container.success(
+                        f"🔗 MERGE WINS — combined ({names}) hit Δ {delta:+.2f}% "
+                        f"(beats individual best `{event.get('individual_best_name')}`)"
+                    )
+                elif outcome == "lost":
+                    delta = event.get("delta_vs_best", 0)
+                    container.info(
+                        f"🔗 Merge tested ({names}) — Δ {delta:+.2f}%, didn't beat "
+                        f"individual `{event.get('individual_best_name')}`"
+                    )
+                elif outcome == "crashed":
+                    container.warning(f"💥 Merge crashed: {names}")
+                else:
+                    container.info(f"🔗 Merge skipped: {event.get('reason', '?')}")
+    elif etype == "iter_done":
+        i = event["iteration"]
+        container = iter_idx_to_container.get(i)
+        outcome = event.get("outcome")
+        if container is not None:
+            with container:
+                if outcome == "accepted":
+                    container.success(
+                        f"✅ ACCEPTED — `{event.get('winner_name')}` "
+                        f"(Δ {event.get('winner_delta', 0):+.2f}%)"
+                    )
+                else:
+                    container.warning(
+                        f"⏭️ ALL REJECTED — best was `{event.get('winner_name')}` "
+                        f"(Δ {event.get('winner_delta', 0):+.2f}%, below threshold)"
+                    )
+        if outcome == "accepted" and event.get("best_metrics"):
+            best_metrics = event["best_metrics"]
+            with metrics_row.container():
+                _render_metrics_row(baseline_metrics, best_metrics)
+        progress_bar.progress(
+            min(0.99, i / expected_iters),
+            text=f"Iter {i} done — best so far: {event.get('best_tps', 0):,.0f} tok/s",
+        )
+    elif etype == "summary":
+        final_summary = event
+        progress_bar.progress(1.0, text="Auto-tune complete")
+    elif etype == "error":
+        st.error(event.get("message", "unknown error"))
+    elif etype == "process_exit":
+        rc = event.get("returncode", "?")
+        st.error(
+            f"Backend subprocess exited (code {rc}): "
+            + event.get("message", "")
+        )
+
+
+# ---- Source the events from either local subprocess or remote SSE ----
+if backend_mode == "Local subprocess":
+    proc = subprocess.Popen(
+        cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        text=True,
+        bufsize=1,
+        cwd=str(REPO_ROOT),
+        env={**os.environ},
+    )
+
+    seen_bytes = 0
+    try:
+        while True:
+            # Pull stdout incrementally so the user sees the raw script log too
+            while True:
+                try:
+                    line = proc.stdout.readline() if proc.stdout else ""
+                except Exception:
+                    line = ""
+                if not line:
+                    break
+                stdout_buffer.append(line.rstrip())
+
+            new_events, seen_bytes = _read_events(events_file, seen_bytes)
+            for event in new_events:
+                _handle_event(event)
+
+            if proc.poll() is not None and not new_events:
+                new_events, seen_bytes = _read_events(events_file, seen_bytes)
+                if not new_events:
+                    break
+            time.sleep(0.4)
+    finally:
+        if proc.poll() is None:
+            proc.terminate()
+            try:
+                proc.wait(timeout=3)
+            except subprocess.TimeoutExpired:
+                proc.kill()
+    return_code = proc.returncode
+else:
+    # Remote SSE: stream from the FastAPI server
+    url = backend_url.rstrip("/") + "/auto-tune"
+    try:
+        response = requests.post(
+            url,
+            json=body,
+            stream=True,
+            timeout=(10, None),  # connect timeout 10s, read timeout indefinite
+            headers={"Accept": "text/event-stream"},
+        )
+        response.raise_for_status()
+    except requests.RequestException as exc:
+        st.error(f"Failed to reach backend at {url}: {exc}")
+        st.stop()
+
+    try:
+        for raw_line in response.iter_lines(decode_unicode=True):
+            if not raw_line:
+                continue
+            stdout_buffer.append(raw_line)
+            # SSE framing: lines starting with `data: <json>`
+            if raw_line.startswith("data:"):
+                payload = raw_line[len("data:"):].strip()
+                try:
+                    event = json.loads(payload)
+                except json.JSONDecodeError:
+                    continue
+                _handle_event(event)
+                if event.get("type") == "summary":
+                    # Final event; the server may still send a process_exit
+                    # but we can stop blocking the UI.
+                    pass
+            elif raw_line.startswith("event:") or raw_line.startswith(":"):
+                # SSE event-name line or comment — ignored
+                continue
+    except requests.RequestException as exc:
+        st.error(f"Stream interrupted: {exc}")
+    return_code = 0 if final_summary is not None else 1
+
+# ---------------------------------------------------------------------------
+# Final summary
+# ---------------------------------------------------------------------------
+
+st.divider()
+st.subheader("Summary")
+
+if final_summary is None:
+    st.error(
+        f"Auto-tune subprocess exited with code {return_code} but emitted "
+        "no summary event. Check the raw stdout below for details."
+    )
+else:
+    base_tps = float(final_summary.get("baseline_tps") or 0)
+    best_tps = float(final_summary.get("best_tps") or 0)
+    improvement_pct = float(final_summary.get("improvement_pct") or 0)
+    base = final_summary.get("baseline_metrics") or {}
+    best = final_summary.get("best_metrics") or {}
+
+    summary_cols = st.columns(4)
+    summary_cols[0].metric(
+        "Baseline tokens/sec",
+        f"{base_tps:,.0f}",
+    )
+    summary_cols[1].metric(
+        "Best tokens/sec",
+        f"{best_tps:,.0f}",
+        delta=f"{best_tps - base_tps:+,.0f}",
+    )
+    summary_cols[2].metric(
+        "Improvement",
+        f"{improvement_pct:+.2f}%",
+    )
+    summary_cols[3].metric(
+        "MFU baseline → best",
+        f"{base.get('mfu_pct', 0):.1f} → {best.get('mfu_pct', 0):.1f} %",
+    )
+
+    chart = _waste_chart(base, best)
+    if chart is not None:
+        st.markdown("**Waste reduction (seconds/step, by bucket)**")
+        st.altair_chart(chart, use_container_width=True)
+        # Per-bucket reduction table
+        diff_rows = []
+        bwb = base.get("waste_budget") or {}
+        cwb = best.get("waste_budget") or {}
+        for bucket in WASTE_BUCKETS:
+            bv = float(bwb.get(bucket, 0.0))
+            cv = float(cwb.get(bucket, 0.0))
+            if bv == 0 and cv == 0:
+                continue
+            diff_rows.append({
+                "bucket": bucket,
+                "baseline (s)": round(bv, 4),
+                "best (s)": round(cv, 4),
+                "Δ (s)": round(cv - bv, 4),
+            })
+        if diff_rows:
+            st.dataframe(pd.DataFrame(diff_rows), use_container_width=True)
+
+    accepted = final_summary.get("accepted") or []
+    rejected = final_summary.get("rejected") or []
+
+    col_a, col_r = st.columns(2)
+    with col_a:
+        st.markdown(f"**✅ Accepted ({len(accepted)})**")
+        if accepted:
+            st.dataframe(
+                pd.DataFrame(accepted)[["name", "tps", "delta_pct"]].rename(
+                    columns={"tps": "tokens/sec", "delta_pct": "Δ %"}
+                ),
+                use_container_width=True,
+                hide_index=True,
+            )
+        else:
+            st.caption("(no experiments accepted)")
+    with col_r:
+        st.markdown(f"**❌ Rejected ({len(rejected)})**")
+        if rejected:
+            st.dataframe(
+                pd.DataFrame(rejected),
+                use_container_width=True,
+                hide_index=True,
+            )
+        else:
+            st.caption("(none)")
+
+    env_vars = final_summary.get("best_env_vars") or {}
+    if env_vars:
+        st.markdown("**Required env vars for best config**")
+        st.code("\n".join(f"export {k}={v}" for k, v in env_vars.items()), language="bash")
+
+    best_path = final_summary.get("best_workload_path")
+    base_path = final_summary.get("baseline_workload_path")
+    if best_path and base_path:
+        st.markdown("**Best workload script**")
+        st.code(f"diff {base_path} {best_path}", language="bash")
+        try:
+            best_text = Path(best_path).read_text()
+            st.download_button(
+                "⬇️ Download best.py",
+                data=best_text,
+                file_name="best.py",
+                mime="text/x-python",
+            )
+        except OSError:
+            pass
+
+# Raw stdout always at the bottom for debugging
+with st.expander("Raw subprocess output"):
+    st.code("\n".join(stdout_buffer), language="text")

workloads/_runtime.py

@@ -86,6 +86,22 @@ def parse_runtime_args() -> RuntimeArgs:
     )
 
 
+# MI300X (CDNA3) peak throughput, dense, bf16/fp16 — both arrive at the
+# same number on this arch since the matrix engine is the same. Source:
+# AMD Instinct MI300X datasheet. With sparsity it's ~2.6 PFLOPS, but
+# transformers training rarely hits the sparse path so we use dense as
+# the realistic peak.
+_MI300X_PEAK_FLOPS_DENSE_BF16 = 1.307e15
+
+# FLOPs per token for forward + backward. The standard 6N approximation
+# (forward 2N + backward 4N for full fine-tuning) slightly overestimates
+# LoRA — pure LoRA backward only computes weight gradients for the small
+# adapter matrices, not the frozen base — so true LoRA flops/token is
+# closer to 4N. We use 6N as the conventional choice and accept a ~30%
+# pessimistic MFU for LoRA. Still useful as a relative metric run-to-run.
+_FLOPS_PER_TOKEN_FACTOR = 6
+
+
 def emit_torch_profile(
     path: str,
     *,
@@ -94,13 +110,16 @@ def emit_torch_profile(
     per_device_batch: int,
     grad_accum: int = 1,
     seq_len_cap: int = 512,
+    model_params: int = 0,
 ) -> None:
     """Write the smallest torch_profile-shape JSON profile_parser will read.
 
     profile_parser._read_torch_profile looks for these top-level fields under
     ``metadata``: tokens_per_sec, mfu_pct, step_time_seconds, pytorch_version.
-    We supply the first three and pytorch_version (mfu_pct is optional and
-    estimated downstream).
+
+    `model_params` is optional — pass `sum(p.numel() for p in
+    model.parameters())` from the workload to get a populated `mfu_pct`.
+    Without it, mfu_pct stays unset (profile_parser will default to 0).
 
     No-ops when ``path`` is empty (script run outside goblin_runner.sh) or
     when ``n_steps`` is 0 (training crashed before finishing a step).
@@ -113,43 +132,21 @@ def emit_torch_profile(
         global_batch = max(1, per_device_batch) * max(1, grad_accum)
         approx_tokens = n_steps * global_batch * seq_len_cap
         tokens_per_sec = approx_tokens / elapsed if elapsed > 0 else 0.0
-        payload = {
-            "metadata": {
-                "tokens_per_sec": round(tokens_per_sec, 2),
-                "step_time_seconds": round(elapsed / n_steps, 4),
-                "pytorch_version": torch.__version__,
-                "n_steps": n_steps,
-            }
+        metadata = {
+            "tokens_per_sec": round(tokens_per_sec, 2),
+            "step_time_seconds": round(elapsed / n_steps, 4),
+            "pytorch_version": torch.__version__,
+            "n_steps": n_steps,
         }
+        if model_params > 0 and tokens_per_sec > 0:
+            flops_per_token = _FLOPS_PER_TOKEN_FACTOR * model_params
+            mfu_pct = (flops_per_token * tokens_per_sec) / _MI300X_PEAK_FLOPS_DENSE_BF16 * 100
+            metadata["mfu_pct"] = round(mfu_pct, 2)
+            metadata["model_params"] = model_params
+        payload = {"metadata": metadata}
         with open(path, "w") as f:
             json.dump(payload, f)
     except Exception as exc:  # pragma: no cover — diagnostic only
         # Don't tank the run on a profile-emit failure; the agent will
         # just see "fake" metrics for this step instead of "live".
         print(f"[workloads._runtime] failed to write {path}: {exc}")
-
-
-def trainer_tokenizer_kwargs(trainer_cls, tokenizer) -> dict:
-    """Return the right kwarg for handing a tokenizer to ``Trainer``.
-
-    transformers ≥ 4.46 renamed ``tokenizer=`` to ``processing_class=`` (the
-    old name is still accepted with a DeprecationWarning, but a future
-    release drops it). Older versions only know ``tokenizer=``. We
-    introspect ``Trainer.__init__`` so the workloads run on either
-    generation without pinning a transformers version.
-
-    Use site:
-
-        trainer = Trainer(
-            model=model,
-            args=training_args,
-            train_dataset=dataset,
-            **trainer_tokenizer_kwargs(Trainer, tokenizer),
-            data_collator=_toy_collate,
-        )
-    """
-    import inspect
-
-    if "processing_class" in inspect.signature(trainer_cls.__init__).parameters:
-        return {"processing_class": tokenizer}
-    return {"tokenizer": tokenizer}

workloads/scenarios/README.md

@@ -0,0 +1,30 @@
+# Test workloads
+
+Hand-built training scripts that demo different misconfiguration patterns.
+The agent should produce **different audit reports** for each one — useful
+for stress-testing the KB, for live demos with variety, and for sanity-
+checking that recommendations track the actual configuration.
+
+| File | Headline misconfiguration | KB rules expected to fire |
+|---|---|---|
+| `train_qwen_full_ft.py` | Full fine-tune (no LoRA), fp32, no gradient checkpointing | `precision.fp32_default_wastes_matrix_cores`, `memory.gradient_checkpointing_for_long_seq`, `attention.sdpa_over_eager` |
+| `train_qwen_long_context.py` | seq_len=8192, no gradient checkpointing, fp16, eager attention | `memory.gradient_checkpointing_for_long_seq`, `attention.flash_rocm_over_eager`, `precision.bf16_over_fp16_on_mi300x` |
+| `train_qwen_distributed_bad.py` | One process driving 8 GPUs (ROCm launch serialization antipattern) | `collectives.one_process_per_gpu`, `env.nccl_min_nchannels`, `env.numa_auto_balancing_disable` |
+| `train_qwen_bnb.py` | Uses bitsandbytes 8-bit Adam (not officially ROCm-supported) | `optimizer.bitsandbytes_not_supported_warning`, `precision.bf16_over_fp16_on_mi300x` |
+| `train_qwen_well_tuned.py` | Already optimized (bf16, flash_rocm, prefetch=4, persistent_workers) | None or only minor suggestions — sanity check that the agent says "nothing to do" instead of inventing problems |
+
+Run any of them through the agent:
+
+```bash
+export GOBLIN_AGENT_BACKEND=qwen-vllm
+export GOBLIN_QWEN_VLLM_URL=http://localhost:8001/v1
+export GOBLIN_QWEN_VLLM_MODEL=Qwen/Qwen3-32B
+python -m agent workloads/scenarios/train_qwen_long_context.py
+```
+
+These are **AST-parseable, optionally executable**. Each one redirects on
+the same fix — `parse_config` only walks the AST, so even if the script
+doesn't actually train cleanly, the agent's audit still works. If you
+want rocprofv3 to capture real numbers (vs FakeRunner fallback), the
+scripts marked "executable: yes" below are runnable; the others rely on
+FakeRunner replay.

workloads/scenarios/train_qwen_bnb.py

@@ -0,0 +1,110 @@
+# Test workload: bitsandbytes 8-bit Adam — the classic "ported from NVIDIA"
+# script that doesn't realize bitsandbytes isn't officially supported on ROCm.
+#
+# Headline misconfigurations the agent should catch:
+#   - optimizer.bitsandbytes_not_supported_warning  (optim="paged_adamw_8bit")
+#   - precision.bf16_over_fp16_on_mi300x            (fp16=True)
+#   - attention.flash_rocm_over_eager               (attn_implementation="eager")
+#   - data.dataloader_workers_zero                   (num_workers=0)
+#
+# Executable: not on ROCm — bitsandbytes will fail at import or on the
+# first 8-bit GEMM. The whole point of this fixture is that the agent
+# warns the user BEFORE they hit that wall. AST parse always works.
+
+import os
+import sys
+import time
+
+sys.path.insert(
+    0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+)
+
+import torch
+from datasets import load_dataset
+from peft import LoraConfig, get_peft_model
+from torch.utils.data import DataLoader
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    Trainer,
+    TrainingArguments,
+)
+
+from workloads._runtime import emit_torch_profile, parse_runtime_args
+
+_runtime = parse_runtime_args()
+
+os.environ["HF_TOKEN"] = "hf_aaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+HF_TOKEN = os.environ["HF_TOKEN"]
+
+MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.float16,
+    attn_implementation="eager",
+    token=HF_TOKEN,
+    load_in_8bit=True,                    # bitsandbytes load — ROCm support unofficial
+)
+
+lora_config = LoraConfig(
+    r=16,
+    lora_alpha=32,
+    target_modules=["q_proj", "v_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, lora_config)
+
+dataset = load_dataset("yahma/alpaca-cleaned", split="train")
+
+train_loader = DataLoader(
+    dataset,
+    batch_size=4,
+    num_workers=0,
+    pin_memory=False,
+    persistent_workers=False,
+)
+
+_RUNTIME_MAX_STEPS = _runtime.max_steps if _runtime.max_steps > 0 else -1
+
+training_args = TrainingArguments(
+    output_dir="./out",
+    per_device_train_batch_size=4,
+    gradient_accumulation_steps=8,
+    num_train_epochs=3,
+    learning_rate=2e-4,
+    warmup_steps=100,
+    fp16=True,
+    optim="paged_adamw_8bit",              # bitsandbytes optimizer — ROCm unofficial
+    logging_steps=10,
+    save_steps=500,
+    dataloader_num_workers=0,
+    dataloader_pin_memory=False,
+    gradient_checkpointing=False,
+    torch_compile=False,
+    report_to="none",
+    push_to_hub=False,
+    max_steps=_RUNTIME_MAX_STEPS,
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+)
+
+if __name__ == "__main__":
+    _t0 = time.time()
+    trainer.train()
+    emit_torch_profile(
+        _runtime.torch_profile_out,
+        elapsed=time.time() - _t0,
+        n_steps=int(getattr(trainer.state, "global_step", 0) or _runtime.max_steps),
+        per_device_batch=training_args.per_device_train_batch_size,
+        grad_accum=training_args.gradient_accumulation_steps,
+        seq_len_cap=512,
+    )

workloads/scenarios/train_qwen_distributed_bad.py

@@ -0,0 +1,122 @@
+# Test workload: multi-GPU LoRA with the "one-process-many-GPUs" antipattern.
+#
+# A single Python process (no torchrun, no accelerate launch) tries to drive
+# multiple MI300X devices via DataParallel. ROCm serializes kernel launches
+# across devices when one process owns multiple HIP streams, so every
+# collective becomes a launch-queue bottleneck.
+#
+# Headline misconfigurations the agent should catch:
+#   - collectives.one_process_per_gpu     (single launcher, multiple devices)
+#   - env.nccl_min_nchannels              (NCCL_MIN_NCHANNELS not set)
+#   - env.numa_auto_balancing_disable     (NUMA balancing left on by default)
+#   - precision.bf16_over_fp16_on_mi300x  (fp16=True)
+#   - attention.flash_rocm_over_eager
+#
+# Executable: requires multiple MI300X visible to one process. Even when
+# runnable, ROCm warns on the launch serialization. AST parse is fine.
+
+import os
+import sys
+import time
+
+sys.path.insert(
+    0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+)
+
+import torch
+import torch.nn as nn
+from datasets import load_dataset
+from peft import LoraConfig, get_peft_model
+from torch.utils.data import DataLoader
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    Trainer,
+    TrainingArguments,
+)
+
+from workloads._runtime import emit_torch_profile, parse_runtime_args
+
+_runtime = parse_runtime_args()
+
+os.environ["HF_TOKEN"] = "hf_aaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+HF_TOKEN = os.environ["HF_TOKEN"]
+os.environ["HSA_FORCE_FINE_GRAIN_PCIE"] = "1"
+# Notably absent: NCCL_MIN_NCHANNELS, GOBLIN_HINT_NUMA_AUTO_BALANCING.
+
+MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.float16,
+    attn_implementation="eager",
+    token=HF_TOKEN,
+)
+
+lora_config = LoraConfig(
+    r=16,
+    lora_alpha=32,
+    target_modules=["q_proj", "v_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, lora_config)
+
+# THE ANTIPATTERN: DataParallel from a single process across all visible GPUs.
+# Production code should use torchrun --nproc_per_node=N or accelerate launch.
+if torch.cuda.device_count() > 1:
+    model = nn.DataParallel(model)
+
+dataset = load_dataset("yahma/alpaca-cleaned", split="train")
+
+train_loader = DataLoader(
+    dataset,
+    batch_size=8,
+    num_workers=4,
+    pin_memory=True,
+    persistent_workers=True,
+)
+
+_RUNTIME_MAX_STEPS = _runtime.max_steps if _runtime.max_steps > 0 else -1
+
+training_args = TrainingArguments(
+    output_dir="./out",
+    per_device_train_batch_size=8,
+    gradient_accumulation_steps=4,
+    num_train_epochs=1,
+    learning_rate=2e-4,
+    warmup_steps=100,
+    fp16=True,
+    optim="adamw_torch",
+    logging_steps=10,
+    save_steps=500,
+    dataloader_num_workers=4,
+    dataloader_pin_memory=True,
+    dataloader_persistent_workers=True,
+    gradient_checkpointing=False,
+    torch_compile=False,
+    report_to="none",
+    push_to_hub=False,
+    max_steps=_RUNTIME_MAX_STEPS,
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+)
+
+if __name__ == "__main__":
+    _t0 = time.time()
+    trainer.train()
+    emit_torch_profile(
+        _runtime.torch_profile_out,
+        elapsed=time.time() - _t0,
+        n_steps=int(getattr(trainer.state, "global_step", 0) or _runtime.max_steps),
+        per_device_batch=training_args.per_device_train_batch_size,
+        grad_accum=training_args.gradient_accumulation_steps,
+        seq_len_cap=512,
+    )

workloads/scenarios/train_qwen_full_ft.py

@@ -0,0 +1,103 @@
+# Test workload: full fine-tune (no LoRA), fp32, no gradient checkpointing.
+#
+# Headline misconfigurations the agent should catch:
+#   - precision.fp32_default_wastes_matrix_cores  (fp16=False, bf16=False)
+#   - memory.gradient_checkpointing_for_long_seq  (gradient_checkpointing=False, seq_len=2048)
+#   - attention.sdpa_over_eager OR flash_rocm     (attn_implementation="eager")
+#   - data.dataloader_workers_zero                (num_workers=0)
+#   - data.pin_memory_false                        (pin_memory=False)
+#
+# Executable: not really — full FT of a 7B model OOMs on a single MI300X
+# without LoRA. AST parse is fine; rocprofv3 will fail and FakeRunner kicks in.
+
+import os
+import sys
+import time
+
+# Bootstrap repo root so `from workloads._runtime import ...` resolves
+# regardless of the cwd goblin_runner.sh launches us from.
+sys.path.insert(
+    0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+)
+
+import torch
+from datasets import load_dataset
+from torch.utils.data import DataLoader
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    Trainer,
+    TrainingArguments,
+)
+
+from workloads._runtime import emit_torch_profile, parse_runtime_args
+
+_runtime = parse_runtime_args()
+
+os.environ["HF_TOKEN"] = "hf_aaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+HF_TOKEN = os.environ["HF_TOKEN"]
+os.environ["HSA_FORCE_FINE_GRAIN_PCIE"] = "1"
+
+MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.float32,            # fp32 — wastes CDNA3 matrix cores
+    attn_implementation="eager",
+    token=HF_TOKEN,
+)
+
+dataset = load_dataset("yahma/alpaca-cleaned", split="train")
+
+train_loader = DataLoader(
+    dataset,
+    batch_size=2,                          # tiny because fp32 + full FT eats HBM
+    num_workers=0,
+    pin_memory=False,
+    persistent_workers=False,
+)
+
+# Literal kwargs so parse_config picks them all up (see canonical workload).
+_RUNTIME_MAX_STEPS = _runtime.max_steps if _runtime.max_steps > 0 else -1
+
+training_args = TrainingArguments(
+    output_dir="./out",
+    per_device_train_batch_size=2,
+    gradient_accumulation_steps=16,
+    num_train_epochs=1,
+    learning_rate=1e-5,
+    warmup_steps=200,
+    fp16=False,                            # no precision flag → fp32 path
+    bf16=False,
+    optim="adamw_torch",
+    max_seq_length=2048,
+    logging_steps=10,
+    save_steps=500,
+    dataloader_num_workers=0,
+    dataloader_pin_memory=False,
+    gradient_checkpointing=False,          # at seq=2048 this leaves a lot of HBM tied up
+    torch_compile=False,
+    report_to="none",
+    push_to_hub=False,
+    max_steps=_RUNTIME_MAX_STEPS,
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+)
+
+if __name__ == "__main__":
+    _t0 = time.time()
+    trainer.train()
+    emit_torch_profile(
+        _runtime.torch_profile_out,
+        elapsed=time.time() - _t0,
+        n_steps=int(getattr(trainer.state, "global_step", 0) or _runtime.max_steps),
+        per_device_batch=training_args.per_device_train_batch_size,
+        grad_accum=training_args.gradient_accumulation_steps,
+        seq_len_cap=2048,
+    )

workloads/scenarios/train_qwen_long_context.py

@@ -0,0 +1,111 @@
+# Test workload: long-context LoRA fine-tune.
+#
+# Headline misconfigurations the agent should catch:
+#   - memory.gradient_checkpointing_for_long_seq  (seq_len=8192, gradient_checkpointing=False)
+#   - attention.flash_rocm_over_eager             (attn_implementation="eager" — KILLS at seq 8K)
+#   - precision.bf16_over_fp16_on_mi300x          (fp16=True)
+#   - data.persistent_workers_false               (persistent_workers=False)
+#   - data.pin_memory_false                        (pin_memory=False)
+#
+# Executable: only with flash attention installed. Without it, eager attention
+# at seq=8192 blows up HBM. AST parse always works.
+
+import os
+import sys
+import time
+
+sys.path.insert(
+    0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+)
+
+import torch
+from datasets import load_dataset
+from peft import LoraConfig, get_peft_model
+from torch.utils.data import DataLoader
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    Trainer,
+    TrainingArguments,
+)
+
+from workloads._runtime import emit_torch_profile, parse_runtime_args
+
+_runtime = parse_runtime_args()
+
+os.environ["HF_TOKEN"] = "hf_aaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+HF_TOKEN = os.environ["HF_TOKEN"]
+os.environ["MIOPEN_FIND_MODE"] = "3"
+
+MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.float16,
+    attn_implementation="eager",          # at seq=8192 this is catastrophic
+    token=HF_TOKEN,
+)
+
+lora_config = LoraConfig(
+    r=32,
+    lora_alpha=64,
+    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, lora_config)
+
+dataset = load_dataset("yahma/alpaca-cleaned", split="train")
+
+train_loader = DataLoader(
+    dataset,
+    batch_size=1,                          # forced down by long context
+    num_workers=4,                         # workers OK; problem is elsewhere
+    pin_memory=False,
+    persistent_workers=False,
+)
+
+_RUNTIME_MAX_STEPS = _runtime.max_steps if _runtime.max_steps > 0 else -1
+
+training_args = TrainingArguments(
+    output_dir="./out",
+    per_device_train_batch_size=1,
+    gradient_accumulation_steps=32,
+    num_train_epochs=2,
+    learning_rate=2e-4,
+    warmup_steps=50,
+    fp16=True,                             # bf16 is the right call on CDNA3
+    optim="adamw_torch",
+    max_seq_length=8192,                   # the long-context part
+    logging_steps=10,
+    save_steps=500,
+    dataloader_num_workers=4,
+    dataloader_pin_memory=False,
+    dataloader_persistent_workers=False,
+    gradient_checkpointing=False,          # missing — at seq 8K activations dominate HBM
+    torch_compile=False,
+    report_to="none",
+    push_to_hub=False,
+    max_steps=_RUNTIME_MAX_STEPS,
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+)
+
+if __name__ == "__main__":
+    _t0 = time.time()
+    trainer.train()
+    emit_torch_profile(
+        _runtime.torch_profile_out,
+        elapsed=time.time() - _t0,
+        n_steps=int(getattr(trainer.state, "global_step", 0) or _runtime.max_steps),
+        per_device_batch=training_args.per_device_train_batch_size,
+        grad_accum=training_args.gradient_accumulation_steps,
+        seq_len_cap=8192,
+    )

workloads/scenarios/train_qwen_well_tuned.py

@@ -0,0 +1,143 @@
+# Test workload: already well-tuned LoRA fine-tune.
+#
+# This is the negative-control fixture. The agent should produce a SHORT
+# audit with few or no recommendations — the goal is to catch hallucinated
+# rule applications. If GPU Goblin invents problems on a clean config, the
+# audit isn't trustworthy.
+#
+# What's already correct here:
+#   - precision = bf16
+#   - attention_impl = flash_rocm (Optimum-AMD path)
+#   - dataloader_num_workers = 8, pin_memory=True, prefetch_factor=4,
+#     persistent_workers=True
+#   - gradient_checkpointing = True (long context)
+#   - torch_compile = True
+#   - NCCL_MIN_NCHANNELS, HSA_FORCE_FINE_GRAIN_PCIE both set
+#
+# Executable: yes (fastest path; meant to actually run cleanly).
+
+import os
+import sys
+import time
+
+sys.path.insert(
+    0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+)
+
+import torch
+from datasets import load_dataset
+from peft import LoraConfig, get_peft_model
+from torch.utils.data import DataLoader
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    Trainer,
+    TrainingArguments,
+)
+
+from workloads._runtime import emit_torch_profile, parse_runtime_args
+
+_runtime = parse_runtime_args()
+
+os.environ["HF_TOKEN"] = "hf_aaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+HF_TOKEN = os.environ["HF_TOKEN"]
+
+# All the env knobs from the KB, set correctly.
+os.environ["HSA_FORCE_FINE_GRAIN_PCIE"] = "1"
+os.environ["MIOPEN_FIND_MODE"] = "3"
+os.environ["NCCL_MIN_NCHANNELS"] = "112"
+
+MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.bfloat16,            # bf16 ✓
+    attn_implementation="flash_attention_2",  # parser maps to flash / flash_rocm
+    token=HF_TOKEN,
+)
+
+# Compile path on — Qwen is on the eager-friendly list.
+model = torch.compile(model, mode="reduce-overhead")
+
+lora_config = LoraConfig(
+    r=16,
+    lora_alpha=32,
+    target_modules=["q_proj", "v_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, lora_config)
+model.gradient_checkpointing_enable()
+
+dataset = load_dataset("yahma/alpaca-cleaned", split="train")
+
+train_loader = DataLoader(
+    dataset,
+    batch_size=12,                          # comfortable on 192 GB HBM
+    num_workers=8,
+    pin_memory=True,
+    prefetch_factor=4,
+    persistent_workers=True,
+)
+
+_RUNTIME_MAX_STEPS = _runtime.max_steps if _runtime.max_steps > 0 else -1
+
+training_args = TrainingArguments(
+    output_dir="./out",
+    per_device_train_batch_size=12,
+    gradient_accumulation_steps=2,
+    num_train_epochs=1,
+    learning_rate=2e-4,
+    warmup_steps=100,
+    bf16=True,                              # bf16 ✓
+    optim="adamw_torch",
+    max_seq_length=4096,
+    logging_steps=10,
+    save_steps=500,
+    dataloader_num_workers=8,
+    dataloader_pin_memory=True,
+    dataloader_prefetch_factor=4,
+    dataloader_persistent_workers=True,
+    gradient_checkpointing=True,
+    torch_compile=True,
+    report_to="none",
+    push_to_hub=False,
+    remove_unused_columns=False,
+    max_steps=_RUNTIME_MAX_STEPS,
+)
+
+
+def _toy_collate(rows):
+    texts = [
+        (r.get("instruction") or "")
+        + ("\n" + r["input"] if r.get("input") else "")
+        + "\n"
+        + (r.get("output") or "")
+        for r in rows
+    ]
+    enc = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
+    enc["labels"] = enc["input_ids"].clone()
+    return enc
+
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+    data_collator=_toy_collate,
+)
+
+if __name__ == "__main__":
+    _t0 = time.time()
+    trainer.train()
+    emit_torch_profile(
+        _runtime.torch_profile_out,
+        elapsed=time.time() - _t0,
+        n_steps=int(getattr(trainer.state, "global_step", 0) or _runtime.max_steps),
+        per_device_batch=training_args.per_device_train_batch_size,
+        grad_accum=training_args.gradient_accumulation_steps,
+        seq_len_cap=4096,
+    )

workloads/train_qwen_lora.py

@@ -30,11 +30,7 @@ from transformers import (
     TrainingArguments,
 )
 
-from workloads._runtime import (
-    emit_torch_profile,
-    parse_runtime_args,
-    trainer_tokenizer_kwargs,
-)
+from workloads._runtime import emit_torch_profile, parse_runtime_args
 
 # Parse the goblin_runner.sh injected flags (--max_steps, --torch_profile_out).
 _runtime = parse_runtime_args()
@@ -150,12 +146,17 @@ trainer = Trainer(
     model=model,
     args=training_args,
     train_dataset=dataset,
-    **trainer_tokenizer_kwargs(Trainer, tokenizer),
+    tokenizer=tokenizer,
     data_collator=_toy_collate,
 )
 
 
 if __name__ == "__main__":
+    # Total parameter count drives MFU. For a peft-wrapped model this
+    # includes both the frozen base (which still does forward) and the
+    # tiny LoRA adapters. The forward+backward FLOPs estimate uses the
+    # full count (the standard 6N rule lives inside emit_torch_profile).
+    _model_params = sum(p.numel() for p in model.parameters())
     _t0 = time.time()
     trainer.train()
     _elapsed = time.time() - _t0
@@ -170,4 +171,5 @@ if __name__ == "__main__":
         per_device_batch=training_args.per_device_train_batch_size,
         grad_accum=training_args.gradient_accumulation_steps,
         seq_len_cap=512,
+        model_params=_model_params,
     )