brainstorming/idea.md

GPU Goblin — Idea

An AI agent that hunts wasted compute on AMD MI300X. Upload a fine-tuning config; the agent profiles a real run, diagnoses inefficiency, recommends ROCm-specific fixes, then re-runs and proves the speedup with hard numbers.

---

The Problem

Most teams fine-tuning LLMs on AMD hardware leave 2–3× of throughput on the floor without realizing it. The waste hides in plain sight:

• HBM under-fed — batch sizes copied from NVIDIA tutorials don't exploit MI300X's 192 GB.
• Wrong precision — fp16 is the reflexive default; bf16 matches throughput on CDNA3 with better numerics.
• Naive attention — teams ship without flash-attn ROCm fork or PyTorch SDPA enabled.
• Generic kernels — hipBLASLt tuning, MIOpen autotune, RCCL collective tweaks all left at defaults.
• Wrong libraries — bitsandbytes is the reflexive choice for 8-bit Adam, but it's not officially supported on ROCm. Optimum-AMD validates Flash Attention 2 + GPTQ/AWQ paths instead.
• Distributed footguns — one-process-driving-many-GPUs serializes kernel launches on ROCm; NUMA auto-balancing causes variance; NCCL_MIN_NCHANNELS left at default.
• Knobs nobody knows about — HSA_FORCE_FINE_GRAIN_PCIE, MIOPEN_FIND_MODE, hipBLASLt cache paths.

Engineers don't fix this not because they're lazy — it's because the knowledge is scattered across ROCm docs, AMD blog posts, GitHub issues, and Discord threads. Nobody has time to read it all before launching a run.

The Solution — GPU Goblin

A tool-using agent that does what an experienced AMD performance engineer would do, in one minute:

1. Read the user's fine-tuning script or HF TrainingArguments.
2. Profile a 10-step warm run on MI300X (torch.profiler + rocprofv3 + amd-smi).
3. Diagnose against a curated knowledge base of ROCm-specific optimizations.
4. Patch the config — concrete diff, not vague advice.
5. Benchmark the patched config on the same MI300X.
6. Report before/after side-by-side with real numbers and citations.

The user keeps interacting in chat: "why bf16?", "what if I can't change the optimizer?", "how much $ does this save per epoch?"

Why This Wins

Most teams build user-facing AI. We build AI for AI builders. That alone is judge-bait.

But the deeper hook is that GPU Goblin is provably useful in a way most hackathon projects aren't. We don't have to convince judges the recommendations are good — we show them, on the same MI300X, with the same model, in the same demo. Tokens/sec: 142 → 318. End of debate.

Track Fit (Track 1: AI Agents & Agentic Workflows)

• Primary track: AI Agents & Agentic Workflows. Real tool-using loop. The agent observes (profile), hypothesizes (KB query), tests (benchmark), refines (patch). Every step visible in chat. Not a one-shot LLM call dressed up as an agent.
• Fine-tuning is what we audit, not the track we enter. Canonical workload is Qwen2.5-7B-Instruct + LoRA on MI300X. Recommendations are fine-tuning specific — batch sizing for LoRA, gradient checkpointing thresholds, optimizer choice (8-bit Adam on ROCm), seq-length packing.

Qwen Technology Partner Challenge

GPU Goblin satisfies the Qwen partner challenge two ways:

1. Qwen as the agent brain. The audit loop runs on Qwen/Qwen2.5-7B-Instruct served via Hugging Face Inference Providers. Every tool call, every recommendation, every chat answer is generated by Qwen.
2. Qwen as the audit target. The canonical demo workload is a Qwen2.5-7B-Instruct LoRA fine-tune. The agent audits the same model family it runs on.

Result: end-to-end Qwen on AMD silicon. Directly answers the judging criterion "How effectively the chosen model(s) are integrated into the solution."

Hugging Face Integration

Hugging Face is the named Technology Partner — both model hub and deployment layer:

• Model hub: Qwen models pulled from HF Hub at audit time + at agent-spin-up time.
• Optimum-AMD: several KB rules cite Optimum-AMD validated paths (Flash Attention 2 on MI300, GPTQ/AWQ ROCm path).
• Inference Providers: the agent's Qwen calls go through HF's Inference Providers router (provider="auto" selects Together / Fireworks-AI / Nebius based on availability).
• Deployment as HF Space: the Streamlit UI ships as a Space within the AMD Developer Hackathon HF Organization, satisfying the "Demo Application Platform" + "Application URL" submission fields. The Space runs in offline-replay mode by default so it works without any live backend.

AMD Differentiation

Every single recommendation cites a ROCm-specific rule, not generic PyTorch advice. The knowledge base is the moat:

• ROCm env-var tuning (HSA_*, MIOPEN_*, NCCL_MIN_NCHANNELS=112)
• hipBLASLt hint logging + offline tuning files; MIOpen MIOPEN_FIND_* autotune
• Flash-attn ROCm fork (validated on MI300 via Optimum-AMD)
• RCCL topology + tensor-parallel placement within a single XGMI island
• bitsandbytes-on-ROCm gotchas (not officially supported — recommend alternatives)
• bf16 vs fp16 on CDNA3 matrix cores
• One-process-per-GPU vs one-process-many-GPUs (ROCm serializes launches in the latter)
• NUMA auto-balancing disable for stable benchmarks
• MI300X-specific: 304 CUs, 192 GB HBM3, ~5.3 TB/s peak bandwidth, native FP8

This is the kind of insight you only get from someone who has actually shipped on MI300X. We bottle it.

Demo Narrative (3 minutes)

Setup (15s): "Most teams waste 50%+ of their MI300X. Watch GPU Goblin find that waste live."

Live demo (2 min):

1. Drop in train_qwen_lora.py — batch=4, fp16, naive attention. (Looks normal.)
2. Agent: "Parsing config…" → extracts hyperparams.
3. Agent: "Running 10-step profile on MI300X…" → shows real metrics: HBM 38%, MFU 24%.
4. Agent: "Querying ROCm playbook…" → 4 issues found, each with citation.
5. Agent: "Generating optimized config…" → diff appears: bf16, batch=12, flash-attn ROCm, hipBLASLt env, packed sequences.
6. Agent: "Benchmarking new config — 50 steps on MI300X…" → live progress.
7. Final report: 142 → 318 tokens/sec (2.24×). MFU 24% → 51%. $X saved per epoch.

Why it works (45s): Show the agent loop. Show the KB. Land the line: "the agent runs Qwen on AMD silicon, the audit target is Qwen on AMD silicon, the optimizations are AMD-specific. End-to-end AMD."

What This Is Not

• Not a generic PyTorch profiler GUI (rocprofv3 + tensorboard already exist)
• Not a chatbot wrapping a static FAQ (the agent runs real benchmarks)
• Not a multi-cloud cost tool (focused on MI300X compute waste, not infra cost)
• Not a fine-tuning trainer itself (we audit others' runs)

One-Line Pitch

GPU Goblin is the AMD performance engineer you wish was on your team — except it costs five minutes and audits any fine-tuning run on MI300X.