brainstorming/goals.md

GPU Goblin — Goals & Implementation Plan

North Star

Win the AMD hackathon (Track 1: AI Agents & Agentic Workflows) by demonstrating a real, reproducible 2×+ throughput improvement on a Qwen2.5-7B LoRA fine-tune on MI300X, driven end-to-end by a Qwen-powered tool-using agent. Hits the Qwen Technology Partner challenge with end-to-end Qwen-on-AMD; deploys as a Hugging Face Space within the hackathon HF Organization.

Everything else is in service of that demo.

Success Criteria

#	Criterion	Measurable As
1	Real MI300X speedup	≥ 2.0× tokens/sec, before vs. after, on the canonical workload
2	Agent is actually agentic	≥ 4 tool calls per audit, visible in UI
3	AMD differentiation	100% of recommendations cite a ROCm-specific KB rule
4	Uplift estimate honesty	Predicted speedup range (low–high) brackets the measured speedup on ≥ 8 of 10 synthetic corpus scenarios
5	Offline parse coverage	≥ 90% of synthetic corpus parsed without manual edits
6	End-to-end reliability	Demo runs cleanly 5× in a row without manual intervention
7	Pitch quality	3-min demo + 1-min architecture + 1-min impact, rehearsed
8	Backup safety	Offline-replay lane works fully when MI300X disconnected

Mapping to Judging Criteria

Hackathon criterion	Where we score
Application of Technology	Tool-using agent loop on MI300X; rocprofv3 + torch.profiler integration; deterministic uplift via waste-budget calculus; ROCm-specific KB
Presentation	Live before/after benchmark on real MI300X; waste-budget bar chart; chat-based "why?" interaction; 90-sec backup video
Business Value	Each audit reports $ saved per training run and time saved per epoch, framed against $1.99/hr public MI300X pricing reference
Originality	"AI for AI builders" — meta-positioning. Real benchmarks, not just LLM advice. Waste-budget decomposition is novel framing

Team Roles (3 people × 4 days)

ROCm / ML Lead

• Day 1: MI300X cloud env, ROCm container, baseline workload (Qwen2.5-7B LoRA on alpaca)
• Day 1-2: Hand-curate 20-25 KB rules from ROCm docs + AMD blog
• Day 2: profile_run + benchmark tools (rocprofv3 wrapper, parser)
• Day 3: Validate end-to-end speedup on canonical demo, generate cached results
• Owns: anything that touches the GPU

Agent / Backend Lead

• Day 1: FastAPI skeleton, tool schemas, Qwen-via-HF tool-use plumbing
• Day 1-2: parse_config, propose_patch, query_rocm_kb, compare_runs tools
• Day 2: Full agent loop, system prompt, SSE streaming
• Day 3: Hardening, error handling, max-steps cap, fallback behaviors
• Owns: agent reasoning quality + tool wiring

Frontend / Demo Lead

• Day 1: Streamlit skeleton, file upload, message log
• Day 2: Tool-call cards (live status), report renderer, diff viewer
• Day 3: Demo polish, charts, golden-run replay mode
• Day 4: Pitch deck, 90-second backup video, dry runs
• Owns: what the judges actually see

Roles overlap on integration days — pair-program when blocked.

Day-by-Day Plan

Day 1 (May 5–6) — Foundations

ROCm Lead

• Provision MI300X cloud instance via AMD Developer Cloud ($100 credits), SSH access, persistent storage
• Pull rocm/pytorch:rocm6.1_* image, verify GPU visible inside container
• Run baseline train_qwen_lora.py (batch=4, fp16, naive attention, alpaca, 100 steps)
• Capture baseline tokens/sec, MFU, HBM peak — this is our "before"
• Generate synthetic corpus — 5-8 misconfigured variants of the canonical workload (FP32, num_workers=0, naive attention, etc.) with cached RunMetrics JSON for each
• Start drafting KB rules (target 10 rules by EOD), each tagged with targets_bucket matching the waste-budget decomposition

Backend Lead

• Repo scaffold per architecture.md layout
• pip install fastapi anthropic sentence-transformers pyyaml pydantic
• Define agent/schemas.py with RunMetrics, WasteBudget, ConfigDict, Patch, Rule, Report as pydantic models — Day-1 priority (blocks all tools)
• Define RunnerProtocol interface; build FakeRunner that loads cached metrics from workloads/synthetic/ (lets backend dev without MI300X)
• FastAPI POST /audit skeleton with SSE
• parse_config tool — handle HF TrainingArguments first; include regex redaction pass for tokens/paths
• Qwen tool-use hello-world (one tool, one round-trip via HF Inference Providers)

Frontend Lead

• Streamlit skeleton with file upload + chat panel
• Hardcoded "fake audit" — render canned tool calls + report
• Pick chart library (Altair recommended for Streamlit)

Day 1 Exit Criteria

• Baseline benchmark numbers in hand
• Synthetic corpus has ≥ 3 cached scenarios (Backend Lead can now dev without GPU)
• Schemas (RunMetrics, WasteBudget, etc.) frozen
• RunnerProtocol + FakeRunner working end-to-end
• Backend can call Qwen with one tool via HF Inference Providers
• UI renders a fake audit
• 10 KB rules drafted

Day 2 (May 6–7) — Core Build

ROCm Lead

• Finish KB to 20-25 rules; hand-tag categories + targets_bucket; pre-embed with sentence-transformers
• Include the high-impact MI300X rules: BF16-over-FP16, AITER-flash-attn-via-Optimum-AMD, NCCL_MIN_NCHANNELS=112, NUMA disable, one-process-per-GPU, hipBLASLt hint logging, MIOpen MIOPEN_FIND_*, bitsandbytes-not-supported-on-ROCm warning, num_workers/pin_memory/prefetch_factor/persistent_workers
• profile_run tool: rocprofv3 wrapper + torch.profiler + amd-smi → RunMetrics with WasteBudget
• benchmark tool: same pipeline, longer run, with version-tagged cache
• Validate on baseline workload: profile output makes sense, MFU is plausible, waste budget sums to ~T_total

Backend Lead

• All 6 tools wired and individually tested with fixtures from synthetic corpus
• Full agent loop with max-steps cap, SSE event types finalized, error envelope per tool ({ok, result, error})
• System prompt iterated against test workloads — include MI300X hardware specs (304 CUs, 192 GB HBM3, ~5.3 TB/s, FP8 native) so the agent reasons quantitatively
• propose_patch deterministic transformer + uplift estimator (waste-budget × bucket recovery) + confidence formula

Frontend Lead

• Live tool-call cards consuming real SSE stream
• Final report layout: side-by-side metrics + diff + kernel chart + waste-budget bar chart (where time was lost, before vs after)
• Lane toggle: Live MI300X vs Offline replay (synthetic corpus) — judges can pick either
• First end-to-end run through the actual backend

Day 2 Exit Criteria

• Real audit runs end-to-end on a toy workload
• Profile + benchmark return real MI300X numbers
• KB has 20+ rules, embeddings pre-computed
• UI streams real agent activity

Day 3 (May 7–8) — Demo Day Prep

All hands

• Run canonical demo (Qwen2.5-7B LoRA) end-to-end → confirm ≥ 2× speedup
• Cache the demo benchmark results — don't burn cloud time on every rehearsal
• Build "golden run" replay mode (read cached SSE events, replay timing)
• Validate uplift accuracy on synthetic corpus: predicted range should bracket measured speedup on ≥ 8 of 10 scenarios
• Polish system prompt for demo-friendly narration ("I'll start by…")
• Tighten error handling — agent should never panic in front of judges; tool failures degrade gracefully with {ok:false} envelopes
• Run with --no-cache once to verify cached results aren't masking real bugs
• 5× clean dry runs, fix anything flaky

Day 3 Exit Criteria

• Canonical demo: cleanly runs in ≤ 4 minutes, ≥ 2× speedup, no manual fixes
• Cached results enable offline replay
• Offline-replay lane works fully when MI300X is disconnected (proven by unplugging cloud)
• Golden-run video recorded as backup

Day 4 (May 8–9) — Pitch & Stretch

Frontend Lead

• Pitch deck (5 slides): problem, agent loop diagram, demo (live), KB rules sample, impact + ask
• Cover image for the submission listing
• Final 90-second backup video
• Submission form filled (title, short + long description, tags), repo public, README crisp
• Build-in-Public bonus track: at least one public update post (X/LinkedIn/Discord) showing the agent's first audit; one ROCm/Optimum-AMD feedback note based on what was rough during build

ROCm + Backend Leads (in parallel, optional stretch)

• vLLM inference workload as second demo (only if rock-solid on Day 3)
• Cost calculator: $ saved per training run line, anchored on $1.99/hr public MI300X reference
• What-if slider panel for batch / precision / attention (chat already does this conversationally, sliders are visual icing)
• Stretch dream: self-host Qwen on MI300X via vLLM (replacing the HF-Inference-Providers path) — closes the loop entirely on AMD silicon. Mention in pitch even if not running live.

Day 4 Exit Criteria

• Submission complete by deadline
• 5+ rehearsed dry runs of the pitch
• Cover image, video, slides, repo all linked from submission form
• Backup video on USB stick, in cloud, on phone

Scope Discipline (YAGNI)

If we're behind schedule, cut in this order:

1. ❌ All stretch goals (vLLM, cost calc, self-hosted agent)
2. ❌ Live tool-call UI animations — static cards work
3. ❌ Some KB categories (keep precision/attention/memory; drop env_vars/collectives)
4. ❌ Multi-file script parsing — single-file only
5. ❌ Dynamic batch-size search — hardcode the recommended batch for demo workload

Never cut:

• Real MI300X benchmark (that's the entire pitch)
• Tool-using agent loop (that's the track fit)
• ROCm-specific KB citations (that's the differentiation)

Risk Register

Risk	Likelihood	Impact	Mitigation
MI300X cloud quota burns out	Medium	High	Cache every benchmark, dev with 10-step traces, full benchmarks only for canonical demo
rocprofv3 flaky / version mismatch	Medium	High	Always run torch.profiler in parallel as fallback. Pre-record golden trace. Use rocprofv3 not deprecated rocprof/rocprofv2
MI300X cloud unreachable on demo day	Low	Critical	Offline-replay lane (synthetic corpus + cached metrics) provides full demo without cloud — judges can't tell the difference for the agent-reasoning portion
Uploaded scripts contain user secrets / tokens	Medium	Medium	Regex redaction pass in parse_config before any persistence or LLM call; ephemeral storage; explicit "we never store weights or datasets" line in UI
Agent loops infinitely	Low	Medium	Hard cap of 8 tool calls, fall back to "best effort" report after cap
Recommendations are generic, not ROCm-specific	Medium	Critical	Hand-curate KB on Day 1 before wiring agent — KB is the moat
Demo crashes during pitch	Low	Critical	Pre-recorded video backup. Golden-run replay mode. USB + cloud + phone copies
Qwen2.5-7B doesn't fit a 12-batch on MI300X	Low	Medium	Have a fallback config in hand (batch=8 + grad_accum=2)
LoRA on alpaca too easy — speedup looks staged	Low	High	Measure on a non-trivial seq_len (1024+), include MFU not just tokens/sec, show kernel breakdown to prove it's real
HF Inference Provider rate limit / outage during demo	Low	Medium	Offline-replay UI lane plays cached_audit.json without any backend; pre-cache a full recorded session; have a backup HF token; provider="auto" already routes around individual provider outages
Team member unavailable (illness, etc.)	Low	High	Pair on critical path (agent loop, KB) so no single point of failure

Definition of Done — MVP

GPU Goblin is "done enough to ship" when all of these are true:

• ✅ A judge can upload train_qwen_lora.py (we provide it) and get a real audit
• ✅ Agent makes ≥ 4 visible tool calls
• ✅ Final report shows ≥ 2× tokens/sec, real numbers from MI300X
• ✅ Every recommendation in the report cites a ROCm KB rule by ID
• ✅ User can ask follow-up questions in chat ("why bf16?") and get cited answers
• ✅ Full audit completes in ≤ 4 minutes
• ✅ 5 consecutive dry runs succeed without manual intervention
• ✅ Backup video and cached replay both work

Stretch Definition of Done

• 🎯 Second canonical workload (vLLM inference) audited end-to-end
• 🎯 Cost calculator: "you save $X per training run, $Y per epoch"
• 🎯 Agent backed by self-hosted Qwen via vLLM on the same MI300X (the ultimate AMD story — replaces today's HF Inference Providers path with on-cluster serving)

Compute Budget — AMD Developer Cloud Credits

Eligible participants get $100 in AMD Developer Cloud credits. Public reference price for MI300X is around $1.99/hr (single VM) or $3.39/hr (8× bare metal). Plan accordingly:

Activity	GPU-hours	Notes
Day-1 baseline + synthetic corpus generation	4–6	One-shot, results cached
Day-2 KB validation runs	2–3	Sanity-check the rules fire on synthetic scenarios
Day-3 canonical demo dry runs (cached)	2–4	Cache hits after the first run
Day-3 --no-cache cold validation	1	Confirms nothing's stale
Day-4 final dry runs + record video	2–3	Lock the demo
Total estimate	~12–17 hrs	Well within $100 even at bare-metal rates

Backend Lead spends zero MI300X time after Day 1 — develops against synthetic corpus + FakeRunner.

Submission Checklist

• Public GitHub repo with clear README + architecture diagram + setup instructions
• 90-second demo video (live agent run, real MI300X numbers)
• Pitch deck (PDF or slides URL)
• Cover image (project listing visual)
• Architecture diagram (PNG)
• Sample audit report PDF (one canonical run, before/after, including waste-budget chart)
• Short description (1-2 sentences) + long description (paragraph) + technology/category tags
• At least one Build-in-Public post + one ROCm/Optimum-AMD feedback note (bonus track)
• Hackathon submission form filled
• Team member credits + contact