GPU Perf Prophet v1 — cross-vendor GPU performance forecasting & recommendation engine

Roofline-physics + XGBoost efficiency-gap model predicting inference throughput
across AMD Instinct and NVIDIA GPUs from MLPerf Inference v4.1-v6.0 data plus
self-run AMD Dev Cloud MI300X calibration, served via FastAPI + Streamlit and
deployed to Hugging Face Spaces.

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,48 @@
+# Private working notes — session journal, plan docs, checklists. Never commit.
+home/
+
+# MLPerf repos — large, not committed; populate via scripts/fetch_mlperf.sh
+data/raw/
+
+# Generated outputs
+data/processed/
+
+# MLflow tracking
+mlruns/
+mlartifacts/
+
+# Model artifacts
+*.pkl
+*.joblib
+*.ubj
+*.bin
+
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+.pytest_cache/
+.coverage
+htmlcov/
+dist/
+build/
+*.egg-info/
+
+# Environments
+.venv/
+venv/
+.env
+
+# Claude Code config (local only — contains machine-specific paths)
+.claude/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+
+# macOS
+.DS_Store
+
+# Jupyter
+.ipynb_checkpoints/

Dockerfile

@@ -0,0 +1,46 @@
+# GPU Perf Prophet — CPU-only HF Spaces compatible image
+# Target: 2 vCPU / 16 GB RAM (HF Spaces free tier)
+# Build context: project root (includes data/models/, data/gpu_specs.yaml,
+#                data/pricing.yaml, src/, app/)
+
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system deps required by some Python packages
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        libgomp1 \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Python deps first (layer cached unless requirements.txt changes)
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Create a non-root user; HF Spaces also runs as UID 1000 by convention.
+RUN useradd -m -u 1000 appuser
+
+# Copy source and data artifacts
+COPY src/       src/
+COPY app/       app/
+COPY data/gpu_specs.yaml  data/gpu_specs.yaml
+COPY data/pricing.yaml    data/pricing.yaml
+COPY data/models/         data/models/
+
+# Transfer ownership before dropping privileges.
+RUN chown -R appuser /app
+
+USER appuser
+
+# HF Spaces expects the app to listen on port 7860
+ENV PORT=7860
+
+# Expose both ports: Streamlit (7860) and FastAPI (8000)
+EXPOSE 7860
+EXPOSE 8000
+
+# Default: run Streamlit UI
+# Override CMD to run FastAPI instead:
+#   docker run ... uvicorn src.api.main:app --host 0.0.0.0 --port 8000
+CMD ["streamlit", "run", "app/streamlit_app.py", \
+     "--server.port=7860", "--server.address=0.0.0.0", \
+     "--server.headless=true", "--browser.gatherUsageStats=false"]

README.md

@@ -0,0 +1,92 @@
+---
+title: GPU Perf Prophet
+emoji: ⚡
+colorFrom: purple
+colorTo: blue
+sdk: docker
+app_port: 7860
+pinned: false
+---
+
+# GPU Perf Prophet
+
+**Cross-vendor LLM inference performance forecasting and hardware recommendation engine.**
+
+Predict throughput (tokens/sec) for major LLM workloads across AMD Instinct and NVIDIA GPU families — then get a Pareto-optimal recommendation ranked by throughput, cost-efficiency, and VRAM fit.
+
+---
+
+## What it does
+
+1. **Forecasts** per-GPU inference throughput for 5 LLM models × 8 GPU SKUs using a roofline physics model corrected by XGBoost trained on MLPerf Inference v4.1–v6.0 (1,112 benchmark rows).
+2. **Recommends** GPUs via multi-objective Pareto ranking across throughput, cost-efficiency (tok/$), and VRAM headroom.
+3. **Covers AMD Instinct** MI300X, MI325X, MI355X alongside NVIDIA H100, H200, A100, L4, RTX 4090.
+
+## Supported workloads
+
+| LLM | Params |
+|-----|--------|
+| Llama 2 70B | 70B |
+| Llama 3.1 8B | 8B |
+| Llama 3.1 405B | 405B |
+| Mixtral 8×7B | 46.7B total / 14.1B active |
+| GPT-J 6B | 6B |
+
+**Scenarios:** Offline · Server  
+**Accuracy tiers:** base (BF16) · 99 (FP8) · 99.9 (FP8 on AMD, FP16 on NVIDIA)  
+**Frameworks:** vLLM · TensorRT-LLM · ROCm/other
+
+## How to use
+
+1. Select an LLM model, scenario, accuracy tier, and framework in the sidebar.
+2. Optionally set a budget cap ($/GPU/hr) or minimum throughput threshold.
+3. Click **Recommend**.
+
+The app returns:
+- **Pareto-optimal GPUs** — no dominated option appears in this list.
+- **Top pick** — the frontier member with the highest cost-efficiency.
+- **Filtered GPUs** — candidates eliminated by VRAM, budget, or throughput constraints.
+- **AMD vs NVIDIA breakdown** — best per-vendor throughput at a glance.
+
+## Model accuracy (LOGO-CV on 5 in-scope GPU SKUs)
+
+| Metric | NVIDIA (H100, H200) | AMD (MI300X, MI325X, MI355X) |
+|--------|--------------------|-----------------------------|
+| Mean MAPE | ~21% | ~25% |
+| Spearman ρ | 0.885 | 0.598 |
+| Roofline violations | 0 / 461 rows | 0 / 188 rows |
+
+**Use for relative ranking and hardware shortlisting, not precise capacity planning.**  
+AMD predictions carry higher uncertainty due to smaller training corpus (188 vs 461 NVIDIA rows in MLPerf).
+
+## Architecture
+
+```
+MLPerf Inference v4.1–v6.0  →  Roofline model (physics ceiling)
+                             →  XGBoost efficiency-gap correction (20 features)
+                             →  Multi-objective Pareto recommender
+                             →  FastAPI backend + Streamlit UI
+```
+
+Key design principle borrowed from [NeuSight](https://arxiv.org/abs/2405.12031): physics-bounded ML generalizes to unseen GPUs; pure ML fails.
+
+## Data sources
+
+- **MLPerf Inference results** v4.1, v5.0, v5.1, v6.0 — [mlcommons/inference_results_*](https://github.com/mlcommons)
+- **GPU specs** — AMD and NVIDIA product pages (HBM bandwidth, TFLOPS, VRAM)
+- **Pricing** — static estimates as of June 2026 from cloud provider list prices
+
+## Limitations
+
+- Prices are static (June 2026). Cloud spot/reserved pricing varies significantly.
+- MI355X predictions have higher variance (50 training rows, CDNA4 architecture with limited cross-GPU training signal).
+- Multi-GPU scaling, training-time workloads, and Blackwell/MI400 families are out of scope.
+- No live API calls — all inference is local to the Docker container.
+
+## Tech stack
+
+Python · XGBoost · FastAPI · Streamlit · Docker · MLflow · SHAP
+
+---
+
+*Built by [Victoria Alabi](https://github.com/vxa8502) · Trained on MLPerf Inference v4.1–v6.0*

app/streamlit_app.py

@@ -0,0 +1,246 @@
+"""GPU Perf Prophet — Streamlit UI."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+# Ensure src/ is importable when running from the project root.
+# Use append (not insert) so the project root is searched LAST — if a file in
+# the project root happened to share a name with a stdlib module, insert(0)
+# would shadow the stdlib for the entire process.
+sys.path.append(str(Path(__file__).parent.parent))
+
+import pandas as pd
+import streamlit as st
+
+from src.models.predictor import GpuPredictor, VALID_MODELS
+from src.recommend.recommender import GpuRecommender
+
+_SORTED_MODELS: list[str] = sorted(VALID_MODELS)
+
+# ---------------------------------------------------------------------------
+# Page config
+# ---------------------------------------------------------------------------
+
+st.set_page_config(
+    page_title="GPU Perf Prophet",
+    layout="wide",
+)
+
+st.title("GPU Perf Prophet")
+st.caption(
+    "Cross-vendor LLM inference forecasting · AMD Instinct + NVIDIA · "
+    "Powered by roofline physics + XGBoost"
+)
+
+# ---------------------------------------------------------------------------
+# Load model (cached so it runs only once)
+# ---------------------------------------------------------------------------
+
+@st.cache_resource(show_spinner="Loading model …")
+def _load() -> tuple[GpuPredictor, GpuRecommender]:
+    pred = GpuPredictor()
+    rec  = GpuRecommender(pred)
+    return pred, rec
+
+
+predictor, recommender = _load()
+
+# ---------------------------------------------------------------------------
+# Sidebar — workload inputs
+# ---------------------------------------------------------------------------
+
+with st.sidebar:
+    st.header("Workload")
+
+    model_name = st.selectbox(
+        "LLM model",
+        options=_SORTED_MODELS,
+        index=_SORTED_MODELS.index("llama2-70b"),
+    )
+    scenario = st.selectbox("Scenario", ["Offline", "Server"])
+    accuracy_tier = st.selectbox(
+        "Accuracy tier",
+        ["99", "99.9", "base"],
+        help="99 → FP8 | 99.9 → FP8 (AMD) / FP16 (NVIDIA) | base → BF16",
+    )
+    framework = st.selectbox(
+        "Framework",
+        ["vllm", "tensorrt", "rocm_other", "other"],
+    )
+
+    st.divider()
+    st.header("Constraints (optional)")
+    budget = st.number_input(
+        "Max $/GPU/hr", min_value=0.0, max_value=20.0,
+        value=0.0, step=0.25,
+        help="Set to 0 to disable budget filter",
+    )
+    min_tput = st.number_input(
+        "Min throughput (tok/s)", min_value=0.0,
+        value=0.0, step=100.0,
+        help="Set to 0 to disable throughput filter",
+    )
+
+    run_btn = st.button("Recommend", use_container_width=True, type="primary")
+
+# ---------------------------------------------------------------------------
+# Main panel
+# ---------------------------------------------------------------------------
+
+if not run_btn:
+    st.info("Configure your workload in the sidebar and click **Recommend**.")
+    st.stop()
+
+with st.spinner("Running predictions …"):
+    result = recommender.recommend(
+        model_name=model_name,
+        scenario=scenario,
+        accuracy_tier=accuracy_tier,
+        framework=framework,
+        budget_per_gpu_hr=budget if budget > 0 else None,
+        min_throughput_tok_per_sec=min_tput if min_tput > 0 else None,
+    )
+
+workload = result["workload"]
+frontier = result["frontier"]
+dominated = result["dominated"]
+filtered = result["filtered"]
+
+# ---------------------------------------------------------------------------
+# Workload summary
+# ---------------------------------------------------------------------------
+
+col1, col2, col3, col4 = st.columns(4)
+col1.metric("Model", workload["model_name"])
+col2.metric("Model size", f"{workload['model_size_gb']:.1f} GB")
+col3.metric("Scenario", workload["scenario"])
+col4.metric("Accuracy tier", workload["accuracy_tier"])
+
+st.divider()
+
+# ---------------------------------------------------------------------------
+# Pareto frontier
+# ---------------------------------------------------------------------------
+
+_ALL_CANDIDATES = frontier + dominated
+
+if not _ALL_CANDIDATES:
+    st.warning("No GPUs passed the hard constraints (VRAM fit / budget / throughput).")
+    if filtered:
+        st.subheader("Filtered GPUs")
+        fdf = pd.DataFrame([
+            {
+                "GPU": r["gpu_name"],
+                "Vendor": r["vendor"].upper(),
+                "Pred. tput (tok/s)": f"{r['pred_throughput_tok_per_sec']:,.0f}",
+                "Reason": r["reject_reason"],
+            }
+            for r in filtered
+        ])
+        st.dataframe(fdf, use_container_width=True, hide_index=True)
+    st.stop()
+
+st.subheader("Pareto-Optimal Recommendations")
+st.caption(
+    "GPUs on the Pareto frontier are not dominated by any other candidate "
+    "across throughput, cost-efficiency (tok/$), and VRAM headroom."
+)
+
+def _make_table(rows: list[dict]) -> pd.DataFrame:
+    return pd.DataFrame([
+        {
+            "GPU":               r["gpu_name"],
+            "Vendor":            r["vendor"].upper(),
+            "Pred. tput (tok/s)": f"{r['pred_throughput_tok_per_sec']:,.0f}",
+            "Roofline (tok/s)":  f"{r['roofline_tput_tok_per_sec']:,.0f}",
+            "Efficiency":        f"{r['efficiency_ratio']:.2f}×",
+            "VRAM (GB)":         r["vram_gb"],
+            "Model (GB)":        f"{r['model_size_gb']:.1f}",
+            "VRAM headroom":     f"{r['vram_headroom']:.0%}",
+            "$/GPU/hr":          f"${r['price_per_gpu_hr']:.2f}" if r["price_per_gpu_hr"] else "—",
+            "Tok/$":             f"{r['cost_efficiency']:,.0f}" if r["cost_efficiency"] else "—",
+        }
+        for r in rows
+    ])
+
+if frontier:
+    st.dataframe(
+        _make_table(frontier),
+        use_container_width=True,
+        hide_index=True,
+    )
+
+    # Highlight the top pick
+    top = frontier[0]
+    st.success(
+        f"**Top pick: {top['gpu_name']}** — "
+        f"{top['pred_throughput_tok_per_sec']:,.0f} tok/s · "
+        f"${top['price_per_gpu_hr']:.2f}/hr · "
+        f"{top['cost_efficiency']:,.0f} tok/$ · "
+        f"{top['vram_headroom']:.0%} VRAM free"
+    )
+else:
+    st.info("No Pareto-optimal candidates after constraints.")
+
+# ---------------------------------------------------------------------------
+# All candidates (dominated)
+# ---------------------------------------------------------------------------
+
+if dominated:
+    with st.expander(f"Other passing GPUs ({len(dominated)} dominated)", expanded=False):
+        st.dataframe(
+            _make_table(dominated),
+            use_container_width=True,
+            hide_index=True,
+        )
+
+# ---------------------------------------------------------------------------
+# Filtered GPUs
+# ---------------------------------------------------------------------------
+
+if filtered:
+    with st.expander(f"Filtered out ({len(filtered)} GPUs)", expanded=False):
+        fdf = pd.DataFrame([
+            {
+                "GPU":    r["gpu_name"],
+                "Vendor": r["vendor"].upper(),
+                "Pred. tput (tok/s)": f"{r['pred_throughput_tok_per_sec']:,.0f}",
+                "Reason": r["reject_reason"],
+            }
+            for r in filtered
+        ])
+        st.dataframe(fdf, use_container_width=True, hide_index=True)
+
+# ---------------------------------------------------------------------------
+# AMD vs NVIDIA context
+# ---------------------------------------------------------------------------
+
+with st.expander("AMD vs NVIDIA breakdown", expanded=False):
+    all_rows = _ALL_CANDIDATES
+    amd_rows  = [r for r in all_rows if r["vendor"] == "amd"]
+    nvidia_rows = [r for r in all_rows if r["vendor"] == "nvidia"]
+
+    c1, c2 = st.columns(2)
+    with c1:
+        st.markdown("**AMD Instinct**")
+        if amd_rows:
+            best_amd = max(amd_rows, key=lambda r: r["pred_throughput_tok_per_sec"])
+            st.metric("Best throughput", f"{best_amd['pred_throughput_tok_per_sec']:,.0f} tok/s", best_amd["gpu_name"])
+        else:
+            st.info("No AMD GPUs passed filters.")
+    with c2:
+        st.markdown("**NVIDIA**")
+        if nvidia_rows:
+            best_nv = max(nvidia_rows, key=lambda r: r["pred_throughput_tok_per_sec"])
+            st.metric("Best throughput", f"{best_nv['pred_throughput_tok_per_sec']:,.0f} tok/s", best_nv["gpu_name"])
+        else:
+            st.info("No NVIDIA GPUs passed filters.")
+
+st.divider()
+st.caption(
+    "Predictions use a roofline physics model + XGBoost trained on MLPerf Inference v4.1–v6.0. "
+    "Prices are static estimates (June 2026). AMD MAPE ≈ 25%, NVIDIA MAPE ≈ 21% — use for "
+    "ranking, not precise capacity planning."
+)

benchmarks/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

benchmarks/merge_calibration_rows.py

@@ -0,0 +1,249 @@
+#!/usr/bin/env python3
+"""
+Merge MI300X calibration results into the GPU Perf Prophet training set.
+
+Run this on your local machine after SCP-ing mi300x_calibration_results.csv
+back from AMD Developer Cloud.
+
+Steps
+-----
+1. Reads mi300x_calibration_results.csv
+2. Constructs a raw-MLPerf-format DataFrame (same schema as the parser output)
+3. Calls build_training_df() to compute all features (roofline, precision, etc.)
+4. Appends to data/processed/mlperf_features.parquet
+5. Re-trains the production model via src/models/train_final.py
+
+Usage
+-----
+    python benchmarks/merge_calibration_rows.py --csv benchmarks/mi300x_calibration_results.csv
+    python benchmarks/merge_calibration_rows.py --csv benchmarks/mi300x_calibration_results.csv --dry-run
+    python benchmarks/merge_calibration_rows.py --csv benchmarks/mi300x_calibration_results.csv --no-retrain
+"""
+from __future__ import annotations
+
+import argparse
+import logging
+import subprocess
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+# Ensure project root is on the path (script is in benchmarks/, project root is parent)
+_PROJECT_ROOT = Path(__file__).parent.parent
+sys.path.append(str(_PROJECT_ROOT))
+
+from src.features.build_features import build_training_df  # noqa: E402
+from src.data.mlperf_parser import TOKENS_PER_SAMPLE       # noqa: E402
+
+log = logging.getLogger(__name__)
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s  %(levelname)-7s  %(message)s",
+    datefmt="%H:%M:%S",
+)
+
+_FEATURES_PARQUET = _PROJECT_ROOT / "data" / "processed" / "mlperf_features.parquet"
+
+# ---------------------------------------------------------------------------
+# Construct raw-MLPerf-format rows from calibration CSV
+# ---------------------------------------------------------------------------
+
+def _build_raw_df(csv_path: Path) -> pd.DataFrame:
+    """Read calibration CSV and return a DataFrame in raw-MLPerf parser format.
+
+    The schema matches what mlperf_parser.py produces, so build_training_df()
+    can process it without modification.
+    """
+    raw = pd.read_csv(csv_path)
+
+    # Drop failed runs (throughput == 0 means the run errored out)
+    failed = raw[raw["throughput_tok_per_sec"].astype(float) == 0.0]
+    if len(failed):
+        log.warning("Dropping %d failed runs: %s", len(failed),
+                    failed[["benchmark_base", "precision_used", "scenario"]].to_dict("records"))
+    raw = raw[raw["throughput_tok_per_sec"].astype(float) > 0.0].copy()
+
+    if raw.empty:
+        log.error("No successful runs in %s", csv_path)
+        sys.exit(1)
+
+    log.info("Loaded %d successful calibration rows from %s", len(raw), csv_path)
+
+    rows = []
+    for _, r in raw.iterrows():
+        bench_base = r["benchmark_base"]
+        tier       = r["benchmark_accuracy_tier"]
+        scenario   = r["scenario"]
+        tput       = float(r["throughput_tok_per_sec"])
+        tps        = TOKENS_PER_SAMPLE.get(bench_base, TOKENS_PER_SAMPLE.get(f"{bench_base}-{tier}"))
+
+        benchmark_name = bench_base if tier == "base" else f"{bench_base}-{tier}"
+
+        rows.append({
+            # MLPerf identity fields
+            "round":            r["round_tag"],
+            "division":         "open",
+            "submitter":        "vxa8502",
+            "system_name":      "1xMI300X_AMD_Dev_Cloud",
+            "gpu_name":         r["gpu_name"],
+            "num_gpus":         1,
+            "vram_gb":          192.0,   # MI300X per-GPU VRAM (matches spec DB)
+            "framework":        f"vLLM {r.get('vllm_version', '')}".strip(),
+            "system_type":      "single_node",
+            "hw_status":        "available",
+
+            # Benchmark identity
+            "benchmark":               benchmark_name,
+            "benchmark_base":          bench_base,
+            "benchmark_accuracy_tier": tier,
+            "scenario":                scenario,
+            "precision":               None,   # 0% populated in real MLPerf data — correct to leave null
+
+            # Throughput (the training target)
+            "throughput_tokens_per_sec":      tput,
+            "throughput_tok_per_sec_per_gpu": tput,   # single GPU
+
+            # Derived from throughput + tokens_per_sample
+            "throughput_samples_per_sec": tput / tps if tps else None,
+            "tokens_per_sample":          tps,
+
+            # Latency fields — not measured in throughput benchmark
+            "latency_mean_ms": None,
+            "latency_p99_ms":  None,
+            "ttft_mean_ms":    None,
+            "ttft_p99_ms":     None,
+            "tpot_mean_ms":    None,
+            "tpot_p99_ms":     None,
+
+            # Validation gate — True because we verified the run succeeded
+            "result_valid": True,
+
+            # Provenance
+            "log_path": f"benchmarks/results/{r.get('round_tag', 'self-run')}/{bench_base}_{tier}_{scenario}",
+        })
+
+    df = pd.DataFrame(rows)
+    log.info("Constructed %d raw rows ready for feature engineering", len(df))
+    return df
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Merge MI300X calibration results into training set")
+    parser.add_argument(
+        "--csv", required=True,
+        help="Path to mi300x_calibration_results.csv from AMD Dev Cloud",
+    )
+    parser.add_argument(
+        "--features-parquet", default=str(_FEATURES_PARQUET),
+        help="Path to existing mlperf_features.parquet (default: data/processed/mlperf_features.parquet)",
+    )
+    parser.add_argument(
+        "--dry-run", action="store_true",
+        help="Build features and print summary but do NOT write to disk or retrain",
+    )
+    parser.add_argument(
+        "--no-retrain", action="store_true",
+        help="Write updated parquet but skip model retraining",
+    )
+    args = parser.parse_args()
+
+    csv_path      = Path(args.csv)
+    parquet_path  = Path(args.features_parquet)
+
+    if not csv_path.exists():
+        log.error("CSV not found: %s", csv_path)
+        sys.exit(1)
+    if not parquet_path.exists():
+        log.error("Features parquet not found: %s", parquet_path)
+        sys.exit(1)
+
+    # 1 — Build raw rows and run feature engineering
+    raw_df = _build_raw_df(csv_path)
+    cal_features = build_training_df(raw_df)
+
+    if cal_features.empty:
+        log.error("build_training_df returned 0 rows — check GPU name aliases and benchmark_base values")
+        sys.exit(1)
+
+    log.info(
+        "Feature engineering complete: %d rows  |  "
+        "efficiency_ratio range [%.3f, %.3f]  |  "
+        "roofline violations: %d",
+        len(cal_features),
+        cal_features["efficiency_ratio"].min(),
+        cal_features["efficiency_ratio"].max(),
+        (cal_features["throughput_tok_per_sec_per_gpu"] > cal_features["roofline_tput"]).sum(),
+    )
+
+    # 2 — Load existing features and check for overlap
+    existing = pd.read_parquet(parquet_path)
+    log.info("Existing training set: %d rows", len(existing))
+
+    # Check for duplicates on (submitter, system_name, benchmark, scenario, round, division)
+    pk_cols = ["submitter", "system_name", "benchmark", "scenario", "round", "division"]
+    overlap_mask = cal_features[pk_cols].apply(tuple, axis=1).isin(
+        existing[pk_cols].apply(tuple, axis=1)
+    )
+    if overlap_mask.any():
+        log.warning(
+            "Dropping %d calibration rows that are already in the training set (PK collision)",
+            overlap_mask.sum(),
+        )
+        cal_features = cal_features[~overlap_mask]
+
+    if cal_features.empty:
+        log.info("No new rows to add — all calibration configs already present.")
+        return
+
+    # 3 — Print summary
+    print("\n=== Calibration rows to add ===")
+    print(
+        cal_features.groupby(["canonical_gpu_id", "benchmark_base", "benchmark_accuracy_tier", "scenario"])
+        [["throughput_tok_per_sec_per_gpu", "efficiency_ratio"]]
+        .agg({"throughput_tok_per_sec_per_gpu": "mean", "efficiency_ratio": "mean"})
+        .round(3)
+        .to_string()
+    )
+    print(f"\nTotal new rows: {len(cal_features)}")
+    print(f"MI300X rows before: {(existing.canonical_gpu_id == 'mi300x').sum()}")
+    print(f"MI300X rows after:  {(existing.canonical_gpu_id == 'mi300x').sum() + len(cal_features[cal_features.canonical_gpu_id == 'mi300x'])}")
+
+    if args.dry_run:
+        log.info("Dry run — no files written.")
+        return
+
+    # 4 — Concatenate and write
+    combined = pd.concat([existing, cal_features], ignore_index=True)
+    log.info("Combined training set: %d rows", len(combined))
+    combined.to_parquet(parquet_path, index=False)
+    log.info("Wrote updated parquet: %s", parquet_path)
+
+    if args.no_retrain:
+        log.info("Skipping retraining (--no-retrain).")
+        return
+
+    # 5 — Retrain
+    log.info("Retraining production model via train_final.py ...")
+    env = {**__import__("os").environ, "PYTHONPATH": str(_PROJECT_ROOT)}
+    result = subprocess.run(
+        [sys.executable, str(_PROJECT_ROOT / "src" / "models" / "train_final.py")],
+        cwd=str(_PROJECT_ROOT),
+        env=env,
+        check=False,
+    )
+    if result.returncode != 0:
+        log.error("train_final.py exited with code %d", result.returncode)
+        sys.exit(result.returncode)
+
+    log.info("Retraining complete. Run pytest to verify gate tests still pass:")
+    log.info("  pytest -m gate -v")
+    log.info("  pytest --cov=src --cov-fail-under=85")
+
+
+if __name__ == "__main__":
+    main()

benchmarks/mi300x_calibration_results.csv

@@ -0,0 +1,25 @@
+gpu_name,benchmark_base,benchmark_accuracy_tier,scenario,precision_used,hf_model_id,input_len,output_len,n_samples,total_output_tokens,elapsed_s,throughput_tok_per_sec,round_tag,vllm_version,rocm_version,notes
+AMD Instinct MI300X,gptj,base,Offline,bf16,EleutherAI/gpt-j-6b,1919,128,256,32768,7.382,4438.81,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,gptj,base,Server,bf16,EleutherAI/gpt-j-6b,1919,128,64,8192,2.683,3052.99,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,gptj,99,Offline,fp8,EleutherAI/gpt-j-6b,1919,128,256,32768,7.047,4649.89,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,gptj,99.9,Offline,fp8,EleutherAI/gpt-j-6b,1919,128,256,32768,7.047,4649.89,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,gptj,99,Server,fp8,EleutherAI/gpt-j-6b,1919,128,64,8192,2.604,3145.90,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,gptj,99.9,Server,fp8,EleutherAI/gpt-j-6b,1919,128,64,8192,2.604,3145.90,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama3.1-8b,base,Offline,bf16,meta-llama/Meta-Llama-3.1-8B,1024,294,128,37632,5.417,6947.42,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama3.1-8b,base,Server,bf16,meta-llama/Meta-Llama-3.1-8B,1024,294,32,9408,2.566,3667.02,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama3.1-8b,99,Offline,fp8,meta-llama/Meta-Llama-3.1-8B,1024,294,128,37632,4.851,7758.06,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama3.1-8b,99.9,Offline,fp8,meta-llama/Meta-Llama-3.1-8B,1024,294,128,37632,4.851,7758.06,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama3.1-8b,99,Server,fp8,meta-llama/Meta-Llama-3.1-8B,1024,294,32,9408,2.281,4124.51,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama3.1-8b,99.9,Server,fp8,meta-llama/Meta-Llama-3.1-8B,1024,294,32,9408,2.281,4124.51,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama2-70b,base,Offline,bf16,meta-llama/Llama-2-70b-hf,1024,294,64,18816,15.682,1199.83,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama2-70b,base,Server,bf16,meta-llama/Llama-2-70b-hf,1024,294,16,4704,13.055,360.31,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama2-70b,99,Offline,fp8,meta-llama/Llama-2-70b-hf,1024,294,64,18816,11.160,1686.04,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama2-70b,99.9,Offline,fp8,meta-llama/Llama-2-70b-hf,1024,294,64,18816,11.160,1686.04,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama2-70b,99,Server,fp8,meta-llama/Llama-2-70b-hf,1024,294,16,4704,9.364,502.35,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,llama2-70b,99.9,Server,fp8,meta-llama/Llama-2-70b-hf,1024,294,16,4704,9.364,502.35,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,mixtral-8x7b,base,Offline,bf16,mistralai/Mixtral-8x7B-v0.1,1024,145,128,18560,4.034,4600.38,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,mixtral-8x7b,base,Server,bf16,mistralai/Mixtral-8x7B-v0.1,1024,145,32,4640,2.755,1684.40,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,mixtral-8x7b,99,Offline,fp8,mistralai/Mixtral-8x7B-v0.1,1024,145,128,18560,3.355,5531.64,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,mixtral-8x7b,99.9,Offline,fp8,mistralai/Mixtral-8x7B-v0.1,1024,145,128,18560,3.355,5531.64,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,mixtral-8x7b,99,Server,fp8,mistralai/Mixtral-8x7B-v0.1,1024,145,32,4640,2.001,2318.51,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,
+AMD Instinct MI300X,mixtral-8x7b,99.9,Server,fp8,mistralai/Mixtral-8x7B-v0.1,1024,145,32,4640,2.001,2318.51,v6.0,0.23.0,ROCM-SMI version: 4.0.0+c2d9476115,

benchmarks/run_mi300x_calibration.py

@@ -0,0 +1,476 @@
+#!/usr/bin/env python3
+"""
+MI300X calibration benchmark runner for GPU Perf Prophet.
+
+Runs on AMD Developer Cloud (ROCm 7.x) with vLLM installed.
+Measures output throughput (tok/s) for each (model, precision, scenario) config
+and appends results to mi300x_calibration_results.csv.
+
+Requirements (AMD Dev Cloud):
+    pip install vllm>=0.6 huggingface_hub
+
+Gated models (llama2-70b, llama3.1-8b) require a HuggingFace token:
+    export HF_TOKEN=hf_...
+    huggingface-cli login --token $HF_TOKEN
+
+Usage:
+    python run_mi300x_calibration.py                         # full sweep
+    python run_mi300x_calibration.py --dry-run               # print configs only
+    python run_mi300x_calibration.py --models gptj mixtral-8x7b   # subset
+    python run_mi300x_calibration.py --resume                # skip completed rows
+
+Output:
+    mi300x_calibration_results.csv  (appended on each run; safe to resume)
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import gc
+import logging
+import os
+import sys
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Optional
+
+log = logging.getLogger(__name__)
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s  %(levelname)-7s  %(message)s",
+    datefmt="%H:%M:%S",
+)
+
+# ---------------------------------------------------------------------------
+# Sweep matrix
+# ---------------------------------------------------------------------------
+
+@dataclass
+class ModelConfig:
+    benchmark_base: str         # key used in GPU Perf Prophet MODEL_PARAMS
+    hf_model_id: str            # HuggingFace model identifier
+    input_len: int              # fixed prompt length in tokens
+    output_len: int             # max output tokens (matches MLPerf standard)
+    n_samples_offline: int      # number of prompts for Offline throughput run
+    n_samples_server: int       # number of prompts for Server (lower-concurrency) run
+    needs_hf_token: bool = False
+    max_model_len: Optional[int] = None  # set if model's default context is too large
+    # trust_remote_code must remain False unless the model repo uses a custom
+    # architecture class not built into vLLM — setting it True allows the model
+    # repo to execute arbitrary Python at load time (RCE if repo is compromised).
+    trust_remote_code: bool = False
+
+
+# MLPerf standard output lengths:
+#   gptj        128 tokens  (OpenOCR fixed)
+#   llama2-70b  294 tokens  (OpenOCR mean, MLPerf v4.1+)
+#   mixtral-8x7b 145 tokens (OpenOCR mean)
+#   llama3.1-8b  294 tokens (same OpenOCR dataset as llama2-70b per MLPerf rules)
+MODEL_CONFIGS: list[ModelConfig] = [
+    ModelConfig(
+        benchmark_base="gptj",
+        hf_model_id="EleutherAI/gpt-j-6b",
+        input_len=1919,
+        output_len=128,
+        n_samples_offline=256,
+        n_samples_server=64,
+        needs_hf_token=False,
+    ),
+    ModelConfig(
+        benchmark_base="llama3.1-8b",
+        hf_model_id="meta-llama/Meta-Llama-3.1-8B",
+        input_len=1024,
+        output_len=294,
+        n_samples_offline=128,
+        n_samples_server=32,
+        needs_hf_token=True,
+    ),
+    ModelConfig(
+        benchmark_base="llama2-70b",
+        hf_model_id="meta-llama/Llama-2-70b-hf",
+        input_len=1024,
+        output_len=294,
+        n_samples_offline=64,
+        n_samples_server=16,
+        needs_hf_token=True,
+    ),
+    ModelConfig(
+        benchmark_base="mixtral-8x7b",
+        hf_model_id="mistralai/Mixtral-8x7B-v0.1",
+        input_len=1024,
+        output_len=145,
+        n_samples_offline=128,
+        n_samples_server=32,
+        needs_hf_token=False,
+        max_model_len=4096,
+    ),
+]
+
+@dataclass
+class PrecisionConfig:
+    name: str                          # "bf16" or "fp8"
+    vllm_dtype: str                    # passed as dtype= to LLM()
+    vllm_quantization: Optional[str]   # passed as quantization= to LLM()
+    # MLPerf benchmark_accuracy_tiers this precision maps to.
+    # fp8 covers both "99" AND "99.9" for AMD (AMD achieves 99.9 accuracy with FP8).
+    tiers: list[str] = field(default_factory=list)
+
+
+PRECISION_CONFIGS: list[PrecisionConfig] = [
+    PrecisionConfig(
+        name="bf16",
+        vllm_dtype="bfloat16",
+        vllm_quantization=None,
+        tiers=["base"],
+    ),
+    PrecisionConfig(
+        name="fp8",
+        vllm_dtype="bfloat16",
+        vllm_quantization="fp8",
+        tiers=["99", "99.9"],   # AMD MI300X achieves both with FP8
+    ),
+]
+
+# MLPerf scenario names
+SCENARIOS = ["Offline", "Server"]
+
+# GPU identifier that must match an alias in data/gpu_specs.yaml
+GPU_NAME = "AMD Instinct MI300X"
+
+# Use the most recent MLPerf round tag so mlperf_round_num maps to 4 (max maturity)
+# in the training pipeline — we're running on a current ROCm/vLLM stack.
+ROUND_TAG = "v6.0"
+
+# ---------------------------------------------------------------------------
+# CSV schema
+# ---------------------------------------------------------------------------
+
+CSV_FIELDS = [
+    "gpu_name",
+    "benchmark_base",
+    "benchmark_accuracy_tier",
+    "scenario",
+    "precision_used",
+    "hf_model_id",
+    "input_len",
+    "output_len",
+    "n_samples",
+    "total_output_tokens",
+    "elapsed_s",
+    "throughput_tok_per_sec",
+    "round_tag",
+    "vllm_version",
+    "rocm_version",
+    "notes",
+]
+
+# ---------------------------------------------------------------------------
+# vLLM helpers
+# ---------------------------------------------------------------------------
+
+def _rocm_version() -> str:
+    """Return ROCm version string, or 'unknown'."""
+    try:
+        import subprocess
+        result = subprocess.run(
+            ["rocm-smi", "--version"], capture_output=True, text=True, timeout=5
+        )
+        for line in result.stdout.splitlines():
+            if "ROCm" in line or "rocm" in line.lower():
+                return line.strip()
+        return result.stdout.strip()[:80] or "unknown"
+    except Exception:
+        return "unknown"
+
+
+def _vllm_version() -> str:
+    try:
+        import vllm
+        return getattr(vllm, "__version__", "unknown")
+    except ImportError:
+        return "not_installed"
+
+
+def _load_model(config: ModelConfig, prec: PrecisionConfig) -> "vllm.LLM":
+    """Load a vLLM LLM instance with the given precision."""
+    from vllm import LLM
+
+    kwargs: dict = {
+        "model": config.hf_model_id,
+        "dtype": prec.vllm_dtype,
+        "gpu_memory_utilization": 0.93,
+        "enforce_eager": False,   # allow graph capture for throughput
+        "disable_log_stats": True,
+    }
+    if config.trust_remote_code:
+        kwargs["trust_remote_code"] = True
+    if prec.vllm_quantization:
+        kwargs["quantization"] = prec.vllm_quantization
+    if config.max_model_len:
+        kwargs["max_model_len"] = config.max_model_len
+
+    log.info(
+        "Loading %s  dtype=%s  quant=%s",
+        config.hf_model_id, prec.vllm_dtype, prec.vllm_quantization or "none",
+    )
+    t0 = time.perf_counter()
+    llm = LLM(**kwargs)
+    log.info("Model loaded in %.1f s", time.perf_counter() - t0)
+    return llm
+
+
+def _unload_model(llm) -> None:
+    """Release GPU memory held by an LLM instance."""
+    try:
+        import torch
+        del llm
+        gc.collect()
+        torch.cuda.empty_cache()
+    except Exception:
+        pass
+
+
+def _make_prompts(n: int, input_len: int) -> list[list[int]]:
+    """Return n identical prompt_token_ids of length input_len.
+
+    Using token ID 1 (BOS in most tokenizers) gives exact token-count control
+    without running a tokenizer.  The content doesn't affect throughput measurement.
+    """
+    # Each prompt must be an independent list — vLLM may mutate the lists
+    # during tokenization, and [prompt] * n would share a single object.
+    prompt = [1] * input_len
+    return [list(prompt) for _ in range(n)]
+
+
+def _run_throughput(
+    llm,
+    config: ModelConfig,
+    n_samples: int,
+    *,
+    warmup: bool = True,
+) -> tuple[float, int]:
+    """Run a throughput pass and return (elapsed_seconds, total_output_tokens).
+
+    If warmup=True, runs one warm-up batch first (results discarded).
+    """
+    from vllm import SamplingParams
+
+    params = SamplingParams(
+        temperature=0.0,
+        max_tokens=config.output_len,
+        ignore_eos=True,   # always generate exactly output_len tokens
+    )
+
+    if warmup:
+        log.info("  Warmup pass (%d samples)...", min(n_samples, 8))
+        warmup_prompts = _make_prompts(min(n_samples, 8), config.input_len)
+        llm.generate(warmup_prompts, sampling_params=params, use_tqdm=False)
+
+    prompts = _make_prompts(n_samples, config.input_len)
+    log.info("  Timed pass (%d samples × %d output tokens)...", n_samples, config.output_len)
+
+    t0 = time.perf_counter()
+    outputs = llm.generate(prompts, sampling_params=params, use_tqdm=False)
+    elapsed = time.perf_counter() - t0
+
+    total_out = sum(len(o.outputs[0].token_ids) for o in outputs)
+    return elapsed, total_out
+
+
+# ---------------------------------------------------------------------------
+# CSV helpers
+# ---------------------------------------------------------------------------
+
+def _completed_keys(csv_path: Path) -> set[tuple]:
+    """Return set of (benchmark_base, precision_used, scenario) already in CSV."""
+    if not csv_path.exists():
+        return set()
+    keys = set()
+    with csv_path.open(newline="") as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            keys.add((row["benchmark_base"], row["precision_used"], row["scenario"]))
+    return keys
+
+
+def _append_row(csv_path: Path, row: dict) -> None:
+    write_header = not csv_path.exists()
+    with csv_path.open("a", newline="") as f:
+        writer = csv.DictWriter(f, fieldnames=CSV_FIELDS)
+        if write_header:
+            writer.writeheader()
+        writer.writerow(row)
+
+
+# ---------------------------------------------------------------------------
+# Main sweep
+# ---------------------------------------------------------------------------
+
+def run_sweep(
+    model_filter: Optional[list[str]],
+    resume: bool,
+    dry_run: bool,
+    output: Path,
+) -> None:
+    models = [m for m in MODEL_CONFIGS if model_filter is None or m.benchmark_base in model_filter]
+    if not models:
+        log.error("No models matched filter %s", model_filter)
+        sys.exit(1)
+
+    vllm_ver  = _vllm_version()
+    rocm_ver  = _rocm_version()
+    completed = _completed_keys(output) if resume else set()
+
+    log.info("vLLM %s  |  ROCm: %s", vllm_ver, rocm_ver)
+    log.info("Output: %s", output)
+
+    # Count total configs
+    total = sum(len(PRECISION_CONFIGS) * len(SCENARIOS) for _ in models)
+    log.info("Sweep: %d model×precision×scenario configs", total)
+
+    if dry_run:
+        for m in models:
+            for prec in PRECISION_CONFIGS:
+                for scenario in SCENARIOS:
+                    n = m.n_samples_offline if scenario == "Offline" else m.n_samples_server
+                    key = (m.benchmark_base, prec.name, scenario)
+                    status = "SKIP (completed)" if key in completed else "RUN"
+                    log.info(
+                        "  [%s] %s | %s | %s | n=%d | input=%d out=%d",
+                        status, m.benchmark_base, prec.name, scenario,
+                        n, m.input_len, m.output_len,
+                    )
+        return
+
+    hf_token = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+
+    for m in models:
+        if m.needs_hf_token and not hf_token:
+            log.warning(
+                "Skipping %s — set HF_TOKEN env var to access gated model.",
+                m.hf_model_id,
+            )
+            continue
+
+        for prec in PRECISION_CONFIGS:
+            # Check if all scenarios for this (model, prec) are already done
+            needed_scenarios = [
+                s for s in SCENARIOS
+                if (m.benchmark_base, prec.name, s) not in completed
+            ]
+            if not needed_scenarios:
+                log.info("Skipping %s/%s — all scenarios already completed.", m.benchmark_base, prec.name)
+                continue
+
+            # Load model once per (model, precision) pair
+            llm = None
+            try:
+                llm = _load_model(m, prec)
+            except Exception as exc:
+                log.error("Failed to load %s/%s: %s", m.benchmark_base, prec.name, exc)
+                if llm:
+                    _unload_model(llm)
+                continue
+
+            for scenario in needed_scenarios:
+                n_samples = m.n_samples_offline if scenario == "Offline" else m.n_samples_server
+                log.info(
+                    "Running %s | prec=%s | scenario=%s | n=%d",
+                    m.benchmark_base, prec.name, scenario, n_samples,
+                )
+                try:
+                    elapsed, total_out = _run_throughput(llm, m, n_samples, warmup=(scenario == "Offline"))
+                    tput = total_out / elapsed
+                    log.info(
+                        "  => %.1f tok/s  (%d tokens in %.1f s)",
+                        tput, total_out, elapsed,
+                    )
+                    # Write one row per (model, precision, scenario, tier) combination.
+                    # fp8 covers both "99" and "99.9" tiers for AMD MI300X.
+                    for tier in prec.tiers:
+                        row = {
+                            "gpu_name":                GPU_NAME,
+                            "benchmark_base":          m.benchmark_base,
+                            "benchmark_accuracy_tier": tier,
+                            "scenario":                scenario,
+                            "precision_used":          prec.name,
+                            "hf_model_id":             m.hf_model_id,
+                            "input_len":               m.input_len,
+                            "output_len":              m.output_len,
+                            "n_samples":               n_samples,
+                            "total_output_tokens":     total_out,
+                            "elapsed_s":               f"{elapsed:.3f}",
+                            "throughput_tok_per_sec":  f"{tput:.2f}",
+                            "round_tag":               ROUND_TAG,
+                            "vllm_version":            vllm_ver,
+                            "rocm_version":            rocm_ver,
+                            "notes":                   "",
+                        }
+                        _append_row(output, row)
+                    completed.add((m.benchmark_base, prec.name, scenario))
+
+                except Exception as exc:
+                    log.error(
+                        "Run failed for %s/%s/%s: %s",
+                        m.benchmark_base, prec.name, scenario, exc,
+                    )
+                    _append_row(output, {
+                        "gpu_name":                GPU_NAME,
+                        "benchmark_base":          m.benchmark_base,
+                        "benchmark_accuracy_tier": prec.tiers[0] if prec.tiers else "?",
+                        "scenario":                scenario,
+                        "precision_used":          prec.name,
+                        "hf_model_id":             m.hf_model_id,
+                        "input_len":               m.input_len,
+                        "output_len":              m.output_len,
+                        "n_samples":               n_samples,
+                        "total_output_tokens":     0,
+                        "elapsed_s":               0,
+                        "throughput_tok_per_sec":  0,
+                        "round_tag":               ROUND_TAG,
+                        "vllm_version":            vllm_ver,
+                        "rocm_version":            rocm_ver,
+                        "notes":                   f"FAILED: {exc}",
+                    })
+
+            _unload_model(llm)
+
+    log.info("Sweep complete. Results: %s", output)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="MI300X calibration benchmark runner")
+    parser.add_argument(
+        "--models", nargs="+",
+        choices=[m.benchmark_base for m in MODEL_CONFIGS],
+        help="Run only these models (default: all)",
+    )
+    parser.add_argument(
+        "--dry-run", action="store_true",
+        help="Print sweep plan without running",
+    )
+    parser.add_argument(
+        "--resume", action="store_true",
+        help="Skip configs already present in the output CSV",
+    )
+    parser.add_argument(
+        "--output", default="mi300x_calibration_results.csv",
+        help="Output CSV path (default: mi300x_calibration_results.csv)",
+    )
+    args = parser.parse_args()
+
+    run_sweep(
+        model_filter=args.models,
+        resume=args.resume,
+        dry_run=args.dry_run,
+        output=Path(args.output),
+    )
+
+
+if __name__ == "__main__":
+    main()

data/data_card.md

@@ -0,0 +1,193 @@
+# GPU Perf Prophet — Data Card v0.1
+
+> **Status:** Hand-authored baseline. This file will be superseded once `build_data_card()`
+> is implemented and `make data` is run.
+> All figures verified against `data/processed/mlperf_raw.parquet` on 2026-06-12.
+
+---
+
+## 1 · Provenance
+
+| Field | Value |
+|-------|-------|
+| Primary source | MLCommons MLPerf Inference results repos v4.1, v5.0, v5.1, v6.0 |
+| Divisions | closed + open |
+| Ingestor | `src/data/mlperf_parser.py` |
+| GPU spec enrichment | `src/data/gpu_spec_db.py` → `data/gpu_specs.yaml` |
+| Processed artifact | `data/processed/mlperf_raw.parquet` |
+| Generated | 2026-06-12 |
+| Total parsed rows | **1,223** |
+| Unique submitters | 32 |
+| Unique system configs | 167 |
+| Result validity | 100% VALID (zero INVALID rows in corpus) |
+
+Source repos are sparse-checked-out git clones. Fetch scripts in `scripts/fetch_mlperf.sh`.
+Every parsed row carries `log_path` back to its originating `mlperf_log_summary.txt`.
+
+---
+
+## 2 · In-scope training corpus
+
+Rows entering model training must pass the `gpu_in_model_scope == True` filter
+(set in `data/gpu_specs.yaml`). CPU-only, heterogeneous, edge, and other
+out-of-scope GPUs are excluded.
+
+**Total in-scope rows: 649**
+
+### Per-GPU breakdown
+
+| Canonical ID | GPU | Vendor | Rounds present | In-scope rows | Spec confidence |
+|---|---|---|---|---|---|
+| `mi300x` | AMD Instinct MI300X | AMD | v4.1, v5.0, v5.1, v6.0 | 56 | verified |
+| `mi325x` | AMD Instinct MI325X | AMD | v5.0, v5.1, v6.0 | 82 | verified |
+| `mi355x` | AMD Instinct MI355X | AMD | **v6.0 only** | **50** | estimated |
+| `h100_sxm` | NVIDIA H100 SXM5 | NVIDIA | v4.1, v5.0, v5.1 | 178 | verified |
+| `h200_sxm` | NVIDIA H200 SXM | NVIDIA | v4.1, v5.0, v5.1 | 283 | verified |
+
+AMD total: **188 rows** (29 %)  
+NVIDIA total: **461 rows** (71 %)
+
+The AMD/NVIDIA imbalance is the reason the AMD-specific MAPE target is relaxed to
+< 20 % (vs. < 15 % overall).
+
+### Per-round in-scope row counts
+
+| Round | mi300x | mi325x | mi355x | h100_sxm | h200_sxm | Round total |
+|-------|-------:|-------:|-------:|---------:|---------:|------------:|
+| v4.1 | 16 | — | — | 116 | 68 | 200 |
+| v5.0 | 4 | 16 | — | 54 | 156 | 230 |
+| v5.1 | 28 | 62 | — | 8 | 59 | 157 |
+| v6.0 | 8 | 4 | **50** | — | — | 62 |
+| **Total** | **56** | **82** | **50** | **178** | **283** | **649** |
+
+---
+
+## 3 · MI355X — v6.0-only GPU
+
+**MI355X is the only in-scope GPU with data from a single round.**  
+This is the primary known limitation of the training corpus.
+
+### What "v6.0-only" means
+
+| Property | Detail |
+|---|---|
+| First MLPerf round | v6.0 (no MI355X rows in v4.1, v5.0, or v5.1) |
+| In-scope training rows | 50 |
+| Benchmarks covered | `llama2-70b-99` (25 rows) and `llama2-70b-99.9` (25 rows) only |
+| Benchmarks *not* covered | `gptj`, `mixtral-8x7b`, `llama3.1-405b` |
+| Scenarios | Offline and Server |
+| Spec confidence | **estimated** — CDNA4 dense FLOPS derived from with-sparsity ÷ 2 |
+
+### Why this matters for the model
+
+1. **No round-over-round variance.** The model cannot check MI355X consistency across
+   rounds, making it harder to detect submission anomalies or performance regressions.
+
+2. **Single benchmark family.** The model sees MI355X only on `llama2-70b`. Predictions
+   for MI355X on `gptj`, `mixtral-8x7b`, or `llama3.1-405b` are extrapolations,
+   not interpolations. SHAP explanations will not decompose MI355X benchmark-level
+   effects from GPU-level effects for those benchmarks.
+
+3. **50 rows < 100-row minimum.** If MI355X does not meet the 100-row minimum gate
+   after further AMD Dev Cloud calibration, it will be marked `enabled: false` in
+   `gpu_specs.yaml` and excluded from v1 recommendations. The current value is
+   `in_model_scope: true` as a placeholder pending that gate.
+
+4. **Estimated spec values.** `gpu_specs.yaml` MI355X entries for `peak_tflops` are
+   `sparse/2` derivations (flagged `spec_confidence: estimated`). These **must be
+   reconfirmed** against the AMD MI350-series whitepaper before the training data
+   is finalized.
+
+### Mitigation plan
+
+| Step | Action |
+|------|--------|
+| 1 | Train with MI355X included; report MI355X MAPE separately; note in model card |
+| 2 | Run MI300X + MI355X benchmarks on AMD Dev Cloud; add calibration rows |
+| 3 | If MI355X row count ≥ 100 → keep `enabled: true`; else set `enabled: false` |
+| 4 | Public writeup explicitly discloses the single-round limitation |
+
+---
+
+## 4 · Excluded rows
+
+| Category | Count | Reason |
+|---|---:|---|
+| CPU-inference (`gpu_name = N/A`) | 26 | Intel EMR and similar; no GPU to predict |
+| NVIDIA Blackwell (B200, B300, GB200, GB300) | 206 | Explicitly out of v1 scope |
+| NVIDIA RTX PRO 6000 Blackwell | 52 | Not in spec DB; out of scope |
+| Heterogeneous multi-GPU | 20 | Mixed SKUs; cannot normalize to a single canonical ID |
+| Intel Arc | 16 | Not in spec DB |
+| Edge GPUs (Jetson) | 4 | Edge, not datacenter |
+| GH200, H100-NVL, H100-PCIe, H200-NVL, L40S | 250 | Insufficient LLM rows or out of form-factor scope |
+| **Total excluded** | **574** | |
+
+Excluded rows are present in `mlperf_raw.parquet` with `gpu_in_model_scope` null or false.
+They are **not removed** — exclusion happens at training time via the scope filter.
+
+---
+
+## 5 · Feature encoding decisions
+
+### `precision` field
+
+`precision` is 0 % populated across the entire corpus. MLPerf system names are hardware
+identifiers (e.g., `8xMI300X_2xEPYC-9374F`), not precision-tagged. Regex extraction
+returns `None` on all real submissions.
+
+**`benchmark_accuracy_tier` is the reliable precision proxy** (100 % populated):
+
+| Tier | Benchmark suffix | Typical precision regime |
+|------|-----------------|--------------------------|
+| `99.9` | `-99.9` | BF16 / FP16 (high-fidelity) |
+| `99` | `-99` | FP8 or better |
+| `base` | none | Throughput-optimized; any precision |
+
+Distribution: `99` → 550 rows, `99.9` → 488 rows, `base` → 185 rows.
+
+### `tokens_per_sample` (verified 2026-06-12)
+
+These values are the divisor in `throughput_tok_per_sec_per_gpu`.
+
+| Benchmark base | Value used | Source-verified | Delta | Source |
+|---|---:|---:|---:|---|
+| `gptj` | 128 | 128 (fixed) | 0.0 % | MLPerf benchmark spec |
+| `llama2-70b` | 294 | 294.45 | < 0.2 % | `mlcommons/inference` language/llama2-70b/README.md |
+| `mixtral-8x7b` | 145 | 144.84 | < 0.1 % | `mlcommons/inference` language/mixtral-8x7b README |
+| `llama3.1-405b` | 294 | 294.45 | < 0.2 % | Same Open ORCA dataset as llama2-70b per MLPerf rules |
+
+All values verified within ± 5 % tolerance. No parquet re-export required.
+
+### `vram_gb` vs `gpu_vram_gb`
+
+`vram_gb` (from the system JSON `accelerator_memory_capacity`) can reflect **total
+system VRAM** for multi-GPU submissions. `gpu_vram_gb` (from `gpu_specs.yaml`) is
+always per-GPU capacity.
+
+**Rule: use `gpu_vram_gb` for the memory-fit constraint. Treat `vram_gb` as unreliable
+for any system with `num_gpus > 1`.**
+
+---
+
+## 6 · Deduplication policy
+
+No deduplication is applied. Rows from different rounds for the same GPU + benchmark
+combination are intentional — they represent genuinely independent performance measurements
+submitted to different MLPerf rounds.
+
+**Primary key:** `(submitter, system_name, benchmark, scenario, round, division)`  
+**Duplicate rows in corpus:** 0
+
+---
+
+## 7 · Known limitations
+
+| ID | Limitation | Impact | Mitigation |
+|----|-----------|--------|-----------|
+| KL-01 | **MI355X v6.0-only** — single round, one benchmark family, 50 rows | AMD MAPE for MI355X has higher variance; no multi-round consistency check | Further AMD Dev Cloud calibration; 100-row gate |
+| KL-02 | `precision` field is 0 % populated | Cannot distinguish FP8 vs FP16 directly | Use `benchmark_accuracy_tier` as proxy |
+| KL-03 | `tokens_per_sample` are rounded estimates | < 0.2 % systematic bias in `throughput_tok_per_sec_per_gpu` | Verified; rounding error is negligible |
+| KL-04 | AMD corpus is 29 % of in-scope rows | AMD predictions are lower-confidence than NVIDIA | Relaxed AMD MAPE gate (< 20 % vs < 15 %) |
+| KL-05 | No INVALID rows in current corpus | INVALID filter in training code is not currently exercised | Filter remains; future rounds may include INVALID |
+| KL-06 | MI355X CDNA4 spec values are estimated | Roofline ceilings for MI355X may be slightly off | Reconfirm against AMD MI350 whitepaper before finalizing training data |
+| KL-07 | `TOKENS_PER_SAMPLE` for llama3.1-405b inherits llama2-70b value | If Open ORCA output lengths differ across models, minor bias | Verify if llama3.1-405b round adds dataset-specific stats |

data/gpu_specs.yaml

@@ -0,0 +1,507 @@
+##############################################################################
+# GPU Perf Prophet — GPU Specification Database
+# schema_version: "1.0"
+#
+# Accuracy policy
+# ───────────────
+# spec_confidence: "verified"   → from vendor spec sheet / whitepaper
+#                  "estimated"  → derived from architecture docs or cross-refs;
+#                                 verify before Phase-2 training freeze
+#
+# peak_tflops are DENSE (no structured sparsity) per design doc ADR-006.
+# AMD "compute" throughput uses MFMA instruction peak.
+# NVIDIA "compute" throughput uses Tensor Core peak for BF16/FP16/FP8;
+#   fp32 is scalar CUDA-core throughput.
+# hbm_bandwidth_tbps is peak theoretical from memory specification.
+# ~ (null) means not supported natively on that GPU.
+#
+# in_model_scope
+# ──────────────
+# true  → GPU is a Phase-1 recommendation target
+# false → GPU appears in the training corpus but is not exposed to users v1
+#         (Blackwell family, edge GPUs, GPU combos without clean single-SKU rows)
+#
+# aliases
+# ───────
+# Every unique accelerator_model_name string that appeared in MLPerf
+# v4.1–v6.0 system JSONs for this GPU, including case variants and
+# suffix variants without the count specifier "(xN)".
+##############################################################################
+
+schema_version: "1.0"
+
+gpus:
+
+  # ── AMD Instinct CDNA3 ─────────────────────────────────────────────────
+
+  - id: mi300x
+    name: "AMD Instinct MI300X"
+    vendor: amd
+    architecture: cdna3
+    memory_type: hbm3
+    vram_gb: 192
+    hbm_bandwidth_tbps: 5.3
+    peak_tflops:
+      fp32: 163.4
+      bf16: 1307.4
+      fp16: 1307.4
+      fp8:  2614.9
+      fp6:  ~             # not supported natively in CDNA3
+      fp4:  ~             # not supported natively in CDNA3
+      int8: 2614.9
+    compute_units: 304
+    l2_cache_mb: 32
+    tdp_w: 750
+    spec_confidence: verified   # AMD MI300X Data Sheet, Dec 2023
+    in_model_scope: true
+    aliases:
+      - "AMD Instinct MI300X"
+      - "AMD Instinct MI300X 192GB HBM3"
+      - "AMD Instinct MI300X-NPS1-SPX-192GB-750W"
+      - "AMD MI300X-NPS1-SPX-192GB-750W"
+
+  - id: mi325x
+    name: "AMD Instinct MI325X"
+    vendor: amd
+    architecture: cdna3             # same compute die as MI300X; HBM3E memory upgrade
+    memory_type: hbm3e
+    vram_gb: 256
+    hbm_bandwidth_tbps: 6.0
+    peak_tflops:
+      fp32: 163.4                   # same CDNA3 compute die
+      bf16: 1307.4
+      fp16: 1307.4
+      fp8:  2614.9
+      fp6:  ~
+      fp4:  ~
+      int8: 2614.9
+    compute_units: 304
+    l2_cache_mb: 32
+    tdp_w: 1000
+    spec_confidence: verified   # AMD MI325X Announcement, Oct 2024
+    in_model_scope: true
+    aliases:
+      - "AMD Instinct MI325X"
+      - "AMD Instinct MI325X 256GB HBM3E"
+      - "AMD Instinct MI325X 256GB HBM3e"     # lowercase variant in system JSONs
+
+  # ── AMD Instinct CDNA4 ─────────────────────────────────────────────────
+
+  - id: mi355x
+    name: "AMD Instinct MI355X"
+    vendor: amd
+    architecture: cdna4
+    memory_type: hbm3e
+    vram_gb: 288
+    hbm_bandwidth_tbps: 8.0         # AMD official spec sheet (amd-instinct-mi355x-gpu-brochure.pdf)
+    peak_tflops:
+      fp32: 157.3                   # AMD official spec sheet — vector FP32; CDNA4 ratio = 16× (vs CDNA3 8×)
+      bf16: 2516.6                  # AMD official spec sheet — matrix
+      fp16: 2516.6                  # AMD official spec sheet — matrix
+      fp8:  5033.2                  # AMD official spec sheet — matrix
+      fp6:  10066.3                 # AMD official spec sheet — CDNA4 native FP6
+      fp4:  20132.6                 # AMD official spec sheet — MXFP4
+      int8: 5033.2                  # AMD official spec sheet
+    compute_units: 304              # (est) — not listed on spec sheet; no change
+    l2_cache_mb: 32                 # (est)
+    tdp_w: 1400                     # AMD official spec sheet (1.4 kW); 1000 W variant observed in MLPerf
+    spec_confidence: verified        # confirmed 2026-06-17 against AMD official spec sheet
+    in_model_scope: true
+    aliases:
+      - "AMD Instinct MI355X"
+      - "AMD Instinct MI355X 288GB HBM3e"
+      - "AMD Instinct MI355X 288GB HBM3e (Power Cap 1000 W)"
+      - "AMD Instinct MI355X 288GB HBM3e (x87)"  # (x87) stripped by normalizer
+
+  - id: mi350x
+    name: "AMD Instinct MI350X"
+    vendor: amd
+    architecture: cdna4
+    memory_type: hbm3e
+    vram_gb: 288
+    hbm_bandwidth_tbps: 8.0         # AMD official spec sheet (amd-instinct-mi350x-gpu-brochure.pdf)
+    peak_tflops:
+      fp32: 144.2                   # vector FP32; 256 CUs × same clock as MI355X die; confirmed via platform FP16-vector 1153.6 TFLOPS ÷ 8
+      bf16: 2307.0                  # matrix dense; derived from confirmed fp8 (4614.0) using CDNA4 2:1 ratio (verified against MI355X)
+      fp16: 2307.0                  # matrix dense; same derivation
+      fp8:  4614.0                  # matrix dense; AMD official spec sheet — platform 36.9 PFLOPS ÷ 8 GPUs = 4612.5 ≈ 4614 TFLOPS
+      fp6:  9228.0                  # matrix dense; derived from confirmed fp8 × 2 (CDNA4 native MXFP6 support confirmed)
+      fp4:  18456.0                 # matrix dense; derived from confirmed fp8 × 4 (CDNA4 native MXFP4 support confirmed)
+      int8: 4614.0                  # same as fp8 per AMD CDNA4 pattern
+    compute_units: 256              # AMD official spec sheet — 8 XCDs × 32 CUs each
+    l2_cache_mb: 32                 # (est)
+    tdp_w: 1000                     # AMD official spec sheet — air-cooled 1000W variant (vs MI355X 1400W)
+    spec_confidence: verified       # confirmed 2026-06-18 against AMD official spec sheet (amd-instinct-mi350x-gpu-brochure.pdf); bf16/fp6/fp4 derived from confirmed fp8 using CDNA4 ratios
+    in_model_scope: false           # spec gate cleared (S33); scope gate blocked: LOGO-CV ρ=0.019 (14 rows, 35 rounds) — wait for v7.0+ MLPerf data
+    aliases:
+      - "AMD Instinct MI350X"
+      - "AMD Instinct MI350X 288GB HBM3e"
+
+  # ── NVIDIA Hopper ──────────────────────────────────────────────────────
+
+  - id: h100_sxm
+    name: "NVIDIA H100 SXM5"
+    vendor: nvidia
+    architecture: hopper
+    memory_type: hbm3
+    vram_gb: 80
+    hbm_bandwidth_tbps: 3.35
+    peak_tflops:
+      fp32: 66.9                    # scalar CUDA-core FP32
+      bf16: 989.4                   # Tensor Core, dense
+      fp16: 989.4
+      fp8:  1978.9
+      fp6:  ~
+      fp4:  ~
+      int8: 3957.8
+    streaming_multiprocessors: 132
+    l2_cache_mb: 50
+    tdp_w: 700
+    spec_confidence: verified   # NVIDIA H100 GPU Architecture Whitepaper
+    in_model_scope: true
+    aliases:
+      - "NVIDIA H100-SXM-80GB"
+      - "Virtualized NVIDIA H100-SXM-80GB"
+
+  - id: h100_pcie
+    name: "NVIDIA H100 PCIe"
+    vendor: nvidia
+    architecture: hopper
+    memory_type: hbm2e
+    vram_gb: 80
+    hbm_bandwidth_tbps: 2.0
+    peak_tflops:
+      fp32: 51.2
+      bf16: 756.5
+      fp16: 756.5
+      fp8:  1513.0
+      fp6:  ~
+      fp4:  ~
+      int8: 3026.0
+    streaming_multiprocessors: 114
+    l2_cache_mb: 50
+    tdp_w: 350
+    spec_confidence: verified   # NVIDIA H100 PCIe Datasheet
+    in_model_scope: false       # not in v1 scope; included for training signal
+    aliases:
+      - "NVIDIA H100-PCIe-80GB"
+
+  - id: h100_nvl
+    name: "NVIDIA H100 NVL"
+    vendor: nvidia
+    architecture: hopper
+    memory_type: hbm3
+    vram_gb: 94
+    hbm_bandwidth_tbps: 3.93        # (est) scaled from H100 SXM die bandwidth
+    peak_tflops:
+      fp32: 66.9
+      bf16: 989.4                   # same compute die as H100 SXM5
+      fp16: 989.4
+      fp8:  1978.9
+      fp6:  ~
+      fp4:  ~
+      int8: 3957.8
+    streaming_multiprocessors: 132
+    l2_cache_mb: 50
+    tdp_w: 700                      # (est)
+    spec_confidence: estimated
+    in_model_scope: false
+    aliases:
+      - "NVIDIA H100-NVL-94GB"
+
+  - id: h200_sxm
+    name: "NVIDIA H200 SXM"
+    vendor: nvidia
+    architecture: hopper            # same Hopper die as H100; HBM3E memory upgrade
+    memory_type: hbm3e
+    vram_gb: 141
+    hbm_bandwidth_tbps: 4.8
+    peak_tflops:
+      fp32: 66.9                    # same compute die as H100 SXM5
+      bf16: 989.4
+      fp16: 989.4
+      fp8:  1978.9
+      fp6:  ~
+      fp4:  ~
+      int8: 3957.8
+    streaming_multiprocessors: 132
+    l2_cache_mb: 50
+    tdp_w: 700
+    spec_confidence: verified   # NVIDIA H200 Tensor Core GPU Datasheet
+    in_model_scope: true
+    aliases:
+      - "NVIDIA H200-SXM-141GB"
+      - "NVIDIA H200-SXM-141GB-CTS"
+      - "Virtualized NVIDIA H200-SXM-141GB"
+
+  - id: h200_nvl
+    name: "NVIDIA H200 NVL"
+    vendor: nvidia
+    architecture: hopper
+    memory_type: hbm3e
+    vram_gb: 141
+    hbm_bandwidth_tbps: 4.8         # same HBM3E spec as H200 SXM
+    peak_tflops:
+      fp32: 66.9
+      bf16: 989.4
+      fp16: 989.4
+      fp8:  1978.9
+      fp6:  ~
+      fp4:  ~
+      int8: 3957.8
+    streaming_multiprocessors: 132
+    l2_cache_mb: 50
+    tdp_w: 600                      # (est) NVL form-factor typically lower TDP
+    spec_confidence: estimated
+    in_model_scope: false           # not in v1 scope; included for training signal
+    aliases:
+      - "NVIDIA H200-NVL-141GB"
+
+  # GH200 = H100 GPU die + Grace CPU on NVLink-C2C.  GPU-side specs only.
+  - id: gh200_96gb
+    name: "NVIDIA GH200 96GB"
+    vendor: nvidia
+    architecture: hopper
+    memory_type: hbm3
+    vram_gb: 96
+    hbm_bandwidth_tbps: 3.35        # H100 GPU die bandwidth
+    peak_tflops:
+      fp32: 66.9
+      bf16: 989.4
+      fp16: 989.4
+      fp8:  1978.9
+      fp6:  ~
+      fp4:  ~
+      int8: 3957.8
+    streaming_multiprocessors: 132
+    l2_cache_mb: 50
+    tdp_w: 900                      # full GH200 module TDP
+    spec_confidence: verified
+    in_model_scope: false
+    aliases:
+      - "NVIDIA GH200 Grace Hopper Superchip 96GB"
+
+  - id: gh200_144gb
+    name: "NVIDIA GH200 144GB"
+    vendor: nvidia
+    architecture: hopper
+    memory_type: hbm3e
+    vram_gb: 144
+    hbm_bandwidth_tbps: 4.9         # (est) HBM3E variant
+    peak_tflops:
+      fp32: 66.9
+      bf16: 989.4
+      fp16: 989.4
+      fp8:  1978.9
+      fp6:  ~
+      fp4:  ~
+      int8: 3957.8
+    streaming_multiprocessors: 132
+    l2_cache_mb: 50
+    tdp_w: 900
+    spec_confidence: estimated
+    in_model_scope: false
+    aliases:
+      - "NVIDIA GH200 Grace Hopper Superchip 144GB"
+
+  # ── NVIDIA Ampere ──────────────────────────────────────────────────────
+  # A100 has zero LLM rows in v4.1–v6.0 corpus; specs kept for the
+  # recommendation engine (roofline-only fallback for unseen GPUs).
+
+  - id: a100_sxm_80gb
+    name: "NVIDIA A100 SXM4 80GB"
+    vendor: nvidia
+    architecture: ampere
+    memory_type: hbm2e
+    vram_gb: 80
+    hbm_bandwidth_tbps: 2.0
+    peak_tflops:
+      fp32: 77.6                    # scalar CUDA-core FP32
+      bf16: 312.0                   # Tensor Core, dense
+      fp16: 312.0
+      fp8:  ~                       # FP8 not supported natively on Ampere
+      fp6:  ~
+      fp4:  ~
+      int8: 624.0
+    streaming_multiprocessors: 108
+    l2_cache_mb: 40
+    tdp_w: 400
+    spec_confidence: verified   # NVIDIA A100 GPU Architecture Whitepaper
+    in_model_scope: true        # in v1 scope; roofline-only prediction (no training rows)
+    aliases:
+      - "NVIDIA A100-SXM4-80GB"
+      - "NVIDIA A100-SXM-80GB"
+      - "NVIDIA A100 80GB"
+
+  # ── NVIDIA Ada Lovelace ────────────────────────────────────────────────
+  # L4 and RTX 4090 have zero LLM rows in corpus; included for recommender.
+
+  - id: l4
+    name: "NVIDIA L4"
+    vendor: nvidia
+    architecture: ada_lovelace
+    memory_type: gddr6
+    vram_gb: 24
+    hbm_bandwidth_tbps: 0.300       # 300 GB/s GDDR6
+    peak_tflops:
+      fp32: 30.3
+      bf16: 121.6                   # Tensor Core, dense
+      fp16: 121.6
+      fp8:  242.6
+      fp6:  ~
+      fp4:  ~
+      int8: 485.0
+    streaming_multiprocessors: 60
+    l2_cache_mb: 48
+    tdp_w: 72
+    spec_confidence: verified   # NVIDIA L4 Tensor Core GPU Datasheet
+    in_model_scope: true        # in v1 scope; roofline-only (no training rows)
+    aliases:
+      - "NVIDIA L4"
+      - "NVIDIA L4 24GB"
+
+  - id: l40s
+    name: "NVIDIA L40S"
+    vendor: nvidia
+    architecture: ada_lovelace
+    memory_type: gddr6
+    vram_gb: 48
+    hbm_bandwidth_tbps: 0.864       # 864 GB/s GDDR6
+    peak_tflops:
+      fp32: 91.6
+      bf16: 362.1
+      fp16: 362.1
+      fp8:  724.1
+      fp6:  ~
+      fp4:  ~
+      int8: 1448.2
+    streaming_multiprocessors: 142
+    l2_cache_mb: 96
+    tdp_w: 350
+    spec_confidence: verified   # NVIDIA L40S Datasheet
+    in_model_scope: false
+    aliases:
+      - "NVIDIA L40S"
+
+  - id: rtx4090
+    name: "NVIDIA GeForce RTX 4090"
+    vendor: nvidia
+    architecture: ada_lovelace
+    memory_type: gddr6x
+    vram_gb: 24
+    hbm_bandwidth_tbps: 1.008       # GDDR6X
+    peak_tflops:
+      fp32: 82.6
+      bf16: 165.2                   # Tensor Core, dense
+      fp16: 165.2
+      fp8:  330.3                   # (est) 2× FP16
+      fp6:  ~
+      fp4:  ~
+      int8: 660.6
+    streaming_multiprocessors: 128
+    l2_cache_mb: 72
+    tdp_w: 450
+    spec_confidence: verified   # NVIDIA RTX 4090 Product Specifications
+    in_model_scope: true        # in v1 scope; roofline-only (no training rows)
+    aliases:
+      - "NVIDIA GeForce RTX 4090"
+      - "NVIDIA RTX 4090"
+
+  # ── NVIDIA Blackwell — out of v1 scope; included for training signal ──
+  # Rows exist in corpus; having roofline bounds enables the model to train
+  # on them.  They are never surfaced as recommendations in v1.
+
+  - id: b200_sxm
+    name: "NVIDIA B200 SXM"
+    vendor: nvidia
+    architecture: blackwell
+    memory_type: hbm3e
+    vram_gb: 180                    # as reported in MLPerf system JSONs
+    hbm_bandwidth_tbps: 8.0         # (est) Blackwell specification
+    peak_tflops:
+      fp32: 147.0                   # (est)
+      bf16: 2250.0                  # (est)
+      fp16: 2250.0                  # (est)
+      fp8:  4500.0                  # (est)
+      fp6:  ~
+      fp4:  9000.0                  # (est) — Blackwell MXFP4
+      int8: 9000.0                  # (est)
+    streaming_multiprocessors: 160  # (est)
+    l2_cache_mb: 96                 # (est)
+    tdp_w: 1000                     # (est)
+    spec_confidence: estimated
+    in_model_scope: false           # Blackwell explicitly out of v1 scope
+    aliases:
+      - "NVIDIA B200-SXM-180GB"
+
+  - id: b300_sxm
+    name: "NVIDIA B300 SXM"
+    vendor: nvidia
+    architecture: blackwell
+    memory_type: hbm3e
+    vram_gb: 270                    # as reported in MLPerf system JSONs
+    hbm_bandwidth_tbps: 10.0        # (est)
+    peak_tflops:
+      fp32: 147.0                   # (est)
+      bf16: 2250.0                  # (est)
+      fp16: 2250.0                  # (est)
+      fp8:  4500.0                  # (est)
+      fp6:  ~
+      fp4:  9000.0                  # (est)
+      int8: 9000.0                  # (est)
+    streaming_multiprocessors: 160  # (est)
+    l2_cache_mb: 96                 # (est)
+    tdp_w: 1200                     # (est)
+    spec_confidence: estimated
+    in_model_scope: false
+    aliases:
+      - "NVIDIA B300-SXM-270GB"
+
+  # GB200 NVL module = 2× B200 GPU dies + 2× Grace CPUs.
+  # Per-module specs listed here; num_gpus normalization treats the module
+  # as the unit (matching how MLPerf system JSONs report accelerators_per_node).
+  - id: gb200
+    name: "NVIDIA GB200 NVL"
+    vendor: nvidia
+    architecture: blackwell
+    memory_type: hbm3e
+    vram_gb: 384                    # 2× 192 GB per NVL72 spec (est)
+    hbm_bandwidth_tbps: 16.0        # (est) 2× B200
+    peak_tflops:
+      fp32: 294.0                   # (est) 2× B200
+      bf16: 4500.0                  # (est) 2× B200
+      fp16: 4500.0                  # (est)
+      fp8:  9000.0                  # (est)
+      fp6:  ~
+      fp4:  18000.0                 # (est)
+      int8: 18000.0                 # (est)
+    streaming_multiprocessors: 320  # (est) 2× B200
+    l2_cache_mb: 192                # (est)
+    tdp_w: 2700                     # (est) full NVL module
+    spec_confidence: estimated
+    in_model_scope: false
+    aliases:
+      - "NVIDIA GB200"
+
+  - id: gb300
+    name: "NVIDIA GB300"
+    vendor: nvidia
+    architecture: blackwell
+    memory_type: hbm3e
+    vram_gb: 384                    # (est)
+    hbm_bandwidth_tbps: 16.0        # (est)
+    peak_tflops:
+      fp32: 294.0                   # (est)
+      bf16: 4500.0                  # (est)
+      fp16: 4500.0                  # (est)
+      fp8:  9000.0                  # (est)
+      fp6:  ~
+      fp4:  18000.0                 # (est)
+      int8: 18000.0                 # (est)
+    streaming_multiprocessors: 320  # (est)
+    l2_cache_mb: 192                # (est)
+    tdp_w: 2700                     # (est)
+    spec_confidence: estimated
+    in_model_scope: false
+    aliases:
+      - "NVIDIA GB300"

data/models/feature_metadata.json

@@ -0,0 +1,37 @@
+{
+  "feature_cols": [
+    "gpu_hbm_bandwidth_tbps",
+    "gpu_vram_gb",
+    "peak_tflops_selected",
+    "compute_ceiling_tok_per_sec",
+    "bandwidth_ceiling_tok_per_sec",
+    "model_total_params_b",
+    "model_compute_params_b",
+    "model_size_gb",
+    "model_to_vram_ratio",
+    "bytes_per_param",
+    "nvidia_arch_gen",
+    "amd_arch_gen",
+    "vendor_is_amd",
+    "scenario_offline",
+    "is_base_tier",
+    "fw_tensorrt",
+    "fw_vllm",
+    "fw_rocm_other",
+    "is_cdna4",
+    "mlperf_round_num"
+  ],
+  "target": "efficiency_ratio",
+  "model_version": "v1",
+  "prod_params": {
+    "n_estimators": 250,
+    "max_depth": 5,
+    "learning_rate": 0.03,
+    "subsample": 0.8,
+    "colsample_bytree": 0.8,
+    "min_child_weight": 5,
+    "tree_method": "hist",
+    "random_state": 42
+  },
+  "n_training_rows": 1136
+}

data/pricing.yaml

@@ -0,0 +1,52 @@
+##############################################################################
+# GPU Perf Prophet — Static Cloud GPU Pricing
+#
+# source_date: 2026-06-17
+# Policy: static snapshot — no live API calls. Prices are per-GPU per-hour
+#         on a single on-demand instance. Costs vary by cloud provider,
+#         region, and contract term; these are representative on-demand
+#         rates for planning purposes only.
+#
+# Sources (June 2026):
+#   AMD Developer Cloud (amd.com/en/developer/resources/cloud-access.html)
+#   Lambda Labs (lambdalabs.com/service/gpu-cloud)
+#   Vast.ai / RunPod community rates (runpod.io/gpu-instance/pricing)
+#   getdeploying.com/gpus/amd-mi355x (multi-provider comparison)
+#
+# price_per_gpu_hr: USD, on-demand, single GPU, no reserved/spot discount
+# source: human-readable provenance note
+##############################################################################
+
+pricing:
+
+  mi300x:
+    price_per_gpu_hr: 1.99
+    source: "AMD Developer Cloud on-demand, June 2026"
+
+  mi325x:
+    price_per_gpu_hr: 2.50
+    source: "AMD Developer Cloud est. (MI325X premium over MI300X), June 2026"
+
+  mi355x:
+    price_per_gpu_hr: 4.50
+    source: "Mid-market on-demand est., June 2026 (range: $2.95 TensorWave – $8.60 OCI; mkt avg $5.45; up 232% since Oct 2025)"
+
+  h100_sxm:
+    price_per_gpu_hr: 2.49
+    source: "Lambda Labs H100 SXM5 80GB on-demand, June 2026"
+
+  h200_sxm:
+    price_per_gpu_hr: 3.99
+    source: "Lambda Labs H200 SXM 141GB on-demand est., June 2026"
+
+  a100_sxm_80gb:
+    price_per_gpu_hr: 1.50
+    source: "Lambda Labs A100 SXM4 80GB on-demand, June 2026"
+
+  l4:
+    price_per_gpu_hr: 0.44
+    source: "Vast.ai / RunPod L4 on-demand est., June 2026"
+
+  rtx4090:
+    price_per_gpu_hr: 0.39
+    source: "RunPod RTX 4090 on-demand, June 2026"

requirements.txt

@@ -0,0 +1,28 @@
+# Data processing
+pandas>=2.0
+numpy>=1.26
+pyarrow>=14.0        # Parquet I/O
+pyyaml>=6.0          # GPU/LLM spec databases
+
+# ML
+scikit-learn>=1.4
+xgboost>=2.0
+lightgbm>=4.0
+shap>=0.45
+optuna>=3.5
+
+# Experiment tracking
+mlflow>=2.10
+
+# API + serving
+fastapi>=0.110
+uvicorn[standard]>=0.27
+pydantic>=2.6
+
+# UI
+streamlit>=1.32
+
+# Testing
+pytest>=8.0
+pytest-cov>=4.1
+httpx>=0.27           # required by FastAPI TestClient (starlette.testclient)

scripts/fetch_mlperf.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Sparse-clone MLPerf Inference results repos for GPU Perf Prophet.
+#
+# Fetches only the files the parser needs from each round:
+#   closed/*/systems/*.json              (GPU hardware descriptions)
+#   closed/*/results/**/mlperf_log_summary.txt  (performance logs)
+#
+# Repos are placed under data/raw/mlperf/<version>/.
+# Full clones of these repos can exceed several GB; sparse checkout
+# keeps each round under ~50 MB.
+#
+# Usage:
+#   bash scripts/fetch_mlperf.sh              # all four rounds
+#   bash scripts/fetch_mlperf.sh v6.0         # single round
+#   ROUNDS="v5.1 v6.0" bash scripts/fetch_mlperf.sh
+
+set -euo pipefail
+
+DEST="data/raw/mlperf"
+ORG="mlcommons"
+ROUNDS="${ROUNDS:-v4.1 v5.0 v5.1 v6.0}"
+
+# If a round argument is passed on the command line, use that instead.
+if [[ $# -ge 1 ]]; then
+    ROUNDS="$*"
+fi
+
+mkdir -p "$DEST"
+
+for ROUND in $ROUNDS; do
+    # Validate before using $ROUND in path or URL construction.
+    # Accepts only the form vN.N (e.g. v6.0) to prevent path traversal.
+    if ! [[ "$ROUND" =~ ^v[0-9]+\.[0-9]+$ ]]; then
+        echo "Error: invalid round tag '${ROUND}' — expected vN.N (e.g. v6.0)" >&2
+        exit 1
+    fi
+
+    REPO="${ORG}/inference_results_${ROUND}"
+    TARGET="${DEST}/${ROUND}"
+
+    if [[ -d "$TARGET/.git" ]]; then
+        echo "→ ${ROUND}: already cloned at ${TARGET}, skipping."
+        continue
+    fi
+
+    echo "→ Cloning ${REPO} (sparse) into ${TARGET} …"
+
+    git clone \
+        --filter=blob:none \
+        --sparse \
+        --depth=1 \
+        --no-checkout \
+        "https://github.com/${REPO}.git" \
+        "$TARGET"
+
+    pushd "$TARGET" > /dev/null
+
+    # Fetch only the paths the parser needs.
+    # --no-cone: use gitignore-style patterns so * wildcards are accepted.
+    # (default cone mode rejects wildcards with "specify directories rather
+    # than patterns")
+    git sparse-checkout set --no-cone \
+        "closed/*/systems" \
+        "closed/*/results/*/llama2-70b/*/performance" \
+        "closed/*/results/*/llama2-70b-99/*/performance" \
+        "closed/*/results/*/llama2-70b-99.9/*/performance" \
+        "closed/*/results/*/mixtral-8x7b/*/performance" \
+        "closed/*/results/*/mixtral-8x7b-99/*/performance" \
+        "closed/*/results/*/mixtral-8x7b-99.9/*/performance" \
+        "closed/*/results/*/llama3.1-405b/*/performance" \
+        "closed/*/results/*/llama3.1-405b-99/*/performance" \
+        "closed/*/results/*/llama3.1-405b-99.9/*/performance" \
+        "closed/*/results/*/gptj/*/performance" \
+        "closed/*/results/*/gptj-99/*/performance" \
+        "closed/*/results/*/gptj-99.9/*/performance"
+
+    git checkout
+
+    popd > /dev/null
+
+    echo "   Done: ${TARGET}"
+done
+
+echo ""
+echo "All requested rounds fetched. Run the parser with:"
+echo "  python -m src.data.mlperf_parser --repos-dir ${DEST} --parquet"

setup.cfg

@@ -0,0 +1,9 @@
+[tool:pytest]
+markers =
+    gate: GATE-level reliability target — blocks merge if failing (see JOURNAL.md §"Reliability Targets")
+
+[flake8]
+max-line-length = 100
+# E203: whitespace before ':' — conflicts with slice notation
+# W503: line break before binary operator — conflicts with Black style
+extend-ignore = E203, W503

src/api/main.py

@@ -0,0 +1,195 @@
+"""
+GPU Perf Prophet — FastAPI application.
+
+Routes
+------
+GET  /health                   liveness probe
+GET  /gpus                     list all GPU ids in the spec DB
+GET  /models                   list all supported LLM model names
+POST /predict                  predict throughput for one (GPU, workload) pair
+POST /predict/batch            vectorised prediction for up to 50 pairs
+POST /recommend                Pareto GPU recommendation for a workload
+"""
+
+from __future__ import annotations
+
+import logging
+from contextlib import asynccontextmanager
+from typing import Annotated
+
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import Response
+from pydantic import Field
+
+from src.api.schemas import (
+    PredictRequest,
+    PredictResponse,
+    RecommendRequest,
+    RecommendResponse,
+)
+from src.data.gpu_spec_db import load_specs
+from src.models.predictor import GpuPredictor, VALID_MODELS
+from src.recommend.recommender import GpuRecommender
+
+# Hard cap on raw request body size — prevents parsing-based DoS before any
+# Pydantic validation runs.  50 PredictRequest objects at ~200 bytes each ≈
+# 10 KB; 1 MB is a generous ceiling that still blocks runaway bodies.
+_MAX_BODY_BYTES = 1 * 1024 * 1024  # 1 MB
+
+log = logging.getLogger(__name__)
+
+_predictor: GpuPredictor | None = None
+_recommender: GpuRecommender | None = None
+_gpu_list: list[dict] | None = None
+_model_list: list[str] = sorted(VALID_MODELS)
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    global _predictor, _recommender, _gpu_list
+    log.info("Loading model …")
+    _predictor = GpuPredictor()
+    _recommender = GpuRecommender(_predictor)
+    specs = load_specs()
+    _gpu_list = [
+        {
+            "id": s["id"],
+            "name": s.get("name", s["id"]),
+            "vendor": s.get("vendor"),
+            "architecture": s.get("architecture"),
+            "vram_gb": s.get("vram_gb"),
+            "in_model_scope": s.get("in_model_scope", False),
+        }
+        for s in specs
+    ]
+    log.info("Ready.")
+    yield
+    _predictor = None
+    _recommender = None
+    _gpu_list = None
+
+
+app = FastAPI(
+    title="GPU Perf Prophet",
+    description=(
+        "Cross-vendor LLM inference performance forecasting "
+        "and workload-aware GPU recommendation."
+    ),
+    version="1.0.0",
+    lifespan=lifespan,
+)
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["GET", "POST"],
+    allow_headers=["Content-Type"],
+)
+
+
+@app.middleware("http")
+async def limit_body_size(request: Request, call_next) -> Response:
+    # Fast path: reject if Content-Length header exceeds the limit.
+    # A malformed or missing Content-Length header is not an error here —
+    # it just means we must check the actual body below.
+    cl = request.headers.get("content-length")
+    if cl is not None:
+        try:
+            if int(cl) > _MAX_BODY_BYTES:
+                return Response(status_code=413, content="Request body too large")
+        except ValueError:
+            return Response(status_code=400, content="Invalid Content-Length header")
+
+    # GET / HEAD / OPTIONS carry no request body — no route handler reads one,
+    # so the size limit is irrelevant and buffering is wasted work.
+    if request.method in {"GET", "HEAD", "OPTIONS"}:
+        return await call_next(request)
+
+    # Slow path: consume the actual body so chunked-transfer or lying clients
+    # cannot bypass the Content-Length check above.  Without this step, a
+    # client sending without Content-Length (chunked encoding) would bypass
+    # the limit entirely because uvicorn imposes no default body size cap.
+    body = await request.body()
+    if len(body) > _MAX_BODY_BYTES:
+        return Response(status_code=413, content="Request body too large")
+
+    # Re-inject the consumed body so downstream route handlers can read it.
+    async def _receive() -> dict:
+        return {"type": "http.request", "body": body, "more_body": False}
+
+    request._receive = _receive
+    return await call_next(request)
+
+
+def _get_predictor() -> GpuPredictor:
+    if _predictor is None:
+        raise HTTPException(status_code=503, detail="Model not loaded")
+    return _predictor
+
+
+def _get_recommender() -> GpuRecommender:
+    if _recommender is None:
+        raise HTTPException(status_code=503, detail="Recommender not loaded")
+    return _recommender
+
+
+# ---------------------------------------------------------------------------
+# Routes
+# ---------------------------------------------------------------------------
+
+@app.get("/health")
+def health() -> dict:
+    return {"status": "ok", "model_loaded": _predictor is not None}
+
+
+@app.get("/gpus")
+def list_gpus() -> dict:
+    return {"gpus": _gpu_list or []}
+
+
+@app.get("/models")
+def list_models() -> dict:
+    return {"models": _model_list}
+
+
+@app.post("/predict", response_model=PredictResponse)
+def predict(req: PredictRequest) -> dict:
+    predictor = _get_predictor()
+    try:
+        return predictor.predict(
+            gpu_id=req.gpu_id,
+            model_name=req.model_name,
+            scenario=req.scenario,
+            accuracy_tier=req.accuracy_tier,
+            framework=req.framework,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+
+
+@app.post("/predict/batch", response_model=list[PredictResponse])
+def predict_batch(
+    requests: Annotated[list[PredictRequest], Field(max_length=50)],
+) -> list[dict]:
+    predictor = _get_predictor()
+    try:
+        return predictor.predict_batch([r.model_dump() for r in requests])
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+
+
+@app.post("/recommend", response_model=RecommendResponse)
+def recommend(req: RecommendRequest) -> dict:
+    recommender = _get_recommender()
+    try:
+        return recommender.recommend(
+            model_name=req.model_name,
+            scenario=req.scenario,
+            accuracy_tier=req.accuracy_tier,
+            framework=req.framework,
+            budget_per_gpu_hr=req.budget_per_gpu_hr,
+            min_throughput_tok_per_sec=req.min_throughput_tok_per_sec,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc

src/api/schemas.py

@@ -0,0 +1,88 @@
+"""Pydantic request/response schemas for the GPU Perf Prophet API."""
+
+from __future__ import annotations
+
+from typing import Literal, Optional
+
+from pydantic import BaseModel, Field, field_validator
+
+
+Scenario      = Literal["Offline", "Server"]
+AccuracyTier  = Literal["base", "99", "99.9"]
+Framework     = Literal["vllm", "tensorrt", "rocm_other", "other"]
+
+
+class PredictRequest(BaseModel):
+    gpu_id:        str          = Field(..., max_length=100, examples=["mi300x"])
+    model_name:    str          = Field(..., max_length=100, examples=["llama2-70b"])
+    scenario:      Scenario     = "Offline"
+    accuracy_tier: AccuracyTier = "99"
+    framework:     Framework    = "vllm"
+
+
+class PredictResponse(BaseModel):
+    gpu_id:                        str
+    model_name:                    str
+    scenario:                      str
+    accuracy_tier:                 str
+    framework:                     str
+    pred_throughput_tok_per_sec:   float
+    roofline_tput_tok_per_sec:     float
+    efficiency_ratio:              float
+    vram_fits:                     bool
+    model_size_gb:                 float
+
+
+class RecommendRequest(BaseModel):
+    model_name:                str          = Field(..., max_length=100, examples=["llama2-70b"])
+    scenario:                  Scenario     = "Offline"
+    accuracy_tier:             AccuracyTier = "99"
+    framework:                 Framework    = "vllm"
+    budget_per_gpu_hr:         Optional[float] = Field(None, gt=0, examples=[4.0])
+    min_throughput_tok_per_sec: Optional[float] = Field(None, gt=0)
+
+    @field_validator("budget_per_gpu_hr", "min_throughput_tok_per_sec", mode="before")
+    @classmethod
+    def _coerce_none(cls, v: object) -> object:
+        return None if v == 0 else v
+
+
+class GpuResult(BaseModel):
+    gpu_id:                      str
+    gpu_name:                    str
+    vendor:                      str
+    model_name:                  str
+    scenario:                    str
+    accuracy_tier:                str
+    framework:                   str
+    pred_throughput_tok_per_sec: float
+    roofline_tput_tok_per_sec:   float
+    efficiency_ratio:            float
+    vram_fits:                   bool
+    model_size_gb:               float
+    vram_gb:                     Optional[float]
+    price_per_gpu_hr:            Optional[float]
+    vram_headroom:               float
+    cost_efficiency:             Optional[float]
+    throughput:                  float
+
+
+class FilteredGpuResult(GpuResult):
+    reject_reason: str
+
+
+class WorkloadSummary(BaseModel):
+    model_name:                str
+    scenario:                  str
+    accuracy_tier:             str
+    framework:                 str
+    model_size_gb:             float
+    budget_per_gpu_hr:         Optional[float]
+    min_throughput_tok_per_sec: Optional[float]
+
+
+class RecommendResponse(BaseModel):
+    frontier:  list[GpuResult]
+    dominated: list[GpuResult]
+    filtered:  list[FilteredGpuResult]
+    workload:  WorkloadSummary

src/data/gpu_spec_db.py

@@ -0,0 +1,294 @@
+"""
+GPU specification database loader for GPU Perf Prophet.
+
+Loads data/gpu_specs.yaml and provides two public entry-points:
+
+    normalize_gpu_name(raw_name, specs) → Optional[str]
+        Maps a raw MLPerf accelerator_model_name to a canonical GPU id.
+        Returns None for heterogeneous multi-GPU strings, unknown GPUs, or
+        explicitly unrecognised names.
+
+    enrich_df(df, spec_path) → pd.DataFrame
+        Adds GPU hardware spec columns to a parsed MLPerf DataFrame.
+        Rows whose gpu_name cannot be normalized get NaN in spec columns.
+"""
+
+from __future__ import annotations
+
+import functools
+import logging
+import re
+import stat as _stat
+import types
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+import yaml
+
+# Real GPU spec files are ~10 KB.  1 MB cap blocks both accidental and
+# adversarial oversized files; mirrors _safe_read_text in mlperf_parser.py.
+_MAX_SPEC_BYTES: int = 1 * 1024 * 1024  # 1 MB
+
+log = logging.getLogger(__name__)
+
+_DEFAULT_SPEC_PATH = Path(__file__).parent.parent.parent / "data" / "gpu_specs.yaml"
+
+# Columns added by enrich_df, in output order.
+SPEC_COLUMNS = [
+    "canonical_gpu_id",
+    "gpu_vendor",
+    "gpu_architecture",
+    "gpu_memory_type",
+    "gpu_vram_gb",
+    "gpu_hbm_bandwidth_tbps",
+    "gpu_peak_fp32_tflops",
+    "gpu_peak_bf16_tflops",
+    "gpu_peak_fp16_tflops",
+    "gpu_peak_fp8_tflops",
+    "gpu_peak_fp6_tflops",
+    "gpu_peak_fp4_tflops",
+    "gpu_peak_int8_tops",
+    "gpu_cu_sm_count",
+    "gpu_l2_cache_mb",
+    "gpu_tdp_w",
+    "gpu_in_model_scope",
+    "gpu_spec_confidence",
+]
+
+# Matches "(x8)", "(x16)", "(x32)", "(x87)" etc.
+_COUNT_SUFFIX_RE = re.compile(r"\s*\(x\d+\)\s*$")
+# Matches "(Power Cap 1000 W)" and similar parenthetical power notes
+_POWER_SUFFIX_RE = re.compile(r"\s*\(Power Cap[^)]*\)\s*$", re.IGNORECASE)
+
+
+@functools.lru_cache(maxsize=4)
+def _build_spec_cache(spec_path_str: str) -> tuple[list, dict, dict]:
+    """
+    Load the spec file and build the two derived lookup structures.
+    Cached by path string so repeated enrich_df calls pay the I/O and
+    parse cost only once per unique path.
+
+    Returns (specs, alias_index, id_map).
+
+    The cache key is the literal path string (not resolved), so the
+    symlink guard in load_specs still fires on first access.
+    Cached objects must not be mutated by callers.
+    """
+    specs  = load_specs(Path(spec_path_str))
+    index  = _build_alias_index(specs)
+    # Wrap id_map values in MappingProxyType so callers cannot mutate the
+    # cached spec rows.  The proxy supports all read operations (get, iter,
+    # subscript) that downstream code uses.
+    id_map = {s["id"]: types.MappingProxyType(_spec_row(s)) for s in specs}
+    return specs, index, id_map
+
+
+@functools.lru_cache(maxsize=8)
+def load_specs(spec_path: Path | str = _DEFAULT_SPEC_PATH) -> list[dict]:
+    """
+    Load and return the list of GPU spec dicts from the YAML file.
+
+    Raises ValueError for symlinks (path traversal guard) or oversized files.
+    Raises FileNotFoundError / OSError for I/O problems.
+    Raises ValueError for malformed YAML that is missing the 'gpus' key.
+    """
+    path = Path(spec_path)
+    try:
+        st = path.lstat()
+    except OSError as exc:
+        raise FileNotFoundError(f"GPU spec DB not found: {path}") from exc
+    if _stat.S_ISLNK(st.st_mode):
+        raise ValueError(f"GPU spec DB path is a symlink (refused): {path}")
+    if st.st_size > _MAX_SPEC_BYTES:
+        raise ValueError(
+            f"GPU spec DB too large ({st.st_size} bytes > {_MAX_SPEC_BYTES}): {path}"
+        )
+    with path.open(encoding="utf-8") as f:
+        data = yaml.safe_load(f)
+    if not isinstance(data, dict) or "gpus" not in data:
+        raise ValueError(
+            f"GPU spec DB at {path} is missing required 'gpus' key "
+            f"(got {type(data).__name__})"
+        )
+    return data["gpus"]
+
+
+def _build_alias_index(specs: list[dict]) -> dict[str, dict]:
+    """Return lowercase-alias → spec dict for fast lookup."""
+    index: dict[str, dict] = {}
+    for spec in specs:
+        for alias in spec.get("aliases", []):
+            key = alias.lower()
+            if key in index and index[key]["id"] != spec["id"]:
+                # Warn only when the same alias maps to two *different* GPU ids.
+                # Duplicate aliases within the same GPU (e.g. "HBM3E" vs "HBM3e")
+                # are harmless after case-folding and should not produce noise.
+                log.warning(
+                    "Conflicting alias %r maps to both %r and %r in GPU spec DB",
+                    alias, index[key]["id"], spec["id"],
+                )
+            index[key] = spec
+    return index
+
+
+def _is_heterogeneous(raw_name: str) -> bool:
+    """Return True for multi-GPU strings mixing different SKUs."""
+    # "A and B" pattern (heterogeneous mix)
+    if " and " in raw_name.lower():
+        return True
+    # "A (xN), B (xM)" pattern
+    if "," in raw_name and re.search(r"\(x\d+\)", raw_name):
+        return True
+    return False
+
+
+def _strip_suffixes(raw_name: str) -> str:
+    """Remove parenthetical count and power suffixes from a GPU name."""
+    name = _COUNT_SUFFIX_RE.sub("", raw_name)
+    name = _POWER_SUFFIX_RE.sub("", name)
+    return name.strip()
+
+
+def normalize_gpu_name(
+    raw_name: str | None,
+    specs: list[dict],
+    *,
+    _index: dict[str, dict] | None = None,
+) -> Optional[str]:
+    """
+    Map a raw MLPerf accelerator_model_name to a canonical GPU id.
+
+    Returns None for:
+      - null / "N/A" names
+      - heterogeneous multi-GPU strings (e.g. "MI300X ... and MI325X ...")
+      - names that don't match any alias in the spec DB (logged at DEBUG)
+    """
+    if not raw_name or raw_name.strip().upper() in {"N/A", "NA", "NONE", ""}:
+        return None
+
+    if _is_heterogeneous(raw_name):
+        log.debug("Skipping heterogeneous GPU string: %r", raw_name)
+        return None
+
+    index = _index if _index is not None else _build_alias_index(specs)
+    clean = _strip_suffixes(raw_name)
+
+    match = index.get(clean.lower())
+    if match is None:
+        log.debug("No GPU spec match for: %r (stripped: %r)", raw_name, clean)
+        return None
+
+    return match["id"]
+
+
+def _spec_row(spec: dict) -> dict:
+    """Flatten one GPU spec dict into the SPEC_COLUMNS schema."""
+    pt = spec.get("peak_tflops") or {}
+    return {
+        "canonical_gpu_id":        spec["id"],
+        "gpu_vendor":              spec.get("vendor"),
+        "gpu_architecture":        spec.get("architecture"),
+        "gpu_memory_type":         spec.get("memory_type"),
+        "gpu_vram_gb":             spec.get("vram_gb"),
+        "gpu_hbm_bandwidth_tbps":  spec.get("hbm_bandwidth_tbps"),
+        "gpu_peak_fp32_tflops":    pt.get("fp32"),
+        "gpu_peak_bf16_tflops":    pt.get("bf16"),
+        "gpu_peak_fp16_tflops":    pt.get("fp16"),
+        "gpu_peak_fp8_tflops":     pt.get("fp8"),
+        "gpu_peak_fp6_tflops":     pt.get("fp6"),
+        "gpu_peak_fp4_tflops":     pt.get("fp4"),
+        "gpu_peak_int8_tops": pt.get("int8"),
+        # AMD uses compute_units; NVIDIA uses streaming_multiprocessors.
+        # Stored under a unified key so feature engineering is vendor-agnostic.
+        "gpu_cu_sm_count": (
+            spec.get("compute_units") or spec.get("streaming_multiprocessors")
+        ),
+        "gpu_l2_cache_mb": spec.get("l2_cache_mb"),
+        "gpu_tdp_w": spec.get("tdp_w"),
+        "gpu_in_model_scope": spec.get("in_model_scope"),
+        "gpu_spec_confidence": spec.get("spec_confidence"),
+    }
+
+
+def enrich_df(
+    df: pd.DataFrame,
+    spec_path: Path | str = _DEFAULT_SPEC_PATH,
+) -> pd.DataFrame:
+    """
+    Add GPU hardware spec columns to a parsed MLPerf DataFrame.
+
+    Joins on the canonical_gpu_id derived from each row's gpu_name.
+    Rows whose gpu_name cannot be normalized receive NaN in all spec columns.
+    The original DataFrame is not modified; a new DataFrame is returned.
+    """
+    # Fix 1: derive lookup structures once per unique path (cached).
+    specs, index, id_map = _build_spec_cache(str(Path(spec_path)))
+
+    # Fix 2: normalize once per unique gpu_name, then broadcast to all rows.
+    # The corpus has ~33 unique names across ~1223 rows; calling
+    # normalize_gpu_name per-row wastes ~1190 redundant lookups.
+    unique_names = {n for n in df["gpu_name"].dropna().unique()}
+    name_to_id = {
+        name: normalize_gpu_name(name, specs, _index=index)
+        for name in unique_names
+    }
+    canonical_ids = df["gpu_name"].map(name_to_id)
+
+    spec_df = pd.DataFrame(
+        [id_map.get(cid, {}) for cid in canonical_ids],
+        index=df.index,
+    )
+
+    # Ensure all expected columns exist even if spec_rows were all empty.
+    for col in SPEC_COLUMNS:
+        if col not in spec_df.columns:
+            spec_df[col] = None
+
+    matched = canonical_ids.notna().sum()
+    unmatched = canonical_ids.isna().sum()
+    log.info(
+        "GPU spec enrichment: %d rows matched, %d unmatched (%.0f%%)",
+        matched, unmatched, 100 * matched / max(len(df), 1),
+    )
+    if unmatched:
+        unseen = set(df.loc[canonical_ids.isna(), "gpu_name"].dropna().unique())
+        # Use %r so control characters (newlines, ANSI escapes) in GPU names
+        # are escaped rather than injected into the log stream.
+        for name in sorted(unseen):
+            log.debug("Unmatched gpu_name: %r", name)
+
+    result = pd.concat([df, spec_df[SPEC_COLUMNS]], axis=1)
+
+    # Warn when the parser's reported vram_gb (from system JSON
+    # accelerator_memory_capacity) differs from the spec DB's gpu_vram_gb.
+    # This happens when submitters report total system VRAM rather than
+    # per-GPU capacity.  Use gpu_vram_gb for the memory-fit constraint;
+    # treat vram_gb as unreliable for multi-GPU submissions.
+    if "vram_gb" in result.columns and "gpu_vram_gb" in result.columns:
+        conflict = (
+            result["vram_gb"].notna()
+            & result["gpu_vram_gb"].notna()
+            & (result["vram_gb"] != result["gpu_vram_gb"])
+        )
+        if conflict.any():
+            n = conflict.sum()
+            examples = (
+                result.loc[conflict, ["gpu_name", "num_gpus", "vram_gb", "gpu_vram_gb"]]
+                .drop_duplicates()
+            )
+            log.warning(
+                "%d rows: vram_gb (parser) != gpu_vram_gb (spec DB). "
+                "Some submitters report total system VRAM. "
+                "Use gpu_vram_gb for the memory-fit constraint.",
+                n,
+            )
+            # Fix 3: itertuples ~10x faster than iterrows; sufficient for
+            # attribute access.  %r quotes gpu_name to prevent log injection.
+            for row in examples.itertuples(index=False):
+                log.warning(
+                    "  vram conflict: gpu=%r  num_gpus=%s  reported=%s  spec=%s",
+                    row.gpu_name, row.num_gpus, row.vram_gb, row.gpu_vram_gb,
+                )
+
+    return result

src/data/mlperf_parser.py

@@ -0,0 +1,602 @@
+"""
+MLPerf Inference results parser for GPU Perf Prophet.
+
+Walks one or more local MLPerf Inference results repos (v4.1–v6.0) and
+produces a flat CSV/Parquet of (system, workload, scenario) performance rows.
+GPU hardware specs and LLM model specs are joined separately (see gpu_spec_db.py).
+
+Directory contract — consistent across v4.1–v6.0, closed and open divisions:
+
+  <repo_root>/
+    {closed,open}/<submitter>/
+      results/<system_name>/<benchmark>/<scenario>/
+        performance/run_<N>/mlperf_log_summary.txt
+      systems/<system_name>.json
+
+Usage (CLI):
+    python -m src.data.mlperf_parser \\
+        --repos-dir data/raw/mlperf \\
+        --rounds v4.1 v5.0 v5.1 v6.0 \\
+        --output data/processed/mlperf_raw.csv --parquet
+
+    Expects repos at data/raw/mlperf/v4.1/, data/raw/mlperf/v5.0/, etc.
+    See scripts/fetch_mlperf.sh for sparse-checkout instructions.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import re
+import stat as _stat
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+
+log = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Reference tables
+# ---------------------------------------------------------------------------
+
+# All LLM inference benchmark names used across MLPerf v4.1–v6.0 (and anticipated v7.0+).
+# Accuracy-constrained variants (e.g. -99, -99.9) share the same dataset.
+LLM_BENCHMARKS: frozenset[str] = frozenset({
+    "gptj", "gptj-99", "gptj-99.9",
+    "llama2-70b", "llama2-70b-99", "llama2-70b-99.9",
+    "mixtral-8x7b", "mixtral-8x7b-99", "mixtral-8x7b-99.9",
+    "llama3.1-405b", "llama3.1-405b-99", "llama3.1-405b-99.9",
+    "llama3.1-8b",   "llama3.1-8b-99",   "llama3.1-8b-99.9",
+})
+
+# Approximate mean output tokens per sample — used to convert samples/sec → tok/sec.
+# Sources: MLPerf Inference LLM benchmark dataset documentation.
+#   gptj:         fixed 128-token output per benchmark spec.
+#   llama2-70b:   ~294 mean output tokens from Open ORCA v1 (MLPerf v4.1+).
+#   mixtral-8x7b: ~145 mean output tokens (Open ORCA; verify per round if needed).
+#   llama3.1-405b: same Open ORCA dataset as llama2-70b per MLPerf rules.
+#   llama3.1-8b:   same Open ORCA dataset as llama2-70b per MLPerf rules.
+# IMPORTANT: these are estimates; flag them in the data card and re-verify if the
+# dataset changes between rounds.
+TOKENS_PER_SAMPLE: dict[str, int] = {
+    "gptj":               128,
+    "gptj-99":            128,
+    "gptj-99.9":          128,
+    "llama2-70b":         294,
+    "llama2-70b-99":      294,
+    "llama2-70b-99.9":    294,
+    "mixtral-8x7b":       145,
+    "mixtral-8x7b-99":    145,
+    "mixtral-8x7b-99.9":  145,
+    "llama3.1-405b":      294,
+    "llama3.1-405b-99":   294,
+    "llama3.1-405b-99.9": 294,
+    "llama3.1-8b":        294,
+    "llama3.1-8b-99":     294,
+    "llama3.1-8b-99.9":   294,
+}
+
+
+def _base_benchmark(benchmark: str) -> str:
+    """Strip accuracy-constraint suffix: 'llama2-70b-99' → 'llama2-70b'."""
+    for suffix in ("-99.9", "-99"):
+        if benchmark.endswith(suffix):
+            return benchmark[: -len(suffix)]
+    return benchmark
+
+
+def _accuracy_tier(benchmark: str) -> str:
+    """
+    Extract the MLPerf accuracy-constraint tier from a benchmark name.
+
+    Returns one of "99.9" | "99" | "base".
+
+    Used as a proxy for precision when direct precision extraction fails
+    (which is almost always — real MLPerf system names are hardware
+    identifiers, not precision-tagged strings).  Correlation:
+      "99.9" → FP16 (high-fidelity submissions; AMD may use FP8 achieving this tier)
+      "99"   → typically FP8 or better
+      "base" → throughput-optimized; any precision acceptable to submitter
+    """
+    if benchmark.endswith("-99.9"):
+        return "99.9"
+    if benchmark.endswith("-99"):
+        return "99"
+    return "base"
+
+
+# ---------------------------------------------------------------------------
+# Safe file reading
+# ---------------------------------------------------------------------------
+
+# Real MLPerf files are < 50 KB. Cap at 512 KB (10× safety margin) to
+# refuse corrupted or adversarial files without loading them into memory.
+_MAX_FILE_BYTES: int = 512 * 1024  # 512 KB
+
+
+def _safe_read_text(path: Path) -> Optional[str]:
+    """
+    Read a text file with three safety checks:
+
+    1. Refuse symlinks — git repos can contain symlinks pointing anywhere on disk.
+    2. Enforce a size cap — prevents memory exhaustion from oversized/crafted files.
+    3. Catch only I/O errors — lets MemoryError and other fatal signals propagate.
+
+    Returns None (logged at WARNING) instead of raising on any guarded condition.
+    """
+    # Single lstat() call supplies both the symlink bit and the file size,
+    # avoiding a second stat() syscall compared to is_symlink() + stat().
+    try:
+        st = path.lstat()
+    except OSError as exc:
+        log.warning("Could not stat %s: %s", path, exc)
+        return None
+    if _stat.S_ISLNK(st.st_mode):
+        log.warning("Refusing to read symlink: %s", path)
+        return None
+    if st.st_size > _MAX_FILE_BYTES:
+        log.warning("Skipping oversized file (%d bytes > %d limit): %s",
+                    st.st_size, _MAX_FILE_BYTES, path)
+        return None
+    try:
+        return path.read_text(encoding="utf-8", errors="replace")
+    except (OSError, UnicodeDecodeError) as exc:
+        log.warning("Could not read %s: %s", path, exc)
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Field-parsing helpers
+# ---------------------------------------------------------------------------
+
+def _parse_vram_gb(s: object) -> Optional[float]:
+    """'192 GB' → 192.0, '80 GiB' → 80.0, None → None."""
+    if not s:
+        return None
+    # Require match to start with a digit so a bare '.' can't slip through.
+    m = re.search(r"(\d[\d.]*)\s*(?:GB|GiB)", str(s), re.IGNORECASE)
+    if m is None:
+        return None
+    try:
+        return float(m.group(1))
+    except ValueError:
+        log.warning("Could not parse VRAM value: %r", s)
+        return None
+
+
+def _parse_int(v: object) -> Optional[int]:
+    try:
+        return int(v)
+    except (TypeError, ValueError):
+        return None
+
+
+def _ns_to_ms(ns: Optional[int]) -> Optional[float]:
+    return ns / 1_000_000 if ns is not None else None
+
+
+# ---------------------------------------------------------------------------
+# system_desc.json parser
+# ---------------------------------------------------------------------------
+
+def _parse_system_json(path: Path) -> dict:
+    """
+    Parse one system description JSON file into a flat dict.
+
+    Field names are consistent across v4.1–v6.0.  Missing fields return None
+    rather than raising, so schema drift between rounds is handled gracefully.
+    """
+    text = _safe_read_text(path)
+    if text is None:
+        return {}
+    try:
+        raw = json.loads(text)
+    except json.JSONDecodeError as exc:
+        log.warning("Malformed JSON in %s: %s", path, exc)
+        return {}
+
+    # json.loads can return any JSON type (list, string, int, …); the caller
+    # expects a dict and calls raw.get(...), which raises AttributeError on
+    # non-dict types.  Guard here so a corrupted or adversarial system JSON
+    # doesn't abort parsing of the entire round directory.
+    if not isinstance(raw, dict):
+        log.warning(
+            "Expected JSON object in %s, got %s — skipping",
+            path, type(raw).__name__,
+        )
+        return {}
+
+    # num_gpus = accelerators_per_node × number_of_nodes.
+    # Multi-node MLPerf submissions report throughput for the entire cluster but
+    # only set accelerators_per_node (= per-node count).  Multiplying by
+    # number_of_nodes gives the total GPU count that throughput was measured
+    # against, which is the correct divisor for per-GPU normalisation.
+    accel_per_node = _parse_int(raw.get("accelerators_per_node")) or 1
+    num_nodes      = max(_parse_int(raw.get("number_of_nodes")) or 1, 1)
+
+    return {
+        "gpu_name":    raw.get("accelerator_model_name") or raw.get("accelerators"),
+        "num_gpus":    accel_per_node * num_nodes,
+        "vram_gb":     _parse_vram_gb(raw.get("accelerator_memory_capacity")),
+        "framework":   raw.get("framework"),
+        "system_type": raw.get("system_type"),   # "datacenter" or "edge"
+        "hw_status":   raw.get("status"),         # "available" / "preview" / "rdi"
+    }
+
+
+# ---------------------------------------------------------------------------
+# mlperf_log_summary.txt parser
+# ---------------------------------------------------------------------------
+
+# Regex patterns — cover formatting variations observed across v4.1–v6.0.
+_RE_SCENARIO     = re.compile(r"Scenario\s*:\s*(\S+)",                              re.I)
+_RE_VALID        = re.compile(r"Result is\s*:\s*(\w+)",                              re.I)
+# Anchored to line-start so it doesn't match "Completed samples per second" in Server logs.
+_RE_OFFLINE_TPUT = re.compile(r"^Samples per second\s*:\s*([\d.]+)",                re.I | re.MULTILINE)
+_RE_SERVER_TPUT  = re.compile(r"Scheduled samples per second\s*:\s*([\d.]+)",       re.I)
+_RE_LAT_MEAN     = re.compile(r"Mean latency\s*\(ns\)\s*:\s*(\d+)",              re.I)
+_RE_LAT_P99      = re.compile(r"99\.00 percentile latency\s*\(ns\)\s*:\s*(\d+)", re.I)
+# LLM-specific metrics (present in v4.1+ for LLM benchmarks)
+_RE_TTFT_MEAN    = re.compile(r"Mean First Token Latency\s*\(ns\)\s*:\s*(\d+)",                      re.I)
+_RE_TTFT_P99     = re.compile(r"99(?:th|\.00) Percentile First Token Latency\s*\(ns\)\s*:\s*(\d+)", re.I)
+_RE_TPOT_MEAN    = re.compile(r"Mean Time Per Output Token\s*\(ns\)\s*:\s*(\d+)",                   re.I)
+_RE_TPOT_P99     = re.compile(r"99(?:th|\.00) Percentile Time Per Output Token\s*\(ns\)\s*:\s*(\d+)", re.I)
+
+
+def _parse_log_summary(path: Path) -> dict:
+    """
+    Parse one mlperf_log_summary.txt into a flat dict of performance metrics.
+    All latencies are converted from nanoseconds → milliseconds.
+    Returns an empty dict on read/parse failure (logged at WARNING).
+    """
+    text = _safe_read_text(path)
+    if text is None:
+        return {}
+
+    def _f(pat: re.Pattern) -> Optional[float]:
+        m = pat.search(text)
+        return float(m.group(1)) if m else None
+
+    def _i(pat: re.Pattern) -> Optional[int]:
+        m = pat.search(text)
+        return int(m.group(1)) if m else None
+
+    scenario_m = _RE_SCENARIO.search(text)
+    valid_m    = _RE_VALID.search(text)
+
+    tput = _f(_RE_OFFLINE_TPUT) or _f(_RE_SERVER_TPUT)
+
+    return {
+        "scenario_from_log": scenario_m.group(1).capitalize() if scenario_m else None,
+        "result_valid":       (valid_m.group(1).upper() == "VALID") if valid_m else None,
+        "throughput_samples_per_sec": tput,
+        "latency_mean_ms": _ns_to_ms(_i(_RE_LAT_MEAN)),
+        "latency_p99_ms":  _ns_to_ms(_i(_RE_LAT_P99)),
+        "ttft_mean_ms":    _ns_to_ms(_i(_RE_TTFT_MEAN)),
+        "ttft_p99_ms":     _ns_to_ms(_i(_RE_TTFT_P99)),
+        "tpot_mean_ms":    _ns_to_ms(_i(_RE_TPOT_MEAN)),
+        "tpot_p99_ms":     _ns_to_ms(_i(_RE_TPOT_P99)),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Precision extraction
+# ---------------------------------------------------------------------------
+
+# Ordered from most-specific to least-specific; first match wins.
+_PRECISION_PATTERNS: list[tuple[str, re.Pattern]] = [
+    ("fp4",   re.compile(r"\bfp4\b|\bmxfp4\b",  re.I)),
+    ("fp6",   re.compile(r"\bfp6\b|\bmxfp6\b",  re.I)),
+    ("fp8",   re.compile(r"\bfp8\b|\be4m3\b|\be5m2\b", re.I)),
+    ("int8",  re.compile(r"\bint8\b|\bw8a8\b",   re.I)),
+    ("bf16",  re.compile(r"\bbf16\b|\bbfloat16\b", re.I)),
+    ("fp16",  re.compile(r"\bfp16\b|\bfloat16\b", re.I)),
+]
+
+
+def _extract_precision(system_name: str, framework: Optional[str]) -> Optional[str]:
+    """
+    Best-effort precision extraction from system_name and framework fields.
+
+    MLPerf doesn't have a dedicated precision field; submitters encode it in
+    the system name (e.g., 'AMD_MI300X_FP8_vLLM') and/or the framework string
+    (e.g., 'ROCm 6.2.4, vLLM 0.5.0-fp8').  Returns None when not found —
+    callers should treat None as 'unknown' rather than a specific precision.
+    """
+    # Replace underscores with spaces so \b word-boundaries work on tokens like
+    # "AMD_MI300X_FP8_vLLM" → "AMD MI300X FP8 vLLM", making \bFP8\b match.
+    search_text = f"{system_name} {framework or ''}".replace("_", " ")
+    for label, pat in _PRECISION_PATTERNS:
+        if pat.search(search_text):
+            return label
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Run-directory resolution
+# ---------------------------------------------------------------------------
+
+def _run_number(log_file: Path) -> int:
+    """Extract the integer N from a 'run_N' directory name, -1 if not found."""
+    m = re.search(r"run_(\d+)", log_file.parent.name)
+    return int(m.group(1)) if m else -1
+
+
+def _find_best_run(scenario_dir: Path) -> Optional[Path]:
+    """
+    Return the mlperf_log_summary.txt from the highest-numbered run_N directory
+    under <scenario_dir>/performance/.  Returns None if none exists.
+    Symlinked run directories are skipped (see _safe_read_text for rationale).
+    """
+    perf_dir = scenario_dir / "performance"
+    if not perf_dir.is_dir() or perf_dir.is_symlink():
+        return None
+
+    # Single pass: filter symlinks at both directory and file level, extract
+    # run number, and discard non-numeric names — all in one list comprehension.
+    runs = [
+        (p, n)
+        for p in perf_dir.glob("run_*/mlperf_log_summary.txt")
+        if not p.parent.is_symlink()
+        and not p.is_symlink()
+        and (n := _run_number(p)) >= 0
+    ]
+    if not runs:
+        return None
+
+    return max(runs, key=lambda x: x[1])[0]
+
+
+# ---------------------------------------------------------------------------
+# Repo walker
+# ---------------------------------------------------------------------------
+
+def parse_repo(
+    repo_root: Path,
+    round_tag: str,
+    divisions: tuple[str, ...] = ("closed", "open"),
+    llm_only: bool = True,
+) -> pd.DataFrame:
+    """
+    Walk one MLPerf Inference results repo and return a DataFrame.
+
+    Parameters
+    ----------
+    repo_root : Local clone of an inference_results_vX.Y repo.
+    round_tag : Version label written into every output row, e.g. "v6.0".
+    divisions : Submission divisions to include ("closed", "open", or both).
+    llm_only  : When True (default), skip non-LLM benchmarks (vision, audio, etc.).
+    """
+    repo_root = Path(repo_root)
+    rows: list[dict] = []
+
+    for division in divisions:
+        div_dir = repo_root / division
+        if not div_dir.is_dir():
+            continue
+
+        for submitter_dir in sorted(div_dir.iterdir()):
+            if not submitter_dir.is_dir() or submitter_dir.is_symlink():
+                continue
+            submitter = submitter_dir.name
+
+            systems_dir = submitter_dir / "systems"
+            results_dir = submitter_dir / "results"
+            if not results_dir.is_dir() or results_dir.is_symlink():
+                continue
+
+            for system_dir in sorted(results_dir.iterdir()):
+                if not system_dir.is_dir() or system_dir.is_symlink():
+                    continue
+                system_name = system_dir.name
+
+                sys_json_path = systems_dir / f"{system_name}.json"
+                # Single lstat() gives existence + symlink bit — avoids the
+                # exists()+is_symlink()+is_symlink() three-call pattern.
+                try:
+                    _st = sys_json_path.lstat()
+                except OSError:
+                    hw = {}
+                    log.debug("No system JSON for %r/%r", submitter, system_name)
+                else:
+                    if _stat.S_ISLNK(_st.st_mode):
+                        hw = {}
+                        log.warning("Skipping symlinked system JSON: %s", sys_json_path)
+                    else:
+                        hw = _parse_system_json(sys_json_path)
+
+                # Precision is a system-level attribute — compute it once here
+                # rather than once per (benchmark, scenario) row.
+                precision = _extract_precision(system_name, hw.get("framework"))
+                # Clamp to ≥ 1: accelerators_per_node = "0" appears for CPU-only
+                # inference systems (e.g. Intel EMR PyTorch); using 0 as the
+                # divisor would produce ±inf throughput_tok_per_sec_per_gpu.
+                # The stored num_gpus must match the divisor used for computation.
+                num_gpus = max(hw.get("num_gpus") or 1, 1)
+
+                for benchmark_dir in sorted(system_dir.iterdir()):
+                    if not benchmark_dir.is_dir() or benchmark_dir.is_symlink():
+                        continue
+
+                    # Lower-case once; reused for LLM_BENCHMARKS check,
+                    # TOKENS_PER_SAMPLE lookup, and storage.
+                    bench_lower = benchmark_dir.name.lower()
+
+                    if llm_only and bench_lower not in LLM_BENCHMARKS:
+                        continue
+
+                    for scenario_dir in sorted(benchmark_dir.iterdir()):
+                        if not scenario_dir.is_dir() or scenario_dir.is_symlink():
+                            continue
+                        scenario = scenario_dir.name  # "Offline", "Server", "SingleStream", …
+
+                        log_path = _find_best_run(scenario_dir)
+                        if log_path is None:
+                            log.debug("No performance run found: %s", scenario_dir)
+                            continue
+
+                        metrics = _parse_log_summary(log_path)
+
+                        # Cross-check: scenario from log should match directory name.
+                        log_scenario = metrics.get("scenario_from_log")
+                        if log_scenario and log_scenario.lower() != scenario.lower():
+                            log.debug(
+                                "Scenario mismatch: dir=%r log=%r (%s)",
+                                scenario, log_scenario, log_path,
+                            )
+
+                        tput    = metrics.get("throughput_samples_per_sec")
+                        tps     = TOKENS_PER_SAMPLE.get(bench_lower)
+                        tok_sec = tput * tps if (tput and tps) else None
+                        # Per-GPU throughput is the model target variable.
+                        # Roofline bounds are computed per-GPU, so we normalize here.
+                        tok_sec_per_gpu = tok_sec / num_gpus if tok_sec else None
+
+                        rows.append({
+                            # Provenance
+                            "round":       round_tag,
+                            "division":    division,
+                            "submitter":   submitter,
+                            "system_name": system_name,
+                            # Hardware (from system JSON)
+                            **hw,
+                            # num_gpus overrides **hw: the clamped value (≥ 1)
+                            # must match the divisor used for tok_sec_per_gpu.
+                            "num_gpus": num_gpus,
+                            # Workload
+                            "benchmark":               bench_lower,
+                            "benchmark_base":          _base_benchmark(bench_lower),
+                            "benchmark_accuracy_tier": _accuracy_tier(bench_lower),
+                            "scenario":                scenario,
+                            # precision: best-effort from system_name/framework regex;
+                            # typically None on real data (system names are hardware
+                            # identifiers, not precision-tagged).  Use
+                            # benchmark_accuracy_tier as the reliable proxy feature.
+                            "precision": precision,
+                            # Token-rate (per-node and per-GPU)
+                            "tokens_per_sample":           tps,
+                            "throughput_tokens_per_sec":   tok_sec,
+                            "throughput_tok_per_sec_per_gpu": tok_sec_per_gpu,
+                            # Performance metrics
+                            **{k: v for k, v in metrics.items() if k != "scenario_from_log"},
+                            # Audit trail — relative path preferred; fall back to
+                            # absolute if log_path resolves outside repo_root
+                            # (e.g. via an unexpected symlink).
+                            "log_path": (
+                                str(log_path.relative_to(repo_root))
+                                if log_path.is_relative_to(repo_root)
+                                else str(log_path)
+                            ),
+                        })
+
+    if not rows:
+        log.warning("No rows parsed from %s — check repo layout and round tag.", repo_root)
+        return pd.DataFrame()
+
+    df = pd.DataFrame(rows)
+    log.info("Parsed %d rows from %s (%s)", len(df), repo_root.name, round_tag)
+    return df
+
+
+def parse_repos(
+    repo_specs: list[tuple[Path | str, str]],
+    llm_only: bool = True,
+) -> pd.DataFrame:
+    """
+    Parse multiple repos and return a single concatenated DataFrame.
+
+    Parameters
+    ----------
+    repo_specs : [(repo_path, round_tag), ...]
+                 e.g. [("data/raw/mlperf/v6.0", "v6.0"), ...]
+                 Paths that don't exist are silently skipped.
+    llm_only   : Passed through to parse_repo.
+    """
+    frames = [
+        parse_repo(Path(p), tag, llm_only=llm_only)
+        for p, tag in repo_specs
+        if Path(p).is_dir()
+    ]
+    frames = [f for f in frames if not f.empty]
+    if not frames:
+        return pd.DataFrame()
+    return pd.concat(frames, ignore_index=True)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def _build_arg_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        description="Parse MLPerf Inference results repos into a flat CSV/Parquet.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    p.add_argument(
+        "--repos-dir",
+        default="data/raw/mlperf",
+        help="Parent directory containing version-named subdirs (v4.1/, v5.0/, …).",
+    )
+    p.add_argument(
+        "--rounds", nargs="+",
+        default=["v4.1", "v5.0", "v5.1", "v6.0"],
+        help="Round tags to look for under --repos-dir.",
+    )
+    p.add_argument(
+        "--output",
+        default="data/processed/mlperf_raw.csv",
+        help="Output CSV path.",
+    )
+    p.add_argument("--parquet", action="store_true",
+                   help="Also write a .parquet file alongside the CSV.")
+    p.add_argument("--all-benchmarks", action="store_true",
+                   help="Include non-LLM benchmarks (ResNet, BERT, DLRM, …).")
+    p.add_argument("--log-level", default="INFO",
+                   choices=["DEBUG", "INFO", "WARNING", "ERROR"])
+    return p
+
+
+def main() -> None:
+    args = _build_arg_parser().parse_args()
+    logging.basicConfig(level=args.log_level, format="%(levelname)s: %(message)s")
+
+    repos_dir  = Path(args.repos_dir)
+    repo_specs = [(repos_dir / tag, tag) for tag in args.rounds]
+    found      = [(p, t) for p, t in repo_specs if p.is_dir()]
+
+    if not found:
+        log.error(
+            "No round directories found in %s. "
+            "Run scripts/fetch_mlperf.sh first, or pass --repos-dir.",
+            repos_dir,
+        )
+        raise SystemExit(1)
+
+    df = parse_repos(found, llm_only=not args.all_benchmarks)
+    if df.empty:
+        log.error("No rows collected — check repo layout and round tags.")
+        raise SystemExit(1)
+
+    out = Path(args.output)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    df.to_csv(out, index=False)
+    log.info("Wrote %d rows → %s", len(df), out)
+
+    if args.parquet:
+        pq = out.with_suffix(".parquet")
+        df.to_parquet(pq, index=False)
+        log.info("Wrote Parquet → %s", pq)
+
+    # Quick summary to stdout
+    if "gpu_name" in df.columns:
+        print("\nGPU coverage:")
+        print(df.groupby(["gpu_name", "round"])["benchmark_base"].nunique()
+                .rename("benchmarks")
+                .to_string())
+    print(f"\nTotal rows: {len(df)}")
+
+
+if __name__ == "__main__":
+    main()

src/features/build_features.py

@@ -0,0 +1,335 @@
+"""
+Feature engineering for GPU Perf Prophet.
+
+Public API
+----------
+build_training_df(raw_df)
+    Filter to in-scope rows, join GPU specs, add roofline features,
+    and return a model-ready DataFrame.
+
+roofline_ceilings(total_params_b, compute_params_b, bytes_per_param, hbm_bw_tbps, peak_tflops)
+    Pure-function roofline computation used by the pipeline and notebooks.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+
+from src.data.gpu_spec_db import enrich_df
+
+log = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Reference tables
+# ---------------------------------------------------------------------------
+
+# benchmark_base → (total_params_b, compute_params_b)
+#
+# total_params_b   : all weights loaded into VRAM — used for the
+#                    memory-bandwidth ceiling and the VRAM-fit check.
+# compute_params_b : active params per forward pass — equals total_params_b
+#                    for dense models; equals active-expert params for MoE.
+#
+# Mixtral 8×7B: 8 experts of ~7B params each, but only 2 are activated per
+# token. Shared layers (attention, embeddings) add ~0.7B. We approximate
+# total = 46.7B and active = 2 × 6.7B + 0.7B ≈ 14.1B. Memory-bandwidth
+# ceiling uses total (all weights reside in HBM); compute ceiling uses active.
+#
+# Sources:
+#   llama2-70b    : Meta "Llama 2" paper, Table 1 (2023) — 70B parameters
+#   llama3.1-405b : Meta "Llama 3" blog (2024) — 405B parameters
+#   llama3.1-8b   : Meta "Llama 3" blog (2024) — 8B parameters
+#   gptj          : EleutherAI GPT-J-6B model card (2021) — 6.05B parameters
+#   mixtral-8x7b  : Mistral AI blog (2023) — 46.7B total / ~14.1B active
+MODEL_PARAMS: dict[str, tuple[float, float]] = {
+    "llama2-70b":    (70.0,  70.0),
+    "llama3.1-405b": (405.0, 405.0),
+    "llama3.1-8b":   (8.03,  8.03),   # anticipated for MLPerf v7.0+; no rows yet
+    "gptj":          (6.05,  6.05),
+    "mixtral-8x7b":  (46.7,  14.1),   # (total, active)
+}
+
+# benchmark_accuracy_tier → precision label used to select peak TFLOPS
+# and compute bytes-per-param.
+#
+# "99.9" requires near-lossless accuracy → FP16 (no quantization risk).
+# "99"   allows modest accuracy drop     → FP8  (halves memory footprint).
+# "base" has the loosest constraint      → BF16 (widely supported baseline).
+#
+# If the selected precision is not supported on a GPU (peak column is NaN or
+# None), _select_peak_tflops falls back to FP16.
+TIER_TO_PRECISION: dict[str, str] = {
+    "99.9": "fp16",
+    "99":   "fp8",
+    "base": "bf16",
+}
+
+# Bytes occupied per stored parameter at each precision.
+BYTES_PER_PARAM: dict[str, float] = {
+    "fp32": 4.0,
+    "bf16": 2.0,
+    "fp16": 2.0,
+    "fp8":  1.0,
+    "fp6":  0.75,
+    "fp4":  0.5,
+    "int8": 1.0,
+}
+
+# GPU peak TFLOPS column name for each precision label.
+_PRECISION_TO_COL: dict[str, str] = {
+    "fp32": "gpu_peak_fp32_tflops",
+    "bf16": "gpu_peak_bf16_tflops",
+    "fp16": "gpu_peak_fp16_tflops",
+    "fp8":  "gpu_peak_fp8_tflops",
+    "fp6":  "gpu_peak_fp6_tflops",
+    "fp4":  "gpu_peak_fp4_tflops",
+    "int8": "gpu_peak_int8_tops",
+}
+
+# Framework string → normalized family label.
+# Matched in order; first hit wins.
+_FRAMEWORK_PATTERNS: list[tuple[re.Pattern, str]] = [
+    (re.compile(r"TensorRT", re.IGNORECASE),  "tensorrt"),
+    (re.compile(r"vLLM",     re.IGNORECASE),  "vllm"),
+    (re.compile(r"ROCm|Mango", re.IGNORECASE), "rocm_other"),
+]
+
+# Per-vendor architecture generation ordinals (higher = newer).
+# Split by vendor so the model cannot learn spurious cross-vendor "newer = better"
+# patterns from a single linear scale (e.g. CDNA3 > Hopper is meaningless).
+# NaN for the other vendor's GPUs is intentional and correct.
+_NVIDIA_ARCH_ORDINAL: dict[str, int] = {
+    "ampere":      1,
+    "ada_lovelace": 2,
+    "hopper":      3,
+    "blackwell":   4,
+}
+_AMD_ARCH_ORDINAL: dict[str, int] = {
+    "cdna3": 1,
+    "cdna4": 2,
+}
+
+# MLPerf round tag → chronological ordinal (higher = more recent submission round).
+# Prices in framework/driver maturity (e.g. ROCm version) as a feature: early
+# rounds for a given GPU tend to run on less-tuned software stacks, which the
+# model can otherwise mistake for a hardware effect. Unrecognized round tags
+# map to NaN rather than raising, the same convention as the arch ordinals
+# above — a future round just needs a new entry here.
+ROUND_ORDINAL: dict[str, int] = {
+    "v4.1": 1,
+    "v5.0": 2,
+    "v5.1": 3,
+    "v6.0": 4,
+}
+
+
+# ---------------------------------------------------------------------------
+# Core roofline computation (pure function — also used by notebooks directly)
+# ---------------------------------------------------------------------------
+
+def roofline_ceilings(
+    total_params_b: float,
+    compute_params_b: float,
+    bytes_per_param: float,
+    hbm_bw_tbps: float,
+    peak_tflops: float,
+) -> tuple[float, float, float]:
+    """Return (bandwidth_ceiling, compute_ceiling, roofline_tput) in tokens/sec.
+
+    bandwidth_ceiling
+        A bandwidth-density proxy: how many tokens/sec HBM bandwidth could
+        sustain if reading the full model weight footprint per token.  Uses
+        *total* params (VRAM footprint), not active params, so that for MoE
+        models the feature reflects how much HBM capacity the model occupies
+        rather than per-token access volume.  Not a hard ceiling for batched
+        inference, but a useful feature capturing GPU memory-bandwidth richness
+        relative to model size.
+
+    compute_ceiling
+        The absolute maximum tokens/sec at any batch size, limited by peak
+        FLOP/s.  Uses *compute* (active) params because only the activated
+        weights participate in each forward pass.  This IS the hard physical
+        ceiling for throughput; MLPerf Offline/Server run large batches and
+        approach this bound in practice.
+
+    roofline_tput = compute_ceiling.
+        The compute ceiling is the correct upper bound for batched LLM
+        inference.  We keep bandwidth_ceiling as a separate feature rather
+        than folding it into the roofline.
+    """
+    model_bytes = total_params_b * 1e9 * bytes_per_param          # bytes
+    bw_bytes_per_sec = hbm_bw_tbps * 1e12                         # bytes/s
+    bw_ceil = bw_bytes_per_sec / model_bytes                       # tokens/s
+
+    flops_per_token = 2.0 * compute_params_b * 1e9                # FLOPs
+    peak_flops_per_sec = peak_tflops * 1e12                        # FLOPs/s
+    compute_ceil = peak_flops_per_sec / flops_per_token            # tokens/s
+
+    return bw_ceil, compute_ceil, compute_ceil
+
+
+# ---------------------------------------------------------------------------
+# Internal helpers
+# ---------------------------------------------------------------------------
+
+def _normalize_framework(raw: Optional[str]) -> str:
+    if not isinstance(raw, str):
+        return "unknown"
+    for pattern, label in _FRAMEWORK_PATTERNS:
+        if pattern.search(raw):
+            return label
+    return "other"
+
+
+def _select_peak_tflops(row: pd.Series, precision: str) -> Optional[float]:
+    """Return the peak TFLOPS for `precision`, falling back to fp16 if absent."""
+    col = _PRECISION_TO_COL.get(precision)
+    val = row.get(col) if col else None
+    if val is None or (isinstance(val, float) and pd.isna(val)):
+        # GPU doesn't support this precision natively — fall back to fp16.
+        val = row.get("gpu_peak_fp16_tflops")
+    return val
+
+
+# ---------------------------------------------------------------------------
+# Public pipeline
+# ---------------------------------------------------------------------------
+
+def build_training_df(
+    raw_df: pd.DataFrame,
+    spec_path: Optional[Path] = None,
+) -> pd.DataFrame:
+    """Filter, enrich, and featurise raw MLPerf data for model training.
+
+    Steps
+    -----
+    1. Enrich with GPU spec columns via ``enrich_df``.
+    2. Filter: ``result_valid == True`` only.  ``gpu_in_model_scope`` is a
+       serving-layer gate, not a training filter.
+    3. Attach model-parameter reference values.
+    4. Compute roofline ceilings (bandwidth, compute ceiling only).
+    5. Derive secondary features (efficiency ratio, VRAM fit, framework family,
+       architecture ordinal, vendor indicator).
+
+    Returns
+    -------
+    pd.DataFrame with all original columns plus the features listed below.
+    Rows that cannot be featurised (unknown benchmark_base or missing GPU
+    specs) are dropped with a warning rather than silently propagating NaN
+    into training.
+    """
+    kwargs = {"spec_path": spec_path} if spec_path is not None else {}
+    df = enrich_df(raw_df, **kwargs).copy()
+
+    # --- filter ---
+    # gpu_in_model_scope gates recommendation exposure, not training inclusion.
+    # Out-of-scope GPUs (e.g. B200, H200 NVL) are valid training signal; they
+    # inform the model about architectural trends even if not served to users v1.
+    df = df[df["result_valid"]]
+    log.info("After result_valid filter: %d rows", len(df))
+
+    # --- model params ---
+    df["model_total_params_b"] = df["benchmark_base"].map(
+        {k: v[0] for k, v in MODEL_PARAMS.items()}
+    )
+    df["model_compute_params_b"] = df["benchmark_base"].map(
+        {k: v[1] for k, v in MODEL_PARAMS.items()}
+    )
+
+    unknown_benchmarks = df[df["model_total_params_b"].isna()]["benchmark_base"].unique()
+    if len(unknown_benchmarks):
+        log.warning("Dropping %d rows with unknown benchmark_base: %s",
+                    df["model_total_params_b"].isna().sum(), unknown_benchmarks)
+        df = df[df["model_total_params_b"].notna()]
+
+    # --- precision selection ---
+    df["selected_precision"] = df["benchmark_accuracy_tier"].map(TIER_TO_PRECISION)
+    # AMD CDNA hardware achieves 99.9 accuracy with FP8, not FP16.  The
+    # TIER_TO_PRECISION default (99.9→fp16) is correct for NVIDIA; override
+    # it for AMD so efficiency_ratio targets are computed against the right
+    # ceiling (2× the TFLOPS of the FP16 ceiling, halving the efficiency
+    # ratio for those rows and eliminating the ceiling violations in training).
+    amd_tier_99_9 = (df["gpu_vendor"] == "amd") & (df["benchmark_accuracy_tier"] == "99.9")
+    df.loc[amd_tier_99_9, "selected_precision"] = "fp8"
+    df["bytes_per_param"] = df["selected_precision"].map(BYTES_PER_PARAM)
+
+    # Vectorised equivalent of _select_peak_tflops over all rows.
+    # Covers the three precision values that TIER_TO_PRECISION produces;
+    # fp16 is the fallback for any unknown precision or for GPUs whose
+    # selected-precision column is NaN (same as the scalar helper).
+    _fp16 = df["gpu_peak_fp16_tflops"]
+    _fp8 = df["gpu_peak_fp8_tflops"].where(df["gpu_peak_fp8_tflops"].notna(), _fp16)
+    _bf16 = df["gpu_peak_bf16_tflops"].where(df["gpu_peak_bf16_tflops"].notna(), _fp16)
+    _prec = df["selected_precision"]
+    df["peak_tflops_selected"] = _fp8.where(
+        _prec == "fp8", _bf16.where(_prec == "bf16", _fp16)
+    )
+
+    # --- roofline ---
+    missing_specs = df[
+        df["gpu_hbm_bandwidth_tbps"].isna() | df["peak_tflops_selected"].isna()
+    ]
+    if len(missing_specs):
+        log.warning("Dropping %d rows with missing GPU specs (cannot compute roofline)",
+                    len(missing_specs))
+        df = df[
+            df["gpu_hbm_bandwidth_tbps"].notna() & df["peak_tflops_selected"].notna()
+        ]
+
+    _model_bytes = df["model_total_params_b"] * 1e9 * df["bytes_per_param"]
+    df["bandwidth_ceiling_tok_per_sec"] = (df["gpu_hbm_bandwidth_tbps"] * 1e12) / _model_bytes
+    df["compute_ceiling_tok_per_sec"] = (df["peak_tflops_selected"] * 1e12) / (
+        2.0 * df["model_compute_params_b"] * 1e9
+    )
+    df["roofline_tput"] = df["compute_ceiling_tok_per_sec"]
+
+    # --- derived features ---
+    df["efficiency_ratio"] = (
+        df["throughput_tok_per_sec_per_gpu"] / df["roofline_tput"]
+    )
+    df["model_size_gb"] = (
+        df["model_total_params_b"] * df["bytes_per_param"]
+    )
+    df["model_to_vram_ratio"] = df["model_size_gb"] / df["gpu_vram_gb"]
+    # Deduplicate framework strings before normalizing — ~10–20 unique values
+    # across ~1112 rows, so .apply() would call _normalize_framework (3 regex
+    # searches) ~1112 times.  Map unique values once, then broadcast.
+    # Parallels the per-unique-gpu-name optimization in enrich_df ("Fix 2").
+    _fw_unique_map = {fw: _normalize_framework(fw) for fw in df["framework"].dropna().unique()}
+    df["framework_family"] = df["framework"].map(_fw_unique_map).fillna("unknown")
+    df["nvidia_arch_gen"] = df["gpu_architecture"].map(_NVIDIA_ARCH_ORDINAL)
+    df["amd_arch_gen"] = df["gpu_architecture"].map(_AMD_ARCH_ORDINAL)
+    df["vendor_is_amd"] = (df["gpu_vendor"] == "amd").astype(int)
+    df["mlperf_round_num"] = df["round"].map(ROUND_ORDINAL)
+    # Binary flag separating the "base" accuracy tier (BF16, loose accuracy
+    # constraint) from the "99" and "99.9" tiers (FP8 / FP16 or the AMD FP8
+    # override).  Replaces the three-level accuracy_tier_ord: after the AMD
+    # FP8 override, tiers "99" and "99.9" are both FP8 for AMD, making
+    # accuracy_tier_ord a spurious discriminator within AMD LOGO folds.
+    # bytes_per_param already captures the FP8 vs FP16 precision split for
+    # NVIDIA; is_base_tier captures the remaining base-vs-non-base distinction
+    # that bytes_per_param cannot express (base BF16 = 2.0, same as NVIDIA FP16).
+    df["is_base_tier"] = (df["benchmark_accuracy_tier"] == "base").astype(int)
+
+    # Efficiency ratio > 1 means actual throughput exceeds the compute ceiling
+    # at the precision we inferred from benchmark_accuracy_tier.  This is
+    # expected for AMD CDNA4 rows at the "99.9" tier: AMD's vLLM/ROCm stack
+    # achieves 99.9 accuracy with FP8 (higher TFLOPS ceiling), but our proxy
+    # maps 99.9 → FP16.  These rows are valid training data — the model learns
+    # that AMD systematically outperforms the FP16 ceiling for this tier.
+    n_violations = (df["throughput_tok_per_sec_per_gpu"] > df["roofline_tput"]).sum()
+    if n_violations:
+        log.warning(
+            "%d rows (%.1f%%) have throughput > compute ceiling at selected "
+            "precision — likely precision-proxy mismatch (AMD FP8 at 99.9 tier).",
+            n_violations,
+            100 * n_violations / len(df),
+        )
+
+    log.info("build_training_df complete: %d rows, %d columns", len(df), df.shape[1])
+    return df.reset_index(drop=True)

src/models/predictor.py

@@ -0,0 +1,360 @@
+"""
+GPU Perf Prophet — inference module.
+
+Public API
+----------
+GpuPredictor(model_dir)
+    Load the trained XGBoost model and metadata from disk.
+
+predictor.predict(gpu_id, model_name, scenario, accuracy_tier, framework)
+    Predict tokens/sec for one (GPU, workload) pair.
+
+predictor.predict_batch(requests)
+    Vectorised prediction for a list of dicts.
+
+Feature construction mirrors the exact encoding used in
+notebooks/03_model_training.ipynb so the serving path cannot diverge.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import logging
+import stat as _stat
+from pathlib import Path
+
+import numpy as np
+import xgboost as xgb
+
+from src.data.gpu_spec_db import load_specs
+from src.features.build_features import (
+    MODEL_PARAMS,
+    TIER_TO_PRECISION,
+    ROUND_ORDINAL,
+    BYTES_PER_PARAM,
+    _NVIDIA_ARCH_ORDINAL,
+    _AMD_ARCH_ORDINAL,
+    roofline_ceilings,
+)
+
+log = logging.getLogger(__name__)
+
+_DEFAULT_MODEL_DIR = Path(__file__).parent.parent.parent / "data" / "models"
+
+# File-guard caps for model artifacts — mirrors load_specs() and _load_pricing().
+_MAX_META_BYTES: int = 1 * 1024 * 1024   # 1 MB; real metadata JSON is ~1 KB
+_MAX_MODEL_BYTES: int = 50 * 1024 * 1024  # 50 MB; matches RT-5.3 disk budget
+
+# At serving time we always predict for the most mature software stack —
+# i.e. the most recent MLPerf round.  This corrects the ROCm-maturity
+# confound without exposing round_tag as an API parameter.
+_SERVING_ROUND: float = float(max(ROUND_ORDINAL.values()))
+
+# Feature column order — must match FEATURE_COLS in the training notebook exactly.
+FEATURE_COLS: list[str] = [
+    # GPU hardware
+    "gpu_hbm_bandwidth_tbps",
+    "gpu_vram_gb",
+    "peak_tflops_selected",
+    "compute_ceiling_tok_per_sec",
+    "bandwidth_ceiling_tok_per_sec",
+    # Model
+    "model_total_params_b",
+    "model_compute_params_b",
+    "model_size_gb",
+    "model_to_vram_ratio",
+    "bytes_per_param",
+    # Architecture
+    "nvidia_arch_gen",
+    "amd_arch_gen",
+    "vendor_is_amd",
+    # Encoded categoricals
+    "scenario_offline",
+    "is_base_tier",
+    "fw_tensorrt",
+    "fw_vllm",
+    "fw_rocm_other",
+    "is_cdna4",
+    # Context
+    "mlperf_round_num",
+]
+
+VALID_SCENARIOS: frozenset[str] = frozenset({"Offline", "Server"})
+VALID_TIERS: frozenset[str] = frozenset({"base", "99", "99.9"})
+VALID_FRAMEWORKS: frozenset[str] = frozenset({"vllm", "tensorrt", "rocm_other", "other"})
+VALID_MODELS: frozenset[str] = frozenset(MODEL_PARAMS.keys())
+
+
+def _build_feature_vector(
+    *,
+    gpu_spec: dict,
+    model_name: str,
+    scenario: str,
+    accuracy_tier: str,
+    framework: str,
+) -> tuple[list[float], float, float]:
+    """Return (feature_vector, roofline_tput, model_size_gb) for one (GPU, workload) pair.
+
+    roofline_tput and model_size_gb are returned so callers don't re-derive
+    them with a second copy of the AMD precision override logic.
+    """
+    total_params_b, compute_params_b = MODEL_PARAMS[model_name]
+
+    selected_precision = TIER_TO_PRECISION[accuracy_tier]
+    # Mirror the AMD 99.9-tier override from build_features.build_training_df.
+    if gpu_spec.get("vendor") == "amd" and accuracy_tier == "99.9":
+        selected_precision = "fp8"
+    bpp = BYTES_PER_PARAM[selected_precision]
+
+    # Peak TFLOPS: use selected precision, fall back to fp16 if None/NaN.
+    pt = gpu_spec.get("peak_tflops") or {}
+    peak_tflops = pt.get(selected_precision)
+    if peak_tflops is None or (isinstance(peak_tflops, float) and np.isnan(peak_tflops)):
+        peak_tflops = pt.get("fp16")
+
+    hbm_bw = gpu_spec["hbm_bandwidth_tbps"]
+    vram_gb = gpu_spec["vram_gb"]
+
+    bw_ceil, compute_ceil, roofline_tput = roofline_ceilings(
+        total_params_b, compute_params_b, bpp, hbm_bw, peak_tflops
+    )
+
+    model_size_gb = total_params_b * bpp
+    model_to_vram_ratio = model_size_gb / vram_gb
+
+    arch = gpu_spec.get("architecture", "")
+    nvidia_arch_gen = _NVIDIA_ARCH_ORDINAL.get(arch)
+    amd_arch_gen = _AMD_ARCH_ORDINAL.get(arch)
+    vendor_is_amd = int(gpu_spec.get("vendor", "") == "amd")
+
+    is_cdna4 = int(amd_arch_gen == 2) if amd_arch_gen is not None else 0
+
+    scenario_offline = int(scenario == "Offline")
+    is_base_tier = int(accuracy_tier == "base")
+    fw_tensorrt = int(framework == "tensorrt")
+    fw_vllm = int(framework == "vllm")
+    fw_rocm_other = int(framework == "rocm_other")
+
+    # NaN for the other vendor's arch ordinal — XGBoost handles missing natively.
+    features: list[float] = [
+        hbm_bw,
+        vram_gb,
+        peak_tflops,
+        compute_ceil,
+        bw_ceil,
+        total_params_b,
+        compute_params_b,
+        model_size_gb,
+        model_to_vram_ratio,
+        bpp,
+        float(nvidia_arch_gen) if nvidia_arch_gen is not None else float("nan"),
+        float(amd_arch_gen) if amd_arch_gen is not None else float("nan"),
+        vendor_is_amd,
+        scenario_offline,
+        is_base_tier,
+        fw_tensorrt,
+        fw_vllm,
+        fw_rocm_other,
+        is_cdna4,
+        _SERVING_ROUND,
+    ]
+    return features, roofline_tput, model_size_gb
+
+
+class GpuPredictor:
+    """Load-once, predict-many XGBoost inference wrapper."""
+
+    def __init__(self, model_dir: Path | str = _DEFAULT_MODEL_DIR) -> None:
+        model_dir = Path(model_dir)
+        meta_path = model_dir / "feature_metadata.json"
+        model_path = model_dir / "prophet_v1.json"
+
+        # File guards — mirrors the policy in load_specs() and _load_pricing().
+        # Symlink checks prevent path-traversal redirects; size caps prevent
+        # unbounded memory consumption.  Applied before open() so the checks
+        # are not compiled away (unlike assert statements under python -O).
+        for _path, _cap in ((meta_path, _MAX_META_BYTES), (model_path, _MAX_MODEL_BYTES)):
+            try:
+                _st = _path.lstat()
+            except OSError as exc:
+                raise FileNotFoundError(f"Model artifact not found: {_path}") from exc
+            if _stat.S_ISLNK(_st.st_mode):
+                raise ValueError(f"Model artifact path is a symlink (refused): {_path}")
+            if _st.st_size > _cap:
+                raise ValueError(
+                    f"Model artifact too large ({_st.st_size} bytes > {_cap}): {_path}"
+                )
+
+        with meta_path.open() as f:
+            self._meta = json.load(f)
+
+        # assert is compiled away under `python -O` / PYTHONOPTIMIZE=1 — use
+        # explicit raise so this check is never a no-op in any deployment mode.
+        if self._meta["feature_cols"] != FEATURE_COLS:
+            raise ValueError(
+                "feature_metadata.json feature_cols mismatch — retrain the model. "
+                f"Expected {len(FEATURE_COLS)} cols, got "
+                f"{len(self._meta.get('feature_cols', []))}."
+            )
+
+        self._model = xgb.XGBRegressor()
+        self._model.load_model(str(model_path))
+
+        specs = load_specs()
+        # Deep-copy each spec dict so _id_map holds independent objects.
+        # load_specs() is lru_cache'd and returns its live list; storing direct
+        # references means any write to a spec value (e.g. caching a derived
+        # field) would silently corrupt the global cache for all callers.
+        self._id_map: dict[str, dict] = {s["id"]: copy.deepcopy(s) for s in specs}
+
+        log.info(
+            "GpuPredictor loaded: model=%s  features=%d  gpus=%d",
+            model_path.name, len(FEATURE_COLS), len(self._id_map),
+        )
+
+    # ------------------------------------------------------------------
+    # Public API
+    # ------------------------------------------------------------------
+
+    def predict(
+        self,
+        *,
+        gpu_id: str,
+        model_name: str,
+        scenario: str = "Offline",
+        accuracy_tier: str = "99",
+        framework: str = "vllm",
+    ) -> dict:
+        """Predict inference throughput for one (GPU, workload) pair.
+
+        Returns
+        -------
+        dict with keys:
+            gpu_id, model_name, scenario, accuracy_tier, framework,
+            pred_throughput_tok_per_sec, roofline_tput_tok_per_sec,
+            efficiency_ratio, vram_fits
+        """
+        self._validate(gpu_id, model_name, scenario, accuracy_tier, framework)
+        gpu_spec = self._id_map[gpu_id]
+
+        features, roofline_tput, model_size_gb = _build_feature_vector(
+            gpu_spec=gpu_spec,
+            model_name=model_name,
+            scenario=scenario,
+            accuracy_tier=accuracy_tier,
+            framework=framework,
+        )
+
+        X = np.array([features], dtype=np.float32)
+        pred_eff = float(self._model.predict(X)[0])
+        pred_tput = pred_eff * roofline_tput
+
+        # Enforce roofline ceiling on the output (< 2% violation rate in CV;
+        # clamp rather than raise so API remains responsive).
+        pred_tput = min(pred_tput, roofline_tput)
+
+        vram_fits = model_size_gb <= gpu_spec["vram_gb"]
+
+        return {
+            "gpu_id": gpu_id,
+            "model_name": model_name,
+            "scenario": scenario,
+            "accuracy_tier": accuracy_tier,
+            "framework": framework,
+            "pred_throughput_tok_per_sec": round(pred_tput, 2),
+            "roofline_tput_tok_per_sec": round(roofline_tput, 2),
+            "efficiency_ratio": round(pred_eff, 4),
+            "vram_fits": vram_fits,
+            "model_size_gb": round(model_size_gb, 2),
+        }
+
+    def predict_batch(self, requests: list[dict]) -> list[dict]:
+        """Vectorised prediction over a list of request dicts.
+
+        Each dict must contain the same keys as predict() keyword args.
+        """
+        if not requests:
+            return []
+
+        feature_matrix: list[list[float]] = []
+        roofline_tputs: list[float] = []
+        meta: list[dict] = []
+
+        for req in requests:
+            gpu_id = req["gpu_id"]
+            model_name = req["model_name"]
+            scenario = req.get("scenario", "Offline")
+            accuracy_tier = req.get("accuracy_tier", "99")
+            framework = req.get("framework", "vllm")
+
+            self._validate(gpu_id, model_name, scenario, accuracy_tier, framework)
+            gpu_spec = self._id_map[gpu_id]
+            features, roofline_tput, model_size_gb = _build_feature_vector(
+                gpu_spec=gpu_spec,
+                model_name=model_name,
+                scenario=scenario,
+                accuracy_tier=accuracy_tier,
+                framework=framework,
+            )
+            feature_matrix.append(features)
+            roofline_tputs.append(roofline_tput)
+
+            meta.append({
+                "gpu_id": gpu_id,
+                "model_name": model_name,
+                "scenario": scenario,
+                "accuracy_tier": accuracy_tier,
+                "framework": framework,
+                "model_size_gb": round(model_size_gb, 2),
+                "vram_fits": model_size_gb <= gpu_spec["vram_gb"],
+            })
+
+        X = np.array(feature_matrix, dtype=np.float32)
+        pred_effs = self._model.predict(X)
+
+        results = []
+        for req_meta, pred_eff, roofline_tput in zip(meta, pred_effs, roofline_tputs):
+            pred_tput = min(float(pred_eff) * roofline_tput, roofline_tput)
+            results.append({
+                **req_meta,
+                "pred_throughput_tok_per_sec": round(pred_tput, 2),
+                "roofline_tput_tok_per_sec": round(roofline_tput, 2),
+                "efficiency_ratio": round(float(pred_eff), 4),
+            })
+        return results
+
+    # ------------------------------------------------------------------
+    # Internal
+    # ------------------------------------------------------------------
+
+    def _validate(
+        self,
+        gpu_id: str,
+        model_name: str,
+        scenario: str,
+        accuracy_tier: str,
+        framework: str,
+    ) -> None:
+        if gpu_id not in self._id_map:
+            raise ValueError(
+                f"Unknown gpu_id {gpu_id!r}. "
+                f"Valid: {sorted(self._id_map)}"
+            )
+        if model_name not in VALID_MODELS:
+            raise ValueError(
+                f"Unknown model_name {model_name!r}. "
+                f"Valid: {sorted(VALID_MODELS)}"
+            )
+        if scenario not in VALID_SCENARIOS:
+            raise ValueError(
+                f"Invalid scenario {scenario!r}. Valid: {sorted(VALID_SCENARIOS)}"
+            )
+        if accuracy_tier not in VALID_TIERS:
+            raise ValueError(
+                f"Invalid accuracy_tier {accuracy_tier!r}. Valid: {sorted(VALID_TIERS)}"
+            )
+        if framework not in VALID_FRAMEWORKS:
+            raise ValueError(
+                f"Invalid framework {framework!r}. Valid: {sorted(VALID_FRAMEWORKS)}"
+            )

src/models/train_final.py

@@ -0,0 +1,133 @@
+"""
+Train and save the production XGBoost model for GPU Perf Prophet.
+
+Reads  : data/processed/mlperf_features.parquet
+Writes : data/models/prophet_v1.json
+         data/models/feature_metadata.json
+
+Run once LOGO-CV evaluation is complete:
+    python -m src.models.train_final
+
+The feature encoding mirrors notebooks/03_model_training.ipynb cell 3 exactly
+so the saved model is compatible with GpuPredictor in src/models/predictor.py.
+
+Training scope
+--------------
+This script trains on ALL rows with a valid target (~1112 rows), including
+out-of-scope GPUs (B200, H200 NVL, etc.).  The training notebook's SHAP
+analysis used only the 649 in-scope rows, but LOGO-CV training folds always
+contained out-of-scope rows, so the production model mirrors what the CV folds
+actually saw.  Out-of-scope GPUs are never served to users (gated by
+gpu_in_model_scope in GpuRecommender); they are included here purely as
+additional training signal about architectural trends.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import xgboost as xgb
+
+from src.models.predictor import FEATURE_COLS
+
+logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+log = logging.getLogger(__name__)
+
+TARGET = "efficiency_ratio"
+THROUGHPUT_COL = "throughput_tok_per_sec_per_gpu"
+
+# Same hyperparameters as the training notebook's final model (no early stopping).
+# n_estimators set to 250 — midpoint of observed best_iteration range across
+# folds (77–341); conservative to avoid overfitting without an eval set.
+PROD_PARAMS: dict = {
+    "n_estimators": 250,
+    "max_depth": 5,
+    "learning_rate": 0.03,
+    "subsample": 0.8,
+    "colsample_bytree": 0.8,
+    "min_child_weight": 5,
+    "tree_method": "hist",
+    "random_state": 42,
+}
+
+FEATURES_PATH = Path("data/processed/mlperf_features.parquet")
+MODEL_DIR = Path("data/models")
+
+
+def _encode(feat: pd.DataFrame) -> pd.DataFrame:
+    """Apply the same categorical encoding as notebook cell 3."""
+    df = feat.copy()
+
+    df["scenario_offline"] = (df["scenario"] == "Offline").astype(int)
+
+    df["fw_tensorrt"] = (df["framework_family"] == "tensorrt").astype(int)
+    df["fw_vllm"] = (df["framework_family"] == "vllm").astype(int)
+    df["fw_rocm_other"] = (df["framework_family"] == "rocm_other").astype(int)
+
+    df["is_cdna4"] = (df["amd_arch_gen"] == 2).astype(int)
+
+    return df
+
+
+def train_and_save(
+    features_path: Path = FEATURES_PATH,
+    model_dir: Path = MODEL_DIR,
+) -> xgb.XGBRegressor:
+    feat = pd.read_parquet(features_path)
+    log.info("Loaded %d rows from %s", len(feat), features_path)
+
+    df = _encode(feat)
+
+    # Drop rows where target is NaN or infinite.
+    valid = df[TARGET].notna() & np.isfinite(df[TARGET])
+    df = df[valid].reset_index(drop=True)
+    log.info("Training rows (valid target): %d", len(df))
+
+    # Variance gate — matches the training notebook's assertion.
+    # assert compiled away under python -O; use explicit raise.
+    dead = [c for c in FEATURE_COLS if df[c].nunique() <= 1]
+    if dead:
+        raise ValueError(f"Dead features (constant across all training rows): {dead}")
+
+    X = df[FEATURE_COLS]
+    y = df[TARGET]
+
+    model = xgb.XGBRegressor(**PROD_PARAMS)
+    model.fit(X, y)
+    log.info("Model trained: %d trees, %d features", PROD_PARAMS["n_estimators"], len(FEATURE_COLS))
+
+    model_dir.mkdir(parents=True, exist_ok=True)
+    model_path = model_dir / "prophet_v1.json"
+    meta_path = model_dir / "feature_metadata.json"
+
+    # Write both artifacts atomically via temp-file + os.replace().
+    # If this process is killed between the two writes, the reader
+    # (GpuPredictor.__init__) sees either the old pair or the new pair —
+    # never a mixed state that would silently mismatch feature_cols.
+    tmp_model = model_path.with_name(model_path.stem + ".tmp.json")
+    model.save_model(str(tmp_model))
+    tmp_model.replace(model_path)
+    log.info("Saved model → %s", model_path)
+
+    meta = {
+        "feature_cols": FEATURE_COLS,
+        "target": TARGET,
+        "model_version": "v1",
+        "prod_params": PROD_PARAMS,
+        "n_training_rows": len(df),
+    }
+    tmp_meta = meta_path.with_name(meta_path.name + ".tmp")
+    with tmp_meta.open("w") as f:
+        json.dump(meta, f, indent=2)
+    tmp_meta.replace(meta_path)
+    log.info("Saved metadata → %s", meta_path)
+
+    return model
+
+
+if __name__ == "__main__":
+    train_and_save()

src/recommend/recommender.py

@@ -0,0 +1,314 @@
+"""
+GPU Perf Prophet — Pareto recommendation engine.
+
+Public API
+----------
+GpuRecommender(predictor, pricing_path)
+    Wraps GpuPredictor and adds Pareto multi-objective ranking.
+
+recommender.recommend(model_name, scenario, accuracy_tier, framework,
+                      budget_per_gpu_hr, min_throughput_tok_per_sec)
+    Return a ranked list of GPU recommendations for the given workload.
+
+Objectives (all higher-is-better after normalisation):
+    1. throughput — predicted tokens/sec
+    2. cost_efficiency — tokens per dollar (throughput / price_per_hr)
+    3. vram_headroom — fraction of VRAM unused (1 - model_to_vram_ratio)
+
+Constraints (hard filters applied before ranking):
+    • VRAM fit:  model must fit on a single GPU (model_size_gb ≤ gpu_vram_gb)
+    • Budget:    price_per_gpu_hr ≤ budget_per_gpu_hr  (if provided)
+    • Min tput:  pred_throughput ≥ min_throughput       (if provided)
+
+Pareto frontier:
+    A GPU is Pareto-dominated if another GPU is at least as good on all
+    three objectives and strictly better on at least one.  The frontier
+    contains all non-dominated GPUs, sorted by cost_efficiency descending
+    (best tokens-per-dollar first).  Dominated GPUs are returned in a
+    separate list so the UI can show them as alternatives.
+"""
+
+from __future__ import annotations
+
+import logging
+import stat as _stat
+from pathlib import Path
+from typing import Optional
+
+import yaml
+
+from src.data.gpu_spec_db import load_specs
+from src.models.predictor import GpuPredictor, MODEL_PARAMS, TIER_TO_PRECISION, BYTES_PER_PARAM
+
+log = logging.getLogger(__name__)
+
+_DEFAULT_PRICING_PATH = Path(__file__).parent.parent.parent / "data" / "pricing.yaml"
+
+# Real pricing files are < 1 KB.  1 MB cap matches gpu_spec_db.py's policy.
+_MAX_PRICING_BYTES: int = 1 * 1024 * 1024  # 1 MB
+
+
+def _load_pricing(path: Path) -> dict[str, float]:
+    # Mirror the symlink and size guards from gpu_spec_db.load_specs so the
+    # pricing file cannot be swapped out via a filesystem symlink.
+    try:
+        st = path.lstat()
+    except OSError as exc:
+        raise FileNotFoundError(f"Pricing DB not found: {path}") from exc
+    if _stat.S_ISLNK(st.st_mode):
+        raise ValueError(f"Pricing DB path is a symlink (refused): {path}")
+    if st.st_size > _MAX_PRICING_BYTES:
+        raise ValueError(
+            f"Pricing DB too large ({st.st_size} bytes > {_MAX_PRICING_BYTES}): {path}"
+        )
+    with path.open() as f:
+        data = yaml.safe_load(f)
+    if not isinstance(data, dict) or "pricing" not in data:
+        raise ValueError(f"pricing.yaml at {path} is missing required 'pricing' key")
+    result: dict[str, float] = {}
+    for gpu_id, entry in data["pricing"].items():
+        if not isinstance(entry, dict) or "price_per_gpu_hr" not in entry:
+            raise ValueError(
+                f"pricing.yaml entry {gpu_id!r} is missing 'price_per_gpu_hr' key"
+            )
+        result[gpu_id] = entry["price_per_gpu_hr"]
+    return result
+
+
+def _pareto_frontier(candidates: list[dict]) -> tuple[list[dict], list[dict]]:
+    """Split candidates into (frontier, dominated).
+
+    Each candidate dict must have numeric keys:
+        throughput, cost_efficiency, vram_headroom  (all higher-is-better)
+
+    None is treated as -inf for dominance comparisons and sorts last.
+    """
+    objectives = ["throughput", "cost_efficiency", "vram_headroom"]
+    frontier: list[dict] = []
+    dominated: list[dict] = []
+
+    # None objective value → treat as -inf so unpriced GPUs are always
+    # dominated by any priced GPU and don't crash comparisons.
+    def _obj(v) -> float:
+        return v if v is not None else float("-inf")
+
+    def _dominates(a: dict, b: dict) -> bool:
+        """Return True if a dominates b: a >= b on all objectives, a > b on ≥1.
+
+        Single pass over objectives: returns False immediately on the first
+        objective where a < b, avoiding the O(2k) double-evaluation of the
+        separate all()/any() generators.
+        """
+        has_strict = False
+        for obj in objectives:
+            ao, bo = _obj(a[obj]), _obj(b[obj])
+            if ao < bo:
+                return False
+            if ao > bo:
+                has_strict = True
+        return has_strict
+
+    for i, cand in enumerate(candidates):
+        is_dominated = any(
+            _dominates(other, cand)
+            for j, other in enumerate(candidates)
+            if j != i
+        )
+        if is_dominated:
+            dominated.append(cand)
+        else:
+            frontier.append(cand)
+
+    # Sort frontier by cost_efficiency descending (best tokens-per-dollar first).
+    # None sorts last so unpriced GPUs appear after all priced GPUs.
+    def _ce(x: dict) -> float:
+        v = x["cost_efficiency"]
+        return v if v is not None else float("-inf")
+
+    frontier.sort(key=_ce, reverse=True)
+    dominated.sort(key=_ce, reverse=True)
+    return frontier, dominated
+
+
+class GpuRecommender:
+    """Multi-objective GPU recommender wrapping GpuPredictor."""
+
+    def __init__(
+        self,
+        predictor: GpuPredictor,
+        pricing_path: Path | str = _DEFAULT_PRICING_PATH,
+    ) -> None:
+        self._predictor = predictor
+        self._pricing = _load_pricing(Path(pricing_path))
+
+        specs = load_specs()
+        self._in_scope_ids: list[str] = [
+            s["id"] for s in specs if s.get("in_model_scope")
+        ]
+        # Re-use the predictor's already-deep-copied spec map — this class never
+        # writes to spec dicts, so sharing is safe and avoids a second full
+        # deepcopy of every spec at init time.
+        self._spec_map: dict[str, dict] = predictor._id_map
+
+        # Fail fast: a missing pricing entry produces cost_efficiency=None,
+        # which causes a TypeError in _pareto_frontier's sort and comparisons.
+        missing = [gid for gid in self._in_scope_ids if gid not in self._pricing]
+        if missing:
+            raise ValueError(
+                f"pricing.yaml is missing entries for in-scope GPUs: {missing}. "
+                "Add a price_per_gpu_hr entry before enabling these GPUs."
+            )
+
+        log.info(
+            "GpuRecommender ready: %d in-scope GPUs, %d pricing entries",
+            len(self._in_scope_ids), len(self._pricing),
+        )
+
+    # ------------------------------------------------------------------
+    # Public API
+    # ------------------------------------------------------------------
+
+    def recommend(
+        self,
+        *,
+        model_name: str,
+        scenario: str = "Offline",
+        accuracy_tier: str = "99",
+        framework: str = "vllm",
+        budget_per_gpu_hr: Optional[float] = None,
+        min_throughput_tok_per_sec: Optional[float] = None,
+    ) -> dict:
+        """Return a recommendation result dict.
+
+        Keys
+        ----
+        frontier : list[dict]   — Pareto-optimal GPUs, best tokens/$ first
+        dominated: list[dict]   — remaining GPUs that passed hard constraints
+        filtered : list[dict]   — GPUs removed by hard constraints (vram / budget)
+        workload : dict         — echoed inputs + model_size_gb
+        """
+        if model_name not in MODEL_PARAMS:
+            raise ValueError(
+                f"Unknown model_name {model_name!r}. Valid: {sorted(MODEL_PARAMS)}"
+            )
+        total_params_b, _ = MODEL_PARAMS[model_name]
+        bpp = BYTES_PER_PARAM[TIER_TO_PRECISION[accuracy_tier]]
+        model_size_gb = total_params_b * bpp  # workload summary (canonical, FP16 for tier 99.9)
+
+        # Per-GPU effective model size — AMD uses FP8 at 99.9 tier, halving the
+        # footprint vs the FP16 default.  The VRAM pre-filter and reject messages
+        # must use this per-GPU value; candidates use pred["model_size_gb"] which
+        # comes from predict_batch() and already applies the same override.
+        def _gpu_model_size_gb(gpu_id: str) -> float:
+            spec = self._spec_map[gpu_id]
+            eff_bpp = bpp
+            if spec.get("vendor") == "amd" and accuracy_tier == "99.9":
+                eff_bpp = BYTES_PER_PARAM["fp8"]
+            return total_params_b * eff_bpp
+
+        # Pre-filter by VRAM before calling predict_batch: skip XGBoost inference
+        # for GPUs where the model provably cannot fit.  Matters most for large
+        # models (llama3.1-405b at fp8 = 405 GB; no in-scope GPU reaches that).
+        # Compute effective model size once per GPU (avoid 4× calls per failing GPU
+        # across the dual list comprehensions and the reject entry builder below).
+        gpu_sizes: dict[str, float] = {
+            gid: _gpu_model_size_gb(gid) for gid in self._in_scope_ids
+        }
+        vram_ok_ids: list[str] = []
+        vram_fail_ids: list[str] = []
+        for gid in self._in_scope_ids:
+            fits = self._spec_map[gid]["vram_gb"] >= gpu_sizes[gid]
+            (vram_ok_ids if fits else vram_fail_ids).append(gid)
+
+        requests = [
+            {
+                "gpu_id": gpu_id,
+                "model_name": model_name,
+                "scenario": scenario,
+                "accuracy_tier": accuracy_tier,
+                "framework": framework,
+            }
+            for gpu_id in vram_ok_ids
+        ]
+        predictions = self._predictor.predict_batch(requests)
+
+        candidates: list[dict] = []
+        filtered: list[dict] = []
+
+        # Build reject entries for VRAM-failing GPUs without running inference.
+        for gpu_id in vram_fail_ids:
+            spec = self._spec_map[gpu_id]
+            price = self._pricing.get(gpu_id)
+            _sz = gpu_sizes[gpu_id]
+            filtered.append({
+                "gpu_id": gpu_id,
+                "gpu_name": spec.get("name", gpu_id),
+                "vendor": spec.get("vendor", ""),
+                "model_name": model_name,
+                "scenario": scenario,
+                "accuracy_tier": accuracy_tier,
+                "framework": framework,
+                "pred_throughput_tok_per_sec": 0.0,
+                "roofline_tput_tok_per_sec": 0.0,
+                "efficiency_ratio": 0.0,
+                "vram_fits": False,
+                "model_size_gb": round(_sz, 2),
+                "vram_gb": spec.get("vram_gb"),
+                "price_per_gpu_hr": price,
+                "vram_headroom": 0.0,
+                "cost_efficiency": None,
+                "throughput": 0.0,
+                "reject_reason": (
+                    f"model too large ({_sz:.1f} GB"
+                    f" > {spec['vram_gb']} GB VRAM)"
+                ),
+            })
+
+        for pred in predictions:
+            gpu_id = pred["gpu_id"]
+            spec = self._spec_map[gpu_id]
+            price = self._pricing.get(gpu_id)
+            pred_tput = pred["pred_throughput_tok_per_sec"]
+
+            reject_reason = None
+            if budget_per_gpu_hr is not None and price is not None and price > budget_per_gpu_hr:
+                reject_reason = f"price ${price:.2f}/hr > budget ${budget_per_gpu_hr:.2f}/hr"
+            elif min_throughput_tok_per_sec is not None and pred_tput < min_throughput_tok_per_sec:
+                reject_reason = (
+                    f"predicted {pred_tput:.0f} tok/s"
+                    f" < minimum {min_throughput_tok_per_sec:.0f} tok/s"
+                )
+
+            entry = {
+                **pred,
+                "gpu_name":        spec.get("name", gpu_id),
+                "vendor":          spec.get("vendor", ""),
+                "vram_gb":         spec.get("vram_gb"),
+                "price_per_gpu_hr": price,
+                "vram_headroom":   max(0.0, 1.0 - pred["model_size_gb"] / spec["vram_gb"]),
+                "cost_efficiency": (pred_tput / price) if price else None,
+                "throughput":      pred_tput,
+            }
+
+            if reject_reason:
+                entry["reject_reason"] = reject_reason
+                filtered.append(entry)
+            else:
+                candidates.append(entry)
+
+        frontier, dominated = _pareto_frontier(candidates)
+
+        return {
+            "frontier":  frontier,
+            "dominated": dominated,
+            "filtered":  filtered,
+            "workload": {
+                "model_name":     model_name,
+                "scenario":       scenario,
+                "accuracy_tier":  accuracy_tier,
+                "framework":      framework,
+                "model_size_gb":  round(model_size_gb, 2),
+                "budget_per_gpu_hr": budget_per_gpu_hr,
+                "min_throughput_tok_per_sec": min_throughput_tok_per_sec,
+            },
+        }

tests/test_api.py

@@ -0,0 +1,218 @@
+"""
+Integration tests for src/api/main.py via FastAPI TestClient.
+"""
+
+from __future__ import annotations
+
+import pytest
+from fastapi.testclient import TestClient
+
+from src.api.main import app
+
+
+@pytest.fixture(scope="module")
+def client() -> TestClient:
+    with TestClient(app) as c:
+        yield c
+
+
+# ---------------------------------------------------------------------------
+# /health
+# ---------------------------------------------------------------------------
+
+class TestHealth:
+    def test_returns_ok(self, client):
+        r = client.get("/health")
+        assert r.status_code == 200
+        assert r.json()["status"] == "ok"
+
+    def test_model_loaded(self, client):
+        r = client.get("/health")
+        assert r.json()["model_loaded"] is True
+
+
+# ---------------------------------------------------------------------------
+# /gpus
+# ---------------------------------------------------------------------------
+
+class TestListGpus:
+    def test_returns_gpu_list(self, client):
+        r = client.get("/gpus")
+        assert r.status_code == 200
+        gpus = r.json()["gpus"]
+        # The spec DB has 8 required in-scope GPUs; len > 0 would pass even if
+        # all but one were dropped.
+        assert len(gpus) >= 8
+
+    def test_mi300x_present(self, client):
+        r = client.get("/gpus")
+        ids = [g["id"] for g in r.json()["gpus"]]
+        assert "mi300x" in ids
+
+    def test_gpu_fields_present(self, client):
+        r = client.get("/gpus")
+        gpu = r.json()["gpus"][0]
+        # Verify keys exist AND values are populated with correct types — a dict
+        # full of None values would pass a bare "in gpu" key-presence check.
+        assert isinstance(gpu["id"], str) and gpu["id"]
+        assert gpu["vendor"] in ("amd", "nvidia")
+        assert isinstance(gpu["vram_gb"], (int, float)) and gpu["vram_gb"] > 0
+        assert isinstance(gpu["in_model_scope"], bool)
+
+
+# ---------------------------------------------------------------------------
+# /models
+# ---------------------------------------------------------------------------
+
+class TestListModels:
+    def test_returns_model_list(self, client):
+        from src.models.predictor import VALID_MODELS
+        r = client.get("/models")
+        assert r.status_code == 200
+        # Compare the complete set — spot-checking 2 models misses regressions
+        # that remove other models (e.g. mixtral-8x7b, llama3.1-405b).
+        assert set(r.json()["models"]) == VALID_MODELS
+
+
+# ---------------------------------------------------------------------------
+# POST /predict
+# ---------------------------------------------------------------------------
+
+class TestPredict:
+    def test_response_has_throughput(self, client):
+        r = client.post("/predict", json={
+            "gpu_id": "h100_sxm",
+            "model_name": "llama2-70b",
+            "scenario": "Offline",
+            "accuracy_tier": "99",
+            "framework": "tensorrt",
+        })
+        assert r.status_code == 200
+        data = r.json()
+        assert data["pred_throughput_tok_per_sec"] >= 0.05 * data["roofline_tput_tok_per_sec"]
+
+    def test_throughput_not_exceeds_roofline(self, client):
+        r = client.post("/predict", json={
+            "gpu_id": "mi300x",
+            "model_name": "llama2-70b",
+        })
+        assert r.status_code == 200
+        data = r.json()
+        assert data["pred_throughput_tok_per_sec"] <= data["roofline_tput_tok_per_sec"] + 1e-2
+
+    def test_unknown_gpu_returns_422(self, client):
+        r = client.post("/predict", json={
+            "gpu_id": "titan_v",
+            "model_name": "llama2-70b",
+        })
+        assert r.status_code == 422
+
+    def test_unknown_model_returns_422(self, client):
+        r = client.post("/predict", json={
+            "gpu_id": "mi300x",
+            "model_name": "gpt5",
+        })
+        assert r.status_code == 422
+
+    def test_invalid_scenario_returns_422(self, client):
+        r = client.post("/predict", json={
+            "gpu_id": "mi300x",
+            "model_name": "gptj",
+            "scenario": "Interactive",
+        })
+        assert r.status_code == 422
+
+    def test_defaults_filled(self, client):
+        r = client.post("/predict", json={
+            "gpu_id": "h200_sxm",
+            "model_name": "gptj",
+        })
+        assert r.status_code == 200
+        data = r.json()
+        assert data["scenario"] == "Offline"
+        assert data["accuracy_tier"] == "99"
+        assert data["framework"] == "vllm"
+
+
+# ---------------------------------------------------------------------------
+# POST /predict/batch
+# ---------------------------------------------------------------------------
+
+class TestPredictBatch:
+    def test_batch_returns_correct_count(self, client):
+        r = client.post("/predict/batch", json=[
+            {"gpu_id": "mi300x", "model_name": "llama2-70b"},
+            {"gpu_id": "h100_sxm", "model_name": "gptj"},
+        ])
+        assert r.status_code == 200
+        results = r.json()
+        assert len(results) == 2
+        # Verify identity, not just count — [mi300x, mi300x] would also have len 2.
+        assert results[0]["gpu_id"] == "mi300x" and results[0]["model_name"] == "llama2-70b"
+        assert results[1]["gpu_id"] == "h100_sxm" and results[1]["model_name"] == "gptj"
+
+    def test_empty_batch_returns_empty(self, client):
+        r = client.post("/predict/batch", json=[])
+        assert r.status_code == 200
+        assert r.json() == []
+
+    def test_batch_over_50_returns_422(self, client):
+        batch = [{"gpu_id": "mi300x", "model_name": "gptj"}] * 51
+        r = client.post("/predict/batch", json=batch)
+        assert r.status_code == 422
+
+    def test_body_too_large_returns_413(self, client):
+        # Send 1 MB + 1 byte — the limit_body_size middleware must intercept
+        # before Pydantic validation (which would return 422, not 413).
+        # The 51-item batch test above only exercises the Pydantic max_length path.
+        from src.api.main import _MAX_BODY_BYTES
+        r = client.post(
+            "/predict/batch",
+            content=b"x" * (_MAX_BODY_BYTES + 1),
+            headers={"Content-Type": "application/json"},
+        )
+        assert r.status_code == 413
+
+
+# ---------------------------------------------------------------------------
+# POST /recommend
+# ---------------------------------------------------------------------------
+
+class TestRecommend:
+    def test_workload_echoed(self, client):
+        r = client.post("/recommend", json={
+            "model_name": "mixtral-8x7b",
+            "scenario": "Server",
+            "accuracy_tier": "base",
+            "framework": "rocm_other",
+        })
+        assert r.status_code == 200
+        wl = r.json()["workload"]
+        # All four echoed fields must round-trip — previously only model_name and
+        # scenario were asserted; accuracy_tier and framework were silently unchecked.
+        assert wl["model_name"] == "mixtral-8x7b"
+        assert wl["scenario"] == "Server"
+        assert wl["accuracy_tier"] == "base"
+        assert wl["framework"] == "rocm_other"
+
+    def test_budget_filter_in_recommend(self, client):
+        r = client.post("/recommend", json={
+            "model_name": "gptj",
+            "budget_per_gpu_hr": 1.0,
+        })
+        assert r.status_code == 200
+        candidates = r.json()["frontier"] + r.json()["dominated"]
+        # L4 ($0.44) and RTX4090 ($0.39) are both under $1.00/hr and fit gptj
+        # (6 GB).  Without this guard the loop below fires zero assertions if
+        # all GPUs exceed the budget (e.g. after a pricing update).
+        assert len(candidates) > 0, (
+            "Expected at least one GPU within $1.00/hr (L4 $0.44, RTX4090 $0.39)"
+        )
+        # All in-scope GPUs have pricing (validated at startup), so
+        # price_per_gpu_hr is never None here — the is-None arm was dead code.
+        for gpu in candidates:
+            assert gpu["price_per_gpu_hr"] <= 1.0
+
+    def test_unknown_model_returns_422(self, client):
+        r = client.post("/recommend", json={"model_name": "invalid_model"})
+        assert r.status_code == 422

tests/test_build_features.py

@@ -0,0 +1,537 @@
+"""
+Tests for src/features/build_features.py.
+
+Strategy: unit-test the pure functions with hand-computed expected values,
+then run a smoke-test of the full pipeline on a synthetic fixture DataFrame
+that mimics the enriched MLPerf schema (avoiding a real parquet dependency).
+"""
+
+from __future__ import annotations
+
+import math
+from pathlib import Path
+from unittest.mock import patch
+
+import pandas as pd
+import pytest
+import yaml
+
+from src.features.build_features import (
+    BYTES_PER_PARAM,
+    MODEL_PARAMS,
+    TIER_TO_PRECISION,
+    _normalize_framework,
+    build_training_df,
+    roofline_ceilings,
+)
+
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+@pytest.fixture()
+def spec_path(tmp_path: Path) -> Path:
+    """Minimal gpu_specs.yaml with two GPUs for pipeline tests."""
+    spec = {
+        "schema_version": "1.0",
+        "gpus": [
+            {
+                "id": "h200_sxm",
+                "name": "NVIDIA H200 SXM",
+                "vendor": "nvidia",
+                "architecture": "hopper",
+                "memory_type": "hbm3e",
+                "vram_gb": 141,
+                "hbm_bandwidth_tbps": 4.8,
+                "peak_tflops": {
+                    "fp32": 66.9,
+                    "bf16": 989.4,
+                    "fp16": 989.4,
+                    "fp8": 1978.9,
+                    "fp6": None,
+                    "fp4": None,
+                    "int8": 3957.8,
+                },
+                "streaming_multiprocessors": 132,
+                "l2_cache_mb": 50,
+                "tdp_w": 700,
+                "spec_confidence": "verified",
+                "in_model_scope": True,
+                "aliases": ["NVIDIA H200-SXM-141GB"],
+            },
+            {
+                "id": "mi300x",
+                "name": "AMD Instinct MI300X",
+                "vendor": "amd",
+                "architecture": "cdna3",
+                "memory_type": "hbm3",
+                "vram_gb": 192,
+                "hbm_bandwidth_tbps": 5.3,
+                "peak_tflops": {
+                    "fp32": 163.4,
+                    "bf16": 1307.4,
+                    "fp16": 1307.4,
+                    "fp8": 2614.9,
+                    "fp6": None,
+                    "fp4": None,
+                    "int8": 2614.9,
+                },
+                "compute_units": 304,
+                "l2_cache_mb": 32,
+                "tdp_w": 750,
+                "spec_confidence": "verified",
+                "in_model_scope": True,
+                "aliases": ["AMD Instinct MI300X 192GB HBM3"],
+            },
+        ],
+    }
+    p = tmp_path / "gpu_specs.yaml"
+    p.write_text(yaml.dump(spec))
+    return p
+
+
+def _make_raw_df(spec_path: Path) -> pd.DataFrame:
+    """Synthetic raw MLPerf DataFrame with the minimal schema expected by build_training_df."""
+    rows = [
+        # H200 SXM, llama2-70b, 99-tier (FP8 selected), Offline, 1 node × 8 GPUs
+        {
+            "round": "v5.0",
+            "division": "closed",
+            "submitter": "TestOrg",
+            "system_name": "8xH200_test",
+            "gpu_name": "NVIDIA H200-SXM-141GB",
+            "num_gpus": 8,
+            "vram_gb": 1128.0,
+            "framework": "TensorRT 10.2.0, CUDA 12.4",
+            "system_type": "datacenter",
+            "hw_status": "available",
+            "benchmark": "llama2-70b-99",
+            "benchmark_base": "llama2-70b",
+            "benchmark_accuracy_tier": "99",
+            "scenario": "Offline",
+            "precision": None,
+            "tokens_per_sample": 294,
+            "throughput_tokens_per_sec": 50000.0,
+            "throughput_tok_per_sec_per_gpu": 6250.0,
+            "result_valid": True,
+            "throughput_samples_per_sec": 170.0,
+            "latency_mean_ms": None,
+            "latency_p99_ms": None,
+            "ttft_mean_ms": None,
+            "ttft_p99_ms": None,
+            "tpot_mean_ms": None,
+            "tpot_p99_ms": None,
+            "log_path": "fake/path",
+        },
+        # MI300X, llama2-70b, 99.9-tier (FP16 selected), Server, 1 node × 8 GPUs
+        {
+            "round": "v5.0",
+            "division": "closed",
+            "submitter": "TestOrg",
+            "system_name": "8xMI300X_test",
+            "gpu_name": "AMD Instinct MI300X 192GB HBM3",
+            "num_gpus": 8,
+            "vram_gb": 1536.0,
+            "framework": "vLLM 0.4.3+rocm614, PyTorch 2.3.0, ROCm 6.1.2",
+            "system_type": "datacenter",
+            "hw_status": "available",
+            "benchmark": "llama2-70b-99.9",
+            "benchmark_base": "llama2-70b",
+            "benchmark_accuracy_tier": "99.9",
+            "scenario": "Server",
+            "precision": None,
+            "tokens_per_sample": 294,
+            "throughput_tokens_per_sec": 24000.0,
+            "throughput_tok_per_sec_per_gpu": 3000.0,
+            "result_valid": True,
+            "throughput_samples_per_sec": 81.6,
+            "latency_mean_ms": None,
+            "latency_p99_ms": None,
+            "ttft_mean_ms": None,
+            "ttft_p99_ms": None,
+            "tpot_mean_ms": None,
+            "tpot_p99_ms": None,
+            "log_path": "fake/path2",
+        },
+        # H200 SXM, llama2-70b, result_valid=False — must be dropped
+        {
+            "round": "v5.0",
+            "division": "closed",
+            "submitter": "TestOrg",
+            "system_name": "8xH200_invalid",
+            "gpu_name": "NVIDIA H200-SXM-141GB",
+            "num_gpus": 8,
+            "vram_gb": 1128.0,
+            "framework": "TensorRT 10.2.0, CUDA 12.4",
+            "system_type": "datacenter",
+            "hw_status": "available",
+            "benchmark": "llama2-70b-99",
+            "benchmark_base": "llama2-70b",
+            "benchmark_accuracy_tier": "99",
+            "scenario": "Offline",
+            "precision": None,
+            "tokens_per_sample": 294,
+            "throughput_tokens_per_sec": 99999.0,
+            "throughput_tok_per_sec_per_gpu": 12499.8,
+            "result_valid": False,
+            "throughput_samples_per_sec": 340.0,
+            "latency_mean_ms": None,
+            "latency_p99_ms": None,
+            "ttft_mean_ms": None,
+            "ttft_p99_ms": None,
+            "tpot_mean_ms": None,
+            "tpot_p99_ms": None,
+            "log_path": "fake/path3",
+        },
+        # Unknown GPU — should be dropped (no spec DB entry → gpu_hbm_bandwidth_tbps is NaN
+        # → caught by the missing-specs guard in build_training_df, not by gpu_in_model_scope)
+        {
+            "round": "v5.0",
+            "division": "closed",
+            "submitter": "TestOrg",
+            "system_name": "8xUnknownGPU",
+            "gpu_name": "SomeFutureGPU 9000",
+            "num_gpus": 8,
+            "vram_gb": 512.0,
+            "framework": "TensorRT 99.0, CUDA 99.0",
+            "system_type": "datacenter",
+            "hw_status": "available",
+            "benchmark": "llama2-70b-99",
+            "benchmark_base": "llama2-70b",
+            "benchmark_accuracy_tier": "99",
+            "scenario": "Offline",
+            "precision": None,
+            "tokens_per_sample": 294,
+            "throughput_tokens_per_sec": 100.0,
+            "throughput_tok_per_sec_per_gpu": 12.5,
+            "result_valid": True,
+            "throughput_samples_per_sec": 0.34,
+            "latency_mean_ms": None,
+            "latency_p99_ms": None,
+            "ttft_mean_ms": None,
+            "ttft_p99_ms": None,
+            "tpot_mean_ms": None,
+            "tpot_p99_ms": None,
+            "log_path": "fake/path4",
+        },
+    ]
+    return pd.DataFrame(rows)
+
+
+# ---------------------------------------------------------------------------
+# roofline_ceilings
+# ---------------------------------------------------------------------------
+
+class TestRooflineCeilings:
+    """Hand-verify the bandwidth and compute ceilings for known GPU–model pairs."""
+
+    def test_h200_llama2_70b_fp16(self):
+        # H200 SXM: HBM BW = 4.8 TB/s, FP16 peak = 989.4 TFLOPS
+        # Llama2-70B: 70B params (dense, total == compute)
+        # BW ceiling  = 4.8e12 / (70e9 × 2)        = 34.29 tok/s
+        # Compute ceil = 989.4e12 / (2 × 70e9)      = 7067.14 tok/s
+        # roofline_tput = compute_ceil (not BW limit)
+        bw, compute, roofline = roofline_ceilings(
+            total_params_b=70.0,
+            compute_params_b=70.0,
+            bytes_per_param=2.0,
+            hbm_bw_tbps=4.8,
+            peak_tflops=989.4,
+        )
+        assert math.isclose(bw, 4.8e12 / (70e9 * 2), rel_tol=1e-6)
+        assert math.isclose(compute, 989.4e12 / (2 * 70e9), rel_tol=1e-6)
+        assert roofline == compute
+
+    def test_mi300x_llama2_70b_fp8(self):
+        # MI300X: HBM BW = 5.3 TB/s, FP8 peak = 2614.9 TFLOPS
+        # BW ceiling  = 5.3e12 / (70e9 × 1)        = 75.71 tok/s
+        # Compute ceil = 2614.9e12 / (2 × 70e9)     = 18677.86 tok/s
+        bw, compute, roofline = roofline_ceilings(
+            total_params_b=70.0,
+            compute_params_b=70.0,
+            bytes_per_param=1.0,
+            hbm_bw_tbps=5.3,
+            peak_tflops=2614.9,
+        )
+        assert math.isclose(bw, 5.3e12 / (70e9 * 1), rel_tol=1e-6)
+        assert math.isclose(compute, 2614.9e12 / (2 * 70e9), rel_tol=1e-6)
+        assert roofline == compute
+
+    def test_moe_uses_active_params_for_compute(self):
+        # Mixtral 8×7B: total=46.7B, active=14.1B
+        # BW ceiling uses total (all weights in HBM): 5.3e12 / (46.7e9 × 2)
+        # Compute ceiling uses active: 1307.4e12 / (2 × 14.1e9)
+        bw, compute, roofline = roofline_ceilings(
+            total_params_b=46.7,
+            compute_params_b=14.1,
+            bytes_per_param=2.0,
+            hbm_bw_tbps=5.3,
+            peak_tflops=1307.4,
+        )
+        assert math.isclose(bw, 5.3e12 / (46.7e9 * 2), rel_tol=1e-6)
+        assert math.isclose(compute, 1307.4e12 / (2 * 14.1e9), rel_tol=1e-6)
+        assert roofline == compute
+
+    def test_roofline_equals_compute_ceiling(self):
+        # roofline_tput is defined as the compute ceiling (a project-level choice:
+        # the model learns the hardware efficiency ratio relative to compute, not BW).
+        # All three cases are compute-bound (compute >> bw) by design — the invariant
+        # is that roofline == compute regardless of what bw_tbps is.
+        for bw_tbps, tflops in [(0.3, 121.6), (4.8, 989.4), (8.0, 2620.0)]:
+            bw, compute, roofline = roofline_ceilings(
+                total_params_b=70.0,
+                compute_params_b=70.0,
+                bytes_per_param=2.0,
+                hbm_bw_tbps=bw_tbps,
+                peak_tflops=tflops,
+            )
+            assert roofline == compute, (
+                f"roofline should be compute ceiling, got roofline={roofline} compute={compute}"
+            )
+
+    def test_bandwidth_ceiling_scales_with_bytes_per_param(self):
+        # Halving bytes_per_param (FP16→FP8) doubles the BW ceiling.
+        bw_fp16, _, _ = roofline_ceilings(70.0, 70.0, 2.0, 4.8, 989.4)
+        bw_fp8, _, _ = roofline_ceilings(70.0, 70.0, 1.0, 4.8, 989.4)
+        assert math.isclose(bw_fp8, 2 * bw_fp16, rel_tol=1e-9)
+
+
+# ---------------------------------------------------------------------------
+# Reference tables
+# ---------------------------------------------------------------------------
+
+class TestReferenceTables:
+    def test_all_model_params_have_positive_values(self):
+        for name, (total, compute) in MODEL_PARAMS.items():
+            assert total > 0, f"{name}: total_params_b must be > 0"
+            assert compute > 0, f"{name}: compute_params_b must be > 0"
+            assert compute <= total, f"{name}: compute_params_b must be ≤ total_params_b"
+
+    def test_mixtral_moe_compute_lt_total(self):
+        total, compute = MODEL_PARAMS["mixtral-8x7b"]
+        assert compute < total
+
+    def test_dense_models_compute_equals_total(self):
+        for name in ("llama2-70b", "llama3.1-405b", "gptj"):
+            total, compute = MODEL_PARAMS[name]
+            assert total == compute, f"{name}: dense model should have compute == total"
+
+    def test_bytes_per_param_ordering(self):
+        # FP4 < FP8 < FP16 == BF16 < FP32
+        assert BYTES_PER_PARAM["fp4"] < BYTES_PER_PARAM["fp8"]
+        assert BYTES_PER_PARAM["fp8"] < BYTES_PER_PARAM["fp16"]
+        assert BYTES_PER_PARAM["fp16"] == BYTES_PER_PARAM["bf16"]
+        assert BYTES_PER_PARAM["bf16"] < BYTES_PER_PARAM["fp32"]
+
+    def test_tier_to_precision_coverage(self):
+        for tier in ("99.9", "99", "base"):
+            assert tier in TIER_TO_PRECISION
+            assert TIER_TO_PRECISION[tier] in BYTES_PER_PARAM
+
+
+# ---------------------------------------------------------------------------
+# _normalize_framework
+# ---------------------------------------------------------------------------
+
+class TestNormalizeFramework:
+    @pytest.mark.parametrize("raw,expected", [
+        ("TensorRT 10.2.0, CUDA 12.4",          "tensorrt"),
+        ("TensorRT 9.3.0, CUDA 12.2",           "tensorrt"),
+        ("vLLM 0.4.3+rocm614, PyTorch 2.3.0",   "vllm"),
+        ("vLLM 0.9.0, Pytorch 2.7, ROCm 6.4",   "vllm"),
+        ("PyTorch 2.9.1+git, ROCm 7.0.0",       "rocm_other"),
+        ("ROCm 6.3.1",                           "rocm_other"),
+        ("Mango LLMBoost, ROCm 6.12",            "rocm_other"),
+        ("SomeUnknownFramework 1.0",             "other"),
+        ("",                                     "other"),
+        (None,                                   "unknown"),
+        (123,                                    "unknown"),
+    ])
+    def test_classification(self, raw, expected):
+        assert _normalize_framework(raw) == expected
+
+
+# ---------------------------------------------------------------------------
+# build_training_df — integration smoke test
+# ---------------------------------------------------------------------------
+
+class TestBuildTrainingDf:
+    def test_filters_invalid_and_missing_spec_rows(self, spec_path, tmp_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        # invalid row (result_valid=False) and unknown-GPU row (no spec → NaN HBM BW) are dropped
+        assert len(feat) == 2
+        # Verify the correct 2 rows survived — a filter bug that kept the wrong pair
+        # (e.g. result_valid=False row + unknown-GPU row) would still give len == 2.
+        assert set(feat["canonical_gpu_id"]) == {"h200_sxm", "mi300x"}
+
+    def test_new_columns_present(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        expected_cols = [
+            "model_total_params_b", "model_compute_params_b",
+            "bytes_per_param", "selected_precision", "peak_tflops_selected",
+            "bandwidth_ceiling_tok_per_sec", "compute_ceiling_tok_per_sec",
+            "roofline_tput", "efficiency_ratio", "model_size_gb",
+            "model_to_vram_ratio", "framework_family",
+            "nvidia_arch_gen", "amd_arch_gen", "vendor_is_amd",
+            "mlperf_round_num", "is_base_tier",
+        ]
+        for col in expected_cols:
+            assert col in feat.columns, f"Missing column: {col}"
+
+    def test_mlperf_round_num_mapping(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        # _make_raw_df uses "round": "v5.0" → ROUND_ORDINAL["v5.0"] == 2
+        feat = build_training_df(raw, spec_path=spec_path)
+        assert (feat["mlperf_round_num"] == 2).all()
+
+    def test_is_base_tier_encoding(self, spec_path):
+        raw = _make_raw_df(spec_path).copy()
+        # Inject a base-tier row for MI300X.
+        base_row = raw[raw["gpu_name"] == "AMD Instinct MI300X 192GB HBM3"].iloc[0].copy()
+        base_row["benchmark_accuracy_tier"] = "base"
+        base_row["benchmark"] = "llama2-70b-base"
+        raw = pd.concat([raw, base_row.to_frame().T], ignore_index=True)
+        feat = build_training_df(raw, spec_path=spec_path)
+        base_rows = feat[feat["benchmark_accuracy_tier"] == "base"]
+        non_base_rows = feat[feat["benchmark_accuracy_tier"] != "base"]
+        assert (base_rows["is_base_tier"] == 1).all()
+        assert (non_base_rows["is_base_tier"] == 0).all()
+
+    def test_mlperf_round_num_unknown_round_is_nan(self, spec_path):
+        raw = _make_raw_df(spec_path).copy()
+        raw["round"] = "v99.0"
+        feat = build_training_df(raw, spec_path=spec_path)
+        assert feat["mlperf_round_num"].isna().all()
+
+    def test_roofline_equals_compute_ceiling(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        pd.testing.assert_series_equal(
+            feat["roofline_tput"],
+            feat["compute_ceiling_tok_per_sec"],
+            check_names=False,
+        )
+
+    def test_h200_fp8_precision_selection(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        h200_row = feat[feat["canonical_gpu_id"] == "h200_sxm"].iloc[0]
+        assert h200_row["selected_precision"] == "fp8"
+        assert h200_row["bytes_per_param"] == BYTES_PER_PARAM["fp8"]
+        # Pin the actual TFLOPS value — catches bugs in the vectorised peak-TFLOPS
+        # selection (e.g. accidentally returning fp16=989.4 instead of fp8=1978.9).
+        assert h200_row["peak_tflops_selected"] == pytest.approx(1978.9)
+
+    def test_amd_tier_99_9_uses_fp8(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        mi_row = feat[feat["canonical_gpu_id"] == "mi300x"].iloc[0]
+        # AMD 99.9 tier → fp8 override (not fp16).  MI300X fp8 = 2614.9 TFLOPS.
+        assert mi_row["selected_precision"] == "fp8"
+        assert mi_row["bytes_per_param"] == BYTES_PER_PARAM["fp8"]
+        assert mi_row["peak_tflops_selected"] == pytest.approx(2614.9)
+
+    def test_nvidia_tier_99_9_still_uses_fp16(self, spec_path):
+        raw = _make_raw_df(spec_path).copy()
+        # Swap the H200 row to tier 99.9 — NVIDIA must NOT be overridden to fp8.
+        mask = raw["gpu_name"] == "NVIDIA H200-SXM-141GB"
+        raw.loc[mask, "benchmark_accuracy_tier"] = "99.9"
+        raw.loc[mask, "benchmark"] = "llama2-70b-99.9"
+        feat = build_training_df(raw, spec_path=spec_path)
+        h200 = feat[feat["canonical_gpu_id"] == "h200_sxm"].iloc[0]
+        assert h200["selected_precision"] == "fp16"
+        assert h200["bytes_per_param"] == BYTES_PER_PARAM["fp16"]
+        assert h200["peak_tflops_selected"] == pytest.approx(989.4)
+
+    def test_vendor_is_amd_flag(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        assert feat.loc[feat["canonical_gpu_id"] == "mi300x",  "vendor_is_amd"].iloc[0] == 1
+        assert feat.loc[feat["canonical_gpu_id"] == "h200_sxm", "vendor_is_amd"].iloc[0] == 0
+
+    def test_framework_family_assigned(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        h200 = feat.loc[feat["canonical_gpu_id"] == "h200_sxm", "framework_family"].iloc[0]
+        mi = feat.loc[feat["canonical_gpu_id"] == "mi300x", "framework_family"].iloc[0]
+        assert h200 == "tensorrt"
+        assert mi == "vllm"
+
+    def test_no_nulls_in_feature_columns(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        feature_cols = [
+            "model_total_params_b", "model_compute_params_b", "bytes_per_param",
+            "bandwidth_ceiling_tok_per_sec", "compute_ceiling_tok_per_sec",
+            "roofline_tput", "efficiency_ratio", "model_size_gb",
+            "model_to_vram_ratio", "vendor_is_amd",
+        ]
+        for col in feature_cols:
+            assert feat[col].notna().all(), f"Nulls found in {col}"
+
+    def test_vendor_arch_gen_ordinals(self, spec_path):
+        raw = _make_raw_df(spec_path)
+        feat = build_training_df(raw, spec_path=spec_path)
+        h200 = feat[feat["canonical_gpu_id"] == "h200_sxm"].iloc[0]
+        mi300 = feat[feat["canonical_gpu_id"] == "mi300x"].iloc[0]
+        # H200 is hopper (nvidia_arch_gen=3); amd_arch_gen must be NaN
+        assert h200["nvidia_arch_gen"] == 3
+        assert pd.isna(h200["amd_arch_gen"])
+        # MI300X is cdna3 (amd_arch_gen=1); nvidia_arch_gen must be NaN
+        assert mi300["amd_arch_gen"] == 1
+        assert pd.isna(mi300["nvidia_arch_gen"])
+
+    def test_unknown_benchmark_dropped(self, spec_path):
+        raw = _make_raw_df(spec_path).copy()
+        # Inject a row with a benchmark_base not in MODEL_PARAMS
+        extra = raw.iloc[0].copy()
+        extra["benchmark_base"] = "future-model-1t"
+        extra["result_valid"] = True
+        raw = pd.concat([raw, extra.to_frame().T], ignore_index=True)
+        feat = build_training_df(raw, spec_path=spec_path)
+        assert "future-model-1t" not in feat["benchmark_base"].values
+        # Guard against a catastrophic bug that drops ALL rows — an empty
+        # DataFrame would also satisfy the assertion above.
+        assert len(feat) == 2
+
+    def test_rows_with_missing_gpu_specs_are_dropped(self, spec_path):
+        # Patch enrich_df in the build_features namespace (where it was imported).
+        import src.features.build_features as bf_mod
+        from src.data.gpu_spec_db import enrich_df as real_enrich
+
+        def patched_enrich(df, **kwargs):
+            enriched = real_enrich(df, **kwargs)
+            mask = enriched["canonical_gpu_id"] == "h200_sxm"
+            enriched.loc[mask, "gpu_hbm_bandwidth_tbps"] = float("nan")
+            return enriched
+
+        raw = _make_raw_df(spec_path)
+        with patch.object(bf_mod, "enrich_df", patched_enrich):
+            feat = build_training_df(raw, spec_path=spec_path)
+
+        # H200 row dropped; only MI300X survives
+        assert len(feat) == 1
+        assert feat["canonical_gpu_id"].iloc[0] == "mi300x"
+
+    def test_violation_rows_logged_not_dropped(self, spec_path, caplog):
+        import logging
+        # When throughput > roofline_tput, rows are kept (not dropped) AND a warning
+        # is emitted.  The test name claims both; both are now asserted.
+        raw = _make_raw_df(spec_path).copy()
+        # Force throughput_tok_per_sec_per_gpu to a huge value to trigger the path.
+        h200_mask = raw["gpu_name"] == "NVIDIA H200-SXM-141GB"
+        raw.loc[h200_mask, "throughput_tok_per_sec_per_gpu"] = 1_000_000.0
+        with caplog.at_level(logging.WARNING, logger="src.features.build_features"):
+            feat = build_training_df(raw, spec_path=spec_path)
+        # Rows are kept, not filtered.
+        assert len(feat) == 2
+        assert (feat.loc[feat["canonical_gpu_id"] == "h200_sxm", "efficiency_ratio"] > 1).all()
+        # Violations must also be logged (the "logged" half of the test name).
+        assert any(
+            "compute ceiling" in r.getMessage().lower()
+            for r in caplog.records
+            if r.levelno == logging.WARNING
+        ), "Expected a WARNING about throughput > compute ceiling but none was emitted"

tests/test_gpu_spec_db.py

@@ -0,0 +1,489 @@
+"""
+Tests for src/data/gpu_spec_db.py.
+
+All tests that require the real data/gpu_specs.yaml use the actual file —
+it is curated reference data, not generated output, so reading it in tests
+is appropriate.  No MLPerf repo data required.
+"""
+
+import logging
+from pathlib import Path
+import pandas as pd
+import pytest
+import yaml
+
+from src.data.gpu_spec_db import (
+    _MAX_SPEC_BYTES,
+    SPEC_COLUMNS,
+    _build_alias_index,
+    _is_heterogeneous,
+    _strip_suffixes,
+    enrich_df,
+    load_specs,
+    normalize_gpu_name,
+)
+
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+@pytest.fixture(scope="module")
+def specs():
+    # Shallow copy so test mutations (e.g. specs.append(...)) do not corrupt
+    # the load_specs() lru_cache, which is shared across the whole test process.
+    return list(load_specs())
+
+
+@pytest.fixture(scope="module")
+def alias_index(specs):
+    return _build_alias_index(specs)
+
+
+# ---------------------------------------------------------------------------
+# YAML integrity
+# ---------------------------------------------------------------------------
+
+class TestYamlIntegrity:
+    REQUIRED_FIELDS = [
+        "id", "name", "vendor", "architecture", "memory_type",
+        "vram_gb", "hbm_bandwidth_tbps", "peak_tflops",
+        "in_model_scope", "spec_confidence", "aliases",
+    ]
+    # Precisions every in-scope GPU supports (A100 Ampere has no FP8 natively,
+    # so fp8 is intentionally absent here; covered by a separate test below).
+    REQUIRED_TFLOPS = ["fp32", "bf16", "fp16"]
+
+    def test_all_ids_unique(self, specs):
+        ids = [s["id"] for s in specs]
+        assert len(ids) == len(set(ids)), "Duplicate GPU ids in spec DB"
+
+    def test_required_fields_present(self, specs):
+        for spec in specs:
+            for field in self.REQUIRED_FIELDS:
+                assert field in spec, (
+                    f"GPU {spec.get('id')!r} missing required field {field!r}"
+                )
+
+    def test_all_gpus_have_valid_spec_confidence(self, specs):
+        # Previously named "test_in_scope_gpus_have_verified_or_estimated_confidence"
+        # but the loop had no in_model_scope guard — it checked all GPUs.
+        # The name now matches the actual behavior.
+        valid = {"verified", "estimated"}
+        for spec in specs:
+            assert spec["spec_confidence"] in valid, (
+                f"{spec['id']}: spec_confidence must be 'verified' or 'estimated', "
+                f"got {spec['spec_confidence']!r}"
+            )
+
+    def test_in_scope_gpus_have_bandwidth_and_bf16(self, specs):
+        """In-scope GPUs must have positive bandwidth and BF16 TFLOPS.
+
+        Zero or None would silently corrupt the roofline model:
+        zero bandwidth → division-by-zero in arithmetic intensity calculation;
+        zero BF16 TFLOPS → roofline ceiling of 0, predicting 0 throughput always.
+        """
+        for spec in specs:
+            if not spec["in_model_scope"]:
+                continue
+            bw = spec["hbm_bandwidth_tbps"]
+            assert bw is not None and bw > 0, (
+                f"{spec['id']}: hbm_bandwidth_tbps must be > 0, got {bw!r}"
+            )
+            bf16 = spec["peak_tflops"].get("bf16")
+            assert bf16 is not None and bf16 > 0, (
+                f"{spec['id']}: peak_tflops.bf16 must be > 0, got {bf16!r}"
+            )
+
+    def test_in_scope_gpus_have_positive_required_tflops(self, specs):
+        """REQUIRED_TFLOPS must be positive for every in-scope GPU.
+
+        Catches accidentally-zero values that would silently make the roofline
+        predict 0 throughput for those precisions.
+        """
+        for spec in specs:
+            if not spec["in_model_scope"]:
+                continue
+            pt = spec.get("peak_tflops") or {}
+            for key in self.REQUIRED_TFLOPS:
+                val = pt.get(key)
+                assert val is not None and val > 0, (
+                    f"{spec['id']}: peak_tflops.{key} must be > 0 for in-scope GPU, "
+                    f"got {val!r}"
+                )
+
+    def test_non_null_tflops_are_positive(self, specs):
+        """Any non-null peak_tflops value (including fp8, fp6, fp4) must be > 0.
+
+        Null means unsupported (e.g. A100 fp8 is null — correct).
+        A non-null zero would silently corrupt roofline calculations.
+        """
+        for spec in specs:
+            pt = spec.get("peak_tflops") or {}
+            for key, val in pt.items():
+                if val is not None:
+                    assert val > 0, (
+                        f"{spec['id']}: peak_tflops.{key} is non-null but not > 0: {val!r}"
+                    )
+
+    def test_no_conflicting_aliases_across_gpus(self, specs):
+        """No alias should map to two different GPU ids after case-folding."""
+        seen: dict[str, str] = {}
+        for spec in specs:
+            for alias in spec.get("aliases", []):
+                key = alias.lower()
+                if key in seen and seen[key] != spec["id"]:
+                    pytest.fail(
+                        f"Alias {alias!r} maps to both {seen[key]!r} and {spec['id']!r}"
+                    )
+                seen[key] = spec["id"]
+
+    def test_amd_gpus_have_compute_units(self, specs):
+        for spec in specs:
+            if spec["vendor"] == "amd":
+                assert "compute_units" in spec, (
+                    f"{spec['id']}: AMD GPU missing compute_units"
+                )
+
+    def test_nvidia_gpus_have_streaming_multiprocessors(self, specs):
+        for spec in specs:
+            if spec["vendor"] == "nvidia":
+                assert "streaming_multiprocessors" in spec, (
+                    f"{spec['id']}: NVIDIA GPU missing streaming_multiprocessors"
+                )
+
+    def test_in_scope_gpu_ids_present(self, specs):
+        """The eight v1-scope GPUs must all be present and in-scope."""
+        required = {"mi300x", "mi325x", "mi355x", "h100_sxm", "h200_sxm",
+                    "a100_sxm_80gb", "l4", "rtx4090"}
+        in_scope_ids = {s["id"] for s in specs if s["in_model_scope"]}
+        missing = required - in_scope_ids
+        assert not missing, f"Required GPUs missing from spec DB: {missing}"
+
+
+# ---------------------------------------------------------------------------
+# Security tests: load_specs guards
+# ---------------------------------------------------------------------------
+
+class TestLoadSpecsSecurity:
+    def _minimal_yaml(self, tmp_path, content: dict) -> Path:
+        p = tmp_path / "specs.yaml"
+        p.write_text(yaml.dump(content), encoding="utf-8")
+        return p
+
+    def test_rejects_symlink(self, tmp_path):
+        real = self._minimal_yaml(tmp_path, {"gpus": []})
+        link = tmp_path / "link.yaml"
+        link.symlink_to(real)
+        with pytest.raises(ValueError, match="symlink"):
+            load_specs(link)
+
+    def test_rejects_symlink_to_outside(self, tmp_path):
+        link = tmp_path / "escape.yaml"
+        link.symlink_to("/etc/hosts")
+        with pytest.raises(ValueError, match="symlink"):
+            load_specs(link)
+
+    def test_rejects_oversized_file(self, tmp_path):
+        p = tmp_path / "huge.yaml"
+        p.write_bytes(b"# " + b"x" * _MAX_SPEC_BYTES)
+        with pytest.raises(ValueError, match="too large"):
+            load_specs(p)
+
+    def test_missing_file_raises_file_not_found(self, tmp_path):
+        with pytest.raises(FileNotFoundError):
+            load_specs(tmp_path / "nonexistent.yaml")
+
+    def test_missing_gpus_key_raises_value_error(self, tmp_path):
+        p = tmp_path / "bad.yaml"
+        p.write_text(yaml.dump({"schema_version": "1.0"}), encoding="utf-8")
+        with pytest.raises(ValueError, match="gpus"):
+            load_specs(p)
+
+    def test_empty_yaml_raises_value_error(self, tmp_path):
+        p = tmp_path / "empty.yaml"
+        p.write_text("", encoding="utf-8")
+        with pytest.raises(ValueError, match="gpus"):
+            load_specs(p)
+
+    def test_valid_minimal_yaml_loads(self, tmp_path):
+        p = self._minimal_yaml(tmp_path, {"gpus": [{"id": "test"}]})
+        result = load_specs(p)
+        assert result == [{"id": "test"}]
+
+
+class TestLogInjection:
+    def test_newline_in_gpu_name_does_not_inject_log_line(self, caplog):
+        """A GPU name containing \\n must not create a spurious second log line."""
+        injected = "NVIDIA H100\n[CRITICAL] Forged log entry"
+        df = pd.DataFrame([{"gpu_name": injected}])
+        with caplog.at_level(logging.DEBUG, logger="src.data.gpu_spec_db"):
+            enrich_df(df)
+        # Every record produced must have been created by the logger itself —
+        # the injected newline must appear repr-quoted, not as a raw newline.
+        for record in caplog.records:
+            assert "\n[CRITICAL]" not in record.getMessage(), (
+                "Log injection: raw newline from GPU name appeared in log output"
+            )
+
+    def test_ansi_escape_in_gpu_name_repr_quoted(self, caplog):
+        """ANSI escape sequences in GPU names must be repr-quoted in debug logs."""
+        ansi_name = "GPU\x1b[31mRED\x1b[0m"
+        df = pd.DataFrame([{"gpu_name": ansi_name}])
+        with caplog.at_level(logging.DEBUG, logger="src.data.gpu_spec_db"):
+            enrich_df(df)
+        raw_ansi = "\x1b[31m"
+        for record in caplog.records:
+            assert raw_ansi not in record.getMessage(), (
+                "ANSI escape from GPU name appeared unescaped in log output"
+            )
+
+
+# ---------------------------------------------------------------------------
+# _strip_suffixes
+# ---------------------------------------------------------------------------
+
+class TestStripSuffixes:
+    @pytest.mark.parametrize("raw,expected", [
+        ("AMD Instinct MI355X 288GB HBM3e (x87)",         "AMD Instinct MI355X 288GB HBM3e"),
+        ("AMD Instinct MI300X 192GB HBM3 (x8)",           "AMD Instinct MI300X 192GB HBM3"),
+        ("AMD Instinct MI355X 288GB HBM3e (Power Cap 1000 W)", "AMD Instinct MI355X 288GB HBM3e"),
+        ("NVIDIA H100-SXM-80GB",                           "NVIDIA H100-SXM-80GB"),
+        ("AMD Instinct MI300X 192GB HBM3",                 "AMD Instinct MI300X 192GB HBM3"),
+    ])
+    def test_strips_known_patterns(self, raw, expected):
+        assert _strip_suffixes(raw) == expected
+
+
+# ---------------------------------------------------------------------------
+# _is_heterogeneous
+# ---------------------------------------------------------------------------
+
+class TestIsHeterogeneous:
+    def test_mixed_and(self):
+        assert _is_heterogeneous(
+            "AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8)"
+        )
+
+    def test_mixed_comma_count(self):
+        assert _is_heterogeneous(
+            "AMD Instinct MI300X 192GB HBM3 (x32), AMD Instinct MI325X 256GB HBM3e (x16)"
+        )
+
+    def test_single_gpu_not_heterogeneous(self):
+        assert not _is_heterogeneous("AMD Instinct MI300X 192GB HBM3")
+
+    def test_comma_without_count_not_flagged(self):
+        # A name with a comma but no "(xN)" is not a multi-SKU string.
+        assert not _is_heterogeneous("NVIDIA H100-SXM-80GB, Rev 2")
+
+
+# ---------------------------------------------------------------------------
+# normalize_gpu_name
+# ---------------------------------------------------------------------------
+
+class TestNormalizeGpuName:
+    # Known exact aliases from the real MLPerf corpus
+    @pytest.mark.parametrize("raw,expected_id", [
+        ("AMD Instinct MI300X 192GB HBM3",           "mi300x"),
+        ("AMD Instinct MI300X-NPS1-SPX-192GB-750W",  "mi300x"),
+        ("AMD MI300X-NPS1-SPX-192GB-750W",           "mi300x"),
+        ("AMD Instinct MI325X 256GB HBM3E",          "mi325x"),  # uppercase E
+        ("AMD Instinct MI325X 256GB HBM3e",          "mi325x"),  # lowercase e
+        ("AMD Instinct MI355X 288GB HBM3e",          "mi355x"),
+        ("AMD Instinct MI350X 288GB HBM3e",          "mi350x"),
+        ("NVIDIA H100-SXM-80GB",                     "h100_sxm"),
+        ("Virtualized NVIDIA H100-SXM-80GB",         "h100_sxm"),
+        ("NVIDIA H200-SXM-141GB",                    "h200_sxm"),
+        ("NVIDIA H200-SXM-141GB-CTS",                "h200_sxm"),
+        ("Virtualized NVIDIA H200-SXM-141GB",        "h200_sxm"),
+        ("NVIDIA H200-NVL-141GB",                    "h200_nvl"),
+        ("NVIDIA H100-PCIe-80GB",                    "h100_pcie"),
+        ("NVIDIA H100-NVL-94GB",                     "h100_nvl"),
+        ("NVIDIA GH200 Grace Hopper Superchip 96GB",  "gh200_96gb"),
+        ("NVIDIA GH200 Grace Hopper Superchip 144GB", "gh200_144gb"),
+        ("NVIDIA L40S",                              "l40s"),
+        ("NVIDIA B200-SXM-180GB",                    "b200_sxm"),
+        ("NVIDIA B300-SXM-270GB",                    "b300_sxm"),
+        ("NVIDIA GB200",                             "gb200"),
+        ("NVIDIA GB300",                             "gb300"),
+    ])
+    def test_known_aliases(self, raw, expected_id, specs, alias_index):
+        result = normalize_gpu_name(raw, specs, _index=alias_index)
+        assert result == expected_id, f"Expected {expected_id!r}, got {result!r} for {raw!r}"
+
+    def test_count_suffix_stripped(self, specs, alias_index):
+        """'(x87)' suffix must be stripped before alias lookup."""
+        result = normalize_gpu_name(
+            "AMD Instinct MI355X 288GB HBM3e (x87)", specs, _index=alias_index
+        )
+        assert result == "mi355x"
+
+    def test_power_suffix_stripped(self, specs, alias_index):
+        result = normalize_gpu_name(
+            "AMD Instinct MI355X 288GB HBM3e (Power Cap 1000 W)",
+            specs, _index=alias_index,
+        )
+        assert result == "mi355x"
+
+    def test_heterogeneous_returns_none(self, specs, alias_index):
+        result = normalize_gpu_name(
+            "AMD Instinct MI300X 192GB HBM3 (x8) and AMD Instinct MI325X 256GB HBM3e (x8)",
+            specs, _index=alias_index,
+        )
+        assert result is None
+
+    def test_na_returns_none(self, specs, alias_index):
+        assert normalize_gpu_name("N/A", specs, _index=alias_index) is None
+
+    def test_none_input_returns_none(self, specs, alias_index):
+        assert normalize_gpu_name(None, specs, _index=alias_index) is None
+
+    def test_unknown_gpu_returns_none(self, specs, alias_index):
+        assert normalize_gpu_name("NVIDIA H9000-FAKE", specs, _index=alias_index) is None
+
+
+# ---------------------------------------------------------------------------
+# enrich_df
+# ---------------------------------------------------------------------------
+
+class TestEnrichDf:
+    def _minimal_df(self, gpu_name: str) -> pd.DataFrame:
+        return pd.DataFrame([{"gpu_name": gpu_name}])
+
+    def test_adds_all_spec_columns(self):
+        df = self._minimal_df("AMD Instinct MI300X 192GB HBM3")
+        enriched = enrich_df(df)
+        for col in SPEC_COLUMNS:
+            assert col in enriched.columns, f"Missing column {col!r}"
+
+    def test_known_gpu_values_populated(self):
+        df = self._minimal_df("NVIDIA H100-SXM-80GB")
+        enriched = enrich_df(df)
+        row = enriched.iloc[0]
+        assert row["canonical_gpu_id"] == "h100_sxm"
+        assert row["gpu_vendor"] == "nvidia"
+        assert row["gpu_architecture"] == "hopper"
+        assert row["gpu_vram_gb"] == pytest.approx(80.0)
+        assert row["gpu_hbm_bandwidth_tbps"] == pytest.approx(3.35)
+        assert row["gpu_peak_bf16_tflops"] == pytest.approx(989.4)
+        assert row["gpu_in_model_scope"] == True  # noqa: E712 — numpy.bool_
+
+    def test_unknown_gpu_yields_null_spec_columns(self):
+        df = self._minimal_df("NVIDIA FAKE-9000")
+        enriched = enrich_df(df)
+        assert enriched.iloc[0]["canonical_gpu_id"] is None
+        assert pd.isna(enriched.iloc[0]["gpu_vram_gb"])
+
+    def test_heterogeneous_gpu_yields_null(self):
+        df = self._minimal_df(
+            "AMD Instinct MI300X 192GB HBM3 (x8) "
+            "and AMD Instinct MI325X 256GB HBM3e (x8)"
+        )
+        enriched = enrich_df(df)
+        assert enriched.iloc[0]["canonical_gpu_id"] is None
+
+    def test_original_df_not_mutated(self):
+        df = self._minimal_df("NVIDIA H200-SXM-141GB")
+        original_cols = set(df.columns)
+        enrich_df(df)
+        assert set(df.columns) == original_cols
+
+    def test_multi_row_df(self):
+        df = pd.DataFrame([
+            {"gpu_name": "AMD Instinct MI300X 192GB HBM3"},
+            {"gpu_name": "NVIDIA H200-SXM-141GB"},
+            {"gpu_name": "N/A"},
+        ])
+        enriched = enrich_df(df)
+        assert len(enriched) == 3
+        assert enriched.iloc[0]["canonical_gpu_id"] == "mi300x"
+        assert enriched.iloc[1]["canonical_gpu_id"] == "h200_sxm"
+        assert pd.isna(enriched.iloc[2]["canonical_gpu_id"])
+
+    def test_out_of_scope_gpu_flagged(self):
+        df = self._minimal_df("NVIDIA B200-SXM-180GB")
+        enriched = enrich_df(df)
+        assert enriched.iloc[0]["gpu_in_model_scope"] == False  # noqa: E712 — numpy.bool_
+
+    def test_cu_sm_count_unified(self):
+        """AMD CUs and NVIDIA SMs both appear under gpu_cu_sm_count."""
+        amd = enrich_df(self._minimal_df("AMD Instinct MI300X 192GB HBM3")).iloc[0]
+        nv = enrich_df(self._minimal_df("NVIDIA H100-SXM-80GB")).iloc[0]
+        assert amd["gpu_cu_sm_count"] == 304
+        assert nv["gpu_cu_sm_count"] == 132
+
+    def test_duplicate_gpu_name_rows_all_enriched(self):
+        """All rows sharing the same gpu_name must get spec columns populated.
+
+        Correctness check: the deduplication must broadcast results to all rows,
+        not just the first occurrence.
+        """
+        df = pd.DataFrame([{"gpu_name": "NVIDIA H200-SXM-141GB"}] * 50)
+        enriched = enrich_df(df)
+        assert len(enriched) == 50
+        assert (enriched["canonical_gpu_id"] == "h200_sxm").all()
+        assert enriched["gpu_hbm_bandwidth_tbps"].notna().all()
+
+    def test_vram_conflict_warning_logged(self, caplog):
+        """enrich_df warns when parser vram_gb differs from spec DB gpu_vram_gb.
+
+        Simulates a multi-GPU system where the submitter reported total VRAM
+        (8 × 80 GB = 640 GB) rather than per-GPU capacity (80 GB).
+        Covers lines 262-282 in gpu_spec_db.py.
+        """
+        df = pd.DataFrame([{
+            "gpu_name": "NVIDIA H100-SXM-80GB",
+            "vram_gb":  640.0,   # 8-GPU total — spec DB stores 80 GB per-GPU
+            "num_gpus": 8,
+        }])
+        with caplog.at_level(logging.WARNING, logger="src.data.gpu_spec_db"):
+            enriched = enrich_df(df)
+        assert enriched.iloc[0]["gpu_vram_gb"] == pytest.approx(80.0)
+        assert any(
+            "vram" in r.getMessage().lower()
+            for r in caplog.records
+            if r.levelno == logging.WARNING
+        )
+
+    def test_no_vram_conflict_no_warning(self, caplog):
+        """No warning when vram_gb already matches the spec DB value."""
+        df = pd.DataFrame([{
+            "gpu_name": "NVIDIA H100-SXM-80GB",
+            "vram_gb":  80.0,   # matches spec DB
+            "num_gpus": 1,
+        }])
+        with caplog.at_level(logging.WARNING, logger="src.data.gpu_spec_db"):
+            enrich_df(df)
+        assert not any(
+            "vram" in r.getMessage().lower()
+            for r in caplog.records
+            if r.levelno == logging.WARNING
+        )
+
+
+# ---------------------------------------------------------------------------
+# _build_alias_index: conflicting alias warning
+# ---------------------------------------------------------------------------
+
+class TestBuildAliasIndex:
+    def test_conflicting_alias_warns(self, caplog):
+        """Same lowercase alias in two different specs emits a warning (line 122)."""
+        specs = [
+            {"id": "gpu_a", "aliases": ["Shared Alias", "only_a"]},
+            {"id": "gpu_b", "aliases": ["shared alias", "only_b"]},  # same after lower()
+        ]
+        with caplog.at_level(logging.WARNING, logger="src.data.gpu_spec_db"):
+            index = _build_alias_index(specs)
+        assert any("conflict" in r.getMessage().lower() for r in caplog.records)
+        # Last writer wins — gpu_b is processed second
+        assert index["shared alias"]["id"] == "gpu_b"
+
+    def test_same_alias_same_gpu_no_warning(self, caplog):
+        """Duplicate alias within one GPU (case variants) must not warn."""
+        specs = [
+            {"id": "gpu_a", "aliases": ["HBM3E", "hbm3e"]},  # same after lower()
+        ]
+        with caplog.at_level(logging.WARNING, logger="src.data.gpu_spec_db"):
+            _build_alias_index(specs)
+        assert not any(r.levelno == logging.WARNING for r in caplog.records)

tests/test_mlperf_parser.py

@@ -0,0 +1,1111 @@
+"""
+Unit tests for src/data/mlperf_parser.py.
+
+All tests use synthetic, in-memory fixture data — no real MLPerf repos required.
+Fixtures mirror the exact directory layout and file content format of
+MLPerf Inference v4.1–v6.0 closed-division submissions.
+"""
+
+import json
+import logging
+import sys
+from pathlib import Path
+
+import pandas as pd
+import pytest
+
+from src.data.mlperf_parser import (
+    LLM_BENCHMARKS,
+    TOKENS_PER_SAMPLE,
+    _MAX_FILE_BYTES,
+    _accuracy_tier,
+    _base_benchmark,
+    _extract_precision,
+    _find_best_run,
+    _parse_log_summary,
+    _parse_system_json,
+    _parse_vram_gb,
+    _run_number,
+    _safe_read_text,
+    main,
+    parse_repo,
+    parse_repos,
+)
+
+
+# ---------------------------------------------------------------------------
+# Fixture content (mirrors real MLPerf file content)
+# ---------------------------------------------------------------------------
+
+SYSTEM_JSON_NVIDIA = {
+    "system_name": "H100-SXM5-80GBx8_TRT-LLM",
+    # Real MLPerf H100 SXM submissions use "H100-SXM-80GB" (no "5").
+    # This must match the aliases in data/gpu_specs.yaml for end-to-end
+    # parse_repo → enrich_df tests to produce a non-null canonical_gpu_id.
+    "accelerator_model_name": "NVIDIA H100-SXM-80GB",
+    "accelerators_per_node": "8",
+    "accelerator_memory_capacity": "80 GB",
+    "framework": "TensorRT-LLM v0.12.0, CUDA 12.6.3",
+    "system_type": "datacenter",
+    "status": "available",
+    "division": "closed",
+}
+
+SYSTEM_JSON_AMD = {
+    "system_name": "AMD_Instinct_MI300Xx8_vLLM",
+    "accelerator_model_name": "AMD Instinct MI300X",
+    "accelerators_per_node": "8",
+    "accelerator_memory_capacity": "192 GB",
+    "framework": "ROCm 6.2.4, vLLM 0.5.0",
+    "system_type": "datacenter",
+    "status": "available",
+    "division": "closed",
+}
+
+LOG_SUMMARY_OFFLINE = """\
+================================================
+MLPerf Results Summary
+================================================
+SUT name : PySUT
+Scenario : Offline
+Mode     : PerformanceOnly
+Samples per second: 25.34
+Result is : VALID
+  Min duration satisfied : Yes
+  Min queries satisfied : Yes
+  Early stopping satisfied: Yes
+================================================
+Additional Stats
+================================================
+Min latency (ns)                : 1000000000
+Max latency (ns)                : 5000000000
+Mean latency (ns)               : 2000000000
+50.00 percentile latency (ns)   : 1800000000
+90.00 percentile latency (ns)   : 3500000000
+95.00 percentile latency (ns)   : 4000000000
+97.00 percentile latency (ns)   : 4200000000
+99.00 percentile latency (ns)   : 4800000000
+99.90 percentile latency (ns)   : 4950000000
+Mean First Token Latency (ns)   : 150000000
+99th Percentile First Token Latency (ns) : 300000000
+Mean Time Per Output Token (ns) : 50000
+99th Percentile Time Per Output Token (ns) : 80000
+"""
+
+LOG_SUMMARY_SERVER = """\
+================================================
+MLPerf Results Summary
+================================================
+SUT name : PySUT
+Scenario : Server
+Mode     : PerformanceOnly
+Scheduled samples per second : 12.56
+Result is : VALID
+  Min duration satisfied : Yes
+  Min queries satisfied : Yes
+  Early stopping satisfied: Yes
+================================================
+Additional Stats
+================================================
+Completed samples per second    : 11.80
+Min latency (ns)                : 2000000000
+Max latency (ns)                : 9000000000
+Mean latency (ns)               : 4000000000
+50.00 percentile latency (ns)   : 3800000000
+90.00 percentile latency (ns)   : 7000000000
+95.00 percentile latency (ns)   : 7800000000
+97.00 percentile latency (ns)   : 8200000000
+99.00 percentile latency (ns)   : 8900000000
+99.90 percentile latency (ns)   : 8990000000
+Mean First Token Latency (ns)   : 200000000
+99th Percentile First Token Latency (ns) : 450000000
+Mean Time Per Output Token (ns) : 60000
+99th Percentile Time Per Output Token (ns) : 95000
+"""
+
+LOG_SUMMARY_INVALID = """\
+================================================
+MLPerf Results Summary
+================================================
+Scenario : Offline
+Mode     : PerformanceOnly
+Samples per second: 99.99
+Result is : INVALID
+"""
+
+
+# ---------------------------------------------------------------------------
+# Fixtures: build a minimal synthetic repo tree on disk
+# ---------------------------------------------------------------------------
+
+def _build_repo(
+    tmp_path: Path,
+    system_json: dict,
+    benchmarks: list[str] | None = None,
+    scenarios: list[str] | None = None,
+) -> Path:
+    """
+    Create a minimal MLPerf-layout repo under tmp_path.
+
+    Returns the repo root path.
+    """
+    if benchmarks is None:
+        benchmarks = ["llama2-70b"]
+    if scenarios is None:
+        scenarios = ["Offline", "Server"]
+
+    repo = tmp_path / "inference_results_v6.0"
+    submitter = "TestSubmitter"
+    system_name = system_json["system_name"]
+
+    # systems/<system>.json
+    systems_dir = repo / "closed" / submitter / "systems"
+    systems_dir.mkdir(parents=True)
+    (systems_dir / f"{system_name}.json").write_text(
+        json.dumps(system_json), encoding="utf-8"
+    )
+
+    # results/<system>/<benchmark>/<scenario>/performance/run_1/mlperf_log_summary.txt
+    for bm in benchmarks:
+        for sc in scenarios:
+            log_content = LOG_SUMMARY_OFFLINE if sc == "Offline" else LOG_SUMMARY_SERVER
+            run_dir = (
+                repo / "closed" / submitter / "results"
+                / system_name / bm / sc / "performance" / "run_1"
+            )
+            run_dir.mkdir(parents=True)
+            (run_dir / "mlperf_log_summary.txt").write_text(log_content, encoding="utf-8")
+
+    return repo
+
+
+# ---------------------------------------------------------------------------
+# Unit tests: helper functions
+# ---------------------------------------------------------------------------
+
+class TestParseVramGb:
+    def test_gb_suffix(self):
+        assert _parse_vram_gb("192 GB") == 192.0
+
+    def test_gib_suffix(self):
+        assert _parse_vram_gb("80 GiB") == 80.0
+
+    def test_case_insensitive(self):
+        assert _parse_vram_gb("80 gb") == 80.0
+
+    def test_none_input(self):
+        assert _parse_vram_gb(None) is None
+
+    def test_empty_string(self):
+        assert _parse_vram_gb("") is None
+
+    def test_unrecognised_format(self):
+        assert _parse_vram_gb("lots") is None
+
+    def test_dot_only_match_returns_none(self):
+        # Regression: [\d.]+ matched ". GB" → float(".") raised ValueError and
+        # crashed the entire parse job.  Fixed by requiring match to start with \d.
+        assert _parse_vram_gb(". GB") is None
+
+    def test_malformed_float_returns_none(self):
+        # "1.2.3" is not a valid float; must return None, not crash.
+        assert _parse_vram_gb("1.2.3 GB") is None
+
+
+class TestBaseBenchmark:
+    @pytest.mark.parametrize("name,expected", [
+        ("llama2-70b",       "llama2-70b"),
+        ("llama2-70b-99",    "llama2-70b"),
+        ("llama2-70b-99.9",  "llama2-70b"),
+        ("gptj-99",          "gptj"),
+        ("mixtral-8x7b-99.9", "mixtral-8x7b"),
+    ])
+    def test_strip_suffix(self, name, expected):
+        assert _base_benchmark(name) == expected
+
+
+class TestAccuracyTier:
+    @pytest.mark.parametrize("name,expected", [
+        ("llama2-70b",        "base"),
+        ("gptj",              "base"),
+        ("llama2-70b-99",     "99"),
+        ("gptj-99",           "99"),
+        ("llama2-70b-99.9",   "99.9"),
+        ("mixtral-8x7b-99.9", "99.9"),
+    ])
+    def test_tier_extraction(self, name, expected):
+        assert _accuracy_tier(name) == expected
+
+
+# ---------------------------------------------------------------------------
+# Unit tests: _parse_system_json
+# ---------------------------------------------------------------------------
+
+class TestParseSystemJson:
+    def test_nvidia_fields(self, tmp_path):
+        p = tmp_path / "H100.json"
+        p.write_text(json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8")
+        hw = _parse_system_json(p)
+        assert hw["gpu_name"] == "NVIDIA H100-SXM-80GB"
+        assert hw["num_gpus"] == 8
+        assert hw["vram_gb"] == 80.0
+        assert "TensorRT-LLM" in hw["framework"]
+        assert hw["system_type"] == "datacenter"
+
+    def test_amd_fields(self, tmp_path):
+        p = tmp_path / "MI300X.json"
+        p.write_text(json.dumps(SYSTEM_JSON_AMD), encoding="utf-8")
+        hw = _parse_system_json(p)
+        assert hw["gpu_name"] == "AMD Instinct MI300X"
+        assert hw["num_gpus"] == 8
+        assert hw["vram_gb"] == 192.0
+        assert "ROCm" in hw["framework"]
+
+    def test_missing_file_returns_empty(self, tmp_path):
+        hw = _parse_system_json(tmp_path / "nonexistent.json")
+        assert hw == {}
+
+    def test_malformed_json_returns_empty(self, tmp_path):
+        p = tmp_path / "bad.json"
+        p.write_text("not json at all", encoding="utf-8")
+        hw = _parse_system_json(p)
+        assert hw == {}
+
+    def test_json_array_returns_empty(self, tmp_path):
+        # Bug fix: json.loads('[1,2,3]') returns a list; raw.get(...) on a list
+        # raised AttributeError (uncaught), aborting the entire round parse.
+        p = tmp_path / "array.json"
+        p.write_text("[1, 2, 3]", encoding="utf-8")
+        hw = _parse_system_json(p)
+        assert hw == {}
+
+    def test_json_scalar_returns_empty(self, tmp_path):
+        p = tmp_path / "scalar.json"
+        p.write_text('"just a string"', encoding="utf-8")
+        hw = _parse_system_json(p)
+        assert hw == {}
+
+
+# ---------------------------------------------------------------------------
+# Unit tests: _parse_log_summary
+# ---------------------------------------------------------------------------
+
+class TestParseLogSummary:
+    def test_offline_throughput(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(LOG_SUMMARY_OFFLINE, encoding="utf-8")
+        m = _parse_log_summary(p)
+        assert m["throughput_samples_per_sec"] == pytest.approx(25.34)
+        assert m["result_valid"] is True
+        assert m["scenario_from_log"] == "Offline"
+
+    def test_server_throughput(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(LOG_SUMMARY_SERVER, encoding="utf-8")
+        m = _parse_log_summary(p)
+        assert m["throughput_samples_per_sec"] == pytest.approx(12.56)
+        assert m["result_valid"] is True
+        assert m["scenario_from_log"] == "Server"
+
+    def test_invalid_result_flag(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(LOG_SUMMARY_INVALID, encoding="utf-8")
+        m = _parse_log_summary(p)
+        assert m["result_valid"] is False
+
+    def test_latency_ns_to_ms_conversion(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(LOG_SUMMARY_OFFLINE, encoding="utf-8")
+        m = _parse_log_summary(p)
+        # 2_000_000_000 ns → 2000.0 ms
+        assert m["latency_mean_ms"] == pytest.approx(2000.0)
+        assert m["latency_p99_ms"] == pytest.approx(4800.0)
+
+    def test_ttft_parsed(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(LOG_SUMMARY_OFFLINE, encoding="utf-8")
+        m = _parse_log_summary(p)
+        # 150_000_000 ns → 150.0 ms
+        assert m["ttft_mean_ms"] == pytest.approx(150.0)
+        assert m["ttft_p99_ms"] == pytest.approx(300.0)
+
+    def test_tpot_parsed(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(LOG_SUMMARY_OFFLINE, encoding="utf-8")
+        m = _parse_log_summary(p)
+        # 50_000 ns → 0.05 ms
+        assert m["tpot_mean_ms"] == pytest.approx(0.05)
+        assert m["tpot_p99_ms"] == pytest.approx(0.08)
+
+    def test_missing_file_returns_empty(self, tmp_path):
+        m = _parse_log_summary(tmp_path / "nonexistent.txt")
+        assert m == {}
+
+    def test_no_llm_metrics_returns_none(self, tmp_path):
+        p = tmp_path / "summary.txt"
+        p.write_text(
+            "Scenario : Offline\nSamples per second: 5.0\nResult is : VALID\n",
+            encoding="utf-8",
+        )
+        m = _parse_log_summary(p)
+        assert m["ttft_mean_ms"] is None
+        assert m["tpot_mean_ms"] is None
+
+
+# ---------------------------------------------------------------------------
+# Integration tests: parse_repo
+# ---------------------------------------------------------------------------
+
+class TestParseRepo:
+    def test_correct_row_count(self, tmp_path):
+        # 2 scenarios × 1 benchmark = 2 rows — verify both scenarios are present,
+        # not just that the total count is right (two Offline rows would also give len==2)
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b"],
+            scenarios=["Offline", "Server"],
+        )
+        df = parse_repo(repo, "v6.0")
+        assert len(df) == 2
+        assert set(df["scenario"]) == {"Offline", "Server"}
+
+    def test_round_tag_propagated(self, tmp_path):
+        repo = _build_repo(tmp_path, SYSTEM_JSON_NVIDIA)
+        df = parse_repo(repo, "v6.0")
+        assert (df["round"] == "v6.0").all()
+
+    def test_gpu_name_extracted(self, tmp_path):
+        repo = _build_repo(tmp_path, SYSTEM_JSON_AMD)
+        df = parse_repo(repo, "v6.0")
+        assert (df["gpu_name"] == "AMD Instinct MI300X").all()
+
+    def test_throughput_tokens_computed(self, tmp_path):
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        row = df.iloc[0]
+        expected = 25.34 * TOKENS_PER_SAMPLE["llama2-70b"]
+        assert row["throughput_tokens_per_sec"] == pytest.approx(expected)
+
+    def test_gptj_tokens_per_sample(self, tmp_path):
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["gptj"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["tokens_per_sample"] == 128
+
+    def test_non_llm_benchmark_excluded_by_default(self, tmp_path):
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["resnet50", "llama2-70b"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0", llm_only=True)
+        assert set(df["benchmark"]) == {"llama2-70b"}
+
+    def test_non_llm_included_when_flag_off(self, tmp_path):
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["resnet50", "llama2-70b"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0", llm_only=False)
+        # Verify both sides: non-LLM included AND LLM not accidentally dropped
+        assert set(df["benchmark"]) == {"resnet50", "llama2-70b"}
+
+    def test_invalid_result_rows_present(self, tmp_path):
+        # Parser includes INVALID rows — filtering is caller's responsibility
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        run_dir = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_INVALID, encoding="utf-8"
+        )
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        df = parse_repo(repo, "v6.0")
+        assert len(df) == 1
+        assert df.iloc[0]["result_valid"] == False  # noqa: E712 — numpy.bool_ != Python bool
+
+    def test_multiple_runs_uses_highest(self, tmp_path):
+        # run_2 has higher throughput; parser should pick run_2
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        base = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Offline" / "performance"
+        )
+
+        (base / "run_1").mkdir(parents=True)
+        (base / "run_1" / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+        run2_content = LOG_SUMMARY_OFFLINE.replace("Samples per second: 25.34",
+                                                   "Samples per second: 99.00")
+        (base / "run_2").mkdir(parents=True)
+        (base / "run_2" / "mlperf_log_summary.txt").write_text(
+            run2_content, encoding="utf-8"
+        )
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["throughput_samples_per_sec"] == pytest.approx(99.00)
+
+    def test_empty_repo_returns_empty_df(self, tmp_path):
+        repo = tmp_path / "empty_repo"
+        repo.mkdir()
+        df = parse_repo(repo, "v6.0")
+        assert df.empty
+
+    def test_benchmark_base_column(self, tmp_path):
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b-99"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["benchmark_base"] == "llama2-70b"
+
+    def test_submitter_field(self, tmp_path):
+        repo = _build_repo(tmp_path, SYSTEM_JSON_AMD)
+        df = parse_repo(repo, "v6.0")
+        assert (df["submitter"] == "TestSubmitter").all()
+
+    def test_benchmark_accuracy_tier_column(self, tmp_path):
+        """benchmark_accuracy_tier must be present and correctly derived."""
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b-99"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        assert "benchmark_accuracy_tier" in df.columns
+        assert df.iloc[0]["benchmark_accuracy_tier"] == "99"
+
+    def test_num_gpus_zero_clamped_to_one(self, tmp_path):
+        """accelerators_per_node=0 must produce num_gpus=1, not 0.
+
+        CPU-only inference systems (e.g. Intel EMR) submit with 0 accelerators.
+        Storing 0 would make the stored column inconsistent with the divisor
+        used to compute throughput_tok_per_sec_per_gpu.
+        """
+        zero_gpu_json = {**SYSTEM_JSON_NVIDIA, "accelerators_per_node": "0"}
+        repo = _build_repo(
+            tmp_path, zero_gpu_json,
+            benchmarks=["llama2-70b"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["num_gpus"] == 1
+        # per-GPU throughput must be finite (not inf/nan)
+        assert df.iloc[0]["throughput_tok_per_sec_per_gpu"] > 0
+
+
+# ---------------------------------------------------------------------------
+# Security tests: _safe_read_text, symlink guards, _run_number
+# ---------------------------------------------------------------------------
+
+class TestSafeReadText:
+    def test_normal_file_read(self, tmp_path):
+        p = tmp_path / "ok.txt"
+        p.write_text("hello", encoding="utf-8")
+        assert _safe_read_text(p) == "hello"
+
+    def test_missing_file_returns_none(self, tmp_path):
+        assert _safe_read_text(tmp_path / "missing.txt") is None
+
+    def test_oversized_file_returns_none(self, tmp_path):
+        p = tmp_path / "huge.txt"
+        # Write a file just over the limit.
+        p.write_bytes(b"x" * (_MAX_FILE_BYTES + 1))
+        assert _safe_read_text(p) is None
+
+    def test_symlink_returns_none(self, tmp_path):
+        target = tmp_path / "real.txt"
+        target.write_text("secret", encoding="utf-8")
+        link = tmp_path / "link.txt"
+        link.symlink_to(target)
+        assert _safe_read_text(link) is None
+
+    def test_symlink_to_outside_returns_none(self, tmp_path):
+        # Simulates a git-repo symlink pointing outside the repo root.
+        link = tmp_path / "escape.txt"
+        link.symlink_to("/etc/hosts")
+        assert _safe_read_text(link) is None
+
+    def test_read_error_returns_none(self, tmp_path, monkeypatch):
+        """OSError from read_text() after lstat() succeeds returns None (lines 140-142)."""
+        p = tmp_path / "ok_to_stat.txt"
+        p.write_text("data", encoding="utf-8")
+
+        def _raise(*args, **kwargs):
+            raise OSError("simulated read failure")
+
+        monkeypatch.setattr(Path, "read_text", _raise)
+        assert _safe_read_text(p) is None
+
+
+class TestRunNumber:
+    def test_extracts_integer(self, tmp_path):
+        p = tmp_path / "run_3" / "mlperf_log_summary.txt"
+        assert _run_number(p) == 3
+
+    def test_double_digit(self, tmp_path):
+        p = tmp_path / "run_12" / "mlperf_log_summary.txt"
+        assert _run_number(p) == 12
+
+    def test_no_match_returns_minus_one(self, tmp_path):
+        p = tmp_path / "audit" / "mlperf_log_summary.txt"
+        assert _run_number(p) == -1
+
+
+class TestSymlinkGuards:
+    def test_symlinked_submitter_dir_skipped(self, tmp_path):
+        """A symlinked submitter directory must not be walked.
+
+        The target is a valid submission tree — without the guard, the parser
+        would follow the symlink and return rows.  With it, df must be empty.
+        """
+        repo = tmp_path / "repo"
+        real_sub = tmp_path / "real_submitter"
+        # Build a complete, parseable submission inside real_sub.
+        run_dir = (
+            real_sub / "results" / "SomeSystem" / "llama2-70b"
+            / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+        sys_dir = real_sub / "systems"
+        sys_dir.mkdir()
+        (sys_dir / "SomeSystem.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        # Symlink real_sub into the repo's closed/ directory.
+        (repo / "closed").mkdir(parents=True)
+        (repo / "closed" / "EvilSub").symlink_to(real_sub)
+        df = parse_repo(repo, "v6.0")
+        assert df.empty
+
+    def test_symlinked_system_json_skipped(self, tmp_path):
+        """A symlinked system JSON must not be read."""
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        # Build the results tree normally.
+        run_dir = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+        # Create a symlinked system JSON instead of a real one.
+        real_json = tmp_path / "real.json"
+        real_json.write_text(json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8")
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").symlink_to(real_json)
+        # Parser should produce a row (from the log) but with no hw fields —
+        # the **hw spread adds no keys when hw == {}, so gpu_name is absent.
+        df = parse_repo(repo, "v6.0")
+        assert len(df) == 1
+        assert "gpu_name" not in df.columns
+
+    def test_parse_log_summary_rejects_symlink(self, tmp_path):
+        """_parse_log_summary returns {} for a symlinked log file."""
+        real = tmp_path / "real.txt"
+        real.write_text(LOG_SUMMARY_OFFLINE, encoding="utf-8")
+        link = tmp_path / "link.txt"
+        link.symlink_to(real)
+        assert _parse_log_summary(link) == {}
+
+
+# ---------------------------------------------------------------------------
+# Unit tests: _extract_precision
+# ---------------------------------------------------------------------------
+
+class TestExtractPrecision:
+    @pytest.mark.parametrize("system_name,framework,expected", [
+        # FP8 in system name
+        ("AMD_MI300X_FP8_vLLM", "ROCm 6.2.4",            "fp8"),
+        # FP16 in framework string
+        ("AMD_MI300X_vLLM",     "ROCm 6.2.4 fp16",       "fp16"),
+        # BF16
+        ("H100_bf16_TRT",       "TRT-LLM",                "bf16"),
+        # FP4 / MXFP4 — should win over FP8 when both present (priority order)
+        ("MI355X_MXFP4",        "ROCm 7.2, fp8 fallback", "fp4"),
+        # INT8
+        ("H100_int8_TRT",       None,                      "int8"),
+        # Nothing recognisable → None
+        ("H100_SXM5_80GBx8",    "TensorRT-LLM v0.12.0",   None),
+        # None framework arg handled gracefully
+        ("MI300X_FP16",         None,                      "fp16"),
+    ])
+    def test_precision_extraction(self, system_name, framework, expected):
+        assert _extract_precision(system_name, framework) == expected
+
+
+# ---------------------------------------------------------------------------
+# Additional parse_repo behavior tests (post-fix)
+# ---------------------------------------------------------------------------
+
+class TestParseRepoPostFix:
+    def test_server_tput_uses_scheduled_not_completed(self, tmp_path):
+        """_RE_OFFLINE_TPUT must not match 'Completed samples per second'."""
+        log_with_diverged_completed = """\
+================================================
+MLPerf Results Summary
+================================================
+Scenario : Server
+Mode     : PerformanceOnly
+Scheduled samples per second : 12.56
+Result is : VALID
+================================================
+Additional Stats
+================================================
+Completed samples per second    : 11.80
+Mean latency (ns)               : 4000000000
+99.00 percentile latency (ns)   : 8900000000
+"""
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        run_dir = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Server" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            log_with_diverged_completed, encoding="utf-8"
+        )
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["throughput_samples_per_sec"] == pytest.approx(12.56)
+
+    def test_per_gpu_throughput_normalized(self, tmp_path):
+        """Both throughput columns are independently pinned to fixture constants.
+
+        Using row["throughput_tokens_per_sec"] / 8 as expected would only test
+        col_a / 8 ≈ col_b — it passes even if both columns are miscalculated by
+        the same factor.  Pin to the known fixture value instead.
+        """
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,   # 8 GPUs
+            benchmarks=["llama2-70b"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        row = df.iloc[0]
+        # 25.34 samples/sec (LOG_SUMMARY_OFFLINE) × 294 tokens/sample
+        per_node = 25.34 * TOKENS_PER_SAMPLE["llama2-70b"]
+        assert row["throughput_tokens_per_sec"] == pytest.approx(per_node)
+        assert row["throughput_tok_per_sec_per_gpu"] == pytest.approx(per_node / 8)
+
+    def test_precision_extracted_from_system_name(self, tmp_path):
+        fp8_json = {**SYSTEM_JSON_NVIDIA, "system_name": "H100_SXM5_80GBx8_FP8_TRT"}
+        repo = _build_repo(tmp_path, fp8_json, scenarios=["Offline"])
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["precision"] == "fp8"
+
+    def test_benchmark_column_is_lowercase(self, tmp_path):
+        """benchmark must be lowercased even when the directory name is uppercase."""
+        # _build_repo default ("llama2-70b") is already lowercase — that fixture
+        # never exercises the .lower() call.  Use an uppercase directory name here.
+        repo = _build_repo(
+            tmp_path, SYSTEM_JSON_NVIDIA,
+            benchmarks=["LLAMA2-70B"],
+            scenarios=["Offline"],
+        )
+        df = parse_repo(repo, "v6.0")
+        assert df.iloc[0]["benchmark"] == "llama2-70b"
+
+
+# ---------------------------------------------------------------------------
+# Coverage gap A: open division
+# ---------------------------------------------------------------------------
+
+class TestOpenDivision:
+    def test_open_division_rows_included(self, tmp_path):
+        """Submissions under open/ must be walked, not only closed/."""
+        repo = tmp_path / "repo"
+        system_name = SYSTEM_JSON_AMD["system_name"]
+        run_dir = (
+            repo / "open" / "OpenSub" / "results"
+            / system_name / "llama2-70b" / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+        sys_dir = repo / "open" / "OpenSub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_AMD), encoding="utf-8"
+        )
+        df = parse_repo(repo, "v6.0", divisions=("open",))
+        assert len(df) == 1
+        assert df.iloc[0]["division"] == "open"
+        assert df.iloc[0]["submitter"] == "OpenSub"
+
+    def test_both_divisions_combined(self, tmp_path):
+        """When both divisions are requested, rows from each are present."""
+        # closed submission
+        closed_repo = _build_repo(
+            tmp_path / "closed_build", SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b"], scenarios=["Offline"],
+        )
+        # graft an open submission into the same repo root
+        open_sys = SYSTEM_JSON_AMD["system_name"]
+        run_dir = (
+            closed_repo / "open" / "OpenSub" / "results"
+            / open_sys / "llama2-70b" / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+        sys_dir = closed_repo / "open" / "OpenSub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{open_sys}.json").write_text(
+            json.dumps(SYSTEM_JSON_AMD), encoding="utf-8"
+        )
+        df = parse_repo(closed_repo, "v6.0", divisions=("closed", "open"))
+        assert set(df["division"]) == {"closed", "open"}
+
+
+# ---------------------------------------------------------------------------
+# Coverage gap B: parse_repos (multi-round entry point)
+# ---------------------------------------------------------------------------
+
+class TestParseRepos:
+    def test_combines_two_rounds(self, tmp_path):
+        """Rows from two different rounds are concatenated."""
+        repo_a = _build_repo(
+            tmp_path / "a", SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b"], scenarios=["Offline"],
+        )
+        repo_b = _build_repo(
+            tmp_path / "b", SYSTEM_JSON_AMD,
+            benchmarks=["gptj"], scenarios=["Offline"],
+        )
+        df = parse_repos([(repo_a, "v5.1"), (repo_b, "v6.0")])
+        assert set(df["round"]) == {"v5.1", "v6.0"}
+        assert len(df) == 2
+
+    def test_missing_round_silently_skipped(self, tmp_path):
+        """A path that doesn't exist must be skipped, not raise."""
+        repo = _build_repo(tmp_path, SYSTEM_JSON_NVIDIA, scenarios=["Offline"])
+        df = parse_repos([
+            (repo, "v6.0"),
+            (tmp_path / "does_not_exist", "v5.1"),
+        ])
+        assert len(df) == 1
+        assert df.iloc[0]["round"] == "v6.0"
+
+    def test_all_missing_returns_empty_df(self, tmp_path):
+        df = parse_repos([(tmp_path / "ghost_a", "v4.1"), (tmp_path / "ghost_b", "v5.0")])
+        assert df.empty
+
+    def test_deduplication_not_silently_applied(self, tmp_path):
+        """parse_repos must NOT silently drop duplicate (GPU, benchmark) pairs
+        that appear in multiple rounds — those are intentional cross-round rows."""
+        repo_a = _build_repo(
+            tmp_path / "a", SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b"], scenarios=["Offline"],
+        )
+        repo_b = _build_repo(
+            tmp_path / "b", SYSTEM_JSON_NVIDIA,
+            benchmarks=["llama2-70b"], scenarios=["Offline"],
+        )
+        df = parse_repos([(repo_a, "v5.1"), (repo_b, "v6.0")])
+        # Same GPU + benchmark in two rounds → 2 rows (no silent dedup)
+        assert len(df) == 2
+        assert set(df["round"]) == {"v5.1", "v6.0"}
+
+
+# ---------------------------------------------------------------------------
+# Reference table sanity checks
+# ---------------------------------------------------------------------------
+
+class TestReferenceTables:
+    def test_all_llm_benchmarks_have_token_count(self):
+        for bm in LLM_BENCHMARKS:
+            assert bm in TOKENS_PER_SAMPLE, f"Missing TOKENS_PER_SAMPLE entry for {bm!r}"
+
+    def test_gptj_is_128(self):
+        assert TOKENS_PER_SAMPLE["gptj"] == 128
+
+    def test_accuracy_variants_match_base(self):
+        for bm in LLM_BENCHMARKS:
+            base = _base_benchmark(bm)
+            if base != bm:
+                assert TOKENS_PER_SAMPLE[bm] == TOKENS_PER_SAMPLE[base], (
+                    f"{bm!r} and {base!r} should have same TOKENS_PER_SAMPLE"
+                )
+
+
+# ---------------------------------------------------------------------------
+# Coverage gap C: _find_best_run early-return paths
+# ---------------------------------------------------------------------------
+
+class TestFindBestRun:
+    def test_no_performance_dir_returns_none(self, tmp_path):
+        """scenario_dir with no performance/ subdirectory returns None (line 304)."""
+        scenario_dir = tmp_path / "Offline"
+        scenario_dir.mkdir()
+        assert _find_best_run(scenario_dir) is None
+
+    def test_no_valid_runs_returns_none(self, tmp_path):
+        """performance/ exists but is empty → no run_* files → returns None (line 316)."""
+        scenario_dir = tmp_path / "Offline"
+        (scenario_dir / "performance").mkdir(parents=True)
+        assert _find_best_run(scenario_dir) is None
+
+
+# ---------------------------------------------------------------------------
+# Coverage gap D: parse_repo continue-guards and debug-log branches
+# ---------------------------------------------------------------------------
+
+class TestParseRepoEdgeCases:
+    """Targets the five directory-guard continue statements and two debug branches."""
+
+    def test_no_system_json_row_still_produced(self, tmp_path):
+        """Missing systems/ dir → OSError branch fires (lines 369-371) → hw = {}.
+
+        The row is still produced but no hardware columns are present because
+        hw == {} contributes nothing to the row dict spread.
+        """
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        run_dir = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+        # Intentionally NO systems/ directory so lstat raises FileNotFoundError
+        df = parse_repo(repo, "v6.0")
+        assert len(df) == 1
+        assert "gpu_name" not in df.columns  # hw spread was empty — no hardware columns
+        # Performance data from the log must still be captured
+        assert df.iloc[0]["benchmark"] == "llama2-70b"
+        assert df.iloc[0]["throughput_tok_per_sec_per_gpu"] > 0
+
+    def test_results_dir_is_file_skipped(self, tmp_path):
+        """A file named 'results' (not a dir) triggers the is_dir() guard (line 357)."""
+        repo = tmp_path / "repo"
+        sub_dir = repo / "closed" / "Sub"
+        sub_dir.mkdir(parents=True)
+        (sub_dir / "results").write_text("not a directory", encoding="utf-8")
+        assert parse_repo(repo, "v6.0").empty
+
+    def test_system_dir_is_file_skipped(self, tmp_path):
+        """A file inside results/ triggers the system-dir is_dir() guard (line 361)."""
+        repo = tmp_path / "repo"
+        results_dir = repo / "closed" / "Sub" / "results"
+        results_dir.mkdir(parents=True)
+        (results_dir / "not_a_system.txt").write_text("file", encoding="utf-8")
+        assert parse_repo(repo, "v6.0").empty
+
+    def test_benchmark_dir_is_file_skipped(self, tmp_path):
+        """A file inside a system dir triggers the benchmark is_dir() guard (line 390).
+
+        The guard must be selective: the file is skipped while a valid benchmark
+        directory next to it is still processed.  assert df.empty would not catch
+        an over-aggressive guard that skipped the entire system.
+        """
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        system_dir = repo / "closed" / "Sub" / "results" / system_name
+        system_dir.mkdir(parents=True)
+
+        # Invalid entry — must be skipped
+        (system_dir / "readme.txt").write_text("file", encoding="utf-8")
+
+        # Valid sibling — must still be processed
+        run_dir = system_dir / "llama2-70b" / "Offline" / "performance" / "run_1"
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(
+            LOG_SUMMARY_OFFLINE, encoding="utf-8"
+        )
+
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        df = parse_repo(repo, "v6.0")
+        assert len(df) == 1                           # file skipped, valid dir kept
+        assert df.iloc[0]["benchmark"] == "llama2-70b"
+
+    def test_scenario_dir_is_file_skipped(self, tmp_path):
+        """A file inside a benchmark dir triggers the scenario is_dir() guard (line 401)."""
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        benchmark_dir = (
+            repo / "closed" / "Sub" / "results" / system_name / "llama2-70b"
+        )
+        benchmark_dir.mkdir(parents=True)
+        (benchmark_dir / "metadata.txt").write_text("file", encoding="utf-8")
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        assert parse_repo(repo, "v6.0").empty
+
+    def test_no_performance_run_row_skipped(self, tmp_path):
+        """Scenario dir with no performance/run_N tree produces no row (lines 406-407)."""
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        # Scenario dir exists but NO performance/ inside it
+        scenario_dir = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Offline"
+        )
+        scenario_dir.mkdir(parents=True)
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        assert parse_repo(repo, "v6.0").empty
+
+    def test_scenario_mismatch_debug_logged(self, tmp_path, caplog):
+        """Scenario in log != directory name emits a debug message (line 414)."""
+        system_name = SYSTEM_JSON_NVIDIA["system_name"]
+        repo = tmp_path / "repo"
+        # Directory is "Offline" but the log file claims "Scenario : Server"
+        mismatched = LOG_SUMMARY_OFFLINE.replace(
+            "Scenario : Offline", "Scenario : Server"
+        )
+        run_dir = (
+            repo / "closed" / "Sub" / "results"
+            / system_name / "llama2-70b" / "Offline" / "performance" / "run_1"
+        )
+        run_dir.mkdir(parents=True)
+        (run_dir / "mlperf_log_summary.txt").write_text(mismatched, encoding="utf-8")
+        sys_dir = repo / "closed" / "Sub" / "systems"
+        sys_dir.mkdir(parents=True)
+        (sys_dir / f"{system_name}.json").write_text(
+            json.dumps(SYSTEM_JSON_NVIDIA), encoding="utf-8"
+        )
+        with caplog.at_level(logging.DEBUG, logger="src.data.mlperf_parser"):
+            df = parse_repo(repo, "v6.0")
+        assert len(df) == 1  # row is still produced
+        assert any("mismatch" in r.getMessage().lower() for r in caplog.records)
+
+
+# ---------------------------------------------------------------------------
+# Coverage gap E: CLI entry point — main() and _build_arg_parser()
+# ---------------------------------------------------------------------------
+
+class TestMainCLI:
+    """Integration tests for main() via monkeypatched sys.argv.
+
+    _build_repo(parent, ...) creates a repo at parent/inference_results_v6.0/,
+    so --repos-dir parent --rounds inference_results_v6.0 is the correct shape.
+    """
+
+    def test_no_round_dirs_exits_1(self, tmp_path, monkeypatch):
+        """--repos-dir with no matching subdirectories → SystemExit(1)."""
+        repos_dir = tmp_path / "repos"
+        repos_dir.mkdir()
+        monkeypatch.setattr(sys, "argv", [
+            "mlperf_parser",
+            "--repos-dir", str(repos_dir),
+            "--rounds", "v6.0",
+        ])
+        with pytest.raises(SystemExit) as exc:
+            main()
+        assert exc.value.code == 1
+
+    def test_empty_round_exits_1(self, tmp_path, monkeypatch):
+        """Round dir exists but is empty → parse_repos returns empty df → SystemExit(1)."""
+        repos_dir = tmp_path / "repos"
+        (repos_dir / "v6.0").mkdir(parents=True)
+        monkeypatch.setattr(sys, "argv", [
+            "mlperf_parser",
+            "--repos-dir", str(repos_dir),
+            "--rounds", "v6.0",
+        ])
+        with pytest.raises(SystemExit) as exc:
+            main()
+        assert exc.value.code == 1
+
+    def test_normal_run_writes_csv(self, tmp_path, monkeypatch, capsys):
+        """Normal invocation writes a CSV and prints a row-count summary."""
+        repos_dir = tmp_path / "repos"
+        _build_repo(repos_dir, SYSTEM_JSON_NVIDIA, scenarios=["Offline"])
+        out_csv = tmp_path / "out.csv"
+        monkeypatch.setattr(sys, "argv", [
+            "mlperf_parser",
+            "--repos-dir", str(repos_dir),
+            "--rounds", "inference_results_v6.0",
+            "--output", str(out_csv),
+        ])
+        main()
+        assert out_csv.exists()
+        result = pd.read_csv(out_csv)
+        assert len(result) == 1
+        assert result.iloc[0]["benchmark"] == "llama2-70b"   # not just any row
+        assert result.iloc[0]["throughput_samples_per_sec"] == pytest.approx(25.34)
+        assert "Total rows" in capsys.readouterr().out
+
+    def test_parquet_flag_writes_parquet(self, tmp_path, monkeypatch):
+        """--parquet produces a .parquet file alongside the CSV."""
+        repos_dir = tmp_path / "repos"
+        _build_repo(repos_dir, SYSTEM_JSON_NVIDIA, scenarios=["Offline"])
+        out_csv = tmp_path / "out.csv"
+        monkeypatch.setattr(sys, "argv", [
+            "mlperf_parser",
+            "--repos-dir", str(repos_dir),
+            "--rounds", "inference_results_v6.0",
+            "--output", str(out_csv),
+            "--parquet",
+        ])
+        main()
+        pq_path = out_csv.with_suffix(".parquet")
+        assert pq_path.exists()
+        assert len(pd.read_parquet(pq_path)) == 1   # file is readable, not just present
+
+    def test_all_benchmarks_flag_includes_non_llm(self, tmp_path, monkeypatch):
+        """--all-benchmarks passes llm_only=False, including non-LLM benchmarks."""
+        repos_dir = tmp_path / "repos"
+        _build_repo(
+            repos_dir, SYSTEM_JSON_NVIDIA,
+            benchmarks=["resnet50"],
+            scenarios=["Offline"],
+        )
+        out_csv = tmp_path / "out.csv"
+        monkeypatch.setattr(sys, "argv", [
+            "mlperf_parser",
+            "--repos-dir", str(repos_dir),
+            "--rounds", "inference_results_v6.0",
+            "--output", str(out_csv),
+            "--all-benchmarks",
+        ])
+        main()
+        assert "resnet50" in pd.read_csv(out_csv)["benchmark"].values

tests/test_predictor.py

@@ -0,0 +1,442 @@
+"""
+Unit and integration tests for src/models/predictor.py.
+
+All tests load the real trained model (data/models/prophet_v1.json) so they
+verify the full inference path, not just the feature-construction logic.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+
+import pytest
+
+from src.models.predictor import (
+    FEATURE_COLS,
+    GpuPredictor,
+    _build_feature_vector,
+    _SERVING_ROUND,
+)
+from src.data.gpu_spec_db import load_specs
+
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+@pytest.fixture(scope="module")
+def predictor() -> GpuPredictor:
+    return GpuPredictor()
+
+
+@pytest.fixture(scope="module")
+def spec_map() -> dict:
+    return {s["id"]: s for s in load_specs()}
+
+
+# ---------------------------------------------------------------------------
+# _build_feature_vector — pure function tests (no model needed)
+# ---------------------------------------------------------------------------
+
+class TestBuildFeatureVector:
+    def test_amd_nvidia_arch_ordinals_mutually_exclusive(self, spec_map):
+        feat_amd, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="gptj",
+            scenario="Server",
+            accuracy_tier="base",
+            framework="rocm_other",
+        )
+        feat_nv, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["h100_sxm"],
+            model_name="gptj",
+            scenario="Server",
+            accuracy_tier="base",
+            framework="tensorrt",
+        )
+        nvidia_idx = FEATURE_COLS.index("nvidia_arch_gen")
+        amd_idx = FEATURE_COLS.index("amd_arch_gen")
+
+        # AMD GPU: nvidia_arch_gen NaN, amd_arch_gen not NaN
+        assert math.isnan(feat_amd[nvidia_idx])
+        assert not math.isnan(feat_amd[amd_idx])
+
+        # NVIDIA GPU: amd_arch_gen NaN, nvidia_arch_gen not NaN
+        assert math.isnan(feat_nv[amd_idx])
+        assert not math.isnan(feat_nv[nvidia_idx])
+
+    def test_scenario_offline_flag(self, spec_map):
+        feat_off, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["h200_sxm"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="vllm",
+        )
+        feat_srv, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["h200_sxm"],
+            model_name="llama2-70b",
+            scenario="Server",
+            accuracy_tier="99",
+            framework="vllm",
+        )
+        off_idx = FEATURE_COLS.index("scenario_offline")
+        assert feat_off[off_idx] == 1
+        assert feat_srv[off_idx] == 0
+
+    def test_is_cdna4_flag(self, spec_map):
+        feat_cdna4, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi355x"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99.9",
+            framework="rocm_other",
+        )
+        feat_cdna3, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99.9",
+            framework="rocm_other",
+        )
+        cdna4_idx = FEATURE_COLS.index("is_cdna4")
+        assert feat_cdna4[cdna4_idx] == 1
+        assert feat_cdna3[cdna4_idx] == 0
+
+    def test_bytes_per_param_precision_selection(self, spec_map):
+        bpp_idx = FEATURE_COLS.index("bytes_per_param")
+        # AMD tier 99   → fp8  → 1.0 byte/param
+        feat_amd_99, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="vllm",
+        )
+        assert feat_amd_99[bpp_idx] == pytest.approx(1.0)
+        # AMD tier 99.9 → fp8 override → 1.0 byte/param (not 2.0)
+        feat_amd_99_9, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99.9",
+            framework="vllm",
+        )
+        assert feat_amd_99_9[bpp_idx] == pytest.approx(1.0)
+        # NVIDIA tier 99.9 → fp16 → 2.0 bytes/param (no override)
+        feat_nv_99_9, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["h100_sxm"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99.9",
+            framework="tensorrt",
+        )
+        assert feat_nv_99_9[bpp_idx] == pytest.approx(2.0)
+
+    def test_fw_other_all_dummies_zero(self, spec_map):
+        # framework="other" must set all three fw_* flags to 0.
+        # Previously untested — a bug that accidentally set one flag to 1 would
+        # be invisible to the model (wrong feature for unknown frameworks).
+        features, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="gptj",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="other",
+        )
+        for fw_col in ("fw_tensorrt", "fw_vllm", "fw_rocm_other"):
+            assert features[FEATURE_COLS.index(fw_col)] == 0, (
+                f"{fw_col} should be 0 for framework='other'"
+            )
+
+    def test_is_base_tier_flag(self, spec_map):
+        base_idx = FEATURE_COLS.index("is_base_tier")
+        feat_base, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="gptj",
+            scenario="Offline",
+            accuracy_tier="base",
+            framework="rocm_other",
+        )
+        feat_99, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="gptj",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="rocm_other",
+        )
+        feat_99_9, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["h100_sxm"],
+            model_name="gptj",
+            scenario="Offline",
+            accuracy_tier="99.9",
+            framework="tensorrt",
+        )
+        assert feat_base[base_idx] == 1
+        assert feat_99[base_idx] == 0
+        assert feat_99_9[base_idx] == 0
+
+    def test_mlperf_round_num_serving_uses_latest(self, spec_map):
+        from src.features.build_features import ROUND_ORDINAL
+        # _SERVING_ROUND must be the maximum defined ordinal — any future round
+        # added to ROUND_ORDINAL automatically raises this floor.
+        assert _SERVING_ROUND == float(max(ROUND_ORDINAL.values()))
+        # The feature vector must include it at the correct position.
+        features, _, _ = _build_feature_vector(
+            gpu_spec=spec_map["mi300x"],
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="vllm",
+        )
+        assert features[FEATURE_COLS.index("mlperf_round_num")] == _SERVING_ROUND
+
+
+# ---------------------------------------------------------------------------
+# GpuPredictor.predict — full inference path
+# ---------------------------------------------------------------------------
+
+class TestGpuPredictorPredict:
+    def test_predict_returns_expected_keys(self, predictor):
+        result = predictor.predict(
+            gpu_id="mi300x",
+            model_name="llama2-70b",
+        )
+        expected = {
+            "gpu_id", "model_name", "scenario", "accuracy_tier", "framework",
+            "pred_throughput_tok_per_sec", "roofline_tput_tok_per_sec",
+            "efficiency_ratio", "vram_fits", "model_size_gb",
+        }
+        assert set(result.keys()) == expected
+
+    def test_predict_throughput_positive(self, predictor):
+        result = predictor.predict(
+            gpu_id="h100_sxm",
+            model_name="llama2-70b",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="tensorrt",
+        )
+        # Must reach at least 1% of the roofline — a degenerate model predicting
+        # ~0.001 tok/s would pass a bare > 0 check but not this threshold.
+        assert result["pred_throughput_tok_per_sec"] >= 0.01 * result["roofline_tput_tok_per_sec"]
+
+    def test_predict_never_exceeds_roofline(self, predictor):
+        for gpu_id in ["mi300x", "h100_sxm", "h200_sxm", "mi325x", "mi355x"]:
+            result = predictor.predict(
+                gpu_id=gpu_id,
+                model_name="llama2-70b",
+                accuracy_tier="99",
+            )
+            assert (
+                result["pred_throughput_tok_per_sec"]
+                <= result["roofline_tput_tok_per_sec"] + 1e-3
+            )
+
+    def test_predict_vram_fits_llama70b_on_mi300x(self, predictor):
+        result = predictor.predict(
+            gpu_id="mi300x",
+            model_name="llama2-70b",
+            accuracy_tier="99",
+        )
+        # llama2-70b at FP8 = 70 GB, MI300X has 192 GB → fits
+        assert result["vram_fits"] is True
+
+    def test_predict_vram_not_fit_405b_on_l4(self, predictor):
+        result = predictor.predict(
+            gpu_id="l4",
+            model_name="llama3.1-405b",
+            accuracy_tier="99",
+        )
+        # 405B FP8 = 405 GB >> 24 GB L4 VRAM
+        assert result["vram_fits"] is False
+
+    def test_unknown_gpu_raises(self, predictor):
+        with pytest.raises(ValueError, match="Unknown gpu_id"):
+            predictor.predict(gpu_id="rtx9090", model_name="gptj")
+
+    def test_unknown_model_raises(self, predictor):
+        with pytest.raises(ValueError, match="Unknown model_name"):
+            predictor.predict(gpu_id="mi300x", model_name="gpt4")
+
+    def test_invalid_scenario_raises(self, predictor):
+        with pytest.raises(ValueError, match="Invalid scenario"):
+            predictor.predict(gpu_id="mi300x", model_name="gptj", scenario="Interactive")
+
+    def test_invalid_tier_raises(self, predictor):
+        with pytest.raises(ValueError, match="Invalid accuracy_tier"):
+            predictor.predict(gpu_id="mi300x", model_name="gptj", accuracy_tier="99.5")
+
+    def test_invalid_framework_raises(self, predictor):
+        with pytest.raises(ValueError, match="Invalid framework"):
+            predictor.predict(gpu_id="mi300x", model_name="gptj", framework="pytorch")
+
+    def test_model_size_gb_correct_fp8(self, predictor):
+        result = predictor.predict(
+            gpu_id="mi300x",
+            model_name="llama2-70b",
+            accuracy_tier="99",
+        )
+        # 70B params × 1 byte/param (FP8) = 70 GB
+        assert result["model_size_gb"] == pytest.approx(70.0)
+
+
+# ---------------------------------------------------------------------------
+# GpuPredictor.predict_batch
+# ---------------------------------------------------------------------------
+
+class TestGpuPredictorBatch:
+    def test_batch_preserves_order_and_identity(self, predictor):
+        reqs = [
+            {"gpu_id": "mi300x",   "model_name": "llama2-70b"},
+            {"gpu_id": "h100_sxm", "model_name": "gptj"},
+            {"gpu_id": "h200_sxm", "model_name": "mixtral-8x7b"},
+        ]
+        results = predictor.predict_batch(reqs)
+        assert len(results) == 3
+        # Count alone doesn't catch a bug that returns the mi300x result 3 times.
+        for req, res in zip(reqs, results):
+            assert res["gpu_id"] == req["gpu_id"]
+            assert res["model_name"] == req["model_name"]
+
+    def test_batch_empty_returns_empty(self, predictor):
+        assert predictor.predict_batch([]) == []
+
+    @pytest.mark.parametrize("req", [
+        # tier 99 (fp8) — baseline case
+        {"gpu_id": "mi325x", "model_name": "llama2-70b",
+         "scenario": "Server", "accuracy_tier": "99", "framework": "rocm_other"},
+        # AMD tier 99.9 — fp8 override path; a batch/single divergence here would
+        # produce the wrong model_size_gb (70 vs 140 GB) and thus wrong vram_fits.
+        {"gpu_id": "mi300x", "model_name": "llama2-70b",
+         "scenario": "Offline", "accuracy_tier": "99.9", "framework": "vllm"},
+    ])
+    def test_batch_results_match_single(self, predictor, req):
+        single = predictor.predict(**{k: v for k, v in req.items()})
+        batch = predictor.predict_batch([req])
+        # Compare every numeric output — the batch path accumulates metadata in a
+        # separate dict; a divergence (e.g. vram_fits always True) was invisible
+        # when only pred_throughput_tok_per_sec was checked.
+        for key in ("pred_throughput_tok_per_sec", "roofline_tput_tok_per_sec",
+                    "efficiency_ratio", "model_size_gb"):
+            assert batch[0][key] == pytest.approx(single[key], rel=1e-4), (
+                f"{key}: batch={batch[0][key]!r}  single={single[key]!r}"
+            )
+        assert batch[0]["vram_fits"] == single["vram_fits"]
+
+    def test_batch_all_in_scope_gpus(self, predictor):
+        from src.data.gpu_spec_db import load_specs
+        in_scope = [s["id"] for s in load_specs() if s.get("in_model_scope")]
+        reqs = [{"gpu_id": g, "model_name": "llama2-70b"} for g in in_scope]
+        results = predictor.predict_batch(reqs)
+        assert len(results) == len(in_scope)
+        for r in results:
+            assert r["pred_throughput_tok_per_sec"] >= 0.01 * r["roofline_tput_tok_per_sec"]
+
+
+# ---------------------------------------------------------------------------
+# Cross-check: _encode() (training) vs _build_feature_vector() (serving)
+# ---------------------------------------------------------------------------
+
+class TestEncodeVsPredictor:
+    """Verify that train_final._encode() and predictor._build_feature_vector()
+    produce identical categorical encodings for all shared columns.
+
+    If these two implementations drift the model is trained on different features
+    than serving produces — a silent, test-invisible bug without this class.
+    """
+
+    @pytest.mark.parametrize("scenario,tier,framework,gpu_id", [
+        ("Offline", "99",   "tensorrt",  "h100_sxm"),
+        ("Server",  "99.9", "vllm",      "mi300x"),
+        ("Offline", "base", "rocm_other", "mi355x"),
+        ("Server",  "99",   "other",     "h200_sxm"),
+        ("Offline", "99",   "vllm",      "mi325x"),
+    ])
+    def test_categorical_encodings_match(self, spec_map, scenario, tier, framework, gpu_id):
+        import math
+        import pandas as pd
+        from src.models.train_final import _encode
+
+        # Get ground-truth feature values from the serving path.
+        feat, _, _ = _build_feature_vector(
+            gpu_spec=spec_map[gpu_id],
+            model_name="gptj",
+            scenario=scenario,
+            accuracy_tier=tier,
+            framework=framework,
+        )
+        amd_arch_gen_raw = feat[FEATURE_COLS.index("amd_arch_gen")]
+
+        # Build a row that mimics what build_training_df produces before _encode().
+        # is_base_tier is computed in build_training_df (not _encode), so inject it directly.
+        row = pd.DataFrame([{
+            "scenario": scenario,
+            "benchmark_accuracy_tier": tier,
+            "framework_family": framework,
+            "amd_arch_gen": None if math.isnan(amd_arch_gen_raw) else amd_arch_gen_raw,
+            "is_base_tier": int(tier == "base"),
+        }])
+        enc = _encode(row).iloc[0]
+
+        # Compare every column that _encode() or build_training_df writes and
+        # _build_feature_vector() computes.
+        for col in ("scenario_offline", "is_base_tier",
+                    "fw_tensorrt", "fw_vllm", "fw_rocm_other", "is_cdna4"):
+            feat_val = feat[FEATURE_COLS.index(col)]
+            enc_val = enc[col]
+            assert feat_val == enc_val, (
+                f"{col} mismatch for gpu={gpu_id} scenario={scenario!r} "
+                f"tier={tier!r} fw={framework!r}: "
+                f"serving={feat_val}, training={enc_val}"
+            )
+
+
+# ---------------------------------------------------------------------------
+# File-guard security tests — GpuPredictor.__init__
+# ---------------------------------------------------------------------------
+
+class TestGpuPredictorFileGuards:
+    """Security tests for the file guards in GpuPredictor.__init__.
+
+    These tests verify three properties:
+    1. Feature-cols mismatch raises ValueError — not AssertionError — so the
+       check is never compiled away under `python -O` / PYTHONOPTIMIZE=1.
+    2. Symlinked artifact files are refused (path-traversal defence).
+    3. Oversized metadata is refused (memory-exhaustion defence).
+    """
+
+    def test_feature_cols_mismatch_raises_valueerror(self, tmp_path):
+        # Both files must pass the file guards (non-symlink, non-oversized)
+        # before the feature-cols check fires.  prophet_v1.json is never
+        # loaded in this test path — the check raises before xgb.load_model.
+        (tmp_path / "feature_metadata.json").write_text(
+            json.dumps({"feature_cols": ["wrong", "cols"], "target": "efficiency_ratio"}),
+            encoding="utf-8",
+        )
+        (tmp_path / "prophet_v1.json").write_text("{}", encoding="utf-8")
+        with pytest.raises(ValueError, match="feature_cols mismatch"):
+            GpuPredictor(model_dir=tmp_path)
+
+    def test_symlink_meta_raises_valueerror(self, tmp_path):
+        link = tmp_path / "feature_metadata.json"
+        link.symlink_to(tmp_path / "nonexistent_target")
+        with pytest.raises(ValueError, match="symlink"):
+            GpuPredictor(model_dir=tmp_path)
+
+    def test_symlink_model_raises_valueerror(self, tmp_path):
+        # meta must pass guards so the loop reaches model_path.
+        (tmp_path / "feature_metadata.json").write_text(
+            json.dumps({"feature_cols": [], "target": ""}), encoding="utf-8"
+        )
+        (tmp_path / "prophet_v1.json").symlink_to(tmp_path / "nonexistent_target")
+        with pytest.raises(ValueError, match="symlink"):
+            GpuPredictor(model_dir=tmp_path)
+
+    def test_oversized_meta_raises_valueerror(self, tmp_path):
+        from src.models.predictor import _MAX_META_BYTES
+        (tmp_path / "feature_metadata.json").write_bytes(b"x" * (_MAX_META_BYTES + 1))
+        (tmp_path / "prophet_v1.json").write_text("{}", encoding="utf-8")
+        with pytest.raises(ValueError, match="too large"):
+            GpuPredictor(model_dir=tmp_path)

tests/test_recommender.py

@@ -0,0 +1,280 @@
+"""
+Unit tests for src/recommend/recommender.py.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from src.models.predictor import GpuPredictor
+from src.recommend.recommender import GpuRecommender, _pareto_frontier
+
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+@pytest.fixture(scope="module")
+def recommender() -> GpuRecommender:
+    pred = GpuPredictor()
+    return GpuRecommender(pred)
+
+
+# ---------------------------------------------------------------------------
+# _pareto_frontier (pure function)
+# ---------------------------------------------------------------------------
+
+class TestParetoFrontier:
+    def _c(self, throughput, cost_efficiency, vram_headroom) -> dict:
+        return {
+            "throughput": throughput,
+            "cost_efficiency": cost_efficiency,
+            "vram_headroom": vram_headroom,
+            "gpu_id": "test",
+        }
+
+    def test_all_on_frontier_when_no_domination(self):
+        # A: best throughput; B: best cost; C: best vram — no one dominates
+        candidates = [
+            self._c(1000, 100, 0.3),   # A
+            self._c(500,  200, 0.2),   # B
+            self._c(600,  150, 0.8),   # C
+        ]
+        frontier, dominated = _pareto_frontier(candidates)
+        assert len(frontier) == 3
+        assert len(dominated) == 0
+
+    def test_dominated_candidate_excluded_from_frontier(self):
+        # D is strictly worse than A on all objectives → dominated
+        candidates = [
+            self._c(1000, 200, 0.8),   # A — best on all
+            self._c(500,  100, 0.3),   # D — dominated by A
+        ]
+        frontier, dominated = _pareto_frontier(candidates)
+        assert len(frontier) == 1
+        assert frontier[0]["throughput"] == 1000
+        assert len(dominated) == 1
+
+    def test_frontier_sorted_by_cost_efficiency_desc(self):
+        candidates = [
+            self._c(800, 100, 0.5),
+            self._c(900, 300, 0.4),
+            self._c(700, 200, 0.6),
+        ]
+        frontier, _ = _pareto_frontier(candidates)
+        efficiencies = [c["cost_efficiency"] for c in frontier]
+        assert efficiencies == sorted(efficiencies, reverse=True)
+
+    def test_empty_input(self):
+        frontier, dominated = _pareto_frontier([])
+        assert frontier == []
+        assert dominated == []
+
+    def test_single_candidate_on_frontier(self):
+        frontier, dominated = _pareto_frontier([self._c(500, 100, 0.5)])
+        assert len(frontier) == 1
+        assert len(dominated) == 0
+
+    def test_none_cost_efficiency_sorts_last(self):
+        # A GPU with cost_efficiency=None (unpriced) must sort last on the
+        # frontier — and must not crash the Pareto comparison with TypeError.
+        # Choose values where neither candidate dominates the other:
+        # first has higher cost_efficiency, second has higher throughput & vram.
+        candidates = [
+            self._c(500, 200, 0.4),   # better cost_efficiency
+            self._c(800, None, 0.8),  # better throughput & vram; unpriced
+        ]
+        frontier, dominated = _pareto_frontier(candidates)
+        assert len(frontier) == 2
+        assert len(dominated) == 0
+        assert frontier[-1]["cost_efficiency"] is None  # None sorts last
+
+    def test_none_cost_efficiency_dominated_by_priced(self):
+        # An unpriced GPU (cost_efficiency=None, treated as -inf) is dominated
+        # by any GPU that beats it on the remaining two objectives.
+        candidates = [
+            self._c(1000, 200, 0.8),  # beats unpriced on all objectives
+            self._c(500,  None, 0.3),  # unpriced; worse on throughput & vram too
+        ]
+        frontier, dominated = _pareto_frontier(candidates)
+        assert len(frontier) == 1
+        assert frontier[0]["cost_efficiency"] == 200
+        assert len(dominated) == 1
+        assert dominated[0]["cost_efficiency"] is None
+
+
+# ---------------------------------------------------------------------------
+# GpuRecommender.recommend
+# ---------------------------------------------------------------------------
+
+class TestGpuRecommender:
+    def test_workload_echoed(self, recommender):
+        result = recommender.recommend(
+            model_name="gptj",
+            scenario="Server",
+            accuracy_tier="base",
+            framework="tensorrt",
+        )
+        wl = result["workload"]
+        assert wl["model_name"] == "gptj"
+        assert wl["scenario"] == "Server"
+        assert wl["accuracy_tier"] == "base"
+        assert wl["framework"] == "tensorrt"
+
+    def test_model_size_in_workload(self, recommender):
+        result = recommender.recommend(model_name="llama2-70b", accuracy_tier="99")
+        # 70B × 1 byte (FP8) = 70 GB
+        assert result["workload"]["model_size_gb"] == pytest.approx(70.0)
+
+    def test_vram_filter_splits_correctly(self, recommender):
+        # llama2-70b at tier 99.9: FP16 for NVIDIA (140 GB), FP8 for AMD (70 GB).
+        # The VRAM filter is vendor-aware, so AMD GPUs use 70 GB for the check.
+        # In-scope GPUs that fit:
+        #   MI355X(288), MI325X(256), MI300X(192) — AMD, 70 GB check
+        #   H200 SXM(141)                          — NVIDIA, 140 GB check (barely fits)
+        # In-scope GPUs that don't fit:
+        #   H100 SXM(80), A100 SXM 80GB(80), L4(24), RTX4090(24) — NVIDIA, 140 GB
+        result = recommender.recommend(model_name="llama2-70b", accuracy_tier="99.9")
+        candidate_ids = {r["gpu_id"] for r in result["frontier"] + result["dominated"]}
+        vram_filtered_ids = {
+            f["gpu_id"] for f in result["filtered"]
+            if "too large" in f.get("reject_reason", "")
+        }
+        # GPUs that must appear in candidates (they fit the 140 GB model)
+        for gid in ("mi355x", "mi325x", "mi300x", "h200_sxm"):
+            assert gid in candidate_ids, f"{gid} should fit 140 GB but is absent from candidates"
+        # GPUs that must be VRAM-filtered (they don't fit 140 GB FP16 model)
+        for gid in ("h100_sxm", "a100_sxm_80gb", "l4", "rtx4090"):
+            assert gid in vram_filtered_ids, f"{gid} should be VRAM-filtered but is missing"
+        # The two sets must be disjoint — no GPU can be both candidate and filtered
+        assert candidate_ids.isdisjoint(vram_filtered_ids)
+
+    def test_405b_fp16_all_filtered(self, recommender):
+        # 405B × 2 bytes = 810 GB; largest GPU is MI355X at 288 GB — none fit
+        result = recommender.recommend(
+            model_name="llama3.1-405b",
+            accuracy_tier="99.9",
+        )
+        assert len(result["frontier"]) == 0
+        assert len(result["dominated"]) == 0
+        assert len(result["filtered"]) > 0
+        for f in result["filtered"]:
+            assert "too large" in f["reject_reason"]
+
+    def test_budget_filter_respected(self, recommender):
+        # Budget $1.50/hr — should filter out expensive GPUs.
+        # All in-scope GPUs have pricing (validated at startup), so
+        # price_per_gpu_hr is never None — the is-None arm was dead code.
+        result = recommender.recommend(
+            model_name="gptj",
+            accuracy_tier="99",
+            budget_per_gpu_hr=1.50,
+        )
+        candidates = result["frontier"] + result["dominated"]
+        # RTX4090 ($0.39) and L4 ($0.44) are both priced under $1.50 and fit
+        # gptj (6 GB). Without this guard the loop below fires zero assertions
+        # if pricing changes so all GPUs exceed the budget.
+        assert len(candidates) > 0, (
+            "Expected at least one GPU within $1.50/hr — RTX4090 ($0.39) and "
+            "L4 ($0.44) are currently below this threshold"
+        )
+        for r in candidates:
+            assert r["price_per_gpu_hr"] <= 1.50
+
+    def test_no_dominated_option_strictly_worse_on_all(self, recommender):
+        result = recommender.recommend(model_name="llama2-70b", accuracy_tier="99")
+        frontier = result["frontier"]
+        dominated = result["dominated"]
+        objectives = ["throughput", "cost_efficiency", "vram_headroom"]
+
+        # Guard: without this assertion the loop below never fires when dominated
+        # is empty, giving zero assertions and false confidence.
+        # llama2-70b FP8 (70 GB) fits 6 in-scope GPUs; with 3 objectives and
+        # diverse pricing ($0.39–$3.99/hr) at least one must be dominated.
+        assert len(dominated) >= 1, (
+            "Expected at least one dominated candidate for llama2-70b fp8 — "
+            "if all 6 VRAM-fitting GPUs are now Pareto-optimal, verify this "
+            "is intentional (pricing or model change) before updating."
+        )
+
+        for dom in dominated:
+            # At least one frontier member must dominate this candidate
+            is_dominated_by_frontier = any(
+                all(f[obj] >= dom[obj] for obj in objectives)
+                and any(f[obj] > dom[obj] for obj in objectives)
+                for f in frontier
+            )
+            assert is_dominated_by_frontier, (
+                f"{dom['gpu_id']} is in dominated list but is not dominated by any frontier member"
+            )
+
+    def test_min_throughput_filter(self, recommender):
+        # High min_throughput should remove low-performing GPUs
+        result = recommender.recommend(
+            model_name="llama2-70b",
+            accuracy_tier="99",
+            min_throughput_tok_per_sec=10_000_000,  # absurdly high
+        )
+        # Nothing should pass
+        assert len(result["frontier"]) == 0
+        assert len(result["dominated"]) == 0
+
+    def test_amd_99_9_vram_headroom_uses_fp8_model_size(self, recommender):
+        # AMD at 99.9 tier uses FP8 (70 GB), not FP16 (140 GB).
+        # vram_headroom for MI300X (192 GB) must be ~63.5% (70 GB model),
+        # not ~27.1% (140 GB model).  This would catch the recommender
+        # computing vram_headroom from the workload-level FP16 model_size_gb
+        # instead of pred["model_size_gb"] which applies the AMD override.
+        result = recommender.recommend(model_name="llama2-70b", accuracy_tier="99.9")
+        candidates = result["frontier"] + result["dominated"]
+        mi300x = next((r for r in candidates if r["gpu_id"] == "mi300x"), None)
+        assert mi300x is not None, "MI300X should be a candidate for llama2-70b tier 99.9"
+        expected_headroom = (192 - 70) / 192
+        got_headroom = mi300x["vram_headroom"]
+        assert got_headroom == pytest.approx(expected_headroom, abs=0.01), (
+            f"MI300X vram_headroom={got_headroom:.3f} — expected ~{expected_headroom:.3f}"
+            " (FP8 70 GB model); got FP16 headroom instead?"
+        )
+
+    def test_frontier_is_pareto_optimal_gptj(self, recommender):
+        # gptj (6.7B, ~6.7 GB FP8) fits all 8 in-scope GPUs — full-field Pareto test
+        # with no VRAM pre-filter. After the MI355X CDNA4 spec update (FP8 5033 TFLOPS,
+        # confirmed from AMD official spec sheet 2026-06-17), MI355X dominates all other
+        # in-scope GPUs on throughput, cost_efficiency, and vram_headroom at current
+        # pricing ($3.50/hr). The test verifies: (1) frontier is non-empty, (2) every
+        # dominated GPU is correctly classified, (3) no frontier member dominates another.
+        result = recommender.recommend(model_name="gptj", accuracy_tier="99")
+        frontier = result["frontier"]
+        dominated = result["dominated"]
+        objectives = ["throughput", "cost_efficiency", "vram_headroom"]
+
+        assert len(frontier) >= 1, "Expected ≥ 1 Pareto-optimal GPU for gptj"
+        assert len(dominated) >= 1, (
+            "Expected ≥ 1 dominated candidate for gptj — "
+            "8 in-scope GPUs with diverse specs; not all can be Pareto-optimal"
+        )
+
+        # Every GPU in dominated must be dominated by at least one frontier member.
+        for dom in dominated:
+            is_dominated_by_frontier = any(
+                all(f[obj] >= dom[obj] for obj in objectives)
+                and any(f[obj] > dom[obj] for obj in objectives)
+                for f in frontier
+            )
+            assert is_dominated_by_frontier, (
+                f"{dom['gpu_id']} is in dominated list but not dominated by any frontier member"
+            )
+
+        # No frontier member may dominate another.
+        for i, a in enumerate(frontier):
+            for j, b in enumerate(frontier):
+                if i == j:
+                    continue
+                dominates = (
+                    all(b[obj] >= a[obj] for obj in objectives)
+                    and any(b[obj] > a[obj] for obj in objectives)
+                )
+                assert not dominates, (
+                    f"Frontier member {b['gpu_id']} dominates {a['gpu_id']} — "
+                    "gptj frontier is not Pareto-optimal"
+                )

tests/test_reliability_gates.py

@@ -0,0 +1,595 @@
+"""
+Reliability-target gate tests  (RT-X.Y IDs from JOURNAL.md §"Reliability Targets").
+
+These are the machine-executable counterpart of the Design Review Checklist.
+Run all gates before merging any PR:
+
+    pytest tests/test_reliability_gates.py -v
+
+Or run only GATE-marked tests across the entire suite:
+
+    pytest -m gate -v
+
+Each test name encodes the RT it protects so a CI failure immediately names
+the violated requirement.
+
+Coverage scope
+--------------
+This file fills the gaps left by the existing per-module test files:
+
+  RT-2.2  MoE models (mixtral-8x7b) use total_params_b for the bandwidth
+          ceiling and compute_params_b (active expert params) for the compute
+          ceiling.  An arg-swap would overestimate mixtral throughput 3.31×.
+          Not covered by any per-module test.
+
+  RT-3.6  Full 20-feature parity (training path vs serving path).
+          test_predictor.py::TestEncodeVsPredictor already covers the 6
+          categorical columns.  This file covers all 13 continuous features
+          (roofline ceilings, model sizes, VRAM ratio, precision TFLOPS)
+          plus vendor_is_amd.
+          A formula divergence there is a silent bug: predictions appear
+          correct but the model is evaluated on features it was never trained on.
+
+  RT-4.1  Pydantic Literal types in schemas.py match VALID_* frozensets in
+          predictor.py.  Both are defined independently; silent divergence
+          means valid API inputs are rejected by the predictor (500) or
+          invalid inputs accepted by Pydantic then fail in the predictor.
+
+  RT-5.3  Model artifact total disk size < 50 MB.
+          No other test verifies this; the limit matters for free-tier HF Spaces
+          deployment (2 vCPU / 16 GB RAM, shared bandwidth).
+"""
+
+from __future__ import annotations
+
+import ast
+import math
+from pathlib import Path
+from typing import get_args
+
+import pandas as pd
+import pytest
+import yaml
+
+from src.api.schemas import AccuracyTier, Framework, Scenario
+from src.data.gpu_spec_db import load_specs
+from src.features.build_features import (
+    BYTES_PER_PARAM,
+    MODEL_PARAMS,
+    TIER_TO_PRECISION,
+    _FRAMEWORK_PATTERNS,
+    _normalize_framework,
+    build_training_df,
+)
+from src.models.predictor import (
+    FEATURE_COLS,
+    VALID_FRAMEWORKS,
+    VALID_SCENARIOS,
+    VALID_TIERS,
+    GpuPredictor,
+    _build_feature_vector,
+)
+from src.recommend.recommender import GpuRecommender
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _nan_match(a: float, b: float) -> bool:
+    """True when both are NaN, or both are finite and within 1e-3 of each other."""
+    a_nan = math.isnan(a)
+    b_nan = math.isnan(b)
+    if a_nan and b_nan:
+        return True
+    if a_nan or b_nan:
+        return False
+    return abs(a - b) < 1e-3
+
+
+def _make_single_row(
+    gpu_alias: str,
+    model_name: str,
+    scenario: str,
+    tier: str,
+    raw_framework: str,
+    throughput: float = 1000.0,
+) -> pd.DataFrame:
+    """One synthetic MLPerf row with every field build_training_df expects."""
+    return pd.DataFrame([{
+        "round":                       "v5.0",
+        "division":                    "closed",
+        "submitter":                   "test",
+        "system_name":                 "test_system",
+        "gpu_name":                    gpu_alias,
+        "num_gpus":                    1,
+        "vram_gb":                     None,
+        "framework":                   raw_framework,
+        "system_type":                 "datacenter",
+        "hw_status":                   "available",
+        "benchmark":                   f"{model_name}-{tier}",
+        "benchmark_base":              model_name,
+        "benchmark_accuracy_tier":     tier,
+        "scenario":                    scenario,
+        "precision":                   None,
+        "tokens_per_sample":           294,
+        "throughput_tokens_per_sec":   throughput,
+        "throughput_tok_per_sec_per_gpu": throughput,
+        "result_valid":                True,
+        "throughput_samples_per_sec":  throughput / 294,
+        "latency_mean_ms":             None,
+        "latency_p99_ms":              None,
+        "ttft_mean_ms":                None,
+        "ttft_p99_ms":                 None,
+        "tpot_mean_ms":                None,
+        "tpot_p99_ms":                 None,
+        "log_path":                    "fake/path",
+    }])
+
+
+# ---------------------------------------------------------------------------
+# RT-2.2  MoE param split — bandwidth ceiling uses total, compute uses active
+# ---------------------------------------------------------------------------
+
+@pytest.mark.gate
+class TestRT22MoEParamSplit:
+    """RT-2.2 GATE: For MoE models, the bandwidth ceiling must use total_params_b
+    (full model footprint in HBM) and the compute ceiling must use compute_params_b
+    (active-expert params per token).
+
+    mixtral-8x7b: total=46.7B, active=14.1B (ratio 3.31×).  Swapping these args
+    in roofline_ceilings() would overestimate mixtral compute throughput 3.31×
+    and underestimate the bandwidth ceiling by the same factor — corrupting both
+    the feature values and the hard roofline cap in predict().
+    """
+
+    @pytest.fixture(scope="class")
+    def mi300x_spec(self) -> dict:
+        return {s["id"]: s for s in load_specs()}["mi300x"]
+
+    def test_mixtral_total_ne_active_params(self) -> None:
+        total_b, active_b = MODEL_PARAMS["mixtral-8x7b"]
+        assert total_b != active_b, "mixtral-8x7b must have distinct total and active params"
+        assert total_b == pytest.approx(46.7, abs=0.1)
+        assert active_b == pytest.approx(14.1, abs=0.1)
+
+    def test_bw_ceiling_uses_total_params(self, mi300x_spec: dict) -> None:
+        """Back-calculate which param count produced the BW ceiling in the serving path."""
+        total_b, active_b = MODEL_PARAMS["mixtral-8x7b"]
+        precision = TIER_TO_PRECISION["99"]   # fp8
+        bpp = BYTES_PER_PARAM[precision]
+        hbm_bw = mi300x_spec["hbm_bandwidth_tbps"]
+
+        feats, _, _ = _build_feature_vector(
+            gpu_spec=mi300x_spec,
+            model_name="mixtral-8x7b",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="vllm",
+        )
+        bw_ceil = feats[FEATURE_COLS.index("bandwidth_ceiling_tok_per_sec")]
+
+        # bw_ceil = (hbm_bw * 1e12) / (params * 1e9 * bpp)  ⟹  params = …
+        implied_params = (hbm_bw * 1e12) / (bw_ceil * 1e9 * bpp)
+        assert implied_params == pytest.approx(total_b, rel=1e-3), (
+            f"RT-2.2: BW ceiling implies {implied_params:.2f}B params in model footprint; "
+            f"expected total_params_b={total_b}B.  "
+            f"Compute ceiling likely received total params instead of active params."
+        )
+
+    def test_compute_ceiling_uses_active_params(self, mi300x_spec: dict) -> None:
+        """Back-calculate which param count produced the compute ceiling in the serving path."""
+        total_b, active_b = MODEL_PARAMS["mixtral-8x7b"]
+        precision = TIER_TO_PRECISION["99"]
+        pt = mi300x_spec.get("peak_tflops") or {}
+        peak_tflops = pt.get(precision) or pt.get("fp16")
+
+        feats, _, _ = _build_feature_vector(
+            gpu_spec=mi300x_spec,
+            model_name="mixtral-8x7b",
+            scenario="Offline",
+            accuracy_tier="99",
+            framework="vllm",
+        )
+        compute_ceil = feats[FEATURE_COLS.index("compute_ceiling_tok_per_sec")]
+
+        # compute_ceil = (peak_tflops * 1e12) / (2 * params * 1e9)  ⟹  params = …
+        implied_params = (peak_tflops * 1e12) / (2.0 * compute_ceil * 1e9)
+        assert implied_params == pytest.approx(active_b, rel=1e-3), (
+            f"RT-2.2: Compute ceiling implies {implied_params:.2f}B active params; "
+            f"expected compute_params_b={active_b}B.  "
+            f"Bandwidth ceiling likely received active params instead of total params."
+        )
+
+
+# ---------------------------------------------------------------------------
+# RT-4.1  Schema/predictor agreement — Literal types match VALID_* frozensets
+# ---------------------------------------------------------------------------
+
+@pytest.mark.gate
+class TestRT41SchemaPredictorAgreement:
+    """RT-4.1 GATE: Pydantic Literal types in src/api/schemas.py must exactly
+    match the VALID_* frozensets in src/models/predictor.py.
+
+    Both are defined independently.  If schemas.py accepts a value that
+    VALID_* rejects, a well-formed API request causes a 500 (predictor raises
+    ValueError).  If VALID_* accepts a value schemas.py rejects, a valid
+    predictor call cannot be reached via the API at all.
+    """
+
+    def test_scenario_literals_match_valid_scenarios(self) -> None:
+        assert set(get_args(Scenario)) == VALID_SCENARIOS, (
+            f"RT-4.1: schemas.Scenario={set(get_args(Scenario))} "
+            f"!= predictor.VALID_SCENARIOS={VALID_SCENARIOS}"
+        )
+
+    def test_accuracy_tier_literals_match_valid_tiers(self) -> None:
+        assert set(get_args(AccuracyTier)) == VALID_TIERS, (
+            f"RT-4.1: schemas.AccuracyTier={set(get_args(AccuracyTier))} "
+            f"!= predictor.VALID_TIERS={VALID_TIERS}"
+        )
+
+    def test_framework_literals_match_valid_frameworks(self) -> None:
+        assert set(get_args(Framework)) == VALID_FRAMEWORKS, (
+            f"RT-4.1: schemas.Framework={set(get_args(Framework))} "
+            f"!= predictor.VALID_FRAMEWORKS={VALID_FRAMEWORKS}"
+        )
+
+
+# ---------------------------------------------------------------------------
+# RT-3.6  Feature vector parity — training path vs serving path
+# ---------------------------------------------------------------------------
+
+@pytest.mark.gate
+class TestRT36FeatureVectorParity:
+    """RT-3.6 GATE: all 20 FEATURE_COLS values match between build_training_df
+    (training path) and _build_feature_vector (serving path), with one
+    intentional exception noted below.
+
+    Column groups verified
+    ----------------------
+    Continuous (this file):
+        gpu_hbm_bandwidth_tbps, gpu_vram_gb, peak_tflops_selected,
+        compute_ceiling_tok_per_sec, bandwidth_ceiling_tok_per_sec,
+        model_total_params_b, model_compute_params_b, model_size_gb,
+        model_to_vram_ratio, bytes_per_param, vendor_is_amd,
+        nvidia_arch_gen, amd_arch_gen
+
+    Categorical (test_predictor.py::TestBuildFeatureVector):
+        scenario_offline, is_base_tier, fw_tensorrt, fw_vllm,
+        fw_rocm_other, is_cdna4
+
+    Intentionally excluded from parity check:
+        mlperf_round_num — training uses the actual submission round ordinal;
+        serving always uses _SERVING_ROUND (max(ROUND_ORDINAL.values())) to
+        predict for the most mature software stack.  Verified separately in
+        test_predictor.py::TestBuildFeatureVector::test_mlperf_round_num_serving_uses_latest.
+    """
+
+    # (gpu_alias, gpu_id, model_name, scenario, tier, raw_framework)
+    # gpu_alias must match a known alias in data/gpu_specs.yaml.
+    # raw_framework is a realistic MLPerf string; _normalize_framework maps it
+    # to the family label that _build_feature_vector expects.
+    CASES = [
+        # NVIDIA Hopper, FP8 tier (99 → fp8, falls back for GPUs without fp8)
+        (
+            "NVIDIA H200-SXM-141GB",          "h200_sxm",  "llama2-70b",
+            "Offline", "99",   "TensorRT 10.2.0",
+        ),
+        # AMD CDNA3, FP8 tier (99.9 → fp8 for AMD; parity check verifies both
+        # paths apply the vendor override consistently)
+        (
+            "AMD Instinct MI300X 192GB HBM3",  "mi300x",    "gptj",
+            "Server",  "99.9", "vLLM 0.4.3+rocm614",
+        ),
+        # AMD CDNA3, BF16 tier (base → bf16); MoE model (active ≠ total params)
+        (
+            "AMD Instinct MI325X",             "mi325x",    "mixtral-8x7b",
+            "Offline", "99",   "ROCm 7.0",
+        ),
+        # AMD CDNA4 — only GPU where amd_arch_gen=2 and is_cdna4=1.
+        # A divergence in CDNA4 arch-ordinal encoding (training vs serving)
+        # cannot be detected by the 3 cases above (all use CDNA3 or NVIDIA).
+        (
+            "AMD Instinct MI355X 288GB HBM3e", "mi355x",    "llama2-70b",
+            "Offline", "99",   "ROCm 7.0",
+        ),
+        # NVIDIA at tier 99.9 — selects fp16 TFLOPS (NOT fp8, no AMD override).
+        # The four cases above are all tier 99 (fp8) or AMD 99.9 (fp8 override);
+        # without this case a divergence in NVIDIA fp16 TFLOPS selection between
+        # the training and serving paths would pass the gate undetected.
+        (
+            "NVIDIA H100-SXM-80GB",            "h100_sxm", "gptj",
+            "Server",  "99.9", "TensorRT 10.4.0",
+        ),
+    ]
+
+    @pytest.fixture(scope="class")
+    def spec_map(self) -> dict:
+        return {s["id"]: s for s in load_specs()}
+
+    @pytest.mark.parametrize(
+        "gpu_alias, gpu_id, model_name, scenario, tier, raw_fw",
+        CASES,
+    )
+    def test_continuous_features_match(
+        self, spec_map, gpu_alias, gpu_id, model_name, scenario, tier, raw_fw,
+    ):
+        """RT-3.6 GATE: every continuous FEATURE_COL has the same value in
+        both paths.  A mismatch means the model evaluates features at serving
+        time that it was never trained on.
+        """
+        # ── Training path ─────────────────────────────────────────────────
+        raw_df = _make_single_row(gpu_alias, model_name, scenario, tier, raw_fw)
+        feat_df = build_training_df(raw_df)
+        assert len(feat_df) == 1, (
+            f"build_training_df dropped the row — check gpu_alias={gpu_alias!r} "
+            f"and benchmark_base={model_name!r}"
+        )
+        train_row = feat_df.iloc[0]
+
+        # ── Serving path ──────────────────────────────────────────────────
+        fw_family = _normalize_framework(raw_fw)
+        serving_feats, _, _ = _build_feature_vector(
+            gpu_spec=spec_map[gpu_id],
+            model_name=model_name,
+            scenario=scenario,
+            accuracy_tier=tier,
+            framework=fw_family,
+        )
+
+        # ── Compare continuous columns ────────────────────────────────────
+        # mlperf_round_num is excluded: serving always uses the latest round
+        # ordinal (_SERVING_ROUND); training uses the actual submission round.
+        # This divergence is intentional — see class docstring.
+        continuous_cols = [
+            c for c in FEATURE_COLS
+            if c not in {
+                "scenario_offline", "is_base_tier",
+                "fw_tensorrt", "fw_vllm", "fw_rocm_other", "is_cdna4",
+                "mlperf_round_num",
+            }
+        ]
+        mismatches = []
+        for col in continuous_cols:
+            idx = FEATURE_COLS.index(col)
+            sv = float(serving_feats[idx])
+            tv_raw = train_row[col]
+            tv = float(tv_raw) if not pd.isna(tv_raw) else float("nan")
+
+            if not _nan_match(sv, tv):
+                mismatches.append(
+                    f"  {col:<40} serving={sv:.6g}  training={tv:.6g}"
+                )
+
+        assert not mismatches, (
+            f"RT-3.6: feature mismatch for gpu={gpu_id!r} model={model_name!r} "
+            f"scenario={scenario!r} tier={tier!r}:\n"
+            + "\n".join(mismatches)
+        )
+
+
+# ---------------------------------------------------------------------------
+# RT-5.3  Model artifact total disk size < 50 MB
+# ---------------------------------------------------------------------------
+
+_MODEL_DIR = Path("data/models")
+_DISK_LIMIT_MB = 50
+
+
+@pytest.mark.gate
+class TestRT53ModelDiskSize:
+    """RT-5.3 GATE: data/models/ total size is < 50 MB.
+
+    Free-tier Hugging Face Spaces (the deploy target) has shared storage
+    and slow cold-start I/O.  50 MB is the deployment budget.
+    """
+
+    def test_model_dir_exists(self) -> None:
+        assert _MODEL_DIR.is_dir(), (
+            f"RT-5.3: {_MODEL_DIR} not found — run train_final.py to generate artifacts"
+        )
+
+    def test_total_size_under_limit(self) -> None:
+        total_bytes = sum(f.stat().st_size for f in _MODEL_DIR.rglob("*") if f.is_file())
+        total_mb = total_bytes / (1024 ** 2)
+        assert total_mb < _DISK_LIMIT_MB, (
+            f"RT-5.3: data/models/ is {total_mb:.1f} MB — exceeds {_DISK_LIMIT_MB} MB limit. "
+            "Trim model artifacts before deploying to HF Spaces."
+        )
+
+    def test_required_artifacts_present(self) -> None:
+        required = ["prophet_v1.json", "feature_metadata.json"]
+        for name in required:
+            assert (_MODEL_DIR / name).exists(), (
+                f"RT-5.3: required artifact {name} is missing from {_MODEL_DIR}"
+            )
+
+
+# ---------------------------------------------------------------------------
+# RT-1.3  Pricing coverage — all in-scope GPUs have pricing entries
+# ---------------------------------------------------------------------------
+
+_PRICING_PATH = Path("data/pricing.yaml")
+
+
+@pytest.mark.gate
+class TestRT13PricingCoverage:
+    """RT-1.3 GATE: every GPU with in_model_scope=True in gpu_specs.yaml has a
+    corresponding entry in data/pricing.yaml.
+
+    GpuRecommender.__init__() raises ValueError for missing pricing, but that
+    failure is only visible at runtime (startup).  This gate catches the gap
+    before the Docker image is built or before AMD Dev Cloud session starts.
+
+    Adding a new in-scope GPU without a pricing entry causes cost_efficiency=None,
+    which raises TypeError in _pareto_frontier's sort — silently breaking all
+    recommendation responses rather than failing cleanly at startup.
+    """
+
+    def test_pricing_file_exists(self) -> None:
+        assert _PRICING_PATH.is_file(), (
+            f"RT-1.3: {_PRICING_PATH} not found — create data/pricing.yaml before deployment"
+        )
+
+    def test_all_in_scope_gpus_have_pricing(self) -> None:
+        with _PRICING_PATH.open() as f:
+            pricing: dict = yaml.safe_load(f).get("pricing", {})
+        in_scope_ids = [s["id"] for s in load_specs() if s.get("in_model_scope")]
+        assert in_scope_ids, (
+            "RT-1.3: no in-scope GPUs found — check gpu_specs.yaml in_model_scope flags"
+        )
+        missing = [gid for gid in in_scope_ids if gid not in pricing]
+        assert not missing, (
+            f"RT-1.3: pricing.yaml missing entries for in-scope GPUs: {missing}. "
+            "GpuRecommender.__init__() will raise ValueError at startup."
+        )
+
+
+# ---------------------------------------------------------------------------
+# RT-3.7  AMD FP8 override — three-way consistency gate
+# ---------------------------------------------------------------------------
+
+@pytest.mark.gate
+class TestRT37AmdFp8Override:
+    """RT-3.7 GATE: the AMD 99.9-tier FP8 override is applied consistently in
+    build_training_df (training), _build_feature_vector (serving), and
+    GpuRecommender._gpu_model_size_gb (VRAM pre-filter).
+
+    A regression in any one of the three paths causes:
+      - Training: efficiency_ratio > 1.0 for MI355X 99.9-tier rows
+        (FP16 ceiling used instead of FP8, ceiling violations in 22/50 rows,
+        reducing MI355X LOGO ρ from ~0.60 to ~0.40)
+      - Serving/recommender: wrong VRAM fit decision for llama2-70b at 99.9 tier
+        on MI300X (140 GB FP16 model exceeds 192 GB VRAM by ~27%, so MI300X
+        would be incorrectly rejected despite fitting comfortably as 70 GB FP8)
+
+    Verified end-to-end via recommender: if vram_headroom ≈ 0.635 (FP8 70 GB),
+    all three paths agree.  If headroom ≈ 0.271 (FP16 140 GB), at least one
+    path has drifted.
+    """
+
+    @pytest.fixture(scope="class")
+    def recommender(self) -> GpuRecommender:
+        return GpuRecommender(GpuPredictor())
+
+    def test_mi300x_vram_headroom_uses_fp8_not_fp16(self, recommender) -> None:
+        result = recommender.recommend(model_name="llama2-70b", accuracy_tier="99.9")
+        candidates = result["frontier"] + result["dominated"]
+        mi300x = next((r for r in candidates if r["gpu_id"] == "mi300x"), None)
+        assert mi300x is not None, (
+            "RT-3.7: MI300X absent from llama2-70b tier-99.9 candidates — "
+            "VRAM filter may be applying FP16 model size (140 GB) instead of FP8 (70 GB), "
+            "causing MI300X (192 GB) to be incorrectly rejected."
+        )
+        fp8_model_gb = 70.0  # 70B params × 1 byte/param (FP8)
+        expected_headroom = (192.0 - fp8_model_gb) / 192.0   # ≈ 0.635
+        assert mi300x["vram_headroom"] == pytest.approx(expected_headroom, abs=0.01), (
+            f"RT-3.7: MI300X vram_headroom={mi300x['vram_headroom']:.3f}, "
+            f"expected {expected_headroom:.3f} (FP8 70 GB model). "
+            "A value near 0.271 indicates FP16 (140 GB) is being used instead."
+        )
+
+
+# ---------------------------------------------------------------------------
+# RT-3.8  Framework normalization — pattern outputs ⊆ VALID_FRAMEWORKS
+# ---------------------------------------------------------------------------
+
+@pytest.mark.gate
+class TestRT38FrameworkNormalization:
+    """RT-3.8 GATE: every label returned by a _FRAMEWORK_PATTERNS entry is in
+    VALID_FRAMEWORKS, and the no-match fallback ("other") is also in VALID_FRAMEWORKS.
+
+    If a new framework pattern is added to _FRAMEWORK_PATTERNS with a label that
+    is NOT in VALID_FRAMEWORKS, that framework family enters the training data but
+    can never be requested via the API — predictor._validate() raises ValueError
+    on what appears to be a perfectly valid input, causing silent 500 errors for
+    any workload that uses that framework.
+    """
+
+    def test_all_pattern_outputs_in_valid_frameworks(self) -> None:
+        labels_from_patterns = {label for _, label in _FRAMEWORK_PATTERNS}
+        unknown_labels = labels_from_patterns - VALID_FRAMEWORKS
+        assert not unknown_labels, (
+            f"RT-3.8: _FRAMEWORK_PATTERNS returns labels not in VALID_FRAMEWORKS: "
+            f"{unknown_labels}. Add to predictor.VALID_FRAMEWORKS or remove from patterns."
+        )
+
+    def test_fallback_other_in_valid_frameworks(self) -> None:
+        fallback = _normalize_framework("some_completely_unknown_framework_xyz")
+        assert fallback == "other", (
+            f"RT-3.8: no-match fallback should be 'other', got {fallback!r}. "
+            "The fallback must be in VALID_FRAMEWORKS so unrecognized frameworks "
+            "can be requested via the API without a 422 validation error."
+        )
+        assert "other" in VALID_FRAMEWORKS, (
+            "RT-3.8: 'other' is _normalize_framework's no-match return value "
+            "but is not in VALID_FRAMEWORKS — unrecognized frameworks fail predictor._validate()."
+        )
+
+
+# ---------------------------------------------------------------------------
+# RT-6.1  Calibration runner/merge script CSV compatibility
+# ---------------------------------------------------------------------------
+
+_RUNNER_PATH = Path("benchmarks/run_mi300x_calibration.py")
+
+# Fields that merge_calibration_rows.py accesses directly from CSV rows
+# (r["field"] or r.get("field", ...)).  Update whenever merge script adds
+# a new field access.
+_MERGE_REQUIRED_FIELDS: frozenset[str] = frozenset({
+    "round_tag",
+    "gpu_name",
+    "benchmark_base",
+    "benchmark_accuracy_tier",
+    "scenario",
+    "throughput_tok_per_sec",
+    "vllm_version",
+    # DataFrame subscript access on line 64 of merge_calibration_rows.py:
+    # failed[["benchmark_base", "precision_used", "scenario"]].to_dict("records")
+    # Not a .get() — raises KeyError if the column is absent from the CSV.
+    "precision_used",
+})
+
+
+def _parse_runner_csv_fields() -> frozenset[str]:
+    """Extract CSV_FIELDS list from runner script via AST — avoids importing
+    the script directly, which would fail without vllm installed."""
+    tree = ast.parse(_RUNNER_PATH.read_text())
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Assign):
+            for target in node.targets:
+                if isinstance(target, ast.Name) and target.id == "CSV_FIELDS":
+                    if isinstance(node.value, ast.List):
+                        return frozenset(
+                            elt.value
+                            for elt in node.value.elts
+                            if isinstance(elt, ast.Constant) and isinstance(elt.value, str)
+                        )
+    raise RuntimeError("CSV_FIELDS constant not found in benchmarks/run_mi300x_calibration.py")
+
+
+@pytest.mark.gate
+class TestRT61RunnerMergeCompatibility:
+    """RT-6.1 GATE: benchmarks/run_mi300x_calibration.py CSV_FIELDS is a superset
+    of the columns that benchmarks/merge_calibration_rows.py reads from each row.
+
+    Drift between the runner's CSV schema and the merge script's column accesses
+    causes KeyError (or silent None via .get()) when processing real AMD Dev Cloud
+    results — after the 25-hr credit clock has already been consumed.  This gate
+    runs locally before starting the timed session.
+    """
+
+    def test_runner_script_exists(self) -> None:
+        assert _RUNNER_PATH.is_file(), (
+            f"RT-6.1: {_RUNNER_PATH} not found — calibration runner script is missing"
+        )
+
+    def test_runner_csv_fields_cover_merge_required(self) -> None:
+        runner_fields = _parse_runner_csv_fields()
+        missing = _MERGE_REQUIRED_FIELDS - runner_fields
+        assert not missing, (
+            f"RT-6.1: merge_calibration_rows.py accesses fields not in runner CSV_FIELDS: "
+            f"{missing}. Add these to CSV_FIELDS in run_mi300x_calibration.py, or "
+            "remove the access from merge_calibration_rows.py."
+        )