| # Benchmarks |
|
|
| The numbers we are chasing on the hero workload, and the framework comparison that goes in the README. |
|
|
| ## Hero workload |
|
|
| **Qwen3-8B SFT, 1× MI300X, bs=8, seq=4096, BF16, AdamW, 1B tokens.** |
|
|
| | Metric | Target | Why | |
| |--------------------------|-----------------|----------------------------------------------------------------| |
| | Throughput | **>15 000 tok/s** | Comparable to AMD's published Llama-3.1-8B numbers. | |
| | MFU | **>40 %** | Floor for "MI300X is being exercised, not idled." | |
| | Time to eval-loss = 1.5 | **<90 minutes** | Lets the demo video show the loop converging in real time. | |
| | Total cost | **<$3** | $1.99 / hr × 1 GPU × ~1.5 hr × safety margin. | |
| | Peak HBM | ~80 GB | Headroom on MI300X's 192 GB; impossible on H100 80 GB. | |
|
|
| Hit those four and the cost slide writes itself: **MI300X $1.99/hr × 1 GPU × 1.5 hr ≈ $3** versus **H100 $4/hr × 2 GPUs × 4 hr ≈ $32** — 4× cheaper for the same workload, and the H100 baseline can't even fit BF16 at this batch/seq combo without quantization. |
|
|
| ## H100 cost baseline |
|
|
| The argument the judges remember is "MI300X is 4× cheaper for this exact workload." The cost numbers come from public list prices and need to hold up under questioning. |
|
|
| | GPU | $/hr | Memory | Qwen3-8B BF16 bs=8 seq=4096 | Cost for 1B tokens | |
| |-----------|-------|---------|-----------------------------|--------------------| |
| | H100 80 GB | $4.00 | 80 GB | OOM unless bs/seq cut | ~$32 (2× GPUs, 4 hr, FP8 fallback) | |
| | H200 141 GB | $6.00 | 141 GB | Fits, ~12k tok/s | ~$24 (1× GPU, 4 hr) | |
| | MI300X 192 GB | $1.99 | 192 GB | Fits with headroom, >15k tok/s | **<$3** (1× GPU, 1.5 hr) | |
|
|
| ## Framework comparison (the README differentiator) |
|
|
| This is the table the README prints. mindxtrain is the only row with all seven cells filled — that is the elevator pitch. |
|
|
| | Framework | One-cmd ROCm 7.2.1 install | MI300X auto-tune | Qwen3.6 day-zero | FP8 via Quark | x402 micropayments | Decentralized fallback | Training-receipt manifest | |
| |----------------|----------------------------|------------------|------------------|---------------|--------------------|------------------------|---------------------------| |
| | Axolotl | ⚠ (community fork) | ✗ | ✓ | △ (torchao) | ✗ | ✗ | ✗ | |
| | LLaMA-Factory | ✓ (AMD tutorial) | ✗ | ✓ | △ | ✗ | ✗ | ✗ | |
| | Unsloth | ✓ (OneClickAMD) | ✗ | △ (single-GPU) | ✗ | ✗ | ✗ | ✗ | |
| | torchtune | ✓ (AMD CI) | ✗ | ✗ (no recipe) | △ | ✗ | ✗ | ✗ | |
| | Primus | ✓ (`rocm/primus:v26.2`) | ✗ | ✗ (pretrain only) | ✓ | ✗ | ✗ | ✗ | |
| | **mindxtrain** | **✓** | **✓ (60s AOT)** | **✓** | **✓** | **✓ (Algorand)** | **✓ (Bacalhau/Akash)** | **✓ (BLAKE3 + INFT)** | |
|
|
| Legend: ✓ = supported · ⚠ = supported via community fork · △ = partial / opt-in · ✗ = not supported. |
|
|
| ## Capturing the numbers |
|
|
| The `mindxtrain` CLI emits structured logs that map onto the metrics above. The output tree: |
|
|
| ``` |
| runs/<run_id>/ |
| ├── config.yaml # input, BLAKE3-hashed in the manifest |
| ├── autotune_plan.json # the AOT plan (the differentiator) |
| ├── train.log # accelerate stdout/stderr |
| ├── metrics.jsonl # one record per logging step: tok_per_s, mfu, hbm_gb, watts |
| ├── checkpoint/ # HF safetensors + tokenizer, BLAKE3-hashed |
| ├── quantized/ # Quark FP8 PTPC, vLLM-loadable |
| ├── eval.json # lm-evaluation-harness output |
| └── manifest.json # mindxtrain.provenance.Manifest with BLAKE3 hashes |
| ``` |
|
|
| `metrics.jsonl` is the source of truth for the benchmark numbers. The cost slide is a one-liner over that file: average `tok_per_s` × seconds × $1.99 / 3600. |
|
|
| ## Regression detection |
|
|
| `eval.regression.threshold_pct: -1.0` in every recipe means **fail the run if any benchmark task drops more than 1 percentage point** versus the base model baseline. That keeps a fine-tune that improves the target distribution but breaks general capability from being silently published. The baseline JSON is computed once per base model and cached alongside the run; comparison happens via `mindxtrain.eval.persona_regression.regression_score` and `mindxtrain.eval.agenda_regression.regression_score`. |
|
|
| ## What's not measured (yet) |
|
|
| - Energy (kWh per training run) — `mindxtrain.operator.telemetry.energy.sample_power_w` wraps `rocm-smi --showpower` (returns 0.0 W gracefully on a CPU dev box). MI300X power baseline is ~750 W under load; a 90-minute run is ~1.1 kWh. Telemetry collection into `metrics.jsonl` is wired but the dashboard integration is post-hackathon work. |
| - Multi-node throughput — out of hackathon scope; the `mindxtrain receipt` manifest accommodates it (`hardware.gpus` field), and the autotune `rccl_probe` is the entry point for the multi-node version. |
|
|
