gpu/single-gpu/home_benjamin_hdbasin/code/README.md

# HD-BasinFlow Experiment Suite

This repository implements the validation plan in `hd_basinflow_experiment_instructions.pdf`.

The suite includes:

- A runnable HD-BasinFlow optimizer for objectives on `[0, 1]^d`.
- Exact synthetic gradients and local weighted ridge gradient estimation support.
- Active-subspace estimation, kNN flow graph construction, basin certificates, repair sampling, and ablations.
- Synthetic benchmarks and baseline comparisons.
- Required CSV logs, processed metrics, plots, and `reports/experiment_report.md`.
- A real-data status probe for the Hugging Face datasets named in the plan. Real ML runs are not silently faked.
- A target-savings benchmark mode for the commercial claim: comparable validation quality with less compute.

## Commercial Validation Path

The narrow commercial claim is:

> HD-BasinFlow can reach comparable validation quality on selected HPO/fine-tuning workloads while using less compute than common search baselines.

The current pass/fail rule is intentionally conservative:

- Compare against Random Search, Sobol/quasi-random, Optuna TPE, and ASHA/Successive Halving.
- Treat HD-BasinFlow as passing a comparison only when median final loss is within 1% of the baseline median final loss.
- Require the target-savings policy to reach that quality with at least 20% fewer evaluations.
- Treat 15-30% savings as the first commercial operating band.
- Report failures and underpowered rows instead of hiding them.

Build the target-savings report from the latest combined raw evaluations:

```bash
python3 scripts/build_target_savings_report.py \
  --input runs/combined_report/raw/evaluations.csv \
  --outdir runs/target_savings_benchmark
```

That produces:

- `target_savings_decisions.csv`: per-dataset/per-baseline pass-fail table.
- `target_savings_summary.csv`: aggregate competitor comparison.
- `target_savings_report.md`: technical report.
- `target_savings_operating_point.pdf`: plot for the report/deck.
- `customer_deck_outline.md`: short commercial deck outline.
- `demo_target_savings.py`: minimal demo command that prints decisions.

Run a clean A100 commercial benchmark suite:

```bash
bash scripts/run_commercial_benchmark.sh runs/commercial_benchmark_a100
```

This stages repeated image, transformer, classic NLP, and tabular workloads on
Fashion-MNIST, CIFAR-10, SST-2, IMDB, AG News, and tabular credit. The script
then aggregates results and rebuilds the target-savings report.

Run the stronger 20-seed internal validation gate with GPU telemetry:

```bash
bash scripts/run_stronger_commercial_benchmark.sh runs/commercial_benchmark_a100_20seed
```

That writes `gpu_telemetry.csv`, `run.log`, per-workload outputs, a combined
report, and a fresh target-savings package. Use this before making external
sales claims.

The commercial/internal whitepaper is:

```bash
make whitepaper
```

Output:

- `paper/hd_basinflow_paper.tex`
- `paper/build/hd_basinflow_paper.pdf`

Convenience targets:

```bash
make test
make report
make report-strong
make figures
make figures-strong
make whitepaper
make all-commercial
make all-commercial-strong
```

## Run

On the remote machine:

```bash
cd ~/hdbasin
python3 run_experiments.py --synthetic --seeds 5 --budget 80 --outdir runs/smoke
```

For a fuller synthetic run:

```bash
python3 run_experiments.py --synthetic --seeds 20 --budget 160 --outdir runs/full
```

Probe real dataset availability without running expensive training:

```bash
python3 run_experiments.py --real-status --outdir runs/real_status
```

Run the lightweight AG News real-data HPO workload. `--seeds` repeats real
workloads with distinct dataset shuffles and optimizer seeds:

```bash
python3 run_experiments.py --ag-news --seeds 3 --budget 8 --ablations full --ag-train-size 400 --ag-val-size 160 --outdir runs/ag_news_diagnostic
```

Run a tiny AG News curve-logging smoke test:

```bash
python3 run_experiments.py --ag-news --seeds 1 --budget 2 --ablations full --ag-train-size 80 --ag-val-size 40 --outdir runs/ag_news_curves_smoke
```

Run the full-size AG News TF-IDF/logistic-regression supplement:

```bash
python3 run_experiments.py --ag-news --seeds 1 --seed-offset 100 --budget 12 --ablations full --ag-train-size 100000 --ag-val-size 20000 --outdir runs/ag_news_full_1x12
```

Run the lightweight Fashion-MNIST tiny-CNN workload on GPU when available:

```bash
python3 run_experiments.py --fashion-mnist --seeds 3 --budget 6 --ablations full --fashion-train-size 256 --fashion-val-size 128 --fashion-epochs 1 --outdir runs/fashion_mnist_smoke
```

Run the lightweight CIFAR-10 tiny-CNN workload on GPU when available:

```bash
python3 run_experiments.py --cifar10 --seeds 3 --budget 6 --ablations full --cifar-train-size 256 --cifar-val-size 128 --cifar-epochs 1 --outdir runs/cifar10_smoke
```

Run the lightweight tabular credit workload:

```bash
python3 run_experiments.py --tabular-credit --seeds 3 --budget 8 --ablations full --tabular-train-size 800 --tabular-val-size 240 --outdir runs/tabular_credit_smoke
```

Extend tabular evidence without duplicating the first 20 seeds:

```bash
python3 run_experiments.py --tabular-credit --seeds 80 --seed-offset 20 --budget 8 --ablations full --tabular-train-size 800 --tabular-val-size 240 --outdir runs/tabular_credit_extra_80x8
```

Run the full-available tabular credit supplement:

```bash
python3 run_experiments.py --tabular-credit --seeds 1 --seed-offset 100 --budget 12 --ablations full --tabular-train-size 13000 --tabular-val-size 3714 --outdir runs/tabular_credit_full_1x12
```

Run the lightweight SST-2 DistilBERT workload on GPU when available:

```bash
python3 run_experiments.py --sst2 --seeds 3 --budget 3 --ablations full --sst2-train-size 48 --sst2-val-size 48 --outdir runs/sst2_smoke
```

Run the lightweight IMDB DistilBERT workload on GPU when available:

```bash
python3 run_experiments.py --imdb --seeds 3 --budget 3 --ablations full --imdb-train-size 48 --imdb-val-size 48 --outdir runs/imdb_smoke
```

Run a synthetic dimension-scaling sweep:

```bash
python3 run_experiments.py --dimension-scaling --seeds 1 --budget 16 --scaling-dims 2,5,10,20 --scaling-active-dims 2,5 --outdir runs/dimension_scaling_smoke
```

Run the current 20-seed fair-seed synthetic statistics pass. HD-BasinFlow
ablations use the same optimizer seed within each objective/outer-seed pair so
the comparison is not confounded by different initial random designs:

```bash
python3 run_experiments.py --synthetic --seeds 20 --budget 80 --ablations full,NoFlow,NoActive,NoShell,NoCurvature,NoLeakage,NoRepair,NoRandom,MidpointOnly,LossOnly --outdir runs/synthetic_fair_20x80
```

Run a synthetic Sobol/quasi-random-only supplement if you want to isolate that
baseline outside the fair-seed synthetic run:

```bash
python3 run_experiments.py --synthetic --seeds 20 --budget 80 --baselines sobol --skip-hdbasinflow --outdir runs/sobol_synthetic_20x80
```

Run the current 20-seed synthetic compute-allocation supplement:

```bash
python3 run_experiments.py --synthetic --seeds 20 --budget 80 --ablations full --baselines asha --outdir runs/allocation_synthetic_20x80
```

Run the current extended dimension-scaling pass:

```bash
python3 run_experiments.py --dimension-scaling --seeds 3 --budget 32 --scaling-dims 2,5,10,20,50,100 --scaling-active-dims 2,5,10,20 --outdir runs/dimension_scaling_extended
```

Run the current dimension-scaling Sobol/quasi-random supplement:

```bash
python3 run_experiments.py --dimension-scaling --seeds 3 --budget 32 --scaling-dims 2,5,10,20,50,100 --scaling-active-dims 2,5,10,20 --baselines sobol --skip-hdbasinflow --outdir runs/sobol_dimension_scaling_3x32
```

Run the current dimension-scaling active-subspace-error supplement:

```bash
python3 run_experiments.py --dimension-scaling --seeds 3 --budget 32 --scaling-dims 2,5,10,20,50,100 --scaling-active-dims 2,5,10,20 --baselines random,sobol --outdir runs/subspace_dimension_scaling_3x32
```

Run the current 20-seed classic real-data pass:

```bash
python3 run_experiments.py --ag-news --fashion-mnist --cifar10 --tabular-credit --seeds 20 --budget 8 --ablations full --ag-train-size 400 --ag-val-size 160 --fashion-train-size 256 --fashion-val-size 128 --fashion-epochs 1 --cifar-train-size 256 --cifar-val-size 128 --cifar-epochs 1 --tabular-train-size 800 --tabular-val-size 240 --outdir runs/real_classic_curves_20x8
```

Run the larger GPU-backed image supplement:

```bash
python3 run_experiments.py --fashion-mnist --cifar10 --seeds 5 --seed-offset 20 --budget 12 --ablations full --fashion-train-size 1024 --fashion-val-size 512 --fashion-epochs 2 --cifar-train-size 1024 --cifar-val-size 512 --cifar-epochs 2 --outdir runs/image_gpu_larger_5x12
```

Run the full-scale Fashion-MNIST GPU supplement:

```bash
python3 run_experiments.py --fashion-mnist --seeds 1 --seed-offset 100 --budget 12 --ablations full --fashion-train-size 50000 --fashion-val-size 10000 --fashion-epochs 2 --outdir runs/fashion_full_1x12
```

Run the full-scale CIFAR-10 GPU supplement:

```bash
python3 run_experiments.py --cifar10 --seeds 1 --seed-offset 100 --budget 12 --ablations full --cifar-train-size 40000 --cifar-val-size 10000 --cifar-epochs 2 --outdir runs/cifar_full_1x12
```

Run the current real-data allocation smoke for the lighter real workloads:

```bash
python3 run_experiments.py --ag-news --fashion-mnist --cifar10 --tabular-credit --seeds 1 --budget 6 --ablations full --ag-train-size 120 --ag-val-size 60 --fashion-train-size 96 --fashion-val-size 48 --fashion-epochs 1 --cifar-train-size 96 --cifar-val-size 48 --cifar-epochs 1 --tabular-train-size 240 --tabular-val-size 80 --outdir runs/real_classic_allocation_smoke
```

Run a larger AG News real-data allocation smoke that exercises HD-BasinFlow
edge-flow allocation:

```bash
python3 run_experiments.py --ag-news --seeds 1 --budget 16 --ablations full --ag-train-size 200 --ag-val-size 80 --outdir runs/ag_news_allocation_basin_smoke
```

Run the current 20-seed transformer pass:

```bash
python3 run_experiments.py --sst2 --imdb --seeds 20 --budget 4 --ablations full --sst2-train-size 64 --sst2-val-size 64 --imdb-train-size 64 --imdb-val-size 64 --outdir runs/transformer_curves_20x4
```

Run the full SST-2 DistilBERT supplement:

```bash
python3 run_experiments.py --sst2 --seeds 1 --seed-offset 100 --budget 8 --ablations full --sst2-train-size 67349 --sst2-val-size 872 --outdir runs/sst2_full_1x8
```

Run the larger IMDB DistilBERT supplement:

```bash
python3 run_experiments.py --imdb --seeds 1 --seed-offset 100 --budget 8 --ablations full --imdb-train-size 25000 --imdb-val-size 2000 --outdir runs/imdb_larger_1x8
```

Run the full-validation IMDB DistilBERT supplement:

```bash
python3 run_experiments.py --imdb --seeds 1 --seed-offset 100 --budget 8 --ablations full --imdb-train-size 25000 --imdb-val-size 25000 --outdir runs/imdb_full_1x8
```

Run the current transformer allocation smoke:

```bash
python3 run_experiments.py --sst2 --imdb --seeds 1 --budget 4 --ablations full --sst2-train-size 32 --sst2-val-size 32 --imdb-train-size 32 --imdb-val-size 32 --outdir runs/transformer_allocation_smoke
```

Aggregate existing run folders into one report:

```bash
python3 -m hdbasin.aggregate --include synthetic_fair_20x80,dimension_scaling_extended,sobol_dimension_scaling_3x32,real_classic_curves_20x8,tabular_credit_extra_80x8,transformer_curves_20x4,image_gpu_larger_5x12,fashion_full_1x12,cifar_full_1x12,ag_news_full_1x12,tabular_credit_full_1x12,sst2_full_1x8,imdb_larger_1x8,imdb_full_1x8 --outdir runs/combined_report
```

The latest combined report in this workspace uses `synthetic_fair_20x80`,
`dimension_scaling_extended`,
`sobol_dimension_scaling_3x32`, `real_classic_curves_20x8`,
`tabular_credit_extra_80x8`, `transformer_curves_20x4`, and
`image_gpu_larger_5x12`, plus `fashion_full_1x12`, `cifar_full_1x12`,
`ag_news_full_1x12`, `tabular_credit_full_1x12`, `sst2_full_1x8`, and
`imdb_larger_1x8`, plus `imdb_full_1x8`. It contains 152,928 evaluation rows, 30
experiments, 133 PNG figures, 133 PDF figures, 20 fair-seed runs for every synthetic method/ablation group
including Sobol/quasi-random, 20 seeds for
AG News/Fashion-MNIST/CIFAR-10 plus 5 larger GPU-backed Fashion-MNIST/CIFAR-10
seeds at 1024/512 train/validation samples and 2 epochs, 100 seeds for
tabular credit, 20 seeds for SST-2/IMDB DistilBERT, one full-scale
Fashion-MNIST seed at 50,000/10,000 train/validation samples and 2 epochs,
one full-scale CIFAR-10 seed at 40,000/10,000 train/validation samples and 2 epochs,
one full-size AG News seed at 100,000/20,000 train/validation examples,
one full-available tabular credit seed at 13,000/3,714 train/validation examples,
one full SST-2 DistilBERT seed at 67,349/872 train/validation examples,
one larger IMDB DistilBERT seed at 25,000/2,000 train/validation examples,
one full-validation IMDB DistilBERT seed at 25,000/25,000 train/validation examples,
and 3 seeds for each
dimension-scaling method group including
the Sobol/quasi-random supplement. The `real_classic_curves_20x8` and
`transformer_curves_20x4` rows have verified nonempty train curves,
validation curves, and final validation metrics; `image_gpu_larger_5x12`
adds the larger image-workload supplement on an A100 GPU, and
`fashion_full_1x12` and `cifar_full_1x12` add full-scale image A100 supplements.
The `ag_news_full_1x12` folder adds the full-size AG News CPU sklearn supplement.
The `tabular_credit_full_1x12` folder adds the full-available tabular CPU sklearn supplement.
The `sst2_full_1x8` folder adds the full SST-2 DistilBERT A100 supplement.
The `imdb_larger_1x8` folder adds the larger IMDB DistilBERT A100 supplement.
The `imdb_full_1x8` folder adds the full-validation IMDB DistilBERT A100 supplement.
The `synthetic_fair_20x80` folder replaces the older synthetic ablation source
in the combined report and shows at least one ablation worse than `full` on all
six synthetic ablation tasks, while still reporting the tasks where an ablation
beats `full`.

The `allocation_synthetic_20x80` supplement contains 19,200 evaluation rows
with explicit HD-BasinFlow `allocation_decision`, `killed_early`, and
`kill_reason` fields. Its `plot_data/early_stopping_kill_metrics.csv` reports
kill precision, kill recall, and good-run false-kill rate for both
HD-BasinFlow and ASHA across the six synthetic tasks. Reports also write
`plot_data/false_kill_threshold_summary.csv` and
`figures/false_kill_threshold_summary.png` / `.pdf`, which classify rows
against the PDF's 5-10 percent good-run false-kill condition.

The `real_classic_allocation_smoke` and `transformer_allocation_smoke` folders
verify the same allocation/kill logging schema on all six real workload
families. At these small budgets HD-BasinFlow initializes and explores
globally, while ASHA contributes real-task kill decisions; larger real budgets
are needed to exercise HD-BasinFlow basin continuation and repair spawning on
the real workloads. The `ag_news_allocation_basin_smoke` folder adds a larger
real AG News run that exercises HD-BasinFlow `explore_edge_flow` allocation.

The `subspace_dimension_scaling_3x32` supplement contains 5,184 evaluation
rows, including 1,728 HD-BasinFlow rows with true Frobenius
`subspace_error=||UU^T-UhatUhat^T||_F` against the known rotated-active
subspace. Its `plot_data/active_subspace_recovery.csv` and
`plot_data/dimension_scaling_summary.csv` include the subspace-error fields.

## Deliverables

Each run writes:

- `raw/evaluations.csv`
- `processed/metrics.csv`
- `processed/real_data_status.csv` when requested
- `figures/*.png` and `figures/*.pdf`
- `plot_data/*.csv`
- `reports/experiment_report.md`

Reports include `plot_data/gpu_time_savings.csv` and
`figures/gpu_time_savings.png`, a wall-clock time-to-target proxy for the
GPU-time savings requirement.

Reports include both `plot_data/best_loss_vs_evaluations.csv` and
`plot_data/best_loss_vs_walltime.csv`, with per-experiment PNG figures for
both views.

HD-BasinFlow ablation reports include both `figures/ablation_best_loss_*.png`
/ `.pdf` and `figures/ablation_time_to_target_*.png` / `.pdf`. They also
write `plot_data/ablation_interpretation.csv`, which states whether `full`
or an ablation wins by median final loss and whether ablated pieces appear to
matter.

Reports with HD-BasinFlow allocation decisions include
`plot_data/basin_allocation_summary.csv` and
`figures/basin_allocation_summary.png`, which summarize the mix of
initialization, global exploration, edge-flow, basin continuation, and repair
sampling.

Reports also write `plot_data/basin_allocation_over_time.csv`; when allocation
decisions are present, they include `figures/basin_allocation_over_time.png`.

Reports with ML curve logs include `plot_data/validation_curves.csv` and
`figures/validation_curves.png`.

Reports with matching `noisy_*` and clean synthetic experiments include
`plot_data/noise_robustness.csv`, `figures/noise_robustness.png`, and
`figures/noise_robustness.pdf`, comparing median final loss and IQR shifts
under observation noise.

Reports generated from known active-subspace synthetic objectives include
`subspace_error` in `raw/evaluations.csv`, `plot_data/active_subspace_recovery.csv`,
`figures/active_subspace_recovery.png` / `.pdf`, and
`plot_data/dimension_scaling_summary.csv`.

Generated reports include an `HD-BasinFlow Compute Savings Verdict` section
an explicit `False-Kill Threshold Check`, an `Ablation Interpretation`
section, and a `What Failed Or Remains Unproven` section, so losses and
missing success criteria are stated explicitly instead of inferred from tables.

HD-BasinFlow evaluation rows now populate `allocation_decision`,
`killed_early`, and `kill_reason` for basin continuation, repair spawning, and
certificate-rejected allocation decisions.

Real ML evaluation rows now populate `train_curve_json`, `val_curve_json`, and
`final_test_metric` from observed model training/evaluation values when those
workloads are rerun. Existing historical run folders keep the values produced
by the code version that generated them.