gpu/single-gpu/home_benjamin_hdbasin/code/README.md

HD-BasinFlow Experiment Suite

This repository implements the validation plan in hd_basinflow_experiment_instructions.pdf.

The suite includes:

• A runnable HD-BasinFlow optimizer for objectives on [0, 1]^d.
• Exact synthetic gradients and local weighted ridge gradient estimation support.
• Active-subspace estimation, kNN flow graph construction, basin certificates, repair sampling, and ablations.
• Synthetic benchmarks and baseline comparisons.
• Required CSV logs, processed metrics, plots, and reports/experiment_report.md.
• A real-data status probe for the Hugging Face datasets named in the plan. Real ML runs are not silently faked.
• A target-savings benchmark mode for the commercial claim: comparable validation quality with less compute.

Commercial Validation Path

The narrow commercial claim is:

HD-BasinFlow can reach comparable validation quality on selected HPO/fine-tuning workloads while using less compute than common search baselines.

The current pass/fail rule is intentionally conservative:

• Compare against Random Search, Sobol/quasi-random, Optuna TPE, and ASHA/Successive Halving.
• Treat HD-BasinFlow as passing a comparison only when median final loss is within 1% of the baseline median final loss.
• Require the target-savings policy to reach that quality with at least 20% fewer evaluations.
• Treat 15-30% savings as the first commercial operating band.
• Report failures and underpowered rows instead of hiding them.

Build the target-savings report from the latest combined raw evaluations:

python3 scripts/build_target_savings_report.py \
  --input runs/combined_report/raw/evaluations.csv \
  --outdir runs/target_savings_benchmark

That produces:

• target_savings_decisions.csv: per-dataset/per-baseline pass-fail table.
• target_savings_summary.csv: aggregate competitor comparison.
• target_savings_report.md: technical report.
• target_savings_operating_point.pdf: plot for the report/deck.
• customer_deck_outline.md: short commercial deck outline.
• demo_target_savings.py: minimal demo command that prints decisions.

Run a clean A100 commercial benchmark suite:

bash scripts/run_commercial_benchmark.sh runs/commercial_benchmark_a100

This stages repeated image, transformer, classic NLP, and tabular workloads on Fashion-MNIST, CIFAR-10, SST-2, IMDB, AG News, and tabular credit. The script then aggregates results and rebuilds the target-savings report.

Run the stronger 20-seed internal validation gate with GPU telemetry:

bash scripts/run_stronger_commercial_benchmark.sh runs/commercial_benchmark_a100_20seed

That writes gpu_telemetry.csv, run.log, per-workload outputs, a combined report, and a fresh target-savings package. Use this before making external sales claims.

The commercial/internal whitepaper is:

make whitepaper

Output:

• paper/hd_basinflow_paper.tex
• paper/build/hd_basinflow_paper.pdf

Convenience targets:

make test
make report
make report-strong
make figures
make figures-strong
make whitepaper
make all-commercial
make all-commercial-strong

Run

On the remote machine:

cd ~/hdbasin
python3 run_experiments.py --synthetic --seeds 5 --budget 80 --outdir runs/smoke

For a fuller synthetic run:

python3 run_experiments.py --synthetic --seeds 20 --budget 160 --outdir runs/full

Probe real dataset availability without running expensive training:

python3 run_experiments.py --real-status --outdir runs/real_status

Run the lightweight AG News real-data HPO workload. --seeds repeats real workloads with distinct dataset shuffles and optimizer seeds:

python3 run_experiments.py --ag-news --seeds 3 --budget 8 --ablations full --ag-train-size 400 --ag-val-size 160 --outdir runs/ag_news_diagnostic

Run a tiny AG News curve-logging smoke test:

python3 run_experiments.py --ag-news --seeds 1 --budget 2 --ablations full --ag-train-size 80 --ag-val-size 40 --outdir runs/ag_news_curves_smoke

Run the full-size AG News TF-IDF/logistic-regression supplement:

python3 run_experiments.py --ag-news --seeds 1 --seed-offset 100 --budget 12 --ablations full --ag-train-size 100000 --ag-val-size 20000 --outdir runs/ag_news_full_1x12

Run the lightweight Fashion-MNIST tiny-CNN workload on GPU when available:

python3 run_experiments.py --fashion-mnist --seeds 3 --budget 6 --ablations full --fashion-train-size 256 --fashion-val-size 128 --fashion-epochs 1 --outdir runs/fashion_mnist_smoke

Run the lightweight CIFAR-10 tiny-CNN workload on GPU when available:

python3 run_experiments.py --cifar10 --seeds 3 --budget 6 --ablations full --cifar-train-size 256 --cifar-val-size 128 --cifar-epochs 1 --outdir runs/cifar10_smoke

Run the lightweight tabular credit workload:

python3 run_experiments.py --tabular-credit --seeds 3 --budget 8 --ablations full --tabular-train-size 800 --tabular-val-size 240 --outdir runs/tabular_credit_smoke

Extend tabular evidence without duplicating the first 20 seeds:

python3 run_experiments.py --tabular-credit --seeds 80 --seed-offset 20 --budget 8 --ablations full --tabular-train-size 800 --tabular-val-size 240 --outdir runs/tabular_credit_extra_80x8

Run the full-available tabular credit supplement:

python3 run_experiments.py --tabular-credit --seeds 1 --seed-offset 100 --budget 12 --ablations full --tabular-train-size 13000 --tabular-val-size 3714 --outdir runs/tabular_credit_full_1x12

Run the lightweight SST-2 DistilBERT workload on GPU when available:

python3 run_experiments.py --sst2 --seeds 3 --budget 3 --ablations full --sst2-train-size 48 --sst2-val-size 48 --outdir runs/sst2_smoke

Run the lightweight IMDB DistilBERT workload on GPU when available:

python3 run_experiments.py --imdb --seeds 3 --budget 3 --ablations full --imdb-train-size 48 --imdb-val-size 48 --outdir runs/imdb_smoke

Run a synthetic dimension-scaling sweep:

python3 run_experiments.py --dimension-scaling --seeds 1 --budget 16 --scaling-dims 2,5,10,20 --scaling-active-dims 2,5 --outdir runs/dimension_scaling_smoke

Run the current 20-seed fair-seed synthetic statistics pass. HD-BasinFlow ablations use the same optimizer seed within each objective/outer-seed pair so the comparison is not confounded by different initial random designs:

python3 run_experiments.py --synthetic --seeds 20 --budget 80 --ablations full,NoFlow,NoActive,NoShell,NoCurvature,NoLeakage,NoRepair,NoRandom,MidpointOnly,LossOnly --outdir runs/synthetic_fair_20x80

Run a synthetic Sobol/quasi-random-only supplement if you want to isolate that baseline outside the fair-seed synthetic run:

python3 run_experiments.py --synthetic --seeds 20 --budget 80 --baselines sobol --skip-hdbasinflow --outdir runs/sobol_synthetic_20x80

Run the current 20-seed synthetic compute-allocation supplement:

python3 run_experiments.py --synthetic --seeds 20 --budget 80 --ablations full --baselines asha --outdir runs/allocation_synthetic_20x80

Run the current extended dimension-scaling pass:

python3 run_experiments.py --dimension-scaling --seeds 3 --budget 32 --scaling-dims 2,5,10,20,50,100 --scaling-active-dims 2,5,10,20 --outdir runs/dimension_scaling_extended

Run the current dimension-scaling Sobol/quasi-random supplement:

python3 run_experiments.py --dimension-scaling --seeds 3 --budget 32 --scaling-dims 2,5,10,20,50,100 --scaling-active-dims 2,5,10,20 --baselines sobol --skip-hdbasinflow --outdir runs/sobol_dimension_scaling_3x32

Run the current dimension-scaling active-subspace-error supplement:

python3 run_experiments.py --dimension-scaling --seeds 3 --budget 32 --scaling-dims 2,5,10,20,50,100 --scaling-active-dims 2,5,10,20 --baselines random,sobol --outdir runs/subspace_dimension_scaling_3x32

Run the current 20-seed classic real-data pass:

python3 run_experiments.py --ag-news --fashion-mnist --cifar10 --tabular-credit --seeds 20 --budget 8 --ablations full --ag-train-size 400 --ag-val-size 160 --fashion-train-size 256 --fashion-val-size 128 --fashion-epochs 1 --cifar-train-size 256 --cifar-val-size 128 --cifar-epochs 1 --tabular-train-size 800 --tabular-val-size 240 --outdir runs/real_classic_curves_20x8

Run the larger GPU-backed image supplement:

python3 run_experiments.py --fashion-mnist --cifar10 --seeds 5 --seed-offset 20 --budget 12 --ablations full --fashion-train-size 1024 --fashion-val-size 512 --fashion-epochs 2 --cifar-train-size 1024 --cifar-val-size 512 --cifar-epochs 2 --outdir runs/image_gpu_larger_5x12

Run the full-scale Fashion-MNIST GPU supplement:

python3 run_experiments.py --fashion-mnist --seeds 1 --seed-offset 100 --budget 12 --ablations full --fashion-train-size 50000 --fashion-val-size 10000 --fashion-epochs 2 --outdir runs/fashion_full_1x12

Run the full-scale CIFAR-10 GPU supplement:

python3 run_experiments.py --cifar10 --seeds 1 --seed-offset 100 --budget 12 --ablations full --cifar-train-size 40000 --cifar-val-size 10000 --cifar-epochs 2 --outdir runs/cifar_full_1x12

Run the current real-data allocation smoke for the lighter real workloads:

python3 run_experiments.py --ag-news --fashion-mnist --cifar10 --tabular-credit --seeds 1 --budget 6 --ablations full --ag-train-size 120 --ag-val-size 60 --fashion-train-size 96 --fashion-val-size 48 --fashion-epochs 1 --cifar-train-size 96 --cifar-val-size 48 --cifar-epochs 1 --tabular-train-size 240 --tabular-val-size 80 --outdir runs/real_classic_allocation_smoke

Run a larger AG News real-data allocation smoke that exercises HD-BasinFlow edge-flow allocation:

python3 run_experiments.py --ag-news --seeds 1 --budget 16 --ablations full --ag-train-size 200 --ag-val-size 80 --outdir runs/ag_news_allocation_basin_smoke

Run the current 20-seed transformer pass:

python3 run_experiments.py --sst2 --imdb --seeds 20 --budget 4 --ablations full --sst2-train-size 64 --sst2-val-size 64 --imdb-train-size 64 --imdb-val-size 64 --outdir runs/transformer_curves_20x4

Run the full SST-2 DistilBERT supplement:

python3 run_experiments.py --sst2 --seeds 1 --seed-offset 100 --budget 8 --ablations full --sst2-train-size 67349 --sst2-val-size 872 --outdir runs/sst2_full_1x8

Run the larger IMDB DistilBERT supplement:

python3 run_experiments.py --imdb --seeds 1 --seed-offset 100 --budget 8 --ablations full --imdb-train-size 25000 --imdb-val-size 2000 --outdir runs/imdb_larger_1x8

Run the full-validation IMDB DistilBERT supplement:

python3 run_experiments.py --imdb --seeds 1 --seed-offset 100 --budget 8 --ablations full --imdb-train-size 25000 --imdb-val-size 25000 --outdir runs/imdb_full_1x8

Run the current transformer allocation smoke:

python3 run_experiments.py --sst2 --imdb --seeds 1 --budget 4 --ablations full --sst2-train-size 32 --sst2-val-size 32 --imdb-train-size 32 --imdb-val-size 32 --outdir runs/transformer_allocation_smoke

Aggregate existing run folders into one report:

python3 -m hdbasin.aggregate --include synthetic_fair_20x80,dimension_scaling_extended,sobol_dimension_scaling_3x32,real_classic_curves_20x8,tabular_credit_extra_80x8,transformer_curves_20x4,image_gpu_larger_5x12,fashion_full_1x12,cifar_full_1x12,ag_news_full_1x12,tabular_credit_full_1x12,sst2_full_1x8,imdb_larger_1x8,imdb_full_1x8 --outdir runs/combined_report

The latest combined report in this workspace uses synthetic_fair_20x80, dimension_scaling_extended, sobol_dimension_scaling_3x32, real_classic_curves_20x8, tabular_credit_extra_80x8, transformer_curves_20x4, and image_gpu_larger_5x12, plus fashion_full_1x12, cifar_full_1x12, ag_news_full_1x12, tabular_credit_full_1x12, sst2_full_1x8, and imdb_larger_1x8, plus imdb_full_1x8. It contains 152,928 evaluation rows, 30 experiments, 133 PNG figures, 133 PDF figures, 20 fair-seed runs for every synthetic method/ablation group including Sobol/quasi-random, 20 seeds for AG News/Fashion-MNIST/CIFAR-10 plus 5 larger GPU-backed Fashion-MNIST/CIFAR-10 seeds at 1024/512 train/validation samples and 2 epochs, 100 seeds for tabular credit, 20 seeds for SST-2/IMDB DistilBERT, one full-scale Fashion-MNIST seed at 50,000/10,000 train/validation samples and 2 epochs, one full-scale CIFAR-10 seed at 40,000/10,000 train/validation samples and 2 epochs, one full-size AG News seed at 100,000/20,000 train/validation examples, one full-available tabular credit seed at 13,000/3,714 train/validation examples, one full SST-2 DistilBERT seed at 67,349/872 train/validation examples, one larger IMDB DistilBERT seed at 25,000/2,000 train/validation examples, one full-validation IMDB DistilBERT seed at 25,000/25,000 train/validation examples, and 3 seeds for each dimension-scaling method group including the Sobol/quasi-random supplement. The real_classic_curves_20x8 and transformer_curves_20x4 rows have verified nonempty train curves, validation curves, and final validation metrics; image_gpu_larger_5x12 adds the larger image-workload supplement on an A100 GPU, and fashion_full_1x12 and cifar_full_1x12 add full-scale image A100 supplements. The ag_news_full_1x12 folder adds the full-size AG News CPU sklearn supplement. The tabular_credit_full_1x12 folder adds the full-available tabular CPU sklearn supplement. The sst2_full_1x8 folder adds the full SST-2 DistilBERT A100 supplement. The imdb_larger_1x8 folder adds the larger IMDB DistilBERT A100 supplement. The imdb_full_1x8 folder adds the full-validation IMDB DistilBERT A100 supplement. The synthetic_fair_20x80 folder replaces the older synthetic ablation source in the combined report and shows at least one ablation worse than full on all six synthetic ablation tasks, while still reporting the tasks where an ablation beats full.

The allocation_synthetic_20x80 supplement contains 19,200 evaluation rows with explicit HD-BasinFlow allocation_decision, killed_early, and kill_reason fields. Its plot_data/early_stopping_kill_metrics.csv reports kill precision, kill recall, and good-run false-kill rate for both HD-BasinFlow and ASHA across the six synthetic tasks. Reports also write plot_data/false_kill_threshold_summary.csv and figures/false_kill_threshold_summary.png / .pdf, which classify rows against the PDF's 5-10 percent good-run false-kill condition.

The real_classic_allocation_smoke and transformer_allocation_smoke folders verify the same allocation/kill logging schema on all six real workload families. At these small budgets HD-BasinFlow initializes and explores globally, while ASHA contributes real-task kill decisions; larger real budgets are needed to exercise HD-BasinFlow basin continuation and repair spawning on the real workloads. The ag_news_allocation_basin_smoke folder adds a larger real AG News run that exercises HD-BasinFlow explore_edge_flow allocation.

The subspace_dimension_scaling_3x32 supplement contains 5,184 evaluation rows, including 1,728 HD-BasinFlow rows with true Frobenius subspace_error=||UU^T-UhatUhat^T||_F against the known rotated-active subspace. Its plot_data/active_subspace_recovery.csv and plot_data/dimension_scaling_summary.csv include the subspace-error fields.

Deliverables

Each run writes:

• raw/evaluations.csv
• processed/metrics.csv
• processed/real_data_status.csv when requested
• figures/*.png and figures/*.pdf
• plot_data/*.csv
• reports/experiment_report.md

Reports include plot_data/gpu_time_savings.csv and figures/gpu_time_savings.png, a wall-clock time-to-target proxy for the GPU-time savings requirement.

Reports include both plot_data/best_loss_vs_evaluations.csv and plot_data/best_loss_vs_walltime.csv, with per-experiment PNG figures for both views.

HD-BasinFlow ablation reports include both figures/ablation_best_loss_*.png / .pdf and figures/ablation_time_to_target_*.png / .pdf. They also write plot_data/ablation_interpretation.csv, which states whether full or an ablation wins by median final loss and whether ablated pieces appear to matter.

Reports with HD-BasinFlow allocation decisions include plot_data/basin_allocation_summary.csv and figures/basin_allocation_summary.png, which summarize the mix of initialization, global exploration, edge-flow, basin continuation, and repair sampling.

Reports also write plot_data/basin_allocation_over_time.csv; when allocation decisions are present, they include figures/basin_allocation_over_time.png.

Reports with ML curve logs include plot_data/validation_curves.csv and figures/validation_curves.png.

Reports with matching noisy_* and clean synthetic experiments include plot_data/noise_robustness.csv, figures/noise_robustness.png, and figures/noise_robustness.pdf, comparing median final loss and IQR shifts under observation noise.

Reports generated from known active-subspace synthetic objectives include subspace_error in raw/evaluations.csv, plot_data/active_subspace_recovery.csv, figures/active_subspace_recovery.png / .pdf, and plot_data/dimension_scaling_summary.csv.

Generated reports include an HD-BasinFlow Compute Savings Verdict section an explicit False-Kill Threshold Check, an Ablation Interpretation section, and a What Failed Or Remains Unproven section, so losses and missing success criteria are stated explicitly instead of inferred from tables.

HD-BasinFlow evaluation rows now populate allocation_decision, killed_early, and kill_reason for basin continuation, repair spawning, and certificate-rejected allocation decisions.

Real ML evaluation rows now populate train_curve_json, val_curve_json, and final_test_metric from observed model training/evaluation values when those workloads are rerun. Existing historical run folders keep the values produced by the code version that generated them.