docs/experiments.md

# How to reproduce the AHD-CMA experiments

Every command assumes the repo root and an activated virtual env::

    git clone <repo>
    cd <repo>
    python3.11 -m venv .venv
    source .venv/bin/activate
    pip install -e ".[dev]"
    pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121

Confirm GPU::

    python -c "import torch; assert torch.cuda.is_available(); print(torch.cuda.get_device_name(0))"

## 1. Unit and integration tests

```bash
pytest tests/unit -q                              # ~1 minute
RUN_INTEGRATION=1 pytest tests/integration -v     # ~10 minutes (downloads CIFAR-10 + ViT-Tiny)
```

The integration test runs the LoRA fitness end-to-end on CIFAR-10
with a tiny ViT and asserts > 50 % accuracy in < 10 minutes.

## 2. CEC-2022 benchmark

Single run::

    python -m ahdcma.cli.run_benchmark \
        --algo ahdcma --func F5_rastrigin --dim 10 --seed 0 \
        --max-evals 30000 --pop 30 --output outputs/runs/cec2022

Full sweep (7 algos x 12 funcs x {10D, 30D} x 30 seeds = 5040 runs,
~19 h on a single 24 GB GPU machine)::

    python scripts/run_cec2022_full.py

Resume-safe: re-running picks up where it stopped via existing-
`result.json` checks.

After the sweep, render figures and tables::

    python -m ahdcma.cli.make_figures

Outputs land in `outputs/figures/` and `outputs/tables/`.

## 3. LoRA hyperparameter sweep

The full pipeline is gated by user approval per CLAUDE.md §10.
Recommended progression:

```bash
# Round 1 -- preliminary (3 algos x 3 tasks x 5 seeds, ~180 GPU-h)
bash scripts/run_full_pipeline.sh round 1

# Round 2 -- full sweep (14 algos x 7 tasks x 5 seeds, ~1960 GPU-h)
bash scripts/run_full_pipeline.sh round 2

# Round 3 -- 30-seed statistical confirmation (top-4 algos x 7 tasks)
bash scripts/run_full_pipeline.sh round 3
```

Single LoRA run::

    python -m ahdcma.cli.run_task \
        --algo ahdcma --task cifar100_vit --seed 0 \
        --pop 8 --gens 10 --num-steps 100

## 4. Ablation study

```bash
python scripts/run_ablation.py \
    --funcs F1_bent_cigar F5_rastrigin F9_composite1 \
    --variants full no_chaotic no_adaptive no_ruggedness no_doa_de \
    --seeds 0 1 2 3 4 \
    --pop 30 --gens 100
```

## 5. Statistical analysis

The CEC-2022 figure generator computes per-dim Wilcoxon vs AHD-CMA
and the Friedman omnibus. Use `cli.make_figures` for the canonical
artifact bundle, or call the lower-level helpers directly:

```python
from ahdcma.stats.tests import wilcoxon_pairwise, friedman_test, cliffs_delta

results = {"ahdcma": ..., "cmaes": ..., ...}  # algo -> per-seed array
df = wilcoxon_pairwise(results, baseline_name="ahdcma")
stat, p = friedman_test(results)
delta = cliffs_delta(results["ahdcma"], results["cmaes"])
```

## 6. Reproducibility checklist

* All entry points call `set_global_seed(seed)` first thing after
  argparse.
* Each run writes a config snapshot to its own `result.json`.
* `make_run_id` includes algo, task, seed, dim and a timestamp.
* The full 30-seed pool used for reported results is `seeds=range(30)`.
* Deviations from the original proposal are catalogued in
  `docs/spec_deviations.md`.