benchmark/RUNNING.md

# Running the FelaTab benchmark

Compares `lowdown-labs/fela-tab` (zero-shot, in-context) against Google TabFM
(`google/tabfm-1.0.0-pytorch`, zero-shot) and trained baselines XGBoost / LightGBM on
8 OpenML classification datasets, and optionally enters FelaTab into the TabArena
leaderboard.

## 1. Environment (CPU/GPU)

```bash
# from the repo root (directory containing fela-tab/ and benchmark/)
python -m venv --system-site-packages .venv   # inherits the system ROCm torch — do NOT pip install stock torch
.venv/bin/pip install -r benchmark/requirements.txt
.venv/bin/pip install "tabfm[pytorch]"        # Google TabFM client
```

GPU: the system torch is a ROCm build (HIP). `torch.cuda.is_available()` maps to the
AMD GPU; no extra setup needed. XGBoost/LightGBM pip builds are CPU-only (normal for
GBMs) — the output tables record the device per model.

TabFM is a **gated, non-commercial** HF repo. Export a token with accepted terms:

```bash
export HF_TOKEN=hf_...
```

## 2. Run the benchmark

```bash
# smoke test: 2 tiny datasets, FelaTab-small + XGB + LGBM (minutes)
.venv/bin/python benchmark/benchmark.py --smoke

# full battery: 8 datasets, FelaTab big+small, TabFM, XGBoost, LightGBM (GPU)
.venv/bin/python benchmark/benchmark.py --device gpu

# useful variants
.venv/bin/python benchmark/benchmark.py --device cpu            # all-CPU shootout
.venv/bin/python benchmark/benchmark.py --skip tabfm            # no TabFM
.venv/bin/python benchmark/benchmark.py --tiers big             # big tier only
.venv/bin/python benchmark/benchmark.py --datasets adult,churn  # subset
```

Outputs (in `benchmark/`):
- `results.csv` — per (dataset, model): ROC-AUC, log loss, accuracy, F1-macro,
  fit time, latency (ms/sample), peak RAM, peak VRAM
- `model_card_snippet.md` — Markdown tables ready to paste into the HF model card
  (per-dataset tables with bolded winners, mean + average-rank summary, efficiency)

Protocol: stratified 80/20 split, seed 42. FelaTab/TabFM get the train split as
in-context support rows (capped at 3000); XGBoost/LightGBM train on the full split.
Datasets with >10 classes are skipped (FelaTab/TabFM hard limit).

## 3. TabArena entry

TabArena (https://huggingface.co/spaces/TabArena/leaderboard) is an AutoGluon-based
living benchmark. FelaTab is wrapped as an AutoGluon model in
`benchmark/tabarena/fela_ag_model.py`.

```bash
# one-time setup (needs Python 3.11-3.13 + uv; already done in benchmark/tabarena/tabarena-repo)
git clone https://github.com/autogluon/tabarena.git benchmark/tabarena/tabarena-repo
cd benchmark/tabarena/tabarena-repo
uv venv --seed --python 3.12 .venv
uv pip install --python .venv/bin/python --prerelease=allow -e "./packages/tabarena[benchmark]"

# run (from benchmark/tabarena/)
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --quickstart   # 3 lite datasets
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --subset lite  # full TabArena-Lite
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --full         # all 51 datasets (hours)
```

Notes:
- **Set TMPDIR to a real disk directory.** AutoGluon pickles one model per CV fold;
  the FelaTab big tier is ~1.6 GB fp32, which overflows a tmpfs `/tmp` (Errno 28).
- The leaderboard comparison + figures land in `benchmark/tabarena/eval/<run-name>/`,
  including `leaderboard_website.md`.
- `fela_ag_model.py` runs tiers small+big as two configs; tasks with >10 classes are
  skipped gracefully (`raise_on_failure=False` in the runner).