Running the FelaTab benchmark

Compares lowdown-labs/fela-tab (zero-shot, in-context) against Google TabFM (google/tabfm-1.0.0-pytorch, zero-shot) and trained baselines XGBoost / LightGBM on 8 OpenML classification datasets, and optionally enters FelaTab into the TabArena leaderboard.

1. Environment (CPU/GPU)

# from the repo root (directory containing fela-tab/ and benchmark/)
python -m venv --system-site-packages .venv   # inherits the system ROCm torch — do NOT pip install stock torch
.venv/bin/pip install -r benchmark/requirements.txt
.venv/bin/pip install "tabfm[pytorch]"        # Google TabFM client

GPU: the system torch is a ROCm build (HIP). torch.cuda.is_available() maps to the AMD GPU; no extra setup needed. XGBoost/LightGBM pip builds are CPU-only (normal for GBMs) — the output tables record the device per model.

TabFM is a gated, non-commercial HF repo. Export a token with accepted terms:

export HF_TOKEN=hf_...

2. Run the benchmark

# smoke test: 2 tiny datasets, FelaTab-small + XGB + LGBM (minutes)
.venv/bin/python benchmark/benchmark.py --smoke

# full battery: 8 datasets, FelaTab big+small, TabFM, XGBoost, LightGBM (GPU)
.venv/bin/python benchmark/benchmark.py --device gpu

# useful variants
.venv/bin/python benchmark/benchmark.py --device cpu            # all-CPU shootout
.venv/bin/python benchmark/benchmark.py --skip tabfm            # no TabFM
.venv/bin/python benchmark/benchmark.py --tiers big             # big tier only
.venv/bin/python benchmark/benchmark.py --datasets adult,churn  # subset

Outputs (in benchmark/):

results.csv — per (dataset, model): ROC-AUC, log loss, accuracy, F1-macro, fit time, latency (ms/sample), peak RAM, peak VRAM
model_card_snippet.md — Markdown tables ready to paste into the HF model card (per-dataset tables with bolded winners, mean + average-rank summary, efficiency)

Protocol: stratified 80/20 split, seed 42. FelaTab/TabFM get the train split as in-context support rows (capped at 3000); XGBoost/LightGBM train on the full split. Datasets with >10 classes are skipped (FelaTab/TabFM hard limit).

3. TabArena entry

TabArena (https://huggingface.co/spaces/TabArena/leaderboard) is an AutoGluon-based living benchmark. FelaTab is wrapped as an AutoGluon model in benchmark/tabarena/fela_ag_model.py.

# one-time setup (needs Python 3.11-3.13 + uv; already done in benchmark/tabarena/tabarena-repo)
git clone https://github.com/autogluon/tabarena.git benchmark/tabarena/tabarena-repo
cd benchmark/tabarena/tabarena-repo
uv venv --seed --python 3.12 .venv
uv pip install --python .venv/bin/python --prerelease=allow -e "./packages/tabarena[benchmark]"

# run (from benchmark/tabarena/)
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --quickstart   # 3 lite datasets
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --subset lite  # full TabArena-Lite
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --full         # all 51 datasets (hours)

Notes:

Set TMPDIR to a real disk directory. AutoGluon pickles one model per CV fold; the FelaTab big tier is ~1.6 GB fp32, which overflows a tmpfs /tmp (Errno 28).
The leaderboard comparison + figures land in benchmark/tabarena/eval/<run-name>/, including leaderboard_website.md.
fela_ag_model.py runs tiers small+big as two configs; tasks with >10 classes are skipped gracefully (raise_on_failure=False in the runner).