File size: 3,562 Bytes
1438ac4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | # Running the FelaTab benchmark
Compares `lowdown-labs/fela-tab` (zero-shot, in-context) against Google TabFM
(`google/tabfm-1.0.0-pytorch`, zero-shot) and trained baselines XGBoost / LightGBM on
8 OpenML classification datasets, and optionally enters FelaTab into the TabArena
leaderboard.
## 1. Environment (CPU/GPU)
```bash
# from the repo root (directory containing fela-tab/ and benchmark/)
python -m venv --system-site-packages .venv # inherits the system ROCm torch — do NOT pip install stock torch
.venv/bin/pip install -r benchmark/requirements.txt
.venv/bin/pip install "tabfm[pytorch]" # Google TabFM client
```
GPU: the system torch is a ROCm build (HIP). `torch.cuda.is_available()` maps to the
AMD GPU; no extra setup needed. XGBoost/LightGBM pip builds are CPU-only (normal for
GBMs) — the output tables record the device per model.
TabFM is a **gated, non-commercial** HF repo. Export a token with accepted terms:
```bash
export HF_TOKEN=hf_...
```
## 2. Run the benchmark
```bash
# smoke test: 2 tiny datasets, FelaTab-small + XGB + LGBM (minutes)
.venv/bin/python benchmark/benchmark.py --smoke
# full battery: 8 datasets, FelaTab big+small, TabFM, XGBoost, LightGBM (GPU)
.venv/bin/python benchmark/benchmark.py --device gpu
# useful variants
.venv/bin/python benchmark/benchmark.py --device cpu # all-CPU shootout
.venv/bin/python benchmark/benchmark.py --skip tabfm # no TabFM
.venv/bin/python benchmark/benchmark.py --tiers big # big tier only
.venv/bin/python benchmark/benchmark.py --datasets adult,churn # subset
```
Outputs (in `benchmark/`):
- `results.csv` — per (dataset, model): ROC-AUC, log loss, accuracy, F1-macro,
fit time, latency (ms/sample), peak RAM, peak VRAM
- `model_card_snippet.md` — Markdown tables ready to paste into the HF model card
(per-dataset tables with bolded winners, mean + average-rank summary, efficiency)
Protocol: stratified 80/20 split, seed 42. FelaTab/TabFM get the train split as
in-context support rows (capped at 3000); XGBoost/LightGBM train on the full split.
Datasets with >10 classes are skipped (FelaTab/TabFM hard limit).
## 3. TabArena entry
TabArena (https://huggingface.co/spaces/TabArena/leaderboard) is an AutoGluon-based
living benchmark. FelaTab is wrapped as an AutoGluon model in
`benchmark/tabarena/fela_ag_model.py`.
```bash
# one-time setup (needs Python 3.11-3.13 + uv; already done in benchmark/tabarena/tabarena-repo)
git clone https://github.com/autogluon/tabarena.git benchmark/tabarena/tabarena-repo
cd benchmark/tabarena/tabarena-repo
uv venv --seed --python 3.12 .venv
uv pip install --python .venv/bin/python --prerelease=allow -e "./packages/tabarena[benchmark]"
# run (from benchmark/tabarena/)
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --quickstart # 3 lite datasets
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --subset lite # full TabArena-Lite
TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --full # all 51 datasets (hours)
```
Notes:
- **Set TMPDIR to a real disk directory.** AutoGluon pickles one model per CV fold;
the FelaTab big tier is ~1.6 GB fp32, which overflows a tmpfs `/tmp` (Errno 28).
- The leaderboard comparison + figures land in `benchmark/tabarena/eval/<run-name>/`,
including `leaderboard_website.md`.
- `fela_ag_model.py` runs tiers small+big as two configs; tasks with >10 classes are
skipped gracefully (`raise_on_failure=False` in the runner).
|