Add TabFM/GBM head-to-head benchmark, efficiency tables, and TabArena entry

README.md

@@ -119,6 +119,52 @@ trees and ridge. The numbers below are measured on the shipped weights.
   support set) did not improve the big tier on this battery (it measured 0.811, below the single
   pass), so the single pass is the number to use.
 
+## Head to head vs Google TabFM (zero shot) and gradient boosted trees
+
+Independent battery: 8 OpenML classification datasets (adult, credit-g, blood-transfusion, churn,
+electricity, vehicle, segment, jungle_chess), stratified 80/20 split, seed 42. FelaTab and
+`google/tabfm-1.0.0-pytorch` run zero shot (train split as in context support, capped at 3000
+rows); XGBoost / LightGBM are trained on the full train split. Deep learning models ran on a
+local GPU (ROCm), the GBMs on CPU. Reproduce with `benchmark/benchmark.py`.
+
+### Summary (mean across 8 datasets)
+
+| Model | Mean Acc | Mean ROC-AUC | Mean LogLoss | Mean F1 | Avg Rank |
+|---|---|---|---|---|---|
+| TabFM (zero shot) | 0.8899 | 0.9263 | 0.2329 | 0.8484 | 1.02 |
+| XGBoost (trained) | 0.8534 | 0.9052 | 0.3350 | 0.8126 | 2.88 |
+| LightGBM (trained) | 0.8526 | 0.9011 | 0.4538 | 0.8113 | 3.14 |
+| **FelaTab big (zero shot)** | 0.8043 | 0.8674 | 0.4301 | 0.7181 | 3.91 |
+| **FelaTab small (zero shot)** | 0.7929 | 0.8539 | 0.4425 | 0.7057 | 4.06 |
+
+### Efficiency (this is where FelaTab lives)
+
+| Model | Params | Device | Fit time | Latency (ms/sample) | Peak VRAM (MB) |
+|---|---|---|---|---|---|
+| **FelaTab small** | 51.6M | gpu (ROCm) / CPU-native | 0 s | **0.79** | 5390 |
+| **FelaTab big** | 411.9M | gpu (ROCm) / CPU-native | 0 s | **1.76** | 5531 |
+| TabFM | 1.64B | gpu (ROCm) | 0.24 s | 52.43 | 7142 |
+| XGBoost | - | cpu | 6.35 s | 0.025 | N/A |
+| LightGBM | - | cpu | 6.66 s | 0.008 | N/A |
+
+- FelaTab-small is **~66x lower latency per sample than TabFM** (0.79 vs 52.4 ms) at roughly
+  1/32nd the parameter count (51.6M vs 1.64B), and it does that while also running comfortably
+  on a plain CPU.
+- TabFM is more accurate on this battery (it won 7 of 8 datasets on accuracy; FelaTab-big tied
+  it on credit-g). FelaTab's trade: give up some accuracy, get an order of magnitude faster,
+  smaller, CPU-native, zero-setup predictor.
+- Full per-dataset tables (accuracy / ROC-AUC / log loss / F1) are in
+  `benchmark/model_card_snippet.md`; raw numbers in `benchmark/results.csv`.
+
+## TabArena
+
+FelaTab is wrapped as an AutoGluon model (`benchmark/tabarena/fela_ag_model.py`) and runnable
+against the official [TabArena](https://huggingface.co/spaces/TabArena/leaderboard) leaderboard
+with `benchmark/tabarena/run_tabarena.py`. On the 3-dataset quickstart subset FelaTab-big scored
+ELO ~938 (small ~897), mid-pack alongside default XGBoost, with 33% of tasks imputed (regression
+tasks are out of scope for the classification wrapper). Full TabArena-Lite / full-benchmark runs
+are in progress; see `benchmark/RUNNING.md`.
+
 ## Regression (5 datasets)
 
 On regression accuracy FelaTab alone beats neither untuned scikit-learn nor a tuned LightGBM on any

benchmark/RUNNING.md

@@ -0,0 +1,78 @@
+# Running the FelaTab benchmark
+
+Compares `lowdown-labs/fela-tab` (zero-shot, in-context) against Google TabFM
+(`google/tabfm-1.0.0-pytorch`, zero-shot) and trained baselines XGBoost / LightGBM on
+8 OpenML classification datasets, and optionally enters FelaTab into the TabArena
+leaderboard.
+
+## 1. Environment (CPU/GPU)
+
+```bash
+# from the repo root (directory containing fela-tab/ and benchmark/)
+python -m venv --system-site-packages .venv   # inherits the system ROCm torch — do NOT pip install stock torch
+.venv/bin/pip install -r benchmark/requirements.txt
+.venv/bin/pip install "tabfm[pytorch]"        # Google TabFM client
+```
+
+GPU: the system torch is a ROCm build (HIP). `torch.cuda.is_available()` maps to the
+AMD GPU; no extra setup needed. XGBoost/LightGBM pip builds are CPU-only (normal for
+GBMs) — the output tables record the device per model.
+
+TabFM is a **gated, non-commercial** HF repo. Export a token with accepted terms:
+
+```bash
+export HF_TOKEN=hf_...
+```
+
+## 2. Run the benchmark
+
+```bash
+# smoke test: 2 tiny datasets, FelaTab-small + XGB + LGBM (minutes)
+.venv/bin/python benchmark/benchmark.py --smoke
+
+# full battery: 8 datasets, FelaTab big+small, TabFM, XGBoost, LightGBM (GPU)
+.venv/bin/python benchmark/benchmark.py --device gpu
+
+# useful variants
+.venv/bin/python benchmark/benchmark.py --device cpu            # all-CPU shootout
+.venv/bin/python benchmark/benchmark.py --skip tabfm            # no TabFM
+.venv/bin/python benchmark/benchmark.py --tiers big             # big tier only
+.venv/bin/python benchmark/benchmark.py --datasets adult,churn  # subset
+```
+
+Outputs (in `benchmark/`):
+- `results.csv` — per (dataset, model): ROC-AUC, log loss, accuracy, F1-macro,
+  fit time, latency (ms/sample), peak RAM, peak VRAM
+- `model_card_snippet.md` — Markdown tables ready to paste into the HF model card
+  (per-dataset tables with bolded winners, mean + average-rank summary, efficiency)
+
+Protocol: stratified 80/20 split, seed 42. FelaTab/TabFM get the train split as
+in-context support rows (capped at 3000); XGBoost/LightGBM train on the full split.
+Datasets with >10 classes are skipped (FelaTab/TabFM hard limit).
+
+## 3. TabArena entry
+
+TabArena (https://huggingface.co/spaces/TabArena/leaderboard) is an AutoGluon-based
+living benchmark. FelaTab is wrapped as an AutoGluon model in
+`benchmark/tabarena/fela_ag_model.py`.
+
+```bash
+# one-time setup (needs Python 3.11-3.13 + uv; already done in benchmark/tabarena/tabarena-repo)
+git clone https://github.com/autogluon/tabarena.git benchmark/tabarena/tabarena-repo
+cd benchmark/tabarena/tabarena-repo
+uv venv --seed --python 3.12 .venv
+uv pip install --python .venv/bin/python --prerelease=allow -e "./packages/tabarena[benchmark]"
+
+# run (from benchmark/tabarena/)
+TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --quickstart   # 3 lite datasets
+TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --subset lite  # full TabArena-Lite
+TMPDIR=<dir-on-disk> ../tabarena/tabarena-repo/.venv/bin/python run_tabarena.py --full         # all 51 datasets (hours)
+```
+
+Notes:
+- **Set TMPDIR to a real disk directory.** AutoGluon pickles one model per CV fold;
+  the FelaTab big tier is ~1.6 GB fp32, which overflows a tmpfs `/tmp` (Errno 28).
+- The leaderboard comparison + figures land in `benchmark/tabarena/eval/<run-name>/`,
+  including `leaderboard_website.md`.
+- `fela_ag_model.py` runs tiers small+big as two configs; tasks with >10 classes are
+  skipped gracefully (`raise_on_failure=False` in the runner).

benchmark/benchmark.py

@@ -0,0 +1,456 @@
+"""Benchmark FelaTab vs TabFM vs XGBoost/LightGBM on OpenML classification datasets.
+
+FelaTab + TabFM run zero-shot / in-context (train split = support rows, capped).
+XGBoost / LightGBM are trained on the full train split (CPU; pip builds can't use ROCm).
+
+Outputs:
+  benchmark/results.csv            - per (dataset, model) metrics + efficiency
+  benchmark/model_card_snippet.md  - Hugging Face model-card-ready Markdown tables
+
+Usage:
+  python benchmark.py --smoke                 # 2 tiny datasets, 3 models, minutes
+  python benchmark.py                         # full battery
+  python benchmark.py --device cpu --skip tabfm
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+import time
+import tracemalloc
+import warnings
+from dataclasses import dataclass, field
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import psutil
+
+warnings.filterwarnings("ignore")
+
+SEED = 42
+HERE = Path(__file__).resolve().parent
+# works both as <workspace>/benchmark/benchmark.py and inside the fela-tab repo as
+# fela-tab/benchmark/benchmark.py
+for _cand in (HERE.parent / "fela-tab", HERE.parent, HERE / ".." / "fela-tab"):
+    if (_cand / "modeling.py").is_file():
+        FELA_DIR = _cand.resolve()
+        break
+else:
+    FELA_DIR = HERE.parent / "fela-tab"
+
+# --------------------------------------------------------------------------
+# Datasets
+# --------------------------------------------------------------------------
+# (display name, openml name, openml version)
+FULL_DATASETS = [
+    ("adult", "adult", 2),
+    ("credit-g", "credit-g", 1),
+    ("blood-transfusion", "blood-transfusion-service-center", 1),
+    ("churn", "churn", 1),
+    ("electricity", "electricity", 1),
+    ("vehicle", "vehicle", 1),
+    ("segment", "segment", 1),
+    ("jungle_chess", "jungle_chess_2pcs_raw_endgame_complete", 1),
+]
+SMOKE_DATASETS = [
+    ("breast-cancer", "wdbc", 1),
+    ("vehicle", "vehicle", 1),
+]
+MAX_CLASSES = 10  # FelaTab / TabFM constraint
+
+
+def load_openml(name: str, version: int):
+    """Return (X DataFrame, y int array, n_classes). Encodes categoricals, imputes."""
+    from sklearn.datasets import fetch_openml
+
+    ds = fetch_openml(name, version=version, as_frame=True, parser="auto")
+    X, y = ds.data, ds.target
+    # encode target
+    y = pd.Categorical(y)
+    y_int = y.codes.astype(np.int64)
+    n_classes = len(y.categories)
+    # encode features
+    X = X.copy()
+    for c in X.columns:
+        if not pd.api.types.is_numeric_dtype(X[c]):
+            X[c] = pd.Categorical(X[c]).codes.astype(np.float64)
+    X = X.astype(np.float64)
+    # impute with column median
+    X = X.fillna(X.median(numeric_only=True)).fillna(0.0)
+    return X, y_int, n_classes
+
+
+# --------------------------------------------------------------------------
+# Efficiency measurement helpers
+# --------------------------------------------------------------------------
+class Meter:
+    """Tracks wall time, peak process RAM (tracemalloc + psutil RSS), peak VRAM."""
+
+    def __init__(self, device: str):
+        self.device = device
+        self.proc = psutil.Process(os.getpid())
+        self._torch = None
+        if device.startswith("cuda"):
+            try:
+                import torch
+
+                self._torch = torch
+                torch.cuda.reset_peak_memory_stats()
+            except Exception:
+                pass
+
+    def __enter__(self):
+        tracemalloc.start()
+        self._rss0 = self.proc.memory_info().rss
+        self._peak_rss = self._rss0
+        self.t0 = time.perf_counter()
+        return self
+
+    def sample(self):
+        self._peak_rss = max(self._peak_rss, self.proc.memory_info().rss)
+
+    def __exit__(self, *exc):
+        self.elapsed = time.perf_counter() - self.t0
+        _, py_peak = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+        self.sample()
+        self.peak_ram_mb = max(py_peak, self._peak_rss - self._rss0) / 1e6
+        self.peak_vram_mb = (
+            self._torch.cuda.max_memory_allocated() / 1e6 if self._torch else None
+        )
+        return False
+
+
+@dataclass
+class Result:
+    dataset: str
+    model: str
+    device: str
+    roc_auc: float = np.nan
+    log_loss: float = np.nan
+    accuracy: float = np.nan
+    f1_macro: float = np.nan
+    fit_s: float = np.nan
+    latency_ms: float = np.nan  # per sample
+    peak_ram_mb: float = np.nan
+    peak_vram_mb: float = np.nan
+    status: str = "ok"
+    notes: str = ""
+
+
+# --------------------------------------------------------------------------
+# Model adapters: uniform fit_predict_proba(Xtr, ytr, Xte) -> (proba, fit_s, infer_s)
+# proba shape [n_test, n_classes], columns aligned with class index 0..K-1
+# --------------------------------------------------------------------------
+class Adapter:
+    name = "base"
+    device = "cpu"
+
+    def fit_predict_proba(self, Xtr, ytr, Xte, n_classes):
+        raise NotImplementedError
+
+
+class FelaTabAdapter(Adapter):
+    def __init__(self, tier: str, device: str):
+        import torch
+
+        sys.path.insert(0, str(FELA_DIR))
+        from modeling import load_model  # fela-tab/modeling.py
+
+        self.tier = tier
+        self.name = f"FelaTab-{tier}"
+        self.device = "gpu" if device.startswith("cuda") else "cpu"
+        dev = torch.device("cuda" if device.startswith("cuda") else "cpu")
+        self._predict_mod = __import__("modeling")
+        self.model = load_model(str(FELA_DIR), tier=tier).to(dev)
+
+    def fit_predict_proba(self, Xtr, ytr, Xte, n_classes):
+        from modeling import predict
+
+        t0 = time.perf_counter()
+        proba = predict(
+            self.model, Xtr, ytr, Xte,
+            task="classification", n_classes=n_classes, support_cap=3000,
+        )
+        total = time.perf_counter() - t0
+        # zero-shot: no fit; count full in-context forward as inference,
+        # report support ingestion separately via fit_s=0
+        return proba, 0.0, total
+
+
+class TabFMAdapter(Adapter):
+    """google/tabfm-1.0.0-pytorch zero-shot via the `tabfm` package.
+
+    Requires HF_TOKEN with accepted license (non-commercial, tabfm-non-commercial-v1.0).
+    Sklearn-style: TabFMClassifier.fit(X_train, y_train) -> predict_proba(X_test).
+    """
+
+    name = "TabFM"
+
+    def __init__(self, device: str):
+        from huggingface_hub import model_info
+
+        self.device = "gpu" if device.startswith("cuda") else "cpu"
+        token = os.environ.get("HF_TOKEN")
+        if not token:
+            raise RuntimeError(
+                "HF_TOKEN not set; google/tabfm-1.0.0-pytorch is a gated repo")
+        model_info("google/tabfm-1.0.0-pytorch", token=token)  # raises if unauthorized
+        import torch
+        from tabfm import TabFMClassifier, tabfm_v1_0_0_pytorch as tabfm_ckpt
+
+        self._torch = torch
+        self._dev = torch.device("cuda" if device.startswith("cuda") else "cpu")
+        model = tabfm_ckpt.load(model_type="classification")
+        try:
+            model = model.to(self._dev)
+        except Exception:
+            pass
+        self.clf = TabFMClassifier(model=model)
+
+    def fit_predict_proba(self, Xtr, ytr, Xte, n_classes):
+        if n_classes > MAX_CLASSES:
+            raise RuntimeError(f"TabFM supports <= {MAX_CLASSES} classes")
+        t0 = time.perf_counter()
+        self.clf.fit(Xtr, ytr)  # in-context: stores support rows, no training
+        fit_s = time.perf_counter() - t0
+        t1 = time.perf_counter()
+        proba = np.asarray(self.clf.predict_proba(Xte), dtype=np.float64)
+        infer_s = time.perf_counter() - t1
+        return proba[:, :n_classes], fit_s, infer_s
+
+
+class SklearnAdapter(Adapter):
+    def __init__(self, model, name):
+        self.model = model
+        self.name = name
+        self.device = "cpu"
+
+    def fit_predict_proba(self, Xtr, ytr, Xte, n_classes):
+        t0 = time.perf_counter()
+        self.model.fit(Xtr, ytr)
+        fit_s = time.perf_counter() - t0
+        t1 = time.perf_counter()
+        proba = self.model.predict_proba(Xte)
+        infer_s = time.perf_counter() - t1
+        return proba, fit_s, infer_s
+
+
+def build_models(device: str, tiers, skip: set) -> list[Adapter]:
+    models: list[Adapter] = []
+    for tier in tiers:
+        try:
+            models.append(FelaTabAdapter(tier, device))
+            print(f"[load] FelaTab-{tier} on {device}")
+        except Exception as e:
+            print(f"[skip] FelaTab-{tier}: {e}")
+    if "tabfm" not in skip:
+        try:
+            models.append(TabFMAdapter(device))
+            print(f"[load] TabFM on {device}")
+        except Exception as e:
+            print(f"[skip] TabFM: {e}")
+    if "xgb" not in skip:
+        from xgboost import XGBClassifier
+
+        models.append(SklearnAdapter(
+            XGBClassifier(n_estimators=300, max_depth=6, learning_rate=0.1,
+                          tree_method="hist", n_jobs=-1, random_state=SEED),
+            "XGBoost"))
+    if "lgbm" not in skip:
+        from lightgbm import LGBMClassifier
+
+        models.append(SklearnAdapter(
+            LGBMClassifier(n_estimators=300, learning_rate=0.1, n_jobs=-1,
+                           random_state=SEED, verbose=-1),
+            "LightGBM"))
+    return models
+
+
+# --------------------------------------------------------------------------
+# Evaluation
+# --------------------------------------------------------------------------
+def evaluate(y_true, proba, n_classes):
+    from sklearn.metrics import accuracy_score, f1_score, log_loss, roc_auc_score
+
+    proba = np.asarray(proba, dtype=np.float64)
+    proba = np.clip(proba, 1e-12, 1.0)
+    proba = proba / proba.sum(1, keepdims=True)
+    pred = proba.argmax(1)
+    acc = accuracy_score(y_true, pred)
+    f1 = f1_score(y_true, pred, average="macro")
+    labels = list(range(n_classes))
+    ll = log_loss(y_true, proba, labels=labels)
+    if n_classes == 2:
+        auc = roc_auc_score(y_true, proba[:, 1])
+    else:
+        auc = roc_auc_score(y_true, proba, multi_class="ovr", average="weighted",
+                            labels=labels)
+    return auc, ll, acc, f1
+
+
+def run(args) -> list[Result]:
+    from sklearn.model_selection import train_test_split
+
+    device = "cuda" if args.device == "gpu" else "cpu"
+    if device == "cuda":
+        import torch
+
+        if not torch.cuda.is_available():
+            print("[warn] GPU requested but unavailable; falling back to CPU")
+            device = "cpu"
+
+    dsets = SMOKE_DATASETS if args.smoke else FULL_DATASETS
+    if args.datasets:
+        keep = set(args.datasets.split(","))
+        dsets = [d for d in dsets if d[0] in keep]
+    tiers = ["small"] if args.smoke else args.tiers.split(",")
+    skip = set(args.skip.split(",")) if args.skip else set()
+    if args.smoke:
+        skip.add("tabfm")
+    models = build_models(device, tiers, skip)
+    if not models:
+        sys.exit("no models available")
+
+    results: list[Result] = []
+    for dname, oml_name, ver in dsets:
+        print(f"\n=== {dname} (openml:{oml_name} v{ver}) ===")
+        try:
+            X, y, n_classes = load_openml(oml_name, ver)
+        except Exception as e:
+            print(f"  [skip dataset] {e}")
+            continue
+        if n_classes > MAX_CLASSES:
+            print(f"  [skip dataset] {n_classes} classes > {MAX_CLASSES}")
+            continue
+        Xtr, Xte, ytr, yte = train_test_split(
+            X, y, test_size=0.2, random_state=SEED, stratify=y)
+        print(f"  train={len(Xtr)} test={len(Xte)} feats={Xtr.shape[1]} classes={n_classes}")
+        for m in models:
+            r = Result(dataset=dname, model=m.name, device=m.device)
+            try:
+                with Meter(device if m.device == "gpu" else "cpu") as meter:
+                    proba, fit_s, infer_s = m.fit_predict_proba(
+                        Xtr.to_numpy(), ytr, Xte.to_numpy(), n_classes)
+                r.fit_s = fit_s
+                r.latency_ms = 1e3 * infer_s / len(Xte)
+                r.peak_ram_mb = meter.peak_ram_mb
+                r.peak_vram_mb = meter.peak_vram_mb or np.nan
+                r.roc_auc, r.log_loss, r.accuracy, r.f1_macro = evaluate(
+                    yte, proba, n_classes)
+                print(f"  {m.name:<14} acc={r.accuracy:.4f} auc={r.roc_auc:.4f} "
+                      f"ll={r.log_loss:.4f} f1={r.f1_macro:.4f} "
+                      f"fit={r.fit_s:.2f}s infer={r.latency_ms:.3f}ms/s "
+                      f"ram={r.peak_ram_mb:.0f}MB vram={r.peak_vram_mb or 0:.0f}MB")
+            except Exception as e:
+                r.status = f"FAILED"
+                r.notes = str(e).split("\n")[0][:120]
+                print(f"  {m.name:<14} FAILED: {r.notes}")
+            results.append(r)
+    return results
+
+
+# --------------------------------------------------------------------------
+# Markdown model-card snippet
+# --------------------------------------------------------------------------
+def _bold_best(df: pd.DataFrame, col: str, higher=True) -> pd.Series:
+    best = df[col].max() if higher else df[col].min()
+    return df[col].map(lambda v: f"**{v:.4f}**" if v == best else f"{v:.4f}")
+
+
+def to_markdown(results: list[Result]) -> str:
+    df = pd.DataFrame([vars(r) for r in results])
+    ok = df[df.status == "ok"]
+    models = [m for m in df.model.unique()]
+    lines = ["## Benchmark results",
+             "",
+             "Zero-shot in-context models (FelaTab, TabFM) vs trained baselines "
+             "(XGBoost, LightGBM). OpenML datasets, stratified 80/20 split, seed 42.",
+             ""]
+
+    # 1) per-dataset performance tables
+    for metric, higher, title in [
+        ("accuracy", True, "Accuracy"), ("roc_auc", True, "ROC-AUC"),
+        ("log_loss", False, "Log Loss"), ("f1_macro", True, "F1 (macro)")]:
+        piv = ok.pivot_table(index="dataset", columns="model", values=metric)
+        lines.append(f"### {title} per dataset")
+        lines.append("")
+        lines.append("| Dataset | " + " | ".join(piv.columns) + " |")
+        lines.append("|" + "---|" * (len(piv.columns) + 1))
+        for d, row in piv.iterrows():
+            best = row.max() if higher else row.min()
+            cells = [f"**{v:.4f}**" if v == best else (f"{v:.4f}" if pd.notna(v) else "—")
+                     for v in row]
+            lines.append(f"| {d} | " + " | ".join(cells) + " |")
+        lines.append("")
+
+    # 2) summary with average rank
+    lines += ["### Summary (mean across datasets)", ""]
+    summ = ok.groupby("model").agg(
+        mean_acc=("accuracy", "mean"), mean_auc=("roc_auc", "mean"),
+        mean_logloss=("log_loss", "mean"), mean_f1=("f1_macro", "mean"))
+    ranks = []
+    for metric, higher in [("accuracy", True), ("roc_auc", True),
+                           ("log_loss", False), ("f1_macro", True)]:
+        piv = ok.pivot_table(index="dataset", columns="model", values=metric)
+        r = piv.rank(axis=1, ascending=not higher).mean()
+        ranks.append(r)
+    avg_rank = pd.concat(ranks, axis=1).mean(axis=1)
+    summ["avg_rank"] = avg_rank
+    lines.append("| Model | Mean Acc | Mean ROC-AUC | Mean LogLoss | Mean F1 | Avg Rank |")
+    lines.append("|---|---|---|---|---|---|")
+    for m in summ.index:
+        s = summ.loc[m]
+        lines.append(f"| {m} | {s.mean_acc:.4f} | {s.mean_auc:.4f} | "
+                     f"{s.mean_logloss:.4f} | {s.mean_f1:.4f} | **{s.avg_rank:.2f}** |")
+    lines.append("")
+
+    # 3) efficiency
+    lines += ["### Efficiency", "",
+              "| Model | Device | Fit time (s) | Latency (ms/sample) | Peak RAM (MB) | Peak VRAM (MB) |",
+              "|---|---|---|---|---|---|"]
+    for m in models:
+        sub = ok[ok.model == m]
+        if sub.empty:
+            continue
+        dev = sub.device.iloc[0]
+        vram = sub.peak_vram_mb.mean()
+        lines.append(f"| {m} | {dev} | {sub.fit_s.mean():.2f} | "
+                     f"{sub.latency_ms.mean():.3f} | {sub.peak_ram_mb.mean():.0f} | "
+                     f"{'N/A' if pd.isna(vram) or vram == 0 else f'{vram:.0f}'} |")
+    lines.append("")
+
+    failed = df[df.status != "ok"]
+    if not failed.empty:
+        lines += ["<details><summary>Failed runs</summary>", "",
+                  "| Dataset | Model | Reason |", "|---|---|---|"]
+        for _, f in failed.iterrows():
+            lines.append(f"| {f.dataset} | {f.model} | {f.notes} |")
+        lines += ["", "</details>", ""]
+    return "\n".join(lines)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--smoke", action="store_true")
+    ap.add_argument("--device", choices=["gpu", "cpu"], default="gpu")
+    ap.add_argument("--tiers", default="big,small")
+    ap.add_argument("--skip", default="")
+    ap.add_argument("--datasets", default="")
+    args = ap.parse_args()
+
+    results = run(args)
+    df = pd.DataFrame([vars(r) for r in results])
+    out_csv = HERE / "results.csv"
+    df.to_csv(out_csv, index=False)
+    md = to_markdown(results)
+    (HERE / "model_card_snippet.md").write_text(md)
+    print(f"\nwrote {out_csv} and {HERE/'model_card_snippet.md'}")
+    print("\n" + md)
+
+
+if __name__ == "__main__":
+    main()

benchmark/model_card_snippet.md

@@ -0,0 +1,75 @@
+## Benchmark results
+
+Zero-shot in-context models (FelaTab, TabFM) vs trained baselines (XGBoost, LightGBM). OpenML datasets, stratified 80/20 split, seed 42.
+
+### Accuracy per dataset
+
+| Dataset | FelaTab-big | FelaTab-small | LightGBM | TabFM | XGBoost |
+|---|---|---|---|---|---|
+| adult | 0.8303 | 0.8172 | 0.8746 | **0.8781** | 0.8772 |
+| blood-transfusion | 0.7867 | 0.7067 | 0.7533 | **0.7933** | 0.7400 |
+| churn | 0.8960 | 0.8970 | 0.9590 | **0.9750** | 0.9580 |
+| credit-g | **0.7350** | 0.7150 | 0.7000 | **0.7350** | 0.7250 |
+| electricity | 0.7087 | 0.7130 | 0.9104 | **0.9629** | 0.9040 |
+| jungle_chess | 0.7773 | 0.7685 | 0.8666 | **0.8881** | 0.8618 |
+| segment | 0.9242 | 0.9372 | 0.9805 | **0.9870** | 0.9784 |
+| vehicle | 0.7765 | 0.7882 | 0.7765 | **0.9000** | 0.7824 |
+
+### ROC-AUC per dataset
+
+| Dataset | FelaTab-big | FelaTab-small | LightGBM | TabFM | XGBoost |
+|---|---|---|---|---|---|
+| adult | 0.8795 | 0.8757 | 0.9293 | **0.9317** | 0.9305 |
+| blood-transfusion | 0.7773 | 0.6701 | 0.7393 | **0.7945** | 0.7617 |
+| churn | 0.8606 | 0.8637 | 0.9283 | **0.9408** | 0.9285 |
+| credit-g | 0.7635 | 0.7685 | 0.7268 | **0.7830** | 0.7431 |
+| electricity | 0.8102 | 0.8099 | 0.9717 | **0.9950** | 0.9692 |
+| jungle_chess | 0.9106 | 0.9050 | 0.9772 | **0.9838** | 0.9754 |
+| segment | 0.9947 | 0.9964 | 0.9995 | **0.9998** | 0.9996 |
+| vehicle | 0.9431 | 0.9422 | 0.9363 | **0.9820** | 0.9340 |
+
+### Log Loss per dataset
+
+| Dataset | FelaTab-big | FelaTab-small | LightGBM | TabFM | XGBoost |
+|---|---|---|---|---|---|
+| adult | 0.3861 | 0.3824 | 0.2743 | **0.2712** | 0.2721 |
+| blood-transfusion | 0.4584 | 0.5843 | 0.6274 | **0.4463** | 0.5444 |
+| churn | 0.2831 | 0.2807 | 0.2441 | **0.1019** | 0.1674 |
+| credit-g | 0.5250 | 0.5162 | 1.0375 | **0.4996** | 0.6554 |
+| electricity | 0.6046 | 0.5942 | 0.2243 | **0.0906** | 0.2327 |
+| jungle_chess | 0.5325 | 0.5482 | 0.2483 | **0.1912** | 0.2564 |
+| segment | 0.2035 | 0.1783 | 0.0862 | **0.0365** | 0.0612 |
+| vehicle | 0.4473 | 0.4559 | 0.8884 | **0.2259** | 0.4907 |
+
+### F1 (macro) per dataset
+
+| Dataset | FelaTab-big | FelaTab-small | LightGBM | TabFM | XGBoost |
+|---|---|---|---|---|---|
+| adult | 0.7445 | 0.7139 | 0.8171 | **0.8207** | 0.8204 |
+| blood-transfusion | 0.6032 | 0.5298 | 0.6445 | **0.6643** | 0.6330 |
+| churn | 0.7279 | 0.7401 | 0.9079 | **0.9453** | 0.9084 |
+| credit-g | 0.6619 | 0.6273 | 0.6429 | **0.6696** | 0.6644 |
+| electricity | 0.7014 | 0.7007 | 0.9083 | **0.9621** | 0.9017 |
+| jungle_chess | 0.6066 | 0.6096 | 0.8149 | **0.8400** | 0.8117 |
+| segment | 0.9231 | 0.9372 | 0.9805 | **0.9871** | 0.9785 |
+| vehicle | 0.7760 | 0.7872 | 0.7745 | **0.8978** | 0.7826 |
+
+### Summary (mean across datasets)
+
+| Model | Mean Acc | Mean ROC-AUC | Mean LogLoss | Mean F1 | Avg Rank |
+|---|---|---|---|---|---|
+| FelaTab-big | 0.8043 | 0.8674 | 0.4301 | 0.7181 | **3.91** |
+| FelaTab-small | 0.7929 | 0.8539 | 0.4425 | 0.7057 | **4.06** |
+| LightGBM | 0.8526 | 0.9011 | 0.4538 | 0.8113 | **3.14** |
+| TabFM | 0.8899 | 0.9263 | 0.2329 | 0.8484 | **1.02** |
+| XGBoost | 0.8534 | 0.9052 | 0.3350 | 0.8126 | **2.88** |
+
+### Efficiency
+
+| Model | Device | Fit time (s) | Latency (ms/sample) | Peak RAM (MB) | Peak VRAM (MB) |
+|---|---|---|---|---|---|
+| FelaTab-big | gpu | 0.00 | 1.755 | 65 | 5531 |
+| FelaTab-small | gpu | 0.00 | 0.791 | 10 | 5390 |
+| TabFM | gpu | 0.24 | 52.431 | 130 | 7142 |
+| XGBoost | cpu | 6.35 | 0.025 | 5 | N/A |
+| LightGBM | cpu | 6.66 | 0.008 | 7 | N/A |

benchmark/requirements.txt

@@ -0,0 +1,10 @@
+scikit-learn>=1.5
+pandas>=2.0
+xgboost>=2.0
+lightgbm>=4.0
+psutil>=5.9
+transformers>=4.45
+safetensors>=0.5
+huggingface-hub>=0.30
+# NOTE: torch with ROCm is already installed system-wide (do NOT pip install stock torch,
+# it would clobber the ROCm build). The venv is created with --system-site-packages.

benchmark/results.csv

@@ -0,0 +1,41 @@
+dataset,model,device,roc_auc,log_loss,accuracy,f1_macro,fit_s,latency_ms,peak_ram_mb,peak_vram_mb,status,notes
+credit-g,FelaTab-big,gpu,0.7635,0.525,0.735,0.6619,0.0,4.551,233.0,5297.0,ok,
+credit-g,FelaTab-small,gpu,0.7685,0.5162,0.715,0.6273,0.0,1.247,18.0,5271.0,ok,
+credit-g,TabFM,gpu,0.783,0.4996,0.735,0.6696,0.27,23.476,126.0,5465.0,ok,
+credit-g,XGBoost,cpu,0.7431,0.6554,0.725,0.6644,5.1,0.065,8.0,,ok,
+credit-g,LightGBM,cpu,0.7268,1.0375,0.7,0.6429,2.38,0.009,4.0,,ok,
+blood-transfusion,FelaTab-big,gpu,0.7773,0.4584,0.7867,0.6032,0.0,2.139,1.0,5287.0,ok,
+blood-transfusion,FelaTab-small,gpu,0.6701,0.5843,0.7067,0.5298,0.0,1.123,1.0,5268.0,ok,
+blood-transfusion,TabFM,gpu,0.7945,0.4463,0.7933,0.6643,0.06,17.228,2.0,5331.0,ok,
+blood-transfusion,XGBoost,cpu,0.7617,0.5444,0.74,0.633,1.23,0.021,1.0,,ok,
+blood-transfusion,LightGBM,cpu,0.7393,0.6274,0.7533,0.6445,6.55,0.017,3.0,,ok,
+churn,FelaTab-big,gpu,0.8606,0.2831,0.896,0.7279,0.0,1.412,5.0,5445.0,ok,
+churn,FelaTab-small,gpu,0.8637,0.2807,0.897,0.7401,0.0,0.84,5.0,5347.0,ok,
+churn,TabFM,gpu,0.9408,0.1019,0.975,0.9453,0.25,29.61,58.0,6145.0,ok,
+churn,XGBoost,cpu,0.9285,0.1674,0.958,0.9084,2.58,0.003,1.0,,ok,
+churn,LightGBM,cpu,0.9283,0.2441,0.959,0.9079,0.94,0.002,4.0,,ok,
+electricity,FelaTab-big,gpu,0.8102,0.6046,0.7087,0.7014,0.0,0.461,17.0,5847.0,ok,
+electricity,FelaTab-small,gpu,0.8099,0.5942,0.713,0.7007,0.0,0.274,17.0,5548.0,ok,
+electricity,TabFM,gpu,0.995,0.0906,0.9629,0.9621,0.23,96.876,231.0,9723.0,ok,
+electricity,XGBoost,cpu,0.9692,0.2327,0.904,0.9017,1.62,0.002,12.0,,ok,
+electricity,LightGBM,cpu,0.9717,0.2243,0.9104,0.9083,2.57,0.001,6.0,,ok,
+vehicle,FelaTab-big,gpu,0.9431,0.4473,0.7765,0.776,0.0,2.63,8.0,5294.0,ok,
+vehicle,FelaTab-small,gpu,0.9422,0.4559,0.7882,0.7872,0.0,1.206,1.0,5270.0,ok,
+vehicle,TabFM,gpu,0.982,0.2259,0.9,0.8978,0.22,22.292,9.0,5416.0,ok,
+vehicle,XGBoost,cpu,0.934,0.4907,0.7824,0.7826,11.57,0.078,1.0,,ok,
+vehicle,LightGBM,cpu,0.9363,0.8884,0.7765,0.7745,8.64,0.024,8.0,,ok,
+segment,FelaTab-big,gpu,0.9947,0.2035,0.9242,0.9231,0.0,1.824,3.0,5362.0,ok,
+segment,FelaTab-small,gpu,0.9964,0.1783,0.9372,0.9372,0.0,1.092,3.0,5305.0,ok,
+segment,TabFM,gpu,0.9998,0.0365,0.987,0.9871,0.24,23.559,25.0,5712.0,ok,
+segment,XGBoost,cpu,0.9996,0.0612,0.9784,0.9785,3.85,0.023,4.0,,ok,
+segment,LightGBM,cpu,0.9995,0.0862,0.9805,0.9805,8.34,0.007,8.0,,ok,
+jungle_chess,FelaTab-big,gpu,0.9106,0.5325,0.7773,0.6066,0.0,0.46,16.0,5839.0,ok,
+jungle_chess,FelaTab-small,gpu,0.905,0.5482,0.7685,0.6096,0.0,0.274,16.0,5545.0,ok,
+jungle_chess,TabFM,gpu,0.9838,0.1912,0.8881,0.84,0.15,94.106,182.0,9121.0,ok,
+jungle_chess,XGBoost,cpu,0.9754,0.2564,0.8618,0.8117,19.06,0.003,2.0,,ok,
+jungle_chess,LightGBM,cpu,0.9772,0.2483,0.8666,0.8149,12.73,0.004,12.0,,ok,
+adult,FelaTab-big,gpu,0.8795,0.3861,0.8303,0.7445,0.0,0.567,234.0,5880.0,ok,
+adult,FelaTab-small,gpu,0.8757,0.3824,0.8172,0.7139,0.0,0.273,21.0,5566.0,ok,
+adult,TabFM,gpu,0.9317,0.2712,0.8781,0.8207,0.52,112.3,407.0,10226.0,ok,
+adult,XGBoost,cpu,0.9305,0.2721,0.8772,0.8204,5.77,0.002,12.0,,ok,
+adult,LightGBM,cpu,0.9293,0.2743,0.8746,0.8171,11.14,0.001,9.0,,ok,

benchmark/tabarena/fela_ag_model.py

@@ -0,0 +1,128 @@
+"""FelaTab as an AutoGluon AbstractModel for TabArena benchmarking.
+
+FelaTab is a zero-shot in-context tabular model: `_fit` stores the train split as
+support rows (capped), `_predict_proba` runs a single in-context forward pass.
+Hard limits: <= 10 classes, <= 100 raw features (PCA-reduced internally beyond that),
+support set capped at 3000 rows.
+
+Must live in an importable module (Ray workers cannot unpickle classes from __main__).
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+import numpy as np
+from autogluon.core.models import AbstractModel
+from autogluon.features import LabelEncoderFeatureGenerator
+
+if TYPE_CHECKING:
+    import pandas as pd
+
+    from tabarena.utils.config_utils import ConfigGenerator
+
+_here = Path(__file__).resolve()
+# works both as <workspace>/benchmark/tabarena/fela_ag_model.py and inside the
+# fela-tab repo as fela-tab/benchmark/tabarena/fela_ag_model.py
+FELA_DIR = None
+for _cand in (_here.parents[2] / "fela-tab", _here.parents[2]):
+    if (_cand / "modeling.py").is_file():
+        FELA_DIR = _cand
+        break
+assert FELA_DIR is not None, "could not locate fela-tab repo (modeling.py)"
+if str(FELA_DIR) not in sys.path:
+    sys.path.insert(0, str(FELA_DIR))
+
+MAX_CLASSES = 10
+SUPPORT_CAP = 3000
+
+
+class FelaTabAGModel(AbstractModel):
+    ag_key = "FELA"
+    ag_name = "FelaTab"
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self._feature_generator = None
+        self._fela_model = None
+        self._X_support = None
+        self._y_support = None
+
+    def _preprocess(self, X: "pd.DataFrame", is_train: bool = False, **kwargs) -> np.ndarray:
+        X = super()._preprocess(X, **kwargs)
+        if is_train:
+            self._feature_generator = LabelEncoderFeatureGenerator(verbosity=0)
+            self._feature_generator.fit(X=X)
+        if self._feature_generator.features_in:
+            X = X.copy()
+            X[self._feature_generator.features_in] = self._feature_generator.transform(X=X)
+        return X.fillna(0).to_numpy(dtype=np.float32)
+
+    def _fit(self, X: "pd.DataFrame", y: "pd.Series", num_cpus: int = 1, **kwargs) -> None:
+        if self.problem_type == "regression":
+            raise ValueError("FelaTabAGModel benchmark wrapper is classification-only")
+        n_classes = len(np.unique(y))
+        if n_classes > MAX_CLASSES:
+            raise ValueError(f"FelaTab supports <= {MAX_CLASSES} classes, got {n_classes}")
+
+        from modeling import load_model
+
+        params = self._get_model_params()
+        tier = params.get("tier", "small")
+        self._fela_model = load_model(str(FELA_DIR), tier=tier)
+        if params.get("device", "cpu") == "gpu":
+            import torch
+
+            if torch.cuda.is_available():
+                self._fela_model = self._fela_model.to("cuda")
+
+        X_np = self.preprocess(X, y=y, is_train=True)
+        y_np = np.asarray(y)
+        if len(X_np) > SUPPORT_CAP:
+            idx = np.random.RandomState(0).permutation(len(X_np))[:SUPPORT_CAP]
+            X_np, y_np = X_np[idx], y_np[idx]
+        self._X_support, self._y_support = X_np, y_np
+
+    def _predict_proba(self, X: "pd.DataFrame", **kwargs) -> np.ndarray:
+        from modeling import predict
+
+        X_np = self.preprocess(X)
+        n_classes = len(self.classes_) if hasattr(self, "classes_") else int(
+            self._y_support.max() + 1)
+        proba = predict(
+            self._fela_model, self._X_support, self._y_support, X_np,
+            task="classification", n_classes=n_classes, support_cap=SUPPORT_CAP,
+        )
+        # guard against NaN/inf or degenerate rows: AutoGluon drops NaN-pred rows when
+        # scoring, which can leave a single class in y_true on small folds
+        proba = np.asarray(proba, dtype=np.float64)
+        proba = np.nan_to_num(proba, nan=1.0 / n_classes, posinf=1.0, neginf=0.0)
+        proba = np.clip(proba, 1e-9, None)
+        proba /= proba.sum(axis=1, keepdims=True)
+        # AutoGluon contract: binary problems must return 1-D positive-class proba
+        return self._convert_proba_to_unified_form(proba)
+
+    def _set_default_params(self) -> None:
+        for param, val in {"tier": "small", "device": "cpu"}.items():
+            self._set_default_param_value(param, val)
+
+    def _get_default_auxiliary_params(self) -> dict:
+        default_auxiliary_params = super()._get_default_auxiliary_params()
+        default_auxiliary_params.update({"valid_raw_types": ["int", "float", "category"]})
+        return default_auxiliary_params
+
+    @classmethod
+    def supported_problem_types(cls) -> list[str]:
+        return ["binary", "multiclass"]
+
+    @classmethod
+    def config_generator(cls) -> "ConfigGenerator":
+        from tabarena.utils.config_utils import ConfigGenerator
+
+        return ConfigGenerator(
+            model_cls=cls,
+            manual_configs=[{"tier": "small"}, {"tier": "big"}],
+            search_space={},
+        )

benchmark/tabarena/run_tabarena.py

@@ -0,0 +1,70 @@
+"""Run FelaTab on TabArena and compare against the official leaderboard.
+
+Usage (from this directory, with the tabarena venv python):
+  ../tabarena-repo/.venv/bin/python run_tabarena.py --quickstart        # 3 small lite datasets
+  ../tabarena-repo/.venv/bin/python run_tabarena.py --subset lite       # full TabArena-Lite
+  ../tabarena-repo/.venv/bin/python run_tabarena.py --full              # all 51 datasets (hours)
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+from tabarena.benchmark.experiment import TabArenaV0pt1ExperimentBundle
+from tabarena.contexts import TabArenaContext
+
+from fela_ag_model import FelaTabAGModel
+
+QUICKSTART_DATASETS = ["blood-transfusion-service-center", "QSAR_fish_toxicity", "anneal"]
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--quickstart", action="store_true", default=True)
+    ap.add_argument("--subset", default=None, help="tabarena subset, e.g. 'lite'")
+    ap.add_argument("--full", action="store_true")
+    ap.add_argument("--run-name", default="felatab_tabarena")
+    args = ap.parse_args()
+
+    here = Path(__file__).parent
+    results_dir = str(here / "experiments" / args.run_name)
+    eval_dir = here / "eval" / args.run_name
+
+    experiments = TabArenaV0pt1ExperimentBundle(
+        models=[
+            (FelaTabAGModel.config_generator(), 0),  # 0 = default/manual configs only
+            ("LightGBM", 0),  # cross-check against a known registry model
+        ],
+    ).build_experiments()
+
+    build_kwargs = {}
+    subset = args.subset
+    if args.full:
+        subset = None
+    elif not subset:
+        subset = "lite"
+        build_kwargs["dataset_names"] = QUICKSTART_DATASETS
+
+    context = TabArenaContext()
+    context.build_and_run_jobs(
+        experiments,
+        expname=results_dir,
+        subset=subset,
+        build_kwargs=build_kwargs,
+        new_result_prefix="[New] ",
+        debug_mode=True,
+        raise_on_failure=False,  # log failed tasks (e.g. >10-class guard) and continue
+    )
+
+    leaderboard = context.compare(output_dir=eval_dir)
+    leaderboard_website = context.leaderboard_to_website_format(leaderboard=leaderboard)
+    print("\n=== TabArena leaderboard (website format) ===")
+    print(leaderboard_website.to_markdown(index=False))
+    (eval_dir / "leaderboard_website.md").write_text(
+        leaderboard_website.to_markdown(index=False))
+    print(f"\nSaved to {eval_dir}")
+
+
+if __name__ == "__main__":
+    main()