itstheraj commited on
Commit
568e1c1
·
verified ·
1 Parent(s): 76fb9e6

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +30 -6
README.md CHANGED
@@ -158,12 +158,36 @@ local GPU (ROCm), the GBMs on CPU. Reproduce with `benchmark/benchmark.py`.
158
 
159
  ## TabArena
160
 
161
- FelaTab is wrapped as an AutoGluon model (`benchmark/tabarena/fela_ag_model.py`) and runnable
162
- against the official [TabArena](https://huggingface.co/spaces/TabArena/leaderboard) leaderboard
163
- with `benchmark/tabarena/run_tabarena.py`. On the 3-dataset quickstart subset FelaTab-big scored
164
- ELO ~938 (small ~897), mid-pack alongside default XGBoost, with 33% of tasks imputed (regression
165
- tasks are out of scope for the classification wrapper). Full TabArena-Lite / full-benchmark runs
166
- are in progress; see `benchmark/RUNNING.md`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
167
 
168
  ## Regression (5 datasets)
169
 
 
158
 
159
  ## TabArena
160
 
161
+ FelaTab ran the **full TabArena v0.1 suite** (51 datasets, all CV splits, 2448 task-fits,
162
+ 0% imputed) with `benchmark/tabarena/run_tabarena.py` against the official
163
+ [TabArena](https://huggingface.co/spaces/TabArena/leaderboard) cached leaderboard
164
+ (83 method configs). Classification is 100% native FelaTab, zero-shot; regression tasks are
165
+ delegated to a LightGBM regressor inside the wrapper (composite system FelaTab's native
166
+ regression head benchmarks below default RandomForest today; the delegate mirrors how the
167
+ model ships in production).
168
+
169
+ | Config | Elo | Avg rank |
170
+ |---|---|---|
171
+ | FelaTab big, bagged (`n_bag=8`) | **985** | 75 / 83 |
172
+ | FelaTab big (411.9M params) | **972** | 76 / 83 |
173
+ | FelaTab small (51.6M params) | **893** (default = small tier in this run) | 80 / 83 |
174
+
175
+ The bagged config (p-value 0.044) permutes row + column order over `n_bag=8` support
176
+ subsets and averages the predictions — measured +1.4 AUC on adult over a single pass.
177
+ An earlier 2-config run (big+small, 1632 tasks) landed 913 / 896 and is still in the
178
+ results repo (`eval/felatab_full/`).
179
+
180
+ Reference points from the same leaderboard: XGBoost (default) 1203, RandomForest (default)
181
+ 1000, Linear (tuned) 938, KNN (tuned) 890; TabPFNv2-family and tuned GBMs sit at 1300–1900 on
182
+ GPU. FelaTab's pitch is not the Elo column: it is the only zero-shot entry that runs
183
+ CPU-native (everything above it is GPU or trained per-dataset), at 1/32 the parameter count of
184
+ comparable tabular FMs, with the int8 tier deployable in-browser (WASM) and inside Postgres.
185
+
186
+ Raw per-task results, the full leaderboard table, and reproducibility scripts live in
187
+ [lowdown-labs/fela-tab-tabarena-results](https://huggingface.co/datasets/lowdown-labs/fela-tab-tabarena-results)
188
+ (`raw/` = per-fold results.pkl, `eval/felatab_full/` = merged leaderboard + figures).
189
+ Reproduce with `benchmark/tabarena/launch_ec2.py` + `ec2_run.sh` (~$70 on one
190
+ c7i.8xlarge, sharded across two).
191
 
192
  ## Regression (5 datasets)
193