Upload benchmark/tabarena/SUBMISSION_DRAFT.md with huggingface_hub

benchmark/tabarena/SUBMISSION_DRAFT.md

@@ -0,0 +1,93 @@
+# DRAFT — TabArena submission issue (DO NOT POST YET)
+
+**Where:** https://github.com/autogluon/tabarena/issues/new
+**Title:** New method submission: FelaTab — zero-shot in-context tabular model (51.6M/411.9M params, CPU-native; composite w/ LightGBM regression)
+
+---
+
+## Summary
+
+I'd like to submit **FelaTab** ([lowdown-labs/fela-tab](https://huggingface.co/lowdown-labs/fela-tab))
+for inclusion in the TabArena leaderboard. FelaTab is a zero-shot, in-context tabular
+foundation model (prior-fitted network with delta-rule linear attention over rows). For
+classification it has no per-dataset training: `_fit` stores support rows and
+`_predict_proba` is a single in-context forward pass.
+
+**Composite disclosure:** FelaTab's native regression head benchmarks below default
+RandomForest on TabArena regression tasks (our smoke run on 5 TabArena regression
+datasets: native head ELO ~400 vs RF-default 1000, per-dataset RMSE below). The
+submitted configuration therefore delegates **regression tasks to a LightGBM
+regressor** inside the same AutoGluon wrapper (`regression_backend="lgbm"`) — this
+mirrors how the model is deployed in our product. Classification is 100% native
+FelaTab, zero-shot. If you prefer a pure-model entry, `regression_backend="native"`
+reproduces the all-FelaTab variant and we can submit that instead (or both).
+
+Notable properties that make it an interesting leaderboard addition:
+
+- **CPU-native / subquadratic**: fixed-size working state (no growing KV cache), flat memory
+  as support rows grow; ships an int8 tier that runs in-browser via WASM and inside Postgres.
+- **Tiny footprint**: 51.6M (small) / 411.9M (big) params — roughly 1/32 and 1/4 the size of
+  comparable zero-shot tabular FMs (e.g. TabFM 1.0 is 1.64B).
+- **Speed**: ~0.8 ms/sample (small) and ~1.8 ms/sample (big) end-to-end inference latency in
+  our OpenML battery (vs 52 ms/sample for TabFM on the same ROCm GPU).
+
+Constraints: classification <= 10 classes, <= 100 raw features, support set capped at
+3000 rows (subsampled deterministically beyond that). Regression handled by the
+LightGBM delegate (see disclosure above).
+
+## Results
+
+- **Full TabArena v0.1** (51 datasets, all splits — 2448 task-fits, 0% imputed):
+  FELA (tuned + ensembled, `n_bag=8` bagged) **Elo 985** (avg rank 75/83), FELA (tuned, big tier)
+  **Elo 972** (rank 76/83), FELA (default, small tier) **Elo 893** (rank 80/83) on the
+  83-config leaderboard. Per-dataset results and figures:
+  **[lowdown-labs/fela-tab-tabarena-results](https://huggingface.co/datasets/lowdown-labs/fela-tab-tabarena-results)**
+  → `eval/felatab_v2_full/` (merged leaderboard), `raw/felatab_v2_full_{od0,od1,spot1-4}`
+  (per-fold `results.pkl`, merged).
+  An earlier 2-config full run (big/small, 1632 tasks, Elo 913/896) is in `eval/felatab_full/`.
+- **TabArena-Lite** (51 datasets, split 0): Elo 941 (tuned+ensembled) / 931 (tuned) / 850 (default),
+  0% imputed, `eval/felatab_v2_lite/`. (Earlier 2-config lite: 875/852, `eval/felatab_ec2/`.)
+- Independent OpenML battery (8 classification datasets, vs TabFM zero-shot, XGBoost,
+  LightGBM): see the [model card](https://huggingface.co/lowdown-labs/fela-tab) and
+  `benchmark/results.csv` in the model repo.
+- Regression smoke comparison (5 TabArena regression datasets, split 0, RMSE):
+
+| dataset | FelaTab native | FelaTab composite (LGBM) | GBM (default) |
+|---|---|---|---|
+| QSAR_fish_toxicity | 0.950 | 0.937 | 0.922 |
+| airfoil_self_noise | 2.610 | 1.580 | 1.485 |
+| concrete_compressive_strength | 7.254 | 4.653 | 4.610 |
+| houses | 0.335 | 0.221 | 0.219 |
+| wine_quality | — | 0.626 | 0.618 |
+
+## Reproduction
+
+- Model wrapper: [`benchmark/tabarena/fela_ag_model.py`](https://huggingface.co/lowdown-labs/fela-tab/blob/main/benchmark/tabarena/fela_ag_model.py)
+  (AutoGluon `AbstractModel`, `ag_key="FELA"`, three manual configs: tier small, tier big,
+  and a bagged big (`n_bag=8`) config; no HPO search space; LightGBM regression delegate on
+  by default).
+- Runner: [`benchmark/tabarena/run_tabarena.py`](https://huggingface.co/lowdown-labs/fela-tab/blob/main/benchmark/tabarena/run_tabarena.py)
+  (standard `TabArenaV0pt1ExperimentBundle` + `build_and_run_jobs`, `debug_mode=True`).
+- Env: `uv pip install --prerelease=allow -e "./packages/tabarena[benchmark]"` on Python 3.12
+  (see `benchmark/tabarena/README.md`).
+- The full run was executed on a plain EC2 c7i.8xlarge (32 vCPU, on-demand) from
+  [`benchmark/tabarena/ec2_run.sh`](https://huggingface.co/lowdown-labs/fela-tab/blob/main/benchmark/tabarena/ec2_run.sh)
+  (launched by `launch_ec2.py`), which reproduces env setup, lite shakedown, full run,
+  and artifact upload end-to-end. (We moved off HF Jobs after repeated job failures.)
+
+## Notes for maintainers
+
+- Both tiers ran as separate configs from one MatFormer nested checkpoint (small is a strict
+  prefix slice of big); happy to submit only the big tier if you prefer one config per method.
+- All classification evaluation was zero-shot, no per-dataset tuning; the model's training
+  prior excludes the benchmarked OpenML datasets by dataset id and name.
+- Regression is a LightGBM delegate (see composite disclosure above); no imputed tasks.
+- License: Apache-2.0 (weights + code).
+
+---
+
+### Checklist before posting
+- [x] Lite results verified in `lowdown-labs/fela-tab-tabarena-results`
+- [x] Full 51-dataset run complete and uploaded
+- [x] Model card TabArena section updated with final ELO/rank
+- [ ] Decide whether to also open a thread on the leaderboard Space Community tab