| --- |
| license: mit |
| pipeline_tag: tabular-classification |
| library_name: tabum |
| tags: |
| - tabular |
| - tabular-classification |
| - tabular-regression |
| - in-context-learning |
| - foundation-model |
| - safetensors |
| --- |
| |
| # tabUM v1.1 |
|
|
| A small tabular in-context-learning foundation model: **14.31M parameters**, |
| zero-shot classification and regression on any table, native missing-value |
| handling, up to **100 classes** and **2,000 features**. MIT licensed, built |
| clean-room (no third-party model code). |
|
|
| **Code, notebook, and full results:** [github.com/heldernoid/tabum](https://github.com/heldernoid/tabum) |
|
|
| ## What it does |
|
|
| tabUM never trains on your data. `fit(X, y)` stores your rows; `predict(X_test)` |
| runs a single forward pass in which every test row attends to every training row |
| (in-context learning). Pretraining happened once, on millions of synthetic tasks |
| sampled from causal graphs, Gaussian processes, and random tree ensembles. |
|
|
| - prediction in seconds, no tuning, no preprocessing pipelines |
| - `NaN` is a first-class value: missingness is signal, never imputed |
| - calibrated probabilities; full regression distributions (any quantile) in one pass |
| - up to 100 classes (most tabular foundation models cap at 10) |
| - optional `finetune()` (~15s per-dataset adaptation) and `explain()` |
| (feature importances + the exact training rows the prediction attended to) |
|
|
| ## Usage |
|
|
| ```bash |
| pip install git+https://github.com/heldernoid/tabum |
| ``` |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from tabum.model import TabUM |
| from tabum.inference import TabUMClassifier, TabUMRegressor |
| |
| model = TabUM.from_pretrained(snapshot_download("helmo/tabum-v1.1"), device="cuda") |
| |
| clf = TabUMClassifier(model=model, n_ensemble=8).fit(X_train, y_train) |
| proba = clf.predict_proba(X_test) |
| |
| reg = TabUMRegressor(model=model, n_ensemble=8).fit(X_train, y_train) |
| p90 = reg.predict_quantile(X_test, 0.9) |
| ``` |
|
|
| ## Evaluation |
|
|
| TabArena-v0.1, all 51 datasets, identical protocol for every model (at most |
| 2,000 train / 1,000 test rows, single split, seed 0; competitors run as |
| black-box pip packages on the same splits). Per-dataset numbers are in the |
| GitHub repository. |
|
|
| | model | params | cls accuracy (38) | reg R² (13) | |
| |---|---|---|---| |
| | logistic / linear regression (fitted) | | 0.8417 | 0.5597 | |
| | tabUM zero-shot, 1 pass | 14M | 0.8523 | 0.6284 | |
| | tabUM zero-shot, `n_ensemble=8` | 14M | 0.8581 | 0.6519 | |
| | **tabUM `finetune()` + `n_ensemble=8`** | 14M | **0.8635** | **0.6907** | |
| | HistGradientBoosting (fitted, default) | | 0.8669 | 0.7091 | |
| | TabPFN v2 (zero-shot) | ~11M | 0.8768 | 0.7471 | |
| | TabICL (zero-shot) | ~500M | 0.8801 | n/a | |
| |
| Strengths and honest limitations: |
| |
| - beats fitted linear baselines everywhere, zero-shot |
| - finetuned, it ties default gradient boosting on classification (20/38 wins) |
| and trails it by 1.8 R² points on regression |
| - 1.3 to 1.7 accuracy points behind TabPFN v2 and TabICL (the state of the art |
| in this class), at 14M parameters and MIT license |
| - many-class is a differentiator: letter (26 classes) scores 0.79 vs 0.74 for |
| fitted logistic regression; TabPFN-class models cap at 10 classes |
| - strongest regime: small noisy tables (under ~1,000 rows), where the learned |
| prior beats boosting; on Titanic with 12% missing cells fed raw it beats an |
| imputed+scaled logistic regression pipeline |
| - context lengths validated to 64k rows (accuracy keeps improving with more |
| context) |
| |
| ## Model details |
| |
| | | | |
| |---|---| |
| | architecture | 3-stage tabular ICL transformer: Fourier cell embeddings with inducing-point column attention, per-row CLS aggregation, 10-block row transformer (train rows bidirectional, test rows attend train-only), retrieval-attention classification head, bar-distribution regression head | |
| | parameters | 14,306,051 | |
| | training | 20,000 steps of continued pretraining from tabUM v1 (50,000 steps), on synthetic tasks only, streamed on the fly (no task seen twice), 50/50 classification/regression, up to 16,384 rows / 2,000 features / 100 classes | |
| | hardware | one NVIDIA DGX Spark (GB10, 128GB unified memory), ~7h | |
| | data | 100% synthetic (causal graphs / Gaussian processes / random trees); no real-world data, no OpenML data, and no other model's outputs were used in training | |
| | license | MIT (weights and code) | |
| |
| ## Intended use and limitations |
| |
| Intended for tabular prediction on datasets from roughly 50 to tens of |
| thousands of rows, up to 2,000 features, up to 100 classes. Not a language |
| model: free-text columns should be dropped or encoded upstream. Regression on |
| large, low-noise datasets is better served by gradient boosting; tabUM's edge |
| is speed-to-first-prediction, small data, many classes, missing values, and |
| built-in explanations. |
| |
