helmo/tabum-v1.1

tabUM v1.1

A small tabular in-context-learning foundation model: 14.31M parameters, zero-shot classification and regression on any table, native missing-value handling, up to 100 classes and 2,000 features. MIT licensed, built clean-room (no third-party model code).

Code, notebook, and full results: github.com/heldernoid/tabum

What it does

tabUM never trains on your data. fit(X, y) stores your rows; predict(X_test) runs a single forward pass in which every test row attends to every training row (in-context learning). Pretraining happened once, on millions of synthetic tasks sampled from causal graphs, Gaussian processes, and random tree ensembles.

• prediction in seconds, no tuning, no preprocessing pipelines
• NaN is a first-class value: missingness is signal, never imputed
• calibrated probabilities; full regression distributions (any quantile) in one pass
• up to 100 classes (most tabular foundation models cap at 10)
• optional finetune() (~15s per-dataset adaptation) and explain() (feature importances + the exact training rows the prediction attended to)

Usage

pip install git+https://github.com/heldernoid/tabum

from huggingface_hub import snapshot_download
from tabum.model import TabUM
from tabum.inference import TabUMClassifier, TabUMRegressor

model = TabUM.from_pretrained(snapshot_download("helmo/tabum-v1.1"), device="cuda")

clf = TabUMClassifier(model=model, n_ensemble=8).fit(X_train, y_train)
proba = clf.predict_proba(X_test)

reg = TabUMRegressor(model=model, n_ensemble=8).fit(X_train, y_train)
p90 = reg.predict_quantile(X_test, 0.9)

Evaluation

TabArena-v0.1, all 51 datasets, identical protocol for every model (at most 2,000 train / 1,000 test rows, single split, seed 0; competitors run as black-box pip packages on the same splits). Per-dataset numbers are in the GitHub repository.

model	params	cls accuracy (38)	reg R² (13)
logistic / linear regression (fitted)		0.8417	0.5597
tabUM zero-shot, 1 pass	14M	0.8523	0.6284
tabUM zero-shot, n_ensemble=8	14M	0.8581	0.6519
tabUM finetune() + n_ensemble=8	14M	0.8635	0.6907
HistGradientBoosting (fitted, default)		0.8669	0.7091
TabPFN v2 (zero-shot)	~11M	0.8768	0.7471
TabICL (zero-shot)	~500M	0.8801	n/a

Strengths and honest limitations:

• beats fitted linear baselines everywhere, zero-shot
• finetuned, it ties default gradient boosting on classification (20/38 wins) and trails it by 1.8 R² points on regression
• 1.3 to 1.7 accuracy points behind TabPFN v2 and TabICL (the state of the art in this class), at 14M parameters and MIT license
• many-class is a differentiator: letter (26 classes) scores 0.79 vs 0.74 for fitted logistic regression; TabPFN-class models cap at 10 classes
• strongest regime: small noisy tables (under ~1,000 rows), where the learned prior beats boosting; on Titanic with 12% missing cells fed raw it beats an imputed+scaled logistic regression pipeline
• context lengths validated to 64k rows (accuracy keeps improving with more context)

Model details

architecture	3-stage tabular ICL transformer: Fourier cell embeddings with inducing-point column attention, per-row CLS aggregation, 10-block row transformer (train rows bidirectional, test rows attend train-only), retrieval-attention classification head, bar-distribution regression head
parameters	14,306,051
training	20,000 steps of continued pretraining from tabUM v1 (50,000 steps), on synthetic tasks only, streamed on the fly (no task seen twice), 50/50 classification/regression, up to 16,384 rows / 2,000 features / 100 classes
hardware	one NVIDIA DGX Spark (GB10, 128GB unified memory), ~7h
data	100% synthetic (causal graphs / Gaussian processes / random trees); no real-world data, no OpenML data, and no other model's outputs were used in training
license	MIT (weights and code)

Intended use and limitations

Intended for tabular prediction on datasets from roughly 50 to tens of thousands of rows, up to 2,000 features, up to 100 classes. Not a language model: free-text columns should be dropped or encoded upstream. Regression on large, low-noise datasets is better served by gradient boosting; tabUM's edge is speed-to-first-prediction, small data, many classes, missing values, and built-in explanations.