---
license: mit
pipeline_tag: tabular-classification
library_name: tabum
tags:
  - tabular
  - tabular-classification
  - tabular-regression
  - in-context-learning
  - foundation-model
  - safetensors
---

# tabUM v1.1

A small tabular in-context-learning foundation model: **14.31M parameters**,
zero-shot classification and regression on any table, native missing-value
handling, up to **100 classes** and **2,000 features**. MIT licensed, built
clean-room (no third-party model code).

**Code, notebook, and full results:** [github.com/heldernoid/tabum](https://github.com/heldernoid/tabum)

## What it does

tabUM never trains on your data. `fit(X, y)` stores your rows; `predict(X_test)`
runs a single forward pass in which every test row attends to every training row
(in-context learning). Pretraining happened once, on millions of synthetic tasks
sampled from causal graphs, Gaussian processes, and random tree ensembles.

- prediction in seconds, no tuning, no preprocessing pipelines
- `NaN` is a first-class value: missingness is signal, never imputed
- calibrated probabilities; full regression distributions (any quantile) in one pass
- up to 100 classes (most tabular foundation models cap at 10)
- optional `finetune()` (~15s per-dataset adaptation) and `explain()`
  (feature importances + the exact training rows the prediction attended to)

## Usage

```bash
pip install git+https://github.com/heldernoid/tabum
```

```python
from huggingface_hub import snapshot_download
from tabum.model import TabUM
from tabum.inference import TabUMClassifier, TabUMRegressor

model = TabUM.from_pretrained(snapshot_download("helmo/tabum-v1.1"), device="cuda")

clf = TabUMClassifier(model=model, n_ensemble=8).fit(X_train, y_train)
proba = clf.predict_proba(X_test)

reg = TabUMRegressor(model=model, n_ensemble=8).fit(X_train, y_train)
p90 = reg.predict_quantile(X_test, 0.9)
```

## Evaluation

TabArena-v0.1, all 51 datasets, identical protocol for every model (at most
2,000 train / 1,000 test rows, single split, seed 0; competitors run as
black-box pip packages on the same splits). Per-dataset numbers are in the
GitHub repository.

| model | params | cls accuracy (38) | reg R² (13) |
|---|---|---|---|
| logistic / linear regression (fitted) | | 0.8417 | 0.5597 |
| tabUM zero-shot, 1 pass | 14M | 0.8523 | 0.6284 |
| tabUM zero-shot, `n_ensemble=8` | 14M | 0.8581 | 0.6519 |
| **tabUM `finetune()` + `n_ensemble=8`** | 14M | **0.8635** | **0.6907** |
| HistGradientBoosting (fitted, default) | | 0.8669 | 0.7091 |
| TabPFN v2 (zero-shot) | ~11M | 0.8768 | 0.7471 |
| TabICL (zero-shot) | ~500M | 0.8801 | n/a |

Strengths and honest limitations:

- beats fitted linear baselines everywhere, zero-shot
- finetuned, it ties default gradient boosting on classification (20/38 wins)
  and trails it by 1.8 R² points on regression
- 1.3 to 1.7 accuracy points behind TabPFN v2 and TabICL (the state of the art
  in this class), at 14M parameters and MIT license
- many-class is a differentiator: letter (26 classes) scores 0.79 vs 0.74 for
  fitted logistic regression; TabPFN-class models cap at 10 classes
- strongest regime: small noisy tables (under ~1,000 rows), where the learned
  prior beats boosting; on Titanic with 12% missing cells fed raw it beats an
  imputed+scaled logistic regression pipeline
- context lengths validated to 64k rows (accuracy keeps improving with more
  context)

## Model details

| | |
|---|---|
| architecture | 3-stage tabular ICL transformer: Fourier cell embeddings with inducing-point column attention, per-row CLS aggregation, 10-block row transformer (train rows bidirectional, test rows attend train-only), retrieval-attention classification head, bar-distribution regression head |
| parameters | 14,306,051 |
| training | 20,000 steps of continued pretraining from tabUM v1 (50,000 steps), on synthetic tasks only, streamed on the fly (no task seen twice), 50/50 classification/regression, up to 16,384 rows / 2,000 features / 100 classes |
| hardware | one NVIDIA DGX Spark (GB10, 128GB unified memory), ~7h |
| data | 100% synthetic (causal graphs / Gaussian processes / random trees); no real-world data, no OpenML data, and no other model's outputs were used in training |
| license | MIT (weights and code) |

## Intended use and limitations

Intended for tabular prediction on datasets from roughly 50 to tens of
thousands of rows, up to 2,000 features, up to 100 classes. Not a language
model: free-text columns should be dropped or encoded upstream. Regression on
large, low-noise datasets is better served by gradient boosting; tabUM's edge
is speed-to-first-prediction, small data, many classes, missing values, and
built-in explanations.