README.md

---
license: apache-2.0
tags:
  - fela
  - felatab
  - tabular
  - in-context-learning
  - prior-fitted-network
  - foundation-model
  - delta-rule
  - cpu
  - on-device
library_name: transformers
pipeline_tag: tabular-classification
model-index:
- name: fela-tab
  results:
  - task:
      type: tabular-classification
      name: Tabular classification
    dataset:
      type: openml
      name: OpenML (8 datasets)
    metrics:
    - type: accuracy
      value: 0.819
      name: mean accuracy (zero shot)
---

> **N.B.** Check us out on [Github](https://github.com/Lowdown-Labs/pg_fela) if you want a
> prepackaged way to deploy this on Postgres. Google Sheets Extension coming soon!!!

# DISCLAIMER

This model is a research preview. It is offered for advancing public science and for evaluation.
It is not a substitute for a domain expert, and it is not a certified decision system.

# FelaTab: point at any table in context tabular model

FelaTab reads a small table you already have and predicts the missing cells. You give it some
example rows with their answers (the support rows) and one or more rows you want filled in (the
query rows); it learns the pattern from your examples in a single pass and returns the answer with
a calibrated confidence range. There is no per table training, no fitting, and no setup: you point
it at a table and it predicts. It runs on a plain CPU with no GPU.

It is a prior fitted network (a "foundation model for tables"), trained on millions of synthetic
tables so that it generalizes zero shot to real tables it has never seen.

# What goes in, what comes out

- Input: one table, split into SUPPORT rows (features + a known label) and QUERY rows (features
  only). Features are plain numbers; the model standardizes them for you. Up to 100 feature
  columns and, for classification, up to 10 classes.
- Output:
  - Classification: a probability for each class on every query row (they sum to one). Pick the
    top class as the prediction and read the probability as the confidence.
  - Regression: a predicted value plus a standard deviation on every query row, an honest error
    bar in the original units of your label, not just a point estimate.
- In plain terms: "these rows look labelled like this; fill the blank ones and tell me how sure
  you are."

# Why we built it this way

The "sequence" the model attends over is the SET of rows of your table. Support rows carry their
label and write into a fixed size working state; query rows are read only (they never write the
state and never see each other, so there is no test time leakage). The mixer is a delta rule
linear attention (an overwrite update that fixes the recall weakness of plain linear attention)
plus a variable chunk landmark attention over pooled support rows. Both are linear in the number
of rows with a fixed working state, so the model scales past the quadratic roughly 10k row cap of
full attention tabular models, and its memory stays flat as the support set grows. That is what
lets it run on a low power CPU and index large tables without a GPU.

## The two tiers (one model, pick your size)

FelaTab ships two tiers from ONE MatFormer nested training run. The small model is a strict
top left prefix slice of the big model's weights (nesting verified bit exact at full
granularity), so you get both from a single set of weights and can pick your size/accuracy
tradeoff.

| Tier  | dim | layers | heads | params | fp32 file | int8 file |
|-------|-----|--------|-------|--------|-----------|-----------|
| big   | 1024 | 28 | 16 | 411.9M | `model_big.safetensors` (1.65 GB) | `model_big_int8.safetensors` (416 MB) |
| small | 512  | 14 | 8  | 51.6M  | `model_small.safetensors` (206 MB) | `model_small_int8.safetensors` (52 MB) |

int8 is close to lossless here (see below), so the int8 files are the recommended deploy
artifact. The in browser demo runs the small int8 tier.

# Performance and footprint

Measured on CPU. The working memory is flat as the support set grows: on the OpenML `adult`
dataset the big model's resident set stays at about **3.79 GB from 500 support rows all the way to
10,000** support rows (the model carries a fixed size state, not a growing key value cache), while
a full attention model would grow quadratically. Row throughput is linear in the support size.

# Accuracy (held out OpenML)

Protocol: zero shot, TabPFN style. The train split of a real OpenML dataset is fed as support
rows, the test split as query rows, in one in context forward, no per dataset training. The
reported battery is 8 classification and 5 regression datasets. Field bars are untuned
scikit-learn and a TUNED LightGBM; the "full ensemble" is FelaTab combined with gradient boosted
trees and ridge. The numbers below are measured on the shipped weights.

## Classification (8 datasets, mean accuracy)

| System | Accuracy |
|---|---|
| FelaTab big, single pass | 0.819 |
| FelaTab small, single pass | 0.810 |
| tuned LightGBM | 0.827 |
| untuned scikit-learn, best | 0.834 |
| full ensemble (FelaTab + GBT + ridge, stacked) | 0.840 |

- With no tuning, FelaTab alone matches or beats a **tuned** LightGBM on **5 of the 8** datasets.
- FelaTab alone reaches about **97%** of the full ensemble's accuracy (0.819 vs 0.840).
- Calibration is good: expected calibration error about **0.051** for FelaTab, and a conformal
  wrapper hits its target coverage (empirical coverage about **0.91** for FelaTab and about **0.92**
  for the ensemble at a 0.90 guarantee).
- Inference time bagging (averaging the frozen model over permuted feature orders and a reshuffled
  support set) did not improve the big tier on this battery (it measured 0.811, below the single
  pass), so the single pass is the number to use.

## Regression (5 datasets)

On regression accuracy FelaTab alone beats neither untuned scikit-learn nor a tuned LightGBM on any
of the 5 datasets (**0 of 5**) but it's not off by far; its mean R2 is 0.813 versus 0.880 for a tuned LightGBM. 

This is a training budget limit of the current model, not a bug. What
FelaTab ships for regression is, however, calibrated uncertainty: a Gaussian mean and standard
deviation per row, plus a conformal wrapper that reaches its coverage target (empirical coverage
about 0.91 at a 0.90 guarantee). If you need maximum regression accuracy, use a gradient boosted
tree; use FelaTab's regression head for a fast zero shot estimate with a trustworthy error bar.

## int8 is close to lossless

Dynamic int8 on the bulk linear layers barely moves accuracy: classification 0.819 fp32 vs 0.816
int8 for the big tier and 0.810 vs 0.809 for the small tier, regression within 0.002 R2 either
way. The int8 files are 4x smaller and are the recommended deploy format.


# How to run it

See `quickstart/` for a runnable example on public sklearn datasets. The short version:

```python
from modeling import load_model, predict

model = load_model("/path/to/repo_dir", tier="small")   # or "big"; or a HF repo id
# Xtr, ytr = your labelled support rows;  Xte = the rows to predict
probs = predict(model, Xtr, ytr, Xte, task="classification", n_classes=3)   # [n_query, 3]
mean, std = predict(model, Xtr, ytr, Xte, task="regression")                # error bars
```

The loader ships `config_big.json` / `config_small.json` (a top level `config.json` mirrors the
big tier) and a self contained `modeling.py` with `load_model` / `from_pretrained`. It
auto detects the int8 bundles and dequantizes them, so you always get a plain fp32 model back. For
an interactive playground, see the Hugging Face Space in `space/`.

## Serving artifacts

- `model_<tier>.safetensors` (fp32) and `model_<tier>_int8.safetensors` (int8), plus the matching
  `config_<tier>.json`.
- `verify.py` runs two fixed bundled tasks (one classification, one regression), checks the output
  shapes, that the regression standard deviations are positive, and that the small tier
  predictions match a stored reference within the shipped int8 tolerance.
- `modeling.py` also exposes `predict_bagged` (the inference bagging path) and the full
  architecture for standard tooling.

FelaTab is embedded natively in the CPU FELA server and in a PostgreSQL extension (`fela_classify`
in database), and runs client side in the browser via WebAssembly (the small int8 tier). Those
paths reproduce the Python predictions to within the int8 tolerance.

## Training data

FelaTab is a prior fitted network: trained on synthetic tables from a structural causal model prior
(about 85% of the stream) with a small mix in of teacher baked real public OpenML tables (about
15%), and evaluated zero shot on held out real OpenML datasets that are excluded from training by
dataset id and name. No customer data and no benchmarked test set is in the training stream. The
synthetic prior is in `prior.py` and the training objective is in `train.py`, which reproduces the
method (`python train.py --smoke`).

# Intended use, limitations, and safety

What it is for: fast, zero shot predictions and autofill on small to medium tables where you do
not want to set up and tune a model: spreadsheet autofill, in database prediction, quick
what if estimates, and calibrated triage where the confidence matters as much as the answer. It is
a good first pass and a strong no tuning baseline.

What it is not: it is not a replacement for a tuned gradient boosted tree when you want maximum
accuracy, especially on regression, where it is behind trees and ships error bars instead of
headline accuracy. It is capped at 100 features and 10 classes. It is trained and validated on
standard public tabular benchmarks; generalization to a very different distribution should be
checked on your own held out data before you rely on it. Do not use it as the sole basis for a
consequential decision.

# How to cite

```bibtex
@misc{lowdownlabs_felatab,
  title  = {FelaTab: an in context tabular foundation model on the FELA linear attention base},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}
```

# Model family

This is part of the FELA family from Lowdown Labs: one linear attention architecture across many
modalities, all CPU native and subquadratic. Sibling repos share no weights, so none carries a
`base_model` link.

# License
Apache-2.0. The model weights and the code are both under Apache-2.0 (see LICENSE): no separate model license and no non commercial restriction. Trained on synthetic data; the OpenML datasets are used only for evaluation, not training. We distribute weights only.