MatrixFormer / README.md
Tabimpute's picture
Update model card for the single-model API and standalone synthetic training
22f1e4b verified
|
Raw History Blame Contribute Delete
8.39 kB
---
license: apache-2.0
tags:
- matrix-completion
- missing-data-imputation
- tabular
- pytorch
- safetensors
---
# MatrixFormer
MatrixFormer is a **single pretrained neural network** for numerical matrix
completion. Supply a 2D NumPy array with `np.nan` for missing entries; the model
fills them while preserving observed entries exactly. No task-specific training
or fine-tuning is required. This release uses one checkpoint, not an ensemble.
- [Code and installation guide](https://github.com/jacobf18/MatrixFormer)
- [Project page](https://jacobf18.github.io/MatrixFormer/)
- [Paper](https://arxiv.org/abs/2610.06751)
- [Standalone synthetic training](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md)
The architecture has 77,998,496 parameters, eight row/column attention blocks,
width 768, 16 attention heads, 12 context tokens per axis, and 5,000 output bins.
Point predictions use the **distributional mean**, not random samples.
## Installation and quick start
Use Python 3.10 or newer. Install the current standalone runtime from GitHub:
```bash
python -m pip install --upgrade "git+https://github.com/jacobf18/MatrixFormer.git"
```
Its runtime dependencies are PyTorch 2.4+, NumPy, and `huggingface-hub`. It does
not import `tabimpute` or require the research `tabular` repository. For a
platform-specific PyTorch installation, use the
[PyTorch installation selector](https://pytorch.org/get-started/locally/).
```python
import numpy as np
from matrixformer import MatrixFormer
matrix = np.array([
[1.0, 2.0, np.nan],
[2.0, np.nan, 6.0],
[3.0, 6.0, 9.0],
])
model = MatrixFormer.from_pretrained(device="cpu", precision="float32")
completed = model.impute(matrix)
print(completed)
```
`from_pretrained()` downloads the final model and matching config from this Hub
repository using a pinned release and verifies their SHA-256 hashes. No private
loader or model selector is needed. The public weights require no Hugging Face
token; later calls reuse the local cache. Allow about 312 MB for the checkpoint,
in addition to PyTorch and its dependencies.
The result is a float64 NumPy array with the same shape and units as the input.
The input is left unchanged. Reuse the model object for multiple matrices.
For a BF16-capable NVIDIA GPU, use
`MatrixFormer.from_pretrained(device="cuda")`; use `precision="float32"` on
older CUDA GPUs. CPU and CUDA are supported, not MPS.
### Command line and offline use
```bash
python -m matrixformer --device cpu --precision float32
python -m matrixformer --input matrix.npy --output completed.npy --device cpu --precision float32
```
The first command runs a small synthetic inference demo. The second completes
your NumPy array and saves the result; an existing output file is overwritten.
Use `python -m matrixformer --help` for options.
To prepare a portable offline checkpoint directory:
```bash
python -m matrixformer --download-only --local-dir ./checkpoints
python -m matrixformer --local-dir ./checkpoints --input matrix.npy --output completed.npy --device cpu --precision float32
```
The offline machine also needs the installed runtime and its dependencies.
`local_dir` must contain `config.json` and `best_validation_checkpoint.pth`
directly. Inference with `local_dir` does not make network requests.
## Input preparation and memory
- Use a nonempty 2D numerical matrix with at least one observed value. Mark
missing values with `np.nan`; zeros and sentinel numbers count as observations.
- Remove IDs and free text and preprocess categorical variables separately.
Strings, infinities, and entirely missing matrices are not supported.
- Inputs are normalized per column using **observed values only** and converted
back to their original units. Keep the standard row/column orientation.
- The default context is up to 2,000 rows by 1,000 columns; larger matrices use
overlapping windows. Adjust `max_rows` and `max_cols` in `from_pretrained()`
to manage memory. Smaller windows can change predictions.
- Attention, feed-forward operations, and decoding are streamed in microbatches
without shortening attention within a context window. The hidden grid still
occupies device memory; larger contexts can require substantial GPU memory.
## Optional synthetic training
The standalone repository now includes a trainer for one randomly initialized
MatrixFormer, using synthetic targets only and no teacher or ensemble:
```bash
git clone https://github.com/jacobf18/MatrixFormer.git
cd MatrixFormer
python -m pip install -e ".[training]"
python -m matrixformer.training --output runs/matrixformer --device cuda
```
The packaged prior combines scientific panels, spectral and latent-factor
imputation tasks with structured missingness, sparse ratings, and score panels.
Each optimizer update processes one matrix from each domain. Their loss weights
are 0.35, 0.35, 0.20, and 0.10, respectively; these are loss weights, not sampling
probabilities. The objective combines Huber, full-support density NLL, and CDF
losses. The trainer includes independent synthetic validation, resumable atomic
checkpoints, and differentiable streamed execution with activation recomputation.
The default standalone recipe runs 250,000 optimizer updates, generating
1,000,000 training matrices. These are the runnable trainer's defaults, not a
guarantee of reproducing the published benchmark scores. For a small CPU wiring
check rather than a full training run:
```bash
python -m matrixformer.training --output runs/smoke --smoke --device cpu --threads 2
```
Load your own training output with
`MatrixFormer.from_checkpoint("runs/matrixformer/best_validation_checkpoint.pth", device="cpu", precision="float32")`.
This reads the adjacent `config.json`; it does not replace the checksum-pinned
published model used by `from_pretrained()`. See the
[training guide](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md)
for the full recipe and memory controls. Training dependencies are optional for
pretrained inference.
## Evaluation scope and limitations
| Benchmark | MatrixFormer NRMSE | Evaluated cases |
| --- | ---: | ---: |
| MissBench | 1.256252 | 2,091 |
| UCI | 1.285373 | 3,058 |
Seeds, rates, and patterns are averaged within each dataset before datasets are
weighted equally. The frozen campaign's primary columnwise NRMSE uses
**held-out-truth column standard deviation**. Separately saved
observed-column-SD NRMSE is 4.000035 for MissBench and 1.045013 for UCI; these
normalization definitions are not interchangeable. Input normalization uses
observed values only; held-out ground truth is used for evaluation scoring,
never to normalize inference inputs.
The checkpoint was selected after inspecting earlier UCI/MissBench monitoring,
so these benchmarks are not untouched selection holdouts. Benchmark masks,
preprocessing, and context settings must match to reproduce results. The generic
wrapper does not implement rating-specific bias centering, user-residual scaling,
or clipping. RecSys and causal evaluations for these weights were pending at
release; no improvement on those tasks is claimed here.
Entirely missing rows or columns have limited observational support. Extreme
values, unusual missingness patterns, or structures outside the synthetic
training distribution can reduce accuracy. Predictions are means, not calibrated
uncertainty estimates. Evaluation datasets and runners are not included in the
standalone package.
## Model files
- `best_validation_checkpoint.pth`: the exact selected checkpoint, with model
weights and scalar provenance metadata; no optimizer state.
- `model.safetensors`: the identical 254 state-dictionary tensors in safetensors
format. This is an alternative serialization, not a Transformers AutoModel.
- `config.json`: minimal inference architecture configuration.
- `release.json`: hashes, architecture, training milestone, and validation details.
Checkpoint SHA-256:
`933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`.
## Citation
```bibtex
@misc{saha2026matrixformerfoundationmodelmatrix,
title={MatrixFormer: A Foundation Model for Matrix Completion},
author={Dwaipayan Saha and Jacob Feitelberg and Kyuseong Choi and Raaz Dwivedi and Anish Agarwal},
year={2026},
eprint={2610.06751},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2610.06751}
}
```