Download README.md from Tabimpute/MatrixFormer: direct link, hf CLI and curl.
- Browser
- Download file 8.39 kB
-
https://huggingface.co/Tabimpute/MatrixFormer/resolve/main/README.md
- Command line
-
hf download hf://Tabimpute/MatrixFormer/README.md
-
curl -L -o README.md https://huggingface.co/Tabimpute/MatrixFormer/resolve/main/README.md
license: apache-2.0
tags:
- matrix-completion
- missing-data-imputation
- tabular
- pytorch
- safetensors
MatrixFormer
MatrixFormer is a single pretrained neural network for numerical matrix
completion. Supply a 2D NumPy array with np.nan for missing entries; the model
fills them while preserving observed entries exactly. No task-specific training
or fine-tuning is required. This release uses one checkpoint, not an ensemble.
The architecture has 77,998,496 parameters, eight row/column attention blocks, width 768, 16 attention heads, 12 context tokens per axis, and 5,000 output bins. Point predictions use the distributional mean, not random samples.
Installation and quick start
Use Python 3.10 or newer. Install the current standalone runtime from GitHub:
python -m pip install --upgrade "git+https://github.com/jacobf18/MatrixFormer.git"
Its runtime dependencies are PyTorch 2.4+, NumPy, and huggingface-hub. It does
not import tabimpute or require the research tabular repository. For a
platform-specific PyTorch installation, use the
PyTorch installation selector.
import numpy as np
from matrixformer import MatrixFormer
matrix = np.array([
[1.0, 2.0, np.nan],
[2.0, np.nan, 6.0],
[3.0, 6.0, 9.0],
])
model = MatrixFormer.from_pretrained(device="cpu", precision="float32")
completed = model.impute(matrix)
print(completed)
from_pretrained() downloads the final model and matching config from this Hub
repository using a pinned release and verifies their SHA-256 hashes. No private
loader or model selector is needed. The public weights require no Hugging Face
token; later calls reuse the local cache. Allow about 312 MB for the checkpoint,
in addition to PyTorch and its dependencies.
The result is a float64 NumPy array with the same shape and units as the input.
The input is left unchanged. Reuse the model object for multiple matrices.
For a BF16-capable NVIDIA GPU, use
MatrixFormer.from_pretrained(device="cuda"); use precision="float32" on
older CUDA GPUs. CPU and CUDA are supported, not MPS.
Command line and offline use
python -m matrixformer --device cpu --precision float32
python -m matrixformer --input matrix.npy --output completed.npy --device cpu --precision float32
The first command runs a small synthetic inference demo. The second completes
your NumPy array and saves the result; an existing output file is overwritten.
Use python -m matrixformer --help for options.
To prepare a portable offline checkpoint directory:
python -m matrixformer --download-only --local-dir ./checkpoints
python -m matrixformer --local-dir ./checkpoints --input matrix.npy --output completed.npy --device cpu --precision float32
The offline machine also needs the installed runtime and its dependencies.
local_dir must contain config.json and best_validation_checkpoint.pth
directly. Inference with local_dir does not make network requests.
Input preparation and memory
- Use a nonempty 2D numerical matrix with at least one observed value. Mark
missing values with
np.nan; zeros and sentinel numbers count as observations. - Remove IDs and free text and preprocess categorical variables separately. Strings, infinities, and entirely missing matrices are not supported.
- Inputs are normalized per column using observed values only and converted back to their original units. Keep the standard row/column orientation.
- The default context is up to 2,000 rows by 1,000 columns; larger matrices use
overlapping windows. Adjust
max_rowsandmax_colsinfrom_pretrained()to manage memory. Smaller windows can change predictions. - Attention, feed-forward operations, and decoding are streamed in microbatches without shortening attention within a context window. The hidden grid still occupies device memory; larger contexts can require substantial GPU memory.
Optional synthetic training
The standalone repository now includes a trainer for one randomly initialized MatrixFormer, using synthetic targets only and no teacher or ensemble:
git clone https://github.com/jacobf18/MatrixFormer.git
cd MatrixFormer
python -m pip install -e ".[training]"
python -m matrixformer.training --output runs/matrixformer --device cuda
The packaged prior combines scientific panels, spectral and latent-factor imputation tasks with structured missingness, sparse ratings, and score panels. Each optimizer update processes one matrix from each domain. Their loss weights are 0.35, 0.35, 0.20, and 0.10, respectively; these are loss weights, not sampling probabilities. The objective combines Huber, full-support density NLL, and CDF losses. The trainer includes independent synthetic validation, resumable atomic checkpoints, and differentiable streamed execution with activation recomputation.
The default standalone recipe runs 250,000 optimizer updates, generating 1,000,000 training matrices. These are the runnable trainer's defaults, not a guarantee of reproducing the published benchmark scores. For a small CPU wiring check rather than a full training run:
python -m matrixformer.training --output runs/smoke --smoke --device cpu --threads 2
Load your own training output with
MatrixFormer.from_checkpoint("runs/matrixformer/best_validation_checkpoint.pth", device="cpu", precision="float32").
This reads the adjacent config.json; it does not replace the checksum-pinned
published model used by from_pretrained(). See the
training guide
for the full recipe and memory controls. Training dependencies are optional for
pretrained inference.
Evaluation scope and limitations
| Benchmark | MatrixFormer NRMSE | Evaluated cases |
|---|---|---|
| MissBench | 1.256252 | 2,091 |
| UCI | 1.285373 | 3,058 |
Seeds, rates, and patterns are averaged within each dataset before datasets are weighted equally. The frozen campaign's primary columnwise NRMSE uses held-out-truth column standard deviation. Separately saved observed-column-SD NRMSE is 4.000035 for MissBench and 1.045013 for UCI; these normalization definitions are not interchangeable. Input normalization uses observed values only; held-out ground truth is used for evaluation scoring, never to normalize inference inputs.
The checkpoint was selected after inspecting earlier UCI/MissBench monitoring, so these benchmarks are not untouched selection holdouts. Benchmark masks, preprocessing, and context settings must match to reproduce results. The generic wrapper does not implement rating-specific bias centering, user-residual scaling, or clipping. RecSys and causal evaluations for these weights were pending at release; no improvement on those tasks is claimed here.
Entirely missing rows or columns have limited observational support. Extreme values, unusual missingness patterns, or structures outside the synthetic training distribution can reduce accuracy. Predictions are means, not calibrated uncertainty estimates. Evaluation datasets and runners are not included in the standalone package.
Model files
best_validation_checkpoint.pth: the exact selected checkpoint, with model weights and scalar provenance metadata; no optimizer state.model.safetensors: the identical 254 state-dictionary tensors in safetensors format. This is an alternative serialization, not a Transformers AutoModel.config.json: minimal inference architecture configuration.release.json: hashes, architecture, training milestone, and validation details.
Checkpoint SHA-256:
933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859.
Citation
@misc{saha2026matrixformerfoundationmodelmatrix,
title={MatrixFormer: A Foundation Model for Matrix Completion},
author={Dwaipayan Saha and Jacob Feitelberg and Kyuseong Choi and Raaz Dwivedi and Anish Agarwal},
year={2026},
eprint={2610.06751},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2610.06751}
}