--- license: apache-2.0 tags: - matrix-completion - missing-data-imputation - tabular - pytorch - safetensors --- # MatrixFormer MatrixFormer is a **single pretrained neural network** for numerical matrix completion. Supply a 2D NumPy array with `np.nan` for missing entries; the model fills them while preserving observed entries exactly. No task-specific training or fine-tuning is required. This release uses one checkpoint, not an ensemble. - [Code and installation guide](https://github.com/jacobf18/MatrixFormer) - [Project page](https://jacobf18.github.io/MatrixFormer/) - [Paper](https://arxiv.org/abs/2610.06751) - [Standalone synthetic training](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md) The architecture has 77,998,496 parameters, eight row/column attention blocks, width 768, 16 attention heads, 12 context tokens per axis, and 5,000 output bins. Point predictions use the **distributional mean**, not random samples. ## Installation and quick start Use Python 3.10 or newer. Install the current standalone runtime from GitHub: ```bash python -m pip install --upgrade "git+https://github.com/jacobf18/MatrixFormer.git" ``` Its runtime dependencies are PyTorch 2.4+, NumPy, and `huggingface-hub`. It does not import `tabimpute` or require the research `tabular` repository. For a platform-specific PyTorch installation, use the [PyTorch installation selector](https://pytorch.org/get-started/locally/). ```python import numpy as np from matrixformer import MatrixFormer matrix = np.array([ [1.0, 2.0, np.nan], [2.0, np.nan, 6.0], [3.0, 6.0, 9.0], ]) model = MatrixFormer.from_pretrained(device="cpu", precision="float32") completed = model.impute(matrix) print(completed) ``` `from_pretrained()` downloads the final model and matching config from this Hub repository using a pinned release and verifies their SHA-256 hashes. No private loader or model selector is needed. The public weights require no Hugging Face token; later calls reuse the local cache. Allow about 312 MB for the checkpoint, in addition to PyTorch and its dependencies. The result is a float64 NumPy array with the same shape and units as the input. The input is left unchanged. Reuse the model object for multiple matrices. For a BF16-capable NVIDIA GPU, use `MatrixFormer.from_pretrained(device="cuda")`; use `precision="float32"` on older CUDA GPUs. CPU and CUDA are supported, not MPS. ### Command line and offline use ```bash python -m matrixformer --device cpu --precision float32 python -m matrixformer --input matrix.npy --output completed.npy --device cpu --precision float32 ``` The first command runs a small synthetic inference demo. The second completes your NumPy array and saves the result; an existing output file is overwritten. Use `python -m matrixformer --help` for options. To prepare a portable offline checkpoint directory: ```bash python -m matrixformer --download-only --local-dir ./checkpoints python -m matrixformer --local-dir ./checkpoints --input matrix.npy --output completed.npy --device cpu --precision float32 ``` The offline machine also needs the installed runtime and its dependencies. `local_dir` must contain `config.json` and `best_validation_checkpoint.pth` directly. Inference with `local_dir` does not make network requests. ## Input preparation and memory - Use a nonempty 2D numerical matrix with at least one observed value. Mark missing values with `np.nan`; zeros and sentinel numbers count as observations. - Remove IDs and free text and preprocess categorical variables separately. Strings, infinities, and entirely missing matrices are not supported. - Inputs are normalized per column using **observed values only** and converted back to their original units. Keep the standard row/column orientation. - The default context is up to 2,000 rows by 1,000 columns; larger matrices use overlapping windows. Adjust `max_rows` and `max_cols` in `from_pretrained()` to manage memory. Smaller windows can change predictions. - Attention, feed-forward operations, and decoding are streamed in microbatches without shortening attention within a context window. The hidden grid still occupies device memory; larger contexts can require substantial GPU memory. ## Optional synthetic training The standalone repository now includes a trainer for one randomly initialized MatrixFormer, using synthetic targets only and no teacher or ensemble: ```bash git clone https://github.com/jacobf18/MatrixFormer.git cd MatrixFormer python -m pip install -e ".[training]" python -m matrixformer.training --output runs/matrixformer --device cuda ``` The packaged prior combines scientific panels, spectral and latent-factor imputation tasks with structured missingness, sparse ratings, and score panels. Each optimizer update processes one matrix from each domain. Their loss weights are 0.35, 0.35, 0.20, and 0.10, respectively; these are loss weights, not sampling probabilities. The objective combines Huber, full-support density NLL, and CDF losses. The trainer includes independent synthetic validation, resumable atomic checkpoints, and differentiable streamed execution with activation recomputation. The default standalone recipe runs 250,000 optimizer updates, generating 1,000,000 training matrices. These are the runnable trainer's defaults, not a guarantee of reproducing the published benchmark scores. For a small CPU wiring check rather than a full training run: ```bash python -m matrixformer.training --output runs/smoke --smoke --device cpu --threads 2 ``` Load your own training output with `MatrixFormer.from_checkpoint("runs/matrixformer/best_validation_checkpoint.pth", device="cpu", precision="float32")`. This reads the adjacent `config.json`; it does not replace the checksum-pinned published model used by `from_pretrained()`. See the [training guide](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md) for the full recipe and memory controls. Training dependencies are optional for pretrained inference. ## Evaluation scope and limitations | Benchmark | MatrixFormer NRMSE | Evaluated cases | | --- | ---: | ---: | | MissBench | 1.256252 | 2,091 | | UCI | 1.285373 | 3,058 | Seeds, rates, and patterns are averaged within each dataset before datasets are weighted equally. The frozen campaign's primary columnwise NRMSE uses **held-out-truth column standard deviation**. Separately saved observed-column-SD NRMSE is 4.000035 for MissBench and 1.045013 for UCI; these normalization definitions are not interchangeable. Input normalization uses observed values only; held-out ground truth is used for evaluation scoring, never to normalize inference inputs. The checkpoint was selected after inspecting earlier UCI/MissBench monitoring, so these benchmarks are not untouched selection holdouts. Benchmark masks, preprocessing, and context settings must match to reproduce results. The generic wrapper does not implement rating-specific bias centering, user-residual scaling, or clipping. RecSys and causal evaluations for these weights were pending at release; no improvement on those tasks is claimed here. Entirely missing rows or columns have limited observational support. Extreme values, unusual missingness patterns, or structures outside the synthetic training distribution can reduce accuracy. Predictions are means, not calibrated uncertainty estimates. Evaluation datasets and runners are not included in the standalone package. ## Model files - `best_validation_checkpoint.pth`: the exact selected checkpoint, with model weights and scalar provenance metadata; no optimizer state. - `model.safetensors`: the identical 254 state-dictionary tensors in safetensors format. This is an alternative serialization, not a Transformers AutoModel. - `config.json`: minimal inference architecture configuration. - `release.json`: hashes, architecture, training milestone, and validation details. Checkpoint SHA-256: `933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`. ## Citation ```bibtex @misc{saha2026matrixformerfoundationmodelmatrix, title={MatrixFormer: A Foundation Model for Matrix Completion}, author={Dwaipayan Saha and Jacob Feitelberg and Kyuseong Choi and Raaz Dwivedi and Anish Agarwal}, year={2026}, eprint={2610.06751}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2610.06751} } ```