|
Download README.md from Tabimpute/MatrixFormer: direct link, hf CLI and curl.
- Browser
- Download file 8.39 kB
-
https://huggingface.co/Tabimpute/MatrixFormer/resolve/main/README.md
- Command line
-
hf download hf://Tabimpute/MatrixFormer/README.md
-
curl -L -o README.md https://huggingface.co/Tabimpute/MatrixFormer/resolve/main/README.md
8.39 kB
| license: apache-2.0 | |
| tags: | |
| - matrix-completion | |
| - missing-data-imputation | |
| - tabular | |
| - pytorch | |
| - safetensors | |
| # MatrixFormer | |
| MatrixFormer is a **single pretrained neural network** for numerical matrix | |
| completion. Supply a 2D NumPy array with `np.nan` for missing entries; the model | |
| fills them while preserving observed entries exactly. No task-specific training | |
| or fine-tuning is required. This release uses one checkpoint, not an ensemble. | |
| - [Code and installation guide](https://github.com/jacobf18/MatrixFormer) | |
| - [Project page](https://jacobf18.github.io/MatrixFormer/) | |
| - [Paper](https://arxiv.org/abs/2610.06751) | |
| - [Standalone synthetic training](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md) | |
| The architecture has 77,998,496 parameters, eight row/column attention blocks, | |
| width 768, 16 attention heads, 12 context tokens per axis, and 5,000 output bins. | |
| Point predictions use the **distributional mean**, not random samples. | |
| ## Installation and quick start | |
| Use Python 3.10 or newer. Install the current standalone runtime from GitHub: | |
| ```bash | |
| python -m pip install --upgrade "git+https://github.com/jacobf18/MatrixFormer.git" | |
| ``` | |
| Its runtime dependencies are PyTorch 2.4+, NumPy, and `huggingface-hub`. It does | |
| not import `tabimpute` or require the research `tabular` repository. For a | |
| platform-specific PyTorch installation, use the | |
| [PyTorch installation selector](https://pytorch.org/get-started/locally/). | |
| ```python | |
| import numpy as np | |
| from matrixformer import MatrixFormer | |
| matrix = np.array([ | |
| [1.0, 2.0, np.nan], | |
| [2.0, np.nan, 6.0], | |
| [3.0, 6.0, 9.0], | |
| ]) | |
| model = MatrixFormer.from_pretrained(device="cpu", precision="float32") | |
| completed = model.impute(matrix) | |
| print(completed) | |
| ``` | |
| `from_pretrained()` downloads the final model and matching config from this Hub | |
| repository using a pinned release and verifies their SHA-256 hashes. No private | |
| loader or model selector is needed. The public weights require no Hugging Face | |
| token; later calls reuse the local cache. Allow about 312 MB for the checkpoint, | |
| in addition to PyTorch and its dependencies. | |
| The result is a float64 NumPy array with the same shape and units as the input. | |
| The input is left unchanged. Reuse the model object for multiple matrices. | |
| For a BF16-capable NVIDIA GPU, use | |
| `MatrixFormer.from_pretrained(device="cuda")`; use `precision="float32"` on | |
| older CUDA GPUs. CPU and CUDA are supported, not MPS. | |
| ### Command line and offline use | |
| ```bash | |
| python -m matrixformer --device cpu --precision float32 | |
| python -m matrixformer --input matrix.npy --output completed.npy --device cpu --precision float32 | |
| ``` | |
| The first command runs a small synthetic inference demo. The second completes | |
| your NumPy array and saves the result; an existing output file is overwritten. | |
| Use `python -m matrixformer --help` for options. | |
| To prepare a portable offline checkpoint directory: | |
| ```bash | |
| python -m matrixformer --download-only --local-dir ./checkpoints | |
| python -m matrixformer --local-dir ./checkpoints --input matrix.npy --output completed.npy --device cpu --precision float32 | |
| ``` | |
| The offline machine also needs the installed runtime and its dependencies. | |
| `local_dir` must contain `config.json` and `best_validation_checkpoint.pth` | |
| directly. Inference with `local_dir` does not make network requests. | |
| ## Input preparation and memory | |
| - Use a nonempty 2D numerical matrix with at least one observed value. Mark | |
| missing values with `np.nan`; zeros and sentinel numbers count as observations. | |
| - Remove IDs and free text and preprocess categorical variables separately. | |
| Strings, infinities, and entirely missing matrices are not supported. | |
| - Inputs are normalized per column using **observed values only** and converted | |
| back to their original units. Keep the standard row/column orientation. | |
| - The default context is up to 2,000 rows by 1,000 columns; larger matrices use | |
| overlapping windows. Adjust `max_rows` and `max_cols` in `from_pretrained()` | |
| to manage memory. Smaller windows can change predictions. | |
| - Attention, feed-forward operations, and decoding are streamed in microbatches | |
| without shortening attention within a context window. The hidden grid still | |
| occupies device memory; larger contexts can require substantial GPU memory. | |
| ## Optional synthetic training | |
| The standalone repository now includes a trainer for one randomly initialized | |
| MatrixFormer, using synthetic targets only and no teacher or ensemble: | |
| ```bash | |
| git clone https://github.com/jacobf18/MatrixFormer.git | |
| cd MatrixFormer | |
| python -m pip install -e ".[training]" | |
| python -m matrixformer.training --output runs/matrixformer --device cuda | |
| ``` | |
| The packaged prior combines scientific panels, spectral and latent-factor | |
| imputation tasks with structured missingness, sparse ratings, and score panels. | |
| Each optimizer update processes one matrix from each domain. Their loss weights | |
| are 0.35, 0.35, 0.20, and 0.10, respectively; these are loss weights, not sampling | |
| probabilities. The objective combines Huber, full-support density NLL, and CDF | |
| losses. The trainer includes independent synthetic validation, resumable atomic | |
| checkpoints, and differentiable streamed execution with activation recomputation. | |
| The default standalone recipe runs 250,000 optimizer updates, generating | |
| 1,000,000 training matrices. These are the runnable trainer's defaults, not a | |
| guarantee of reproducing the published benchmark scores. For a small CPU wiring | |
| check rather than a full training run: | |
| ```bash | |
| python -m matrixformer.training --output runs/smoke --smoke --device cpu --threads 2 | |
| ``` | |
| Load your own training output with | |
| `MatrixFormer.from_checkpoint("runs/matrixformer/best_validation_checkpoint.pth", device="cpu", precision="float32")`. | |
| This reads the adjacent `config.json`; it does not replace the checksum-pinned | |
| published model used by `from_pretrained()`. See the | |
| [training guide](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md) | |
| for the full recipe and memory controls. Training dependencies are optional for | |
| pretrained inference. | |
| ## Evaluation scope and limitations | |
| | Benchmark | MatrixFormer NRMSE | Evaluated cases | | |
| | --- | ---: | ---: | | |
| | MissBench | 1.256252 | 2,091 | | |
| | UCI | 1.285373 | 3,058 | | |
| Seeds, rates, and patterns are averaged within each dataset before datasets are | |
| weighted equally. The frozen campaign's primary columnwise NRMSE uses | |
| **held-out-truth column standard deviation**. Separately saved | |
| observed-column-SD NRMSE is 4.000035 for MissBench and 1.045013 for UCI; these | |
| normalization definitions are not interchangeable. Input normalization uses | |
| observed values only; held-out ground truth is used for evaluation scoring, | |
| never to normalize inference inputs. | |
| The checkpoint was selected after inspecting earlier UCI/MissBench monitoring, | |
| so these benchmarks are not untouched selection holdouts. Benchmark masks, | |
| preprocessing, and context settings must match to reproduce results. The generic | |
| wrapper does not implement rating-specific bias centering, user-residual scaling, | |
| or clipping. RecSys and causal evaluations for these weights were pending at | |
| release; no improvement on those tasks is claimed here. | |
| Entirely missing rows or columns have limited observational support. Extreme | |
| values, unusual missingness patterns, or structures outside the synthetic | |
| training distribution can reduce accuracy. Predictions are means, not calibrated | |
| uncertainty estimates. Evaluation datasets and runners are not included in the | |
| standalone package. | |
| ## Model files | |
| - `best_validation_checkpoint.pth`: the exact selected checkpoint, with model | |
| weights and scalar provenance metadata; no optimizer state. | |
| - `model.safetensors`: the identical 254 state-dictionary tensors in safetensors | |
| format. This is an alternative serialization, not a Transformers AutoModel. | |
| - `config.json`: minimal inference architecture configuration. | |
| - `release.json`: hashes, architecture, training milestone, and validation details. | |
| Checkpoint SHA-256: | |
| `933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`. | |
| ## Citation | |
| ```bibtex | |
| @misc{saha2026matrixformerfoundationmodelmatrix, | |
| title={MatrixFormer: A Foundation Model for Matrix Completion}, | |
| author={Dwaipayan Saha and Jacob Feitelberg and Kyuseong Choi and Raaz Dwivedi and Anish Agarwal}, | |
| year={2026}, | |
| eprint={2610.06751}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.LG}, | |
| url={https://arxiv.org/abs/2610.06751} | |
| } | |
| ``` | |