File size: 8,392 Bytes
f0beeeb
 
4991727
 
 
 
 
 
f0beeeb
4991727
 
 
 
22f1e4b
 
 
4991727
22f1e4b
 
 
 
4991727
22f1e4b
 
 
4991727
22f1e4b
4991727
22f1e4b
4991727
 
22f1e4b
4991727
 
22f1e4b
 
 
 
 
4991727
 
22f1e4b
 
 
 
 
 
 
 
 
4991727
22f1e4b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4991727
 
22f1e4b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4991727
 
 
22f1e4b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4991727
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
---
license: apache-2.0
tags:
- matrix-completion
- missing-data-imputation
- tabular
- pytorch
- safetensors
---

# MatrixFormer

MatrixFormer is a **single pretrained neural network** for numerical matrix
completion. Supply a 2D NumPy array with `np.nan` for missing entries; the model
fills them while preserving observed entries exactly. No task-specific training
or fine-tuning is required. This release uses one checkpoint, not an ensemble.

- [Code and installation guide](https://github.com/jacobf18/MatrixFormer)
- [Project page](https://jacobf18.github.io/MatrixFormer/)
- [Paper](https://arxiv.org/abs/2610.06751)
- [Standalone synthetic training](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md)

The architecture has 77,998,496 parameters, eight row/column attention blocks,
width 768, 16 attention heads, 12 context tokens per axis, and 5,000 output bins.
Point predictions use the **distributional mean**, not random samples.

## Installation and quick start

Use Python 3.10 or newer. Install the current standalone runtime from GitHub:

```bash
python -m pip install --upgrade "git+https://github.com/jacobf18/MatrixFormer.git"
```

Its runtime dependencies are PyTorch 2.4+, NumPy, and `huggingface-hub`. It does
not import `tabimpute` or require the research `tabular` repository. For a
platform-specific PyTorch installation, use the
[PyTorch installation selector](https://pytorch.org/get-started/locally/).

```python
import numpy as np
from matrixformer import MatrixFormer

matrix = np.array([
    [1.0, 2.0, np.nan],
    [2.0, np.nan, 6.0],
    [3.0, 6.0, 9.0],
])

model = MatrixFormer.from_pretrained(device="cpu", precision="float32")
completed = model.impute(matrix)
print(completed)
```

`from_pretrained()` downloads the final model and matching config from this Hub
repository using a pinned release and verifies their SHA-256 hashes. No private
loader or model selector is needed. The public weights require no Hugging Face
token; later calls reuse the local cache. Allow about 312 MB for the checkpoint,
in addition to PyTorch and its dependencies.

The result is a float64 NumPy array with the same shape and units as the input.
The input is left unchanged. Reuse the model object for multiple matrices.
For a BF16-capable NVIDIA GPU, use
`MatrixFormer.from_pretrained(device="cuda")`; use `precision="float32"` on
older CUDA GPUs. CPU and CUDA are supported, not MPS.

### Command line and offline use

```bash
python -m matrixformer --device cpu --precision float32
python -m matrixformer --input matrix.npy --output completed.npy --device cpu --precision float32
```

The first command runs a small synthetic inference demo. The second completes
your NumPy array and saves the result; an existing output file is overwritten.
Use `python -m matrixformer --help` for options.

To prepare a portable offline checkpoint directory:

```bash
python -m matrixformer --download-only --local-dir ./checkpoints
python -m matrixformer --local-dir ./checkpoints --input matrix.npy --output completed.npy --device cpu --precision float32
```

The offline machine also needs the installed runtime and its dependencies.
`local_dir` must contain `config.json` and `best_validation_checkpoint.pth`
directly. Inference with `local_dir` does not make network requests.

## Input preparation and memory

- Use a nonempty 2D numerical matrix with at least one observed value. Mark
  missing values with `np.nan`; zeros and sentinel numbers count as observations.
- Remove IDs and free text and preprocess categorical variables separately.
  Strings, infinities, and entirely missing matrices are not supported.
- Inputs are normalized per column using **observed values only** and converted
  back to their original units. Keep the standard row/column orientation.
- The default context is up to 2,000 rows by 1,000 columns; larger matrices use
  overlapping windows. Adjust `max_rows` and `max_cols` in `from_pretrained()`
  to manage memory. Smaller windows can change predictions.
- Attention, feed-forward operations, and decoding are streamed in microbatches
  without shortening attention within a context window. The hidden grid still
  occupies device memory; larger contexts can require substantial GPU memory.

## Optional synthetic training

The standalone repository now includes a trainer for one randomly initialized
MatrixFormer, using synthetic targets only and no teacher or ensemble:

```bash
git clone https://github.com/jacobf18/MatrixFormer.git
cd MatrixFormer
python -m pip install -e ".[training]"
python -m matrixformer.training --output runs/matrixformer --device cuda
```

The packaged prior combines scientific panels, spectral and latent-factor
imputation tasks with structured missingness, sparse ratings, and score panels.
Each optimizer update processes one matrix from each domain. Their loss weights
are 0.35, 0.35, 0.20, and 0.10, respectively; these are loss weights, not sampling
probabilities. The objective combines Huber, full-support density NLL, and CDF
losses. The trainer includes independent synthetic validation, resumable atomic
checkpoints, and differentiable streamed execution with activation recomputation.

The default standalone recipe runs 250,000 optimizer updates, generating
1,000,000 training matrices. These are the runnable trainer's defaults, not a
guarantee of reproducing the published benchmark scores. For a small CPU wiring
check rather than a full training run:

```bash
python -m matrixformer.training --output runs/smoke --smoke --device cpu --threads 2
```

Load your own training output with
`MatrixFormer.from_checkpoint("runs/matrixformer/best_validation_checkpoint.pth", device="cpu", precision="float32")`.
This reads the adjacent `config.json`; it does not replace the checksum-pinned
published model used by `from_pretrained()`. See the
[training guide](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md)
for the full recipe and memory controls. Training dependencies are optional for
pretrained inference.

## Evaluation scope and limitations

| Benchmark | MatrixFormer NRMSE | Evaluated cases |
| --- | ---: | ---: |
| MissBench | 1.256252 | 2,091 |
| UCI | 1.285373 | 3,058 |

Seeds, rates, and patterns are averaged within each dataset before datasets are
weighted equally. The frozen campaign's primary columnwise NRMSE uses
**held-out-truth column standard deviation**. Separately saved
observed-column-SD NRMSE is 4.000035 for MissBench and 1.045013 for UCI; these
normalization definitions are not interchangeable. Input normalization uses
observed values only; held-out ground truth is used for evaluation scoring,
never to normalize inference inputs.

The checkpoint was selected after inspecting earlier UCI/MissBench monitoring,
so these benchmarks are not untouched selection holdouts. Benchmark masks,
preprocessing, and context settings must match to reproduce results. The generic
wrapper does not implement rating-specific bias centering, user-residual scaling,
or clipping. RecSys and causal evaluations for these weights were pending at
release; no improvement on those tasks is claimed here.

Entirely missing rows or columns have limited observational support. Extreme
values, unusual missingness patterns, or structures outside the synthetic
training distribution can reduce accuracy. Predictions are means, not calibrated
uncertainty estimates. Evaluation datasets and runners are not included in the
standalone package.

## Model files

- `best_validation_checkpoint.pth`: the exact selected checkpoint, with model
  weights and scalar provenance metadata; no optimizer state.
- `model.safetensors`: the identical 254 state-dictionary tensors in safetensors
  format. This is an alternative serialization, not a Transformers AutoModel.
- `config.json`: minimal inference architecture configuration.
- `release.json`: hashes, architecture, training milestone, and validation details.

Checkpoint SHA-256:
`933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`.

## Citation

```bibtex
@misc{saha2026matrixformerfoundationmodelmatrix,
  title={MatrixFormer: A Foundation Model for Matrix Completion},
  author={Dwaipayan Saha and Jacob Feitelberg and Kyuseong Choi and Raaz Dwivedi and Anish Agarwal},
  year={2026},
  eprint={2610.06751},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.06751}
}
```