Release single MatrixFormer checkpoint after 250k additional synthetic matrices
Browse filesSelected fixed snapshot: 62,500 continuation updates. Exact checkpoint plus identical safetensors, minimal inference config, provenance and loading instructions. Existing Tabimpute/TabImpute artifacts remain unchanged.
- README.md +101 -0
- best_validation_checkpoint.pth +3 -0
- config.json +21 -0
- model.safetensors +3 -0
- release.json +56 -0
README.md
CHANGED
|
@@ -1,3 +1,104 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- matrix-completion
|
| 5 |
+
- missing-data-imputation
|
| 6 |
+
- tabular
|
| 7 |
+
- pytorch
|
| 8 |
+
- safetensors
|
| 9 |
---
|
| 10 |
+
|
| 11 |
+
# MatrixFormer
|
| 12 |
+
|
| 13 |
+
MatrixFormer is a **single pretrained neural network** for numerical matrix
|
| 14 |
+
completion. This release contains the selected checkpoint after **250,000
|
| 15 |
+
additional synthetic training matrices** (62,500 continuation optimizer updates).
|
| 16 |
+
These counts describe this continuation, not the model's entire training history.
|
| 17 |
+
It is not an ensemble and does not load multiple specialist networks.
|
| 18 |
+
|
| 19 |
+
The architecture has 77,998,496 parameters, eight alternating row/column attention
|
| 20 |
+
layers, width 768, 16 attention heads, 12 learned context tokens and 5,000 output
|
| 21 |
+
bins. Point predictions use the distributional mean, not random samples.
|
| 22 |
+
|
| 23 |
+
## Files
|
| 24 |
+
|
| 25 |
+
- `best_validation_checkpoint.pth`: the exact selected checkpoint, unchanged,
|
| 26 |
+
with model weights and scalar provenance metadata; no optimizer state.
|
| 27 |
+
- `model.safetensors`: the identical 254 state-dictionary tensors in safetensors format.
|
| 28 |
+
- `config.json`: minimal inference architecture configuration. Training-machine
|
| 29 |
+
paths, dataset inventories and the full training config are not published.
|
| 30 |
+
- `release.json`: hashes, architecture, training milestone and validation details.
|
| 31 |
+
|
| 32 |
+
Original checkpoint SHA256:
|
| 33 |
+
`933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`.
|
| 34 |
+
|
| 35 |
+
## Loading
|
| 36 |
+
|
| 37 |
+
The runtime is [jacobf18/MatrixFormer](https://github.com/jacobf18/MatrixFormer).
|
| 38 |
+
Existing SDK versions may still pin an older release. The following explicit
|
| 39 |
+
loader selects these weights regardless of the SDK's default checkpoint:
|
| 40 |
+
|
| 41 |
+
```bash
|
| 42 |
+
pip install git+https://github.com/jacobf18/MatrixFormer.git
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
```python
|
| 46 |
+
import json
|
| 47 |
+
import numpy as np
|
| 48 |
+
from huggingface_hub import hf_hub_download
|
| 49 |
+
from matrixformer._inference import Expert
|
| 50 |
+
|
| 51 |
+
repo_id = "Tabimpute/MatrixFormer"
|
| 52 |
+
revision = "main"
|
| 53 |
+
config_path = hf_hub_download(repo_id, "config.json", revision=revision)
|
| 54 |
+
checkpoint_path = hf_hub_download(repo_id, "best_validation_checkpoint.pth", revision=revision)
|
| 55 |
+
with open(config_path) as handle:
|
| 56 |
+
config = json.load(handle)["model"]
|
| 57 |
+
|
| 58 |
+
model = Expert(
|
| 59 |
+
checkpoint_path, config, device="cpu", precision="float32",
|
| 60 |
+
max_rows=2000, max_cols=1000,
|
| 61 |
+
)
|
| 62 |
+
matrix = np.array([[1.0, 2.0, np.nan], [2.0, np.nan, 6.0], [3.0, 6.0, 9.0]])
|
| 63 |
+
completed = model.impute(matrix)
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
Pin `revision` to this repository's commit hash for reproducible downloads.
|
| 67 |
+
`Expert` is the runtime's internal single-network inference class; its name does
|
| 68 |
+
not indicate routing or an ensemble. CUDA inference can use `device="cuda"`
|
| 69 |
+
and `precision="amp_bfloat16"` on supported hardware. The safetensors file is
|
| 70 |
+
an alternative state-dictionary serialization, not a Transformers AutoModel.
|
| 71 |
+
|
| 72 |
+
## Evaluation scope and limitations
|
| 73 |
+
|
| 74 |
+
The checkpoint was selected after inspecting earlier UCI/MissBench monitoring;
|
| 75 |
+
these benchmarks are not untouched selection holdouts. Matched evaluations
|
| 76 |
+
completed on 2,091 MissBench and 3,058 UCI eligible cases. The frozen campaign's
|
| 77 |
+
primary columnwise NRMSE (held-out-truth column SD) decreased from 1.415619 to
|
| 78 |
+
1.256252 on MissBench and from 1.442289 to 1.285373 on UCI. Under separately
|
| 79 |
+
saved observed-column-SD normalization, the corresponding scores decreased
|
| 80 |
+
from 4.108524 to 4.000035 and from 1.139899 to 1.045013. These normalization
|
| 81 |
+
definitions are not interchangeable.
|
| 82 |
+
|
| 83 |
+
The generic inference recipe normalizes observed values per column, preserves
|
| 84 |
+
observed entries, and uses the standard orientation. Benchmark-specific masks,
|
| 85 |
+
chunking and preprocessing still matter. In particular, the generic wrapper
|
| 86 |
+
does not reproduce rating-specific bias centering and user-residual scaling.
|
| 87 |
+
RecSys and causal evaluations for this checkpoint were not complete at release;
|
| 88 |
+
no improvement on those tasks is claimed here. Predictions are not calibrated
|
| 89 |
+
uncertainty estimates. Inputs must be numeric matrices with `NaN` marking missing
|
| 90 |
+
entries; categorical variables require preprocessing.
|
| 91 |
+
|
| 92 |
+
## Citation
|
| 93 |
+
|
| 94 |
+
```bibtex
|
| 95 |
+
@misc{saha2026matrixformerfoundationmodelmatrix,
|
| 96 |
+
title={MatrixFormer: A Foundation Model for Matrix Completion},
|
| 97 |
+
author={Dwaipayan Saha and Jacob Feitelberg and Kyuseong Choi and Raaz Dwivedi and Anish Agarwal},
|
| 98 |
+
year={2026},
|
| 99 |
+
eprint={2610.06751},
|
| 100 |
+
archivePrefix={arXiv},
|
| 101 |
+
primaryClass={cs.LG},
|
| 102 |
+
url={https://arxiv.org/abs/2610.06751}
|
| 103 |
+
}
|
| 104 |
+
```
|
best_validation_checkpoint.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859
|
| 3 |
+
size 312096387
|
config.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": {
|
| 3 |
+
"attention_dropout": 0.0,
|
| 4 |
+
"drop_path_rate": 0.0,
|
| 5 |
+
"embedding_dropout": 0.0,
|
| 6 |
+
"embedding_size": 768,
|
| 7 |
+
"ffn_dropout": 0.0,
|
| 8 |
+
"mlp_hidden_size": 1536,
|
| 9 |
+
"num_attention_heads": 16,
|
| 10 |
+
"num_cls": 12,
|
| 11 |
+
"num_layers": 8,
|
| 12 |
+
"num_outputs": 5000,
|
| 13 |
+
"positional_damping_factor": 0.1,
|
| 14 |
+
"residual_scale_init": 1.0,
|
| 15 |
+
"rms_norm_eps": 1e-06,
|
| 16 |
+
"rope_base": 10000.0,
|
| 17 |
+
"rope_fraction": 0.5,
|
| 18 |
+
"use_absolute_positional_embeddings": false,
|
| 19 |
+
"use_rope": true
|
| 20 |
+
}
|
| 21 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5b581f6408d42d8f91f59110765f4113d8f094c21a7dd362133bc44515f63717
|
| 3 |
+
size 312026288
|
release.json
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_name": "MatrixFormer",
|
| 3 |
+
"repository": "Tabimpute/MatrixFormer",
|
| 4 |
+
"single_model": true,
|
| 5 |
+
"additional_training_matrices": 250000,
|
| 6 |
+
"continuation_optimizer_updates": 62500,
|
| 7 |
+
"parameters": 77998496,
|
| 8 |
+
"state_dict_tensors": 254,
|
| 9 |
+
"architecture": {
|
| 10 |
+
"attention_dropout": 0.0,
|
| 11 |
+
"drop_path_rate": 0.0,
|
| 12 |
+
"embedding_dropout": 0.0,
|
| 13 |
+
"embedding_size": 768,
|
| 14 |
+
"ffn_dropout": 0.0,
|
| 15 |
+
"mlp_hidden_size": 1536,
|
| 16 |
+
"num_attention_heads": 16,
|
| 17 |
+
"num_cls": 12,
|
| 18 |
+
"num_layers": 8,
|
| 19 |
+
"num_outputs": 5000,
|
| 20 |
+
"positional_damping_factor": 0.1,
|
| 21 |
+
"residual_scale_init": 1.0,
|
| 22 |
+
"rms_norm_eps": 1e-06,
|
| 23 |
+
"rope_base": 10000.0,
|
| 24 |
+
"rope_fraction": 0.5,
|
| 25 |
+
"use_absolute_positional_embeddings": false,
|
| 26 |
+
"use_rope": true
|
| 27 |
+
},
|
| 28 |
+
"original_checkpoint_sha256": "933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859",
|
| 29 |
+
"original_training_config_sha256": "c2349948b2e735caa63184ec618a24219664de8d9d11ab000869c188b707ec0d",
|
| 30 |
+
"selection": "Fixed 250k continuation snapshot selected after UCI/MissBench monitoring; not a blind holdout selection.",
|
| 31 |
+
"inference": {
|
| 32 |
+
"point_prediction": "distributional mean",
|
| 33 |
+
"orientation": "standard",
|
| 34 |
+
"default_normalization": "observed-column mean and standard deviation"
|
| 35 |
+
},
|
| 36 |
+
"validation": {
|
| 37 |
+
"strict_load": true,
|
| 38 |
+
"safetensors_equal": true,
|
| 39 |
+
"cpu_float32_smoke": true,
|
| 40 |
+
"observed_entries_preserved": true
|
| 41 |
+
},
|
| 42 |
+
"files": {
|
| 43 |
+
"best_validation_checkpoint.pth": {
|
| 44 |
+
"sha256": "933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859",
|
| 45 |
+
"bytes": 312096387
|
| 46 |
+
},
|
| 47 |
+
"config.json": {
|
| 48 |
+
"sha256": "4d9f0e56d57fda7a4d7077ef2ece91c80ae206542ecc585fe919b93892c0a48b",
|
| 49 |
+
"bytes": 503
|
| 50 |
+
},
|
| 51 |
+
"model.safetensors": {
|
| 52 |
+
"sha256": "5b581f6408d42d8f91f59110765f4113d8f094c21a7dd362133bc44515f63717",
|
| 53 |
+
"bytes": 312026288
|
| 54 |
+
}
|
| 55 |
+
}
|
| 56 |
+
}
|