Tabimpute commited on
Commit
22f1e4b
·
verified ·
1 Parent(s): 8bd72f6

Update model card for the single-model API and standalone synthetic training

Browse files
Files changed (1) hide show
  1. README.md +150 -58
README.md CHANGED
@@ -11,80 +11,172 @@ tags:
11
  # MatrixFormer
12
 
13
  MatrixFormer is a **single pretrained neural network** for numerical matrix
14
- completion.
 
 
15
 
16
- The architecture has 77,998,496 parameters, eight alternating row/column attention
17
- layers, width 768, 16 attention heads, 12 learned context tokens and 5,000 output
18
- bins. Point predictions use the distributional mean, not random samples.
 
19
 
20
- ## Files
 
 
21
 
22
- - `best_validation_checkpoint.pth`: the exact selected checkpoint, unchanged,
23
- with model weights and scalar provenance metadata; no optimizer state.
24
- - `model.safetensors`: the identical 254 state-dictionary tensors in safetensors format.
25
- - `config.json`: minimal inference architecture configuration. Training-machine
26
- paths, dataset inventories and the full training config are not published.
27
- - `release.json`: hashes, architecture, training milestone and validation details.
28
 
29
- Original checkpoint SHA256:
30
- `933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`.
31
-
32
- ## Loading
33
-
34
- The runtime is [jacobf18/MatrixFormer](https://github.com/jacobf18/MatrixFormer).
35
- Existing SDK versions may still pin an older release. The following explicit
36
- loader selects these weights regardless of the SDK's default checkpoint:
37
 
38
  ```bash
39
- pip install git+https://github.com/jacobf18/MatrixFormer.git
40
  ```
41
 
 
 
 
 
 
42
  ```python
43
- import json
44
  import numpy as np
45
- from huggingface_hub import hf_hub_download
46
- from matrixformer._inference import Expert
47
-
48
- repo_id = "Tabimpute/MatrixFormer"
49
- revision = "main"
50
- config_path = hf_hub_download(repo_id, "config.json", revision=revision)
51
- checkpoint_path = hf_hub_download(repo_id, "best_validation_checkpoint.pth", revision=revision)
52
- with open(config_path) as handle:
53
- config = json.load(handle)["model"]
54
-
55
- model = Expert(
56
- checkpoint_path, config, device="cpu", precision="float32",
57
- max_rows=2000, max_cols=1000,
58
- )
59
- matrix = np.array([[1.0, 2.0, np.nan], [2.0, np.nan, 6.0], [3.0, 6.0, 9.0]])
60
  completed = model.impute(matrix)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
  ```
62
 
63
- Pin `revision` to this repository's commit hash for reproducible downloads.
64
- `Expert` is the runtime's internal single-network inference class; its name does
65
- not indicate routing or an ensemble. CUDA inference can use `device="cuda"`
66
- and `precision="amp_bfloat16"` on supported hardware. The safetensors file is
67
- an alternative state-dictionary serialization, not a Transformers AutoModel.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
 
69
  ## Evaluation scope and limitations
70
 
71
- The checkpoint was selected after inspecting earlier UCI/MissBench monitoring;
72
- these benchmarks are not untouched selection holdouts. Matched evaluations
73
- completed on 2,091 MissBench and 3,058 UCI eligible cases. The frozen campaign's
74
- primary columnwise NRMSE (held-out-truth column SD) decreased from 1.415619 to
75
- 1.256252 on MissBench and from 1.442289 to 1.285373 on UCI. Under separately
76
- saved observed-column-SD normalization, the corresponding scores decreased
77
- from 4.108524 to 4.000035 and from 1.139899 to 1.045013. These normalization
78
- definitions are not interchangeable.
79
-
80
- The generic inference recipe normalizes observed values per column, preserves
81
- observed entries, and uses the standard orientation. Benchmark-specific masks,
82
- chunking and preprocessing still matter. In particular, the generic wrapper
83
- does not reproduce rating-specific bias centering and user-residual scaling.
84
- RecSys and causal evaluations for this checkpoint were not complete at release;
85
- no improvement on those tasks is claimed here. Predictions are not calibrated
86
- uncertainty estimates. Inputs must be numeric matrices with `NaN` marking missing
87
- entries; categorical variables require preprocessing.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
 
89
  ## Citation
90
 
 
11
  # MatrixFormer
12
 
13
  MatrixFormer is a **single pretrained neural network** for numerical matrix
14
+ completion. Supply a 2D NumPy array with `np.nan` for missing entries; the model
15
+ fills them while preserving observed entries exactly. No task-specific training
16
+ or fine-tuning is required. This release uses one checkpoint, not an ensemble.
17
 
18
+ - [Code and installation guide](https://github.com/jacobf18/MatrixFormer)
19
+ - [Project page](https://jacobf18.github.io/MatrixFormer/)
20
+ - [Paper](https://arxiv.org/abs/2610.06751)
21
+ - [Standalone synthetic training](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md)
22
 
23
+ The architecture has 77,998,496 parameters, eight row/column attention blocks,
24
+ width 768, 16 attention heads, 12 context tokens per axis, and 5,000 output bins.
25
+ Point predictions use the **distributional mean**, not random samples.
26
 
27
+ ## Installation and quick start
 
 
 
 
 
28
 
29
+ Use Python 3.10 or newer. Install the current standalone runtime from GitHub:
 
 
 
 
 
 
 
30
 
31
  ```bash
32
+ python -m pip install --upgrade "git+https://github.com/jacobf18/MatrixFormer.git"
33
  ```
34
 
35
+ Its runtime dependencies are PyTorch 2.4+, NumPy, and `huggingface-hub`. It does
36
+ not import `tabimpute` or require the research `tabular` repository. For a
37
+ platform-specific PyTorch installation, use the
38
+ [PyTorch installation selector](https://pytorch.org/get-started/locally/).
39
+
40
  ```python
 
41
  import numpy as np
42
+ from matrixformer import MatrixFormer
43
+
44
+ matrix = np.array([
45
+ [1.0, 2.0, np.nan],
46
+ [2.0, np.nan, 6.0],
47
+ [3.0, 6.0, 9.0],
48
+ ])
49
+
50
+ model = MatrixFormer.from_pretrained(device="cpu", precision="float32")
 
 
 
 
 
 
51
  completed = model.impute(matrix)
52
+ print(completed)
53
+ ```
54
+
55
+ `from_pretrained()` downloads the final model and matching config from this Hub
56
+ repository using a pinned release and verifies their SHA-256 hashes. No private
57
+ loader or model selector is needed. The public weights require no Hugging Face
58
+ token; later calls reuse the local cache. Allow about 312 MB for the checkpoint,
59
+ in addition to PyTorch and its dependencies.
60
+
61
+ The result is a float64 NumPy array with the same shape and units as the input.
62
+ The input is left unchanged. Reuse the model object for multiple matrices.
63
+ For a BF16-capable NVIDIA GPU, use
64
+ `MatrixFormer.from_pretrained(device="cuda")`; use `precision="float32"` on
65
+ older CUDA GPUs. CPU and CUDA are supported, not MPS.
66
+
67
+ ### Command line and offline use
68
+
69
+ ```bash
70
+ python -m matrixformer --device cpu --precision float32
71
+ python -m matrixformer --input matrix.npy --output completed.npy --device cpu --precision float32
72
  ```
73
 
74
+ The first command runs a small synthetic inference demo. The second completes
75
+ your NumPy array and saves the result; an existing output file is overwritten.
76
+ Use `python -m matrixformer --help` for options.
77
+
78
+ To prepare a portable offline checkpoint directory:
79
+
80
+ ```bash
81
+ python -m matrixformer --download-only --local-dir ./checkpoints
82
+ python -m matrixformer --local-dir ./checkpoints --input matrix.npy --output completed.npy --device cpu --precision float32
83
+ ```
84
+
85
+ The offline machine also needs the installed runtime and its dependencies.
86
+ `local_dir` must contain `config.json` and `best_validation_checkpoint.pth`
87
+ directly. Inference with `local_dir` does not make network requests.
88
+
89
+ ## Input preparation and memory
90
+
91
+ - Use a nonempty 2D numerical matrix with at least one observed value. Mark
92
+ missing values with `np.nan`; zeros and sentinel numbers count as observations.
93
+ - Remove IDs and free text and preprocess categorical variables separately.
94
+ Strings, infinities, and entirely missing matrices are not supported.
95
+ - Inputs are normalized per column using **observed values only** and converted
96
+ back to their original units. Keep the standard row/column orientation.
97
+ - The default context is up to 2,000 rows by 1,000 columns; larger matrices use
98
+ overlapping windows. Adjust `max_rows` and `max_cols` in `from_pretrained()`
99
+ to manage memory. Smaller windows can change predictions.
100
+ - Attention, feed-forward operations, and decoding are streamed in microbatches
101
+ without shortening attention within a context window. The hidden grid still
102
+ occupies device memory; larger contexts can require substantial GPU memory.
103
+
104
+ ## Optional synthetic training
105
+
106
+ The standalone repository now includes a trainer for one randomly initialized
107
+ MatrixFormer, using synthetic targets only and no teacher or ensemble:
108
+
109
+ ```bash
110
+ git clone https://github.com/jacobf18/MatrixFormer.git
111
+ cd MatrixFormer
112
+ python -m pip install -e ".[training]"
113
+ python -m matrixformer.training --output runs/matrixformer --device cuda
114
+ ```
115
+
116
+ The packaged prior combines scientific panels, spectral and latent-factor
117
+ imputation tasks with structured missingness, sparse ratings, and score panels.
118
+ Each optimizer update processes one matrix from each domain. Their loss weights
119
+ are 0.35, 0.35, 0.20, and 0.10, respectively; these are loss weights, not sampling
120
+ probabilities. The objective combines Huber, full-support density NLL, and CDF
121
+ losses. The trainer includes independent synthetic validation, resumable atomic
122
+ checkpoints, and differentiable streamed execution with activation recomputation.
123
+
124
+ The default standalone recipe runs 250,000 optimizer updates, generating
125
+ 1,000,000 training matrices. These are the runnable trainer's defaults, not a
126
+ guarantee of reproducing the published benchmark scores. For a small CPU wiring
127
+ check rather than a full training run:
128
+
129
+ ```bash
130
+ python -m matrixformer.training --output runs/smoke --smoke --device cpu --threads 2
131
+ ```
132
+
133
+ Load your own training output with
134
+ `MatrixFormer.from_checkpoint("runs/matrixformer/best_validation_checkpoint.pth", device="cpu", precision="float32")`.
135
+ This reads the adjacent `config.json`; it does not replace the checksum-pinned
136
+ published model used by `from_pretrained()`. See the
137
+ [training guide](https://github.com/jacobf18/MatrixFormer/blob/main/TRAINING.md)
138
+ for the full recipe and memory controls. Training dependencies are optional for
139
+ pretrained inference.
140
 
141
  ## Evaluation scope and limitations
142
 
143
+ | Benchmark | MatrixFormer NRMSE | Evaluated cases |
144
+ | --- | ---: | ---: |
145
+ | MissBench | 1.256252 | 2,091 |
146
+ | UCI | 1.285373 | 3,058 |
147
+
148
+ Seeds, rates, and patterns are averaged within each dataset before datasets are
149
+ weighted equally. The frozen campaign's primary columnwise NRMSE uses
150
+ **held-out-truth column standard deviation**. Separately saved
151
+ observed-column-SD NRMSE is 4.000035 for MissBench and 1.045013 for UCI; these
152
+ normalization definitions are not interchangeable. Input normalization uses
153
+ observed values only; held-out ground truth is used for evaluation scoring,
154
+ never to normalize inference inputs.
155
+
156
+ The checkpoint was selected after inspecting earlier UCI/MissBench monitoring,
157
+ so these benchmarks are not untouched selection holdouts. Benchmark masks,
158
+ preprocessing, and context settings must match to reproduce results. The generic
159
+ wrapper does not implement rating-specific bias centering, user-residual scaling,
160
+ or clipping. RecSys and causal evaluations for these weights were pending at
161
+ release; no improvement on those tasks is claimed here.
162
+
163
+ Entirely missing rows or columns have limited observational support. Extreme
164
+ values, unusual missingness patterns, or structures outside the synthetic
165
+ training distribution can reduce accuracy. Predictions are means, not calibrated
166
+ uncertainty estimates. Evaluation datasets and runners are not included in the
167
+ standalone package.
168
+
169
+ ## Model files
170
+
171
+ - `best_validation_checkpoint.pth`: the exact selected checkpoint, with model
172
+ weights and scalar provenance metadata; no optimizer state.
173
+ - `model.safetensors`: the identical 254 state-dictionary tensors in safetensors
174
+ format. This is an alternative serialization, not a Transformers AutoModel.
175
+ - `config.json`: minimal inference architecture configuration.
176
+ - `release.json`: hashes, architecture, training milestone, and validation details.
177
+
178
+ Checkpoint SHA-256:
179
+ `933850d7941afe58e5bdf838424ec7160cef9764fbcb26c6462c61c5648a3859`.
180
 
181
  ## Citation
182