UniRes

Universal Residual Predictor for Neural Network Weight Reconstruction

UniRes is a universal residual network that reconstructs approximate higher-precision (FP32 / FP64) weights from low-precision (BF16 / FP16) tensors of arbitrary shape and size. It operates completely architecture-agnostically and generalises across model families never seen during training (Llama, BERT, T5, Gemma, VAE, CLIP, etc.).

Model Description

UniRes demonstrates cross-quantization transfer on an entirely unseen Qwen VAE model. Despite having no training exposure to the Qwen VAE architecture or its tensor shapes, UniRes reconstructed individual FP32 weights more accurately than direct FP16 conversion on approximately 3.9% of evaluated values and more accurately than direct BF16 conversion on approximately 8.3% of evaluated values. The recurrence of particular parameter groups across both FP16 and BF16 experiments suggests that the effect may reflect statistical properties of the underlying trained weights rather than being exclusively tied to one quantization format. However, UniRes remains substantially worse in aggregate absolute error, so these results do not establish general lossless or superior reconstruction.

Evaluation (Completely Out-of-Distribution)

Ground-truth vs BF16 cast

• Tensors: 194
• Elements: 126 892 531
• Total absolute error: (6.6856248102 \times 10^{2})
• Maximum absolute error: (3.4952163696 \times 10^{-3})

Ground-truth vs UniRes prediction

• Total absolute error: (1.4362314089 \times 10^{3})
• Maximum absolute error: (3.4807920456 \times 10^{-3})

Per-element comparison

• UniRes better than BF16: 10 539 994 elements (8.306 %)
• BF16 better than UniRes: 116 350 729 elements (91.692 %)
• Equal: 1 808 elements (0.001 %)

Ratio of total absolute errors: UniRes / BF16 = 2.148×

While UniRes does not improve aggregate absolute error, it produces a strictly lower absolute deviation on more than ten million individual elements.

Interpretation

The minority of element-wise wins is consistent with the persistence of correlated Gaussian structure in neural-network weight matrices well beyond initialisation (Hirst & Ramgoolam, 2026). These residual correlations—local spatial structure, higher-order moments and permutation-invariant dependencies—transfer to a limited degree across architectures, allowing a universal residual predictor to capture useful signal on a non-negligible fraction of elements.

Citation

If you use UniRes, please cite:

@article{unires2026,
  title={Persistent Correlated Structure in Neural Network Weights Enables Partial Out-of-Distribution Residual Prediction from Low-Precision Representations},
  author={UniRes Authors},
  year={2026}
}

@article{hirst2026gaussianity,
  title={Approximate Gaussianity Beyond Initialisation in Neural Networks},
  author={Hirst, Edward and Ramgoolam, Sanjaye},
  journal={Machine Learning: Science and Technology},
  volume={7},
  number={3},
  pages={035038},
  year={2026},
  eprint={2510.05218}
}