Description

This repository contains our proposed MD-SVD (Modality-Decoupled Singular Value Decomposition) initialization weights extracted from Stage 1 checkpoints for initializing Stage 2 MHA2MLA-VLM models, which independently compresses visual and textual KV spaces, enabling efficient compression while maintaining model performance.

Available Weight Files

File Name	Latent Dimension (d_kv)
`Qwen2.5-VL-7B-rope32-d_kv_32.pt`	32
`Qwen2.5-VL-7B-rope32-d_kv_64.pt`	64
`Qwen2.5-VL-7B-rope32-d_kv_128.pt`	128

Citation

@misc{fan2026mha2mlavlmenablingdeepseekseconomical,
      title={MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models}, 
      author={Xiaoran Fan and Zhichao Sun and Tao Ji and Lixing Shen and Tao Gui},
      year={2026},
      eprint={2601.11464},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.11464}, 
}