PMRL / README.md
xhLiu's picture
Create README.md
1b6066e verified
|
Raw
History Blame Contribute Delete
2.62 kB
---
license: mit
---
<div align="center">
<h1><a color="red" href="https://arxiv.org/pdf/2507.17343">Principled Multimodal Representation Learning (PMRL)</a></h1>
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
![License](https://img.shields.io/badge/Accepted-TPAMI'2026-purple)
[![License: MIT](https://img.shields.io/badge/Github-PMRL-black.svg)](https://github.com/Xiaohao-Liu/PMRL)
*A Novel Framework for Representation Learning Across Multiple Modalities*
</div>
---
## โœจ Overview
![](img/top.png)
**Principled Multimodal Representation Learning (PMRL)** addresses the fundamental challenges in multimodal representation learning by proposing a novel framework that achieves simultaneous alignment of multiple modalities without anchor dependency.
### ๐Ÿ’ก Our Solution
PMRL introduces a principled approach grounded in **theoretical insights**:
> **Key Insight**: Full alignment corresponds to a rank-1 Gram matrix
Our framework optimizes the dominant singular value of the representation matrix to align modalities along a shared leading direction.
---
## ๐ŸŽฏ Key Features
๐Ÿ”„ **Simultaneous Multi-Modal Alignment**
- No predefined anchor modality required
- Unified representation space for all modalities
๐Ÿงฎ **Softmax-based Loss Function**
- Treats singular values as logits
- Prioritizes the largest singular value for stable optimization
๐ŸŽฏ **Instance-wise Contrastive Regularization**
- Maintains inter-instance separability
- Prevents representation collapse
โšก **Distributed Training Support**
- Multi-GPU training capabilities
- Efficient data parallel processing
๐Ÿ“Š **Comprehensive Evaluation**
- Extensive benchmarking across diverse tasks
- Quantitative and qualitative analysis tools
---
## ๐Ÿ—๏ธ Architecture
![](img/framework.png)
The PMRL framework consists of three main components:
1. **๐Ÿ”ง Multi-Modal Encoder**: Processes different input modalities
2. **๐ŸŽฏ Singular Value Optimization**: Aligns representations via dominant singular value
3. **๐Ÿ”„ Principled Regularization**: Maintains instance-level discrimination
## Citation
If you find this work useful, please consider citing:
```bibtex
@article{liu2026principled,
title={Principled multimodal representation learning},
author={Liu, Xiaohao and Xia, Xiaobo and Ng, See-Kiong and Chua, Tat-Seng},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2026},
publisher={IEEE}
}
```
<div align="center">
**[๐Ÿ” Back to Top](#-principled-multimodal-representation-learning-pmrl)**
</div>