README.md

---
license: apache-2.0
base_model:
- Qwen/Qwen3-0.6B
---
<div align="center">

# 🪨 Rosetta: Composable Native Multimodal Pretraining

<p align="center">
  <a href="https://rosetta-lmm.github.io/">
    <img src="https://img.shields.io/badge/Rosetta-Website-2F6DBD?logo=safari&logoColor=white" style="display: inline-block; vertical-align: middle;" alt="Rosetta Website" />
  </a>
  <a href="https://arxiv.org/abs/2607.00293">
    <img src="https://img.shields.io/badge/Paper-arXiv-B5212F?logo=arxiv&logoColor=white" style="display: inline-block; vertical-align: middle;" alt="Paper" />
  </a>
  <a href="https://github.com/Lxiangyue/Rosetta" target="_blank" style="margin: 2px;">
    <img src="https://img.shields.io/badge/Rosetta-Codebase-536af5?logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;" alt="Rosetta Codebase" />
  </a>
  <a href="https://huggingface.co/tencent/Rosetta-inference">
    <img src="https://img.shields.io/badge/🤗%20HuggingFace-Model-d96902" style="display: inline-block; vertical-align: middle;" alt="HuggingFace" />
  </a>
  <!-- <img src="https://img.shields.io/badge/License-Apache%202.0-green" style="display: inline-block; vertical-align: middle;" alt="License" /> -->
</p>

<p align="center">
  <b>Escaping the Forgetting-Synergy Dilemma in Native Multimodal Pretraining</b>
</p>

<p align="center">
  <a href="https://xiangyueliu.github.io/">Xiangyue Liu</a><sup>1</sup>,
  <a href="https://scholar.google.com/citations?user=TZ0nnhgAAAAJ&hl=zh-CN">Zijian Zhang</a><sup>2</sup>,
  <a href="https://scholar.google.com/citations?user=Miles_Yang2">Miles Yang</a><sup>2</sup>,
  <a href="https://scholar.google.com/citations?user=igtXP_kAAAAJ&hl=en">Zhao Zhong</a><sup>2</sup>,
  <a href="https://scholar.google.com/citations?user=FJwtMf0AAAAJ&hl=en">Liefeng Bo</a><sup>2</sup>,
  <a href="https://scholar.google.com/citations?user=XhyKVFMAAAAJ&hl=en">Ping Tan</a><sup>1*</sup>
</p>
<p align="center">
  <sup>1</sup>HKUST &nbsp;&nbsp; <sup>2</sup>Tencent Hunyuan
</p>

</div>

<p align="center">
  <img src="https://rosetta-lmm.github.io/assets/figures/teaser.jpg" width="100%">
</p>

<p align="justify" style="font-size: 0.92em; color: #555555; line-height: 1.5;"><b>Figure 1.</b> <i>(Left)</i> Performance on MMLU (language ability) across composable pretraining stages (LM → +MMU → +T2I). Standard MoE and structurally-isolated MoT suffer catastrophic routing collapse upon integrating continuous generative objectives. <b>Rosetta maintains a stable semantic anchor throughout all stages.</b> <i>(Right)</i> Qualitative image generation results from the Rosetta model.</p>

<!-- --- -->

## 🏗️ Architecture

<p align="center">
  <img src="https://rosetta-lmm.github.io/assets/figures/architecture.png" width="100%">
</p>

<p align="justify" style="font-size: 0.92em; color: #555555; line-height: 1.5;"><b>Figure 2. Rosetta FFN.</b> Three mechanisms enable non-destructive modality expansion: <b>(1) Unified Attention</b> — globally shared QKV projections preserve dense cross-modal interactions. <b>(2) Composable FFN</b> — modality-specific plug-and-play experts (Text / ViT / VAE) are bridged by a single Global Shared Expert that anchors foundational knowledge. <b>(3) Conflict-Free Optimization (MAOP)</b> — surgically neutralizes destructive gradients with zero memory overhead.</p>

<!-- --- -->

## 📊 Benchmarks

<p align="center">
  <img src="https://rosetta-lmm.github.io/assets/figures/table.png" alt="Comprehensive Performance Evaluations" width="100%">
</p>


## ✍️ Citation
```bibtex
@misc{liu2026rosettacomposablenativemultimodal,
      title={Rosetta: Composable Native Multimodal Pretraining}, 
      author={Xiangyue Liu and Zijian Zhang and Miles Yang and Zhao Zhong and Liefeng Bo and Ping Tan},
      year={2026},
      eprint={2607.00293},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00293}, 
}
```