--- license: apache-2.0 base_model: - Qwen/Qwen3-0.6B ---

# 🪨 Rosetta: Composable Native Multimodal Pretraining

Escaping the Forgetting-Synergy Dilemma in Native Multimodal Pretraining

Xiangyue Liu¹, Zijian Zhang², Miles Yang², Zhao Zhong², Liefeng Bo², Ping Tan^1*

¹HKUST ²Tencent Hunyuan

Figure 1. (Left) Performance on MMLU (language ability) across composable pretraining stages (LM → +MMU → +T2I). Standard MoE and structurally-isolated MoT suffer catastrophic routing collapse upon integrating continuous generative objectives. Rosetta maintains a stable semantic anchor throughout all stages. (Right) Qualitative image generation results from the Rosetta model.

## 🏗️ Architecture

Figure 2. Rosetta FFN. Three mechanisms enable non-destructive modality expansion: (1) Unified Attention — globally shared QKV projections preserve dense cross-modal interactions. (2) Composable FFN — modality-specific plug-and-play experts (Text / ViT / VAE) are bridged by a single Global Shared Expert that anchors foundational knowledge. (3) Conflict-Free Optimization (MAOP) — surgically neutralizes destructive gradients with zero memory overhead.

## 📊 Benchmarks

Comprehensive Performance Evaluations

## ✍️ Citation ```bibtex @misc{liu2026rosettacomposablenativemultimodal, title={Rosetta: Composable Native Multimodal Pretraining}, author={Xiangyue Liu and Zijian Zhang and Miles Yang and Zhao Zhong and Liefeng Bo and Ping Tan}, year={2026}, eprint={2607.00293}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2607.00293}, } ```