|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- muiltimodal |
|
|
- discrete-flow-matching |
|
|
- unifed-model |
|
|
--- |
|
|
|
|
|
## 1. Introduction |
|
|
The rapid progress of large language models (LLMs) has catalyzed the emergence of multimodal large language models (MLLMs) that unify visual understanding and image generation within a single framework. However, most existing MLLMs rely on autoregressive (AR) architectures, which impose inherent limitations on future development, such as the raster-scan order in image generation and restricted reasoning abilities in causal context modeling. In this work, we challenge the dominance of AR-based approaches by introducing **FUDOKI**, a unified multimodal model purely based on discrete flow matching, as an alternative to conventional AR paradigms. By leveraging metric-induced probability paths with kinetic optimal velocities, our framework goes beyond the previous masking-based corruption process, enabling iterative refinement with self-correction capability and richer bidirectional context integration during generation. To mitigate the high cost of training from scratch, we initialize FUDOKI from pre-trained AR-based MLLMs and adaptively transition to the discrete flow matching paradigm. Experimental results show that FUDOKI achieves performance comparable to state-of-the-art AR-based MLLMs across both visual understanding and image generation tasks, highlighting its potential as a foundation for next-generation unified multimodal models. Furthermore, we show that applying test-time scaling techniques to FUDOKI yields significant performance gains, further underscoring its promise for future enhancement through reinforcement learning. |
|
|
|
|
|
<table> |
|
|
<tr> |
|
|
<td width="25%"><img src="teaser.png" alt="image"></td> |
|
|
<td width="30%"><img src="understanding.gif" alt="image"></td> |
|
|
<td width="29%"><img src="generation.gif" alt="image"></td> |
|
|
</tr> |
|
|
</table> |
|
|
|
|
|
[FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities](https://arxiv.org/abs/2505.20147). |
|
|
|
|
|
|
|
|
## 2. Quick Start |
|
|
|
|
|
Please refer to [**Github Repository**](https://github.com/fudoki-hku/FUDOKI) |
|
|
|
|
|
## 3. Citation |
|
|
|
|
|
``` |
|
|
@article{wang2025fudokidiscreteflowbasedunified, |
|
|
title={FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities}, |
|
|
author={Jin Wang and Yao Lai and Aoxue Li and Shifeng Zhang and Jiacheng Sun and Ning Kang and Chengyue Wu and Zhenguo Li and Ping Luo}, |
|
|
year={2025}, |
|
|
eprint={2505.20147}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CV}, |
|
|
url={https://arxiv.org/abs/2505.20147} |
|
|
} |
|
|
``` |
|
|
|
|
|
## 4. Contact |
|
|
|
|
|
**Point of Contact:** [Jin Wang](mailto:wj0529@connect.hku.hk) |
|
|
|
