Abstract
Visual image-generation training can expand pretrained multimodal models' knowledge, with alignment techniques improving text-based access to that visually acquired information.
Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.
Community
We introduce Knowledge Transfer Probing (KTP) to test whether new facts supplied only through images during generation training become accessible through text-only question answering.
Across three unified multimodal models and seven knowledge categories, we find partial but uneven transfer: models often reproduce facts in generated images yet struggle to express them in words.
Interleaved image–text alignment improves average QA accuracy even on entities without QA supervision, outperforming text-only alignment and making visually acquired knowledge more accessible.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Distilling Visual Reasoning into Text Space (2026)
- Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System (2026)
- Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Models (2026)
- Can Learning to Answer Spatial Questions Improve Novel-View Generation? (2026)
- Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces (2026)
- CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation (2026)
- MERIT: Multimodal Evidence Reasoning over Image Tables (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2610.21459 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper