De-mystifying Multimodal Learning: Enabiling Vision in Language Models

🤗 Comunity Article, 📝 Blogpost

Introduction

In this first installment of our series, De-mystifying Multimodal Learning, we break down the mechanics of how images become language-compatible vectors. To truly understand how a Large Language Model (LLM) "sees", we must look at the mathematics defining the problem, the training objectives that align vision and text, and the specific architectural steps that process raw pixels, introducing Vision Language Models (VLMs).

We will therefore cover:

Mathematical Formulation: The theoretical foundation and formal definitions of VLMs.

Vision Enocoder Breakdown: A detailed overview of the image processing operated by the ViT-CLIP based Vision Encoders.

Contrastive Learning: Uncovering how CLIP models learn to algin the images and text representations into the same space.

VLM Architecture and Flow: Putting it all together, diving deep in the architectural components of VLMs, detailing the birth of Visual Tokens, the source of sigths for LLMs.

Mathematical Formulation

To understand Vision-Language Models (VLMs), we first need to define the notation and the transformation pipeline formally.

Let $\mathbf{X}\in {\mathbb{R}}^{C\times H\times W}\backslash mathbf\{X\}\; \backslash in\; \backslash mathbb\{R\}^\{C\; \backslash times\; H\backslash times\; W\}$ be an image and $t\in \mathrm{\Sigma }t\; \backslash in\; \backslash Sigma$ be a language instruction input, where $\mathrm{\Sigma }\backslash Sigma$ is the input space of character sequences. Let $s_{\theta ,\gamma ,\varphi }s\_\{\backslash theta,\; \backslash gamma,\; \backslash phi\}$ be an VLM parametrized by $\theta ,\gamma ,\varphi \backslash theta,\; \backslash gamma,\; \backslash phi$. We define $f_{v\theta }f\_\{v\backslash theta\}$ as a contrastively pre-trained Vision Encoder model:

$$f_{v\theta }:{\mathbb{R}}^{C\times H\times W}\to {\mathbb{R}}^{V\times F},f\_\{v\backslash theta\}:\; \backslash mathbb\{R\}^\{C\; \backslash times\; H\; \backslash times\; W\}\; \backslash rightarrow\; \backslash mathbb\{R\}^\{V\; \backslash times\; F\},$$

where $VV$ is the number of visual tokens and $FF$ is their hidden size. $f_{t{\theta }^{\mathrm{\prime }}}f\_\{t\backslash theta\text{'}\}$ represents the corresponding Text Encoder used during the pre-training phase.

To bridge the gap between vision and language, we use a connector $m_{\gamma }:{\mathbb{R}}^{V\times F}\to {\mathbb{R}}^{V\times D}m\_\backslash gamma:\; \backslash mathbb\{R\}^\{V\; \backslash times\; F\}\; \backslash rightarrow\; \backslash mathbb\{R\}^\{V\; \backslash times\; D\}$, typically a Multi-Layer Perceptron (MLP). The token vocabulary for the model is defined as:

$$\mathcal{V}={\mathcal{V}}_{\text{vision}}\cup {\mathcal{V}}_{\text{text}}\backslash mathcal\{V\}\backslash ;=\backslash ;\backslash mathcal\{V\}\_\{\backslash text\{vision\}\}\backslash ;\backslash cup\backslash ;\backslash mathcal\{V\}\_\{\backslash text\{text\}\}$$

The Large Language Model itself is defined as:

$$g_{\varphi }:={\mathcal{D}}_d\circ \mathrm{softmax}\circ F_{{\varphi }^{\mathrm{\prime }}}:{\mathbb{R}}^{J\times D}⟶{\mathcal{V}}^J,\varphi =({\varphi }^{\mathrm{\prime }},d),g\_\{\backslash phi\}\backslash ;:=\backslash ;\backslash mathcal\{D\}\_d\backslash ;\backslash circ\backslash ;\backslash operatorname\{softmax\}\backslash ;\backslash circ\backslash ;F\_\{\backslash phi\text{'}\}\backslash ;\backslash ;:\backslash ;\backslash mathbb\{R\}^\{J\backslash times\; D\}\backslash ;\backslash longrightarrow\backslash ;\backslash mathcal\{V\}^\{J\},\; \backslash qquad\; \backslash phi=\backslash bigl(\backslash phi\text{'},d\backslash bigr),$$

where $F_{{\varphi }^{\mathrm{\prime }}}F\_\{\backslash phi\text{'}\}$ is the transformer that produces logits, and ${\mathcal{D}}_d\backslash mathcal\{D\}\_d$ is a decoding operator (such as greedy, top- $kk$, or nucleus sampling) with hyper-parameters $dd$. Thus, $g_{\varphi }g\_\{\backslash phi\}$ maps an embedded input token sequence to an output token sequence.

Vision Enocoder Breakdown

Now that we have established the mathematical setting, let's look at the architectural implementation of the Vision Encoder $f_{v\theta }f\_\{v\backslash theta\}$, visually represented in Figure 2. Practically, the processing flow of $f_{v\theta }f\_\{v\backslash theta\}$ is broken down into the following steps:

1. Patch Partitioning

The first step is breaking the high-resolution image $\mathbf{X}\backslash mathbf\{X\}$ into a grid of fixed-size patches.Assuming our image has $336\times 336336\; \backslash times\; 336$ pixels and we use a patch size of $P=14P=14$, standard ${}^{\ast }^\{*\}$ vision encoders divide the image into $24\times 24=57624\; \backslash times\; 24\; =\; 576$ distinct squares. Mathematically, the image is reshaped from $\mathbf{X}\in {\mathbb{R}}^{C\times H\times W}\backslash mathbf\{X\}\; \backslash in\; \backslash mathbb\{R\}^\{C\; \backslash times\; H\; \backslash times\; W\}$ into a sequence of flattened 2D patches ${\mathbf{x}}_p\in {\mathbb{R}}^{N\times (P^2\cdot C)}\backslash mathbf\{x\}\_p\; \backslash in\; \backslash mathbb\{R\}^\{N\; \backslash times\; (P^2\; \backslash cdot\; C)\}$, where $NN$ is the total number of patches.

${}^{\ast }^*$ Standard stands for CLIP-like Vision Encoders (Radford et al., 2021, Zhai et al., 2024).

2. Linear Projection and Position Embeddings

These patches are simply raw pixel values. To convert them into vectors, $f_{v\theta }f\_\{v\backslash theta\}$ projects each flattened patch into a latent representation through a linear layer. Given the lack of spatial priors in Vision Transformers (ViT) (Dosovitskiy et al., 2021), these vectors are equipped with learnable positional encodings, injecting "GPS-like" coordinates so the model knows where each patch belongs in the original image.

3. Transformer Layers

The resulting vectors are passed through several Transformer Layers consisting of Multi-Head Self-Attention and MLPs. The output is a sequence of vectors where each vector represents a patch within the context of the whole image. This full process produces the representations ${\mathbf{X}}^{\mathbf{\prime }}=f_{v\theta }(\mathbf{X})\in {\mathbb{R}}^{V\times F}\backslash mathbf\{X\text{'}\}\; =\; f\_\{v\backslash theta\}(\backslash mathbf\{X\})\; \backslash in\; \backslash mathbb\{R\}^\{V\backslash times\; F\}$.

Contrastive Learning

Before the Vision Encoder $f_{v\theta }f\_\{v\backslash theta\}$ can be used in the VLM pipeline, it must learn to extract features that are semantically aligned with text. This is achieved through Contrastive Learning, (extra sources here, here and here) a learning process through which Vision Encoders learn to be powerful feature extractors, compressing visual information into vectors (tokens) semantically aligned with language.
Mathematically, during this pre-training phase, each encoder ( $f_{v\theta }f\_\{v\backslash theta\}$, $f_{t{\theta }^{\mathrm{\prime }}}f\_\{t\backslash theta\text{'}\}$) extracts feature representations for a batch of image-text pairs. Let $t^{\mathrm{\prime }}=f_{t{\theta }^{\mathrm{\prime }}}(t)t\text{'}\; =\; f\_\{t\backslash theta\text{'}\}(t)$ be the text features and ${\mathbf{X}}^{\mathrm{\prime }}=f_{v\theta }(\mathbf{X})\backslash mathbf\{X\}\text{'}\; =\; f\_\{v\backslash theta\}(\backslash mathbf\{X\})$ be the image features. These are normalized as follows

$${\mathbf{X}}_e^{\mathrm{\prime }}=\frac{{\mathbf{X}}^{\mathrm{\prime }}}{\mathrm{\parallel }{\mathbf{X}}^{\mathrm{\prime }}{\mathrm{\parallel }}_2},t_e^{\mathrm{\prime }}=\frac{t^{\mathrm{\prime }}}{\mathrm{\parallel }{t}^{\mathrm{\prime }}{\mathrm{\parallel }}_2}\backslash mathbf\{X\}\text{'}\_\{e\}\; =\; \backslash frac\{\backslash mathbf\{X\}\text{'}\}\{\backslash |\backslash mathbf\{X\}\text{'}\backslash |\_2\},\; \backslash quad\; t\text{'}\_\{e\}\; =\; \backslash frac\{t\text{'}\}\{\backslash |t\text{'}\backslash |\_2\}$$

These normalized features are used to compute the pairwise cosine similarities:

$$\mathit{\text{logits}}=({\mathbf{X}}_e^{\mathrm{\prime }}\cdot t_e^{\mathrm{\prime }T})\cdot e^{\tau }\backslash textit\{logits\}\; =\; (\backslash mathbf\{X\}\_e\text{'}\; \backslash cdot\; t\_e\text{'}^T\; )\; \backslash cdot\; e^\{\backslash tau\}$$

where $t_e^{\mathrm{\prime }T}t\_e\text{'}^\{T\}$ is the transpose of $t_e^{\mathrm{\prime }}t\_e\text{'}$, and $\tau \backslash tau$ is a learnable temperature parameter.These logits are finally used to compute the joint loss function using cross-entropy (CE). The model attempts to maximize the similarity of correct image-text pairs (the diagonal of the matrix) while minimizing others:

Here, labels are the ground truths for that sample, and $\text{axis}=i,\text{with }i\in \{0,1\}\backslash text\{axis\}=i,\; \backslash text\{with\; \}\; i\; \backslash in\; \backslash \{0,1\backslash \}$ represents the dimension along which the loss is computed.

VLM Architecture and Flow

Once the Vision Encoder is pre-trained, we can assemble the full model. Architecturally, Vision Language Models are constituted by three major components:

• Vision Encoders ( $f_{v\theta }f\_\{v\backslash theta\}$), usually a CLIP-like image encoder (Dosovitskiy et al., 2021,Radford et al., 2021, Zhai et al., 2023, Bai et al., 2025), but it can vary in architecture and training style. See this extensive survey for more information.
• Modality Connectors ( $m_{\gamma }m\_\backslash gamma$), often simple Multi-Layer Perceptron, with some architectures employing attention blocks (Li et al., 2023) and other alternatives (Tong et al., 2024, Nulli et al,. 2025).
• Large Language Models ( $g_{\varphi }g\_\backslash phi$) like Qwen3 Yang An, et al. 2025, Llama3 Abhimanyu, et al. 2024, Vicuna Wei-Lin, et al. 2023 and more.

Vision-Language Modeling Pipeline

Putting everything together, we can finally describe the classic VLM pipeline during inference, as depicted in Figure 1. In our calculations below we assume:

• A fixed token count. We defer to our next blogpost "The Hidden Inefficiency in Vision Language Modelling" (coming soon), for an analysis of image pre-processing (Li et al., 2024) or other kinds of spatial merging (QwenTeam, 2025, Gemma-Team, 2025) impacting the total visual token count.
• A batch size of 1.

As per earlier, Vision Encoders $f_{v\theta }f\_\{v\backslash theta\}$ are used to encode an image $\mathbf{X}\backslash mathbf\{X\}$ into a representation:

$${\mathbf{X}}^{\mathrm{\prime }}=f_{v\theta }(\mathbf{X})\in {\mathbb{R}}^{V\times F}\backslash mathbf\{X\}\text{'}\; =\; f\_\{v\backslash theta\}(\backslash mathbf\{X\})\; \backslash in\; \backslash mathbb\{R\}^\{V\; \backslash times\; F\}$$

Here, $FF$ is the feature dimension and $VV$ is the vision encoder hidden dimension, calculated as
$V=(\frac{\mathit{\text{image}}\mathit{\text{resolution}}}{\mathit{\text{patch}}\mathit{\text{size}}}{)}^{2}V\; =\; (\backslash frac\{\backslash textit\{image\; resolution\}\}\{\backslash textit\{patch\; size\}\})^2$ ${}^{\ast \ast }^\{**\}$.

Subsequently, ${\mathbf{X}}^{\mathrm{\prime }}\backslash mathbf\{X\}\text{'}$ is transformed through the connector $m_{\gamma }m\_\backslash gamma$ into Visual Tokens ( $\mathbf{V}\mathbf{T}\backslash mathbf\{VT\}$):

$$\mathbf{V}\mathbf{T}=m_{\gamma }({\mathbf{X}}^{\mathrm{\prime }})\in {\mathbb{R}}^{V\times D}\backslash mathbf\{VT\}\; =\; m\_\backslash gamma(\backslash mathbf\{X\}\text{'})\; \backslash in\; \backslash mathbb\{R\}^\{V\; \backslash times\; D\}$$

Crucially, these tokens now exist in the input embedding space of the Large Language Model. In parallel, a Tokenizer $\mathcal{T}:\mathrm{\Sigma }\to {\mathcal{V}}^J\backslash mathcal\{T\}:\; \backslash Sigma\; \backslash rightarrow\; \backslash mathcal\{V\}^\{J\}$ and a learned embedding $E:{\mathcal{V}}^J⟶{\mathbb{R}}^{D}E:\backslash mathcal\{V\}^\{J\}\backslash ;\backslash longrightarrow\backslash ;\backslash mathbb\{R\}^\{D\}$ turn the text input $tt$ into textual tokens: $TT=E^{\otimes }(\mathcal{T}(t))\in {\mathbb{R}}^{J\times D}\backslash mathit\{TT\}\; =\; E^\{\backslash otimes\}(\backslash mathcal\{T\}(t))\; \backslash in\; \backslash mathbb\{R\}^\{J\; \backslash times\; D\}$, where $E^{\otimes }E^\{\backslash otimes\}$ is the sequence-wise lifting of operator $EE$. Lastly, the visual tokens $\mathbf{V}\mathbf{T}\backslash mathbf\{VT\}$ are concatenated with the textual tokens $TT\backslash mathit\{TT\}$ and provided as input to the LLM $g_{\varphi }g\_\backslash phi$ to obtain the output tokens ${\mathbf{T}}_a\backslash mathbf\{T\}\_a$:

$${\mathbf{T}}_a=g_{\varphi }(\mathbf{V}\mathbf{T}\oplus TT)\in {\mathcal{V}}^J\mathrm{.}\backslash mathbf\{T\}\_a\; =\; g\_\{\backslash phi\}(\backslash mathbf\{VT\}\; \backslash oplus\; \backslash mathit\{TT\})\; \backslash in\; \backslash mathcal\{V\}^\{J\}.$$

${}^{\ast \ast }^\{**\}$ An crucial approximation, which we'll tackle in our blogpost "The Hidden Inefficiency in Vision Language Modelling" (coming soon).

Conclusions

Through the pipeline we've explored, we have witnessed a transformation: raw pixels, once just a grid of intensity values, have been flattened, projected, and semantically aligned to emerge as Visual Tokens. These tokens are the "universal language" that allows an LLM to treat an image not as a foreign file type, but as a sequence of concepts—no different from the words in this sentence. By projecting visual data into the same $DD$-dimensional embedding space as text, we have effectively given the LLM a pair of eyes.

What’s Next: The Efficiency Bottleneck

While we have successfully "digitized" sight for our models, a massive challenge remains. The impact of the amount Visual Tokens created by the vision encoding pipeline.

In our next post, "The Hidden Inefficiency in Vision Language Modelling" (coming soon), we will dive deep into the cost of producing Visual Tokens on Inference Time & Memory Requirements. We will break down how token count impacts self-attention $O(N^2)O(N^2)$ and explore why reducing the visual token count is the secret to building faster, leaner, and more capable multimodal systems.

Citation

If you use this work, please cite:

@misc{nulli2026enabling,
  title={De-mystifying Multimodal Learning: Enabiling Vision in Language Models},
  author={Nulli, Matteo},
  year={2026},
  url={https://huggingface.co/blog/MatteoNulli/de-mystifying-multimodal-learning-enabiling-vision},
  howpublished={Available at \url{https://matteonulli.github.io/blog/2026/demystifying0/} and \url{https://huggingface.co/blog/MatteoNulli/de-mystifying-multimodal-learning-enabiling-vision}},
  note={Hugging Face Blog}
}

References