README.md

---
title: "Vision-Language Models: From CLIP to Multimodal Agents"
thumbnail: /blog/assets/vlm-technical-guide/thumbnail.png
authors:
- user: silveroupti
---

# Vision-Language Models: From CLIP to Multimodal Agents

*An applied and research-oriented guide for computer vision engineers and researchers.*

## TL;DR

Vision-Language Models, or VLMs, connect visual perception with natural language. They can describe images, answer visual questions, retrieve images from text, perform zero-shot classification, understand documents, read text in images, localize visual concepts, and support multimodal agents.

In this article, we will cover:

* what Vision-Language Models are;
* how they evolved from CLIP-like contrastive models to multimodal LLMs;
* the main architectural families;
* how to choose an open-source VLM;
* how to run inference with `transformers`;
* how to think about fine-tuning;
* how to evaluate hallucination, grounding, OCR, robustness and latency;
* key papers and PhD theses for deeper reading.

This post is inspired by the educational style of Hugging Face articles on VLMs, while taking a more computer-vision-oriented and research-driven perspective.

*Author: Mahmoud Soua — Computer Vision / Machine Learning R&D Engineer and PhD.*  
*Hugging Face: [silveroupti](https://huggingface.co/silveroupti)*

---

## Introduction

Computer vision has traditionally been formulated around closed tasks: image classification, object detection, semantic segmentation, instance segmentation, pose estimation, tracking, and optical character recognition. These tasks remain essential, but they usually require a predefined label space, annotated datasets, and task-specific model heads.

Vision-Language Models change this interface.

Instead of training a model to answer only:

> “Which class is present in this image?”

we can ask:

> “Is there a damaged component near the left edge of the image?”
> “Which object is occluding the pedestrian?”
> “Extract the table content from this document.”
> “Find images that match this textual description.”
> “What could be the cause of this visual anomaly?”

This shift is important for computer vision engineers because VLMs introduce a language-driven interface to visual understanding. They allow us to combine perception, semantics, retrieval, reasoning, and interaction in a single pipeline.

However, VLMs are not magic. They can hallucinate objects, misunderstand spatial relations, fail on small visual details, produce overconfident answers, or break under domain shift. For real-world systems, the key question is not only whether a VLM can answer a question, but whether it can answer reliably, efficiently, and with measurable failure modes.

Historically, VLMs emerged from transformer-based multimodal pretraining, including VisualBERT, ViLBERT, LXMERT and UNITER [1–4]. The field then shifted toward scalable image-text contrastive learning with CLIP and ALIGN [5, 6], followed by generative and instruction-tuned systems such as Flamingo, BLIP-2, InstructBLIP and LLaVA [8, 10–12]. Figure 1 summarizes this evolution.

---

## What is a Vision-Language Model?

A Vision-Language Model is a multimodal model that processes visual and textual information jointly.

Depending on its architecture, a VLM may produce:

* image and text embeddings;
* textual answers;
* image captions;
* bounding boxes;
* segmentation masks;
* OCR-like text extraction;
* visual reasoning outputs;
* actions in an environment.

Common VLM tasks include:

| Task                         | Description                                                     |
| ---------------------------- | --------------------------------------------------------------- |
| Image captioning             | Generate a natural language description of an image             |
| Visual Question Answering    | Answer a question about visual content                          |
| Image-text retrieval         | Retrieve images using text, or retrieve text using images       |
| Zero-shot classification     | Classify images using natural language labels                   |
| Document understanding       | Read and reason over scanned documents, forms, tables or slides |
| OCR reasoning                | Extract and interpret text present in images                    |
| Visual grounding             | Link language expressions to regions in the image               |
| Open-vocabulary detection    | Detect objects from arbitrary text queries                      |
| Open-vocabulary segmentation | Segment objects described by text                               |
| Multimodal agents            | Use images, text and tools to act in an environment             |

At a high level, VLMs answer one central question:

> How can we align visual representations and language representations in a way that supports useful downstream behavior?

---

## Why should computer vision engineers care?

For a computer vision engineer, VLMs are not just “LLMs with images.” They represent a different way of designing visual systems.

Traditional pipelines often look like this:

```text
Image → Vision backbone → Task-specific head → Fixed prediction
```

A VLM-oriented pipeline can look like this:

```text
Image + Text instruction → Multimodal model → Flexible response
```

This has several consequences.

First, language becomes a control interface. A model can be queried with natural language instead of being restricted to a fixed output space.

Second, the same model can often support multiple tasks: captioning, VQA, OCR, retrieval, grounding or visual dialogue.

Third, VLMs can reduce the need for task-specific annotations in some settings, especially for zero-shot or few-shot prototyping.

Fourth, VLMs can be integrated into larger systems: RAG pipelines, document AI systems, robotics stacks, visual inspection tools, multimodal chatbots or agents.

But this flexibility has a cost. VLMs are harder to evaluate than classical supervised models. A bounding box detector can be evaluated with mAP. A segmentation model can be evaluated with IoU. A VLM response may be linguistically plausible but visually wrong.

This is why evaluation, robustness and grounding are central topics.

---

## A short history of Vision-Language Models

The VLM landscape evolved through several waves, as illustrated in Figure 1.

| Period | Representative models | Main idea |
|---|---|---|
| 2019–2020 | VisualBERT [1], ViLBERT [2], LXMERT [3], UNITER [4] | Transformer-based fusion of visual regions and textual tokens |
| 2021 | CLIP [5], ALIGN [6] | Large-scale image-text contrastive learning |
| 2022 | Flamingo [8], CoCa [9], BLIP [10] | Generative and few-shot vision-language learning |
| 2023 | BLIP-2 [11], InstructBLIP [12], LLaVA [13], SigLIP [7] | Efficient bridges to LLMs and visual instruction tuning |
| 2024 | PaliGemma [15], Florence-2 [16], Qwen-VL [17], InternVL [18] | Open-source VLMs, dense vision tasks, OCR and general-purpose multimodal assistants |
| 2025+ | SmolVLM [19], RT-2 [24], VisRAG [25] | Smaller, faster, more agentic and more deployable multimodal systems |

Early vision-language models such as VisualBERT [1], ViLBERT [2], LXMERT [3] and UNITER [4] explored transformer-based multimodal fusion between visual regions and textual tokens.

The second wave was driven by contrastive image-text learning. CLIP [5] and ALIGN [6] demonstrated that large-scale image-text contrastive learning could produce transferable visual representations and strong zero-shot capabilities. SigLIP [7] later revisited this family of approaches by replacing the softmax contrastive objective with a pairwise sigmoid loss.

Generative and few-shot VLMs then connected visual inputs to language generation. Flamingo [8] introduced a visual language model for few-shot learning over interleaved image-text sequences. CoCa [9] combined contrastive and captioning losses in a unified image-text foundation model. BLIP [10] proposed a unified framework for vision-language understanding and generation, while BLIP-2 [11] introduced an efficient bridge between frozen image encoders and frozen large language models through a Querying Transformer, or Q-Former.

The next major step was visual instruction tuning. InstructBLIP [12] studied instruction tuning on top of BLIP-2-style architectures, while LLaVA [13] popularized a simple and influential recipe for connecting visual encoders to LLMs using multimodal instruction-following data.
A useful way to understand this evolution is to separate representation learning from instruction-following.

CLIP-like models learn a shared image-text embedding space. They are excellent for retrieval, zero-shot classification and representation learning.

LLaVA-like models connect a vision encoder to a language model and are optimized for conversation and instruction following.

Grounded VLMs go one step further by linking language to regions, boxes, masks or visual coordinates.

Agentic VLMs extend the paradigm toward tools, user interfaces, robotics or action spaces.

---

## Figure 1 — Evolution of Vision-Language Models

![Evolution of Vision-Language Models](vlm-evolution.png)

**Figure 1.** Simplified evolution of Vision-Language Models, from early vision-language transformers to contrastive image-text learning, generative VLMs, vision-to-LLM bridges, visual instruction tuning and multimodal agents.

---

## Main families of VLMs

Vision-Language Models can be grouped into five practical families, as shown in Figure 2.

### 1. Contrastive VLMs

Contrastive VLMs learn to align images and texts in a shared embedding space.

Representative models:

- CLIP [5];
- ALIGN [6];
- SigLIP [7].

Typical use cases:

- image-text retrieval;
- zero-shot image classification;
- semantic search;
- dataset filtering;
- visual representation learning.

These models are usually not designed to produce long natural language answers. Their strength is representation alignment.

### 2. Generative Vision-Language Models

Generative VLMs are trained to generate text conditioned on visual inputs.

Representative models:

- Flamingo [8];
- CoCa [9];
- BLIP [10];
- BLIP-2 [11].

Typical use cases:

- image captioning;
- VQA;
- image-to-text generation;
- multimodal pretraining;
- few-shot multimodal learning.

These models are important because they connect visual encoding with language generation.

### 3. Instruction-tuned Large Vision-Language Models

Instruction-tuned LVLMs connect an image encoder to a large language model and train the system to follow multimodal instructions.

Representative models:

- InstructBLIP [12];
- LLaVA [13];
- IDEFICS [14];
- PaliGemma [15];
- Qwen-VL [17];
- InternVL [18];
- SmolVLM [19].

Typical use cases:

- visual chat;
- multimodal assistants;
- document understanding;
- visual reasoning;
- OCR-like reasoning;
- domain-specific assistants.

These models are closer to the current “chat with an image” experience.

### 4. Grounded and open-vocabulary VLMs

Grounded VLMs connect text to spatial elements in the image.

Representative models and systems:

- Kosmos-2 [20];
- Grounding DINO [21];
- Segment Anything / SAM [22];
- OWL-ViT [23];
- Florence-2 [16].

Typical use cases:

- visual grounding;
- open-vocabulary detection;
- open-vocabulary segmentation;
- phrase grounding;
- referring expression comprehension;
- document layout understanding.

Typical outputs:

- bounding boxes;
- segmentation masks;
- region-text alignments;
- visual coordinates.

Grounding is critical when a system must not only describe an object, but also localize it.

### 5. Agentic and action-oriented VLMs

Agentic VLMs use visual inputs as part of a broader decision-making system.

Representative directions:

- RT-2-style Vision-Language-Action models [24];
- VisRAG-style multimodal retrieval-augmented generation [25];
- multimodal agents;
- GUI agents;
- robotics systems.

Typical use cases:

- tool use;
- visual planning;
- action prediction;
- robotic control;
- visual navigation;
- multimodal RAG.

This family is still evolving quickly. It is especially relevant when the model must act, not only answer.

---


## Figure 2 — Taxonomy of VLMs

![Taxonomy of Vision-Language Models](vlm-taxonomy.png)

**Figure 2.** Practical taxonomy of Vision-Language Models. Contrastive VLMs focus on aligned embeddings and retrieval. Generative VLMs produce text conditioned on images. Instruction-tuned LVLMs support multimodal dialogue and task following. Grounded VLMs connect language to spatial regions. Agentic VLMs extend perception toward tool use, planning and action.

The taxonomy in Figure 2 is useful because the term “VLM” is overloaded. A CLIP-like model, a BLIP-2-like model and a LLaVA-like model are all Vision-Language Models, but they are optimized for different objectives and downstream behaviors. Contrastive models are usually better suited for retrieval and zero-shot classification [5–7], while instruction-tuned LVLMs are more appropriate for visual dialogue, document understanding and open-ended question answering [11, 12].

---

## Architecture of a modern VLM

A modern VLM usually contains four components: Figure 3 summarizes this generic architecture.

1. a vision encoder;
2. a multimodal projection or alignment module;
3. a language model;
4. a tokenizer and chat/instruction template.

A simplified architecture looks like this:

```text
Image > Vision Encoder > Visual Embeddings > Multimodal Projector > Language Model ← Text Prompt > Generated Answer
``` 

The vision encoder is often a Vision Transformer or a CLIP/SigLIP-like encoder. The multimodal projector maps visual embeddings into a representation that the language model can consume. The language model then generates text conditioned on both visual tokens and textual tokens.

Different models implement this bridge differently:

| Bridge type                | Description                                                | Example direction        |
| -------------------------- | ---------------------------------------------------------- | ------------------------ |
| Linear projector           | Simple mapping from visual features to LLM embedding space | LLaVA-style models       |
| MLP projector              | More expressive visual-to-text projection                  | Instruction-tuned LVLMs  |
| Q-Former                   | Query-based transformer that extracts visual tokens        | BLIP-2                   |
| Perceiver Resampler        | Resamples visual features into a fixed number of tokens    | Flamingo-like approaches |
| Native multimodal backbone | Model designed for multimodal inputs from the beginning    | Some recent VLMs         |

The design of the visual token interface matters. High-resolution images, document understanding and small-object perception often require many visual tokens. But more visual tokens increase memory usage and latency.

This creates a fundamental trade-off:

> More visual detail usually means more computation.

---

## Figure 3 — Generic VLM architecture

![Architecture of a modern VLM](vlm-architecture.png)

**Figure 3.** Generic architecture of a modern VLM. The image is encoded into visual features, projected into the language-model token space, and fused with textual tokens before generation.

The vision encoder is often a Vision Transformer, CLIP-like encoder or SigLIP-like encoder. The multimodal projector maps visual representations into a token space that can be consumed by the language model. This bridge can be implemented as a linear projection, an MLP, a Q-Former or a resampler. BLIP-2 uses a Querying Transformer to connect frozen image encoders with frozen LLMs [10], while LLaVA-style models use a simpler visual projection strategy for multimodal instruction tuning [12]. Flamingo-like approaches use architectural mechanisms designed to handle interleaved visual and textual sequences [8].

---

## Open-source VLM landscape

The open-source VLM landscape is large and changes quickly. Instead of trying to rank every model, it is more useful to map models to use cases.

| Model family | Strengths | Typical use |
|---|---|---|
| CLIP [5] | Strong image-text embeddings | Retrieval, zero-shot classification |
| ALIGN [6] | Large-scale noisy image-text contrastive learning | Retrieval, representation learning |
| SigLIP [7] | Sigmoid-based contrastive alignment | Retrieval, representation learning |
| Flamingo [8] | Few-shot multimodal learning over interleaved sequences | Multimodal few-shot prompting |
| CoCa [9] | Joint contrastive and captioning training | Image-text foundation modeling |
| BLIP [10] | Unified vision-language understanding and generation | Captioning, VQA, retrieval |
| BLIP-2 [11] | Efficient bridge between frozen vision encoder and LLM | Captioning, VQA, image-to-text generation |
| InstructBLIP [12] | Instruction-following on visual tasks | VQA, multimodal instruction following |
| LLaVA [13] | Simple and influential visual instruction tuning recipe | Visual chat, research prototypes |
| IDEFICS [14] | Open reproduction of Flamingo-like visual language modeling | Multimodal generation and chat |
| PaliGemma [15] | Compact and fine-tunable VLM family | Applied CV, transfer, fine-tuning |
| Florence-2 [16] | Prompt-based dense vision foundation model | Captioning, detection, grounding, segmentation |
| Qwen-VL [17] | OCR, localization and text-reading abilities | Document AI, multilingual VLM tasks |
| InternVL [18] | General-purpose visual-linguistic foundation model | Multimodal understanding and dialogue |
| SmolVLM [19] | Small and efficient VLM family | Local inference, edge-oriented prototyping |
| Kosmos-2 [20] | Grounded multimodal generation | Grounding and phrase-region alignment |
| Grounding DINO [21] | Open-set object detection with language prompts | Open-vocabulary detection |
| SAM [22] | Promptable segmentation | Segmentation masks and visual annotation |
| OWL-ViT [23] | Open-vocabulary object detection | Text-conditioned object detection |
| RT-2 [24] | Vision-Language-Action transfer to robotics | Robotic control |
| VisRAG [25] | Vision-based retrieval-augmented generation | Multimodal document RAG |

Figure 2 helps clarify why model selection should be task-driven rather than leaderboard-driven. CLIP and SigLIP are strong choices for image-text retrieval and zero-shot classification [5, 7]. BLIP-2 and InstructBLIP are more suitable when the goal is to connect visual inputs to language generation or instruction following [10, 11]. PaliGemma, Florence-2, Qwen-VL and InternVL broaden the scope toward compact transfer, dense vision tasks, OCR, localization and general-purpose multimodal understanding [13–16].
A practical engineering decision is rarely “Which model is best?” The better question is:

> Which model is reliable enough for my task, hardware, latency budget, license constraints and evaluation protocol?

---

## How to choose a VLM

Here is a practical decision table.

| Requirement | Recommended direction |
|---|---|
| Image-text retrieval | CLIP [5], ALIGN [6], SigLIP [7] |
| Zero-shot classification | CLIP [5], SigLIP [7] |
| Visual chat | LLaVA [13], IDEFICS [14], Qwen-VL [17], InternVL [18] |
| OCR and document understanding | Qwen-VL [17], PaliGemma [15], DocVQA-style evaluation [28] |
| Fine-tuning on custom visual data | PaliGemma [15], LLaVA-style models [13], SmolVLM [19] |
| Low GPU memory | SmolVLM [19] |
| Open-vocabulary detection | Grounding DINO [21], OWL-ViT [23], Florence-2 [16] |
| Grounded generation | Kosmos-2 [20], Florence-2 [16] |
| Open-vocabulary segmentation | SAM [22] combined with Grounding DINO [21] or OWL-ViT [23] |
| Industrial visual inspection | Small, testable, domain-adapted VLMs such as PaliGemma [15] or SmolVLM [19] |
| Robotics or GUI control | RT-2-style Vision-Language-Action models [24] |
| Multimodal RAG | VisRAG-style vision-based retrieval-augmented generation [25] |
| Video understanding | SmolVLM-style small multimodal models [19] |

When choosing a VLM, I usually check the following criteria:

1. **Task fit**: Is the model designed for retrieval, generation, OCR, grounding, or chat?
2. **Image resolution**: Can it preserve the details required by the task?
3. **OCR ability**: Can it read small text, tables, forms or screenshots?
4. **Grounding**: Can it localize what it describes?
5. **Language support**: Does it work only in English or also in French and other languages?
6. **License**: Can it be used commercially?
7. **Memory footprint**: Can it run on the target hardware?
8. **Latency**: Is it fast enough for the application?
9. **Fine-tuning support**: Does it work with LoRA, QLoRA, PEFT or TRL?
10. **Evaluation evidence**: Are there public benchmarks, ablations or domain-specific tests?

---

## Running inference with Transformers

Let us run a simple VLM inference example using `transformers`.

The current `image-text-to-text` interface supports instruction-tuned VLMs through `AutoProcessor` and `AutoModelForImageTextToText`.

### Installation

```bash
pip install -q transformers accelerate pillow
```

For some models and GPUs, Flash Attention can improve performance:

```bash
pip install flash-attn --no-build-isolation
```

### Single-image inference

```python
from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "HuggingFaceTB/SmolVLM2-256M-Video-Instruct"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "http://images.cocodataset.org/val2017/000000039769.jpg",
            },
            {
                "type": "text",
                "text": "Describe this image and mention the main objects.",
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=128)

generated_text = processor.batch_decode(
    output_ids,
    skip_special_tokens=True,
)[0]

print(generated_text)
```

### A more computer-vision-oriented prompt

For applied vision tasks, prompts should be explicit about uncertainty and visual evidence.

```python
conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "path": "inspection_sample.jpg",
            },
            {
                "type": "text",
                "text": (
                    "Analyze this image as a visual inspection engineer. "
                    "List visible defects only. "
                    "If you are uncertain, say so explicitly. "
                    "Do not invent objects that are not clearly visible."
                ),
            },
        ],
    }
]
```

This will not eliminate hallucinations, but it makes the expected behavior clearer.

---

## Prompting VLMs: useful patterns

Prompting is not a substitute for evaluation, but it matters.

### Bad prompt

```text
What is wrong with this image?
```

This is too open-ended. The model may speculate.

### Better prompt

```text
Inspect the image for visible surface defects. 
Return only defects that are visually supported by the image. 
For each defect, provide:
1. defect type;
2. approximate location;
3. confidence: low, medium, or high.
If no defect is visible, answer: "No visible defect."
```

### Structured output prompt

```text
Return the answer as JSON with the following schema:

{
  "visible_defects": [
    {
      "type": "...",
      "location": "...",
      "visual_evidence": "...",
      "confidence": "low | medium | high"
    }
  ],
  "uncertainties": ["..."]
}
```

Structured outputs are useful when the VLM is part of a larger pipeline.

---

## Fine-tuning strategy

Fine-tuning a VLM can be useful when:

* your domain differs from web images;
* the model must learn domain-specific vocabulary;
* the output format must be strict;
* OCR must improve on a specific document type;
* visual anomalies are rare or domain-specific;
* the model must reduce hallucination in a controlled setting.

A practical VLM fine-tuning stack usually includes:

* `transformers`;
* `datasets`;
* `peft`;
* `trl`;
* LoRA or QLoRA;
* curated image-instruction pairs;
* a validation set designed around failure modes.

### Example dataset format

A fine-tuning example can be represented as an image plus a conversation.

```json
{
  "image": "images/sample_001.jpg",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image"},
        {
          "type": "text",
          "text": "Inspect the image and list visible surface defects."
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "One scratch is visible near the lower-right area. No other visible defect is clearly identifiable."
        }
      ]
    }
  ]
}
```

### Fine-tuning checklist

Before fine-tuning, ask:

1. Do I have enough high-quality image-text pairs?
2. Are the target outputs consistent?
3. Are negative examples included?
4. Are uncertain cases labeled as uncertain?
5. Is there a held-out validation set?
6. Does the validation set include hard cases?
7. Do I measure hallucination, not only accuracy?
8. Do I compare against prompt-only baselines?
9. Do I track latency and memory after fine-tuning?
10. Does the license allow the intended use?

Fine-tuning should not be treated as a default solution. Sometimes, better prompting, retrieval augmentation, image preprocessing, higher resolution or task decomposition may be more effective.

---

## Evaluation: beyond generic benchmarks

Evaluating a VLM is difficult because the output is often open-ended text. Figure 4 proposes a practical evaluation pipeline for moving from generic benchmarks to application-specific validation.

For real-world use, I recommend evaluating at least nine dimensions.

| Evaluation axis   | What to measure                                                |
| ----------------- | -------------------------------------------------------------- |
| Accuracy          | Is the answer correct?                                         |
| Hallucination     | Does the model invent objects, attributes or facts?            |
| Grounding         | Can the model localize what it describes?                      |
| OCR               | Can it read text in images, documents or screenshots?          |
| Spatial reasoning | Does it understand position, relation, occlusion and counting? |
| Robustness        | Does it resist blur, noise, compression and domain shift?      |
| Calibration       | Does the model express uncertainty appropriately?              |
| Latency           | Is inference fast enough for the application?                  |
| Memory            | Can it run on the target hardware?                             |

### Benchmark examples

Useful benchmarks and evaluation directions include:

- MMMU [26] for multidisciplinary multimodal reasoning;
- MMBench [27] for general multimodal capabilities;
- DocVQA [28] for document visual question answering;
- MathVista [29] for visual mathematical reasoning;
- OCRBench [30] for OCR-oriented VLM evaluation;
- POPE [31] for object hallucination evaluation;
- custom domain-specific test sets for industrial, medical, satellite or document-oriented deployment.

Generic benchmarks are useful but insufficient. MMMU [26] evaluates multimodal models on multidisciplinary reasoning tasks requiring expert-level subject knowledge. MMBench [27] focuses on broad multimodal capability evaluation through a structured benchmark design. DocVQA [28] evaluates visual question answering on document images. MathVista [29] focuses on mathematical reasoning in visual contexts. OCRBench [30] evaluates OCR-related capabilities in large multimodal models. POPE [31] was proposed to evaluate object hallucination in large vision-language models.

Benchmarks are useful, but they are not enough. If the model will be used on industrial images, medical images, satellite images, retail shelves, invoices, drawings, or inspection images, then a domain-specific validation set is mandatory.
Generic benchmarks are useful but insufficient. MMMU evaluates multimodal models on multidisciplinary reasoning tasks requiring expert-level subject knowledge [17]. MMBench focuses on broad multimodal capability evaluation through a structured benchmark design [18]. OCRBench evaluates OCR-related capabilities in large multimodal models across text recognition, document VQA, key information extraction and related tasks [19]. POPE was proposed to evaluate object hallucination in large vision-language models through polling-based object probing [20].

---

## Failure modes

VLMs can fail in ways that are different from classical vision models.

### 1. Object hallucination

The model describes an object that is not present in the image.

Example:

> “There is a dog next to the car.”

when no dog is visible.

### 2. Attribute hallucination

The object exists, but the model invents its attributes.

Example:

> “The car is red.”

when the car is actually black.

### 3. Spatial errors

The model misunderstands object position or relation.

Example:

> “The cup is to the left of the laptop.”

when it is on the right.

### 4. Counting errors

Counting remains difficult, especially with occlusion, small objects or repeated patterns.

### 5. OCR errors

Text in images can be misread, especially when it is small, distorted, handwritten, rotated or low-resolution.

### 6. Overconfidence

The model gives a fluent answer even when the visual evidence is weak.

### 7. Domain shift

A model trained mostly on web images may fail on industrial, medical, satellite, microscopic or infrared imagery.

---

## Practical deployment considerations

For deployment, a VLM should be treated as one component in a larger system.

A robust deployment pipeline may include:

```text
Input image > Image quality checks > Optional preprocessing > VLM inference Post-processing / validation > Confidence and uncertainty handling > Human review if needed > Final decision or report
```

Important engineering questions:

| Question                              | Why it matters                            |
| ------------------------------------- | ----------------------------------------- |
| What is the maximum image resolution? | Determines detail preservation and memory |
| How many visual tokens are used?      | Impacts latency and GPU memory            |
| Can outputs be constrained?           | Important for automation                  |
| Are uncertain cases detected?         | Important for safety and review           |
| Is there a fallback model?            | Useful for production robustness          |
| Are prompts versioned?                | Prompt changes can change behavior        |
| Are outputs logged?                   | Needed for debugging and monitoring       |
| Is human validation required?         | Critical for high-stakes use cases        |

---

## Figure 4 — Evaluation pipeline

![VLM evaluation pipeline](vlm-evaluation-pipeline.png)

**Figure 4.** Practical evaluation pipeline for Vision-Language Models. A reliable evaluation should combine diverse datasets, controlled prompt sets, inference logging, multimodal metrics and deployment decisions with human review when needed.

---

## Research directions

VLM research is moving quickly. From a computer vision perspective, I find the following directions especially important.

### 1. Reducing hallucination

A VLM should not invent objects, attributes or relations. This is still a major limitation for deployment.

### 2. Better grounding

Models should connect generated language to visual evidence: regions, boxes, masks, points or coordinates.

### 3. Domain adaptation

Industrial, medical, satellite, robotics and document data often differ strongly from web images.

### 4. Small and efficient VLMs

Large models are powerful, but many real applications need small, fast and memory-efficient models.

### 5. Multimodal RAG

Retrieval-augmented generation can be extended to images, documents, diagrams, screenshots and videos.

### 6. Video-Language Models

Video introduces temporal reasoning, event understanding, motion, causality and long-context memory.

### 7. Vision-Language-Action models

Robotics and GUI agents require models that do not only describe the world, but act in it.

### 8. Evaluation and uncertainty

The field needs better tools to measure uncertainty, calibration, failure detection and visual reliability.

---

## References

### Early vision-language transformers

[1] Li et al., 2019 — [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://arxiv.org/abs/1908.03557)

[2] Lu et al., 2019 — [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://arxiv.org/abs/1908.02265)

[3] Tan and Bansal, 2019 — [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://arxiv.org/abs/1908.07490)

[4] Chen et al., 2020 — [UNITER: UNiversal Image-TExt Representation Learning](https://arxiv.org/abs/1909.11740)

### Contrastive image-text learning

[5] Radford et al., 2021 — [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020)

[6] Jia et al., 2021 — [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://arxiv.org/abs/2102.05918)

[7] Zhai et al., 2023 — [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343)

### Generative and few-shot VLMs

[8] Alayrac et al., 2022 — [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198)

[9] Yu et al., 2022 — [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://arxiv.org/abs/2205.01917)

[10] Li et al., 2022 — [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086)

[11] Li et al., 2023 — [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597)

### Instruction-tuned and Flamingo-like LVLMs

[12] Dai et al., 2023 — [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500)

[13] Liu et al., 2023 — [LLaVA: Visual Instruction Tuning](https://arxiv.org/abs/2304.08485)

[14] Laurençon et al., 2023 — [IDEFICS: An Open Reproduction of Flamingo](https://huggingface.co/blog/idefics)

### Recent open-source and general-purpose VLMs

[15] Beyer et al., 2024 — [PaliGemma: A Versatile 3B VLM for Transfer](https://arxiv.org/abs/2407.07726)

[16] Xiao et al., 2023 — [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://arxiv.org/abs/2311.06242)

[17] Bai et al., 2023 — [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://arxiv.org/abs/2308.12966)

[18] Chen et al., 2023 — [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://arxiv.org/abs/2312.14238)

[19] Marafioti et al., 2025 — [SmolVLM: Redefining Small and Efficient Multimodal Models](https://arxiv.org/abs/2504.05299)

### Grounding, open-vocabulary detection and segmentation

[20] Peng et al., 2023 — [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://arxiv.org/abs/2306.14824)

[21] Liu et al., 2023 — [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://arxiv.org/abs/2303.05499)

[22] Kirillov et al., 2023 — [Segment Anything](https://arxiv.org/abs/2304.02643)

[23] Minderer et al., 2022 — [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230)

### Agentic, VLA and multimodal RAG systems

[24] Brohan et al., 2023 — [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://arxiv.org/abs/2307.15818)

[25] Yu et al., 2024 — [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://arxiv.org/abs/2410.10594)

### Evaluation and hallucination

[26] Yue et al., 2023 — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://arxiv.org/abs/2311.16502)

[27] Liu et al., 2023 — [MMBench: Is Your Multi-modal Model an All-around Player?](https://arxiv.org/abs/2307.06281)

[28] Mathew et al., 2020 — [DocVQA: A Dataset for VQA on Document Images](https://arxiv.org/abs/2007.00398)

[29] Lu et al., 2023 — [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://arxiv.org/abs/2310.02255)

[30] Liu et al., 2023 — [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://arxiv.org/abs/2305.07895)

[31] Li et al., 2023 — [Evaluating Object Hallucination in Large Vision-Language Models](https://arxiv.org/abs/2305.10355)

### Surveys

[32] Du et al., 2022 — [A Survey of Vision-Language Pre-Trained Models](https://arxiv.org/abs/2202.10936)

[33] Yin et al., 2023 — [A Survey on Multimodal Large Language Models](https://arxiv.org/abs/2306.13549)

[34] Caffagni et al., 2024 — [The Revolution of Multimodal Large Language Models: A Survey](https://arxiv.org/abs/2402.12451)

[35] Ghosh et al., 2024 — [Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions](https://arxiv.org/abs/2404.07214)

---

## Relevant PhD theses and dissertations

A strong VLM article should not only cite papers. Recent PhD theses provide deeper and more systematic analyses, especially on multimodal understanding, robustness, adaptation, retrieval, open-vocabulary recognition and trustworthy VLMs.

Suggested theses to explore:

* Élodie Salin, 2023 — *Study of the Multimodal Understanding of Vision-Language Transformer Models*.
* Hugo Laurençon, 2025 — *Foundation Vision-Language Models*.
* Marc Lafon, 2025 — *Robustesse et adaptation des modèles vision-langage pré-entraînés*.
* Ivona Najdenkoska, 2025 — *Learning from Context with Multimodal Foundation Models*.
* M. Y. Hendriksen, 2024 — *Multimodal Machine Learning for Information Retrieval: A Vision and Language Perspective*.
* Shuang Ao, 2025 — *Building Trustworthy AI: Uncertainty Quantification and Failure Detection in Large Vision-Language Models*.
* Philipp Kaul, 2024 — *Developing Object Perception in the Low Data Regime*.
* Alessandro Conti, 2025 — *Learning without Labels*.
* Monika Wysoczańska, 2025 — *Task Adaptation Strategies for Vision-Language Models*.
* Maria Bravo Sarmiento, 2025 — *Advancing Vision-Language Models for Open-Vocabulary Recognition and Generative Evaluation*.

These theses are useful because they provide a more systematic research perspective than short benchmark-driven model reports.

---

## Conclusion

Vision-Language Models are reshaping computer vision by moving from fixed-label prediction to language-guided visual understanding.

For computer vision engineers, they provide a powerful interface between visual perception, language, retrieval, reasoning and interaction. But they also introduce new engineering challenges: hallucination, grounding, prompt sensitivity, OCR failures, uncertainty, domain shift, latency and memory cost.

The practical goal is not simply to use the largest VLM available. The goal is to select, evaluate, adapt and deploy a VLM that is reliable enough for the target visual system.

In my view, the next generation of useful VLM systems will not be defined only by benchmark scores. They will be defined by how well they combine:

* visual grounding;
* calibrated uncertainty;
* efficient inference;
* domain adaptation;
* strong evaluation;
* human-in-the-loop validation;
* and integration into real computer vision pipelines.

VLMs are not the end of classical computer vision. They are a new layer on top of it: a language interface for visual intelligence.

---

*Author: Mahmoud Soua — Computer Vision / Machine Learning R&D Engineer and PhD.*  
*Hugging Face: [silveroupti](https://huggingface.co/silveroupti)*