---
license: other
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - vllm
  - vision-language
  - image-captioning
  - fastvlm
base_model:
  - apple/FastVLM-0.5B
  - LiquidAI/LFM2.5-VL-450M
---

# FastLFM2.5-VL-0.5B

**FastLFM2.5-VL-0.5B retains 92% of its teacher's average captioning quality while delivering 1.5x throughput and using 4x fewer visual tokens.**

This model combines the vision encoder from [Apple FastVLM-0.5B](https://huggingface.co/apple/FastVLM-0.5B) with the language model from [LiquidAI LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M). A small residual connector maps the FastVLM image features into the LFM2 embedding space.

The Apple vision encoder and Liquid language model remained frozen. Only the connector path, consisting of the residual MLP and its learned 2D positional embeddings, was trained. Training used supervised fine-tuning (SFT) followed by on-policy distillation (OPD) from the LFM2.5-VL-450M teacher.

## Results

| Model | Input size | COCO CIDEr | COCO retention | NoCaps CIDEr | NoCaps retention | Visual tokens | Throughput vs. teacher |
|---|---:|---:|---:|---:|---:|---:|---:|
| LiquidAI/LFM2.5-VL-450M | 512x512 | 0.9109 | 100% | 0.9443 | 100% | 256 (1x) | 1.00x |
| apple/FastVLM-0.5B | 512x512 | 0.7606 | 83.5% | 0.8103 | 85.8% | 64 (4x fewer) | 1.53x |
| **FastLFM2.5-VL-0.5B** | **512x512** | **0.8309** | **91.2%** | **0.8782** | **93.0%** | **64 (4x fewer)** | **1.51x** |

The two FastLFM retention scores average to **92.1%**. Retention is computed independently for each benchmark against the 512x512 LFM teacher.

Throughput was measured on 1,000 unique, already-loaded 512x512 images on an NVIDIA RTX PRO 6000 Blackwell Server Edition, using batches of 250 and exactly 32 generated tokens after three warmup batches.

## Architecture

```text
512x512 image; 64 visual tokens; d_vision=3072; d_LLM=1024

                    ┌─────────────────────┐
                    │ FastVLM Encoder     │  frozen
                    └──────────┬──────────┘
                               │ [B, 64, 3072]
              ┌────────────────┴────────────────┐
              │                                 │
    ┌─────────┴──────────┐           ┌──────────┴─────────┐
    │ Skip Linear        │           │ Standardize        │
    │ 3072 → 1024        │           │ Linear 3072 → 3072 │
    └─────────┬──────────┘           │ ReLU²              │
              │                      │ Linear 3072 → 1024 │
              │                      │ Scale × alpha      │
              │                      └──────────┬─────────┘
              └────────────────┬────────────────┘
                               │ residual addition
                    ┌──────────┴──────────┐
                    │ Learned 8x8 2D      │
                    │ positions           │
                    └──────────┬──────────┘
                               │ [B, 64, 1024]
                    ┌──────────┴──────────┐
                    │ LFM2 Language Model │  frozen
                    └─────────────────────┘
```

| Component | Parameters | Training status |
|---|---:|---|
| Apple FastVLM-derived vision encoder | 125,073,504 (125.1M) | Frozen |
| Residual connector MLP and learned 8x8 2D positions | 15,750,145 (15.75M) | Trained: SFT, then OPD |
| Liquid LFM2 language model | 354,483,968 (354.5M) | Frozen |
| **Total** | **495,307,617 (495.31M)** | **15.75M trainable (3.18%)** |

## Maximum-throughput vLLM

For large offline captioning jobs, use the included split vLLM path. It compiles the vision encoder, processes 250 images together, and passes the resulting 64-token prompt embeddings to vLLM's native LFM2 decoder. No patched or custom vLLM installation is required.

```bash
pip install "vllm==0.25.1" "transformers==5.13.1" \
  huggingface_hub safetensors pillow numpy

hf download irotem98/FastLFM2.5-VL-0.5B \
  --local-dir FastLFM2.5-VL-0.5B

python FastLFM2.5-VL-0.5B/fast_vllm.py \
  --model FastLFM2.5-VL-0.5B \
  --image-dir ./images \
  --output captions.jsonl \
  --batch-size 250 \
  --max-tokens 128 \
  --gpu-memory-utilization 0.85
```

## Training

Training updated only the 15.75M-parameter connector path:

1. **SFT:** supervised alignment of the FastVLM visual features to the frozen LFM2 language model.
2. **OPD:** on-policy distillation from the LFM2.5-VL-450M teacher, refining the connector while keeping both foundation components frozen.

## Intended use and limitations

This checkpoint is optimized for fast, one-sentence English image descriptions at 512x512 resolution. It always emits an 8x8 grid of 64 visual tokens. Quality and throughput figures above are captioning results under the stated setup, not guarantees for other prompts, resolutions, GPUs, batch sizes, or generation lengths.

## License and attribution

This repository combines components derived from [Apple FastVLM-0.5B](https://huggingface.co/apple/FastVLM-0.5B) and [LiquidAI LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M). Review and preserve both upstream licenses and usage terms before redistribution. The bundled FastVLM implementation retains its upstream Apache-2.0 source header; this model card does not replace either upstream license.