README.md

---
library_name: keras-hub
pipeline_tag: text-generation
---
### Model Overview
## Model Summary

BLIP-2 (Bootstrapping Language-Image Pre-training) is a generic and efficient pre-training strategy that bridges the modality gap between frozen image encoders and frozen large language models (LLMs). It was introduced by Salesforce in the paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models".

The architecture consists of three main components: a frozen image encoder (EVA-CLIP ViT-g/14), a lightweight querying transformer (Q-Former) that acts as an information bottleneck to extract the most relevant visual features, and a frozen LLM (like OPT or Flan-T5) that handles the text generation. Because the heavy vision and language models are kept frozen during pre-training, BLIP-2 achieves state-of-the-art performance on various vision-language tasks with significantly fewer trainable parameters than existing methods.

Key Features:
- State-of-the-art vision-language pre-training method
- Combines frozen EVA-CLIP vision encoder with frozen LLMs (OPT, Flan-T5)
- Highly efficient pre-training via the lightweight Q-Former
- Capable of visual question answering, image captioning, and conversational image understanding

## Model Details

- Model Family: BLIP-2
- Architecture: Vision-Language Model (EVA-CLIP ViT + Q-Former + LLM)
- Developer: Salesforce
- Paper: BLIP-2: [Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597)
- Source: [Salesforce BLIP-2 Collection](https://huggingface.co/collections/Salesforce/blip2-models)
- Task Type: Image-to-Text / Vision-Language Generation
- Max Sequence Length: Varies by LLM (typically 512+ tokens)

## Architecture Details (BLIP-2 General)

- Vision Encoder: EVA-CLIP ViT-g/14 (Frozen)
- Vision Encoder Parameters: ~1 Billion
- Q-Former: Querying Transformer (Trainable)
- Q-Former Parameters: ~188 Million
- Language Model: Varies (OPT-2.7B, OPT-6.7B, Flan-T5-XL, Flan-T5-XXL) (Frozen)
- Total Parameters: ~3B to ~12B (depending on the LLM variant)

* [BLIP2 Quickstart Notebook](coming soon..!)
* [BLIP2 API Documentation](coming soon..!)
* [BLIP2 Model Card](https://huggingface.co/collections/Salesforce/blip2-models)
* [BLIP2 Technical Paper](https://arxiv.org/abs/2301.12597)
* [KerasHub Beginner Guide](https://keras.io/guides/keras_hub/getting_started/)
* [KerasHub Model Publishing Guide](https://keras.io/guides/keras_hub/upload/)

## Installation

Keras and KerasHub can be installed with:

```
pip install -U -q keras-hub
pip install -U -q keras

```

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the [Keras Getting Started](https://keras.io/getting_started/) page.


## Preset Table

| Preset | Architecture | Vision Encoder | Language Model | Description |
|---|---|---|---|---|
| `blip2_opt_2.7b` | BLIP-2 | EVA-CLIP ViT-g/14 | OPT-2.7B | BLIP-2 model using OPT-2.7B as the frozen language model. |
| `blip2_opt_6.7b` | BLIP-2 | EVA-CLIP ViT-g/14 | OPT-6.7B | BLIP-2 model using OPT-6.7B as the frozen language model. |
| `blip2_flan_t5_xl` | BLIP-2 | EVA-CLIP ViT-g/14 | Flan-T5-XL | BLIP-2 model using Flan-T5-XL (~3B) as the frozen language model. |
| `blip2_flan_t5_xxl` | BLIP-2 | EVA-CLIP ViT-g/14 | Flan-T5-XXL | BLIP-2 model using Flan-T5-XXL (~11B) as the frozen language model. |

## Example Usage
BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either `BLIP2CausalLM` (for OPT models) or `BLIP2Seq2SeqLM` (for Flan-T5 models).

The `BLIP2Seq2SeqLM` class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the `"encoder_text"` key.

```python
import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("blip2_opt_2.7b")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))
```

## Example Usage with Hugging Face URI

BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either `BLIP2CausalLM` (for OPT models) or `BLIP2Seq2SeqLM` (for Flan-T5 models).

The `BLIP2Seq2SeqLM` class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the `"encoder_text"` key.

```python
import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("hf://keras/blip2_opt_2.7b")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))
```