---
license: mit
datasets:
- google-research-datasets/conceptual_captions
language:
- en
base_model:
- openai/clip-vit-base-patch32
- openai-community/gpt2
pipeline_tag: image-to-text
model-index:
  - name: ClipCap (Conceptual Captions)
    results: []
---
# ClipCap

This is an implementation of the [ClipCap](https://arxiv.org/abs/2111.09734) model — a captioning system that connects CLIP vision features to a GPT-2 language model via a learnable prefix.

The provided checkpoint (`coco_prefix_best_200k.pt`) was trained on **203,914 samples from the [Conceptual Captions](https://ai.google.com/research/ConceptualCaptions)** dataset using prefix tuning.

## Model Architecture

- Vision Encoder: [CLIP](https://openai.com/research/clip)
- Language Model: GPT-2 (via Hugging Face Transformers)
- Connector: Multi-Layer Perceptron (MLP) to map CLIP embeddings to GPT-2 prefix tokens
- 
## Usage

To use this model, define the `ClipCapModel` architecture as described in the main.py file and load the checkpoint into your model instance. You’ll also need to obtain CLIP embeddings of the image as input.

Refer to the original [ClipCap repository](https://github.com/rmokady/CLIP_prefix_caption) for preprocessing and full inference pipeline details.

## Reference

> Mokady, R., Hertz, A., & Bermano, A. H. (2021). *ClipCap: CLIP Prefix for Image Captioning*. arXiv preprint arXiv:2111.09734.