metadata
license: mit
datasets:
- google-research-datasets/conceptual_captions
language:
- en
base_model:
- openai/clip-vit-base-patch32
- openai-community/gpt2
pipeline_tag: image-to-text
model-index:
- name: ClipCap (Conceptual Captions)
results: []
ClipCap
This is an implementation of the ClipCap model — a captioning system that connects CLIP vision features to a GPT-2 language model via a learnable prefix.
The provided checkpoint (coco_prefix_best_200k.pt) was trained on 203,914 samples from the Conceptual Captions dataset using prefix tuning.
Model Architecture
- Vision Encoder: CLIP
- Language Model: GPT-2 (via Hugging Face Transformers)
- Connector: Multi-Layer Perceptron (MLP) to map CLIP embeddings to GPT-2 prefix tokens
Usage
To use this model, define the ClipCapModel architecture as described in the main.py file and load the checkpoint into your model instance. You’ll also need to obtain CLIP embeddings of the image as input.
Refer to the original ClipCap repository for preprocessing and full inference pipeline details.
Reference
Mokady, R., Hertz, A., & Bermano, A. H. (2021). ClipCap: CLIP Prefix for Image Captioning. arXiv preprint arXiv:2111.09734.