| --- |
| license: mit |
| datasets: |
| - google-research-datasets/conceptual_captions |
| language: |
| - en |
| base_model: |
| - openai/clip-vit-base-patch32 |
| - openai-community/gpt2 |
| pipeline_tag: image-to-text |
| model-index: |
| - name: ClipCap (Conceptual Captions) |
| results: [] |
| --- |
| # ClipCap |
|
|
| This is an implementation of the [ClipCap](https://arxiv.org/abs/2111.09734) model — a captioning system that connects CLIP vision features to a GPT-2 language model via a learnable prefix. |
|
|
| The provided checkpoint (`coco_prefix_best_200k.pt`) was trained on **203,914 samples from the [Conceptual Captions](https://ai.google.com/research/ConceptualCaptions)** dataset using prefix tuning. |
|
|
| ## Model Architecture |
|
|
| - Vision Encoder: [CLIP](https://openai.com/research/clip) |
| - Language Model: GPT-2 (via Hugging Face Transformers) |
| - Connector: Multi-Layer Perceptron (MLP) to map CLIP embeddings to GPT-2 prefix tokens |
| - |
| ## Usage |
|
|
| To use this model, define the `ClipCapModel` architecture as described in the main.py file and load the checkpoint into your model instance. You’ll also need to obtain CLIP embeddings of the image as input. |
|
|
| Refer to the original [ClipCap repository](https://github.com/rmokady/CLIP_prefix_caption) for preprocessing and full inference pipeline details. |
|
|
| ## Reference |
|
|
| > Mokady, R., Hertz, A., & Bermano, A. H. (2021). *ClipCap: CLIP Prefix for Image Captioning*. arXiv preprint arXiv:2111.09734. |
