# Image Captioning with Transformers — Complete Learning Guide

---

## 1. What is Image Captioning?

**Image Captioning** is the task of automatically generating a natural language description for a given image. It bridges **computer vision** and **natural language processing**.

**Example:**
- **Input:** A photo of a dog catching a frisbee in a park
- **Output:** *"A dog jumps to catch a flying disc in a grassy field."*

---

## 2. Why Transformers for Image Captioning?

Traditional approaches used CNN + RNN (LSTM/GRU), but Transformers changed the game:

| Aspect | CNN + RNN | Vision-Language Transformers |
|--------|-----------|------------------------------|
| Long-range dependencies | Weak (vanishing gradients) | Strong (self-attention) |
| Parallelization | Sequential (slow) | Fully parallel (fast) |
| Pretraining | Limited | Massive (web-scale) |
| Transfer learning | Hard | Easy (one model, many tasks) |

---

## 3. Core Architecture: Encoder-Decoder

The standard transformer-based image captioning model has two parts:

```
┌─────────────────┐      ┌──────────────────┐      ┌─────────────────┐
│   Input Image   │─────▶│  Vision Encoder  │─────▶│  Image Features │
│   (H × W × 3)   │      │  (ViT/Swin/ResNet│      │  (N × D)        │
└─────────────────┘      └──────────────────┘      └─────────────────┘
                                                            │
                                                            ▼
┌─────────────────┐      ┌──────────────────┐      ┌─────────────────┐
│ Generated Text  │◀─────│  Text Decoder    │◀─────│  Cross-Attention │
│  "A dog..."     │      │  (GPT/BERT-style)│      │  (Image ⟷ Text) │
└─────────────────┘      └──────────────────┘      └─────────────────┘
```

### Key Components:

1. **Vision Encoder**: Extracts visual features from images
   - **ViT** (Vision Transformer): Patch-based, most common
   - **Swin Transformer**: Hierarchical, better for multi-scale
   - **CLIP Vision Encoder**: Pre-trained on image-text pairs

2. **Text Decoder**: Generates captions autoregressively
   - **GPT-style**: Autoregressive (most common for generation)
   - **BERT-style**: Masked (less common for captioning)

3. **Cross-Attention**: Connects vision and language
   - Decoder attends to image features when generating each word

---

## 4. Popular Models

### 4.1 BLIP (Bootstrapping Language-Image Pre-training)
- **Paper**: "BLIP: Bootstrapping Language-Image Pre-training" (Salesforce, 2022)
- **Encoder**: ViT
- **Decoder**: Transformer decoder (causal LM)
- **Key Feature**: Unified architecture for understanding + generation
- **Strength**: Strong zero-shot captioning, filter noisy web data
- **Variants**: BLIP (base), BLIP-2 (with Q-Former for frozen LLMs)

### 4.2 GIT (Generative Image-to-text Transformer)
- **Paper**: "GIT: A Generative Image-to-text Transformer for Vision and Language" (Microsoft, 2022)
- **Architecture**: Simple single-stream transformer (image patches as tokens)
- **Strength**: Simpler than BLIP, very strong performance
- **Pre-training**: Large-scale image-text pairs

### 4.3 BLIP-2
- **Paper**: "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models" (2023)
- **Innovation**: Q-Former bridges frozen vision encoder and frozen LLM
- **LLM Backbones**: OPT, Flan-T5
- **Best for**: When you want to leverage large LLMs without training them

### 4.4 ViT-GPT2 / Vision Encoder-Decoder (Hugging Face)
- **Architecture**: Any ViT encoder + any GPT decoder
- **Easy to use**: Hugging Face `VisionEncoderDecoderModel`
- **Best for**: Learning, fine-tuning on custom datasets

### 4.5 LLaVA, MiniGPT-4, InstructBLIP
- **Type**: Instruction-tuned multimodal models
- **Best for**: Conversational image understanding, not just captioning

---

## 5. Hands-On: Using Pre-trained Models (Hugging Face)

### 5.1 Quick Start with BLIP

```python
from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import requests

# Load model and processor
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")

# Load image
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco.png"
image = Image.open(requests.get(url, stream=True).raw).convert('RGB')

# Generate caption
inputs = processor(image, return_tensors="pt")
out = model.generate(**inputs)
caption = processor.decode(out[0], skip_special_tokens=True)

print(f"Caption: {caption}")
# Output: "a soccer game with a player in yellow and white uniforms"
```

### 5.2 Using BLIP-2 (More Powerful)

```python
from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained(
    "Salesforce/blip-image-captioning-large"
).to(device)

image = Image.open("your_image.jpg").convert("RGB")

# Conditional generation (start with a prompt)
text = "a photography of"
inputs = processor(image, text, return_tensors="pt").to(device)

out = model.generate(**inputs, max_new_tokens=50)
caption = processor.decode(out[0], skip_special_tokens=True)
print(caption)
```

### 5.3 Using VisionEncoderDecoder (Flexible)

```python
from transformers import VisionEncoderDecoderModel, ViTImageProcessor, AutoTokenizer
from PIL import Image

# Load a ViT-GPT2 model
model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
feature_extractor = ViTImageProcessor.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
tokenizer = AutoTokenizer.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

image = Image.open("image.jpg").convert("RGB")
pixel_values = feature_extractor(images=image, return_tensors="pt").pixel_values

generated_ids = model.generate(pixel_values, max_length=50)
generated_caption = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_caption)
```

---

## 6. Fine-Tuning on Your Own Dataset

### 6.1 Dataset Preparation (COCO-style)

```python
from datasets import load_dataset
from torch.utils.data import Dataset
from PIL import Image

# Example: COCO Captions dataset
dataset = load_dataset("yerevann/coco-karpathy", "default")

class ImageCaptioningDataset(Dataset):
    def __init__(self, images, captions, processor):
        self.images = images
        self.captions = captions
        self.processor = processor

    def __len__(self):
        return len(self.images)

    def __getitem__(self, idx):
        image = self.images[idx]
        caption = self.captions[idx]
        
        # Process image and text
        encoding = self.processor(
            images=image,
            text=caption,
            padding="max_length",
            return_tensors="pt"
        )
        
        # Remove batch dimension added by processor
        encoding = {k: v.squeeze(0) for k, v in encoding.items()}
        return encoding
```

### 6.2 Training Loop

```python
from transformers import BlipForConditionalGeneration, BlipProcessor
from torch.utils.data import DataLoader
import torch
from tqdm import tqdm

# Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
model.to(device)

# Create dataloader (assuming you have images and captions)
# train_dataset = ImageCaptioningDataset(images, captions, processor)
# train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

model.train()
for epoch in range(3):
    for batch in tqdm(train_dataloader):
        input_ids = batch["input_ids"].to(device)
        pixel_values = batch["pixel_values"].to(device)
        attention_mask = batch["attention_mask"].to(device)
        
        outputs = model(
            input_ids=input_ids,
            pixel_values=pixel_values,
            attention_mask=attention_mask,
            labels=input_ids
        )
        
        loss = outputs.loss
        loss.backward()
        
        optimizer.step()
        optimizer.zero_grad()
        
    print(f"Epoch {epoch} | Loss: {loss.item():.4f}")

# Save model
model.save_pretrained("./my-captioning-model")
processor.save_pretrained("./my-captioning-model")
```

---

## 7. Key Datasets for Image Captioning

| Dataset | Size | Images | Captions/Image | Domain |
|---------|------|--------|----------------|--------|
| **COCO Captions** | ~120K | 120K | 5 | General |
| **Flickr30K** | 30K | 30K | 5 | General |
| **Flickr8K** | 8K | 8K | 5 | General (small) |
| **Conceptual Captions (CC3M/CC12M)** | 3M/12M | - | 1 | Web-scraped |
| **LAION-400M** | 400M | - | 1 | Web-scale |
| **TextCaps** | 28K | - | 1 | Text in images |
| ** nocaps** | 15K | - | 10 | Novel objects |

**Recommended for beginners:** COCO Captions or Flickr8K (small, manageable)

**Recommended for pre-training:** Conceptual Captions (CC12M) or LAION

---

## 8. Evaluation Metrics

| Metric | Description | Range | Good Score |
|--------|-------------|-------|------------|
| **BLEU-4** | N-gram precision | 0-1 | >0.35 |
| **METEOR** | Synonym/paraphrase aware | 0-1 | >0.28 |
| **ROUGE-L** | Longest common subsequence | 0-1 | >0.55 |
| **CIDEr** | TF-IDF weighted n-grams | 0-10 | >1.0 |
| **SPICE** | Scene graph matching | 0-1 | >0.20 |

```python
from evaluate import load

# Using Hugging Face evaluate library
bleu = load("bleu")
meteor = load("meteor")
rouge = load("rouge")

predictions = ["a dog plays with a frisbee"]
references = [["a dog is catching a frisbee in the park"]]

results = bleu.compute(predictions=predictions, references=references)
print(results)
```

---

## 9. Advanced Topics

### 9.1 Beam Search vs. Nucleus Sampling

```python
# Beam Search (more deterministic, higher quality)
out = model.generate(
    **inputs,
    num_beams=5,
    max_length=50,
    early_stopping=True
)

# Nucleus Sampling (more diverse, creative)
out = model.generate(
    **inputs,
    do_sample=True,
    top_p=0.9,
    temperature=0.7,
    max_length=50
)
```

### 9.2 Multi-GPU Training (DistributedDataParallel)

```python
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

# Initialize process group
dist.init_process_group("nccl")
model = DDP(model, device_ids=[local_rank])
# ... training loop ...
```

### 9.3 Quantization for Inference (Faster, Smaller)

```python
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = BlipForConditionalGeneration.from_pretrained(
    "Salesforce/blip-image-captioning-large",
    quantization_config=bnb_config
)
```

---

## 10. Architecture Deep Dive: How Cross-Attention Works

```python
import torch
import torch.nn as nn
import math

class CrossAttention(nn.Module):
    """
    Cross-attention: Text queries attend to Image keys/values
    """
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.d_head = d_model // num_heads
        
        self.q_proj = nn.Linear(d_model, d_model)  # From text
        self.k_proj = nn.Linear(d_model, d_model)  # From image
        self.v_proj = nn.Linear(d_model, d_model)  # From image
        self.out_proj = nn.Linear(d_model, d_model)
    
    def forward(self, text_hidden, image_features, text_mask=None):
        batch_size = text_hidden.size(0)
        
        # Project
        Q = self.q_proj(text_hidden)    # (B, T, D)
        K = self.k_proj(image_features)  # (B, N, D)
        V = self.v_proj(image_features)  # (B, N, D)
        
        # Reshape for multi-head
        Q = Q.view(batch_size, -1, self.num_heads, self.d_head).transpose(1, 2)
        K = K.view(batch_size, -1, self.num_heads, self.d_head).transpose(1, 2)
        V = V.view(batch_size, -1, self.num_heads, self.d_head).transpose(1, 2)
        
        # Attention scores
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_head)
        
        if text_mask is not None:
            scores = scores.masked_fill(text_mask.unsqueeze(1).unsqueeze(1) == 0, float('-inf'))
        
        attn = torch.softmax(scores, dim=-1)
        context = torch.matmul(attn, V)
        
        # Concatenate heads
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
        return self.out_proj(context)
```

---

## 11. Project Ideas for Practice

1. **Instagram Caption Generator** — Fine-tune on social media captions
2. **Medical Image Captioning** — Train on radiology reports + X-rays
3. **News Image Captioning** — Generate journalistic captions
4. **Art Description** — Describe paintings/artwork in detail
5. **E-commerce Product Description** — Generate product descriptions from images
6. **Accessibility Tool** — Screen reader for visually impaired users
7. **Meme Caption Generator** — Understand humor + image context

---

## 12. Essential Papers to Read

| Paper | Authors | Year | Why Read |
|-------|---------|------|----------|
| **"Attention Is All You Need"** | Vaswani et al. | 2017 | Foundation of Transformers |
| **"An Image is Worth 16x16 Words"** | Dosovitskiy et al. | 2020 | ViT - Vision Transformer |
| **"BLIP"** | Li et al. (Salesforce) | 2022 | Best unified V+L model |
| **"BLIP-2"** | Li et al. (Salesforce) | 2023 | Bridging vision and LLMs |
| **"GIT"** | Wang et al. (Microsoft) | 2022 | Simple and strong baseline |
| **"Show, Attend and Tell"** | Xu et al. | 2015 | CNN+RNN classic (historical) |
| **"CLIP"** | Radford et al. (OpenAI) | 2021 | Contrastive pretraining |

---

## 13. Quick Reference: Hugging Face Model Hub

| Model | Path | Size | Best For |
|-------|------|------|----------|
| BLIP Base | `Salesforce/blip-image-captioning-base` | ~400M | Fast inference |
| BLIP Large | `Salesforce/blip-image-captioning-large` | ~1B | Better quality |
| ViT-GPT2 | `nlpconnect/vit-gpt2-image-captioning` | ~300M | Learning/fine-tuning |
| BLIP-2 OPT-2.7B | `Salesforce/blip2-opt-2.7b` | ~2.7B | Strong captions |
| BLIP-2 Flan-T5-XL | `Salesforce/blip2-flan-t5-xl` | ~3B | Instruction following |
| GIT Base | `microsoft/git-base-coco` | ~300M | COCO fine-tuned |
| GIT Large | `microsoft/git-large-coco` | ~800M | Best quality |

---

## 14. Common Pitfalls & Tips

### ⚠️ Pitfalls:
1. **Forgetting to resize images** — Models expect specific sizes (e.g., 224x224 for ViT)
2. **Not handling special tokens** — `<pad>`, `<eos>`, `<unk>` must be properly managed
3. **Evaluating on training data** — Always split train/val/test properly
4. **Ignoring CIDEr/SPICE** — BLEU alone is misleading for caption quality
5. **Not using mixed precision** — Training without `fp16` is 2-3x slower

### ✅ Tips:
1. **Start with pre-trained models** — Don't train from scratch initially
2. **Use gradient checkpointing** — Trade compute for memory on large models
3. **Data augmentation** — Random crops, flips, color jitter help generalization
4. **Label smoothing** — Improves generation diversity
5. **Ensemble decoding** — Average multiple model outputs for best results

---

## 15. Full Example: End-to-End Pipeline

```python
"""
Complete image captioning pipeline using BLIP
"""
import torch
from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import os

class ImageCaptioner:
    def __init__(self, model_name="Salesforce/blip-image-captioning-base", device=None):
        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
        self.processor = BlipProcessor.from_pretrained(model_name)
        self.model = BlipForConditionalGeneration.from_pretrained(model_name).to(self.device)
        self.model.eval()
    
    def caption(self, image_path, num_captions=1, max_length=50):
        """Generate caption(s) for an image."""
        image = Image.open(image_path).convert("RGB")
        inputs = self.processor(image, return_tensors="pt").to(self.device)
        
        with torch.no_grad():
            if num_captions == 1:
                outputs = self.model.generate(
                    **inputs,
                    max_length=max_length,
                    num_beams=5,
                    early_stopping=True
                )
            else:
                outputs = self.model.generate(
                    **inputs,
                    max_length=max_length,
                    num_return_sequences=num_captions,
                    num_beams=num_captions * 2,
                    do_sample=True,
                    temperature=0.8,
                    top_p=0.9
                )
        
        captions = self.processor.batch_decode(outputs, skip_special_tokens=True)
        return captions[0] if num_captions == 1 else captions
    
    def caption_folder(self, folder_path, output_file="captions.txt"):
        """Caption all images in a folder."""
        results = []
        for fname in os.listdir(folder_path):
            if fname.lower().endswith(('.png', '.jpg', '.jpeg')):
                path = os.path.join(folder_path, fname)
                caption = self.caption(path)
                results.append(f"{fname}: {caption}")
                print(f"{fname} -> {caption}")
        
        with open(output_file, "w") as f:
            f.write("\n".join(results))
        return results

# Usage
if __name__ == "__main__":
    captioner = ImageCaptioner()
    caption = captioner.caption("photo.jpg")
    print(caption)
    
    # Multiple diverse captions
    captions = captioner.caption("photo.jpg", num_captions=3)
    for i, cap in enumerate(captions, 1):
        print(f"{i}. {cap}")
```

---

## 16. Learning Roadmap

### Week 1: Foundations
- [ ] Read "Attention Is All You Need"
- [ ] Understand ViT architecture
- [ ] Run pre-trained BLIP on sample images
- [ ] Explore the Hugging Face model hub

### Week 2: Hands-On
- [ ] Load COCO dataset
- [ ] Fine-tune BLIP-base on a small subset
- [ ] Evaluate with BLEU/ROUGE metrics
- [ ] Experiment with beam search vs. sampling

### Week 3: Advanced
- [ ] Implement custom dataset loader
- [ ] Train on full COCO/Flickr30K
- [ ] Try different model combinations (ViT + GPT2, etc.)
- [ ] Implement gradient checkpointing for larger models

### Week 4: Production
- [ ] Optimize inference (ONNX/TensorRT)
- [ ] Build a simple API (FastAPI/Flask)
- [ ] Deploy with Docker
- [ ] Add batch processing support

---

## Resources

- **Hugging Face Transformers Docs**: https://huggingface.co/docs/transformers
- **BLIP GitHub**: https://github.com/salesforce/BLIP
- **COCO Dataset**: https://cocodataset.org/
- **Papers with Code**: https://paperswithcode.com/task/image-captioning
- **Course**: Stanford CS231n (CNNs), CS224N (NLP)

---

*Happy Learning! Start with the pre-trained models and work your way up to training from scratch.*