Image_Captioning_Transformers_Guide.md

Image Captioning with Transformers — Complete Learning Guide

---

1. What is Image Captioning?

Image Captioning is the task of automatically generating a natural language description for a given image. It bridges computer vision and natural language processing.

Example:

• Input: A photo of a dog catching a frisbee in a park
• Output: "A dog jumps to catch a flying disc in a grassy field."

---

2. Why Transformers for Image Captioning?

Traditional approaches used CNN + RNN (LSTM/GRU), but Transformers changed the game:

Aspect	CNN + RNN	Vision-Language Transformers
Long-range dependencies	Weak (vanishing gradients)	Strong (self-attention)
Parallelization	Sequential (slow)	Fully parallel (fast)
Pretraining	Limited	Massive (web-scale)
Transfer learning	Hard	Easy (one model, many tasks)

---

3. Core Architecture: Encoder-Decoder

The standard transformer-based image captioning model has two parts:

┌─────────────────┐      ┌──────────────────┐      ┌─────────────────┐
│   Input Image   │─────▶│  Vision Encoder  │─────▶│  Image Features │
│   (H × W × 3)   │      │  (ViT/Swin/ResNet│      │  (N × D)        │
└─────────────────┘      └──────────────────┘      └─────────────────┘
                                                            │
                                                            ▼
┌─────────────────┐      ┌──────────────────┐      ┌─────────────────┐
│ Generated Text  │◀─────│  Text Decoder    │◀─────│  Cross-Attention │
│  "A dog..."     │      │  (GPT/BERT-style)│      │  (Image ⟷ Text) │
└─────────────────┘      └──────────────────┘      └─────────────────┘

Key Components:

1. Vision Encoder: Extracts visual features from images

   • ViT (Vision Transformer): Patch-based, most common
   • Swin Transformer: Hierarchical, better for multi-scale
   • CLIP Vision Encoder: Pre-trained on image-text pairs

2. Text Decoder: Generates captions autoregressively

   • GPT-style: Autoregressive (most common for generation)
   • BERT-style: Masked (less common for captioning)

3. Cross-Attention: Connects vision and language

   • Decoder attends to image features when generating each word

---

4. Popular Models

4.1 BLIP (Bootstrapping Language-Image Pre-training)

• Paper: "BLIP: Bootstrapping Language-Image Pre-training" (Salesforce, 2022)
• Encoder: ViT
• Decoder: Transformer decoder (causal LM)
• Key Feature: Unified architecture for understanding + generation
• Strength: Strong zero-shot captioning, filter noisy web data
• Variants: BLIP (base), BLIP-2 (with Q-Former for frozen LLMs)

4.2 GIT (Generative Image-to-text Transformer)

• Paper: "GIT: A Generative Image-to-text Transformer for Vision and Language" (Microsoft, 2022)
• Architecture: Simple single-stream transformer (image patches as tokens)
• Strength: Simpler than BLIP, very strong performance
• Pre-training: Large-scale image-text pairs

4.3 BLIP-2

• Paper: "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models" (2023)
• Innovation: Q-Former bridges frozen vision encoder and frozen LLM
• LLM Backbones: OPT, Flan-T5
• Best for: When you want to leverage large LLMs without training them

4.4 ViT-GPT2 / Vision Encoder-Decoder (Hugging Face)

• Architecture: Any ViT encoder + any GPT decoder
• Easy to use: Hugging Face VisionEncoderDecoderModel
• Best for: Learning, fine-tuning on custom datasets

4.5 LLaVA, MiniGPT-4, InstructBLIP

• Type: Instruction-tuned multimodal models
• Best for: Conversational image understanding, not just captioning

---

5. Hands-On: Using Pre-trained Models (Hugging Face)

5.1 Quick Start with BLIP

from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import requests

# Load model and processor
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")

# Load image
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco.png"
image = Image.open(requests.get(url, stream=True).raw).convert('RGB')

# Generate caption
inputs = processor(image, return_tensors="pt")
out = model.generate(**inputs)
caption = processor.decode(out[0], skip_special_tokens=True)

print(f"Caption: {caption}")
# Output: "a soccer game with a player in yellow and white uniforms"

5.2 Using BLIP-2 (More Powerful)

from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained(
    "Salesforce/blip-image-captioning-large"
).to(device)

image = Image.open("your_image.jpg").convert("RGB")

# Conditional generation (start with a prompt)
text = "a photography of"
inputs = processor(image, text, return_tensors="pt").to(device)

out = model.generate(**inputs, max_new_tokens=50)
caption = processor.decode(out[0], skip_special_tokens=True)
print(caption)

5.3 Using VisionEncoderDecoder (Flexible)

from transformers import VisionEncoderDecoderModel, ViTImageProcessor, AutoTokenizer
from PIL import Image

# Load a ViT-GPT2 model
model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
feature_extractor = ViTImageProcessor.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
tokenizer = AutoTokenizer.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

image = Image.open("image.jpg").convert("RGB")
pixel_values = feature_extractor(images=image, return_tensors="pt").pixel_values

generated_ids = model.generate(pixel_values, max_length=50)
generated_caption = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_caption)

---

6. Fine-Tuning on Your Own Dataset

6.1 Dataset Preparation (COCO-style)

from datasets import load_dataset
from torch.utils.data import Dataset
from PIL import Image

# Example: COCO Captions dataset
dataset = load_dataset("yerevann/coco-karpathy", "default")

class ImageCaptioningDataset(Dataset):
    def __init__(self, images, captions, processor):
        self.images = images
        self.captions = captions
        self.processor = processor

    def __len__(self):
        return len(self.images)

    def __getitem__(self, idx):
        image = self.images[idx]
        caption = self.captions[idx]
        
        # Process image and text
        encoding = self.processor(
            images=image,
            text=caption,
            padding="max_length",
            return_tensors="pt"
        )
        
        # Remove batch dimension added by processor
        encoding = {k: v.squeeze(0) for k, v in encoding.items()}
        return encoding

6.2 Training Loop

from transformers import BlipForConditionalGeneration, BlipProcessor
from torch.utils.data import DataLoader
import torch
from tqdm import tqdm

# Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
model.to(device)

# Create dataloader (assuming you have images and captions)
# train_dataset = ImageCaptioningDataset(images, captions, processor)
# train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

model.train()
for epoch in range(3):
    for batch in tqdm(train_dataloader):
        input_ids = batch["input_ids"].to(device)
        pixel_values = batch["pixel_values"].to(device)
        attention_mask = batch["attention_mask"].to(device)
        
        outputs = model(
            input_ids=input_ids,
            pixel_values=pixel_values,
            attention_mask=attention_mask,
            labels=input_ids
        )
        
        loss = outputs.loss
        loss.backward()
        
        optimizer.step()
        optimizer.zero_grad()
        
    print(f"Epoch {epoch} | Loss: {loss.item():.4f}")

# Save model
model.save_pretrained("./my-captioning-model")
processor.save_pretrained("./my-captioning-model")

---

7. Key Datasets for Image Captioning

Dataset	Size	Images	Captions/Image	Domain
COCO Captions	~120K	120K	5	General
Flickr30K	30K	30K	5	General
Flickr8K	8K	8K	5	General (small)
Conceptual Captions (CC3M/CC12M)	3M/12M	-	1	Web-scraped
LAION-400M	400M	-	1	Web-scale
TextCaps	28K	-	1	Text in images
** nocaps**	15K	-	10	Novel objects

Recommended for beginners: COCO Captions or Flickr8K (small, manageable)

Recommended for pre-training: Conceptual Captions (CC12M) or LAION

---

8. Evaluation Metrics

Metric	Description	Range	Good Score
BLEU-4	N-gram precision	0-1	>0.35
METEOR	Synonym/paraphrase aware	0-1	>0.28
ROUGE-L	Longest common subsequence	0-1	>0.55
CIDEr	TF-IDF weighted n-grams	0-10	>1.0
SPICE	Scene graph matching	0-1	>0.20

from evaluate import load

# Using Hugging Face evaluate library
bleu = load("bleu")
meteor = load("meteor")
rouge = load("rouge")

predictions = ["a dog plays with a frisbee"]
references = [["a dog is catching a frisbee in the park"]]

results = bleu.compute(predictions=predictions, references=references)
print(results)

---

9. Advanced Topics

9.1 Beam Search vs. Nucleus Sampling

# Beam Search (more deterministic, higher quality)
out = model.generate(
    **inputs,
    num_beams=5,
    max_length=50,
    early_stopping=True
)

# Nucleus Sampling (more diverse, creative)
out = model.generate(
    **inputs,
    do_sample=True,
    top_p=0.9,
    temperature=0.7,
    max_length=50
)

9.2 Multi-GPU Training (DistributedDataParallel)

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

# Initialize process group
dist.init_process_group("nccl")
model = DDP(model, device_ids=[local_rank])
# ... training loop ...

9.3 Quantization for Inference (Faster, Smaller)

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = BlipForConditionalGeneration.from_pretrained(
    "Salesforce/blip-image-captioning-large",
    quantization_config=bnb_config
)

---

10. Architecture Deep Dive: How Cross-Attention Works

import torch
import torch.nn as nn
import math

class CrossAttention(nn.Module):
    """
    Cross-attention: Text queries attend to Image keys/values
    """
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.d_head = d_model // num_heads
        
        self.q_proj = nn.Linear(d_model, d_model)  # From text
        self.k_proj = nn.Linear(d_model, d_model)  # From image
        self.v_proj = nn.Linear(d_model, d_model)  # From image
        self.out_proj = nn.Linear(d_model, d_model)
    
    def forward(self, text_hidden, image_features, text_mask=None):
        batch_size = text_hidden.size(0)
        
        # Project
        Q = self.q_proj(text_hidden)    # (B, T, D)
        K = self.k_proj(image_features)  # (B, N, D)
        V = self.v_proj(image_features)  # (B, N, D)
        
        # Reshape for multi-head
        Q = Q.view(batch_size, -1, self.num_heads, self.d_head).transpose(1, 2)
        K = K.view(batch_size, -1, self.num_heads, self.d_head).transpose(1, 2)
        V = V.view(batch_size, -1, self.num_heads, self.d_head).transpose(1, 2)
        
        # Attention scores
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_head)
        
        if text_mask is not None:
            scores = scores.masked_fill(text_mask.unsqueeze(1).unsqueeze(1) == 0, float('-inf'))
        
        attn = torch.softmax(scores, dim=-1)
        context = torch.matmul(attn, V)
        
        # Concatenate heads
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
        return self.out_proj(context)

---

11. Project Ideas for Practice

1. Instagram Caption Generator — Fine-tune on social media captions
2. Medical Image Captioning — Train on radiology reports + X-rays
3. News Image Captioning — Generate journalistic captions
4. Art Description — Describe paintings/artwork in detail
5. E-commerce Product Description — Generate product descriptions from images
6. Accessibility Tool — Screen reader for visually impaired users
7. Meme Caption Generator — Understand humor + image context

---

12. Essential Papers to Read

Paper	Authors	Year	Why Read
"Attention Is All You Need"	Vaswani et al.	2017	Foundation of Transformers
"An Image is Worth 16x16 Words"	Dosovitskiy et al.	2020	ViT - Vision Transformer
"BLIP"	Li et al. (Salesforce)	2022	Best unified V+L model
"BLIP-2"	Li et al. (Salesforce)	2023	Bridging vision and LLMs
"GIT"	Wang et al. (Microsoft)	2022	Simple and strong baseline
"Show, Attend and Tell"	Xu et al.	2015	CNN+RNN classic (historical)
"CLIP"	Radford et al. (OpenAI)	2021	Contrastive pretraining

---

13. Quick Reference: Hugging Face Model Hub

Model	Path	Size	Best For
BLIP Base	Salesforce/blip-image-captioning-base	~400M	Fast inference
BLIP Large	Salesforce/blip-image-captioning-large	~1B	Better quality
ViT-GPT2	nlpconnect/vit-gpt2-image-captioning	~300M	Learning/fine-tuning
BLIP-2 OPT-2.7B	Salesforce/blip2-opt-2.7b	~2.7B	Strong captions
BLIP-2 Flan-T5-XL	Salesforce/blip2-flan-t5-xl	~3B	Instruction following
GIT Base	microsoft/git-base-coco	~300M	COCO fine-tuned
GIT Large	microsoft/git-large-coco	~800M	Best quality

---

14. Common Pitfalls & Tips

⚠️ Pitfalls:

1. Forgetting to resize images — Models expect specific sizes (e.g., 224x224 for ViT)
2. Not handling special tokens — <pad>, <eos>, <unk> must be properly managed
3. Evaluating on training data — Always split train/val/test properly
4. Ignoring CIDEr/SPICE — BLEU alone is misleading for caption quality
5. Not using mixed precision — Training without fp16 is 2-3x slower

✅ Tips:

1. Start with pre-trained models — Don't train from scratch initially
2. Use gradient checkpointing — Trade compute for memory on large models
3. Data augmentation — Random crops, flips, color jitter help generalization
4. Label smoothing — Improves generation diversity
5. Ensemble decoding — Average multiple model outputs for best results

---

15. Full Example: End-to-End Pipeline

"""
Complete image captioning pipeline using BLIP
"""
import torch
from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import os

class ImageCaptioner:
    def __init__(self, model_name="Salesforce/blip-image-captioning-base", device=None):
        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
        self.processor = BlipProcessor.from_pretrained(model_name)
        self.model = BlipForConditionalGeneration.from_pretrained(model_name).to(self.device)
        self.model.eval()
    
    def caption(self, image_path, num_captions=1, max_length=50):
        """Generate caption(s) for an image."""
        image = Image.open(image_path).convert("RGB")
        inputs = self.processor(image, return_tensors="pt").to(self.device)
        
        with torch.no_grad():
            if num_captions == 1:
                outputs = self.model.generate(
                    **inputs,
                    max_length=max_length,
                    num_beams=5,
                    early_stopping=True
                )
            else:
                outputs = self.model.generate(
                    **inputs,
                    max_length=max_length,
                    num_return_sequences=num_captions,
                    num_beams=num_captions * 2,
                    do_sample=True,
                    temperature=0.8,
                    top_p=0.9
                )
        
        captions = self.processor.batch_decode(outputs, skip_special_tokens=True)
        return captions[0] if num_captions == 1 else captions
    
    def caption_folder(self, folder_path, output_file="captions.txt"):
        """Caption all images in a folder."""
        results = []
        for fname in os.listdir(folder_path):
            if fname.lower().endswith(('.png', '.jpg', '.jpeg')):
                path = os.path.join(folder_path, fname)
                caption = self.caption(path)
                results.append(f"{fname}: {caption}")
                print(f"{fname} -> {caption}")
        
        with open(output_file, "w") as f:
            f.write("\n".join(results))
        return results

# Usage
if __name__ == "__main__":
    captioner = ImageCaptioner()
    caption = captioner.caption("photo.jpg")
    print(caption)
    
    # Multiple diverse captions
    captions = captioner.caption("photo.jpg", num_captions=3)
    for i, cap in enumerate(captions, 1):
        print(f"{i}. {cap}")

---

16. Learning Roadmap

Week 1: Foundations

• Read "Attention Is All You Need"
• Understand ViT architecture
• Run pre-trained BLIP on sample images
• Explore the Hugging Face model hub

Week 2: Hands-On

• Load COCO dataset
• Fine-tune BLIP-base on a small subset
• Evaluate with BLEU/ROUGE metrics
• Experiment with beam search vs. sampling

Week 3: Advanced

• Implement custom dataset loader
• Train on full COCO/Flickr30K
• Try different model combinations (ViT + GPT2, etc.)
• Implement gradient checkpointing for larger models

Week 4: Production

• Optimize inference (ONNX/TensorRT)
• Build a simple API (FastAPI/Flask)
• Deploy with Docker
• Add batch processing support

---

Resources

• Hugging Face Transformers Docs: https://huggingface.co/docs/transformers
• BLIP GitHub: https://github.com/salesforce/BLIP
• COCO Dataset: https://cocodataset.org/
• Papers with Code: https://paperswithcode.com/task/image-captioning
• Course: Stanford CS231n (CNNs), CS224N (NLP)

---

Happy Learning! Start with the pre-trained models and work your way up to training from scratch.