tahamajs/Deep_UT/CA7_Advanced_Topics/Image_Captioning - Storage Bucket

Advanced Image Captioning with Attention Mechanisms

Course: Neural Networks and Deep Learning
Assignment: CAe - Question 2
Student: Taha Majlesi (810101504)
Institution: University of Tehran, Faculty of Electrical and Computer Engineering

---

📋 Table of Contents

1. Abstract
2. Introduction
3. Theoretical Framework
4. Dataset Preparation
5. Model Architecture
6. Training Process
7. Advanced Techniques
8. Results and Evaluation
9. Conclusion
10. Files and Resources

---

Abstract

This project presents a comprehensive implementation of an advanced image captioning system using deep learning techniques. The system employs an encoder-decoder architecture with attention mechanisms to generate natural language descriptions for images in Persian (Farsi). We explore and compare three key attention mechanisms:

1. Traditional Attention (Additive Attention)
2. Scheduled Sampling with Teacher Forcing decay
3. Scaled Dot-Product Attention

Each method's performance is evaluated and analyzed to understand their strengths and limitations. The model successfully generates fluent Persian descriptions with proper attention to relevant image regions.

---

Introduction

Image captioning is a fundamental task in computer vision and natural language processing that aims to automatically generate natural language descriptions of images. This task bridges the gap between visual understanding and linguistic expression, requiring models to not only recognize objects and scenes in images but also understand their relationships and express them in coherent natural language.

Problem Definition

Given an input image I, the goal is to generate a sequence of words S = (s₁, s₂, ..., sₙ) that accurately describes the visual content of the image:

$$\widehat{S}=\arg \underset{S}{\max }P(S\mathrm{\mid }I)\backslash hat\{S\}\; =\; \backslash arg\backslash max\_S\; P(S|I)$$

Key Challenges

• Semantic Gap: Bridging visual features to linguistic concepts
• Compositionality: Understanding spatial and temporal relationships
• Diversity: Generating varied and creative descriptions
• Accuracy: Ensuring factual correctness of generated captions
• Multilingual Support: Handling different languages and cultural contexts

---

Theoretical Framework

Encoder-Decoder Architecture

The encoder-decoder framework is the foundation of modern image captioning systems:

Encoder: Converts the input image into a rich visual representation $$V=\text{CNN}(I)V\; =\; \backslash text\{CNN\}(I)$$

Decoder: Generates captions conditioned on the visual features

Attention Mechanism

Attention mechanisms address the limitation of fixed-length representations by allowing the decoder to dynamically focus on different parts of the image:

$${\alpha }_t=\text{softmax}(e_t)\backslash alpha\_t\; =\; \backslash text\{softmax\}(e\_t)$$ $$c_t=\sum _{i=1}^L{\alpha }_{t,i}{v}_{i}c\_t\; =\; \backslash sum\_\{i=1\}^\{L\}\; \backslash alpha\_\{t,i\}\; v\_i$$

Where:

• $e_t$ are attention energies
• $\alpha_t$ are attention weights
• $c_t$ is the context vector
• $v_i$ are visual features

Mathematical Formulation

1. Visual Encoding: $V = {v_1, v_2, ..., v_L}$ where $L$ is the number of spatial locations
2. Attention Computation: $e_{t,i} = f_{att}(h_{t-1}, v_i)$
3. Context Vector: $c_t = \sum_{i=1}^{L} \alpha_{t,i} v_i$
4. Hidden State Update: $h_t = f_{lstm}(h_{t-1}, [s_{t-1}; c_t])$
5. Word Prediction: $P(s_t|s_{<t}, I) = \text{softmax}(W_o h_t + b_o)$

---

Dataset Preparation

Dataset: COCO-Flickr-FA-40k

• Total Images: 40,000 images from COCO dataset
• Captions: One Persian caption per image
• Image Dimensions: Range from 72×51 to 673×664 pixels
• Train/Val/Test Split: 10,000 / 500 / 500 samples

Data Preprocessing

Sample Images from Dataset

Example images from the COCO-Flickr Persian dataset

Caption Length Distribution

Distribution of caption lengths in the dataset

Preprocessing Pipeline

1. Text Normalization: Using Hazm library for Persian text normalization
2. Emoji Removal: Cleaning text from emoji characters
3. Tokenization: Word-level tokenization with Hazm
4. Vocabulary Building: Building vocabulary with frequency filtering

Tokenizer Statistics

• Context Length: 40 tokens
• Total Unique Tokens: 5,166 (including unknown words)
• Dictionary Size: 2,971 tokens (excluding unknown)
• Unknown Words: 2,195 tokens (42.5% of unique tokens)
• Total Tokens in Training Set: 110,881

Dataset Structure

The COCODataset class:

• Loads images from specified path
• Applies necessary transforms (EfficientNet-B4 preprocessing)
• Converts captions to token IDs
• Allows switching transforms for visualization

---

Model Architecture

Architecture Overview

• Encoder: EfficientNet-B4 (pre-trained, frozen)
• Decoder: LSTM with attention mechanism (trainable)
• Embedding Dimension: 300
• Decoder Hidden Dimension: 512
• Attention Dimension: 512
• Total Parameters: 28.4 million
  • Encoder (non-trainable): 17.5 million
  • Decoder (trainable): 10.8 million

Encoder Design

Input: RGB image with dimensions 380×380
Output: Visual features with dimensions [batch_size, 144, 1792]

• 144: Number of spatial locations
• 1792: Feature dimension for each location

All EfficientNet parameters are frozen to leverage pre-trained features.

Decoder Design

The Decoder consists of three main components:

1. Classic Attention Layer

The classic attention mechanism (Additive Attention):

• Inputs: Encoder features (a) and decoder hidden state (h)
• Attention Energy: Linear combination of features and hidden state
• Attention Weights: Softmax over energies
• Context Vector: Weighted sum of visual features
• Beta Coefficient: A gate to control the importance of the context vector

2. Custom LSTM Cell

A manual implementation of LSTM:

• Four gates: Input (i), Forget (f), Output (o), Gate (g)
• Uses previous word embedding, hidden state, and attention context vector
• Incorporates visual information at each step

3. Output Layers

• Embedding: Converts word IDs to dense vectors (300-dim)
• LSTM: Processes sequences while considering attention
• Output: Combines embedding, hidden state, and context vector

Generation Methods

1. Greedy Decoding: Selects the word with highest probability at each step
2. Beam Search: Maintains k best candidates for better search (beam_width=3)

---

Training Process

Loss Function

The loss function combines two components:

1. Cross-Entropy Loss: For correct word prediction (main loss)
2. Regularization Loss: To ensure attention weights are properly normalized $$\mathcal{L}{reg} = \lambda \cdot \frac{1}{B} \sum{b=1}^{B} \left(1 - \sum_{i=1}^{L} \alpha_{b,t,i}\right)^2$$

Final loss: $$\mathcal{L}={\mathcal{L}}_{CE}+\lambda \cdot {\mathcal{L}}_{reg}\backslash mathcal\{L\}\; =\; \backslash mathcal\{L\}\_\{CE\}\; +\; \backslash lambda\; \backslash cdot\; \backslash mathcal\{L\}\_\{reg\}$$

where $\lambda = 1$ controls regularization importance.

Training Configuration

• Optimizer: Adam with initial learning rate 0.01
• Learning Rate Scheduler: ExponentialLR with decay factor 0.98
• Batch Size: 64
• Max Epochs: 50
• Early Stopping: Patience = 10, warmup = 3 epochs
• Weight Decay: 1e-5 for regularization

Training Progress - Base Model

Loss and BLEU score progression during training

Validation BLEU score improvement over epochs

Evaluation Metrics

• BLEU-1 to BLEU-4: Measures n-gram overlap between generated and reference captions
• BLEU-4: Most accurate metric as it considers sentence structure

---

Advanced Techniques

1. Scheduled Sampling

Problem with Teacher Forcing

In standard training, the model uses ground truth words at each step, causing:

1. Exposure Bias: During inference, model must use its own predictions
2. Divergence: Errors propagate to subsequent steps

Solution: Scheduled Sampling

Gradually decreases probability of using ground truth words:

• Epoch 1: $p_{teacher} = 1.0$ (always use ground truth)
• Each Epoch: $p_{teacher} = p_{teacher} - 0.02$ (gradual decrease)
• Final Epochs: $p_{teacher} \approx 0.0$ (only use predictions)

Training with Scheduled Sampling

Loss progression with scheduled sampling

BLEU score improvement with scheduled sampling

Benefits

• Better adaptation to inference conditions
• More robust to errors
• Improved generation performance

2. Scaled Dot-Product Attention

Advantages over Classic Attention

1. Computational Efficiency: Uses matrix multiplication instead of multiple linear layers
2. Simplicity: Fewer parameters
3. Scaling: Division by $\sqrt{d_k}$ prevents gradient explosion
4. Better Performance: Often outperforms classic attention

Implementation

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V\backslash text\{Attention\}(Q,\; K,\; V)\; =\; \backslash text\{softmax\}\backslash left(\backslash frac\{QK^T\}\{\backslash sqrt\{d\_k\}\}\backslash right)V$$

Where:

• Q (Query): Extracted from decoder hidden state
• K (Key): Extracted from encoder features
• V (Value): The encoder features themselves

Training with Scaled Dot-Product Attention

Loss convergence with scaled dot-product attention

Improvements

• Faster convergence
• Higher BLEU scores
• More stable training

---

Results and Evaluation

Sample Caption Generation

Base Model Results

Sample generated captions using Greedy decoding

Comparison of Greedy vs Beam Search decoding

Scheduled Sampling Results

Generated captions with scheduled sampling

Improved caption quality with scheduled sampling

Scaled Dot-Product Attention Results

Generated captions with scaled dot-product attention

Quantitative Evaluation

Base Model Performance

The base model with traditional attention achieves:

• Steady loss decrease during training
• Gradual BLEU score improvement
• Good generalization (small train-val gap)

Scheduled Sampling Performance

• Gradual improvement in BLEU scores
• Reduced exposure bias
• More diverse and natural captions
• Longer and more complete descriptions

Scaled Dot-Product Attention Performance

• Faster convergence
• Higher BLEU scores
• Better training stability
• Improved caption accuracy

Attention Map Visualization

Attention maps show which parts of the image the model focuses on when generating each word. Red/orange regions indicate highest attention, while blue/green regions have less attention.

Base Model Attention Maps

Attention visualization for sample 1

Attention visualization for sample 2

Attention visualization for sample 3

Attention visualization for sample 4

Scheduled Sampling Attention Maps

Attention with scheduled sampling - sample 1

Attention with scheduled sampling - sample 2

Attention with scheduled sampling - sample 3

Scaled Dot-Product Attention Maps

Scaled dot-product attention - sample 1

Scaled dot-product attention - sample 2

Scaled dot-product attention - sample 3

Scaled dot-product attention - sample 4

Scaled dot-product attention - sample 5

Detailed Caption Comparison

The following images show detailed comparisons of generated captions:

Detailed caption generation analysis

Training progress showing generated captions at different epochs

Method Comparison

Method	Advantages	Disadvantages
Traditional Attention	Simple implementation, easy to understand	May have lower performance in complex cases
Scheduled Sampling	Reduced exposure bias, more diverse captions	Requires careful tuning of decay schedule
Scaled Dot-Product	Higher efficiency, better performance, faster learning	May require more tuning in some cases

Key Observations

1. Scaled Dot-Product Attention usually achieves the best BLEU scores
2. Scheduled Sampling produces more diverse and natural captions
3. Beam Search performs better than Greedy in all methods
4. All methods successfully generate meaningful Persian captions
5. Attention maps show models learn to attend to correct regions

---

Conclusion

Project Summary

This project successfully implements a complete Persian image captioning system using deep learning. The system uses an encoder-decoder architecture with EfficientNet-B4 as the encoder and LSTM with attention mechanism as the decoder.

Main Achievements

1. Complete Implementation: All pipeline components from preprocessing to evaluation
2. Method Comparison: Examination of three different attention methods and Scheduled Sampling
3. Comprehensive Evaluation: Use of BLEU metrics and attention visualization
4. Persian Processing: Appropriate implementation for Persian language

Key Learnings

• Attention Mechanisms: Play a key role in generation quality
• Scheduled Sampling: Gradually reducing Teacher Forcing improves performance
• Scaled Dot-Product: Simplicity and higher efficiency compared to classic attention
• Beam Search: Better method than Greedy for caption generation

Future Directions

• Using Transformer instead of LSTM
• Implementing Self-Attention in decoder
• Using BERT or similar models for improved language processing
• Increasing dataset size
• Fine-tuning Encoder for the specific task

Final Notes

This project demonstrates that with appropriate architectures and advanced techniques, efficient image captioning systems for Persian language can be built. The key to success lies in the proper combination of architecture, appropriate loss function, and training techniques.

---

Files and Resources

Code Files

• code/NNDL_CAe_2.ipynb: Complete Persian image captioning implementation with all three attention mechanisms
• code/NNDL_CAe_2_Complete_IEEE.ipynb: IEEE format version
• extract_images.py: Script to extract images from notebook

Documentation

• README.md: This comprehensive documentation
• description/NNDL_HWe.pdf: Assignment description
• report/NNDL_UT_CA7_2.pdf: Project report

Reference Papers

• paper/1502.03044v3.pdf: Show and Tell: A Neural Image Caption Generator
• paper/1506.03099v3.pdf: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Images Directory

All extracted images from the notebook are stored in the images/ directory:

• Dataset visualizations
• Training progress plots
• Generated caption examples
• Attention map visualizations
• Method comparison charts

---

Technical Details

Requirements

torch>=1.9.0
torchvision>=0.10.0
matplotlib>=3.3.0
numpy>=1.19.0
PIL>=8.0.0
hazm>=0.7.0
arabic-reshaper>=3.0.0
python-bidi>=0.4.2
nltk>=3.6.0

Model Checkpoints

• checkpoint_loss.pth: Base model checkpoint
• checkpoint_scheduled_sampling.pth: Scheduled sampling model checkpoint
• scaled_dot_prod_attn.pth: Scaled dot-product attention model checkpoint

---

Acknowledgments

This project is part of the Neural Networks and Deep Learning course at the University of Tehran. Special thanks to the course instructors and the developers of the libraries used in this implementation.

---

Last Updated: 2024
Version: 1.0