python_files/CA5_Vision_Transformers/README.md

CA5: Vision Transformers & Large Language Diffusion Models

This project implements two main components:

1. Image Re-identification using Transformers (BotNet): A ResNet50-based model with Multi-Head Self-Attention for person re-identification.

2. Large Language Diffusion (LLaDA): A diffusion-based text generation model for Text-to-SQL tasks.

Project Structure

CA5_Vision_Transformers/
├── data.py              # Data loading, preprocessing, and dataset classes
├── models.py            # Model architectures (ResNet, BotNet, MHSA)
├── utils.py             # Utility functions (visualization, interpretability)
├── train.py             # Training and evaluation functions
├── generate.py          # Generation functions for LLaDA
├── evaluation.py        # Advanced evaluation metrics (CMC, mAP)
├── ablation.py          # Ablation study functions
├── optimization.py      # Model optimization and deployment
├── research.py          # Advanced research demonstrations
├── main.py              # Main script with comprehensive examples
└── requirements.txt     # Dependencies

Installation

pip install -r requirements.txt

Core Components

1. Person Re-identification (Re-ID)

Models

• ResNet50: Baseline CNN architecture
• BotNet50: ResNet50 enhanced with Multi-Head Self-Attention (MHSA)

Key Features

• Attention visualization for interpretability
• Comprehensive evaluation with CMC and mAP metrics
• Ablation studies comparing different architectures

2. Large Language Diffusion (LLaDA)

Features

• Forward diffusion masking process
• LoRA fine-tuning on quantized LLaMA model
• Block diffusion sampling for generation
• Exact match scoring for SQL evaluation

Usage Examples

Basic Re-ID Training

from main import *

# Models are initialized in main.py
train_model(resnet_model, resnet_optimizer, train_loader)
evaluate_model(resnet_model, test_loader)

Advanced Evaluation

from evaluation import evaluate_reid_advanced

# Comprehensive evaluation with CMC and mAP
cmc_scores, map_score = evaluate_reid_advanced(botnet_model, test_loader, device, "BotNet50")

Interpretability Analysis

from utils import comprehensive_interpretability_analysis

# Full interpretability analysis
sample_image, _ = next(iter(test_loader))
results = comprehensive_interpretability_analysis(botnet_model, sample_image, pil_image, "BotNet50")

Ablation Studies

from ablation import ablation_study_reid, plot_ablation_results

# Compare different model configurations
model_configs = {
    'ResNet_Pretrained': {'model_type': 'resnet_pretrained', 'num_classes': num_classes},
    'BotNet_4Heads': {'model_type': 'botnet_4', 'num_classes': num_classes},
}
results = ablation_study_reid(model_configs, train_loader, test_loader)
plot_ablation_results(results)

LLaDA Generation

from main import *

example = dataset['test'][0]
prompt, _ = format_example(example, tokenizer)
generated = generate_block_diffusion(model, tokenizer, prompt)
print("Generated SQL:", generated)

Model Optimization

from optimization import benchmark_inference_speed, quantize_model_dynamic

# Benchmark inference speed
benchmark_results = benchmark_inference_speed(model, test_loader, device, "BotNet50")

# Quantize model for efficiency
quantized_model = quantize_model_dynamic(model)

Research Demonstrations

from research import demonstrate_clip_reid_integration, research_directions_summary

# Explore advanced research directions
demonstrate_clip_reid_integration()
research_directions_summary()

Key Functions

Data Processing

• download_market1501(): Download Market-1501 dataset
• ReIDDataset: Custom dataset class for Re-ID
• format_example(): Prepare examples for LLaDA training

Model Architectures

• get_resnet50(): Create ResNet50 model
• MHSA: Multi-Head Self-Attention module
• BotNet50: Bottleneck Transformer network

Training & Evaluation

• train_model(): Basic training loop
• evaluate_model(): Basic accuracy evaluation
• evaluate_reid_advanced(): CMC and mAP evaluation
• train_step(): Single training step for LLaDA

Analysis & Visualization

• visualize_attention(): Attention map visualization
• comprehensive_interpretability_analysis(): Full model interpretability
• plot_cmc_curve(): Plot CMC evaluation results

Advanced Features

• ablation_study_reid(): Compare model configurations
• benchmark_inference_speed(): Performance benchmarking
• generate_block_diffusion(): LLaDA text generation

Requirements

• Python 3.8+
• PyTorch 2.0+
• CUDA-compatible GPU (recommended)
• 8GB+ VRAM for LLaDA model
• See requirements.txt for full dependencies

Research Contributions

This implementation explores:

• Attention mechanisms in CNN architectures
• Diffusion models for text generation
• Model interpretability techniques
• Efficient training methods
• Advanced evaluation metrics

The code provides a foundation for research in vision transformers, generative diffusion models, and their applications in computer vision and natural language processing.

Key Components

BotNet Architecture

• Replaces ResNet's final convolutional stage with Multi-Head Self-Attention
• Captures global spatial dependencies for better Re-ID performance
• Includes attention visualization for interpretability

LLaDA Implementation

• Forward masking diffusion process for discrete text
• Block-wise iterative generation
• LoRA fine-tuning for efficiency
• SQL normalization and exact match evaluation

Results

• Re-ID: BotNet achieves superior accuracy over baseline ResNet through global attention
• LLaDA: Generates coherent SQL queries via diffusion-based denoising

Authors

[Your Name] - University of Tehran