Leacb4
/

gap-clip

@@ -1,990 +1,68 @@
-# GAP-CLIP: Guaranteed Attribute Positioning in CLIP Embeddings
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
-[![PyTorch 2.0+](https://img.shields.io/badge/pytorch-2.0+-ee4c2c.svg)](https://pytorch.org/)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Hugging Face](https://img.shields.io/badge/🤗-Hugging%20Face-yellow)](https://huggingface.co/Leacb4/gap-clip)
-**Advanced multimodal fashion search model combining specialized color embeddings, hierarchical category embeddings, and CLIP for intelligent fashion item retrieval.**
 ---
-## 🚀 Quick Start
-### Installation (< 1 minute)
-```bash
-# Clone the repository
-git clone https://github.com/Leacb4/gap-clip.git
-cd gap-clip
-# Install package with pip
-pip install -e .
-# Or just install dependencies
-pip install -r requirements.txt
-```
-### Try It Now (< 2 minutes)
-```python
-from example_usage import load_models_from_hf
-# Load pre-trained models from Hugging Face
-models = load_models_from_hf("Leacb4/gap-clip")
-# Search with text
-import torch.nn.functional as F
-text_query = "red summer dress"
-text_inputs = models['processor'](text=[text_query], padding=True, return_tensors="pt")
-text_inputs = {k: v.to(models['device']) for k, v in text_inputs.items()}
-with torch.no_grad():
-    text_features = models['main_model'](**text_inputs).text_embeds
-# Extract specialized embeddings
-color_emb = text_features[:, :16]      # Color (dims 0-15)
-category_emb = text_features[:, 16:80]  # Category (dims 16-79)
-general_emb = text_features[:, 80:]     # General CLIP (dims 80-511)
-print(f"✅ Successfully extracted embeddings!")
-print(f"   Color: {color_emb.shape}, Category: {category_emb.shape}, General: {general_emb.shape}")
-```
 ---
-## 📋 Description
-This project implements an advanced fashion search system based on CLIP, with three specialized models:
-1. **Color Model** (`color_model.pt`) : Specialized CLIP model for extracting reduced-size color embeddings from text and images
-2. **Hierarchy Model** (`hierarchy_model.pth`) : Model for classifying and encoding reduced-size categorical hierarchy of fashion items
-3. **Main CLIP Model** (`gap_clip.pth`) : Main CLIP model based on LAION, trained with color and hierarchy embeddings
-### Architecture
-The main model's embedding structure:
-- **Dimensions 0-15** (16 dims): Color embeddings aligned with specialized color model
-- **Dimensions 16-79** (64 dims): Hierarchy embeddings aligned with specialized hierarchy model
-- **Dimensions 80-511** (432 dims): Standard CLIP embeddings for general visual-semantic understanding
-**Total: 512 dimensions** per embedding (text or image)
-**Key Innovation**: The first 80 dimensions are explicitly trained to align with specialized models through direct MSE and cosine similarity losses, ensuring guaranteed attribute positioning (GAP) while maintaining full CLIP capabilities in the remaining dimensions.
-### Loss Functions
-**1. Enhanced Contrastive Loss** (`enhanced_contrastive_loss`):
-Combines multiple objectives:
-- **Original Triple Loss**: Text-image-attributes contrastive learning
-- **Color Alignment**: Forces dims 0-15 to match color model embeddings
-- **Hierarchy Alignment**: Forces dims 16-79 to match hierarchy model embeddings
-- **Reference Loss**: Optional regularization to stay close to base CLIP
-**2. Alignment Components**:
-```python
-# Color alignment (text & image)
-color_text_mse = F.mse_loss(main_color_dims, color_model_emb)
-color_text_cosine = 1 - F.cosine_similarity(main_color_dims, color_model_emb).mean()
-# Hierarchy alignment (text & image)
-hierarchy_text_mse = F.mse_loss(main_hierarchy_dims, hierarchy_model_emb)
-hierarchy_text_cosine = 1 - F.cosine_similarity(main_hierarchy_dims, hierarchy_model_emb).mean()
-# Combined alignment
-alignment_loss = (color_alignment + hierarchy_alignment) / 2
-```
-**3. Final Loss**:
-```python
-total_loss = (1 - α) * contrastive_loss + α * alignment_loss + β * reference_loss
-```
-Where:
-- α (alignment_weight) = 0.2 : Balances contrastive and alignment objectives
-- β (reference_weight) = 0.1 : Keeps text space close to base CLIP
-## 🚀 Installation
-### Prerequisites
-- Python 3.8 or higher
-- PyTorch 2.0+ (with CUDA for GPU support, optional but recommended)
-- 16GB RAM minimum (32GB recommended for training)
-- ~5GB disk space for models and data
-### Method 1: Install as Package (Recommended)
-```bash
-# Clone repository
-git clone https://github.com/Leacb4/gap-clip.git
-cd gap-clip
-# Install in development mode
-pip install -e .
-# Or install with optional dependencies
-pip install -e ".[dev]"      # With development tools
-pip install -e ".[optuna]"   # With hyperparameter optimization
-pip install -e ".[all]"      # With all extras
-```
-### Method 2: Install Dependencies Only
-```bash
-pip install -r requirements.txt
-```
-### Method 3: From Hugging Face (Model Only)
-```python
-from example_usage import load_models_from_hf
-models = load_models_from_hf("Leacb4/gap-clip")
-```
-### Main Dependencies
-| Package | Version | Purpose |
-|---------|---------|---------|
-| `torch` | ≥2.0.0 | Deep learning framework |
-| `transformers` | ≥4.30.0 | Hugging Face CLIP models |
-| `huggingface-hub` | ≥0.16.0 | Model download/upload |
-| `pillow` | ≥9.0.0 | Image processing |
-| `pandas` | ≥1.5.0 | Data manipulation |
-| `scikit-learn` | ≥1.3.0 | ML metrics & evaluation |
-| `tqdm` | ≥4.65.0 | Progress bars |
-| `matplotlib` | ≥3.7.0 | Visualization |
-### Verify Installation
-```python
-# Test that everything works
-import config
-config.print_config()
-# Check device
-print(f"Using device: {config.device}")
-```
-## 📁 Project Structure
-```
-.
-├── color_model.py              # Color model architecture and training
-├── hierarchy_model.py          # Hierarchy model architecture and training
-├── main_model.py               # Main GAP-CLIP model with enhanced loss functions
-├── train_main_model.py         # Training script with optimized hyperparameters
-├── config.py                   # Configuration for paths and parameters
-├── example_usage.py            # Usage examples and HuggingFace loading
-├── tokenizer_vocab.json        # Tokenizer vocabulary for color model
-├── models/
-│   ├── color_model.pt          # Trained color model checkpoint
-│   ├── hierarchy_model.pth     # Trained hierarchy model checkpoint
-│   └── gap_clip.pth            # Main GAP-CLIP model checkpoint
-├── evaluation/                 # Comprehensive evaluation scripts
-│   ├── main_model_evaluation.py         # Main evaluation with 3 datasets
-│   ├── evaluate_color_embeddings.py     # Color embedding analysis
-│   ├── hierarchy_evaluation.py          # Hierarchy classification tests
-│   ├── fashion_search.py                # Interactive search demos
-│   ├── 0_shot_classification.py         # Zero-shot classification
-│   ├── heatmap_color_similarities.py    # Color similarity visualization
-│   ├── tsne_images.py                   # t-SNE embedding visualization
-│   └── basic_test_generalized.py        # Basic functionality tests
-├── data/
-│   ├── data_with_local_paths.csv        # Training dataset with annotations
-│   ├── fashion-mnist_test.csv           # Fashion-MNIST evaluation data
-│   ├── download_data.py                 # Dataset download utilities
-│   └── get_csv_from_chunks.py           # Dataset preprocessing
-├── optuna/                     # Hyperparameter optimization
-│   ├── optuna_optimisation.py           # Optuna optimization script
-│   ├── optuna_study.pkl                 # Saved optimization study
-│   ├── optuna_results.txt               # Best hyperparameters
-│   ├── optuna_optimization_history.png  # Optimization visualization
-│   ├── optuna_param_importances.png     # Parameter importance plot
-│   └── optuna_guide.md                  # Optuna usage guide
-├── upload_hf/                  # HuggingFace Hub upload utilities
-│   ├── upload_to_huggingface.py         # Professional upload script (rewritten)
-│   └── README_UPLOAD.md                 # Complete upload guide
-├── requirements.txt            # Python dependencies (organized)
-├── setup.py                    # Package installation (NEW)
-├── __init__.py                 # Package initialization (NEW)
-├── .gitignore                  # Git ignore rules (NEW)
-└── README.md                   # This documentation
-```
-### Key Files Description
-**Core Model Files**:
-- `color_model.py`: ResNet18-based color embedding model (16 dims) - Bug fixed ✨
-- `hierarchy_model.py`: ResNet18-based hierarchy classification model (64 dims)
-- `main_model.py`: GAP-CLIP implementation with enhanced contrastive loss - Bug fixed ✨
-- `train_main_model.py`: Training with Optuna-optimized hyperparameters - Improved ✨
-**Configuration & Setup** (✨ New/Improved):
-- `config.py`: ✨ Completely rewritten with type hints, auto device detection, validation utilities
-- `setup.py`: ✨ NEW - Professional package installer with CLI entry points
-- `__init__.py`: ✨ NEW - Package initialization for easy imports
-- `.gitignore`: ✨ NEW - Comprehensive Git ignore rules
-- `requirements.txt`: ✨ Improved - Organized with comments and categories
-- `tokenizer_vocab.json`: Vocabulary for color model's text encoder
-**Upload Tools** (✨ Rewritten):
-- `upload_hf/upload_to_huggingface.py`: ✨ Complete professional rewrite with:
-  - Object-oriented design
-  - Multiple authentication methods
-  - Category-based uploads (models, code, docs, etc.)
-  - Progress tracking
-  - Automatic model card generation
-  - Detailed error handling
-- `upload_hf/README_UPLOAD.md`: ✨ NEW - Complete upload guide
-**Evaluation Suite**:
-- `main_model_evaluation.py`: Comprehensive evaluation across Fashion-MNIST, KAGL, and local datasets
-- `evaluation/run_all_evaluations.py`: ✨ NEW - Automated evaluation runner with reports
-- Other scripts provide specialized analysis (color, hierarchy, search, t-SNE, etc.)
-**Training Data**:
-- `data_with_local_paths.csv`: Main training dataset with text, color, hierarchy, and image paths
-- `fashion-mnist_test.csv`: Evaluation dataset for zero-shot generalization testing
-**CLI Commands** (✨ New):
-After installation with `pip install -e .`, you can use:
-```bash
-gap-clip-train      # Start training
-gap-clip-example    # Run usage examples
-```
-## 🔧 Configuration
-Main parameters are defined in `config.py` (✨ completely rewritten with improvements):
-```python
-import config
-# Automatic device detection (CUDA > MPS > CPU)
-device = config.device  # Automatically selects best available device
-# Embedding dimensions
-color_emb_dim = config.color_emb_dim           # 16 dims (0-15)
-hierarchy_emb_dim = config.hierarchy_emb_dim   # 64 dims (16-79)
-main_emb_dim = config.main_emb_dim             # 512 dims total
-# Default training hyperparameters
-batch_size = config.DEFAULT_BATCH_SIZE         # 32
-learning_rate = config.DEFAULT_LEARNING_RATE   # 1.5e-5
-temperature = config.DEFAULT_TEMPERATURE       # 0.09
-# Utility functions
-config.print_config()      # Print current configuration
-config.validate_paths()    # Validate that all files exist
-```
-### New Features in config.py ✨
-- **Automatic device detection**: Selects CUDA > MPS > CPU automatically
-- **Type hints**: Full type annotations for better IDE support
-- **Validation**: `validate_paths()` checks all model files exist
-- **Print utility**: `print_config()` shows current settings
-- **Constants**: Pre-defined default hyperparameters
-- **Documentation**: Comprehensive docstrings for all settings
-### Model Paths
-Default paths configured in `config.py`:
-- `models/color_model.pt` : Trained color model checkpoint
-- `models/hierarchy_model.pth` : Trained hierarchy model checkpoint
-- `models/gap_clip.pth` : Main GAP-CLIP model checkpoint
-- `tokenizer_vocab.json` : Tokenizer vocabulary for color model
-- `data_with_local_paths.csv` : Training/validation dataset
-### Dataset Format
-The training dataset CSV should contain:
-- `text`: Text description of the fashion item
-- `color`: Color label (e.g., "red", "blue", "black")
-- `hierarchy`: Category label (e.g., "dress", "shirt", "shoes")
-- `local_image_path`: Path to the image file
-Example:
-```csv
-text,color,hierarchy,local_image_path
-"red summer dress with floral pattern",red,dress,data/images/001.jpg
-"blue denim jeans casual style",blue,jeans,data/images/002.jpg
-```
-## 📦 Usage
-### 1. Load Models from Hugging Face
-If your models are already uploaded to Hugging Face:
-```python
-from example_usage import load_models_from_hf
-# Load all models
-models = load_models_from_hf("your-username/your-model")
-color_model = models['color_model']
-hierarchy_model = models['hierarchy_model']
-main_model = models['main_model']
-processor = models['processor']
-device = models['device']
-```
-### 2. Text Search
-```python
-import torch
-from transformers import CLIPProcessor
-# Prepare text query
-text_query = "red dress"
-text_inputs = processor(text=[text_query], padding=True, return_tensors="pt")
-text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
-# Get main model embeddings
-with torch.no_grad():
-    outputs = main_model(**text_inputs)
-    text_features = outputs.text_embeds
-# Get specialized embeddings
-color_emb = color_model.get_text_embeddings([text_query])
-hierarchy_emb = hierarchy_model.get_text_embeddings([text_query])
-```
-### 3. Image Search
-```python
-from PIL import Image
-# Load image
-image = Image.open("path/to/image.jpg").convert("RGB")
-image_inputs = processor(images=[image], return_tensors="pt")
-image_inputs = {k: v.to(device) for k, v in image_inputs.items()}
-# Get embeddings
-with torch.no_grad():
-    outputs = main_model(**image_inputs)
-    image_features = outputs.image_embeds
-```
-### 4. Using the Example Script
-The `example_usage.py` provides ready-to-use examples for loading and using GAP-CLIP:
-```bash
-# Load from HuggingFace and search with text
-python example_usage.py \
-    --repo-id Leacb4/gap-clip \
-    --text "red summer dress"
-# Search with image
-python example_usage.py \
-    --repo-id Leacb4/gap-clip \
-    --image path/to/image.jpg
-# Both text and image
-python example_usage.py \
-    --repo-id Leacb4/gap-clip \
-    --text "blue denim jeans" \
-    --image path/to/image.jpg
-```
-This script demonstrates:
-- Loading models from HuggingFace Hub
-- Extracting text and image embeddings
-- Accessing color and hierarchy subspaces
-- Measuring alignment quality with specialized models
-## 🎯 Model Training
-### Train the Color Model
-```python
-from color_model import ColorCLIP, train_color_model
-# Configuration
-model = ColorCLIP(vocab_size=10000, embedding_dim=16)
-# ... dataset configuration ...
-# Training
-train_color_model(model, train_loader, val_loader, num_epochs=20)
-```
-### Train the Hierarchy Model
-```python
-from hierarchy_model import Model as HierarchyModel, train_hierarchy_model
-# Configuration
-model = HierarchyModel(num_hierarchy_classes=10, embed_dim=64)
-# ... dataset configuration ...
-# Training
-train_hierarchy_model(model, train_loader, val_loader, num_epochs=20)
-```
-### Train the Main CLIP Model
-The main model trains with both specialized models using an enhanced contrastive loss.
-**Option 1: Train with optimized hyperparameters (recommended)**:
-```bash
-python train_main_model.py
-```
-This uses hyperparameters optimized with Optuna (Trial 29, validation loss ~0.1129).
-**Option 2: Train with default parameters**:
-```bash
-python main_model.py
-```
-This runs the main training loop with manually configured parameters.
-**Default Training Parameters** (in `main_model.py`):
-- `num_epochs = 20` : Number of training epochs
-- `learning_rate = 1.5e-5` : Learning rate with AdamW optimizer
-- `temperature = 0.09` : Temperature for softer contrastive learning
-- `alignment_weight = 0.2` : Weight for color/hierarchy alignment loss
-- `weight_decay = 5e-4` : L2 regularization to prevent overfitting
-- `batch_size = 32` : Batch size
-- `subset_size = 20000` : Dataset size for better generalization
-- `reference_weight = 0.1` : Weight for base CLIP regularization
-**Enhanced Loss Function**:
-The training uses `enhanced_contrastive_loss` which combines:
-1. **Triple Contrastive Loss** (weighted):
-   - Text-Image alignment (70%)
-   - Text-Attributes alignment (15%)
-   - Image-Attributes alignment (15%)
-2. **Direct Alignment Loss** (combines color & hierarchy):
-   - MSE loss between main model color dims (0-15) and color model embeddings
-   - MSE loss between main model hierarchy dims (16-79) and hierarchy model embeddings
-   - Cosine similarity losses for both color and hierarchy
-   - Applied to both text and image embeddings
-3. **Reference Model Loss** (optional):
-   - Keeps text embeddings close to base CLIP
-   - Improves cross-domain generalization
-**Training Features**:
-- Enhanced data augmentation (rotation, color jitter, blur, affine transforms)
-- Gradient clipping (max_norm=1.0) to prevent exploding gradients
-- ReduceLROnPlateau scheduler (patience=3, factor=0.5)
-- Early stopping (patience=7)
-- Automatic best model saving with checkpoints
-- Detailed metrics logging (alignment losses, cosine similarities)
-- Overfitting detection and warnings
-- Training curves visualization with 3 plots (losses, overfitting gap, comparison)
-### Hyperparameter Optimization
-The project includes Optuna-based hyperparameter optimization in `optuna/`:
-```bash
-cd optuna
-python optuna_optimisation.py
-```
-This optimizes:
-- Learning rate
-- Temperature for contrastive loss
-- Alignment weight
-- Weight decay
-Results are saved in `optuna_study.pkl` and visualizations in `optuna_optimization_history.png` and `optuna_param_importances.png`.
-The best hyperparameters from Optuna optimization are used in `train_main_model.py`.
-## 📊 Models
-### Color Model
-- **Architecture** : ResNet18 (image encoder) + Embedding (text encoder)
-- **Embedding dimension** : 16
-- **Trained on** : Fashion data with color annotations
-- **Usage** : Extract color embeddings from text or images
-### Hierarchy Model
-- **Architecture** : ResNet18 (image encoder) + Embedding (hierarchy encoder)
-- **Embedding dimension** : 64
-- **Hierarchy classes** : shirt, dress, pant, shoe, bag, etc.
-- **Usage** : Classify and encode categorical hierarchy
-### Main CLIP Model (GAP-CLIP)
-- **Architecture** : CLIP ViT-B/32 (LAION)
-- **Base Model** : `laion/CLIP-ViT-B-32-laion2B-s34B-b79K`
-- **Training Approach** : Enhanced contrastive loss with direct attribute alignment
-- **Embedding Dimensions** : 512 total
-  - Color subspace: dims 0-15 (16 dims)
-  - Hierarchy subspace: dims 16-79 (64 dims)
-  - General CLIP: dims 80-511 (432 dims)
-- **Training Dataset** : 20,000 fashion items with color and hierarchy annotations
-- **Validation Split** : 80/20 train-validation split
-- **Optimizer** : AdamW with weight decay (5e-4)
-- **Best Checkpoint** : Automatically saved based on validation loss
-- **Features** :
-  - Multi-modal text-image search
-  - Guaranteed attribute positioning (GAP) in specific dimensions
-  - Direct alignment with specialized color and hierarchy models
-  - Maintains general CLIP capabilities for cross-domain tasks
-  - Reduced overfitting through augmentation and regularization
-## 🔍 Advanced Usage Examples
-### Search with Combined Embeddings
-```python
-import torch
-import torch.nn.functional as F
-# Text query
-text_query = "red dress"
-text_inputs = processor(text=[text_query], padding=True, return_tensors="pt")
-text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
-# Main model embeddings
-with torch.no_grad():
-    outputs = main_model(**text_inputs)
-    text_features = outputs.text_embeds  # Shape: [1, 512]
-# Extract specialized embeddings from main model
-main_color_emb = text_features[:, :16]  # Color dimensions (0-15)
-main_hierarchy_emb = text_features[:, 16:80]  # Hierarchy dimensions (16-79)
-main_clip_emb = text_features[:, 80:]  # General CLIP dimensions (80-511)
-# Compare with specialized models
-color_emb = color_model.get_text_embeddings([text_query])
-hierarchy_emb = hierarchy_model.get_text_embeddings([text_query])
-# Measure alignment quality
-color_similarity = F.cosine_similarity(color_emb, main_color_emb, dim=1)
-hierarchy_similarity = F.cosine_similarity(hierarchy_emb, main_hierarchy_emb, dim=1)
-print(f"Color alignment: {color_similarity.item():.4f}")
-print(f"Hierarchy alignment: {hierarchy_similarity.item():.4f}")
-# For search, you can use different strategies:
-# 1. Use full embeddings for general search
-# 2. Use color subspace for color-specific search
-# 3. Use hierarchy subspace for category search
-# 4. Weighted combination of subspaces
-```
-### Search in an Image Database
 ```python
-import numpy as np
 import torch
-import torch.nn.functional as F
-from tqdm import tqdm
-# Step 1: Pre-compute image embeddings (do this once)
-image_paths = [...]  # List of image paths
-image_features_list = []
-print("Computing image embeddings...")
-for img_path in tqdm(image_paths):
-    image = Image.open(img_path).convert("RGB")
-    image_inputs = processor(images=[image], return_tensors="pt")
-    image_inputs = {k: v.to(device) for k, v in image_inputs.items()}
-    with torch.no_grad():
-        outputs = main_model(**image_inputs)
-        features = outputs.image_embeds  # Shape: [1, 512]
-        image_features_list.append(features.cpu())
-# Stack all features
-image_features = torch.cat(image_features_list, dim=0)  # Shape: [N, 512]
-# Step 2: Search with text query
-query = "red dress"
-text_inputs = processor(text=[query], padding=True, return_tensors="pt")
-text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
-with torch.no_grad():
-    outputs = main_model(**text_inputs)
-    text_features = outputs.text_embeds  # Shape: [1, 512]
-# Step 3: Calculate similarities
-# Normalize embeddings for cosine similarity
-text_features_norm = F.normalize(text_features, dim=-1)
-image_features_norm = F.normalize(image_features.to(device), dim=-1)
-# Compute cosine similarities
-similarities = (text_features_norm @ image_features_norm.T).squeeze(0)  # Shape: [N]
-# Step 4: Get top-k results
-top_k = 10
-top_scores, top_indices = similarities.topk(top_k, largest=True)
-# Display results
-print(f"\nTop {top_k} results for query: '{query}'")
-for i, (idx, score) in enumerate(zip(top_indices, top_scores)):
-    print(f"{i+1}. {image_paths[idx]} (similarity: {score.item():.4f})")
-# Optional: Filter by color or hierarchy
-# Extract color embeddings from query
-query_color_emb = text_features[:, :16]
-# Extract hierarchy embeddings from query
-query_hierarchy_emb = text_features[:, 16:80]
-# Use these for more targeted search
-```
-## 📝 Evaluation
-### Comprehensive Model Evaluation
-The main evaluation script `evaluation/main_model_evaluation.py` provides extensive testing across multiple datasets:
-```bash
-python evaluation/main_model_evaluation.py
 ```
-**Evaluation Datasets**:
-1. **Fashion-MNIST** (~10,000 samples) - Grayscale fashion items
-2. **KAGL Marqo** (HuggingFace dataset) - Real fashion images with metadata
-3. **Local Validation Dataset** - Custom validation set with local images
-**Evaluation Metrics**:
-For each dataset, the evaluation measures:
-1. **Color Embeddings Performance** (dimensions 0-15):
-   - Nearest Neighbor (NN) Accuracy: Classification accuracy using nearest neighbor
-   - Centroid Accuracy: Classification using cluster centroids
-   - Separation Score: How well color embeddings separate different classes
-2. **Hierarchy Embeddings Performance** (dimensions 16-79):
-   - Nearest Neighbor (NN) Accuracy: Classification accuracy for fashion categories
-   - Centroid Accuracy: Cluster-based classification
-   - Separation Score: Class separation quality
-3. **Full Embeddings Performance** (all 512 dimensions):
-   - Evaluates complete embedding space
-   - Compares subspace (color/hierarchy) vs full embedding effectiveness
-**Baseline Comparison**:
-The evaluation includes comparison against `patrickjohncyh/fashion-clip`:
-- Direct performance comparison on the same datasets
-- Improvement metrics calculation
-- Statistical significance analysis
-**Key Evaluation Functions**:
-- `evaluate_fashion_mnist()` : Test on Fashion-MNIST dataset
-- `evaluate_kaggle_marqo()` : Test on real fashion images
-- `evaluate_local_validation()` : Test on local validation set
-- `evaluate_baseline_fashion_mnist()` : Baseline model on Fashion-MNIST
-- `evaluate_baseline_kaggle_marqo()` : Baseline model on KAGL
-- `evaluate_full_embeddings()` : Test complete 512D space
-- `analyze_baseline_vs_trained_performance()` : Comparative analysis
-- `compare_subspace_vs_full_embeddings()` : Subspace effectiveness
-**Visualization Outputs** (saved in analysis directory):
-- Confusion matrices for color and hierarchy classification
-- t-SNE projections of embeddings
-- Similarity heatmaps
-- Performance comparison charts
-**Other Evaluation Scripts**:
-- `evaluate_color_embeddings.py` : Focused color embeddings evaluation
-- `fashion_search.py` : Interactive fashion search tests
-- `hierarchy_evaluation.py` : Hierarchy classification analysis
-- `0_shot_classification.py` : Zero-shot classification tests
-## 📊 Performance & Results
-The evaluation framework (`main_model_evaluation.py`) tests the model across three datasets with comparison to a baseline fashion CLIP model.
-### Evaluation Metrics
-**Color Classification** (dimensions 0-15):
-- Nearest Neighbor Accuracy
-- Centroid-based Accuracy
-- Separation Score (class separability)
-**Hierarchy Classification** (dimensions 16-79):
-- Nearest Neighbor Accuracy
-- Centroid-based Accuracy
-- Separation Score
-**Full Embedding Quality** (all 512 dims):
-- Tests whether the full space maintains performance
-- Compares subspace vs full embedding effectiveness
-### Datasets Used for Evaluation
-1. **Fashion-MNIST**: 10,000 grayscale fashion item images
-   - 10 categories (T-shirt, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot)
-   - Mapped to model's hierarchy classes
-2. **KAGL Marqo Dataset**: Real-world fashion images from HuggingFace
-   - Diverse fashion items with rich metadata
-   - Color and category annotations
-   - Realistic product images
-3. **Local Validation Set**: Custom validation dataset
-   - Fashion items with local image paths
-   - Annotated with colors and hierarchies
-   - Domain-specific evaluation
-### Comparative Analysis
-The evaluation includes:
-- **Baseline comparison**: GAP-CLIP vs `patrickjohncyh/fashion-clip`
-- **Subspace analysis**: Dedicated dimensions (0-79) vs full space (0-511)
-- **Cross-dataset generalization**: Performance consistency across datasets
-- **Alignment quality**: How well specialized dimensions match expert models
-All visualizations (confusion matrices, t-SNE plots, heatmaps) are automatically saved in the analysis directory.
-## 📄 Citation
-If you use GAP-CLIP in your research, please cite:
 ```bibtex
 @misc{gap-clip-2024,
   title={GAP-CLIP: Guaranteed Attribute Positioning in CLIP Embeddings for Fashion Search},
   author={Sarfati, Lea Attia},
   year={2024},
-  note={A multi-loss framework combining contrastive learning with direct attribute alignment},
-  howpublished={\url{https://huggingface.co/Leacb4/gap-clip}},
-  abstract={GAP-CLIP introduces a novel training approach that guarantees specific embedding
-            dimensions encode color (dims 0-15) and hierarchy (dims 16-79) information through
-            direct alignment with specialized models, while maintaining full CLIP capabilities
-            in the remaining dimensions (80-511).}
 }
 ```
-### Key Contributions
-- **Guaranteed Attribute Positioning**: Specific dimensions reliably encode color and hierarchy
-- **Multi-Loss Training**: Combines contrastive learning with MSE and cosine alignment losses
-- **Specialized Model Alignment**: Direct supervision from expert color and hierarchy models
-- **Preserved Generalization**: Maintains base CLIP capabilities for cross-domain tasks
-- **Comprehensive Evaluation**: Tested across multiple datasets with baseline comparisons
-## ❓ FAQ & Troubleshooting
-### Q: What are the minimum hardware requirements?
-**A**:
-- **GPU**: Recommended for training (CUDA or MPS). CPU training is very slow.
-- **RAM**: Minimum 16GB, recommended 32GB for training
-- **Storage**: ~5GB for models and datasets
-### Q: Why are my embeddings not aligned?
-**A**: Check that:
-1. You're using the correct dimension ranges (0-15 for color, 16-79 for hierarchy)
-2. The model was trained with alignment_weight > 0
-3. Color and hierarchy models were properly loaded during training
-### Q: How do I use only the color or hierarchy subspace for search?
-**A**:
-```python
-# Extract and use only color embeddings
-text_color_emb = text_features[:, :16]
-image_color_emb = image_features[:, :16]
-color_similarity = F.cosine_similarity(text_color_emb, image_color_emb)
-# Extract and use only hierarchy embeddings
-text_hierarchy_emb = text_features[:, 16:80]
-image_hierarchy_emb = image_features[:, 16:80]
-hierarchy_similarity = F.cosine_similarity(text_hierarchy_emb, image_hierarchy_emb)
-```
-### Q: Can I add more attributes beyond color and hierarchy?
-**A**: Yes! The architecture is extensible:
-1. Train a new specialized model for your attribute
-2. Reserve additional dimensions in the embedding space
-3. Add alignment losses for these dimensions in `enhanced_contrastive_loss`
-4. Update `config.py` with new dimension ranges
-### Q: How do I evaluate on my own dataset?
-**A**:
-1. Format your dataset as CSV with columns: `text`, `color`, `hierarchy`, `local_image_path`
-2. Update `config.local_dataset_path` in `config.py`
-3. Run the evaluation: `python evaluation/main_model_evaluation.py`
-### Q: Training loss is decreasing but validation loss is increasing. What should I do?
-**A**: This indicates overfitting. Try:
-- Increase `weight_decay` (e.g., from 5e-4 to 1e-3)
-- Reduce `alignment_weight` (e.g., from 0.2 to 0.1)
-- Increase dataset size (`subset_size`)
-- Add more data augmentation in `CustomDataset`
-- Enable or increase early stopping patience
-### Q: Can I fine-tune GAP-CLIP on a specific domain?
-**A**: Yes! Load the checkpoint and continue training:
-```python
-checkpoint = torch.load('models/gap_clip.pth')
-model.load_state_dict(checkpoint['model_state_dict'])
-# Continue training with your domain-specific data
-```
-## 📦 Upload to Hugging Face
-The project includes a **professional upload script** (✨ completely rewritten) for easy deployment:
-```bash
-cd upload_hf
-# Authenticate (first time only)
-huggingface-cli login
-# Upload everything
-python upload_to_huggingface.py --repo-id your-username/gap-clip --categories all
-# Or upload specific categories
-python upload_to_huggingface.py --repo-id your-username/gap-clip --categories models code
-# Create private repository
-python upload_to_huggingface.py --repo-id your-username/gap-clip --private
-```
-**Features**:
-- ✨ Object-oriented design with `HuggingFaceUploader` class
-- ✨ Multiple authentication methods (token, saved, interactive)
-- ✨ Category-based uploads: models, code, docs, data, optuna, evaluation
-- ✨ Progress tracking with tqdm
-- ✨ Automatic model card generation
-- ✨ Detailed error handling and recovery
-- ✨ Upload statistics and summary
-See `upload_hf/README_UPLOAD.md` for complete documentation.
-## 🧪 Testing & Evaluation
-### Quick Test
-```bash
-# Test configuration
-python -c "import config; config.print_config()"
-# Test model loading
-python example_usage.py --repo-id Leacb4/gap-clip --text "red dress"
-```
-### Full Evaluation Suite
-```bash
-# Run all evaluations
-cd evaluation
-python run_all_evaluations.py --repo-id Leacb4/gap-clip
-# Results will be saved to evaluation_results/ with:
-# - summary.json: Detailed metrics
-# - summary_comparison.png: Visual comparison
-```
-## 🐛 Known Issues & Fixes
-### Fixed Issues ✨
-1. **Color model image loading bug** (Fixed in `color_model.py`)
-   - Previous: `Image.open(config.column_local_image_path)`
-   - Fixed: `Image.open(img_path)` - Now correctly gets path from dataframe
-2. **Function naming in training** (Fixed in `main_model.py` and `train_main_model.py`)
-   - Previous: `train_one_epoch_enhanced`
-   - Fixed: `train_one_epoch` - Consistent naming
-3. **Device compatibility** (Improved in `config.py`)
-   - Now automatically detects and selects best device (CUDA > MPS > CPU)
-## 🎓 Learning Resources
-### Documentation Files
-- **README.md** (this file): Complete project documentation
-- **upload_hf/README_UPLOAD.md**: Upload guide for Hugging Face
-- **evaluation/**: Multiple evaluation examples
-### Code Examples
-- **example_usage.py**: Basic usage with Hugging Face Hub
-- **evaluation/fashion_search.py**: Interactive search examples
-- **evaluation/tsne_images.py**: Visualization examples
-## 🤝 Contributing
-We welcome contributions! Here's how:
-1. **Report bugs**: Open an issue with detailed description
-2. **Suggest features**: Describe your idea in an issue
-3. **Submit PR**: Fork, create branch, commit, and open pull request
-4. **Improve docs**: Help make documentation clearer
-### Development Setup
-```bash
-# Install with dev dependencies
-pip install -e ".[dev]"
-# Run tests (if available)
-pytest
-# Format code
-black .
-flake8 .
-```
-## 📊 Project Statistics
-- **Language**: Python 3.8+
-- **Framework**: PyTorch 2.0+
-- **Models**: 3 specialized models (color, hierarchy, main)
-- **Embedding Size**: 512 dimensions
-- **Training Data**: 20,000+ fashion items
-- **Lines of Code**: 5,000+ (including documentation)
-- **Documentation**: Comprehensive docstrings and guides
-## 🔗 Links
-- **Hugging Face Hub**: [Leacb4/gap-clip](https://huggingface.co/Leacb4/gap-clip)
-- **GitHub**: [github.com/Leacb4/gap-clip](https://github.com/Leacb4/gap-clip)
-- **Contact**: lea.attia@gmail.com
-## 📧 Contact & Support
-**Author**: Lea Attia Sarfati
-**Email**: lea.attia@gmail.com
-**Hugging Face**: [@Leacb4](https://huggingface.co/Leacb4)
-For questions, issues, or suggestions:
-- 🐛 **Bug reports**: Open an issue on GitHub
-- 💡 **Feature requests**: Open an issue with [Feature Request] tag
-- 📧 **Direct contact**: lea.attia@gmail.com
-- 💬 **Discussions**: Hugging Face Discussions
----
-## 📜 License
-This project is licensed under the MIT License - see the LICENSE file for details.
-## 🙏 Acknowledgments
-- LAION team for the base CLIP model
-- Hugging Face for transformers library and model hosting
-- PyTorch team for the deep learning framework
-- Fashion-MNIST dataset creators
-- All contributors and users of this project
----
-**⭐ If you find this project useful, please consider giving it a star on GitHub!**
-**📢 Version**: 1.0.0 | **Status**: Production Ready ✅ | **Last Updated**: December 2024

 ---
+language: en
+tags:
+- fashion
+- clip
+- multimodal
+- image-search
+- text-search
+- embeddings
+- contrastive-learning
+license: mit
+datasets:
+- custom
+metrics:
+- accuracy
+- cosine-similarity
+library_name: transformers
 ---
+# GAP-CLIP: Guaranteed Attribute Positioning in CLIP Embeddings
+This model is part of the GAP-CLIP project for fashion search with guaranteed attribute positioning.
+## Model Description
+GAP-CLIP is a multi-modal search model for fashion that combines:
+- **Color embeddings** (16 dimensions): Specialized for color representation
+- **Hierarchy embeddings** (64 dimensions): Specialized for category classification
+- **General CLIP embeddings** (432 dimensions): General visual-semantic understanding
+**Total embedding size**: 512 dimensions
+## Quick Start
 ```python
+from transformers import CLIPProcessor, CLIPModel
+from huggingface_hub import hf_hub_download
 import torch
+# Load model
+model = CLIPModel.from_pretrained("Leacb4/gap-clip")
+processor = CLIPProcessor.from_pretrained("laion/CLIP-ViT-B-32-laion2B-s34B-b79K")
+# Process text
+text = "red dress"
+inputs = processor(text=[text], return_tensors="pt", padding=True)
+text_features = model.get_text_features(**inputs)
+# Extract subspaces
+color_emb = text_features[:, :16]  # Color dimensions
+hierarchy_emb = text_features[:, 16:80]  # Hierarchy dimensions
+general_emb = text_features[:, 80:]  # General CLIP dimensions
 ```
+## Citation
 ```bibtex
 @misc{gap-clip-2024,
   title={GAP-CLIP: Guaranteed Attribute Positioning in CLIP Embeddings for Fashion Search},
   author={Sarfati, Lea Attia},
   year={2024},
+  url={https://huggingface.co/Leacb4/gap-clip}
 }
 ```
+## License
+MIT License - See LICENSE file for details.