Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,332 @@
+# Fashion Search Model - GAP-CLIP
+
+Multimodal search model for fashion, combining color embeddings, categorical hierarchy embeddings, and a main CLIP model for fashion item search.
+
+## 📋 Description
+
+This project implements an advanced fashion search system based on CLIP, with three specialized models:
+
+1. **Color Model** (`color_model.pt`) : Specialized CLIP model for extracting reduced-size color embeddings from text and images
+2. **Hierarchy Model** (`hierarchy_model.pth`) : Model for classifying and encoding reduced-size categorical hierarchy of fashion items
+3. **Main CLIP Model** (`gap_clip.pth`) : Main CLIP model based on LAION, trained with color and hierarchy embeddings
+
+### Architecture
+
+The main model combines:
+- **16 dimensions** : Color embeddings (first 16 dimensions)
+- **64 dimensions** : Hierarchy embeddings (dimensions 16-80)
+- **512 dimensions** : Standard CLIP embeddings (following dimensions)
+
+Total : **512 dimensions** for each embedding
+
+## 🚀 Installation
+
+### Prerequisites
+
+- Python 3.8+
+- PyTorch 2.0+
+- CUDA (optional, for GPU)
+
+### Installing Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### Main Dependencies
+
+- `torch>=2.0.0` : PyTorch for deep learning
+- `transformers>=4.30.0` : Hugging Face Transformers for CLIP
+- `huggingface-hub>=0.16.0` : To download/upload models
+- `pillow>=9.0.0` : Image processing
+- `pandas>=1.5.0` : Data manipulation
+- `scikit-learn>=1.3.0` : Evaluation metrics
+
+## 📁 Project Structure
+
+```
+.
+├── color_model.py              # Color model definition
+├── hierarchy_model.py           # Hierarchy model definition
+├── main_model.py               # Main CLIP model and training functions
+├── config.py                   # Configuration for paths and parameters
+├── tokenizer_vocab.json        # Tokenizer vocabulary
+├── example_usage.py            # Usage examples
+├── Models/
+│   ├── color_model.pt          # Trained color model
+│   ├── hierarchy_model.pth     # Trained hierarchy model
+│   └── gap_clip.pth            # Main CLIP model
+├── Evaluation/                 # Evaluation scripts
+│   ├── evaluate_color_embeddings.py
+│   ├── main_model_evaluation.py
+│   └── ...
+├── Data/                       # Training data
+│   ├── download_data.py
+├── requirements.txt            # Python dependencies
+└── README.md                   # This documentation
+```
+
+## 🔧 Configuration
+
+Main parameters are defined in `config.py`:
+
+```python
+color_emb_dim = 16          # Color embedding dimension
+hierarchy_emb_dim = 64      # Hierarchy embedding dimension
+device = torch.device("mps")  # Device (cuda, mps, cpu)
+```
+
+### Model Paths
+
+Default paths are:
+- `Models/color_model.pt` : Color model
+- `Models/hierarchy_model.pth` : Hierarchy model
+- `Models/gap_clip.pth` : Main CLIP model
+- `tokenizer_vocab.json` : Tokenizer vocabulary
+
+## 📦 Usage
+
+### 1. Load Models from Hugging Face
+
+If your models are already uploaded to Hugging Face:
+
+```python
+from example_usage import load_models_from_hf
+
+# Load all models
+models = load_models_from_hf("your-username/your-model")
+
+color_model = models['color_model']
+hierarchy_model = models['hierarchy_model']
+main_model = models['main_model']
+processor = models['processor']
+device = models['device']
+```
+
+### 2. Text Search
+
+```python
+import torch
+from transformers import CLIPProcessor
+
+# Prepare text query
+text_query = "red dress"
+text_inputs = processor(text=[text_query], padding=True, return_tensors="pt")
+text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
+
+# Get main model embeddings
+with torch.no_grad():
+    outputs = main_model(**text_inputs)
+    text_features = outputs.text_embeds
+
+# Get specialized embeddings
+color_emb = color_model.get_text_embeddings([text_query])
+hierarchy_emb = hierarchy_model.get_text_embeddings([text_query])
+```
+
+### 3. Image Search
+
+```python
+from PIL import Image
+
+# Load image
+image = Image.open("path/to/image.jpg").convert("RGB")
+image_inputs = processor(images=[image], return_tensors="pt")
+image_inputs = {k: v.to(device) for k, v in image_inputs.items()}
+
+# Get embeddings
+with torch.no_grad():
+    outputs = main_model(**image_inputs)
+    image_features = outputs.image_embeds
+```
+
+### 4. Using the Example Script
+
+```bash
+python example_usage.py \
+    --repo-id your-username/your-model \
+    --text "blue jacket" \
+    --image path/to/image.jpg
+```
+
+## 🎯 Model Training
+
+### Train the Color Model
+
+```python
+from color_model import ColorCLIP, train_color_model
+
+# Configuration
+model = ColorCLIP(vocab_size=10000, embedding_dim=16)
+# ... dataset configuration ...
+
+# Training
+train_color_model(model, train_loader, val_loader, num_epochs=20)
+```
+
+### Train the Hierarchy Model
+
+```python
+from hierarchy_model import Model as HierarchyModel, train_hierarchy_model
+
+# Configuration
+model = HierarchyModel(num_hierarchy_classes=10, embed_dim=64)
+# ... dataset configuration ...
+
+# Training
+train_hierarchy_model(model, train_loader, val_loader, num_epochs=20)
+```
+
+### Train the Main CLIP Model
+
+The main model trains with both specialized models:
+
+```bash
+python main_model.py
+```
+
+**Training Parameters** (in `main_model.py`):
+- `num_epochs = 20` : Number of epochs
+- `learning_rate = 1e-5` : Learning rate
+- `temperature = 0.07` : Temperature for contrastive loss
+- `alignment_weight = 0.5` : Weight for embedding alignment
+- `batch_size = 32` : Batch size
+- `use_enhanced_loss = True` : Use enhanced loss with alignment
+
+**Features**:
+- Triple contrastive loss (text-image-attributes)
+- Direct alignment between specialized models and main model
+- Early stopping with patience
+- Automatic learning rate reduction
+- Automatic best model saving
+
+
+## 📊 Models
+
+### Color Model
+
+- **Architecture** : ResNet18 (image encoder) + Embedding (text encoder)
+- **Embedding dimension** : 16
+- **Trained on** : Fashion data with color annotations
+- **Usage** : Extract color embeddings from text or images
+
+### Hierarchy Model
+
+- **Architecture** : ResNet18 (image encoder) + Embedding (hierarchy encoder)
+- **Embedding dimension** : 64
+- **Hierarchy classes** : shirt, dress, pant, shoe, bag, etc.
+- **Usage** : Classify and encode categorical hierarchy
+
+### Main CLIP Model
+
+- **Architecture** : CLIP ViT-B/32 (LAION)
+- **Base** : `laion/CLIP-ViT-B-32-laion2B-s34B-b79K`
+- **Trained with** : Triple contrastive loss including color and hierarchy embeddings
+- **Dimensions** : 592 (512 CLIP + 16 color + 64 hierarchy)
+- **Features** :
+  - Text-image search
+  - Specialized embeddings (color + hierarchy)
+  - Alignment with specialized models
+
+## 🔍 Advanced Usage Examples
+
+### Search with Combined Embeddings
+
+```python
+import torch.nn.functional as F
+
+# Text query
+text_query = "red dress"
+text_inputs = processor(text=[text_query], padding=True, return_tensors="pt")
+text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
+
+# Main model embeddings
+with torch.no_grad():
+    outputs = main_model(**text_inputs)
+    text_features = outputs.text_embeds
+
+# Extract specialized embeddings from main model
+main_color_emb = text_features[:, :16]  # First 16 dimensions
+main_hierarchy_emb = text_features[:, 16:80]  # Dimensions 16-80
+main_clip_emb = text_features[:, 80:]  # CLIP dimensions (80+)
+
+# Compare with specialized models
+color_emb = color_model.get_text_embeddings([text_query])
+hierarchy_emb = hierarchy_model.get_text_embeddings([text_query])
+
+# Cosine similarity
+color_similarity = F.cosine_similarity(color_emb, main_color_emb, dim=1)
+hierarchy_similarity = F.cosine_similarity(hierarchy_emb, main_hierarchy_emb, dim=1)
+```
+
+### Search in an Image Database
+
+```python
+import numpy as np
+
+# Load all images from database
+image_paths = [...]  # List of image paths
+image_features_list = []
+
+for img_path in image_paths:
+    image = Image.open(img_path).convert("RGB")
+    image_inputs = processor(images=[image], return_tensors="pt")
+    image_inputs = {k: v.to(device) for k, v in image_inputs.items()}
+    
+    with torch.no_grad():
+        outputs = main_model(**image_inputs)
+        features = outputs.image_embeds
+        image_features_list.append(features.cpu().numpy())
+
+# Convert to numpy array
+image_features = np.vstack(image_features_list)
+
+# Search
+query = "red dress"
+# ... get text_features ...
+
+# Calculate similarities
+similarities = F.cosine_similarity(
+    text_features, 
+    torch.from_numpy(image_features).to(device), 
+    dim=1
+)
+
+# Sort by similarity
+top_k = 10
+top_indices = similarities.argsort(descending=True)[:top_k]
+```
+
+## 📝 Evaluation
+
+Evaluation scripts are available in `Evaluation/`:
+
+- `evaluate_color_embeddings.py` : Color embeddings evaluation
+- `main_model_evaluation.py` : Main model evaluation
+- `fashion_search.py` : Fashion search tests
+
+
+## 📄 Citation
+
+If you use these models, please cite:
+
+```bibtex
+@misc{fashion-search-model,
+  title={GAP (Guaranteed Attribute Position) CLIP: A Multi-Loss Framework for Attribute-Aware Fashion Embeddings},
+  author={ },
+  year={2024},
+  howpublished={\url{https://huggingface.co/Leacb4/gap-clip}}
+}
+```
+
+## 🤝 Contributing
+
+Contributions are welcome! Feel free to open an issue or a pull request.
+
+## 📧 Contact
+
+Lea Sarfati lea.attia@gmail.com
+
+---
+
+**Note** : This project is under active development. For any questions or issues, please open an issue on the repository.