README.md

metadata

title: CLIP Search Edit
emoji: 📚
colorFrom: gray
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit

CLIP Search & Edit Engine

A multimodal application combining semantic image search with lightweight, text-guided image editing. This project utilizes OpenAI's CLIP model for retrieving images from the Flickr30k dataset and a custom FiLM-conditioned U-Net for performing style transfer based on text prompts.

🚀 Features

• Semantic Image Search: Search through thousands of images in the Flickr30k dataset using natural language queries (e.g., "a dog on a boat", "neon lights").
• Text-Guided Image Editing: Apply artistic styles (Sketch, Van Gogh, Cyberpunk) to images using a lightweight U-Net architecture conditioned on CLIP text embeddings.
• Efficiency: Includes benchmarking tools to measure model FLOPs, parameters, and inference speed.
• Streamed Dataset: Uses Deep Lake to stream dataset images, eliminating the need for massive local downloads.

🛠️ Installation

Prerequisites

• Python 3.8+
• CUDA-enabled GPU (recommended for faster indexing and inference)
• Git Xet: Required for cloning large files from Hugging Face.

Steps

1. Setup Git Xet and Clone: To clone the repository from Hugging Face Spaces, ensure git-xet is installed to handle large files.

# Install git-xet (macOS example)
brew install git-xet
git xet install

# Clone the repository
git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit

Alternative: Clone without large files (pointers only):

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit

2. Install dependencies:

   cd CLIP-Search-Edit
   pip install -r requirements.txt

⚙️ Setup & Usage

1. Build the Search Index

Before using the search functionality, you must index the dataset. This script downloads image features from Flickr30k and saves them locally.

python indexer.py

• Note: This will create a file named flickr_embeddings.pt containing the CLIP embeddings for the images.
• Config: You can adjust INDEX_LIMIT in indexer.py to change the number of images indexed (default is 5,000).

2. Download/Place Model Weights

The image editing module requires pre-trained U-Net weights. Ensure the following files are in your root directory (or update the paths in app.py):

• unet_charcoal-sketch.pth
• unet_van-gogh-painting.pth
• unet_neon-cyberpunk.pth

3. Run the Application

Launch the Gradio web interface:

python app.py

• Open the link provided in the terminal (usually http://127.0.0.1:7860).
• Tab 1 (Retrieval): Enter a text query to find semantically related images.
• Tab 2 (Image Editing): Upload an image and select a style to transform it.

📊 Benchmarking & Evaluation

• Measure Efficiency: Run the benchmark script to calculate the U-Net model's parameters, FLOPs, and FPS on CPU/GPU.

  python benchmark_efficiency.py

• Measure Search Recall: Calculate retrieval metrics (R@1, R@5, R@10) for the indexed dataset.

  python measure_recall.py

📂 Project Structure

File	Description
app.py	Main entry point. Launches the Gradio UI for Search and Editing.
indexer.py	Generates CLIP embeddings for the dataset and saves them to flickr_embeddings.pt.
dataset.py	Handles streaming images from the Deep Lake Flickr30k dataset.
cnn1.py	Defines the lightweight UNet architecture with FiLM layers for text conditioning.
clip_styler.py	Logic for applying style transfer using the U-Net and CLIP models.
style_net.py	Alternative/Legacy ResNet-based style network architecture.
util.py	Utility functions for image normalization, loading, and loss calculations.
requirements.txt	List of Python dependencies.

🧠 Model Architecture

The editing module uses a custom U-Net (cnn1.py) enhanced with FiLM (Feature-wise Linear Modulation) layers.

1. Encoder: Extracts image features.
2. FiLM Layers: Modulate the feature maps based on the CLIP text embedding of the target style (e.g., "charcoal sketch").
3. Decoder: Reconstructs the stylized image.

This approach allows for fast, text-controllable style transfer without requiring heavy diffusion models.