---
title: CLIP Search Edit
emoji: 📚
colorFrom: gray
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit
---

# CLIP Search & Edit Engine

A multimodal application combining semantic image search with lightweight, text-guided image editing. This project utilizes OpenAI's CLIP model for retrieving images from the Flickr30k dataset and a custom FiLM-conditioned U-Net for performing style transfer based on text prompts.

## 🚀 Features

* **Semantic Image Search**: Search through thousands of images in the Flickr30k dataset using natural language queries (e.g., "a dog on a boat", "neon lights").
* **Text-Guided Image Editing**: Apply artistic styles (Sketch, Van Gogh, Cyberpunk) to images using a lightweight U-Net architecture conditioned on CLIP text embeddings.
* **Efficiency**: Includes benchmarking tools to measure model FLOPs, parameters, and inference speed.
* **Streamed Dataset**: Uses Deep Lake to stream dataset images, eliminating the need for massive local downloads.

## 🛠️ Installation

### Prerequisites
* Python 3.8+
* CUDA-enabled GPU (recommended for faster indexing and inference)
* **Git Xet**: Required for cloning large files from Hugging Face.

### Steps

1. **Setup Git Xet and Clone:**
   To clone the repository from Hugging Face Spaces, ensure `git-xet` is installed to handle large files.

<pre><code># Install git-xet (macOS example)
brew install git-xet
git xet install

# Clone the repository
git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit</code></pre>

   *Alternative: Clone without large files (pointers only):*
<pre><code>GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit</code></pre>

2. **Install dependencies:**
<pre><code>cd CLIP-Search-Edit
pip install -r requirements.txt</code></pre>


## ⚙️ Setup & Usage

### 1. Build the Search Index
Before using the search functionality, you must index the dataset. This script downloads image features from Flickr30k and saves them locally.

<pre><code>python indexer.py</code></pre>

* **Note:** This will create a file named `flickr_embeddings.pt` containing the CLIP embeddings for the images.
* **Config:** You can adjust `INDEX_LIMIT` in `indexer.py` to change the number of images indexed (default is 5,000).

### 2. Download/Place Model Weights
The image editing module requires pre-trained U-Net weights. Ensure the following files are in your root directory (or update the paths in `app.py`):
* `unet_charcoal-sketch.pth`
* `unet_van-gogh-painting.pth`
* `unet_neon-cyberpunk.pth`

### 3. Run the Application
Launch the Gradio web interface:

<pre><code>python app.py</code></pre>

* Open the link provided in the terminal (usually `http://127.0.0.1:7860`).
* **Tab 1 (Retrieval):** Enter a text query to find semantically related images.
* **Tab 2 (Image Editing):** Upload an image and select a style to transform it.

## 📊 Benchmarking & Evaluation

* **Measure Efficiency:**
Run the benchmark script to calculate the U-Net model's parameters, FLOPs, and FPS on CPU/GPU.
<pre><code>python benchmark_efficiency.py</code></pre>

* **Measure Search Recall:**
Calculate retrieval metrics (R@1, R@5, R@10) for the indexed dataset.
<pre><code>python measure_recall.py</code></pre>

## 📂 Project Structure

| File | Description |
| :--- | :--- |
| `app.py` | Main entry point. Launches the Gradio UI for Search and Editing. |
| `indexer.py` | Generates CLIP embeddings for the dataset and saves them to `flickr_embeddings.pt`. |
| `dataset.py` | Handles streaming images from the Deep Lake Flickr30k dataset. |
| `cnn1.py` | Defines the lightweight `UNet` architecture with FiLM layers for text conditioning. |
| `clip_styler.py` | Logic for applying style transfer using the U-Net and CLIP models. |
| `style_net.py` | Alternative/Legacy ResNet-based style network architecture. |
| `util.py` | Utility functions for image normalization, loading, and loss calculations. |
| `requirements.txt` | List of Python dependencies. |

## 🧠 Model Architecture

The editing module uses a custom **U-Net** (`cnn1.py`) enhanced with **FiLM (Feature-wise Linear Modulation)** layers.
1.  **Encoder:** Extracts image features.
2.  **FiLM Layers:** Modulate the feature maps based on the CLIP text embedding of the target style (e.g., "charcoal sketch").
3.  **Decoder:** Reconstructs the stylized image.

This approach allows for fast, text-controllable style transfer without requiring heavy diffusion models.