Spaces:
Sleeping
Sleeping
File size: 4,631 Bytes
6878318 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | ---
title: CLIP Search Edit
emoji: 📚
colorFrom: gray
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit
---
# CLIP Search & Edit Engine
A multimodal application combining semantic image search with lightweight, text-guided image editing. This project utilizes OpenAI's CLIP model for retrieving images from the Flickr30k dataset and a custom FiLM-conditioned U-Net for performing style transfer based on text prompts.
## 🚀 Features
* **Semantic Image Search**: Search through thousands of images in the Flickr30k dataset using natural language queries (e.g., "a dog on a boat", "neon lights").
* **Text-Guided Image Editing**: Apply artistic styles (Sketch, Van Gogh, Cyberpunk) to images using a lightweight U-Net architecture conditioned on CLIP text embeddings.
* **Efficiency**: Includes benchmarking tools to measure model FLOPs, parameters, and inference speed.
* **Streamed Dataset**: Uses Deep Lake to stream dataset images, eliminating the need for massive local downloads.
## 🛠️ Installation
### Prerequisites
* Python 3.8+
* CUDA-enabled GPU (recommended for faster indexing and inference)
* **Git Xet**: Required for cloning large files from Hugging Face.
### Steps
1. **Setup Git Xet and Clone:**
To clone the repository from Hugging Face Spaces, ensure `git-xet` is installed to handle large files.
<pre><code># Install git-xet (macOS example)
brew install git-xet
git xet install
# Clone the repository
git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit</code></pre>
*Alternative: Clone without large files (pointers only):*
<pre><code>GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit</code></pre>
2. **Install dependencies:**
<pre><code>cd CLIP-Search-Edit
pip install -r requirements.txt</code></pre>
## ⚙️ Setup & Usage
### 1. Build the Search Index
Before using the search functionality, you must index the dataset. This script downloads image features from Flickr30k and saves them locally.
<pre><code>python indexer.py</code></pre>
* **Note:** This will create a file named `flickr_embeddings.pt` containing the CLIP embeddings for the images.
* **Config:** You can adjust `INDEX_LIMIT` in `indexer.py` to change the number of images indexed (default is 5,000).
### 2. Download/Place Model Weights
The image editing module requires pre-trained U-Net weights. Ensure the following files are in your root directory (or update the paths in `app.py`):
* `unet_charcoal-sketch.pth`
* `unet_van-gogh-painting.pth`
* `unet_neon-cyberpunk.pth`
### 3. Run the Application
Launch the Gradio web interface:
<pre><code>python app.py</code></pre>
* Open the link provided in the terminal (usually `http://127.0.0.1:7860`).
* **Tab 1 (Retrieval):** Enter a text query to find semantically related images.
* **Tab 2 (Image Editing):** Upload an image and select a style to transform it.
## 📊 Benchmarking & Evaluation
* **Measure Efficiency:**
Run the benchmark script to calculate the U-Net model's parameters, FLOPs, and FPS on CPU/GPU.
<pre><code>python benchmark_efficiency.py</code></pre>
* **Measure Search Recall:**
Calculate retrieval metrics (R@1, R@5, R@10) for the indexed dataset.
<pre><code>python measure_recall.py</code></pre>
## 📂 Project Structure
| File | Description |
| :--- | :--- |
| `app.py` | Main entry point. Launches the Gradio UI for Search and Editing. |
| `indexer.py` | Generates CLIP embeddings for the dataset and saves them to `flickr_embeddings.pt`. |
| `dataset.py` | Handles streaming images from the Deep Lake Flickr30k dataset. |
| `cnn1.py` | Defines the lightweight `UNet` architecture with FiLM layers for text conditioning. |
| `clip_styler.py` | Logic for applying style transfer using the U-Net and CLIP models. |
| `style_net.py` | Alternative/Legacy ResNet-based style network architecture. |
| `util.py` | Utility functions for image normalization, loading, and loss calculations. |
| `requirements.txt` | List of Python dependencies. |
## 🧠 Model Architecture
The editing module uses a custom **U-Net** (`cnn1.py`) enhanced with **FiLM (Feature-wise Linear Modulation)** layers.
1. **Encoder:** Extracts image features.
2. **FiLM Layers:** Modulate the feature maps based on the CLIP text embedding of the target style (e.g., "charcoal sketch").
3. **Decoder:** Reconstructs the stylized image.
This approach allows for fast, text-controllable style transfer without requiring heavy diffusion models. |