Spaces:
Sleeping
Sleeping
| title: CLIP Search Edit | |
| emoji: 📚 | |
| colorFrom: gray | |
| colorTo: purple | |
| sdk: gradio | |
| sdk_version: 5.49.1 | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| # CLIP Search & Edit Engine | |
| A multimodal application combining semantic image search with lightweight, text-guided image editing. This project utilizes OpenAI's CLIP model for retrieving images from the Flickr30k dataset and a custom FiLM-conditioned U-Net for performing style transfer based on text prompts. | |
| ## 🚀 Features | |
| * **Semantic Image Search**: Search through thousands of images in the Flickr30k dataset using natural language queries (e.g., "a dog on a boat", "neon lights"). | |
| * **Text-Guided Image Editing**: Apply artistic styles (Sketch, Van Gogh, Cyberpunk) to images using a lightweight U-Net architecture conditioned on CLIP text embeddings. | |
| * **Efficiency**: Includes benchmarking tools to measure model FLOPs, parameters, and inference speed. | |
| * **Streamed Dataset**: Uses Deep Lake to stream dataset images, eliminating the need for massive local downloads. | |
| ## 🛠️ Installation | |
| ### Prerequisites | |
| * Python 3.8+ | |
| * CUDA-enabled GPU (recommended for faster indexing and inference) | |
| * **Git Xet**: Required for cloning large files from Hugging Face. | |
| ### Steps | |
| 1. **Setup Git Xet and Clone:** | |
| To clone the repository from Hugging Face Spaces, ensure `git-xet` is installed to handle large files. | |
| <pre><code># Install git-xet (macOS example) | |
| brew install git-xet | |
| git xet install | |
| # Clone the repository | |
| git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit</code></pre> | |
| *Alternative: Clone without large files (pointers only):* | |
| <pre><code>GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/spaces/CISC473-Group19/CLIP-Search-Edit</code></pre> | |
| 2. **Install dependencies:** | |
| <pre><code>cd CLIP-Search-Edit | |
| pip install -r requirements.txt</code></pre> | |
| ## ⚙️ Setup & Usage | |
| ### 1. Build the Search Index | |
| Before using the search functionality, you must index the dataset. This script downloads image features from Flickr30k and saves them locally. | |
| <pre><code>python indexer.py</code></pre> | |
| * **Note:** This will create a file named `flickr_embeddings.pt` containing the CLIP embeddings for the images. | |
| * **Config:** You can adjust `INDEX_LIMIT` in `indexer.py` to change the number of images indexed (default is 5,000). | |
| ### 2. Download/Place Model Weights | |
| The image editing module requires pre-trained U-Net weights. Ensure the following files are in your root directory (or update the paths in `app.py`): | |
| * `unet_charcoal-sketch.pth` | |
| * `unet_van-gogh-painting.pth` | |
| * `unet_neon-cyberpunk.pth` | |
| ### 3. Run the Application | |
| Launch the Gradio web interface: | |
| <pre><code>python app.py</code></pre> | |
| * Open the link provided in the terminal (usually `http://127.0.0.1:7860`). | |
| * **Tab 1 (Retrieval):** Enter a text query to find semantically related images. | |
| * **Tab 2 (Image Editing):** Upload an image and select a style to transform it. | |
| ## 📊 Benchmarking & Evaluation | |
| * **Measure Efficiency:** | |
| Run the benchmark script to calculate the U-Net model's parameters, FLOPs, and FPS on CPU/GPU. | |
| <pre><code>python benchmark_efficiency.py</code></pre> | |
| * **Measure Search Recall:** | |
| Calculate retrieval metrics (R@1, R@5, R@10) for the indexed dataset. | |
| <pre><code>python measure_recall.py</code></pre> | |
| ## 📂 Project Structure | |
| | File | Description | | |
| | :--- | :--- | | |
| | `app.py` | Main entry point. Launches the Gradio UI for Search and Editing. | | |
| | `indexer.py` | Generates CLIP embeddings for the dataset and saves them to `flickr_embeddings.pt`. | | |
| | `dataset.py` | Handles streaming images from the Deep Lake Flickr30k dataset. | | |
| | `cnn1.py` | Defines the lightweight `UNet` architecture with FiLM layers for text conditioning. | | |
| | `clip_styler.py` | Logic for applying style transfer using the U-Net and CLIP models. | | |
| | `style_net.py` | Alternative/Legacy ResNet-based style network architecture. | | |
| | `util.py` | Utility functions for image normalization, loading, and loss calculations. | | |
| | `requirements.txt` | List of Python dependencies. | | |
| ## 🧠 Model Architecture | |
| The editing module uses a custom **U-Net** (`cnn1.py`) enhanced with **FiLM (Feature-wise Linear Modulation)** layers. | |
| 1. **Encoder:** Extracts image features. | |
| 2. **FiLM Layers:** Modulate the feature maps based on the CLIP text embedding of the target style (e.g., "charcoal sketch"). | |
| 3. **Decoder:** Reconstructs the stylized image. | |
| This approach allows for fast, text-controllable style transfer without requiring heavy diffusion models. |