--- title: Nature Recommender emoji: 👀 colorFrom: green colorTo: yellow sdk: gradio sdk_version: 6.14.0 python_version: '3.13' app_file: app.py pinned: false license: mit --- # 🌿 Nature Scene Recommender A visual similarity search app — upload a nature image and get the 3 most similar scenes from a dataset of 15,000 nature images, using CLIP embeddings and cosine similarity. --- ## 🎬 Presentation Video ## 📁 Part 1: Dataset **Dataset:** [mertcobanov/nature-dataset](https://huggingface.co/datasets/mertcobanov/nature-dataset) **Source:** Hugging Face — a curated collection of high-quality nature and landscape images. **Size:** 50,000 images with natural scene captions. **Features:** - `image` — RGB photo of a natural landscape (512x512px) - `caption` — a short textual description of the scene (e.g., "a large mountain with a very tall peak") **Use case:** Given an input landscape image, the app finds the Top 3 most visually similar nature scenes using image embeddings and cosine similarity. --- ## 📊 Part 2: Exploratory Data Analysis (EDA) The dataset contains 50,000 nature images — mountains, deserts, forests, waterfalls and beaches — each with a text caption. Every image is 512x512px. **Caption statistics:** - Total captions: 50,000 - Average caption length: ~10 words - Min: 4 words | Max: 16 words **Sanity Checks:** No missing images, no missing captions, all images are RGB and the same size. The dataset is clean and ready. ### Most Common Words in Captions The 15 most common words reveal the diversity of the dataset — mountain, desert, trees, waterfall, forest... ![Most Common Words](pic1.png) ### Sample Images from Dataset 8 sample images showing the variety of nature scenes in the dataset, each with its caption. ![Sample Images](pic2.png) --- ## 🧠 Part 3: Embeddings **Model:** `openai/clip-vit-base-patch32` — trained on millions of image-text pairs, enabling it to understand complex visual relationships. **Sample size:** 15,000 images out of 50,000. **Embedding size:** 768 dimensions per image — each image is converted to a numerical vector representing its visual content. **How it works:** Two images of mountains will get very similar vectors. A mountain image and a beach image will get very different vectors. ### 3.1 PCA Visualization PCA reduces the 768 dimensions to 2 for visualization. The first plot shows the sample index as color, the second shows the K-Means cluster assignment. ![PCA - Sample Index](pic3.png) ### 3.2 K-Means Clustering Applied K-Means with **8 clusters**. Each cluster represents a different type of nature scene. ![PCA - K-Means Clusters](pic4.png) ### t-SNE Visualization t-SNE provides a clearer picture of the clusters — the 8 groups are much more distinct here. CLIP successfully grouped visually and semantically similar nature scenes together without any manual labeling. ![t-SNE Clusters](pic5.png) **Cluster distribution:** | Cluster | Size | Theme | |---------|------|-------| | Cluster 0 | 1,487 | 🏜️ Desert & People | | Cluster 1 | 3,261 | 🏔️💧 Waterfalls & Mountains | | Cluster 2 | 900 | 🌲 Dense Forests & Trees | | Cluster 3 | 1,810 | 🪨 Rocky Landscapes | | Cluster 4 | 1,016 | 🌊 Coastal & Open Scenes | | Cluster 5 | 2,042 | ⬛ Mixed B&W & Rocky | | Cluster 6 | 1,831 | 🌵 Desert Vegetation | | Cluster 7 | 2,653 | ⛰️ Mountain Forests | ### 3.4 Saved Embeddings The embeddings were saved to a `nature_embeddings.parquet` file containing the original image index, caption, and embedding vector — so the Space can use them directly without recomputing every time. --- ## ⚙️ Part 4: Inputs & Outputs ### 4.1 Prepare Embeddings Loaded the embeddings file — extracting the embedding matrix, captions, and original image indices. - Loaded embeddings shape: **(15,000, 768)** ### 4.2 User Input A function that converts any uploaded image into a 768-dimensional embedding vector using CLIP — exactly like we did for the dataset images. ### 4.3 Similarity Scores A function that calculates cosine similarity between the uploaded image's vector and all 15,000 vectors in the dataset — returning a score between 0 (completely different) and 1 (identical). ### 4.4 Top 3 Elements The main recommendation function — converts the image to an embedding, calculates similarity against all 15,000 images, and returns the 3 most similar ones with their captions and similarity scores. --- ## 🚀 Part 5: HuggingFace Space The app is built with **Gradio** and hosted on HuggingFace Spaces. **How to use:** 1. Upload a nature image 2. Click **Find Similar Scenes 🔍** 3. Get the Top 3 most similar scenes with scene type, caption, and similarity score 🔗 [Open the App](https://huggingface.co/spaces/Danielhalali/nature-recommender) --- ## 🎁 Bonus ### Bonus 1: Similarity Score Visualization Visualizing the top 10 cosine similarity scores for a query image — higher scores indicate greater visual similarity between embeddings. ![Similarity Scores](pic6.png) ### Bonus 2: Auto-Classification of Nature Scenes Each image is automatically classified into a nature scene category based on caption keywords: Mountain, Water, Forest, Desert, Coastal. **Distribution:** - 🏔️ Mountain: 6,796 images (45.3%) - 💧 Water: 3,147 images (21.0%) - 🌲 Forest: 1,835 images (12.2%) - 🌿 Other Nature: 1,341 images (8.9%) - 🏜️ Desert: 1,323 images (8.8%) - 🌊 Coastal: 558 images (3.7%) ![Scene Classification](pic7.png) ### Bonus 3: Interactive Cluster Visualization An interactive t-SNE plot where each point represents an image embedding, colored by cluster. Hover over any point to see the image caption. ![Interactive t-SNE](pic8.png) --- ## 📝 Submission Submitted on Moodle: `https://huggingface.co/spaces/Danielhalali/nature-recommender`