README.md

metadata

title: Wonder Finder
emoji: 🌍
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 4.44.1
python_version: '3.10'
app_file: app.py
pinned: false
license: mit

🌍 Wonder Finder

A cross-modal visual recommender for the 12 Wonders of the World. Upload a travel photo or describe a place in words — the app returns the three most semantically similar wonders from a catalog of 11,544 images.

Built for Assignment 3 (Embeddings, RecSys, Spaces).

---

📁 Dataset

Source: chavajaz/wonders_dataset on HuggingFace Hub License: CC0-1.0 (public domain) Size: 11,544 images across 12 classes (train: 3,852 · validation: 3,846 · test: 3,846 — all combined into one retrieval catalog) Modality: Image-only (vision) Features: image (PIL, variable resolution) + label (integer 0–11)

The 12 wonder classes

Burj Khalifa · Chichén Itzá · Christ the Redeemer · Eiffel Tower · Great Wall of China · Machu Picchu · Pyramids of Giza · Roman Colosseum · Statue of Liberty · Stonehenge · Taj Mahal · Venezuela's Angel Falls

Why this dataset

Famous landmarks are an ideal testbed for CLIP-based retrieval: their visual identity is distinctive, CLIP's pretraining corpus contained many examples (so zero-shot recognition is strong), and the 12 ground-truth labels give us a way to quantitatively evaluate the unsupervised clustering against true categories. The CC0 license eliminates IP concerns for a public Space.

---

🔍 Exploratory Data Analysis

Class distribution

The dataset is moderately imbalanced. Eight classes are well-represented (~1,000–1,200 images each), while four are notably smaller:

Group	Approximate count	Wonders
Well-represented	~1,000–1,200	Burj Khalifa, Chichén Itzá, Christ the Redeemer, Eiffel Tower, Great Wall of China, Machu Picchu, Pyramids of Giza, Roman Colosseum
Under-represented	~475–760	Statue of Liberty (720), Venezuela's Angel Falls (760), Stonehenge (610), **Taj Mahal (475)**

Implication: Taj Mahal has roughly 40% the volume of an average well-represented class. We predicted this would weaken its cluster recovery downstream — a prediction that was later confirmed in clustering analysis (see below).

Image resolution

Image dimensions are right-skewed with median dimensions around 300 × 300 pixels, ranging from ~100 px up to ~4500 × 3700 px. The tight cluster around 300 px and long tail suggest the dataset was scraped from web thumbnails and social media rather than curated photography collections. This doesn't affect embedding quality — CLIP's preprocessor resizes all inputs to 224 × 224 regardless — but it's a relevant note on data provenance.

Visual sanity check

Sampling 30 random images and four per class confirmed:

• ✅ All sampled images load and decode successfully via PIL
• ✅ Class labels visually match the images
• ⚠️ A small number of corrupted images exist (e.g. one all-black sample surfaced during cluster inspection, demonstrating that semantic clustering doubles as a data-quality audit)

---

🧠 Embeddings

Model: CLIP ViT-B/32

We use openai/clip-vit-base-patch32 (~151M parameters, 605 MB) for three specific reasons:

1. Small-medium model — fits comfortably on the Colab T4 used for embedding the catalog, and runs cheaply on a CPU-basic HF Space for inference.
2. Joint image–text embedding space — CLIP was trained contrastively on 400M image-caption pairs, so images and natural-language descriptions live in the same 512-D vector space. This is what makes our dual-input feature (image OR text → same recommender) possible with a single model.
3. Landmark coverage — CLIP's pretraining data included famous wonders, so zero-shot text queries like "ancient stone pyramid" or "waterfall in the jungle" surface the right results without any fine-tuning.

Embedding pipeline

1. Each image is converted to RGB, resized to 224 × 224 by CLIP's processor, and passed through the vision encoder.
2. The resulting 512-D vector is L2-normalized so that cosine similarity reduces to a dot product (faster and the standard convention for CLIP).
3. All 11,544 embeddings are saved to a single wonders_embeddings.parquet file (~25 MB), which the Space loads on startup into a GPU/CPU tensor for fast retrieval.

---

📊 Clustering analysis

We ran two clustering algorithms (K-Means and DBSCAN) on the 512-D embeddings, with UMAP used purely for 2-D visualization. The 12 ground-truth labels were not shown to either algorithm — they're used only to evaluate the results.

K-Means (k=12) — primary result

Metric	Value	Interpretation
Silhouette score	0.233	Moderate intrinsic cohesion (typical for 512-D image embeddings)
Adjusted Rand Index (ARI)	0.890	Near-perfect alignment with ground-truth wonder classes
Normalized Mutual Information (NMI)	0.927	Strong information-theoretic agreement with true labels

Headline finding: unsupervised K-Means almost perfectly rediscovered the 12 wonder categories without ever seeing the labels. This tells us CLIP's pretrained embedding space already encodes the semantic structure of world landmarks. The choice of k=12 is supported empirically by a clear elbow at k=12 in the inertia plot.

Cluster purity (most clusters separated cleanly; two are interestingly mixed)

Cluster	Size	Purity	Dominant class	Note
7	912	100.0%	Christ the Redeemer	Perfect separation
8	903	100.0%	Burj Khalifa	Perfect separation
11	1,170	100.0%	Roman Colosseum	Perfect separation
1	1,131	99.7%	Machu Picchu	Near-perfect
3	1,107	99.5%	Pyramids of Giza	Near-perfect
4	699	99.6%	Statue of Liberty	Near-perfect
5	1,131	99.5%	Great Wall of China	Near-perfect
9	612	99.5%	Stonehenge	Near-perfect
10	1,026	98.5%	Chichén Itzá	Near-perfect
0	777	96.9%	Angel Falls	Slight Great Wall bleed
6	1,578	70.5%	Eiffel Tower (+ Taj Mahal absorbed)	See below
2	498	53.6%	Burj Khalifa + Eiffel Tower + Christ Redeemer mix	See below

Interpreting the two mixed clusters

• Cluster 6 (70.5% purity) absorbed Taj Mahal (~462 of its ~475 images) into the Eiffel Tower cluster. This directly confirms the EDA prediction: with only ~40% the volume of well-represented classes, Taj Mahal's centroid couldn't establish itself as its own cluster.

• Cluster 2 (53.6% purity) grouped together vertical/tall silhouettes: Burj Khalifa (skyscraper), Eiffel Tower (lattice tower), and Christ the Redeemer (tall vertical statue). CLIP appears to encode "tall vertical structure" as a shared visual feature that cuts across our human-labeled categories. This is a real semantic insight, not a bug.

DBSCAN comparison

We swept DBSCAN's eps hyperparameter from 0.10 to 0.25. Best result: eps=0.10, min_samples=15 → 11 clusters, 24.3% noise, ARI = 0.750.

K-Means outperformed DBSCAN on this dataset (ARI 0.890 vs 0.750). Two reasons:

1. The wonder classes are roughly balanced in size and form globular clusters in embedding space — favorable conditions for K-Means.
2. DBSCAN had to discard 24.3% of points as noise to achieve its score; K-Means used the full catalog.

DBSCAN's strength is finding arbitrarily-shaped clusters and rejecting outliers, but here the trade-off (losing a quarter of the catalog) isn't worth it.

---

🎯 Inputs & Outputs (the recommender)

The app accepts either an image or a text query. Both are embedded via the same CLIP model into the same 512-D space, then ranked against the catalog by cosine similarity. A diversity filter (similarity threshold 0.98) suppresses near-duplicate results so users don't get three copies of the same photo back.

Catalog quality observation

During testing we discovered the catalog contains exact duplicate images — some landmark photos have ~3 copies scraped from multiple web sources. Cosine similarity surfaces these immediately (a useful side-effect of semantic search for data-quality audits). The diversity filter mitigates this at query time.

Example queries (worth trying)

• "ancient stone pyramid in the desert" → Pyramids of Giza
• "tall modern skyscraper at night" → Burj Khalifa
• "waterfall in the tropical jungle" → Angel Falls
• "ancient Roman amphitheater" → Roman Colosseum
• "statue of a religious figure with outstretched arms" → Christ the Redeemer

---

🛠️ How to use the app

1. Open the App tab above
2. Choose 📷 Search by Image to upload a photo, or 💬 Search by Description to type a description
3. Click Find Similar Wonders / Find Matching Wonders
4. The top-3 matches appear as a gallery with similarity percentages

---

🎬 Project presentation

A short walkthrough is embedded at the bottom of the app, covering dataset selection, embedding analysis, clustering findings, and a live demo.

---

📦 Technical stack

• Embedding model: openai/clip-vit-base-patch32 (HuggingFace Transformers)
• Dataset loader: HuggingFace datasets
• Clustering & viz: scikit-learn (K-Means, DBSCAN, silhouette/ARI/NMI metrics), UMAP, matplotlib
• App framework: Gradio 4.44.1
• Deployment: HuggingFace Spaces (CPU basic, Python 3.10)

---

⚠️ Notes on deployment

This Space pins specific versions of gradio, gradio-client, fastapi, and starlette because Gradio 4.44.x has a known incompatibility with newer Starlette versions, and gradio_client 1.3.0 has a schema-generation bug on certain component types. Defensive monkey-patches in app.py further harden the Space against the schema bug. See requirements.txt for the exact compatible version set.