data/knowledge_base/clustering_guide.md

Clustering Guide

When to Use Clustering

Clustering is unsupervised — there is no target variable. Use it to:

• Discover natural groupings in data (customer segments, document topics)
• Compress or summarize data before supervised learning
• Anomaly detection (points that don't belong to any cluster)
• Preprocessing for semi-supervised learning

Clustering is exploratory. Results depend heavily on the algorithm, distance metric, and hyperparameters. Always validate clusters with domain knowledge, not just metrics.

---

K-Means

How it works: Assign each point to the nearest centroid. Recompute centroids as cluster means. Repeat until convergence.

Strengths: Fast, scalable, simple to understand. Works well when clusters are roughly spherical and similar in size.

Weaknesses: Must specify K in advance. Sensitive to outliers (centroid is pulled toward them). Fails on non-spherical clusters (crescents, rings). Assumes clusters have similar variance.

Hyperparameters:

• n_clusters (K): Most important parameter. Choose via elbow method or silhouette score.
• n_init: Number of random initializations (default 10). Run multiple times, keep best inertia.
• init: "k-means++" (default) is much better than random initialization.
• max_iter: Convergence iterations (default 300 is usually enough).

Choosing K — elbow method: Plot inertia (within-cluster sum of squares) vs K. Look for the "elbow" where adding more clusters gives diminishing returns. The elbow is often subjective — silhouette score is more reliable.

Choosing K — silhouette score: Silhouette score ranges from -1 to 1. Higher = better defined clusters. Try K from 2 to 10, pick the K with the highest average silhouette.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

scores = []
for k in range(2, 11):
    km = KMeans(n_clusters=k, init="k-means++", n_init=10, random_state=42)
    labels = km.fit_predict(X_scaled)
    scores.append(silhouette_score(X_scaled, labels))
best_k = range(2, 11)[scores.index(max(scores))]

Always scale features before K-Means. Euclidean distance is not meaningful when features have different units or scales.

---

DBSCAN

How it works: Groups points that are densely packed together (within eps distance and at least min_samples neighbors). Points in low-density regions are labeled as noise (-1).

Strengths: Does not require specifying K. Finds arbitrarily shaped clusters. Naturally handles outliers — they become noise points rather than distorting clusters.

Weaknesses: Sensitive to eps and min_samples. Struggles when clusters have very different densities. Poor performance in high dimensions (curse of dimensionality makes all distances similar).

Hyperparameters:

• eps: Neighborhood radius. Critical. Too small → everything is noise. Too large → everything is one cluster.
• min_samples: Minimum neighbors to form a dense region. Rule of thumb: 2 * n_features for low-dimensional data.

Choosing eps: Plot the k-nearest-neighbor distance for each point (k = min_samples), sorted in ascending order. Look for the "knee" — that distance is a good eps.

from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors
import numpy as np

k = 2 * X_scaled.shape[1]
nbrs = NearestNeighbors(n_neighbors=k).fit(X_scaled)
distances, _ = nbrs.kneighbors(X_scaled)
distances = np.sort(distances[:, k-1])
# Plot distances to find the knee

Noise label: DBSCAN assigns label -1 to noise points. These are outliers. Do not pass them to downstream models without handling.

---

Hierarchical Clustering (Agglomerative)

How it works: Start with each point as its own cluster. Iteratively merge the two closest clusters. Produces a dendrogram (tree) showing all merge decisions.

Strengths: No need to specify K in advance (can cut the dendrogram at any level). Deterministic. Produces a full hierarchy of clusterings.

Weaknesses: O(n²) memory and O(n² log n) time — not suitable for >10k samples. Greedy merges cannot be undone.

Hyperparameters:

• n_clusters: Cut the dendrogram at this level.
• linkage: How to measure distance between clusters.
  • ward: Minimizes variance within clusters. Usually best for compact clusters.
  • complete: Maximum distance between cluster members. Creates compact clusters.
  • average: Average distance. Compromise between ward and complete.
  • single: Minimum distance. Prone to "chaining" (long thin clusters).

from sklearn.cluster import AgglomerativeClustering
model = AgglomerativeClustering(n_clusters=4, linkage="ward")
labels = model.fit_predict(X_scaled)

---

Gaussian Mixture Models (GMM)

How it works: Assumes data comes from a mixture of K Gaussian distributions. Fits means, covariances, and mixing weights via Expectation-Maximization.

Strengths: Soft assignments (probability of belonging to each cluster). Handles elliptical clusters (covariance can be estimated). More flexible than K-Means.

Weaknesses: More parameters to estimate — needs more data. Sensitive to initialization. Can overfit with full covariance matrices on small datasets.

from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components=3, covariance_type="full", random_state=42)
labels = gmm.fit_predict(X_scaled)
probs = gmm.predict_proba(X_scaled)  # soft assignments

---

Evaluation Metrics

Silhouette Score

Measures how similar each point is to its own cluster vs the nearest other cluster. Range: -1 (wrong cluster) to 1 (perfect). Above 0.5 is considered good.

from sklearn.metrics import silhouette_score
score = silhouette_score(X_scaled, labels)

Davies-Bouldin Index

Ratio of within-cluster scatter to between-cluster separation. Lower is better. More intuitive than silhouette for comparing cluster compactness.

Calinski-Harabasz Score

Ratio of between-cluster dispersion to within-cluster dispersion. Higher is better. Fast to compute.

When ground truth is available

• Adjusted Rand Index (ARI): Agreement between predicted and true labels, adjusted for chance. 1.0 = perfect.
• Normalized Mutual Information (NMI): Information shared between labels. 1.0 = perfect.

---

Algorithm Selection Guide

Situation	Recommended
Unknown K, roughly spherical clusters	K-Means with silhouette to pick K
Unknown K, arbitrary shapes	DBSCAN
Need hierarchy/dendrogram	Agglomerative
Need soft cluster assignments	GMM
Large dataset (>100k rows)	K-Means (MiniBatchKMeans for speed)
High noise/outliers expected	DBSCAN
Very high dimensions (>50 features)	Reduce with PCA first, then cluster

---

Practical Tips

1. Always scale features — K-Means, DBSCAN, and hierarchical clustering use distance metrics that are sensitive to feature scale.

2. Reduce dimensionality first for high-dimensional data — PCA to 10–20 components improves both speed and cluster quality.

3. Validate with domain knowledge — a clustering with silhouette 0.6 that doesn't make business sense is useless. A clustering with 0.3 that produces actionable segments is valuable.

4. Try multiple algorithms — if K-Means and DBSCAN agree, you have more confidence in the clusters.

5. Visualize with t-SNE or UMAP — project to 2D after clustering to visually verify cluster separation.