Spaces:
Running
Running
File size: 7,538 Bytes
47509f4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | # Clustering Guide
## When to Use Clustering
Clustering is unsupervised — there is no target variable. Use it to:
- Discover natural groupings in data (customer segments, document topics)
- Compress or summarize data before supervised learning
- Anomaly detection (points that don't belong to any cluster)
- Preprocessing for semi-supervised learning
Clustering is exploratory. Results depend heavily on the algorithm, distance metric, and hyperparameters. Always validate clusters with domain knowledge, not just metrics.
---
## K-Means
**How it works**: Assign each point to the nearest centroid. Recompute centroids as cluster means. Repeat until convergence.
**Strengths**: Fast, scalable, simple to understand. Works well when clusters are roughly spherical and similar in size.
**Weaknesses**: Must specify K in advance. Sensitive to outliers (centroid is pulled toward them). Fails on non-spherical clusters (crescents, rings). Assumes clusters have similar variance.
**Hyperparameters**:
- `n_clusters` (K): Most important parameter. Choose via elbow method or silhouette score.
- `n_init`: Number of random initializations (default 10). Run multiple times, keep best inertia.
- `init`: `"k-means++"` (default) is much better than random initialization.
- `max_iter`: Convergence iterations (default 300 is usually enough).
**Choosing K — elbow method**:
Plot inertia (within-cluster sum of squares) vs K. Look for the "elbow" where adding more clusters gives diminishing returns. The elbow is often subjective — silhouette score is more reliable.
**Choosing K — silhouette score**:
Silhouette score ranges from -1 to 1. Higher = better defined clusters. Try K from 2 to 10, pick the K with the highest average silhouette.
```python
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
scores = []
for k in range(2, 11):
km = KMeans(n_clusters=k, init="k-means++", n_init=10, random_state=42)
labels = km.fit_predict(X_scaled)
scores.append(silhouette_score(X_scaled, labels))
best_k = range(2, 11)[scores.index(max(scores))]
```
**Always scale features before K-Means.** Euclidean distance is not meaningful when features have different units or scales.
---
## DBSCAN
**How it works**: Groups points that are densely packed together (within `eps` distance and at least `min_samples` neighbors). Points in low-density regions are labeled as noise (-1).
**Strengths**: Does not require specifying K. Finds arbitrarily shaped clusters. Naturally handles outliers — they become noise points rather than distorting clusters.
**Weaknesses**: Sensitive to `eps` and `min_samples`. Struggles when clusters have very different densities. Poor performance in high dimensions (curse of dimensionality makes all distances similar).
**Hyperparameters**:
- `eps`: Neighborhood radius. Critical. Too small → everything is noise. Too large → everything is one cluster.
- `min_samples`: Minimum neighbors to form a dense region. Rule of thumb: `2 * n_features` for low-dimensional data.
**Choosing eps**: Plot the k-nearest-neighbor distance for each point (k = min_samples), sorted in ascending order. Look for the "knee" — that distance is a good `eps`.
```python
from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors
import numpy as np
k = 2 * X_scaled.shape[1]
nbrs = NearestNeighbors(n_neighbors=k).fit(X_scaled)
distances, _ = nbrs.kneighbors(X_scaled)
distances = np.sort(distances[:, k-1])
# Plot distances to find the knee
```
**Noise label**: DBSCAN assigns label `-1` to noise points. These are outliers. Do not pass them to downstream models without handling.
---
## Hierarchical Clustering (Agglomerative)
**How it works**: Start with each point as its own cluster. Iteratively merge the two closest clusters. Produces a dendrogram (tree) showing all merge decisions.
**Strengths**: No need to specify K in advance (can cut the dendrogram at any level). Deterministic. Produces a full hierarchy of clusterings.
**Weaknesses**: O(n²) memory and O(n² log n) time — not suitable for >10k samples. Greedy merges cannot be undone.
**Hyperparameters**:
- `n_clusters`: Cut the dendrogram at this level.
- `linkage`: How to measure distance between clusters.
- `ward`: Minimizes variance within clusters. Usually best for compact clusters.
- `complete`: Maximum distance between cluster members. Creates compact clusters.
- `average`: Average distance. Compromise between ward and complete.
- `single`: Minimum distance. Prone to "chaining" (long thin clusters).
```python
from sklearn.cluster import AgglomerativeClustering
model = AgglomerativeClustering(n_clusters=4, linkage="ward")
labels = model.fit_predict(X_scaled)
```
---
## Gaussian Mixture Models (GMM)
**How it works**: Assumes data comes from a mixture of K Gaussian distributions. Fits means, covariances, and mixing weights via Expectation-Maximization.
**Strengths**: Soft assignments (probability of belonging to each cluster). Handles elliptical clusters (covariance can be estimated). More flexible than K-Means.
**Weaknesses**: More parameters to estimate — needs more data. Sensitive to initialization. Can overfit with full covariance matrices on small datasets.
```python
from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components=3, covariance_type="full", random_state=42)
labels = gmm.fit_predict(X_scaled)
probs = gmm.predict_proba(X_scaled) # soft assignments
```
---
## Evaluation Metrics
### Silhouette Score
Measures how similar each point is to its own cluster vs the nearest other cluster. Range: -1 (wrong cluster) to 1 (perfect). Above 0.5 is considered good.
```python
from sklearn.metrics import silhouette_score
score = silhouette_score(X_scaled, labels)
```
### Davies-Bouldin Index
Ratio of within-cluster scatter to between-cluster separation. Lower is better. More intuitive than silhouette for comparing cluster compactness.
### Calinski-Harabasz Score
Ratio of between-cluster dispersion to within-cluster dispersion. Higher is better. Fast to compute.
### When ground truth is available
- **Adjusted Rand Index (ARI)**: Agreement between predicted and true labels, adjusted for chance. 1.0 = perfect.
- **Normalized Mutual Information (NMI)**: Information shared between labels. 1.0 = perfect.
---
## Algorithm Selection Guide
| Situation | Recommended |
|-----------|-------------|
| Unknown K, roughly spherical clusters | K-Means with silhouette to pick K |
| Unknown K, arbitrary shapes | DBSCAN |
| Need hierarchy/dendrogram | Agglomerative |
| Need soft cluster assignments | GMM |
| Large dataset (>100k rows) | K-Means (MiniBatchKMeans for speed) |
| High noise/outliers expected | DBSCAN |
| Very high dimensions (>50 features) | Reduce with PCA first, then cluster |
---
## Practical Tips
1. **Always scale features** — K-Means, DBSCAN, and hierarchical clustering use distance metrics that are sensitive to feature scale.
2. **Reduce dimensionality first** for high-dimensional data — PCA to 10–20 components improves both speed and cluster quality.
3. **Validate with domain knowledge** — a clustering with silhouette 0.6 that doesn't make business sense is useless. A clustering with 0.3 that produces actionable segments is valuable.
4. **Try multiple algorithms** — if K-Means and DBSCAN agree, you have more confidence in the clusters.
5. **Visualize with t-SNE or UMAP** — project to 2D after clustering to visually verify cluster separation.
|