File size: 21,217 Bytes
f7c7e26 42001a3 f7c7e26 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 |
{% extends "layout.html" %}
{% block content %}
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Study Guide: Gaussian Mixture Models (GMM)</title>
<!-- MathJax for rendering mathematical formulas -->
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<style>
/* General Body Styles */
body {
background-color: #ffffff; /* White background */
color: #000000; /* Black text */
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
font-weight: normal;
line-height: 1.8;
margin: 0;
padding: 20px;
}
/* Container for centering content */
.container {
max-width: 800px;
margin: 0 auto;
padding: 20px;
}
/* Headings */
h1, h2, h3 {
color: #000000;
border: none;
font-weight: bold;
}
h1 {
text-align: center;
border-bottom: 3px solid #000;
padding-bottom: 10px;
margin-bottom: 30px;
font-size: 2.5em;
}
h2 {
font-size: 1.8em;
margin-top: 40px;
border-bottom: 1px solid #ddd;
padding-bottom: 8px;
}
h3 {
font-size: 1.3em;
margin-top: 25px;
}
/* Main words are even bolder */
strong {
font-weight: 900;
}
/* Paragraphs and List Items with a line below */
p, li {
font-size: 1.1em;
border-bottom: 1px solid #e0e0e0; /* Light gray line below each item */
padding-bottom: 10px; /* Space between text and the line */
margin-bottom: 10px; /* Space below the line */
}
/* Remove bottom border from the last item in a list for cleaner look */
li:last-child {
border-bottom: none;
}
/* Ordered lists */
ol {
list-style-type: decimal;
padding-left: 20px;
}
ol li {
padding-left: 10px;
}
/* Unordered Lists */
ul {
list-style-type: none;
padding-left: 0;
}
ul li::before {
content: "•";
color: #000;
font-weight: bold;
display: inline-block;
width: 1em;
margin-left: 0;
}
/* Code block styling */
pre {
background-color: #f4f4f4;
border: 1px solid #ddd;
border-radius: 5px;
padding: 15px;
white-space: pre-wrap;
word-wrap: break-word;
font-family: "Courier New", Courier, monospace;
font-size: 0.95em;
font-weight: normal;
color: #333;
border-bottom: none;
}
/* GMM Specific Styling */
.story-gmm {
background-color: #f0f8ff;
border-left: 4px solid #005f73; /* Dark Cyan accent for GMM */
margin: 15px 0;
padding: 10px 15px;
font-style: italic;
color: #555;
font-weight: normal;
border-bottom: none;
}
.story-gmm p, .story-gmm li {
border-bottom: none;
}
.example-gmm {
background-color: #e6f7f7;
padding: 15px;
margin: 15px 0;
border-radius: 5px;
border-left: 4px solid #0a9396; /* Lighter Cyan accent for GMM */
}
.example-gmm p, .example-gmm li {
border-bottom: none !important;
}
/* Table Styling */
table {
width: 100%;
border-collapse: collapse;
margin: 25px 0;
}
th, td {
border: 1px solid #ddd;
padding: 12px;
text-align: left;
}
th {
background-color: #f2f2f2;
font-weight: bold;
}
/* --- Mobile Responsive Styles --- */
@media (max-width: 768px) {
body, .container {
padding: 10px;
}
h1 { font-size: 2em; }
h2 { font-size: 1.5em; }
h3 { font-size: 1.2em; }
p, li { font-size: 1em; }
pre { font-size: 0.85em; }
table, th, td { font-size: 0.9em; }
}
</style>
</head>
<body>
<div class="container">
<h1>🌌 Study Guide: Gaussian Mixture Models (GMM)</h1>
<!-- button -->
<div>
<!-- Audio Element -->
<!-- Note: Browsers may block audio autoplay if the user hasn't interacted with the document first,
but since this is triggered by a click, it should work fine. -->
<a
href="/gaussian-mixture-three"
target="_blank"
onclick="playSound()"
class="
cursor-pointer
inline-block
relative
bg-blue-500
text-white
font-bold
py-4 px-8
rounded-xl
text-2xl
transition-all
duration-150
/* 3D Effect (Hard Shadow) */
shadow-[0_8px_0_rgb(29,78,216)]
/* Pressed State (Move down & remove shadow) */
active:shadow-none
active:translate-y-[8px]
">
Tap Me!
</a>
</div>
<script>
function playSound() {
const audio = document.getElementById("clickSound");
if (audio) {
audio.currentTime = 0;
audio.play().catch(e => console.log("Audio play failed:", e));
}
}
</script>
<!-- button -->
<h2>🔹 Core Concepts</h2>
<div class="story-gmm">
<p><strong>Story-style intuition: The Expert Fruit Sorter</strong></p>
<p>Imagine you have a pile of fruit containing two types that can be tricky to separate: <strong>lemons</strong> and <strong>limes</strong>. They look similar, and their sizes overlap. A simple sorter (like K-Means) might draw a hard line: anything yellow is a lemon. But what about a greenish lemon or a yellowish lime? GMM is an expert. It knows that limes are, *on average*, smaller and rounder, while lemons are *on average* larger and more oval. GMM models each fruit type as a flexible, oval-shaped "cloud of probability." For a fruit that's right on the border, GMM can say, "I'm <strong>70% sure</strong> this is a lemon and <strong>30% sure</strong> it's a lime." This is called <strong>soft clustering</strong>.</p>
</div>
<p>A <strong>Gaussian Mixture Model (GMM)</strong> is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions (bell curves). In simple terms, it believes the data is a mix of several different groups, where each group has a sort of "center point" and a particular shape (which can be circular or oval).</p>
<div class="example-gmm">
<p><strong>Example:</strong> Analyzing customer data. You might have one group of customers who spend a lot but visit rarely (an oval cluster) and another group who spend a little but visit often (a different oval cluster). GMM is great at finding these non-circular groups.</p>
</div>
<h2>🔹 Mathematical Foundation</h2>
<div class="story-gmm">
<p>Think of it like a recipe. The final probability of any data point is a "mixture" of probabilities from each group's individual recipe. Each group's recipe defines its center, its shape, and its overall importance in the mix.</p>
</div>
<ul>
<li><strong>Probability Density Function of a Gaussian:</strong> This is the complex-looking formula for a single bell curve (the recipe for one fruit type).
<p>$$ \mathcal{N}(x|\mu, \Sigma) = \text{A formula defining a bell curve} $$</p>
<p>You don't need to memorize it! Just know it's the math for creating one of those oval "probability clouds."</p>
</li>
<li><strong>Mixture of Gaussians:</strong> The total probability is a weighted sum of all the individual bell curves.
<p>$$ p(x) = (\text{Weight}_A \times \text{Prob from A}) + (\text{Weight}_B \times \text{Prob from B}) + \dots $$</p>
Where:
<ul>
<li>\( K \): The number of groups (e.g., 2 types of fruit).</li>
<li>\( \pi_k \): The "mixing weight" (e.g., maybe 60% of our pile is lemons).</li>
<li>\( \mu_k \): The "mean" (the center of the fruit group).</li>
<li>\( \Sigma_k \): The "covariance" (the shape and orientation of the fruit group—is it round or a tilted oval?).</li>
</ul>
</li>
</ul>
<h2>🔹 Expectation-Maximization (EM) Algorithm</h2>
<div class="story-gmm">
<p><strong>Story: The "Guess and Check" Method</strong></p>
<p>Imagine you have the fruit pile but don't know the exact size and shape of lemons and limes. You use a two-step "guess and check" process:
<br><strong>1. The "Guess" Step (Expectation):</strong> You make a starting guess for the oval shapes of the two fruit types. Then, for every single fruit in the pile, you calculate the probability it belongs to each shape. (e.g., "This one is 80% likely a lemon, 20% a lime").
<br><strong>2. The "Check & Update" Step (Maximization):</strong> After guessing for all the fruit, you update your oval shapes. You calculate the average size and shape of all the fruits you labeled as "mostly lemon" to get a *better* lemon shape. You do the same for limes.
<br>You repeat these "Guess" and "Check & Update" steps. Each time, your oval shape descriptions get more accurate, until they settle on the best possible fit for the data.</p>
</div>
<ol>
<li><strong>Initialize</strong> the parameters (the oval shapes) with a random guess.</li>
<li><strong>E-step (Expectation):</strong> The "Guess" step. Calculate the probability that each data point belongs to each cluster.</li>
<li><strong>M-step (Maximization):</strong> The "Check & Update" step. Update the oval shapes based on the probabilities from the E-step.</li>
<li><strong>Repeat</strong> until the oval shapes stop changing.</li>
</ol>
<h2>🔹 Types of Covariance Structures</h2>
<div class="example-gmm">
<p><strong>Example: The Cookie Cutter Analogy</strong></p>
<p>The `covariance_type` parameter in the code controls the flexibility of your "oval shapes" or cookie cutters.</p>
<ul>
<li><strong>Spherical:</strong> Least flexible. Clusters must be circles. (Round cookie cutters of different sizes).</li>
<li><strong>Diagonal:</strong> A bit more flexible. Clusters are ovals, but they must be aligned with the axes. (Oval cutters that can't be tilted).</li>
<li><strong>Full:</strong> Most flexible. Clusters can be ovals of any shape and tilted in any direction. (The best, but also the most complex, type of cookie cutter).</li>
<li><strong>Tied:</strong> A special rule where all clusters must have the exact same shape and size. (You must use the same cookie cutter for every group).</li>
</ul>
</div>
<h2>🔹 Comparison</h2>
<table>
<thead>
<tr>
<th>Model</th>
<th>GMM vs. K-Means</th>
<th>GMM vs. Hierarchical</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Cluster Assignment</strong></td>
<td>GMM is <strong>soft</strong> (probabilistic). A point is 70% in Cluster A, 30% in B. K-Means is <strong>hard</strong> (100% in Cluster A).</td>
<td>GMM is probabilistic. Hierarchical is distance-based and deterministic.</td>
</tr>
<tr>
<td><strong>Cluster Shape</strong></td>
<td>GMM can model <strong>elliptical</strong> clusters. K-Means assumes <strong>spherical</strong> clusters.</td>
<td>GMM models clusters as distributions. Hierarchical can produce any shape depending on linkage.</td>
</tr>
<tr>
<td><strong>Scalability</strong></td>
<td>Both scale well, but GMM is more computationally intensive per iteration.</td>
<td>GMM scales much better to large datasets than hierarchical clustering.</td>
</tr>
</tbody>
</table>
<h2>🔹 Model Selection</h2>
<p>GMM requires you to specify the number of clusters (K). Information criteria are used to help find the optimal K by balancing model fit with model complexity.</p>
<div class="example-gmm">
<p><strong>Story Example: Goldilocks and the Three Models</strong></p>
<p>You test three GMMs: one with too few clusters (underfit), one with too many (overfit), and one that's just right.
<br>• <strong>AIC (Akaike Information Criterion)</strong> and <strong>BIC (Bayesian Information Criterion)</strong> are like judges who score each model. They give points for fitting the data well but subtract points for being too complex. The model with the lowest score is the one that's "just right."</p>
</div>
<h2>🔹 Strengths & Weaknesses</h2>
<h3>Advantages:</h3>
<ul>
<li>✅ <strong>Flexible Cluster Shapes:</strong> Can find clusters that aren't simple circles. <strong>Example:</strong> Identifying a long, thin cluster of "commuter" customers on a map.</li>
<li>✅ <strong>Soft Clustering:</strong> Tells you the probability that a point belongs to each cluster, which is great for understanding uncertainty.</li>
</ul>
<h3>Disadvantages:</h3>
<ul>
<li>❌ <strong>Requires specifying K:</strong> You have to tell it how many clusters to look for.</li>
<li>❌ <strong>Sensitive to Initialization:</strong> A bad starting guess can sometimes lead to a bad final result.</li>
<li>❌ <strong>Can be slow:</strong> The "Guess and Check" process can take time, especially with a lot of data.</li>
</ul>
<h2>🔹 Real-World Applications</h2>
<ul>
<li><strong>Image Segmentation:</strong> Grouping pixels of similar color to separate a person from the background in a photo.</li>
<li><strong>Speaker Recognition:</strong> Identifying who is speaking by modeling the unique properties of their voice.</li>
<li><strong>Anomaly Detection:</strong> Finding unusual credit card transactions by seeing which ones don't fit well into any normal spending clusters.</li>
</ul>
<h2>🔹 Python Implementation (Beginner Example)</h2>
<div class="story-gmm">
<p>This simple example shows the core steps: create data, create a GMM model, train it (`.fit`), and then use it to predict which cluster new data belongs to (`.predict`) and the probabilities for each cluster (`.predict_proba`).</p>
</div>
<pre><code>
import numpy as np
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture
from sklearn.datasets import make_blobs
# --- 1. Create Sample Data ---
# We'll create 300 data points, grouped into 3 "blobs" or clusters.
X, y_true = make_blobs(n_samples=300, centers=3, cluster_std=1.0, random_state=42)
# --- 2. Create and Train the GMM ---
# We tell the model to look for 3 clusters (n_components=3).
# random_state ensures we get the same result every time we run the code.
gmm = GaussianMixture(n_components=3, random_state=42)
# Train the model on our data. This is where the EM algorithm runs.
gmm.fit(X)
# --- 3. Make Predictions ---
# Predict the cluster for each data point in our original dataset.
labels = gmm.predict(X)
# Let's create a new, unseen data point to test our model.
new_point = np.array([[-5, -5]])
# Predict which cluster the new point belongs to.
new_point_label = gmm.predict(new_point)
print(f"The new point belongs to cluster: {new_point_label[0]}")
# --- 4. Get Probabilities (The "Soft" Part) ---
# This is the most powerful feature of GMM.
# It tells us the probability of the new point belonging to EACH of the 3 clusters.
probabilities = gmm.predict_proba(new_point)
print(f"Probabilities for each cluster: {np.round(probabilities, 3)}") # e.g., [[0.95, 0.05, 0.0]]
# --- 5. Visualize the Results ---
# Let's plot our data points, colored by the cluster labels GMM assigned.
plt.figure(figsize=(8, 6))
plt.scatter(X[:, 0], X[:, 1], c=labels, s=40, cmap='viridis')
# Let's also plot our new point as a big red star to see where it landed.
plt.scatter(new_point[:, 0], new_point[:, 1], c='red', s=200, marker='*')
plt.title('GMM Clustering Results')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.grid(True)
plt.show()
</code></pre>
<h2>🔹 Best Practices</h2>
<ul>
<li><strong>Scale Features:</strong> If your features are on different scales (e.g., age and income), scale them before fitting GMM so one doesn't unfairly dominate the other.</li>
<li><strong>Use AIC/BIC:</strong> To choose the best number of clusters (K), run your model with several different values for `n_components` and pick the one with the lowest AIC or BIC score.</li>
<li><strong>Use `n_init` Parameter:</strong> To prevent a bad random start from ruining your model, set `n_init` to a value like 10. This tells scikit-learn to run the whole process 10 times and keep the best result.</li>
</ul>
<h2>🔹 Key Terminology Explained (GMM)</h2>
<div class="story-gmm">
<p><strong>The Story: Decoding the Fruit Sorter's Toolkit</strong></p>
<p>Let's clarify the advanced tools our expert fruit sorter uses.</p>
</div>
<ul>
<li>
<strong>Probabilistic Model:</strong>
<br>
<strong>What it is:</strong> A model that uses probabilities to handle uncertainty. It gives you the "chance" of something happening, not a definite yes or no.
<br>
<strong>Story Example:</strong> A weather forecast saying "80% chance of rain" is a <strong>probabilistic model</strong>. GMM uses this same idea to assign a "chance of belonging" to each cluster.
</li>
<li>
<strong>Gaussian Distribution (Bell Curve):</strong>
<br>
<strong>What it is:</strong> The classic bell-shaped curve. It describes data where most values are clustered around an average.
<br>
<strong>Story Example:</strong> The heights of adults in a city follow a <strong>Gaussian distribution</strong>. Most people are near the average height, and very tall or very short people are rare.
</li>
<li>
<strong>Covariance:</strong>
<br>
<strong>What it is:</strong> A measure of how two variables are related. It defines the shape and tilt of the cluster.
<br>
<strong>Story Example:</strong> Ice cream sales and temperature have a positive <strong>covariance</strong>: when one goes up, the other tends to go up. This relationship creates an oval shape in the data, which the covariance matrix describes.
</li>
<li>
<strong>Likelihood:</strong>
<br>
<strong>What it is:</strong> A score of how well the model's "oval shapes" explain the actual data. The "Guess and Check" algorithm works to make this score as high as possible.
<br>
<strong>Story Example:</strong> If our fruit sorter's oval shape for "lemons" perfectly covers all the actual lemons in the pile, it has a high <strong>likelihood</strong>. If it's a bad fit, it has a low likelihood.
</li>
</ul>
</div>
</body>
</html>
{% endblock %}
|