Predicting continuous values with a straight line
Linear regression is the simplest supervised learning algorithm that models the relationship between input features and a continuous output variable using a straight line (in 2D) or hyperplane (in higher dimensions).
Analogy: Like drawing the best-fit line through scattered points on a graph to predict future values based on the trend.
Step-by-step intuition:
β₀ = intercept, β₁ = slope, ε = error
A real estate company has data on 5 houses. Predict the price of a 2500 sq ft house.
Calculate Means
x̄ = (1000 + 1500 + 2000 + 3000) / 4 = 1875 sq ft
ȳ = (150 + 200 + 250 + 350) / 4 = 237.5 ($1000s)
We exclude the house we're predicting from training
Calculate Deviations
(x - x̄): -875, -375, 125, 1125
(y - ȳ): -87.5, -37.5, 12.5, 112.5
Find how much each point differs from the mean
Calculate Slope (β₁)
Numerator: (-875)(-87.5) + (-375)(-37.5) + (125)(12.5) + (1125)(112.5)
= 76562.5 + 14062.5 + 1562.5 + 126562.5 = 218750
Denominator: (-875)² + (-375)² + (125)² + (1125)²
= 765625 + 140625 + 15625 + 1265625 = 2187500
β₁ = 218750 / 2187500 = 0.10
Slope tells us price change per sq ft
Calculate Intercept (β₀)
β₀ = ȳ - β₁ × x̄
β₀ = 237.5 - 0.10 × 1875
β₀ = 237.5 - 187.5 = 50
Base price when size = 0
Write Prediction Equation
Price = 50 + 0.10 × Size
For 2500 sq ft:
Price = 50 + 0.10 × 2500 = 50 + 250 = 300
$300,000 predicted price
Calculate R² Score
Predictions: 150, 200, 250, 350
Residuals: 0, 0, 0, 0 (perfect fit!)
R² = 1 - (SS_res / SS_tot) = 1.0
R² = 1.0 means perfect linear fit
The model fits perfectly (R²=1.0). Each additional sq ft adds $100 to the price. The $50k base price represents fixed costs.
Answers:
from sklearn.linear_model import LinearRegression + import numpy as np + # Training data + X = np.array([[1000], [1500], [2000], [3000]]) + y = np.array([150, 200, 250, 350]) + # Create and train model + model = LinearRegression() + model.fit(X, y) + # Make prediction + prediction = model.predict([[2500]]) + print(f"Predicted price: ${prediction[0]}k") + # Model parameters + print(f"Slope: {model.coef_[0]:.3f}") + print(f"Intercept: {model.intercept_:.2f}")
Classification by majority vote of nearest neighbors
K-Nearest Neighbors is a simple, non-parametric algorithm that classifies data points based on how their neighbors are classified. It finds the K closest training examples and uses majority vote.
Analogy: "You are the average of the 5 people you spend the most time with." KNN says "You're similar to your closest neighbors in feature space!"
Most common distance metric for KNN
Alternative: sum of absolute differences
Most frequent class among K neighbors
Classify a new iris flower with sepal length=5.0cm, sepal width=3.5cm. Use K=3.
Define New Point
New flower: x_new = [5.0, 3.5]
K = 3 (we'll find 3 nearest neighbors)
The flower we want to classify
Calculate Distances to All Points
d₁ = √[(5.0-5.1)² + (3.5-3.5)²] = √[0.01 + 0] = 0.10
d₂ = √[(5.0-4.9)² + (3.5-3.0)²] = √[0.01 + 0.25] = 0.51
d₃ = √[(5.0-7.0)² + (3.5-3.2)²] = √[4.0 + 0.09] = 2.02
d₄ = √[(5.0-6.4)² + (3.5-3.2)²] = √[1.96 + 0.09] = 1.43
d₅ = √[(5.0-5.0)² + (3.5-3.6)²] = √[0 + 0.01] = 0.10
Euclidean distance to each training point
Sort by Distance
Select top 3 (highlighted) for K=3
Take Majority Vote
3 nearest neighbors:
Neighbor 1: Setosa (distance 0.10)
Neighbor 2: Setosa (distance 0.10)
Neighbor 3: Setosa (distance 0.51)
Vote count: Setosa = 3, Versicolor = 0
Winner: Setosa (unanimous!)
Majority class wins
Make Prediction
Predicted Class: Setosa
Confidence: 3/3 = 100%
All neighbors agree
The new flower is extremely close to known Setosa examples (distances 0.10, 0.10, 0.51). The unanimous vote gives us high confidence in this classification.
from sklearn.neighbors import KNeighborsClassifier + import numpy as np + # Training data + X = np.array([[5.1,3.5], [4.9,3.0], [7.0,3.2], [6.4,3.2], [5.0,3.6]]) + y = np.array(['Setosa', 'Setosa', 'Versicolor', 'Versicolor', 'Setosa']) + # Create and train model + model = KNeighborsClassifier(n_neighbors=3) + model.fit(X, y) + # Make prediction + new_flower = np.array([[5.0, 3.5]]) + prediction = model.predict(new_flower) + proba = model.predict_proba(new_flower) + print(f"Predicted: {prediction[0]}") + print(f"Confidence: {proba[0].max():.2%}")
Tree-based decisions using feature splits
Decision Trees make predictions by asking a series of yes/no questions about features, creating a flowchart-like structure from root to leaves.
Analogy: Like a game of 20 Questions - each question (split) narrows down possibilities until you reach a final decision (leaf).
pᵢ = proportion of class i. Measures disorder.
Choose split with highest information gain
Used by CART algorithm. Faster to compute.
Build decision tree for loan approval. Dataset:
Calculate Root Entropy
Total: 6 samples
Approved (Yes): 3/6 = 0.5
Denied (No): 3/6 = 0.5
H(root) = -[0.5 log₂(0.5) + 0.5 log₂(0.5)]
H(root) = -[0.5(-1) + 0.5(-1)] = 1.0
Maximum entropy = maximum disorder
Test Split on Credit Score
If Credit = Good: 2 Yes, 0 No → H = 0 (pure!)
If Credit = Poor: 1 Yes, 3 No → H = -[0.25log₂(0.25) + 0.75log₂(0.75)]
H(Poor) = -[0.25(-2) + 0.75(-0.415)] = 0.5 + 0.311 = 0.811
Weighted avg: (3/6)×0 + (4/6)×0.811 = 0.541
IG(Credit) = 1.0 - 0.541 = 0.459
Information gain from splitting on Credit Score
Test Split on Income
If Income = High: 2 Yes, 1 No → H = 0.918
If Income = Low: 1 Yes, 2 No → H = 0.918
Weighted: (3/6)×0.918 + (3/6)×0.918 = 0.918
IG(Income) = 1.0 - 0.918 = 0.082
Income provides less information gain
Choose Best Split
IG(Credit Score) = 0.459 ← HIGHEST!
IG(Income) = 0.082
Best first split: Credit Score
Choose feature with highest information gain
Build Tree Recursively
Root: Credit Score = Good?
├─ YES → Approved (pure node)
└─ NO → Split on Income
├─ Income = High? → Denied
└─ Income = Low? → Denied (majority)
Continue splitting until pure or stopping criterion
Make Predictions
New applicant: Credit=Good, Income=High
Follow path: Credit=Good → Approved ✓
Decision rule: IF Credit Score is Good THEN Approve
Traverse tree from root to leaf
The tree correctly classifies all training examples. Credit Score is the most important feature with IG=0.459.
from sklearn.tree import DecisionTreeClassifier + from sklearn import tree + import matplotlib.pyplot as plt + # Create and train + model = DecisionTreeClassifier(max_depth=3, criterion='entropy') + model.fit(X_train, y_train) + # Predict + predictions = model.predict(X_test) + # Visualize tree + tree.plot_tree(model, filled=True, feature_names=['Income','Credit','Age'])
Partitioning data into K distinct clusters
K-Means is an unsupervised learning algorithm that groups similar data points into K clusters by minimizing within-cluster variance.
Analogy: Organizing a messy room by grouping similar items together. K-Means finds natural groupings in unlabeled data.
Sum of squared distances from points to centroids
Mean of all points assigned to cluster k
Assign to nearest centroid
Cluster 6 customers into K=2 groups based on [Age, Income]. Data:
Initialize K=2 Random Centroids
C₁ (initial) = [25, 40] (customer A)
C₂ (initial) = [60, 90] (customer E)
Start with random points or use K-means++
Assign Points to Nearest Centroid
Distance from A to C₁: √[(25-25)² + (40-40)²] = 0
Distance from A to C₂: √[(25-60)² + (40-90)²] = √[1225+2500] = 61.0
A → Cluster 1 (closer to C₁)
Similarly calculate for all:
B [30,50] → C₁ (dist=11.2 vs 47.2)
C [28,45] → C₁ (dist=5.8 vs 50.9)
D [55,80] → C₂ (dist=42.7 vs 11.2)
E [60,90] → C₂ (dist=0)
F [52,75] → C₂ (dist=37.3 vs 17.0)
Cluster 1: {A, B, C}
Cluster 2: {D, E, F}
Each point goes to its nearest centroid
Recalculate Centroids
New C₁ = mean of {A, B, C}
Age: (25 + 30 + 28)/3 = 27.67
Income: (40 + 50 + 45)/3 = 45
C₁ = [27.67, 45]
New C₂ = mean of {D, E, F}
Age: (55 + 60 + 52)/3 = 55.67
Income: (80 + 90 + 75)/3 = 81.67
C₂ = [55.67, 81.67]
Centroids move to center of their clusters
Check Convergence
Re-assign with new centroids:
All points stay in same clusters!
Centroids don't change → CONVERGED ✓
Algorithm stops when assignments don't change
Calculate Within-Cluster Sum of Squares
WCSS₁ = Σ dist² to C₁ = 0² + 11.2² + 5.8² = 158.88
WCSS₂ = Σ dist² to C₂ = 11.2² + 0² + 17.0² = 414.24
Total WCSS = 573.12
Measures cluster compactness (lower = better)
Algorithm converged in 1 iteration. Clear separation: younger customers with lower income vs older customers with higher income.
from sklearn.cluster import KMeans + import numpy as np + # Create model + kmeans = KMeans(n_clusters=2, random_state=42) + kmeans.fit(X) + # Get predictions + labels = kmeans.labels_ + centroids = kmeans.cluster_centers_ + # Predict for new point + new_customer = np.array([[32, 55]]) + cluster = kmeans.predict(new_customer) + print(f"Assigned to cluster: {cluster[0]}")
Reliable model evaluation technique
Cross-validation is a resampling technique that evaluates model performance by training and testing on different subsets of data multiple times.
Analogy: Testing a student on multiple different exams instead of just one - gives more reliable assessment of their true knowledge.
Average performance across K folds
σ = standard deviation of K scores
Evaluate a model using 5-fold CV. Dataset has 100 samples. After running, fold accuracies are: 0.85, 0.90, 0.88, 0.87, 0.90. Calculate mean accuracy and standard error.
Understand the Setup
Total samples: n = 100
Number of folds: K = 5
Each fold size: 100/5 = 20 samples
Each iteration: Train on 80, Test on 20
Divide data into 5 equal parts
Record Fold Results
Performance on each test fold
Calculate Mean Accuracy
Mean = (0.85 + 0.90 + 0.88 + 0.87 + 0.90) / 5
Mean = 4.40 / 5 = 0.88
Average accuracy: 88%
This is our best estimate of model performance
Calculate Standard Deviation
Deviations: (0.85-0.88), (0.90-0.88), (0.88-0.88), (0.87-0.88), (0.90-0.88)
= -0.03, 0.02, 0, -0.01, 0.02
Squared: 0.0009, 0.0004, 0, 0.0001, 0.0004
Variance = 0.0018 / 4 = 0.00045
SD = √0.00045 = 0.021
Measures variability across folds
Calculate Standard Error
SE = SD / √K = 0.021 / √5
SE = 0.021 / 2.236 = 0.0094
SE ≈ 0.0094 or 0.94%
Precision of our mean estimate
Report Results with Confidence
Mean accuracy: 0.88 ± 0.009
95% CI (approx): 0.88 ± 2×0.009 = [0.862, 0.898]
Model performs between 86.2% and 89.8% with 95% confidence
Final performance estimate with uncertainty
Low variability (SD=0.021) indicates stable model performance. Every test fold performed similarly, suggesting the model generalizes well.
from sklearn.model_selection import cross_val_score + from sklearn.tree import DecisionTreeClassifier + model = DecisionTreeClassifier() + # 5-fold cross-validation + scores = cross_val_score(model, X, y, cv=5, scoring='accuracy') + print(f"Fold scores: {scores}") + print(f"Mean: {scores.mean():.3f}") + print(f"Std: {scores.std():.3f}") + print(f"95% CI: [{scores.mean()-2*scores.std():.3f}, {scores.mean()+2*scores.std():.3f}]")
Fitting non-linear relationships with polynomial curves
Polynomial regression extends linear regression by adding polynomial terms (x², x³, etc.) to capture non-linear, curved relationships in data.
Analogy: When a straight line won't fit your data (like trajectory of a thrown ball), use a curved line instead!
Temperature (°C): [10, 15, 20, 25, 30]. Sales ($100s): [2, 5, 12, 22, 35]. Fit quadratic model and predict sales at 27°C.
Set Up Polynomial Model
y = β₀ + β₁x + β₂x² + Where x = temperature, y = sales + Need to find β₀, β₁, β₂
Create Design Matrix
x | x² | y + 10 | 100 | 2 + 15 | 225 | 5 + 20 | 400 | 12 + 25 | 625 | 22 + 30 | 900 | 35
Solve Using Normal Equations (simplified)
Using least squares: β = (XᵀX)⁻¹Xᵀy + Result: β₀ = 15, β₁ = -2, β₂ = 0.06
Write Equation
y = 15 - 2x + 0.06x²
Predict at x = 27°C
y = 15 - 2(27) + 0.06(27)² + y = 15 - 54 + 0.06(729) + y = 15 - 54 + 43.74 = 4.74 + But wait! Let me recalculate properly... + Actual better fit: y = 0.06x² - 1.4x + 11 + y = 0.06(729) - 1.4(27) + 11 + y = 43.74 - 37.8 + 11 = 16.94
from sklearn.preprocessing import PolynomialFeatures + from sklearn.linear_model import LinearRegression + import numpy as np + X = np.array([10, 15, 20, 25, 30]).reshape(-1, 1) + y = np.array([2, 5, 12, 22, 35]) + # Create polynomial features (degree 2) + poly = PolynomialFeatures(degree=2) + X_poly = poly.fit_transform(X) + # Fit model + model = LinearRegression() + model.fit(X_poly, y) + # Predict + X_new = poly.transform([[27]]) + print(f"Sales at 27°C: ${model.predict(X_new)[0]:.0f}")
Preventing overfitting with L2 penalty
Ridge regression adds an L2 penalty term to the loss function, shrinking coefficient magnitudes to prevent overfitting.
Formula: J = MSE + α Σβᵢ²
Compare linear vs ridge regression. Data prone to overfitting. α = 0.1
Linear Regression Cost
J = (1/n)Σ(y - ŷ)²
Ridge Cost Function
J_ridge = (1/n)Σ(y - ŷ)² + α Σβᵢ²Penalty term shrinks large coefficients
from sklearn.linear_model import Ridge + model = Ridge(alpha=0.1) + model.fit(X_train, y_train) + predictions = model.predict(X_test)
Feature selection through L1 penalty
Lasso adds L1 penalty: J = MSE + α Σ|βᵢ|. Can shrink coefficients to exactly zero, performing automatic feature selection.
5 features, but only 2 are relevant. Use Lasso with α = 0.5
Linear Regression (No Penalty)
All coefficients non-zero: [3.2, 0.5, 5.1, 0.3, 0.1]
Apply Lasso Penalty
J = MSE + 0.5 Σ|βᵢ|Small coefficients penalized heavily
Lasso Result
Coefficients: [3.1, 0, 5.0, 0, 0]Features 2, 4, 5 eliminated!
from sklearn.linear_model import Lasso + model = Lasso(alpha=0.5) + model.fit(X_train, y_train) + print(f"Non-zero features: {np.sum(model.coef_ != 0)}")
Combining L1 and L2 penalties
Combines L1 and L2: J = MSE + α₁Σ|βᵢ| + α₂Σβᵢ². Best of both Ridge and Lasso.
from sklearn.linear_model import ElasticNetmodel = ElasticNet(alpha=0.1, l1_ratio=0.5)model.fit(X, y)
Robust regression with margin tolerance
SVR finds hyperplane that fits data within margin ε. Points outside margin contribute to loss. Robust to outliers.
from sklearn.svm import SVRmodel = SVR(kernel='rbf', C=1.0, epsilon=0.1)model.fit(X, y)
Binary classification with sigmoid function
Binary classification using sigmoid: P(y=1) = 1/(1+e^(-z)) where z = β₀+β₁x. Despite name, it's for classification!
from sklearn.linear_model import LogisticRegressionmodel = LogisticRegression()model.fit(X, y)proba = model.predict_proba(X_new)
Maximum margin classification
SVM finds hyperplane that maximally separates classes. Uses support vectors (closest points) and kernel trick for non-linear boundaries.
from sklearn.svm import SVCmodel = SVC(kernel='rbf', C=1.0, gamma='auto')model.fit(X, y)
Probabilistic classifier using Bayes' Theorem
Applies Bayes' Theorem with "naive" independence assumption. P(y|x) ∝ P(y)ΠP(xᵢ|y). Extremely fast for text classification.
from sklearn.naive_bayes import GaussianNBmodel = GaussianNB()model.fit(X, y)predictions = model.predict(X_test)
Ensemble of decision trees
Ensemble of decision trees. Each tree trained on random subset (bootstrap) with random features. Final prediction by majority vote.
from sklearn.ensemble import RandomForestClassifiermodel = RandomForestClassifier(n_estimators=100, max_depth=10)model.fit(X, y)feature_importance = model.feature_importances_
Sequential ensemble method
Sequentially builds trees, each correcting errors of previous. Predictions: F(x) = f₁(x) + f₂(x) + ... + f_n(x).
from xgboost import XGBClassifiermodel = XGBClassifier(n_estimators=100, learning_rate=0.1)model.fit(X, y)
Universal function approximators
Layers of connected neurons. Each neuron: z = Σwᵢxᵢ + b, then activation function σ(z). Trained via backpropagation + gradient descent.
from sklearn.neural_network import MLPClassifiermodel = MLPClassifier(hidden_layer_sizes=(100, 50), activation='relu')model.fit(X, y)
Builds hierarchy of clusters (dendrogram). Agglomerative: merge closest clusters. Divisive: split clusters. No need to specify K upfront.
Density-Based Spatial Clustering. Groups points with many neighbors (dense regions). Can find arbitrarily-shaped clusters and outliers.
Soft clustering: each point has probability of belonging to each cluster. Mixture of K Gaussian distributions. Uses EM algorithm.
See Data Science Topic 77 for complete details. Reduces dimensions by finding directions of maximum variance.
t-Distributed Stochastic Neighbor Embedding. Non-linear dimensionality reduction for visualization. Preserves local structure better than PCA.
Neural network that learns compressed representation. Encoder: reduces dimensions. Decoder: reconstructs input. Latent space = compressed features.
Reinforcement learning: agent learns optimal actions through trial and error. Q-table stores expected reward for each state-action pair.
Combines Q-Learning with deep neural networks. Neural net approximates Q-function. Used by DeepMind for Atari games.
Directly optimizes policy π(a|s). Gradient ascent on expected reward. REINFORCE algorithm: update based on return.
GridSearch: Try all combinations of hyperparameters. RandomSearch: Sample random combinations. Both use cross-validation.
Optimizing model settings: learning rate, regularization, tree depth, etc. Methods: Grid search, random search, Bayesian optimization.
Classification: Accuracy, Precision, Recall, F1-Score, ROC-AUC. Regression: MSE, RMSE, MAE, R². Choose based on problem.
L1 (Lasso): Sparse. L2 (Ridge): Smooth. Dropout: Random neuron deactivation. Early Stopping: Stop when validation error increases.
Total Error = Bias² + Variance + Noise. High Bias: Underfitting (too simple). High Variance: Overfitting (too complex). Goal: balance both.
Bagging: Parallel models, average predictions (Random Forest). Boosting: Sequential, correct errors (XGBoost). Stacking: Meta-model combines base models.
Creating new features from existing ones. Techniques: polynomial features, interaction terms, binning, encoding categoricals, domain-specific features.
When one class dominates: SMOTE (synthetic minority oversampling), undersampling, class weights, or use metrics like F1/ROC-AUC instead of accuracy.
Sequential data with temporal dependency. Models: ARIMA, LSTM, Prophet. Key: train/test split must respect time order (no shuffling!).
Finding rare, unusual observations. Methods: Isolation Forest, One-Class SVM, Autoencoders (reconstruction error), statistical methods (z-score, IQR).
Use pre-trained model on new task. Take model trained on ImageNet, adapt to your problem. Faster training, needs less data.
Start with pre-trained weights, continue training on new data. Lower learning rate, selectively unfreeze layers. Balances speed and customization.
SHAP: SHapley Additive exPlanations. Assigns each feature an importance value for prediction. Based on game theory (Shapley values).
Adam: Adaptive learning rates per parameter + momentum. Most popular optimizer. RMSprop: Divides by moving average of gradient squared.
Batch Norm: Normalizes layer inputs, stabilizes training. Dropout: Randomly drops neurons during training (p=0.5 typical), prevents overfitting.