Bias-Variance Tradeoff in Regression

Visualizing how different models fit the same data with varying bias and variance

High Bias, High Variance

Complex model, insufficient data: Models vary wildly between datasets (high variance) but average prediction is far from truth (high bias). Often seen in deep networks with very little training data.

High Bias, Low Variance

Underfitting (Linear model for nonlinear data): All models are similar (low variance) but consistently miss the true relationship (high bias). Too simple to capture complexity.

Low Bias, High Variance

Overfitting (High-degree polynomial): Models vary significantly (high variance) but average prediction is close to truth (low bias). Too flexible, memorizes noise.

Low Bias, Low Variance

Well-regularized model: Models are consistent across datasets (low variance) and average prediction matches truth well (low bias). The sweet spot!

🔬 Mathematical Foundation of Bias and Variance

📐 Bias Definition

Bias²[f̂(x)] = (E[f̂(x)] - f(x))²
Bias measures the difference between the expected prediction of our model and the true value.

• E[f̂(x)]: Average prediction across all possible training sets
• f(x): True function we're trying to learn
• High bias: Model systematically under/over-estimates

📊 Variance Definition

Var[f̂(x)] = E[(f̂(x) - E[f̂(x)])²]
Variance measures how much predictions vary when we train on different datasets.

• f̂(x): Prediction from a specific training set
• E[f̂(x)]: Average prediction across training sets
• High variance: Predictions change dramatically with new data
🎯 The Fundamental Decomposition of Expected Error
E[(y - f̂(x))²] = Bias²[f̂(x)] + Var[f̂(x)] + σ²

🔍 Breaking Down the Error Components:

  • Bias²: Error due to oversimplifying assumptions in the learning algorithm
  • Variance: Error due to sensitivity to small fluctuations in the training set
  • σ² (Irreducible Error): Noise inherent in the problem that no model can eliminate
💡 Key Insight: As model complexity increases, bias typically decreases but variance increases. The art of machine learning is finding the optimal balance to minimize total error.

🎯 In Practice: Estimation

For a regression model trained on dataset D:

Bias estimation:
Train many models on different samples, compute average prediction, compare to true function

Variance estimation:
Measure how much individual model predictions deviate from the average prediction

⚖️ The Tradeoff

Simple models:
• Low variance (consistent)
• High bias (may miss complexity)

Complex models:
• High variance (sensitive to data)
• Low bias (can capture complexity)

Goal: Find the sweet spot that minimizes total error