lectures/2026-08-19_Machine-Learning_Backpropagation.md

metadata

date: 2026-08-19T00:00:00.000Z
course: '[[Machine-Learning]]'
topic: '[[Backpropagation]]'
source_file: Machine-Learning_Backpropagation_2026-08-19.pdf
model_used: gemini-3.1-flash-lite
tags:
  - course/Machine-Learning
  - topic/Backpropagation
  - graduate-notes

Machine-Learning: Backpropagation

1. Executive Summary & Conceptual Mind Map

• Backpropagation provides a computationally efficient mechanism to compute gradients of a scalar loss function with respect to weights in a neural network.
• It leverages the multivariable chain rule to propagate error terms backwards through a Directed Acyclic Graph (DAG).
• The process relies on caching intermediate activations from the forward pass to compute weight gradients during the backward pass.
• Saturated activation functions (like Sigmoid) can lead to the "Vanishing Gradient" problem when their derivative approaches zero.

graph TD
    A[Backpropagation] --> B[Chain Rule]
    A --> C[Efficiency]
    B --> D[Compute Error Delta]
    C --> E[Caching Activations]
    D --> F[Weight Gradient dL/dW]

2. Mathematical Definitions, Derivations & Proofs

The objective is to calculate the gradient of the scalar loss $L$ with respect to the weight matrix $W^{(l)}$ at layer $l$.

Forward Pass: The transformation at layer $l$ is defined as:

Backward Pass (Error Term): By applying the multivariable chain rule, the error term $\delta^{(l)}$ is defined as the partial derivative of the loss with respect to the pre-activation: $${\delta }^{(l)}=\frac{\mathrm{\partial }L}{\mathrm{\partial }{Z}^{(l)}}=(W^{(l+1)}{)}^T{\delta }^{(l+1)}\odot {\sigma }^{\mathrm{\prime }}(Z^{(l)})\backslash delta^\{(l)\}\; =\; \backslash frac\{\backslash partial\; L\}\{\backslash partial\; Z^\{(l)\}\}\; =\; (W^\{(l+1)\})^T\; \backslash delta^\{(l+1)\}\; \backslash odot\; \backslash sigma\text{'}(Z^\{(l)\})$$ Note: $\odot$ denotes the element-wise (Hadamard) product.

Weight Gradient: The gradient of the loss with respect to the weights at layer $l$ is: $$\frac{\mathrm{\partial }L}{\mathrm{\partial }{W}^{(l)}}={\delta }^{(l)}(A^{(l-1)}{)}^T\backslash frac\{\backslash partial\; L\}\{\backslash partial\; W^\{(l)\}\}\; =\; \backslash delta^\{(l)\}\; (A^\{(l-1)\})^T$$

3. High-Yield Exam Notes & Professor Emphasis

Exam Pitfalls & Professor Warnings

• Dimension Analysis: Always perform a dimensional consistency check. Ensure the resulting matrix $\frac{\partial L}{\partial W^{(l)}}$ matches the dimensions of the original weight matrix $W^{(l)}$.
• Saturated Activations: Be prepared to explain the "Vanishing Gradient" phenomenon. Specifically, identify that when $\sigma'(Z) \to 0$ (e.g., in the tails of a Sigmoid function), the gradient signal effectively vanishes, preventing effective learning in deeper layers.
• Caching: Do not forget that the backward pass is strictly dependent on the activations stored during the forward pass; this is a common "trick" question regarding memory complexity.

4. Key Concept Q&A Flashcards

Q1: What is the primary purpose of backpropagation in a neural network? A1: It is an efficient algorithm to compute the gradient of a scalar loss function with respect to all model weights using the chain rule.

Q2: What is the significance of the DAG structure in this context? A2: The Directed Acyclic Graph allows for a defined topological ordering, ensuring that gradients can be computed sequentially from the output layer back to the input.

Q3: Why is the Hadamard product ($\odot$) used in the calculation of $\delta^{(l)}$? A3: Because the activation function $\sigma$ acts element-wise on $Z^{(l)}$, the chain rule requires element-wise multiplication with the derivative $\sigma'(Z^{(l)})$.

Q4: What causes the Vanishing Gradient problem? A4: It occurs when the derivative of the activation function, $\sigma'(Z)$, is very small (near 0). In deep networks, the chain rule involves repeated multiplication of these small values, causing the gradient to shrink exponentially.

Q5: Why is caching $A^{(l-1)}$ mandatory? A5: The weight gradient calculation $\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (A^{(l-1)})^T$ explicitly requires the activation value from the previous layer to determine how much the weight contributed to the loss.

5. Chronological / Sectional Breakdown

• [Section 1: Executive Overview]: Introduction to Backpropagation as a multivariable chain rule application on DAGs.
• [Section 2: Mathematical Formulation]: Formal definitions of the forward pass equations ($Z^{(l)}, A^{(l)}$) and the backward pass derivation ($\delta^{(l)}$ and gradient of $W^{(l)}$).
• [Section 3: High-Yield Exam Notes]: Strategy for exams, focusing on dimension checking and understanding gradient saturation.
• [Section 4: Practice Check]: Conceptual validation regarding the necessity of activation caching for gradient computation.