lectures/2026-10-15_Machine_Learning_Backpropagation.md

metadata

date: 2026-10-15T00:00:00.000Z
course: '[[Machine Learning]]'
topic: '[[Backpropagation]]'
source_file: Machine-Learning_Backpropagation_2026-10-15.pdf
model_used: gemini-3.6-flash
tags:
  - course/MachineLearning
  - topic/Backpropagation
  - graduate-notes

Machine Learning: Backpropagation

1. Executive Summary

• Algorithmic Basis: Backpropagation is an efficient linear-time application of the multivariable chain rule over computational topologies represented as Directed Acyclic Graphs (DAGs).
• Forward vs. Backward Pass: The forward pass computes pre-activations $Z^{(l)}$ and non-linear activations $A^{(l)}$ sequentially, caching intermediate states required to compute parameter updates during the backward pass.
• Error Propagation: The error signal vector $\delta^{(l)}$ is recursively propagated backward from layer $l+1$ to layer $l$ via transposition of the weight matrix and element-wise scaling by activation derivatives.
• Gradient Construction: Weight gradients $\frac{\partial L}{\partial W^{(l)}}$ are expressed directly as the outer product of the current layer's error vector $\delta^{(l)}$ and the transposed activation vector from the preceding layer $(A^{(l-1)})^T$.
• Pathological Dynamics: Saturating activation functions (e.g., Sigmoid) drive derivative terms $\sigma'(Z^{(l)}) \to 0$, inducing the vanishing gradient problem in deep architectures.

---

2. Mathematical Definitions, Derivation & Equations

State Variables & Dimensionality

Let the network at layer $l \in {1, \dots, L}$ be defined with the following variables:

• $L \in \mathbb{R}$: Scalar loss objective function.
• $W^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$: Weight matrix for layer $l$.
• $b^{(l)} \in \mathbb{R}^{n_l}$: Bias vector for layer $l$.
• $Z^{(l)} \in \mathbb{R}^{n_l}$: Linear combination / pre-activation vector at layer $l$.
• $A^{(l)} \in \mathbb{R}^{n_l}$: Post-activation output vector at layer $l$ (where $A^{(0)} = X$, the input data).
• $\sigma(\cdot)$: Pointwise non-linear activation function.
• $\odot$: Hadamard (element-wise) matrix product.
• $\delta^{(l)} \equiv \frac{\partial L}{\partial Z^{(l)}} \in \mathbb{R}^{n_l}$: Layer error vector.

---

Forward Pass Equations

For layer $l$: $$Z^{(l)}=W^{(l)}{A}^{(l-1)}+b^{(l)}Z^\{(l)\}\; =\; W^\{(l)\}\; A^\{(l-1)\}\; +\; b^\{(l)\}$$ $$A^{(l)}=\sigma \left(Z^{(l)}\right)A^\{(l)\}\; =\; \backslash sigma\backslash left(Z^\{(l)\}\backslash right)$$

---

Backward Pass & Gradient Equations

1. Layer Error Propagation Term ($\delta^{(l)}$)

By applying the vector-valued multivariable chain rule across computational steps $Z^{(l)} \to A^{(l)} \to Z^{(l+1)} \to L$:

$${\delta }^{(l)}=\frac{\mathrm{\partial }L}{\mathrm{\partial }{Z}^{(l)}}={\left(W^{(l+1)}\right)}^T{\delta }^{(l+1)}\odot {\sigma }^{\mathrm{\prime }}\left(Z^{(l)}\right)\backslash delta^\{(l)\}\; =\; \backslash frac\{\backslash partial\; L\}\{\backslash partial\; Z^\{(l)\}\}\; =\; \backslash left(W^\{(l+1)\}\backslash right)^T\; \backslash delta^\{(l+1)\}\; \backslash odot\; \backslash sigma\text{'}\backslash left(Z^\{(l)\}\backslash right)$$

2. Weight Parameter Gradient ($\frac{\partial L}{\partial W^{(l)}}$)

Applying the derivative with respect to the matrix parameters $W^{(l)}$:

$$\frac{\mathrm{\partial }L}{\mathrm{\partial }{W}^{(l)}}={\delta }^{(l)}{\left(A^{(l-1)}\right)}^T\backslash frac\{\backslash partial\; L\}\{\backslash partial\; W^\{(l)\}\}\; =\; \backslash delta^\{(l)\}\; \backslash left(A^\{(l-1)\}\backslash right)^T$$

Dimension Verification Check:

• $\delta^{(l)} \in \mathbb{R}^{n_l \times 1}$
• $(A^{(l-1)})^T \in \mathbb{R}^{1 \times n_{l-1}}$
• $\frac{\partial L}{\partial W^{(l)}} \in \mathbb{R}^{n_l \times n_{l-1}} = \operatorname{dim}\left(W^{(l)}\right)$ $\quad \checkmark$

---

3. High-Yield Exam Notes & Professor Emphasis

Exam Pitfalls & Professor Warnings

• Matrix Dimension Verification: Dimension checks on parameter gradients are guaranteed exam questions. Always explicitly verify that $\operatorname{dim}\left(\frac{\partial L}{\partial W^{(l)}}\right) = \operatorname{dim}\left(W^{(l)}\right) = (n_l \times n_{l-1})$.
• Outer Product Order: A common mistake is swapping the order in the weight gradient formula. Remember: $\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (A^{(l-1)})^T$, NOT $A^{(l-1)} (\delta^{(l)})^T$.
• Vanishing Gradient Pathology: Occurs when pre-activations $Z^{(l)}$ enter the saturated regime of activation functions like Sigmoid ($\sigma'(Z^{(l)}) \approx 0$). Multiplying by near-zero terms recursively drives $\delta^{(l)} \to 0$ for early layers.
• Memory Caching Necessity: Intermediate activations $A^{(l-1)}$ MUST be stored in RAM during the forward pass; without caching, computing $\frac{\partial L}{\partial W^{(l)}}$ requires redundant re-computation of the forward pass.

---

4. Key Concept Q&A Flashcards

Q1: What underlying graph representation is assumed when executing backpropagation?
A1: Directed Acyclic Graphs (DAGs), over which multivariable chain rule operations can be evaluated deterministically.

Q2: What is the formal definition of the error vector $\delta^{(l)}$ at layer $l$?
A2: $\delta^{(l)}$ is defined as the partial derivative of the scalar loss $L$ with respect to the pre-activation vector $Z^{(l)}$, i.e., $\delta^{(l)} = \frac{\partial L}{\partial Z^{(l)}}$.

Q3: Why are intermediate activations $A^{(l-1)}$ cached during the forward pass?
A3: Because the weight gradient update $\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (A^{(l-1)})^T$ explicitly depends on the activation outputs from the preceding layer.

Q4: How does the Hadamard product ($\odot$) function in the error term formula?
A4: It performs element-wise multiplication between the backpropagated error vector $(W^{(l+1)})^T \delta^{(l+1)}$ and the element-wise derivative of the activation function evaluated at $Z^{(l)}$.

Q5: What mathematical condition causes the vanishing gradient problem in networks with Sigmoid activations?
A5: When $|Z^{(l)}|$ becomes large, the activation function saturates, causing its derivative $\sigma'(Z^{(l)}) \to 0$. Since $\delta^{(l)}$ scales directly with $\sigma'(Z^{(l)})$, early layers receive near-zero updates.

Q6: What is the shape of the gradient matrix $\frac{\partial L}{\partial W^{(l)}}$ for a layer with $n_{l-1}$ inputs and $n_l$ output neurons?
A6: It has dimensions $(n_l \times n_{l-1})$, matching the shape of the weight matrix $W^{(l)}$.

---

5. Chronological / Sectional Breakdown

Section 1: Executive Overview

• Focus: Fundamental definition of backpropagation.
• Key Concept: Backpropagation as an algorithmic realization of the multivariable chain rule on directed acyclic computational graphs.

Section 2: Mathematical Formulation

• Focus: Formal matrix/vector definitions for feedforward propagation and error backpropagation.
• Key Formulas:
  • Forward activation steps: $Z^{(l)} = W^{(l)} A^{(l-1)} + b^{(l)}$ and $A^{(l)} = \sigma(Z^{(l)})$.
  • Error update rule: $\delta^{(l)} = (W^{(l+1)})^T \delta^{(l+1)} \odot \sigma'(Z^{(l)})$.
  • Weight gradient update rule: $\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (A^{(l-1)})^T$.

Section 3: High-Yield Exam Notes

• Focus: Core exam test points and common implementation errors.
• Key Takeaways:
  • Mandatory dimensional matching checks for gradient matrices vs. parameter matrices.
  • Theoretical underpinnings of the vanishing gradient phenomenon due to $\sigma'(Z) \approx 0$.

Section 4: Practice Check

• Focus: Applied conceptual validation.
• Key Takeaway: Explanation of memory trade-offs ( caching $A^{(l-1)}$ in forward pass) to enable linear time computation during backward pass.