How Gradient Descent actually update model parameters (Example)?

Gradient Descent: Under the Hood

At a concrete level, we can walk through the gradient descent steps using a small e-commerce dataset containing five examples, utilizing Mean Squared Error (MSE) as the loss metric, mathematically denoted as \( J(w, b) \):

Ad Spend in $1000s (feature: \( x \)) Sales in $1000s (label: \( y \))
1.002.0
2.004.0
3.005.0
4.007.0
5.009.0
1

Initialize Parameters

The algorithm begins by setting the weight (\( w \)) and bias (\( b \)) to zero:

\( w = 0 \) \( b = 0 \) \( \hat{y} = 0 + 0(x) \)

Where \( \hat{y} \) (y-hat) represents the model’s predicted value.

2

Calculate Initial Loss

Calculate the MSE loss function, \( J(w, b) \), with the initial parameters against all 5 examples:

\[ J(w,b) = \frac{1}{N} \sum_{i=1}^{N} (y^{(i)} – \hat{y}^{(i)})^2 \]
Loss = 35.00
3

Calculate the Gradient (Slope)

Calculate the steepness and direction of the loss function relative to both parameters:

\( \frac{\partial J}{\partial w} = -39.2 \) \( \frac{\partial J}{\partial b} = -10.8 \)
Expand to see calculus derivation

To determine the local slope for the parameter updates, we take the partial derivative of the MSE function with respect to \( w \) and \( b \).

Weight Derivative

The partial derivative of the loss function with respect to the weight (\( w \)):

\[ \frac{\partial J}{\partial w} = \frac{-2}{N} \sum_{i=1}^{N} x^{(i)} (y^{(i)} – \hat{y}^{(i)}) \]

We compute the error for each example, multiply by its feature value \( x \), sum the results, and scale by \(-2/N\). Substituting \( w=0 \) and \( b=0 \) into this dataset yields: \((-2/5) \times 98 = -39.2\).

Bias Derivative

The partial derivative of the loss function with respect to the bias (\( b \)):

\[ \frac{\partial J}{\partial b} = \frac{-2}{N} \sum_{i=1}^{N} (y^{(i)} – \hat{y}^{(i)}) \]

We simply sum the raw errors and scale by \(-2/N\). Substituting the initial zeros yields: \((-2/5) \times 27 = -10.8\).

4

Update Parameters

Step in the opposite direction of the gradient to descend the error curve. We apply the update rule using a learning rate (\( \alpha \)) of 0.02:

Weight Update
\[ w_{new} = w_{old} – \alpha \left( \frac{\partial J}{\partial w} \right) \]
0 - (0.02 × -39.2)
\( w = 0.784 \)
Bias Update
\[ b_{new} = b_{old} – \alpha \left( \frac{\partial J}{\partial b} \right) \]
0 - (0.02 × -10.8)
\( b = 0.216 \)

We use the updated weight (\( w = 0.78 \)) and bias (\( b = 0.22 \)) to recalculate the loss and repeat the gradient computation. Projecting this out for six total iterations produces the following convergence trajectory:

Iteration Weight (\( w \)) Bias (\( b \)) Loss \( J(w,b) \)
10.000.0035.00
20.780.2213.42
31.280.355.35
41.580.422.30
51.750.461.14
61.850.480.69
Convergence Characteristics: The loss falls dramatically during the initial large steps down the steep walls of the error curve, but the reduction slows as the parameters approach the flatter valley floor (the minimum). When the algorithm fully converges, the partial derivatives will equal zero, meaning the loss has stabilized at its lowest possible numerical value.

Deriving the Gradients: The Chain Rule

1. The Setup

To optimize the model, we must find the partial derivatives of the Mean Squared Error (MSE) loss function with respect to the weight (\( w \)) and the bias (\( b \)). We start with our two foundational equations:

Prediction Function
\[ f_{w,b}(x^{(i)}) = w \cdot x^{(i)} + b \]
MSE Loss Function
\[ J(w,b) = \frac{1}{M} \sum_{i=1}^{M} (f_{w,b}(x^{(i)}) – y^{(i)})^2 \]

The Chain Rule: Because the linear prediction function is nested inside the squared error function, we take the derivative of the outer function and multiply it by the derivative of the inner function.

2. The Weight Derivative

\( \frac{\partial J}{\partial w} \)
  1. Step A: The Outer Derivative

    Treat the inner parenthesis as a single variable (\(u^2\)). Applying the power rule drops the exponent 2 to the front.

    \( 2 \cdot (w \cdot x^{(i)} + b – y^{(i)}) \)
  2. Step B: The Inner Derivative (w.r.t \( w \))

    Derive the inside function strictly with respect to \( w \). Constants \( b \) and \( y \) become 0. The derivative of \( wx \) is simply \( x \).

    \( \frac{\partial}{\partial w} (w \cdot x^{(i)} + b – y^{(i)}) = \mathbf{x^{(i)}} \)
  3. Step C: Multiply & Reformat

    Multiply Step A by Step B, place it back inside the summation, and swap the prediction notation back in.

    \[ \frac{\partial J}{\partial w} = \frac{1}{M} \sum_{i=1}^{M} (f_{w,b}(x^{(i)}) – y^{(i)}) \cdot 2x^{(i)} \]

3. The Bias Derivative

\( \frac{\partial J}{\partial b} \)
  1. Step A: The Outer Derivative

    This remains exactly the same as the weight derivative. The exponent 2 drops to the front.

    \( 2 \cdot (w \cdot x^{(i)} + b – y^{(i)}) \)
  2. Step B: The Inner Derivative (w.r.t \( b \))

    Derive the inside function strictly with respect to \( b \). The \( wx \) and \( y \) terms are treated as constants (becoming 0). The derivative of a standalone \( b \) is 1.

    \( \frac{\partial}{\partial b} (w \cdot x^{(i)} + b – y^{(i)}) = \mathbf{1} \)
  3. Step C: Multiply & Reformat

    Multiply Step A by Step B. Because multiplying by 1 changes nothing, the \( x \) feature vanishes completely from this formula.

    \[ \frac{\partial J}{\partial b} = \frac{1}{M} \sum_{i=1}^{M} (f_{w,b}(x^{(i)}) – y^{(i)}) \cdot 2 \]
Calculation steps involved in above derivatives:

Reference

error: