What is Gradient Descent?

Optimization Algorithm

  • Iterative in nature

    Minimizes specific objective functions. For example:

    • Minimize error of regression model
    • Maximize accuracy of classification task
  • Core Purpose

    Reduces loss functions and optimizes parameters.

Understanding Gradient Descent

Gradient descent is an iterative optimization algorithm used to find the minimum of a function. In machine learning and artificial intelligence, it is the core engine used to train models by minimizing a cost function (or loss function), which measures how far off a model’s predictions are from actual data.

Model Weight Total Error (Loss) Global Minimum Start Large step Small step

The “Walking Downhill” Analogy

Imagine you are blindfolded on a foggy mountain and need to reach the lowest valley:

  1. Find the slope: You feel around with your feet to find which direction slopes the steepest downward. In mathematics, this slope is the gradient.
  2. Take a step: You move a short distance in the opposite direction of the gradient (the downhill path). Notice in the graph above how steps get smaller as the slope flattens out.
  3. Repeat: You pause, check the slope again, and take another step. You keep repeating this until the ground becomes flat, meaning you have reached the bottom (the minimum error).
The Basic Update Rule
New Weight = Old Weight − (Learning Rate × Gradient)

Key Components

  • Learning Rate (α): A hyperparameter that controls how big your steps are down the mountain.
    • Too large You might step too far and leap right past the bottom (causing overshooting or divergence).
    • Too small You will take forever to reach the bottom (causing slow convergence).
  • Cost / Loss Function: A mathematical formula that calculates your model’s total error (such as Mean Squared Error).

Main Types of Gradient Descent

Batch

Evaluates the gradient using the entire dataset before updating parameters. It is very stable, but can be slow and memory-heavy with massive datasets.

Stochastic (SGD)

Evaluates and updates parameters using only one randomly chosen example at a time. It is fast and noisy, which can occasionally help jump out of tricky valleys.

Mini-Batch

Splits the dataset into small random batches for each update. It is the standard approach used in deep learning because it balances speed and stability.

References

error: