
Optimization Algorithm
-
Iterative in nature
Minimizes specific objective functions. For example:
- Minimize error of regression model
- Maximize accuracy of classification task
-
Core Purpose
Reduces loss functions and optimizes parameters.
Understanding Gradient Descent
Gradient descent is an iterative optimization algorithm used to find the minimum of a function. In machine learning and artificial intelligence, it is the core engine used to train models by minimizing a cost function (or loss function), which measures how far off a model’s predictions are from actual data.
The “Walking Downhill” Analogy
Imagine you are blindfolded on a foggy mountain and need to reach the lowest valley:
- Find the slope: You feel around with your feet to find which direction slopes the steepest downward. In mathematics, this slope is the gradient.
- Take a step: You move a short distance in the opposite direction of the gradient (the downhill path). Notice in the graph above how steps get smaller as the slope flattens out.
- Repeat: You pause, check the slope again, and take another step. You keep repeating this until the ground becomes flat, meaning you have reached the bottom (the minimum error).
Key Components
-
Learning Rate (α): A hyperparameter that controls how big your steps are down the mountain.
- Too large You might step too far and leap right past the bottom (causing overshooting or divergence).
- Too small You will take forever to reach the bottom (causing slow convergence).
- Cost / Loss Function: A mathematical formula that calculates your model’s total error (such as Mean Squared Error).
Main Types of Gradient Descent
Batch
Evaluates the gradient using the entire dataset before updating parameters. It is very stable, but can be slow and memory-heavy with massive datasets.
Stochastic (SGD)
Evaluates and updates parameters using only one randomly chosen example at a time. It is fast and noisy, which can occasionally help jump out of tricky valleys.
Mini-Batch
Splits the dataset into small random batches for each update. It is the standard approach used in deep learning because it balances speed and stability.