In machine learning, cost functions (also called loss or objective functions) quantify how wrong a model’s predictions are. They are divided primarily into functions for predicting continuous numbers (Regression) and functions for sorting data into categories (Classification).
Regression Functions (Predicting continuous values)
Mean Squared Error (MSE) / L2 Loss
Most CommonMeasures the average squared difference between predictions and actual values. Heavily penalizes large errors (outliers) because the errors are squared.
Mean Absolute Error (MAE) / L1 Loss
RobustMeasures the average absolute difference between predictions and actual values. It treats all errors proportionally, making it highly robust to outliers.
Huber Loss
A piecewise function that combines the best of MSE and MAE. It acts like MSE for small errors (smooth gradient) and acts like MAE for large errors (ignoring extreme outliers). It requires a tunable hyperparameter, δ (delta).
Classification Functions (Predicting categories)
Binary Cross-Entropy (Log Loss)
Used for binary (two-class) problems. It penalizes confident wrong predictions exponentially. It compares the model’s output probability (0.0 to 1.0) against the true label (0 or 1).
Categorical Cross-Entropy
The expansion of Log Loss for multi-class problems. It measures the distance between the predicted probability distribution across all classes and the actual true class (usually one-hot encoded).
Hinge Loss
SVMsPrimarily used for Support Vector Machines (SVMs). Instead of outputting probabilities, it aims to create a “maximum margin” boundary between classes. It penalizes predictions that are on the wrong side of the margin, even if they are technically classified correctly.
References
-
“Huber loss,” Wikipedia, Apr. 15, 2026. https://en.wikipedia.org/w/index.php?title=Huber_loss&oldid=1349030864
-
“Loss Functions — ML Glossary documentation,” ml-cheatsheet.readthedocs.io, 2017. https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html