Objective (Cost) Functions in Machine Learning

In machine learning, cost functions (also called loss or objective functions) quantify how wrong a model’s predictions are. They are divided primarily into functions for predicting continuous numbers (Regression) and functions for sorting data into categories (Classification).

Regression Functions (Predicting continuous values)

Mean Squared Error (MSE) / L2 Loss

Most Common

Measures the average squared difference between predictions and actual values. Heavily penalizes large errors (outliers) because the errors are squared.

J = 1⁄N ∑ (yi − ŷi)2
Example: Predicting real estate prices. If a house is priced $100k off, the squared penalty forces the model to adjust rapidly.

Mean Absolute Error (MAE) / L1 Loss

Robust

Measures the average absolute difference between predictions and actual values. It treats all errors proportionally, making it highly robust to outliers.

J = 1⁄N ∑ |yi − ŷi|
Example: Predicting pizza delivery times. A 60-minute outlier won’t aggressively drag the entire model’s baseline away from the 20-minute average.

Huber Loss

A piecewise function that combines the best of MSE and MAE. It acts like MSE for small errors (smooth gradient) and acts like MAE for large errors (ignoring extreme outliers). It requires a tunable hyperparameter, δ (delta).

Example: Stock market price forecasting, where you want to ignore massive, unpredictable flash crashes (outliers) but fine-tune around normal daily volatility.

Classification Functions (Predicting categories)

Binary Cross-Entropy (Log Loss)

Used for binary (two-class) problems. It penalizes confident wrong predictions exponentially. It compares the model’s output probability (0.0 to 1.0) against the true label (0 or 1).

J = −1⁄N ∑ [yi log(ŷi) + (1 − yi) log(1 − ŷi)]
Example: Email Spam Filtering. Is this email Spam (1) or Not Spam (0)?

Categorical Cross-Entropy

The expansion of Log Loss for multi-class problems. It measures the distance between the predicted probability distribution across all classes and the actual true class (usually one-hot encoded).

J = − ∑ yi · log(ŷi)
Example: Image classification, like the MNIST dataset (classifying handwritten digits into 10 buckets: 0 through 9).

Hinge Loss

SVMs

Primarily used for Support Vector Machines (SVMs). Instead of outputting probabilities, it aims to create a “maximum margin” boundary between classes. It penalizes predictions that are on the wrong side of the margin, even if they are technically classified correctly.

J = max(0, 1 − yi · ŷi)
Example: Text categorization (e.g., distinguishing between “Sports” and “Politics” news articles using a hard mathematical boundary).

References

error: