# Understanding Machine Learning Metrics

# **Introduction**

Machine learning metrics are the cornerstone of model evaluation, playing a critical role in the development and validation phases. These metrics provide quantifiable measures of a model’s performance and are crucial for comparing different models, tuning parameters, and ultimately selecting the best model for deployment.

# **Section 1: Classification Metrics**

# **Accuracy**

**Definition:** Accuracy is the most intuitive performance measure and it is simply a ratio of correctly predicted observation to the total observations. It is useful when the class distribution is similar.

**Formula:** Accuracy=Number of Correct PredictionsTotal Number of **PredictionsAccuracy**\=Total Number of PredictionsNumber of Correct Predictions​

**Limitations:** Accuracy is not suitable for imbalanced class problems. For instance, if a model is designed to detect fraud where 95% of the data is non-fraudulent, a naive model predicting ‘non-fraud’ will still achieve a 95% accuracy.

![](https://miro.medium.com/v2/resize:fit:1050/1*Dzct3xT48ZE0UGM77ZOHZw.png align="left")

A confusion matrix can be visualized showing true positives, true negatives, false positives, and false negatives

# **Precision and Recall**

Precision is the ratio of correctly predicted positive observations to the total predicted positives. High precision relates to a low false positive rate.

Recall (Sensitivity) is the ratio of correctly predicted positive observations to the all observations in actual class.

**Trade-off:** Increasing precision lowers recall and vice versa. This trade-off can be combined into the F1 Score.

![](https://miro.medium.com/v2/resize:fit:1050/1*yhnoCH2Em7WpBeghsCFKGw.png align="left")

Precision and recall can be plotted on a bar chart for a comparative view of multiple classifiers.

# **F1 Score**

Definition: The F1 Score is the 2\*((precision\*recall)/(precision+recall)). It is an overall measure of a model’s accuracy that combines precision and recall.

Use-case: When you want a balance between Precision and Recall and there is an uneven class distribution (large number of Actual Negatives).

![](https://miro.medium.com/v2/resize:fit:1050/1*WE7SFPJklfiT98L--jmTSQ.png align="left")

F1 scores for different thresholds or classes can be plotted to show their variation with changing thresholds.

# **ROC Curve and AUC**

ROC Curve (Receiver Operating Characteristic curve) is a graph showing the performance of a classification model at all classification thresholds.

AUC (Area Under the Curve) represents the degree or measure of separability. It tells how much the model is capable of distinguishing between classes.

Illustration: A graph with True Positive Rate on the y-axis and False Positive Rate on the x-axis, with the area under the curve highlighted.

![](https://miro.medium.com/v2/resize:fit:1050/1*6qz6Et_2WbkgSrRQor6slA.png align="left")

The ROC curve is a plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied.

# **Section 2: Regression Metrics**

# **Mean Absolute Error (MAE)**

Definition: MAE measures the average magnitude of the errors in a set of predictions, without considering their direction.

Formula:

![](https://miro.medium.com/v2/resize:fit:461/1*S-4VTsyQ-NNwoD_CbJkpbw.png align="left")

**Interpretation:** It’s the average over the test sample of the absolute differences between prediction and actual observation where all individual differences have equal weight.

![](https://miro.medium.com/v2/resize:fit:1050/1*QnvSj0GRfbes4l3apitL8g.png align="left")

# **Mean Squared Error (MSE) and Root Mean Squared Error (RMSE)**

MSE is the average of the squares of the errors. It’s more popular than MAE because MSE “punishes” larger errors.

RMSE is the square root of the mean of the squared errors. RMSE is even more popular than MSE because RMSE is interpretable in the “y” units.

![](https://miro.medium.com/v2/resize:fit:1050/1*8GAMVhPN7WmZkeeron01Nw.png align="left")

Plotting actual vs. predicted values with a line of perfect fit can visually represent how MSE and RMSE relate to variance in the data.

# **R-squared and Adjusted R-squared**

R-squared measures how well the observed outcomes are replicated by the model, based on the proportion of total variation of outcomes explained by the model.

Adjusted R-squared also takes into account the number of predictors in the model and adjusts for the number of terms in the model.

![](https://miro.medium.com/v2/resize:fit:1050/1*80BRjWuCkBOp70gUnZKBEQ.png align="left")

# **Section 3: Clustering Metrics**

# **Silhouette Coefficient**

Definition: Measures how similar an object is to its own cluster compared to other clusters.

![](https://miro.medium.com/v2/resize:fit:1050/1*Kk9MkVoWvN6xvDTtTFL8Jg.png align="left")

**Davies-Bouldin Index**

![](https://miro.medium.com/v2/resize:fit:1050/1*9PVd5GBgaYzZGnVSM28t_A.png align="left")

Definition: The average ‘similarity’ between each cluster and the most similar one. Lower values indicate better clustering.

# **Section 4: Ranking Metrics**

# **Mean Reciprocal Rank**

Definition: The average of the reciprocal ranks of the first relevant answer.

![](https://miro.medium.com/v2/resize:fit:1050/1*y9H8AMPLG1hRbKyPtf41dw.png align="left")

# **Normalized Discounted Cumulative Gain (NDCG)**

Definition: Measures the performance of a ranking system based on the graded relevance of the recommended items.

![](https://miro.medium.com/v2/resize:fit:1050/1*H3-Amo31cGB29OaIdNEP4g.png align="left")

# **Section 5: Advanced Metrics and Considerations**

Discuss advanced metrics such as Log Loss, which measures the performance of a classification model where the prediction input is a probability value between 0 and 1, and Cohen’s Kappa, which measures the agreement between two raters.

**Log Loss:**

![](https://miro.medium.com/v2/resize:fit:1050/1*gLM0zpPp6vgc8sRz_P9wzA.png align="left")

**Cohen’s Kappa:**

![](https://miro.medium.com/v2/resize:fit:1050/1*tcga9jMhg8FNltoDy8B-qw.png align="left")

# **Conclusion**

Choosing the right metric is paramount to developing powerful machine-learning models. This comprehensive overview provides a foundation, but the learning journey in the ever-evolving field of machine learning is continuous.