Evaluation Metrics

In machine learning, "accuracy" is rarely enough to judge a model's performance, especially in imbalanced datasets. We need a more granular set of metrics to evaluate our classifiers.

1. The Confusion Matrix

The Confusion Matrix provides a breakdown of all possible prediction outcomes for a binary classifier:

• TP: Model correctly predicted the positive class.
• TN: Model correctly predicted the negative class.
• FP (Type I Error): Model predicted positive, but it was negative.
• FN (Type II Error): Model predicted negative, but it was positive.

2. Standard Metrics

Accuracy

The proportion of total predictions that are correct.

Precision (Positive Predictive Value)

Of all predicted positives, how many were actually positive? Focuses on reliability.

Recall (Sensitivity or True Positive Rate)

Of all actual positives, how many did the model find? Focuses on completeness.

F1-Score

The harmonic mean of Precision and Recall. It balances the tradeoff between them.

3. ROC and AUC

The Receiver Operating Characteristic (ROC) Curve plots the True Positive Rate (Recall) against the False Positive Rate ($FP / (FP + TN)$) as we vary the classification threshold.

• AUC (Area Under the Curve): A single value from 0 to 1 representing the probability that the model will rank a random positive sample higher than a random negative one.
• AUC = 0.5: Random guessing.
• AUC = 1.0: Perfect classifier.

4. Multi-class Averaging

For multi-class classification, we can calculate metrics for each class individually and then average them:

• Macro Averaging: Treat all classes equally by taking the simple average of their scores.
• Micro Averaging: Weight classes by their size (more frequent classes have a bigger impact). Use this if you care more about overall performance than smaller classes.

5. Ranking Metrics (Rank Learning)

In many systems (like search engines or recommendation engines), we don't just care if a document is "relevant" or not; we care about the order in which results are presented.

NDCG (Normalized Discounted Cumulative Gain)

NDCG is the industry standard for evaluating ranked results. It builds on two ideas:

1. Cumulative Gain: Highly relevant documents are more valuable.
2. Discounting: Relevant documents are much more valuable at the top of the list (Rank 1) than at the bottom.

6. Strategic Metric Selection

When building production systems, you often have multiple metrics (e.g., Accuracy, Latency, Memory). How do you choose?

Satisfying vs. Optimizing Metrics

• Optimizing Metric: The one metric you want to be as "best" as possible (e.g., Accuracy). You pick only one.
• Satisfying Metrics: Metrics that just need to be "good enough" or below a threshold (e.g., Latency must be $< 100ms$).

Data Consistency

A fundamental principle of evaluation is that your Training, Validation, and Test sets must come from the same distribution. If you train on high-res professional photos but test on blurry mobile uploads, your metrics will be meaningless.