Evaluating models properly

Classification metrics

The confusion matrix: true positives (TP), false positives (FP), true negatives (TN), false negatives (FN).

• Accuracy = (TP + TN) / all. Misleading on imbalanced data: if 1% of transactions are fraud, predicting "not fraud" always gives 99% accuracy.
• Precision = TP / (TP + FP): of those flagged, how many were right? Prioritise when false alarms are costly (flagging genuine customers).
• Recall = TP / (TP + FN): of the real positives, how many did we catch? Prioritise when misses are costly (cancer screening, fraud).
• F1 = harmonic mean of precision and recall: one number for imbalanced problems.
• ROC-AUC: how well the model ranks positives above negatives across all thresholds. PR-AUC is more informative when positives are rare.

The threshold trades precision against recall. Choose it from business costs, not the default 0.5.

Regression metrics

• MAE: average absolute error, in the target's units; robust to outliers.
• RMSE: penalises large errors more.
• R²: share of variance explained (1 is perfect; 0 is no better than predicting the mean).
• MAPE: percentage error; breaks down near zero.

Imbalanced data

Use stratified splits; class weights (class_weight="balanced"); resampling (undersample the majority, oversample or SMOTE the minority) applied only to training data; the right metric (recall, F1, PR-AUC); threshold tuning.

Validation strategy

• Random k-fold for independent rows.
• Stratified k-fold for classification.
• Group k-fold when rows belong to the same entity (the same user should not be in both train and test).
• Time-based split for anything over time: train on the past, test on the future.

Beyond the metric

• Compare against the baseline.
• Error analysis: look at the worst mistakes; they often reveal data problems.
• Fairness: check performance across groups (gender, region, college tier) for models that affect people, such as hiring or lending.
• Calibration: when the model says 70%, is it right 70% of the time?

Check yourself

• A fraud model has 99% accuracy. Why might it still be useless?
• When do you prefer precision over recall? Give one example of each.
• How would you validate a sales-forecasting model?