Evaluation & RL
Threshold, Confusion Matrix & ROC
Understand how decision thresholds impact precision, recall, and the ROC curve.
Drag one threshold and watch the confusion matrix, precision, recall and the ROC point all move together. Then push it to either extreme and see one metric go perfect while the other collapses.
Stage 1 of 3: Two distributions
The model gave every example a score. Positives and negatives both spread out, and they overlap. That overlap is the entire problem — no threshold can separate what the scores do not. The threshold is 0.50. Accuracy is 72%.
- Actually positive
- Actually negative
- Your threshold
- Misclassified
How far apart the two score distributions sit.
The model gave every example a score. Positives and negatives both spread out, and they overlap. That overlap is the entire problem — no threshold can separate what the scores do not.
The Threshold is Not the Model
A classifier does not natively output "positive" or "negative". Instead, it outputs a continuous score—a probability or confidence level. The actual prediction comes from picking a number and saying, "Everything above this is positive."
That number is the decision threshold, and tuning it is one of the most consequential choices in deploying a machine learning model.
The Precision-Recall Trade-off
Because the score distributions for positive and negative examples overlap, no single threshold can perfectly separate them. Moving the threshold forces a trade-off:
- Raising the threshold means you make fewer positive predictions, but the ones you do make are more likely to be correct. Precision goes up, but Recall goes down because you miss genuine positives.
- Lowering the threshold casts a wider net. You catch more genuine positives (Recall goes up), but you also sweep up more false positives (Precision goes down).
The Confusion Matrix
The confusion matrix breaks down exactly what happens at a specific threshold by counting four outcomes:
- True Positives (TP): Predicted positive, actually positive.
- False Positives (FP): Predicted positive, actually negative (Type I error).
- True Negatives (TN): Predicted negative, actually negative.
- False Negatives (FN): Predicted negative, actually positive (Type II error).
Metrics like Precision, Recall, and Accuracy are just arithmetic on these four boxes.
The ROC Curve
If you plotted the True Positive Rate (Recall) against the False Positive Rate at every possible threshold, you would trace the Receiver Operating Characteristic (ROC) curve.
The ROC curve summarizes the model's intrinsic ability to separate the classes, independent of any specific threshold. A perfect model shoots straight up the Y-axis. A model that guesses randomly traces the diagonal.
Reference
- Precision
- TP / (TP + FP) — of what you flagged, how much was right
- Recall
- TP / (TP + FN) — of what was there, how much you found
- Accuracy
- (TP + TN) / total — misleading whenever the classes are imbalanced
- F1
- 2·P·R / (P + R) — the harmonic mean, so one bad value drags it down
- TPR
- same as recall — the ROC y axis
- FPR
- FP / (FP + TN) — the ROC x axis
- ROC
- every threshold at once, plotted as (FPR, TPR)
- AUC
- the probability a random positive outscores a random negative
Break it on purpose
Slide the threshold to the bottom and everything is predicted positive: recall hits 1.00 and precision falls to the base rate. Slide it to the top and precision looks perfect on a handful of predictions while recall approaches zero. Both are defensible-looking numbers, and both models are useless.