Accuracy vs AUC vs F1

The Short Answer

When evaluating a classification model (e.g., predicting Spam vs. Not Spam), you need a number to tell you how good it is.

• Accuracy is the simplest: Total Correct / Total Guesses. It is highly misleading on imbalanced datasets.
• F1 Score is the harmonic mean of Precision and Recall. It is much better for imbalanced datasets, but it requires you to pick a specific cutoff threshold (e.g., if probability > 0.5, predict Spam).
• AUC (Area Under the Curve) measures how well the model separates the classes regardless of what threshold you pick. It essentially asks: "If I pick a random spam email and a random real email, what is the probability the model gave the spam email a higher score?"

Where They Differ

Feature	Accuracy	F1 Score	AUC (ROC)
Threshold Dependent?	Yes. Requires hard predictions (0 or 1).	Yes. Requires hard predictions (0 or 1).	No. Uses raw probabilities (0.0 to 1.0).
Handles Imbalance?	Horribly.	Extremely well.	Very well.
Best Used For	Communicating with non-technical stakeholders on balanced data.	When you must deploy a model with a strict cutoff and False Positives/Negatives both matter.	Comparing the inherent predictive power of two different models.

The Problem of Thresholds

Most classification algorithms (like Logistic Regression or Random Forests) don't actually output "Yes" or "No". They output a continuous probability (e.g., 0.82).

To calculate Accuracy or F1, you have to draw a line in the sand—a Threshold. The default is usually 0.5. If the probability is > 0.5, you predict "Yes".

The problem is, 0.5 might not be the best threshold for your business problem! If you are predicting cancer, you might want to set the threshold to 0.1 to catch every possible case (maximizing Recall). If you calculate F1 at 0.5, you are evaluating a configuration of the model you won't even use in production.

This is why AUC is so powerful. It evaluates the model across all possible thresholds, from 0.0 to 1.0. It grades the model on its fundamental ability to rank positive items higher than negative items.

Choose Accuracy When

• Your classes are perfectly balanced: You have exactly 50% cats and 50% dogs in your dataset.
• False Positives and False Negatives have the exact same cost.
• Warning: Never use Accuracy on imbalanced data. If 99% of transactions are legitimate, a model that simply hardcodes "Predict Legitimate" every single time will achieve 99% Accuracy, despite being entirely useless.

Choose F1 When

• You have an imbalanced dataset and you care equally about catching the minority class (Recall) and not making false alarms (Precision).
• You are forced to deliver a hard prediction label rather than a probability score.
• Information Retrieval: F1 is the standard metric for search engines (you want all relevant documents, and only relevant documents).

Choose AUC When

• You want to evaluate the pure predictive power of the model before making a business decision about where to set the threshold.
• You are comparing two different algorithms (e.g., Random Forest vs XGBoost) and want to know which one separated the data better overall.
• Note: AUC is sometimes insensitive to highly extreme imbalances. In those rare cases, PR-AUC (Area Under the Precision-Recall Curve) is preferred over ROC-AUC.

What People Get Wrong

Optimizing the wrong metric during training

During model training (e.g., the .fit() step in Python), the model is actually optimizing its internal Loss function (like Log Loss/Cross Entropy). Accuracy, AUC, and F1 are Evaluation Metrics, calculated after training to help humans understand performance. You cannot strictly optimize a neural network for Accuracy directly because Accuracy is not a differentiable mathematical function.