Beyond Accuracy: Precision, Recall, and F1

What you will learn in this lesson:

• Why accuracy alone is misleading on imbalanced datasets
• How to read a confusion matrix and what each of its four cells means
• Precision, recall, and F1, explained intuitively and mathematically
• A worked numeric example, and how to reproduce it with scikit-learn
• When to prioritize precision versus recall in real applications

The Trap of Accuracy

Accuracy, the fraction of predictions a classifier gets right, feels like the obvious metric to optimize. It's simple, intuitive, and the first thing most people compute. But it can be dangerously misleading whenever your classes are imbalanced, meaning one outcome is much rarer than the other, which describes the large majority of interesting real-world classification problems.

Imagine building a model to detect a rare disease that affects 1% of patients. A model that simply predicts "no disease" for every single patient, without looking at any data at all, achieves 99% accuracy. It sounds excellent on paper, but it is completely useless: it catches zero actual cases of the disease, which is the entire point of building the model in the first place.

This kind of trap shows up constantly in fraud detection, rare-event prediction, spam filtering, and medical screening, anywhere one class vastly outnumbers the other. The more imbalanced the classes, the more impressive-looking, and the more meaningless, a high accuracy number becomes.

The Confusion Matrix

To understand the metrics that fix this problem, you need the confusion matrix, a simple table that breaks predictions into four categories for a binary classification problem:

A true positive is a correctly caught case (the model said "disease" and the patient actually had it). A false negative is a missed case (the model said "no disease" but the patient actually had it, a dangerous miss). A false positive is a false alarm (the model said "disease" but the patient was healthy). A true negative is a correctly cleared case.

Every single prediction a binary classifier makes falls into exactly one of these four buckets, and accuracy is simply (TP + TN) / (TP + TN + FP + FN), every other metric in this lesson is just a different, more targeted way of combining these same four counts.

In our disease example, a model that always predicts "no disease" would have zero true positives and zero false positives, but a very large number of false negatives, every actual case slips through entirely uncaught. The confusion matrix exposes this failure immediately and explicitly, where accuracy alone hides it behind a single reassuring number.

Precision and Recall

Precision answers: of all the cases the model flagged as positive, how many were actually positive?

Precision = TP / (TP + FP)

High precision means few false alarms. A spam filter with low precision would constantly flag legitimate emails as spam, annoying users even if it catches most real spam, the cost of a false positive (a real email buried) is what precision is protecting against.

Recall (also called sensitivity, or the true positive rate) answers: of all the cases that were actually positive, how many did the model successfully catch?

Recall = TP / (TP + FN)

High recall means few missed cases. A disease screening test with low recall would let many genuinely sick patients walk away thinking they're healthy, the cost of a false negative (a missed case) is what recall is protecting against.

Precision and recall typically trade off against each other, and that trade-off is usually controlled by a decision threshold sitting underneath the classifier (most classifiers output a probability, and you choose some cutoff, often 0.5 by default, above which you call the prediction "positive").

You can usually increase recall by lowering that threshold (catch more real cases, but also more false alarms), or increase precision by raising it (fewer false alarms, but more missed cases). The F1 score combines both into a single number using their harmonic mean, rather than a simple average:

F1 = 2 * (Precision * Recall) / (Precision + Recall)

The harmonic mean matters here: F1 is high only when both precision and recall are reasonably high. A model with precision 1.0 and recall 0.01 would have an arithmetic average of about 0.5, looking deceptively reasonable, but its F1 score is roughly 0.02, correctly flagging that the model is nearly useless despite "perfect" precision. F1 punishes models that achieve one metric at the severe expense of the other, making it a more honest single-number summary than accuracy for imbalanced problems.

A Quick Numeric Walkthrough

Suppose out of 1,000 patients, 10 actually have the disease. Your model flags 15 patients as positive, of which 8 are genuinely sick (true positives), and 7 are false alarms (false positives). That leaves 2 actually sick patients the model missed (false negatives), and the remaining 983 patients are correctly identified as healthy (true negatives).

• Precision = 8 / (8 + 7) = 8/15 ≈ 0.53 (about half of the flagged patients are truly sick)
• Recall = 8 / (8 + 2) = 8/10 = 0.80 (the model catches 80% of actual cases)
• F1 = 2 * (0.53 * 0.80) / (0.53 + 0.80) ≈ 0.64

Meanwhile, the overall accuracy here would be (8 true positives + 983 true negatives) / 1000 = 99.1%, a number that looks reassuring but tells you almost nothing useful about how well the model handles the rare, important class it was actually built to catch. The F1 score of 0.64 paints a far more honest picture of how the model is actually performing on the task that matters.

Computing These Metrics in Code

In practice, you'll rarely tally up TP, FP, FN, and TN by hand, scikit-learn computes all of these directly from a model's predictions. The snippet below reproduces something close to the disease-screening scenario above on a small synthetic dataset, and prints the full classification report:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
)

np.random.seed(0)

# Simulate a strongly imbalanced dataset: ~1% positive class, like the disease example
n_samples = 1000
y_true = (np.random.rand(n_samples) < 0.01).astype(int)

# A feature that's somewhat informative but noisy
X = np.random.randn(n_samples, 1) + y_true.reshape(-1, 1) * 2.5

model = LogisticRegression()
model.fit(X, y_true)
y_pred = model.predict(X)

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print("TP: {}  FP: {}  FN: {}  TN: {}".format(tp, fp, fn, tn))

print("Accuracy:  {:.4f}".format(accuracy_score(y_true, y_pred)))
print("Precision: {:.4f}".format(precision_score(y_true, y_pred, zero_division=0)))
print("Recall:    {:.4f}".format(recall_score(y_true, y_pred, zero_division=0)))
print("F1:        {:.4f}".format(f1_score(y_true, y_pred, zero_division=0)))

Run this a few times with different random seeds and watch how accuracy stays stubbornly high near 99% almost no matter what the model does, while precision, recall, and F1 swing around far more, reflecting how well the model is actually doing on the rare class that matters. This is the core lesson in concrete form: accuracy is nearly uninformative here, while the other three metrics are where the real signal lives.

When to Prioritize Precision vs. Recall

Which metric matters more depends entirely on the relative cost of the two types of mistakes in your specific application, there is no universally correct answer.

In fraud detection, a false positive (flagging a legitimate transaction as fraud) is annoying but recoverable, the customer gets a verification call. A false negative (letting actual fraud through) directly costs money. Many fraud systems are tuned to favor recall, sometimes accepting more false alarms to make sure fewer real fraud cases slip through, though the exact balance also depends on how costly false alarms are to customer trust and support load.

In medical screening, missing a real case (false negative) can be life-threatening, so recall is often prioritized heavily, it's better to flag some healthy patients for a follow-up test than to send a sick patient home undiagnosed. Precision matters too, of course, since too many false alarms creates unnecessary anxiety and strains healthcare resources, but the cost asymmetry usually pushes screening tests toward favoring recall over precision.

In contrast, an email spam filter usually prioritizes precision: users tolerate the occasional spam message slipping through (a false negative) far better than they tolerate an important email being wrongly buried in the spam folder (a false positive). There is no universally "correct" metric, the right choice depends entirely on which type of mistake costs you more in your specific application, and that cost should be decided by domain experts and stakeholders, not derived from the data alone.