Classification Metrics: 95.6% Accuracy, Yet Most Spam Warnings Are Wrong

Imagine a spam filter evaluated on 1,000 messages. Only 20 are spam. The filter catches 16 of those, misses four, and incorrectly flags 40 legitimate messages.

These are invented numbers, chosen to make the trade-off visible:

	Actually spam	Actually legitimate
Flagged as spam	16	40
Allowed through	4	940

The filter makes 956 correct decisions. Its accuracy is 95.6%. Yet of the 56 messages it flags, 40 are legitimate. A person opening the spam folder sees a system whose warnings are mostly wrong.

Both descriptions are numerically correct. They answer different questions.

Start with the denominator

Accuracy counts correct decisions among all decisions: (16 + 940) / 1000. In this example, a filter that never flags anything would score 98% accuracy. That baseline would miss every spam message, but it exposes how little accuracy alone tells us here.

Precision asks how many flagged messages really are spam: 16 / (16 + 40), or about 28.6%. It speaks to the reliability of the warning.

Recall asks how much of the actual spam was caught: 16 / (16 + 4), or 80%. It speaks to coverage of the target class.

False positives are the 40 legitimate messages flagged. False negatives are the four spam messages missed. Which is more costly depends on what the product does with its prediction. A warning banner and permanent deletion should not share a decision threshold merely because they use the same model.

A threshold turns a score into an action

Many classifiers produce a score. The system then chooses a cutoff for flagging a message. Lowering that cutoff usually catches more spam and also flags more legitimate mail. The exact effect must be measured on representative data; an uncalibrated score should not automatically be read as a probability.

A precision–recall curve shows the relationship across thresholds. It is particularly useful when the positive class is rare and the burden of false alerts matters. Saito and Rehmsmeier’s 2015 paper explains why precision–recall plots can be more informative than ROC plots on imbalanced datasets.

A ROC curve compares true-positive rate with false-positive rate. Its ideal corner is (0, 1): no false positives, all positives detected. An AUC value summarizes ranking across thresholds, but it does not choose an operating threshold or tell you that the resulting workflow is affordable.

There is no universal AUC cutoff that makes a classifier “good” regardless of the task.

One combined number still leaves things out

F1 is the harmonic mean of precision and recall. For this filter, it is 2 × 16 / (2 × 16 + 40 + 4), approximately 42.1%.

It penalizes a large imbalance between precision and recall, which can make it useful for comparisons. It does not account for true negatives, and it does not encode your actual costs. A system optimized for F1 can therefore be inappropriate for the product’s decision.

For a spam folder, begin with operational questions: how many legitimate messages may be misplaced, how much spam is acceptable, and can users recover a mistake? Select a threshold on validation data, then report its performance on held-out data with the confusion matrix.

“95.6% accurate” fits on a slide. The four cells tell the reader what living with the filter would feel like.