Bias in AI: Equal Accuracy Can Hide Unequal Errors

Two groups receive predictions from the same system. Each group gets 90% accuracy. The dashboard looks reassuringly even.

Now look at the mistakes.

Imagine a tool that flags urgent support requests for faster review. In each group of 100 requests, 20 genuinely need urgent attention. These numbers are synthetic, not a study of any demographic group:

Result	Group A	Group B
Urgent, correctly flagged	18	10
Urgent, missed	2	10
Non-urgent, incorrectly flagged	8	0
Non-urgent, correctly left unflagged	72	80

Both columns contain 90 correct decisions. But the system catches 90% of urgent requests in A and only 50% in B. Equal accuracy has concealed a substantial difference in access to timely help.

A disparity is a reason to investigate

The table does not tell us what caused the gap. Perhaps the training data contains fewer examples of how B describes urgency. Perhaps the labeling process misunderstands certain expressions. Perhaps the input channel strips information from one group’s requests.

Start by examining those possibilities instead of jumping directly to a mathematical adjustment. Inspect missed cases, confirm that the outcome labels mean the same thing, and check sample sizes and uncertainty. A tiny subgroup estimate should not be presented with the confidence of a large one.

Performance can differ across intersections too. A system might appear acceptable when evaluated separately by age and by language while failing a particular age–language combination. Which breakdowns are relevant depends on the use, the available evidence, and the people affected.

The Gender Shades study, published in 2018, is an influential example of examining commercial gender-classification performance across skin-tone and gender groups. Its findings concern the systems and task studied; they are not a universal error estimate for all AI.

Historical labels are not automatically neutral

Suppose “urgent” was assigned whenever an agent escalated a ticket. Those labels may partly describe agent behavior, staffing levels, and who was believed—not just the underlying urgency.

A model that reproduces the labels faithfully could reproduce the same unequal treatment. Better prediction against the historical target would then be an incomplete measure of improvement.

Removing a sensitive attribute also does not guarantee fairness. Other variables may act as proxies, and excluding an attribute from prediction does not remove the need to evaluate relevant disparities where that evaluation is appropriate and lawful.

There is no single fairness dial

We could require similar recall across groups, similar false-positive rates, well-calibrated risk estimates, or other properties. These criteria answer different concerns and cannot always be satisfied together. Kleinberg, Mullainathan, and Raghavan show formal tensions between particular fairness conditions under specified assumptions.

That does not excuse a bad system. It makes the decision explicit. In the support example, missed urgent requests and unnecessary escalations have different consequences. People responsible for the service must decide which harms to prioritize, with input from affected users, rather than letting a default metric decide silently.

Possible remedies include improving coverage, revising labels, changing the task, adding review, or choosing a different model. Changing group-specific thresholds is one possible intervention in some settings, not a universal prescription.

The next version should be evaluated on fresh evidence and on the actual workflow. If a flag now reaches a queue nobody monitors, improved recall has not delivered faster help. Fairness work continues past the score table, into what happens to the person behind the request.