What Is AI Bias?

AI bias is when a system produces systematically worse outcomes for some groups of people than others, in a way nobody intended and nobody noticed at first. It is rarely a line of code saying who to disadvantage. It is almost always a consequence of what the system was trained on, what it was asked to predict, and how its accuracy was measured. Stanford HAI defines it as a system producing results that favor or discriminate against certain groups.

Five cases that are actually documented

A health algorithm that used cost as a stand-in for illness. In October 2019, researchers led by Ziad Obermeyer published a study in Science of a commercial algorithm US hospitals used to flag patients for extra care. It predicted future healthcare spending, on the assumption that sicker people cost more. But Black patients at the same level of illness had historically received less care, so they cost less, so the algorithm scored them as healthier. Nature's coverage reported that 17.7% of patients flagged for extra help were Black, against 46.5% if it had tracked illness. Algorithms of this kind are used to help manage care for about 200 million people in the United States each year.

A risk score used in criminal courts. ProPublica's "Machine Bias" investigation, published 23 May 2016, analyzed more than 7,000 COMPAS risk assessments from Broward County, Florida. Among defendants who did not go on to reoffend, 44.9% of Black defendants had been labeled higher risk, against 23.5% of white defendants. Northpointe, the company behind COMPAS, disputed the methodology, arguing that the score was equally calibrated by race, so that a given score carried the same likelihood of reoffending for a Black defendant and a white one. Both measurements held. The conflict between them turns out to be mathematical rather than a matter of one side measuring badly.

Face recognition that failed unevenly. NIST's Face Recognition Vendor Test Part 3, published in December 2019, found false positive rates in one-to-one matching ten to one hundred times higher for Asian and African American faces than for white faces, and, in the one-to-many searches police use, higher false positive rates for African American women, though NIST noted that not all algorithms showed this and that the most equitable ones were also among the most accurate. The same report found that a number of algorithms developed in China produced low false positive rates on East Asian faces, sometimes lower than on Caucasian faces, which points at the training data rather than anything intrinsic to the faces. Joy Buolamwini and Timnit Gebru's 2018 "Gender Shades" study had already found gender classification error rates up to 34.7% for darker-skinned women against 0.8% for lighter-skinned men. In January 2020, Detroit police arrested Robert Williams in front of his family after a search matched a blurry shop camera image to his driving licence photo. He was held thirty hours, and the case settled in June 2024.

Automated rejection in hiring. The live test of the learned kind is Mobley v. Workday, filed in February 2023, alleging that screening software discriminated on race, age and disability. A federal court granted preliminary collective certification of the age claim in May 2025, and the opt-in period for that collective closed in March 2026. The case has not been decided, and nothing has been proven. The contrast is a case that has already resolved. In September 2023 the US Equal Employment Opportunity Commission settled a case against iTutorGroup for $365,000. The EEOC alleged the company had programmed its application software to automatically reject female applicants aged 55 and over and male applicants aged 60 and over, turning away more than 200 qualified people in the United States. The suit was resolved by a consent decree, so no court ruled on the allegation. What was alleged there was an explicit rule, not a learned pattern, which is the older kind of discrimination that law already knew how to name. The harder problem, and the one this article is about, is the kind nobody wrote down.

A regulator penalising a government department. In December 2021 the Dutch Data Protection Authority fined the Netherlands Tax Administration 2.75 million euros over its handling of nationality data, covering three unlawful processing practices, one of which was using nationality as an indicator in an automated risk-scoring system.

Two completely different things are called bias

In statistics and machine learning, bias is a technical quantity with no moral content: the error a model makes because its assumptions are too simple to capture the real pattern. A model that assumes a straight line where the truth curves has high bias. That is one half of the bias-variance tradeoff, the balance between a model too rigid to learn the shape of the data and one so flexible it memorises accidents.

In the social sense, bias means the system disadvantages people, which is what every case above describes. The two senses are unrelated. Reducing statistical bias will not make a model fair, and a perfectly fitted model can be flagrantly unfair if it fits an unfair pattern accurately. Neither sense is the same as AI hallucinations, where a model states something that is simply false: a biased system can be accurate about everything it says and still sort people badly. When someone says they reduced bias, ask which kind. NIST's 2022 report splits the ground a third way: systemic bias from institutional practice, statistical bias from unrepresentative samples, and human bias from how people judge. Its argument is that AI bias is socio-technical and cannot be fixed inside the model.

Four mechanisms that produce it

Unrepresentative training data. If a face dataset is mostly light-skinned, the model learns those faces best. Buolamwini and Gebru found the benchmarks were already skewed before anyone trained on them. What ends up in a training set is rarely a considered sample of the world, and usually reflects where AI gets its training data, which is largely whatever could be collected at scale. This is the mechanism people know, and the easiest to fix.

A proxy standing in for the thing you care about. Obermeyer and colleagues call this label choice bias, and argue in their 2021 Algorithmic Bias Playbook that it is more common than unrepresentative data. No database has a column called "future health need", so the team picks a measurable stand-in, such as cost. The same problem runs through ordinary annotation: supervised systems learn from labeled data, and every label is a human judgment about which category something belongs in, made against a guideline that somebody wrote. Change the guideline and you change what the model learns.

Historical patterns treated as the target. Train a system on past hiring decisions and you teach it to reproduce them, including the parts nobody would defend out loud. That is pattern-matching against what happened before, which is how AI works. It also compounds. A system that sends attention to one neighborhood generates more records from that neighborhood, and those records become the next round of training data, which is the standing concern about predictive policing.

Evaluation that measures the wrong thing. A model can be 95% accurate overall and near-useless for a group making up 4% of the test set. The systems Gender Shades tested were reported as accurate, and were, on average.

Does ChatGPT have bias?

In August 2024, Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky and Sharese King published a study in Nature on dialect prejudice in large language models. Asked about African Americans directly, the models produced overtly positive descriptions. Asked to judge text written in African American English, with no mention of race, the same models assigned less prestigious jobs and reached harsher verdicts in hypothetical criminal scenarios. Conviction came out at 68.7% for the African American English text against 62.1% for Standard American English. In first-degree murder scenarios, death sentences came out at 27.7% against 22.8%. Those were research scenarios, not court outcomes. What matters most is that training with human feedback reduced the overt stereotypes while leaving the covert ones intact, so visible and real improvement can come apart.

OpenAI ran its own English-language measurement in October 2024 and reported harmful gender stereotype rates below 0.1% on typical prompts, but above 2% on open-ended creative tasks in one older model.

Automation bias: the human half

A biased output only becomes a biased decision if a person acts on it. Automation bias is the tendency to over-trust a computer's recommendation even when other evidence contradicts it. Kate Goddard, Abdul Roudsari and Jeremy Wyatt's systematic review in the Journal of the American Medical Informatics Association screened 13,821 papers and analyzed the 74 that met their criteria. Automation bias rises under workload, time pressure and task complexity, and varies with how far the user trusts the system. It falls with training, with explicit user accountability, and when the system presents information rather than a recommendation. Knowing when to override a system is one of the capabilities every published definition of AI literacy names.

That last finding is the practical one. A tool that says "here are three similar cases" produces better judgment than one that says "reject this application". The Williams arrest shows the chain: a system returned a candidate match, officers treated it as an identification, and the checks that should have caught it did not happen.

What mitigation can and cannot do

The Obermeyer team worked with the algorithm's manufacturer to retrain it on measures of health rather than cost, and reported an 84% reduction in bias. That is a substantial result and it is not elimination. The deeper difficulty is that fairness targets conflict mathematically. Northpointe's calibration claim and ProPublica's error rate finding were both correct at the same time. Alexandra Chouldechova showed in 2016 that when the underlying rates differ between groups, a score cannot satisfy both criteria at once. Choosing between them is a policy decision rather than a technical one, and it belongs with the other things AI cannot do on your behalf.

Three things help, none sufficient: accuracy reported for each subgroup rather than overall, documentation such as a model card stating intended use and results by group, and a reviewer with authority to override, which only works if automation bias is designed against, not assumed away.

Where that leaves the question

Bias in AI is not a fault that a better model removes. It is what happens when a system learns from records of decisions people already made, and is then judged by a single accuracy figure that hides who it fails. Chouldechova's result is the part worth carrying away: where the underlying rates differ between groups, no scoring system can be equally calibrated and equally accurate for everyone at once. Someone has to decide which kind of error matters more, and that is a choice a person should be answerable for rather than one a system makes quietly.

So the useful question to ask of anything that sorts people is not whether it is biased. It is whether anyone has measured what it does to each group, written the result down, and given a named person the standing to overrule it. Where that was done, as when the Obermeyer team retrained the health algorithm, the harm fell sharply. Where it was not, the harm is usually still there and simply unmeasured.