Guides

A guide by Vitor Araujo

Algorithmic bias

The 10 questions people ask most about algorithmic bias.

Japanese print: blind monks feel an elephant, each on a different part of its body.
Blind monks examining an elephant, by Hanabusa Itchō. Each touches one part and describes a different animal. Public domain. Wikimedia Commons

What you will read

01

What is algorithmic bias?

Algorithmic bias is a computer system’s tendency to repeatedly produce unfair outcomes, such as favoring one group over another. It can arise from design choices and from how data is collected, organized or used to train the system.

Engraving: a woman operates a counting machine in front of a panel of dials.
Hollerith’s tabulating machine counted the 1890 US census (Scientific American, 1890). Public domain. Wikimedia Commons

The problem matters in uses such as screening job applications, where AI can unfairly exclude people. According to a report by NIST, the US standards institute, addressing bias requires looking at human, institutional and social factors, not just data and software.

  1. Wikipedia records that Amazon shut down an AI application-screening system after finding that it discriminated against women, excluding resumes containing the word “women’s” and applicants from all-women’s colleges.Algorithmic bias · Wikipedia contributors, Wikipedia, 2026
  2. The NIST report identifies systemic, statistical and human biases and warns that human oversight helps but does not eliminate the risk of bias.Towards a Standard for Identifying and Managing Bias in Artificial Intelligence · Reva Schwartz et al., NIST Special Publication 1270, 2022
  3. A study published at the EAAMO conference identifies seven sources of harm across data collection, development and deployment of machine learning systems.A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle · Harini Suresh, John V. Guttag, EAAMO 2021, 2021

What I think: I recommend measuring who is favored or harmed before expanding an AI system’s use, and reviewing both its data and the human choices behind it.

02

What is algorithmic racism?

Algorithmic racism is the reproduction or reinforcement of racism through technologies such as search engines and image-analysis systems. Researcher Tarcízio Silva describes it as an update of structural racism for the age of data.

The issue matters because these systems can reinforce stereotypes and harm the representation of Black people. The discussion concerns both the data used and the categories the technology adopts to organize the world.

  1. A study by Tarcízio Silva collects public cases of racist results from image-analysis systems and examines how Black people appear or become invisible in their data.Visão computacional e racismo algorítmico: branquitude e opacidade no aprendizado de máquina · Tarcízio Silva, Revista da ABPN 12(31), 2020
  2. In an interview, Tarcízio Silva explains that technologies can strengthen the racially unequal distribution of knowledge, resources, space and violence against non-white groups.O racismo algorítmico é uma espécie de atualização do racismo estrutural. Entrevista com Tarcízio Silva · Daiane Batista (entrevista com Tarcízio Silva), Instituto Humanitas Unisinos (IHU), republicada do CEE-Fiocruz, 2023
  3. According to Wikipedia's entry on her book, Safiya Umoja Noble argues that search engines reflect the prejudices of society and their creators, reinforcing stereotypes about Black, Latina and Asian women.Algorithms of Oppression · Wikipedia contributors, Wikipedia, 2026

What I think: I favor evaluating how these systems treat different racial groups before expanding their use, without assuming their results are neutral.

03

Where does AI bias come from?

AI bias comes from training data, the choices of people who build and use it, and inequalities in society. It can also arise when a system pursues a target that does not reflect people's actual needs.

This already has consequences beyond the screen: a widely used health algorithm disadvantaged Black patients by predicting medical spending instead of illness. Even ordinary text can teach AI historical biases about race and gender.

  1. A report by NIST, the US standards institute, warns that human, institutional and social sources of bias receive less attention than data and algorithms.Towards a Standard for Identifying and Managing Bias in Artificial Intelligence · Reva Schwartz et al., NIST Special Publication 1270, 2022
  2. Researchers identified seven sources of harm that can arise during data collection, development and deployment of machine learning systems.A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle · Harini Suresh, John V. Guttag, EAAMO 2021, 2021
  3. A study published in Science showed that a model trained on ordinary text learned human biases along with the meaning of words.Semantics derived automatically from language corpora contain human-like biases · Aylin Caliskan, Joanna J. Bryson, Arvind Narayanan, Science, 2017
  4. A study published in Science found a health algorithm that assigned Black patients the same risk as healthier White patients because it predicted spending, and less was spent on Black patients.Dissecting racial bias in an algorithm used to manage the health of populations · Ziad Obermeyer et al., Science, 2019

What I think: I advocate evaluating the data, the chosen target and the effects on people before expanding AI use, rather than treating the technology as neutral.

04

Is facial recognition biased?

Yes, facial recognition can have different error rates depending on race and sex, and those differences vary by algorithm.

In policing, a mistaken identification can lead to handcuffs and questioning. In Sergipe, a mistaken police stop led to the system being suspended in 2024.

  1. A NIST study found differences between demographic groups in most systems tested, with higher false identification rates for Asian and African American faces than white faces.NIST Study Evaluates Effects of Race, Age, Sex on Face Recognition Software · National Institute of Standards and Technology (NIST), NIST news release on NISTIR 8280, 2019
  2. Olhar Digital reported that Thaís Santos was handcuffed after a system twice wrongly identified her as a fugitive.Foi presa! IA confunde mulher com foragida da Justiça em Aracaju · Alessandro Di Lorenzo, Olhar Digital, 2023
  3. According to Folhapress, Sergipe suspended the system after police searched and questioned a fan who had been wrongly identified.Sergipe suspende uso de reconhecimento facial após abordagem errada da PM · Folhapress, TNH1, 2024

What I think: I recommend measuring errors by group and requiring human review of identifications before expanding use.

05

What are the best-known cases of algorithmic bias?

Well-known cases include Amazon’s résumé screening tool and COMPAS, used in criminal justice. Others involve facial gender classification and a health care algorithm that disadvantaged Black patients.

The problem goes beyond a wrong answer: these systems can treat groups unequally in decisions about employment, criminal justice and health care. Those differences can arise from choices about data or what a system is designed to predict.

  1. Wikipedia records that Amazon shut down an AI hiring tool that excluded résumés containing the word “women’s” and applicants from all-women’s colleges.Algorithmic bias · Wikipedia contributors, Wikipedia, 2026
  2. The Gender Shades study found error rates of up to 34.7% for darker-skinned women across three commercial gender classifiers, compared with at most 0.8% for lighter-skinned men.Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification · Joy Buolamwini, Timnit Gebru, Proceedings of Machine Learning Research 81, 77-91 (Conference on Fairness, Accountability and Transparency), 2018
  3. A ProPublica investigation found that COMPAS wrongly flagged Black defendants as future criminals at almost twice the rate of white defendants; its maker, Northpointe, disputed the analysis.Machine Bias · Julia Angwin et al., ProPublica, 2016
  4. A study published in Science found that a health care algorithm underestimated Black patients’ needs because it predicted health care spending rather than illness, and less was spent on those patients.Dissecting racial bias in an algorithm used to manage the health of populations · Ziad Obermeyer et al., Science, 2019

What I think: I recommend measuring errors by group and checking what a system is trying to predict before expanding its use.

06

Can you fix bias in a model?

Specific biases in a model can be reduced, including by correcting what it tries to predict, but that does not guarantee removing all bias. Different definitions of fairness can also conflict, requiring choices.

That distinction matters when models guide who receives help: in health care, predicting spending rather than illness can disadvantage Black patients. And a technique presented as a fix may merely hide bias rather than remove it.

  1. A study found that a health algorithm disadvantaged Black patients by predicting spending rather than illness; correcting this would raise the share of Black patients receiving extra help from 17.7% to 46.5%.Dissecting racial bias in an algorithm used to manage the health of populations · Ziad Obermeyer et al., Science, 2019
  2. Researchers proved that, except in highly constrained special cases, no method can satisfy three common fairness conditions for risk scores at the same time.Inherent Trade-Offs in the Fair Determination of Risk Scores · Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan, ITCS 2017, 2016
  3. A study found that two methods for correcting gender bias in word representations mostly hid the bias rather than removed it.Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them · Hila Gonen, Yoav Goldberg, NAACL 2019, 2019

What I think: I recommend reviewing what the model should predict and measuring each new correction’s effects on the groups affected before promising that it is fair.

07

How do you test whether an AI is biased?

One way to test whether an AI is biased is to compare its decisions about equally qualified résumés, changing only names, presentation order, or wording. Differences in the results can reveal a preference for features that do not represent competence.

In résumé screening, language models have shown preferences for certain names and for the résumé presented first. A convincing explanation therefore cannot replace comparing results when only one detail changes.

  1. A study using over 500 résumés across nine occupations found that the systems tested favored White-associated names in 85.1% of cases.Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval · Kyra Wilson, Aylin Caliskan, AIES 2024, 2024
  2. A study using equally qualified résumés across 70 occupations found that 21 of 22 models preferred the résumé presented first.Gender and positional biases in LLM-based hiring decisions: evidence from comparative CV/résumé evaluations · David Rozado, PeerJ Computer Science 12:e3628, 2026
  3. A study still under review found that rewriting résumés without changing competence reversed 29.6% of decisions in Llama 3.1 8B and 41.4% in Mistral 7B.Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening · Q. Chen, Y. Xiao, arXiv, em revisão, 2026

What I think: I would test names, order, and wording before expanding AI use, without treating a well-written answer as proof of a fair decision.

08

Does human review fix AI bias?

Human review can help, but it does not eliminate AI bias: people can also copy the AI’s errors. Having someone review a decision does not guarantee it will be free of bias.

Policies requiring human oversight of government algorithms can create a false sense of security when they expect people to perform reviews they cannot carry out. The problem also involves human, institutional and social choices, not just data and algorithms.

  1. A report by NIST, the US standards institute, says keeping a person involved in oversight helps but does not solve everything, and zero risk of bias is not possible.Towards a Standard for Identifying and Managing Bias in Artificial Intelligence · Reva Schwartz et al., NIST Special Publication 1270, 2022
  2. A study of 41 policies requiring human oversight of government algorithms found that they expect people to perform reviews that the evidence suggests they cannot carry out.The Flaws of Policies Requiring Human Oversight of Government Algorithms · Ben Green, Computer Law & Security Review 45, 2022
  3. In a study involving simulated medical diagnosis, people copied a biased AI’s errors and continued making them after it stopped offering suggestions.Humans inherit artificial intelligence biases · Lucía Vicente, Helena Matute, Scientific Reports 13, 15737, 2023

What I think: I would test whether people and AI repeat the same errors before expanding use, rather than treat human review as a guarantee.

09

Does the order of options change an AI's answer?

Changing the order of options can change an AI's answer, even when the options themselves stay the same. In multiple-choice questions, models can favor certain labels, such as option A.

This effect is not limited to test questions: it also appears in résumé comparisons. Position can influence an AI's choice even when people have equivalent qualifications.

  1. A study tested 20 language models and found sensitivity to option position, linked to a preference for certain labels.Large Language Models Are Not Robust Multiple Choice Selectors · Chujie Zheng et al., ICLR 2024, 2024
  2. A preliminary study found changes of roughly 13% to 75% in model performance simply by reordering options, depending on the test.Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions · Pouya Pezeshkpour, Estevam Hruschka, arXiv, 2023
  3. A study comparing two equally qualified résumés across 70 occupations found that 21 of 22 models preferred the résumé shown first.Gender and positional biases in LLM-based hiring decisions: evidence from comparative CV/résumé evaluations · David Rozado, PeerJ Computer Science 12:e3628, 2026

What I think: I recommend testing the same comparison with the options in different orders before trusting AI with a selection task.

10

What does Brazilian law say about automated decisions?

Brazil’s data protection law, the LGPD, gives people the right to request a review of decisions made solely through automated processing of their personal data that affect their interests. The law does not require that review to be carried out by a person.

It is important to distinguish the law in force from proposed changes: the AI bill approved by the Senate provides for human review in certain cases, but, as of November 10, 2026, it was still awaiting a report in the Chamber of Deputies.

  1. Article 20 of the LGPD also guarantees clear information about the criteria used, subject to trade secrets, and allows the national authority to audit for discrimination if that information is withheld on those grounds.Lei nº 13.709, de 14 de agosto de 2018 (Lei Geral de Proteção de Dados Pessoais), art. 20 · Brasil, Planalto, texto compilado, 2018
  2. In July 2019, the President vetoed a requirement for review by a person, arguing that it would make many companies’ business models unviable and affect credit risk analysis.Mensagem nº 288, de 8 de julho de 2019 (veto parcial ao projeto que originou a Lei nº 13.853/2019) · Presidência da República, Câmara dos Deputados, Legislação Informatizada, 2019
  3. According to Agência Senado, the bill approved in December 2024 provides for explanations and human review of decisions with relevant legal impact.Senado aprova regulamentação da inteligência artificial; texto vai à Câmara · Agência Senado, Agência Senado, 2024
  4. As of November 10, 2026, the Chamber listed PL 2338/2023 as awaiting its rapporteur’s report in the Special Committee on Artificial Intelligence and still subject to a vote of the full Chamber.PL 2338/2023, ficha de tramitação · Câmara dos Deputados, Câmara dos Deputados, 2026

What I think: I recommend asking for clear criteria and review by a person while new rules are being debated, without confusing that request with a human-review requirement already established by the LGPD.

Who writes this

I’m Vitor Araujo, an engineer. I build AI products for companies, and I speak and write about AI ethics and the philosophy of technology. More about me.

Other guides