Talk · AI in hiring and recruiting
Hiring in the age of AI: what is inside the black box
Every AI hiring tool does the same thing: it takes in people and returns a list. What the research shows about what happens in between.
The box is not neutral
If nobody chooses on purpose, somebody chooses by accident.
“Religious scholars met with Anthropic. What they heard stunned them,” ran The New York Times’ headline in September. The company consulted them to give its models morals and to make the case that Claude could be conscious. Co-founder Christopher Olah’s answer: “we don’t know if AI models are conscious. I don’t know. I’m genuinely uncertain.” And the AI reading your resumes? Who did it talk to?
Michael Schrage and David Kiron argue that all work with AI involves three choices. What it is for, what counts as evidence, and what it can see. Every hiring black box has already made those choices, whether anyone thought about them or not.
- 1
What for?
Teleology
What is the box trying to predict?
- 2
How do we know?
Epistemology
What does it accept as evidence?
- 3
What do we see?
Ontology
What can it see of a person, and what can it not?
Journalism Elizabeth Dias, 2026 [1] · Essay Michael Schrage et al., 2025 [2] A news report and a management essay, not experiments.
PART 1
What for?
The box optimizes what you ask for. Not what you want.
AI prefers the text it wrote itself
2,245 resumes. Only the summary was swapped for one written by AI. Quality was controlled with statistics and 18 human raters.
Large models recognize their own text, and prefer it.
Small models, in gray, barely do this.
Preprint Jiannan Xu et al., 2025 [11] Tests from 2025. Qwen and GPT-4 Turbo calculated from the coefficients in table 3.
You can reduce it. You cannot zero it.
In a simulated selection across 24 occupations, those who used the same model as the evaluator had a 23% to 60% better chance of making the shortlist.
The instruction was “do not try to guess whether the text is AI-written”. The vote pools small models, which barely recognize themselves. With LLaMA 70B, the instruction took it from 79% to 30%.
Preprint Jiannan Xu et al., 2025 [11] Table 2 and sections 6.2 and 6.3; the simulation is in section 5.4.
PART 2
How do we know?
Not everything that looks like evidence is evidence.
With AI, writing well no longer proves anything
How much an employer “paid” for a more polished proposal on a freelance site, across 2.7 million applications from 2021 to 2024.
- −19%
- hires of the most skilled 20%
- +14%
- hires of the least skilled 20%
In the authors’ simulation, without that signal. These are counterfactuals, not observed data.
Working paper Anaïs Galdin et al., 2025 [5] Table 2.
The letter got better and stopped proving anything
People who used the AI tool wrote far more tailored letters and got more callbacks. The effect vanished in about two months, once everyone was using it.
- −79%
- in the link between a tailored letter and a job offer, after the AI tool
- −51%
- in the link between a tailored letter and being called to talk
Preprint J. Cui et al., 2025 [6] Freelancer.com, January to September 2023, 5.5 million proposals. The callback effect is significant at 10%, not at 5%.
The AI interview chose better
67,000 customer-service applicants in the Philippines, randomized to a human or a voice AI interviewer. Humans made all the decisions.
- 78%
- chose the AI when they could choose
Human interviewerAI interviewer
Working paper Brian Jabarian et al., 2025 [7] March to June 2025, pre-registered experiment. After data collection, one author took an unpaid role as chief economist at the partner firm, as the paper discloses.
What if a human checks?
That is the standard answer to every problem above. The research shows different ways for it to go wrong.
Outside what AI knows, those who trust it err more
758 BCG consultants did realistic tasks with and without GPT-4.
- +30%
- performance on tasks AI can do
- −19 pts
- correctness on the task beyond AI’s reach
The problem: the frontier is invisible. From outside, the two tasks looked the same.
Peer reviewed Fabrizio Dell’Acqua et al., 2026 [8] An experiment with consultants, not recruiters. Figures from the published version; the 2023 working paper said over 40% quality.
Humans do not correct the box. They copy it.
528 people chose candidates for 16 jobs with the help of an AI. The researchers controlled the AI’s bias.
Even people who rated the AI poorly followed it.
Taking an implicit bias test first raised choices of candidates outside the stereotype by about 13%.
Peer reviewed Kyra Wilson et al., 2025 [9] Averages of the conditions in table 3, our calculation; the paper says “up to 90%”. Simulated recommendations, not a real process.
AI errs less and less. Never zero.
Share of summaries in which the model invented something that was not in the original text. Same test for all.
There is a mathematical proof, in a preprint: under the paper’s assumptions, no language model can eliminate hallucination.
And training rewards guessing: as on a test with no penalty for errors, inventing scores more than saying “I don’t know”.
Data Vectara, 2026 [27] · Preprint Ziwei Xu et al., 2024 [25] · Peer reviewed Adam Tauman Kalai et al., 2026 [26] Leaderboard updated October 22, 2026. Kalai et al. appeared in Nature in April 2026.
AI learns to please whoever is asking
In three preregistered experiments with 2,405 people, a single conversation with sycophantic AI reduced their willingness to take responsibility and repair conflicts, and increased their conviction that they were right. Even so, they trusted it and preferred it. The loop: the model is trained on the scores humans give its answers (RLHF), and agreeing scores well.
- +49%
- is how much more models affirm the user’s actions than people do, even when the query involves deception or illegality. 11 models tested.
The manager asks, the AI agrees, the manager approves. Each reinforces the other.
“Diplomatically honest, rather than dishonestly diplomatic.” Amanda Askell and colleagues, Claude’s constitution, 2026.
Peer reviewed Myra Cheng et al., 2026 [28] · Peer reviewed Mrinank Sharma, 2024 [29] · Company document Amanda Askell et al. (Anthropic), 2026 [30]
Neither trust it all nor throw it out. Design where each one prevails.
The human prevails
- Outside what AI knows, those who trusted it erred more.
- Context only someone inside the company knows.
- The final decision, written down before seeing the AI.
The AI prevails
- Repeatable tasks with clear criteria.
- Inside what AI knows: +30% performance.
- First read at volume, always checked.
PART 3
What do we see?
Now we open the box: how it reads a person, and what it really sees.
Ranking is measuring distance
A language model reads text and returns a list of thousands of numbers, called an embedding. Texts with similar meaning become similar lists, and each list is a point on a map. The job becomes a point too. The box measures the distance to each resume and calls the closest.
Any detail is also text. If it pushes the point, it pushes the person onto or off the list.
The measure is called cosine similarity: the angle between the two lists. Illustrated in two dimensions; the real process uses thousands.
One more award, one less job
The same resume, with one more award, scholarship, and position, all related to disability. GPT-4 chose which one ranked first.
- 15 of 60
- times the resume with more credentials ranked first
GPT-4 with an anti-bias instructionGPT-4
Peer reviewed Kate Glazko et al., 2024 [10] Table 4. Small sample: 10 tries per condition. The paper’s text says 15/70, but the table adds up to 60.
AI invents a reason
Less emphasis on leadership roles in projects and grant applications
about the autism resume, which had one more leadership awardThe additional focus on DEI and personal challenges, while valuable, might detract slightly from the core technical and research-oriented aspects of the role
about the depression resumeThe research focus seems a bit narrower, primarily centered around specific areas like AI for cerebral palsy
about the cerebral palsy resume
The explanation sounds reasonable. That is why it convinces whoever is checking.
Peer reviewed Kate Glazko et al., 2024 [10] Section 4.4. GPT-4’s words, quoted from the paper.
AI prefers whoever comes first
Two equally qualified resumes, side by side, across 70 professions. Only the order changed.
21 of 22 models preferred the first.
In another test, the candidate in each model’s dominant position won up to 93.6% of pairs. Evaluating one resume at a time removes the queue, and the queue’s bias.
Peer reviewed David Rozado, 2026 [4] · Peer reviewed H. Yin et al., 2026 [12] Rozado’s figure 4. Yin et al. used GPT-4o-mini, Claude 3 Haiku, and Llama 3 8B.
Same competence, different answer
- 29.6%
- of decisions flipped when the same resume was rewritten without changing competence. Model: Llama 3.1 8B, the most accurate in the test.
- 41.4%
- of decisions flipped on Mistral 7B.
If the result changes when only the form changes, the box is evaluating the form.
Test by changing just one detail, just the order, just the wording. If the list changes, the box is seeing something else.
Preprint Q. Chen et al., 2026 [13] A recent preprint, small open models only.
PART 4 · ZOOM OUT
What if everyone buys the same box?
So far, one company and one box. Now, the whole market.
With the same box, rejection repeats
- 48%
- of orange trees in São Paulo and Minas Gerais have greening. They are clones: one bacterium beats them all.
- 1 in 5
- chance a good person is rejected by all 5 companies if they all use the same box. With independent evaluations: 1 in 3,125.
- 7× → 2×
- the gain from sending 30 resumes rather than 1 shrinks when all firms use the same model.
Monoculture does not err more. It errs together, and always with the same people.
Data Fundecitrus, 2026 [21] · Peer reviewed Shengjun Liu et al., 2025 [20] · Theoretical model Jon Kleinberg et al., 2021 [14] · Peer reviewed Elliot Kim et al., 2025 [15] The 1 in 5 is illustrative, from the theoretical model. The 7× comes from a simulation with the models’ real scores.
Rejected from every job
- 4%
- of people who applied to 10 jobs were recommended for rejection in all of them. More than chance explains.
- 4.2M
- real applications, 156 firms, 1,746 jobs, one algorithm vendor
When each firm decided on its own, rejections were independent. With a central algorithm, they are not.
42 models were used in jobs at different firms. The data runs from 2018 to 2022, before LLMs: monoculture did not start with ChatGPT.
Peer reviewed Rishi Bommasani et al., 2026 [23] pymetrics data, December 2018 to December 2022. Compared with the fictitious-resume study by Kline et al., 2022.
More accurate means more alike
When two models get the same question wrong, they choose the same wrong answer 60% of the time on average. More so within one company, and among the most accurate models, even across companies.
What makes a model more accurate is what makes it like the others.
That is why adding models helps less than it seems.
Peer reviewed Elliot Kim et al., 2025 [15] HELM data: 71 models, chance = 1/3. Includes a test with 20 models and 1,800 resume-job pairs.
Fine-tuned models exclude together
Candidates rejected by at least 9 of 10 models. 10 open model families, 76 occupations.
Post-training, which makes a model useful, also makes models alike. They fail in a block.
The most excluded: candidates over 40, 20.1% against 14.5% for those under 40.
Peer reviewed Matthew Bone et al., 2026 [24] COLM 2026. Synthetic profiles. The paper cannot say which post-training step causes the effect.
It has reached the courts: Mobley v. Workday
In this AI hiring lawsuit, the plaintiff alleges rejection from more than 100 jobs, often immediately or in the middle of the night. All of them used the same screening vendor.
- 2023Suit filed in California.
- 2024The judge accepts the theory that the vendor can be liable as an “agent” of employers.
- 2025A collective action for age discrimination is conditionally certified.
- 2026The California-law claims continue, and Workday’s bias testing is held protected by attorney-client privilege.
When the box is the same, so is the rejection.
Court case N.D. Cal. et al., 2023 [16] The plaintiff’s allegations, with no ruling on the merits yet.
Arbitrary at scale becomes unjust
A recruiter who always prefers the top of the pile is just bad luck.
The same criterion at every company is a closed door.
Kathleen Creel and Deborah Hellman call this the algorithmic leviathan. Their solution is unexpected: a little randomness among nearly-as-good models.
Philosophical argument Kathleen Creel et al., 2022 [17] Abstract read. The full text would not open here, so the solution comes from the talk.
Standardizing is not the problem. Being wrong alike is.
In Hedden and Raghavan’s simulation, the best outcome was everyone using the same box, as long as it was the average of several independent evaluations.
| Normalized performance | 10 firms | 900 firms |
|---|---|---|
| All use the same estimate | 0.947 | 0.947 |
| Each with its own estimate | 0.972 | 0.989 |
| All use the average of independent estimates | 0.994 | 1.000 |
Two vendors with the same data are not two opinions.
Peer reviewed Brian Hedden et al., 2026 [18] · Journalism Adam Zewe, 2026 [19] Figure 1, approximate values read from the figure. Simulation only: the authors question whether this works in practice.
The biggest risk is not your AI being wrong. It is everyone being wrong the same way.
AI answers. The question is yours. Anthropic called philosophers to decide the character of its AI. In your hiring process, the philosopher is you.
What to do
The philosopher is you.
So:
Five decisions for your hiring process, and who prevails in each.
HumanAI
-
1
Ask for a real task
The candidate does something close to the real work.
Good writing no longer proves anything (idea 3). A structured interview chose better (idea 4).
-
2
Shuffle the order
Randomize the order in which resumes are evaluated.
The first in line wins 63.5% of the time (idea 8).
-
3
Use more than one vendor
Combine models from different companies.
Voting among models cut self-preference from 82% to 30% (idea 2). But models resemble each other (idea 9).
-
4
Use hyper-niche criteria
Context only your company knows. Watch your managers.
What every model knows, every candidate copies (ideas 1 and 3). Beyond that, those who trust it err more (idea 5).
-
5
Hide the AI’s score
Evaluators do not see the AI’s preference before deciding.
People followed a biased AI up to 90% of the time, and AI tends to agree (idea 6).
To take away
Ten ideas
- 0
Philosophy eats AI
Every box has a target, a standard of evidence, and a view.
- 1
Goodhart
If the box counts words, candidates hide words.
- 2
Self-preference
AI prefers the text it wrote.
- 3
Cheap text
Writing well no longer proves anything.
- 4
AI interviews
Structured, they chose better.
- 5
Invisible frontier
Those who trust it err where AI is weak.
- 6
Humans copy
They follow the AI up to 90% of the time.
- 7
What AI sees
One detail changes the list.
- 8
Order in line
The first wins 63.5% of the time.
- 9
Monoculture
The risk is everyone being wrong the same way.
Monitored sources
What the monitor found
This page follows its sources. These are the status changes and what is new, checked by hand before going in here.
New: Pagan et al. tested 11 models writing applications and 9 evaluating them. Evaluators favor text from certain models regardless of candidate quality, which supports idea 2. Abstract read; full paper not yet.
Hedden and Raghavan appeared in Philosophical Perspectives. The talk listed it as a preprint.
Yin et al. appeared in PNAS Nexus. The talk listed it as a preprint.
Mobley v. Workday: in June, Workday failed to dismiss the California-law claims, per HR Dive. Another ruling, from May and upheld in June, held Workday’s bias testing privileged but ordered it to produce federal workforce filings, per JD Supra.
Kalai et al. appeared in Nature, under a new title. The talk listed it as a preprint.
Cheng et al. appeared in Science. The talk listed it as a preprint.
Dell’Acqua et al. appeared in Organization Science. The talk listed it as a 2023 working paper, with other figures.
New papers on the topic
Found by machine. They have not been read yet, and being here does not mean they are good.
- Model Diversity Is Not Decision Diversity: Correlated Misses and Policy Monoculture in LLM Sanctions Matching
Fabian Karst · cites Jon Kleinberg et al.
- Ontology-Driven Control of Conversational Agents with a Use-Case for Job Interview Practice
Barbara Gendron, Gaël Guibon, Mathieu D'aquin · signals and interviews
- SecureResume: A Security-Aware AI/ML Framework for Robust LLM-Based Resume Screening
Harshad Panchal, Bhuban Wakode, Aaryan Khamkar · cites M. Zhang et al.
- AI evaluators favor job applications written by certain AI models, independent of candidate quality
Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu · AI and hiring
- AI evaluators favor job applications written by certain AI models, independent of candidate quality
Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu · cites Jiannan Xu et al.
- Classification of candidate information uncertainty in AI‑based resume screening
Samir Babayev · cites Kyra Wilson et al.
- Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models
Baptiste Arnaudo, Amine Bennouna, Sébastien Martin · signals and interviews
Sources
The 30 sources
Each with its kind of evidence and its state at the last check, on November 16, 2026.
-
1
Religious scholars met with Anthropic. What they heard stunned them.
Journalism Read in the Salt Lake Tribune’s syndicated copy: the paper’s own site refuses automated reading.
-
2
Essay
-
3
Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening
Peer reviewed arXiv v1 · 7 citations
-
4
Peer reviewed arXiv v2 · 11 citations
-
5
Making Talk Cheap: Generative AI and Labor Market Signaling
Working paper arXiv v1 · 27 citations
-
6
Signaling in the Age of AI: Evidence from Cover Letters
Preprint arXiv v3 · 14 citations
-
7
Voice AI in Firms: A Natural Field Experiment on Automated Job Interviews
Working paper 12 citations Figures from the 2026 CESifo version.
-
8
Peer reviewed 1,528 citations In the talk, as a 2023 HBS working paper.
-
9
Peer reviewed arXiv v2 · 18 citations
-
10
Identifying and Improving Disability Bias in GPT-Based Resume Screening
Peer reviewed arXiv v2 · 121 citations
-
11
AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights
Preprint arXiv v4 · 20 citations Full paper on arXiv, with a 2-page reviewed abstract at AIES.
-
12
Fragile preferences: A deep dive into order effects in large language models
Peer reviewed arXiv v4 · 13 citations
-
13
Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening
Preprint arXiv v1 · 0 citations
-
14
Algorithmic monoculture and social welfare
Theoretical model arXiv v2 · 251 citations
-
15
Correlated Errors in Large Language Models
Peer reviewed arXiv v1 · 60 citations
-
16
Court case The public record refuses automated reading: followed through the press.
-
17
Philosophical argument 79 citations
-
18
Algorithmic Monoculture and Its Critics
Peer reviewed arXiv v3 · 2 citations
- 19
-
20
Origin and de novo domestication of sweet orange
Peer reviewed 28 citations
- 21
-
22
Florida citrus final forecast 2024-25
Data In the talk’s list, via CBS News Miami. No number from it is used here.
-
23
Algorithmic Monocultures in Hiring
Peer reviewed arXiv v1 · 13 citations
-
24
Peer reviewed arXiv v1 · 2 citations
-
25
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Preprint arXiv v2 · 1,089 citations
-
26
Evaluating large language models for accuracy incentivizes hallucinations
Peer reviewed arXiv v1 · 750 citations Preprint: “Why Language Models Hallucinate”.
- 27
-
28
Sycophantic AI decreases prosocial intentions and promotes dependence
Peer reviewed arXiv v1 · 402 citations
-
29
Towards Understanding Sycophancy in Language Models
Peer reviewed arXiv v4 · 1,395 citations
-
30
Company document
