Incident 146: Research Prototype AI, Delphi, Reportedly Gave Racially Biased Answers on Ethics

Description: A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.


New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History
Alleged: Allen Institute for AI developed and deployed an AI system, which harmed Minority Groups.

Incident Stats

Incident ID
Report Count
Incident Date
Sean McGregor, Khoa Lam

GMF Taxonomy Classifications

Taxonomy Details

Known AI Goal

Question Answering

Known AI Technology

Distributional Learning, Language Modeling

Potential AI Technology


Known AI Technical Failure

Distributional Bias, Gaming Vulnerability

Potential AI Technical Failure

Overfitting, Robustness Failure, Context Misidentification, Limited Dataset

The AI oracle of Delphi uses the problems of Reddit to offer dubious moral advice · 2021

Got a moral quandary you don’t know how to solve? Fancy making it worse? Why not turn to the wisdom of artificial intelligence, aka Ask Delphi: an intriguing research project from the Allen Institute for AI that offers answers to ethical di…

Scientists Built an AI to Give Ethical Advice, But It Turned Out Super Racist · 2021

We’ve all been in situations where we had to make tough ethical decisions. Why not dodge that pesky responsibility by outsourcing the choice to a machine learning algorithm?

That’s the idea behind Ask Delphi, a machine-learning model from t…

Can a Machine Learn Morality? · 2021

Researchers at an artificial intelligence lab in Seattle called the Allen Institute for AI unveiled new technology last month that was designed to make moral judgments. They called it Delphi, after the religious oracle consulted by the anci…


A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents