Incident 146: Research Prototype AI, Delphi, Reportedly Gave Racially Biased Answers on Ethics

Description: A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.


New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover
Alleged: Allen Institute for AI developed and deployed an AI system, which harmed Minority Groups.

Incident Stats

Incident ID
Report Count
Incident Date
Sean McGregor, Khoa Lam

GMF Taxonomy Classifications

Taxonomy Details

Known AI Goal

Question Answering

Known AI Technology

Distributional Learning, Language Modeling

Potential AI Technology


Known AI Technical Failure

Distributional Bias, Gaming Vulnerability

Potential AI Technical Failure

Overfitting, Robustness Failure, Context Misidentification, Limited Dataset


A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents