Incident 43: Racist AI behaviour is not a new problem

Description: From 1982 to 1986, St George's Hospital Medical School used a program to automate a portion of their admissions process that resulted in discrimination against women and members of ethnic minorities.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover
Alleged: Dr. Geoffrey Franglen developed an AI system deployed by St George's Hospital Medical School, which harmed Women and Minority Groups.

Incident Stats

Incident ID
43
Report Count
4
Incident Date
1998-03-05
Editors
Sean McGregor

CSET Taxonomy Classifications

Taxonomy Details

Full Description

From 1982 to 1986, St George's Hospital Medical School used a program to autonomously select candidates for admissions interviews. The system, designed by staff member Dr. Geoffrey Franglen, used past admission data to select potential students based on their standardized university applications. After the program achieved 90-95% match with the admission panel’s selection of interview candidates, it was entrusted as the primary method to conduct initial applicant screening. In 1986, lecturers at the school recognized that the system was biased against women and members of ethnic minorities and reported the issue to Britain’s Commission for Racial Equality.

Short Description

From 1982 to 1986, St George's Hospital Medical School used a program to automate a portion of their admissions process that resulted in discrimination against women and members of ethnic minorities.

Severity

Moderate

Harm Distribution Basis

Race, Sex

Harm Type

Harm to civil liberties

AI System Description

A custom designed statistical analysis program that used data from past admissions decisions to select which university applicants to be given admissions interviews.

System Developer

Dr. Geoffrey Franglen

Sector of Deployment

Human health and social work activities

Relevant AI functions

Cognition

AI Techniques

Machine learning

AI Applications

decision support

Location

London, England

Named Entities

St George’s Hospital Medical School, Dr. Geoffrey Franglen, University Central Council for Admission, Commission for Racial Equality

Technology Purveyor

St George’s Hospital Medical School, Dr. Geoffrey Franglen

Beginning Date

1982-01-01T00:00:00.000Z

Ending Date

1986-01-01T00:00:00.000Z

Near Miss

Harm caused

Intent

Accident

Lives Lost

No

Laws Implicated

United Kingdom's Race Relations Act

Data Inputs

Standardized university admission form, Previous admission and regection decisions

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents