Incident 83: Spam filters are efficient and uncontroversial. Until you look at them.

Description: Gmail, Yahoo, Outlook, GMX, and LaPoste email inbox sites showed racial and content-based biases when AlgorithmWatch tested their spam box filtering algorithms.

Entities

View all entities
Alleged: Gmail , Outlook , Yahoo , GMX and LaPoste developed and deployed an AI system, which harmed email users.

Suggested citation format

Dickinson, Ingrid. (2020-10-15) Incident Number 83. in McGregor, S. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
83
Report Count
1
Incident Date
2020-10-15
Editors
Sean McGregor, Khoa Lam

Reports Timeline

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover

CSET Taxonomy Classifications

Taxonomy Details

Full Description

Gmail, Yahoo, Outlook, GMX, and LaPoste email inbox sites showed racial and content-based biases when AlgorithmWatch tested their spam box filtering algorithms. AlgorithmWatch sent hundreds of emails to 10 email accounts on the listed sites, and noticed emails would be filtered into the spam box if certain words were within the body of the email. A Nigerian students internship application was marked spam, but when the word "Nigeria" was removed it was delivered to the inbox. The same applied to a "sex education" email that was forwarded to inbox after removing "sex". A Joe Biden speech went through when the words "loan, investment, billion" were removed.

Short Description

Gmail, Yahoo, Outlook, GMX, and LaPoste email inbox sites showed racial and content-based biases when AlgorithmWatch tested their spam box filtering algorithms.

Severity

Unclear/unknown

Harm Distribution Basis

Race, National origin or immigrant status

Harm Type

Harm to civil liberties

AI System Description

Machine learning algorithms used to filter spam emails out of inboxes

System Developer

Gmail, Outlook, Yahoo, GMX, LaPoste

Sector of Deployment

Information and communication

Relevant AI functions

Perception, Cognition, Action

AI Techniques

Language recognition, content filtering

AI Applications

spam filtering

Named Entities

Gmail, Yahoo, Outlook, GMX, LaPoste, SpamAssassin, AlgorithmWatch

Technology Purveyor

Gmail, Yahoo, Outlook

Beginning Date

2020-10-22

Ending Date

2020-10-22

Near Miss

Unclear/unknown

Intent

Unclear

Lives Lost

No

Infrastructure Sectors

Communications

Data Inputs

inbound emails

Spam filters are efficient and uncontroversial. Until you look at them.

Spam filters are efficient and uncontroversial. Until you look at them.

algorithmwatch.org

Incident Reports

Spam filters are efficient and uncontroversial. Until you look at them.

An experiment reveals that Microsoft Outlook marks messages as spam on the basis of a single word, such as “Nigeria”. Spam filters are largely unaudited and could discriminate unfairly.

In an experiment, AlgorithmWatch sent a few hundred em…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents

Sexist and Racist Google Adsense Advertisements

· 27 reports

High-Toxicity Assessed on Text Involving Women and Minority Groups

· 9 reports

Facebook’s Political Ad Detection Reportedly Showed High and Geographically Uneven Error Rates

· 5 reports