Incident 83: Spam filters are efficient and uncontroversial. Until you look at them.

Description: Gmail, Yahoo, Outlook, GMX, and LaPoste email inbox sites showed racial and content-based biases when AlgorithmWatch tested their spam box filtering algorithms.
Alleged: Gmail , Outlook , Yahoo , GMX and LaPoste developed and deployed an AI system, which harmed email users.

Suggested citation format

Dickinson, Ingrid. (2020-10-15) Incident Number 83. in McGregor, S. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
83
Report Count
1
Incident Date
2020-10-15
Editors
Sean McGregor, Khoa Lam

Tools

New ReportNew ReportDiscoverDiscover

CSET Taxonomy Classifications

Taxonomy Details

Full Description

Gmail, Yahoo, Outlook, GMX, and LaPoste email inbox sites showed racial and content-based biases when AlgorithmWatch tested their spam box filtering algorithms. AlgorithmWatch sent hundreds of emails to 10 email accounts on the listed sites, and noticed emails would be filtered into the spam box if certain words were within the body of the email. A Nigerian students internship application was marked spam, but when the word "Nigeria" was removed it was delivered to the inbox. The same applied to a "sex education" email that was forwarded to inbox after removing "sex". A Joe Biden speech went through when the words "loan, investment, billion" were removed.

Short Description

Gmail, Yahoo, Outlook, GMX, and LaPoste email inbox sites showed racial and content-based biases when AlgorithmWatch tested their spam box filtering algorithms.

Severity

Unclear/unknown

Harm Distribution Basis

Race, National origin or immigrant status

Harm Type

Harm to civil liberties

AI System Description

Machine learning algorithms used to filter spam emails out of inboxes

System Developer

Gmail, Outlook, Yahoo, GMX, LaPoste

Sector of Deployment

Information and communication

Relevant AI functions

Perception, Cognition, Action

AI Techniques

Language recognition, content filtering

AI Applications

spam filtering

Named Entities

Gmail, Yahoo, Outlook, GMX, LaPoste, SpamAssassin, AlgorithmWatch

Technology Purveyor

Gmail, Yahoo, Outlook

Beginning Date

2020-10-22

Ending Date

2020-10-22

Near Miss

Unclear/unknown

Intent

Unclear

Lives Lost

No

Infrastructure Sectors

Communications

Data Inputs

inbound emails

Incidents Reports

An experiment reveals that Microsoft Outlook marks messages as spam on the basis of a single word, such as “Nigeria”. Spam filters are largely unaudited and could discriminate unfairly.

In an experiment, AlgorithmWatch sent a few hundred emails to 10 email inboxes at Gmail, Yahoo, Outlook, GMX and LaPoste (the last two are used by millions of Germans and French, respectively). All accounts were created specifically for the experiment.

The results, which are available online, show that Microsoft Outlook considers the following as spam:

An internship application from a Nigerian student. The same email with the word “Nigeria” removed was delivered to the inbox.

A description of a sex education program. The same email was delivered to the inbox after removing all instances of “sex” (but leaving just one directed the email to the spam folder).

An excerpt from a speech by Joe Biden on student debt. Removing the words “loan”, “investment” and “billion” from a similar email resulted in its delivery in the inbox.

Spam detectors at other providers did not display the same behavior. Outlook was the only provider where we could identify the words that triggered the spam filter.

Microsoft declined to comment. It is unlikely that an Outlook engineer made an explicit rule to mark any message that contains “Nigeria” as spam. Instead, a machine learning algorithm probably identified “Nigeria” as a strong discriminator between spam and non-spam messages. Microsoft does not make the training data set of its spam filter available to researchers.

SpamAssassin’s creed

SpamAssassin is a spam filter developed by the Apache Software Foundation. It is widely used by organizations that maintain their own email servers. Unlike most commercial offerings, SpamAssassin’s code is open-source and can be reviewed.

While SpamAssassin’s rules change daily, its default configuration files single out words like “Ivory Coast”, “Nigeria” or “Nigerian government” as spammy. The phrase “Oprah!”, an African-American entertainer, is listed as potentially spammy, though the rule is currently inactive.

Rules are changed based on daily checks on training data submitted by users. No effort seems to be made to ensure that user-submitted data does not discriminate unfairly.

User-submitted data is not available, but some of the training data sets are. SpamAssassin published a public corpus of spam and non-spam mail (which the anti-spam community calls ham) which, while over 15 years old, is still widely used. In the spam folder, 59 emails out of 1,397 are from Nigerians. In the ham folder, none are.

The SpamAssassin Project Management Committee did not answer our questions but stated that problems with specific rules were managed by “the community”.

White privilege

SpamAssassin’s leadership is aware of the racism and white privilege embedded in software. In July, it announced that its next release would use “welcomelist” and “blocklist” to replace the racially-charged terms that were used until then.

However, while SpamAssassin says that “[they have] a particular self-interest in attracting contributors from a diversity of cultures”, its Project Management Committee seems to be composed exclusively of white men (some members use pseudonyms and could not be verified with certainty). And at least one of its members routinely signs the emails he posts on the SpamAssassin’s mailing list with anti-feminist quotes from a far-right columnist.

Spam filters are efficient and uncontroversial. Until you look at them.