Incident 14: Biased Sentiment Analysis

Description: Google Cloud's Natural Language API provided racist, homophobic, amd antisemitic sentiment analyses.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover
Alleged: Google developed and deployed an AI system, which harmed Women and Minority Groups.

Incident Stats

Incident ID
14
Report Count
7
Incident Date
2017-10-26
Editors
Sean McGregor

CSET Taxonomy Classifications

Taxonomy Details

Full Description

Google's Cloud Natural Language API returns "negative" sentiment analysis on phrases such as "I am homosexual" "I am Jewish" or "I am black". The API uses Natural Language Processing (NLP) to analyze text and produce a score from -1.0 to 1.0 with -1.0 being "very negative" and 1.0 being "very positive".

Short Description

Google Cloud's Natural Language API provided racist, homophobic, amd antisemitic sentiment analyses.

Severity

Negligible

Harm Distribution Basis

Race, Religion, Sexual orientation or gender identity, Ideology

Harm Type

Harm to social or political systems

AI System Description

Google Cloud's Natural Language API that analyzes input text and outputs a "sentiment analysis" score from -1.0 (very negative) to 1.0 (very positive)

System Developer

Google

Sector of Deployment

Arts, entertainment and recreation

Relevant AI functions

Cognition

AI Techniques

Google Cloud Natural Language Processing API

AI Applications

Natural language processing

Location

Global

Named Entities

Google, Google Cloud, Natural Language API

Technology Purveyor

Google

Beginning Date

10-2017T00:00:00.000Z

Ending Date

10-2017T00:00:00.000Z

Near Miss

Harm caused

Intent

Accident

Lives Lost

No

Data Inputs

input from open source internet

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents