Incident 11: Northpointe Risk Models

Description: An algorithm developed by Northpointe and used in the penal system is two times more likely to incorrectly label a black person as a high-risk re-offender and is two times more likely to incorrectly label a white person as low-risk for reoffense according to a ProPublica review.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover
Alleged: Northpointe developed and deployed an AI system, which harmed Accused People.

Incident Stats

Incident ID
11
Report Count
15
Incident Date
2016-05-23
Editors
Sean McGregor

CSET Taxonomy Classifications

Taxonomy Details

Full Description

An algorithm developed by Northpointe and used in the penal system is shown to be inaccurate and produces racially-skewed results according to a review by ProPublica. The review shows how the 137-question survey given following an arrest is inaccurate and skewed against people of color. While there is not question regarding race in the survey, the algorithm is two times more likely to incorrectly label a black person as a high-risk re-offender (False Positive) and is also two times more likely to incorrectly label a white person as low-risk for reoffense (False Negative) than actual statistics support. Overall, the algorithm is 61% effective at predicting reoffense. This system is used in Broward County, Florida to help judges make decisions surrounding pre-trial release and sentencing post-trial.

Short Description

An algorithm developed by Northpointe and used in the penal system is two times more likely to incorrectly label a black person as a high-risk re-offender and is two times more likely to incorrectly label a white person as low-risk for reoffense according to a ProPublica review.

Severity

Unclear/unknown

Harm Distribution Basis

Race

Harm Type

Harm to civil liberties, Other:Reputational harm; False incarceration

AI System Description

An algorithm, developed by Northpointe designed to assign a risk score associated with a person's likelihood of reoffending after their original arrest.

System Developer

Northpointe

Sector of Deployment

Public administration and defence

Relevant AI functions

Perception, Cognition

AI Techniques

law enforcement algorithm, crime prediction algorithm

AI Applications

risk assesment, crime projection

Location

Broward County, Florida

Named Entities

ProPublica, Northpointe, COMPAS, Broward County, FL

Technology Purveyor

Northpointe

Beginning Date

2016-01-01T00:00:00.000Z

Ending Date

2019-01-01T00:00:00.000Z

Near Miss

Harm caused

Intent

Accident

Lives Lost

No

Infrastructure Sectors

Government facilities

Data Inputs

137-question survey

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents