Incident 92: Apple Card's Credit Assessment Algorithm Allegedly Discriminated against Women

Description: Apple Card's credit assessment algorithm was reported by Goldman-Sachs customers to have shown gender bias, in which men received significantly higher credit limits than women with equal credit qualifications.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover
Alleged: Apple developed an AI system deployed by Goldman-Sachs, which harmed Apple Card female users and Apple Card female credit applicants.

Incident Stats

Incident ID
92
Report Count
6
Incident Date
2019-11-11
Editors
Sean McGregor, Khoa Lam

CSET Taxonomy Classifications

Taxonomy Details

Full Description

In November 2019, customers of Goldman-Sachs and Apple's Apple Card, the first credit offering by Goldman-Sachs, claimed that there was gender discrimination in the credit assessment algorithm that distributes credit lines, with men receiving significantly higher credit limits than women with equal credit qualifications. Apple co-founder Steve Wozniak confirmed this also happened with him and his wife and the New York Department for Financial Services have launched an investigation regarding the discrimination claim. In response to this incident, Goldman Sachs made a statement that it has not and will never make decisions based on factors like gender, race, age, sexual orientation or any other legally prohibited factors when determining credit worthiness.

Short Description

In November 2019, Apple Card clients claimed that the credit assessment algorithm possesses a gender bias in favor of men.

Severity

Minor

Harm Distribution Basis

Sex

Harm Type

Financial harm

AI System Description

Goldman-Sachs uses a credit assessment algorithm that factors credit score, credit report, and reported income to determine credit lines for clients

System Developer

Goldman-Sachs

Sector of Deployment

Financial and insurance activities

Relevant AI functions

Perception, Cognition, Action

AI Techniques

machine learning, daya analytics

AI Applications

data analytics, recommendation engine, decision support

Location

United States

Named Entities

Goldman Sachs, Apple Card, Apple, Steve Wozniak, New York Department of Financial Services

Technology Purveyor

Apple, Goldman-Sachs

Beginning Date

11/2019

Ending Date

11/2019

Near Miss

Harm caused

Intent

Accident

Lives Lost

No

Laws Implicated

The Equal Credit Opportunity Act (ECOA) prohibits credit discrimination on the basis of race, color, religion, national origin, sex, marital status, age, or because you get public assistance.

Data Inputs

credit score, credit report, reported income

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.