Incident 94: Court Rules Deliveroo Used 'Discriminatory' Algorithm

Description: In December 2020, an Italian court ruled that Deliveroo’s employee ‘reliability’ algorithm illegally discriminated against workers with legitimate reasons for cancelling shifts.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History
Alleged: Deliveroo developed and deployed an AI system, which harmed Deliveroo workers with legitimate reasons for cancelling shifts and Deliveroo workers.

Incident Stats

Incident ID
94
Report Count
2
Incident Date
2020-11-27
Editors
Sean McGregor, Khoa Lam

CSETv0 Taxonomy Classifications

Taxonomy Details

Full Description

In December 2020, an Italian court ruled that an algorithm used by Deliveroo, a popular app-based bicycle food delivery company, resulted in discriminatory employee management. The case – brought by Italian labor group Confederazione Generale Italiana del Lavoro (CGIL) – centered on the app’s algorithm to assign delivery workers a “reliability rating.” The rating was based in part on how much notice the employees gave before cancelling shifts and was used to determine preference order in future shift scheduling. The Tribunale Ordinario di Bologna decided that because the employees were not allowed to give justification for cancellations, Deliveroo was illegally discrimination against those with legitimate reasons for rescheduling.

Short Description

In December 2020, an Italian court ruled that Deliveroo’s employee ‘reliability’ algorithm illegally discriminated against workers with legitimate reasons for cancelling shifts.

Severity

Moderate

Harm Distribution Basis

Other:Those with emegencies, illness, unexpected circumstances

Harm Type

Harm to civil liberties

AI System Description

Deliveroo's machine learning algorithm used to calculate employees' 'reliability' rating.

System Developer

Deliveroo

Sector of Deployment

Accommodation and food service activities

Relevant AI functions

Cognition, Action

AI Techniques

machine learning

AI Applications

data analytics, optimization, decision making

Location

Italy

Named Entities

Deliveroo, Confederazione Generale Italiana del Lavoro, Tribunale Ordinario di Bologna

Technology Purveyor

Deliveroo

Beginning Date

11/2020

Ending Date

12/2020

Near Miss

Harm caused

Intent

Accident

Lives Lost

No

Laws Implicated

Italian employee fairness law

Data Inputs

employee activity history, shift schedules

Italian court rules against ‘discriminatory’ Deliveroo rider-ranking algorithm
techcrunch.com · 2021

A court in Italy has dealt a blow to unalloyed algorithmic management after a legal challenge brought by three unions. The Bologna court ruled that a reputational-ranking algorithm used by on-demand food delivery platform Deliveroo discrimi…

Court Rules Deliveroo Used 'Discriminatory' Algorithm
vice.com · 2021

An algorithm used by the popular European food delivery app Deliveroo to rank and offer shifts to riders is discriminatory, an Italian court ruled late last week, in what some experts are calling a historic decision for the gig economy. The…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.