Incident 44: Machine Personal Assistants Failed to Maintain Social Norms

Description: During an experiment of software personal assistants at the Information Sciences Institute (ISI) at the University of Southern California (USC), researchers found that the assistants violated the privacy of their principals and were unable to respect the social norms of the office.
Alleged: USC Information Sciences Institute developed and deployed an AI system, which harmed USC Information Sciences Institute.

Suggested citation format

Yampolskiy, Roman. (2008-07-01) Incident Number 44. in McGregor, S. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
44
Report Count
1
Incident Date
2008-07-01
Editors
Sean McGregor

Tools

New ReportNew ReportDiscoverDiscover

CSET Taxonomy Classifications

Taxonomy Details

Full Description

From June to December 2000, researchers at the Information Sciences Institute (ISI) at the University of Southern California (USC) deployed a team of 12 software agents in their office to act as administrative assistants facilitating routine office operations. The agents, also known as Electronic Elves and nicknamed ‘Friday’, were designed to assist their human principal in scheduling meetings, facilitating informal meetings, auctioning group tasks, and ordering meals as a means to testing agent teamwork dynamics. During the experiment, the agents violated the privacy and social norms of the office by publishing their principal’s location information and revealing an employee value hierarchy used to deconflict meetings.

Short Description

During an experiment of software personal assistants at the Information Sciences Institute (ISI) at the University of Southern California (USC), researchers found that the assistants violated the privacy of their principals and were unable to respect the social norms of the office.

Severity

Negligible

Harm Type

Psychological harm

AI System Description

A team of software agents (agent organization) designed to facilate routine office operations, such as scheduling, informal collaboration, and meeting shared goals.

System Developer

University of Southern California Information Sciences Institute

Sector of Deployment

Administrative and support service activities

Relevant AI functions

Perception, Cognition

AI Techniques

Machine learning

AI Applications

decision support, resource optimization, personalization

Location

Marina del Rey, California

Named Entities

University of Southern California Information Sciences Institute

Technology Purveyor

University of Southern California Information Sciences Institute

Beginning Date

6/2000

Ending Date

12/2000

Near Miss

Near miss

Intent

Accident

Lives Lost

No

Data Inputs

Schedule data, cellphone GPS data

Incidents Reports

Abstract: Software personal assistants continue to be a topic of significant research interest. This article outlines some of the important lessons learned from a successfully deployed team of personal assistant agents (Electric Elves) in an office environment. In the Electric Elves project, a team of almost a dozen personal assistant agents were continually active for seven months. Each elf (agent) represented one person and assisted in daily activities in an actual office environment. This project led to several important observations about privacy, adjustable autonomy, and social norms in office environments. In addition to outlining some of the key lessons learned we outline our continued research to address some of the concerns raised.

The topic of software personal assistants, particularly for office environments, is of continued and growing research interest (Scerri, Pynadath, and Tambe 2002; Maheswaran et al. 2004; Modi and Veloso 2005; Pynadath and Tambe 2003). The goal is to provide software agent assistants for individuals in an office as well as software agents that represent shared office resources. The resulting set of agents coordinate as a team to facilitate routine office activities. This article outlines some key lessons learned during the successful deployment of a team of a dozen agents, called Electric Elves (EElves), which ran continually from June 2000 to December 2000 at the Information Sciences Institute (ISI) at the University of Southern California (USC) (Scerri, Pynadath, and Tambe 2002; Chalupsky et al. 2002; Pynadath and Tambe 2003, 2001; Pynadath et al. 2000). Each elf (agent) acted as an assistant to one person and aided in the daily activities of an actual office environment. Originally, the E-Elves project was designed to focus on team coordination among software agents. However, while team coordination remained an interesting challenge, several other unanticipated research issues came to the fore. Among these new issues were adjustable autonomy (agents dynamically adjusting their own level of autonomy), as well as privacy and social norms in office environments. Several earlier publications outline the primary technical contributions of E-Elves and research inspired by E-Elves in detail. However, the goal of this article is to highlight some of what went wrong in the E-Elves project and provide a broad overview of technical advances in the areas of concern without providing specific technical details.

Electric Elves: What Went Wrong and Why