Description: The National Eating Disorders Association (NEDA) has shut down its chatbot named Tessa after it gave weight-loss advice to users seeking help for eating disorders. The incident has raised concerns about the risks of using chatbots and AI assistants in healthcare settings, particularly in addressing sensitive issues like eating disorders. NEDA is investigating the matter, emphasizing the need for caution and accuracy when utilizing technology to provide mental health support.
Entities
View all entitiesAlleged: Cass developed an AI system deployed by National Eating Disorders Association and Cass, which harmed People with eating disorders.
Incident Stats
Incident ID
545
Report Count
46
Incident Date
2023-05-29
Editors
Daniel Atherton
Applied Taxonomies
CSETv1 Taxonomy Classifications
Taxonomy DetailsIncident Number
The number of the incident in the AI Incident Database.
545
Special Interest Intangible Harm
An assessment of whether a special interest intangible harm occurred. This assessment does not consider the context of the intangible harm, if an AI was involved, or if there is characterizable class or subgroup of harmed entities. It is also not assessing if an intangible harm occurred. It is only asking if a special interest intangible harm occurred.
yes
CSETv1_Annotator-2 Taxonomy Classifications
Taxonomy DetailsIncident Number
The number of the incident in the AI Incident Database.
545
Special Interest Intangible Harm
An assessment of whether a special interest intangible harm occurred. This assessment does not consider the context of the intangible harm, if an AI was involved, or if there is characterizable class or subgroup of harmed entities. It is also not assessing if an intangible harm occurred. It is only asking if a special interest intangible harm occurred.
yes
Notes (AI special interest intangible harm)
If for 5.5 you select unclear or leave it blank, please provide a brief description of why.
You can also add notes if you want to provide justification for a level.
5.2: Tessa was originally a closed, rules-based system (based on decision trees). Cass AI later implemented an AI component without the knowledge or consent of the co-developers.
Date of Incident Year
The year in which the incident occurred. If there are multiple harms or occurrences of the incident, list the earliest. If a precise date is unavailable, but the available sources provide a basis for estimating the year, estimate. Otherwise, leave blank.
Enter in the format of YYYY
2023