Incident 41: All Image Captions Produced are Violent
CSETv0 Taxonomy ClassificationsTaxonomy Details
In 2018, MIT Media Lab researchers created an AI-powered "psychopath" text-generating algorithm named Norman. Norman was trained on caption data from a Reddit community that contained graphic images and videos about people dying. Following this training, they then showed Norman and a regular image recognition algorithm trained on the MSCOCO dataset a series of Rorschach inkblots, which psychologists have used to detect disorders. Norman's responses consistently described gruesome scenes, compared to innocent-sounding descriptions from the other algorithm; for example, "a black and white photo of a small bird," vs. "man gets pulled into dough machine." The researchers created Norman to demonstrate the influence training data has on how machine learning algorithms perform in the real world, and how poor data may lead to unreliable and untrustworthy outputs.
MIT Media Lab researchers create AI-powered "psychopath" named Norman by training a model on "dark corners" of Reddit.
AI System Description
"Norman" is a text generating algorithm trained on disturbing content in order to demonstrate how training data can negatively affect an AI model. The comparison model is a regular text generation model.
MIT Media Lab
Sector of Deployment
Information and communication
Relevant AI functions