Incident 41: All Image Captions Produced are Violent

Description: MIT Media Lab researchers create AI-powered "psychopath" named Norman by training a model on "dark corners" of Reddit.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History
Alleged: MIT Media Lab developed and deployed an AI system, which harmed unknown.

Incident Stats

Incident ID
41
Report Count
28
Incident Date
2018-04-02
Editors
Sean McGregor

CSETv0 Taxonomy Classifications

Taxonomy Details

Full Description

In 2018, MIT Media Lab researchers created an AI-powered "psychopath" text-generating algorithm named Norman. Norman was trained on caption data from a Reddit community that contained graphic images and videos about people dying. Following this training, they then showed Norman and a regular image recognition algorithm trained on the MSCOCO dataset a series of Rorschach inkblots, which psychologists have used to detect disorders. Norman's responses consistently described gruesome scenes, compared to innocent-sounding descriptions from the other algorithm; for example, "a black and white photo of a small bird," vs. "man gets pulled into dough machine." The researchers created Norman to demonstrate the influence training data has on how machine learning algorithms perform in the real world, and how poor data may lead to unreliable and untrustworthy outputs.

Short Description

MIT Media Lab researchers create AI-powered "psychopath" named Norman by training a model on "dark corners" of Reddit.

Severity

Negligible

Harm Type

Psychological harm

AI System Description

"Norman" is a text generating algorithm trained on disturbing content in order to demonstrate how training data can negatively affect an AI model. The comparison model is a regular text generation model.

System Developer

MIT Media Lab

Sector of Deployment

Information and communication

Relevant AI functions

Perception, Cognition, Action

AI Techniques

Machine learning

AI Applications

Text generation

Location

Cambridge, MA

Named Entities

MIT Media Lab, Reddit, Norman, Massachusetts Intstitute of Technology

Technology Purveyor

MIT Media Lab

Beginning Date

2018-04-01

Ending Date

2018-06-01

Near Miss

Unclear/unknown

Intent

Unclear

Lives Lost

No

Data Inputs

Violent content from Reddit for the Norman algorithm, MSCOCO dataset for the control algorithm.

CSETv1 Taxonomy Classifications

Taxonomy Details

Harm Distribution Basis

none

Sector of Deployment

professional, scientific and technical activities

Meet 'Norman,' the Darkest, Most Disturbed AI the World Has Ever Seen
livescience.com · 2018

A neural network named "Norman" is disturbingly different from other types of artificial intelligence (AI).

Housed at MIT Media Lab, a research laboratory that investigates AI and machine learning, Norman's computer brain was allegedly warp…

Are you scared yet? Meet Norman, the psychopathic AI
bbc.co.uk · 2018

Image copyright MIT Image caption Norman was named after Alfred Hitchcock's Norman Bates from his classic horror film Psycho

Norman is an algorithm trained to understand pictures but, like its namesake Hitchcock's Norman Bates, it does not …

MIT Creates An AI Psychopath Because Someone Had To Eventually
geek.com · 2018

In one of the big musical numbers from The Life Of Brian, Eric Idle reminds us to “always look on the bright side of life.” Norman, a new artificial intelligence project from MIT, doesn’t know how to do that.

That’s because Norman is a psyc…

Norman the psychopathic AI offers a warning on biased data
internetofbusiness.com · 2018

Many people are concerned about the potential rise of malignant AI, with UK newspapers, in particular, worried about the ‘Terminator’ scenario of machines that are hostile to humanity.

Researchers at MIT have decided to explore this concept…

MIT Scientists Create Norman, The World's First "Psychopathic" AI
iflscience.com · 2018

A team of scientists at the Massachusetts Institute of Technology (MIT) have built a psychopathic AI using image captions pulled from Reddit. Oh, and they’ve named it Norman after Alfred Hitchcock’s Norman Bates. This is how our very own Te…

Norman Bates-a ‘psychopath’ AI created by MIT-was trained by studying reddit posts of “gruesome deaths.”
reddit.com · 2018

I appreciate what MIT has done here. However, the work described in the article is useless, misleading, or both. All the training methods I've encountered in the literature do not maintain any kind of measure of the mutual information betwe…

MIT Creates AI-Powered Psychopath Called 'Norman'
extremetech.com · 2018

Artificial intelligence researchers have thus far attempted to make well-rounded algorithms that can be helpful to humanity. However, a team from MIT has undertaken a project to do the exact opposite. Researchers from the MIT Media Lab have…

MIT Trains Psychopath Robot 'Norman' Using Only Gruesome Reddit Images
newsweek.com · 2018

Scientists at the Massachusetts Institute of Technology (MIT) trained an artificial intelligence algorithm dubbed "Norman" to become a psychopath by only exposing it to macabre Reddit images of gruesome deaths and violence, according to a n…

MIT Creates A Psychopathic AI Named Norman, Because Why Not
analyticsindiamag.com · 2018

The Massachusets Institute of Technology’s website for Norman, world’s first psychopathic AI is oddly cheerful and optimistic. A creepy combination of Norman Bates (from the 1960 Alfred Hitchcock Movie Psycho) and a robot stares at you and …

Meet Norman, the world's first 'psychopathic' AI
zdnet.com · 2018

Norman: MIT's artificial intelligence with psychopathic traits The experiment is based on the 1921 Rorschach test, which identifies psychopathic traits based on users' perception of inkblots.

Researchers at the Massachusetts Institute of Te…

MIT researchers use Reddit to create the first "psychopath AI"
cnet.com · 2018

MIT Media Lab

Meet Norman.

He's not your everyday AI. His algorithms won't help filter through your Facebook feed or recommend you new songs to listen to on Spotify.

Nope -- Norman is a "psychopath AI", created by researchers at the MIT Med…

'Psychopath AI' Offers A Cautionary Tale for Technologists
blogs.discovermagazine.com · 2018

Researchers at MIT have created a psychopath. They call him Norman. He's a computer. Actually, that's not really right. Though the team calls Norman a psychopath (and the chilling lead graphic on their homepage certainly backs that up), wha…

MIT's Psycho Norman AI Proves Reddit Can Make You Insane
hothardware.com · 2018

MIT has created a new AI that we sincerely hope never escapes into the wild because this IS how you get Skynet. This AI is called Norman and it is a psychopath. How did Norman turn into a psycho? All the image data MIT fed Norman came from …

MIT Scientists Create 'Psychopath' AI Named Norman
fortune.com · 2018

MIT scientists’ newest artificial intelligence algorithm endeavor birthed a “psychopath” by the name of Norman.

Scientists, Pinar Yanardag, Manuel Cebrian, and Iyad Rahwan, exposed AI Norman, named after Anthony Perkins’ character in Alfred…

MIT fed an AI data from Reddit, and now it only thinks about murder
theverge.com · 2018

For some, the phrase “artificial intelligence” conjures nightmare visions — something out of the ’04 Will Smith flick I, Robot, perhaps, or the ending of Ex Machina — like a boot smashing through the glass of a computer screen to stamp on a…

MIT scientists created an AI-powered 'psychopath' named Norman
money.cnn.com · 2018

Norman always sees the worst in things.

That's because Norman is a "psychopath" powered by artificial intelligence and developed by the MIT Media Lab.

Norman is an algorithm meant to show how the data behind AI matters deeply.

MIT researche…

MIT scientists created a “psychopath” AI by feeding it violent content from Reddit
news.avclub.com · 2018

Photo: Matt Cardy (Getty Images)

We’ve all seen evil machines in The Terminator or The Matrix, but how does a machine become evil? Is it like Project Satan from Futurama, where scientists combined parts from various evil cars to create the …

MIT Scientists Unveil First Psychopath AI, ‘Norman’
rollingstone.com · 2018

Scientists at the Massachusetts Institute of Technology unveiled the first artificial intelligence algorithm trained to be a psychopath. The AI was fittingly dubbed “Norman” after Norman Bates, the notorious killer in Alfred Hitchcock’s Psy…

MIT Trained an A.I. With Reddit, and it Became a Psychopath
digitaltrends.com · 2018

Share

Science fiction has given us many iconic malevolent A.I. characters. However, these are often figures like Terminator’s T-800 or Alien’s Ash who commit emotionless murder to pursue an end goal. Those which exhibit more unhinged parano…

We Asked MIT Researchers Why They Made a ‘Psychotic AI' That Only Sees Death
motherboard.vice.com · 2018

MIT researchers created an artificial intelligence that they call “psychopathic” in order to show the biases that are inherent in AI research. When asked to look at Rorschach test blots, “Norman” always sees death. Here’s what Norman saw, c…

Meet Norman, the 'psychopath AI' that's here to teach us a lesson
independent.co.uk · 2018

The development of artificial intelligence, Stephen Hawking once warned, will be “either the best or the worst thing ever to happen to humanity”. A new AI algorithm exposed to the most macabre corners of the internet demonstrates how we cou…

Scientists have created a murder-obsessed 'psychopath' AI called Norman -- and it learned everything it knows from Reddit
businessinsider.com.au · 2018

Researchers at MIT have programmed an AI using exclusively violent and gruesome content from Reddit.

They called it “Norman.”

As a result, Norman only sees death in everything.

This isn’t the first time an AI has been turned dark by the int…

Scientists Made an Psychopathic AI Based on Reddit
popularmechanics.com · 2018

Scientists at MIT have created an AI psychopath trained on images from a particularly disturbing thread on Reddit. Norman is designed to illustrate that the data used for machine learning can significantly impact its outcome. “Norman suffer…

Meet Norman, the world’s first psychopath AI
livemint.com · 2018

Artificial Intelligence (AI) is solving many problems for humans. But, as Google CEO Sundar Pichai said in the company’s manifesto for AI, such a powerful technology “raises equally powerful questions about its use". Google (Alphabet Inc.) …

'Norman,' when artificial intelligence goes psycho
phys.org · 2018

Researchers at MIT have created 'Norman', the first psychopathic artificial intelligence to explain how algorithms are made, and to make people aware of AI's potential dangers

No, it's not a new horror film. It's Norman: also known as the f…

Meet Norman - the world's first 'psychopathic artificial intelligence' unveiled by MIT
news.sky.com · 2018

Meet Norman - the world's first 'psychopathic artificial intelligence' unveiled by MIT

Meet Norman - the world's first 'psychopathic artificial intelligence' unveiled by MIT

It shares his name with the knife-wielding killer in Alfred Hitchc…

"Norman" Is MIT's New Psychopathic AI
curiosity.com · 2018

This experiment with Norman shows that it doesn't matter how rigorous the actual algorithm is if it's overexposed to harmful content. Algorithms like pymetrics actually do have a beneficial effect, but it's vital that it's looked over by a …

This Is Why You Should Know About Norman the AI Psychopath
inc.com · 2018

You can learn a lot from a psychopath, and you don't even have to binge-watch Netflix's "Mindhunter" to do it. You just need to study Norman, the artificial intelligence psychopath aptly named for the pivotal figure in Alfred Hitchcock's "P…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents

TayBot

TayBot

· 28 reports