Incident 1: Google’s YouTube Kids App Presents Inappropriate Content

Description: YouTube’s content filtering and recommendation algorithms exposed children to disturbing and inappropriate videos.


New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History
Alleged: YouTube developed and deployed an AI system, which harmed Children.

Incident Stats

Incident ID
Report Count
Incident Date
Sean McGregor

CSETv0 Taxonomy Classifications

Taxonomy Details

Full Description

The content filtering system for YouTube's children's entertainment app, which incorporated algorithmic filters and human reviewers, failed to screen out inappropriate material, exposing an unknown number of children to videos that included sex, drugs, violence, profanity, and conspiracy theories. Many of the videos, which apparently numbered in the thousands, closely resembled popular children's cartoons such as Peppa Pig, but included disturbing or age-inappropriate content. Additional filters provided by YouTube, such as a "restricted mode" filter, failed to block all of these videos, and YouTube's recommendation algorithm recommended them to child viewers, increasing the harm. The problem was reported as early as 2015 and was ongoing through 2018.

Short Description

YouTube’s content filtering and recommendation algorithms exposed children to disturbing and inappropriate videos.



Harm Distribution Basis


Harm Type

Psychological harm

AI System Description

"A content filtering system incorporating machine learning algorithms and human reviewers. The system was meant to screen out videos that were unsuitable for children to view or that violated YouTube's terms of service. These videos were initially collected either algorithmically or on the basis of user reports. A recommendation system that suggested videos to viewers based on their viewing history on the platform."

System Developer


Sector of Deployment

Arts, entertainment and recreation

Relevant AI functions

Perception, Cognition, Action

AI Techniques

machine learning

AI Applications

content filtering, decision support, curation, recommendation engine



Named Entities

Google, YouTube, YouTube Kids

Technology Purveyor

Google, YouTube

Beginning Date


Ending Date


Near Miss




Lives Lost


Data Inputs


GMF Taxonomy Classifications

Taxonomy Details

Known AI Goal

Content Recommendation, Content Search, Hate Speech Detection, NSFW Content Detection

Known AI Technology

Content-based Filtering, Collaborative Filtering

Potential AI Technology

Classification, Ensemble Aggregation, Distributional Learning

Known AI Technical Failure

Tuning Issues, Lack of Adversarial Robustness, Adversarial Data

Potential AI Technical Failure

Concept Drift, Generalization Failure, Misconfigured Aggregation, Distributional Bias, Misaligned Objective

CSETv1 Taxonomy Classifications

Taxonomy Details

Harm Distribution Basis


Sector of Deployment

Arts, entertainment and recreation, information and communication

Google’s YouTube Kids App Criticized for ‘Inappropriate Content’ · 2015

Child and consumer advocacy groups complained to the Federal Trade Commission Tuesday that Google’s new YouTube Kids app contains “inappropriate content,” including explicit sexual language and jokes about pedophilia.

Google launched the ap…

Remove YouTube Kids app until it eliminates its inappropriate content · 2015

Videos filled with profanity, sexually explicit material, alcohol, smoking, and drug references - this is what parents are finding on Google’s YouTube Kids app. That’s right - its kids app. Now, parents across the country are calling on Go…

Disturbing YouTube Kids video shows Mickey Mouse with gun · 2016

Update, Nov. 7, 2017: TODAY Parents is resharing this story from 2016 because a new series of inappropriate videos have cropped up on YouTube Kids and are making headlines. While the channel names and characters used may be different, the p…

The disturbing YouTube videos that are tricking children · 2017

Media playback is unsupported on your device

Thousands of videos on YouTube look like versions of popular cartoons but contain disturbing and inappropriate content not suitable for children.

If you're not paying much attention, it might loo…

YouTube has thousands of disturbing videos targeted at kids: Report · 2017

An off-brand Paw Patrol video called "Babies Pretend to Die Suicide" features several disturbing scenarios. YouTube

Parents who let their kids watch YouTube unattended might want to pay closer attention to what they're viewing.

YouTube is s…

YouTube to crack down on inappropriate content masked as kids’ cartoons · 2017

Recent news stories and blog posts highlighted the underbelly of YouTube Kids, Google's children-friendly version of the wide world of YouTube. While all content on YouTube Kids is meant to be suitable for children under the age of 13, some…

YouTube says it will crack down on bizarre videos targeting children · 2017

Earlier this week, a report in The New York Times and a blog post on Medium drew a lot of attention to a world of strange and sometimes disturbing videos on YouTube aimed at young children. The genre, which we reported on in February of thi…

YouTube Kids has been a problem since 2015 - why did it take this long to address? · 2017

InIn the last few weeks, the world has learned through a number of reports that YouTube is plagued by problems with children’s content. The company has ramped up moderation in recent weeks to fight the wave of inappropriate content, but thi…

YouTube Kids app is STILL showing disturbing videos · 2018

Google-owned YouTube has apologised again after more disturbing videos surfaced on its YouTube Kids app.

Investigators found several unsuitable videos including one of a burning aeroplane from the cartoon Paw Patrol and footage explaining h…

YouTube suggested conspiracy videos to children using its Kids app · 2018

Children were able to watch David Icke's conspiracy videos through YouTube Kids. Flickr/Tyler Merbler

YouTube's app specifically for children is meant to filter out adult content and provide a "world of learning and fun," but Business Insid…

Children's YouTube is still churning out blood, suicide and cannibalism · 2018

Video still of a reproduced version of Minnie Mouse, which appeared on the now-suspended Simple Fun channel Simple Fun / WIRED

YouTube videos using child-oriented search terms are evading the company's attempts to control them. In one carto…

YouTube Kids, Criticized for Content, Introduces New Parental Controls · 2018

YouTube Kids, which has been criticized for inadvertently recommending disturbing videos to children, said Wednesday that it would introduce several ways for parents to limit what can be watched on the popular app.

Beginning this week, pare…

What parents should know about inappropriate content on YouTube · 2018

After multiple reports of inappropriate content on YouTube over the past year, “Good Morning America” wanted to take a closer look at the site. How often do kids end up seeing inappropriate content on the video platform? We talked with a gr…

YouTube Kids Is Nowhere Near as Innocent As It Seems · 2018

YouTube is both a massive industry and browsing staple that people use to fill their educational and entertainment needs. According to Business Insider, the website accumulates around 1.8 billion logged-in users per month. With its high tra…


A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents