Incident 1: Google’s YouTube Kids App Presents Inappropriate Content
Entities
View all entitiesIncident Stats
CSET Taxonomy Classifications
Taxonomy DetailsFull Description
The content filtering system for YouTube's children's entertainment app, which incorporated algorithmic filters and human reviewers, failed to screen out inappropriate material, exposing an unknown number of children to videos that included sex, drugs, violence, profanity, and conspiracy theories. Many of the videos, which apparently numbered in the thousands, closely resembled popular children's cartoons such as Peppa Pig, but included disturbing or age-inappropriate content. Additional filters provided by YouTube, such as a "restricted mode" filter, failed to block all of these videos, and YouTube's recommendation algorithm recommended them to child viewers, increasing the harm. The problem was reported as early as 2015 and was ongoing through 2018.
Short Description
YouTube’s content filtering and recommendation algorithms exposed children to disturbing and inappropriate videos.
Severity
Moderate
Harm Distribution Basis
Age
Harm Type
Psychological harm
AI System Description
"A content filtering system incorporating machine learning algorithms and human reviewers. The system was meant to screen out videos that were unsuitable for children to view or that violated YouTube's terms of service. These videos were initially collected either algorithmically or on the basis of user reports. A recommendation system that suggested videos to viewers based on their viewing history on the platform."
System Developer
YouTube
Sector of Deployment
Arts, entertainment and recreation
Relevant AI functions
Perception, Cognition, Action
AI Techniques
machine learning
AI Applications
content filtering, decision support, curation, recommendation engine
Location
Global
Named Entities
Google, YouTube, YouTube Kids
Technology Purveyor
Google, YouTube
Beginning Date
2015-01-01T00:00:00.000Z
Ending Date
2018-12-31T00:00:00.000Z
Near Miss
Unclear/unknown
Intent
Accident
Lives Lost
No
Data Inputs
Videos
GMF Taxonomy Classifications
Taxonomy DetailsKnown AI Goal
Content Recommendation, Content Search, Hate Speech Detection, NSFW Content Detection
Known AI Technology
Content-based Filtering, Collaborative Filtering
Potential AI Technology
Classification, Ensemble Aggregation, Distributional Learning
Known AI Technical Failure
Tuning Issues, Lack of Adversarial Robustness, Adversarial Data
Potential AI Technical Failure
Concept Drift, Generalization Failure, Misconfigured Aggregation, Distributional Bias, Misaligned Objective
Incident Reports
Reports Timeline
Child and consumer advocacy groups complained to the Federal Trade Commission Tuesday that Google’s new YouTube Kids app contains “inappropriate content,” including explicit sexual language and jokes about pedophilia.
Google launched the ap…
Videos filled with profanity, sexually explicit material, alcohol, smoking, and drug references - this is what parents are finding on Google’s YouTube Kids app. That’s right - its kids app. Now, parents across the country are calling on Go…
Update, Nov. 7, 2017: TODAY Parents is resharing this story from 2016 because a new series of inappropriate videos have cropped up on YouTube Kids and are making headlines. While the channel names and characters used may be different, the p…
Media playback is unsupported on your device
Thousands of videos on YouTube look like versions of popular cartoons but contain disturbing and inappropriate content not suitable for children.
If you're not paying much attention, it might loo…
An off-brand Paw Patrol video called "Babies Pretend to Die Suicide" features several disturbing scenarios. YouTube
Parents who let their kids watch YouTube unattended might want to pay closer attention to what they're viewing.
YouTube is s…
Recent news stories and blog posts highlighted the underbelly of YouTube Kids, Google's children-friendly version of the wide world of YouTube. While all content on YouTube Kids is meant to be suitable for children under the age of 13, some…
Earlier this week, a report in The New York Times and a blog post on Medium drew a lot of attention to a world of strange and sometimes disturbing videos on YouTube aimed at young children. The genre, which we reported on in February of thi…
InIn the last few weeks, the world has learned through a number of reports that YouTube is plagued by problems with children’s content. The company has ramped up moderation in recent weeks to fight the wave of inappropriate content, but thi…
Google-owned YouTube has apologised again after more disturbing videos surfaced on its YouTube Kids app.
Investigators found several unsuitable videos including one of a burning aeroplane from the cartoon Paw Patrol and footage explaining h…
Children were able to watch David Icke's conspiracy videos through YouTube Kids. Flickr/Tyler Merbler
YouTube's app specifically for children is meant to filter out adult content and provide a "world of learning and fun," but Business Insid…
Video still of a reproduced version of Minnie Mouse, which appeared on the now-suspended Simple Fun channel Simple Fun / WIRED
YouTube videos using child-oriented search terms are evading the company's attempts to control them. In one carto…
YouTube Kids, which has been criticized for inadvertently recommending disturbing videos to children, said Wednesday that it would introduce several ways for parents to limit what can be watched on the popular app.
Beginning this week, pare…
After multiple reports of inappropriate content on YouTube over the past year, “Good Morning America” wanted to take a closer look at the site. How often do kids end up seeing inappropriate content on the video platform? We talked with a gr…
YouTube is both a massive industry and browsing staple that people use to fill their educational and entertainment needs. According to Business Insider, the website accumulates around 1.8 billion logged-in users per month. With its high tra…
Variants
Similar Incidents
Did our AI mess up? Flag the unrelated incidents
Alexa Plays Pornography Instead of Kids Song
Amazon Censors Gay Books
Amazon Alexa Responding to Environmental Inputs
Similar Incidents
Did our AI mess up? Flag the unrelated incidents
Alexa Plays Pornography Instead of Kids Song
Amazon Censors Gay Books