Skip to Content
logologo
AI Incident Database
Open TwitterOpen RSS FeedOpen FacebookOpen LinkedInOpen GitHub
Open Menu
Discover
Submit
  • Welcome to the AIID
  • Discover Incidents
  • Spatial View
  • Table View
  • List view
  • Entities
  • Taxonomies
  • Submit Incident Reports
  • Submission Leaderboard
  • Blog
  • AI News Digest
  • Risk Checklists
  • Random Incident
  • Sign Up
Collapse
Discover
Submit
  • Welcome to the AIID
  • Discover Incidents
  • Spatial View
  • Table View
  • List view
  • Entities
  • Taxonomies
  • Submit Incident Reports
  • Submission Leaderboard
  • Blog
  • AI News Digest
  • Risk Checklists
  • Random Incident
  • Sign Up
Collapse

Report 11

Associated Incidents

Incident 113 Report
Google’s YouTube Kids App Presents Inappropriate Content

Loading...
Children's YouTube is still churning out blood, suicide and cannibalism
wired.co.uk · 2018

Video still of a reproduced version of Minnie Mouse, which appeared on the now-suspended Simple Fun channel Simple Fun / WIRED

YouTube videos using child-oriented search terms are evading the company's attempts to control them. In one cartoon, a woman with a Minnie Mouse head tumbles down an escalator before becoming trapped in its machinery, spurting blood, while her children (baby Mickey and Minnie characters) cry.

The cartoon, Minnie Mouse Mommy Has Pregnancy Problem & Doctor Treats Episodes! Mickey Mouse, Donald Duck Cartoon, racked up over three million views in a single day. It could be viewed even with YouTube's family-friendly restricted mode enabled and existed, along with plenty of similarly distressing content, on Simple Fun, a channel that had been in operation since July 2017.

Advertisement

The channel has now been removed by YouTube "due to multiple or severe violations of YouTube's policy against spam, deceptive practices and misleading content or other Terms of Service violations."

WIRED found videos containing violence against child characters, age-inappropriate sexualisation, Paw Patrol characters attempting suicide and Peppa Pig being tricked into eating bacon. These were discovered by following recommendations in YouTube's sidebar or simply allowing children's videos to autoplay, starting with legitimate content.

Read next Inside the prophetic, angry mind of Black Mirror's Charlie Brooker Inside the prophetic, angry mind of Black Mirror's Charlie Brooker

"Recommendations are designed to optimize watch time, there is no reason that it shows content that is actually good for kids. It might sometimes, but if it does it is coincidence," says former YouTube engineer Guillaume Chaslot, who founded AlgoTransparency, a project that aims to highlight and explain the impact of algorithms in determining what we see online. "Working at YouTube on recommendations, I felt I was the bad guy in Pinocchio: showing kids a colourful and fun world, but actually turning them into donkeys to maximise revenue."

The videos WIRED found were reported to YouTube and were removed or restricted by the Google-owned company before the publication of this article. The company explained that it is increasing its efforts to control content that violates its terms and conditions.

Advertisement

Weird children's YouTube

If you can imagine it, there's a deeply weird Finger Family version of it Munnik TV / WIRED

YouTube is home to millions of hours of children's entertainment – part of the 400 hours of video uploaded to the service every minute – ranging from CBeebies and Disney to the incomprehensibly successful Little Baby Bum, a UK-based YouTube-native children's channel devoted to 3D animated songs and nursery rhymes for pre-schoolers in numerous languages.

Content for pre-school children, in particular, can be lucrative for ad-funded channels, as small children will readily watch and poke at whatever videos YouTube suggests, while harried parents are often unable to fully supervise every minute of their child's media consumption.

Read next The strange story of Section 230, the obscure law that created our flawed, broken internet The strange story of Section 230, the obscure law that created our flawed, broken internet

AlgoTransparency regularly indexes the kids' videos most likely to be recommended by YouTube. Its lists show that YouTube's most-suggested children's videos lean disproportionately towards a combination of YouTube-native songs and nursery rhymes designed for a US audience; long edited-together compilations of TV series such as Peppa Pig, and strange, low-budget 2D and 3D animated mash-ups of animals, characters and voice samples.

Advertisement

Previously described in James Bridle's Something is wrong on the internet Medium post as "decidedly off", the latter type of content can be loosely and collectively categorised as 'weird children's YouTube'.

Titles are typically a word salad designed to attract children's and parents' searches, while the videos' content leans heavily on generic 2D or 3D animated models, usually incongruously combined with familiar figures from hit Disney or superhero franchises. The sheer number of them on the platform is staggering, and many have millions or even hundreds of millions of views.

The video Disney Frozen Finger Family Collection Disney Frozen Finger Family Songs Nursery Rhymes has 43m views, while LEARN COLOR BMX & MotorCycles JUMP! for kids w/ Superheroes Cartoon for children Nursery rhymes has 176m.

Neither video contains any content more distressing than badly-animated video and the intensely annoying Finger Family song, but both are good examples of videos that use popular franchises and the promise of education to target searches that parents and children are likely to carry out.

Read next With AI and DNA, Massive Attack are hacking a new kind of music With AI and DNA, Massive Attack are hacking a new kind of music

Based on what YouTube insiders have said about how

Read the Source

Research

  • Defining an “AI Incident”
  • Defining an “AI Incident Response”
  • Database Roadmap
  • Related Work
  • Download Complete Database

Project and Community

  • About
  • Contact and Follow
  • Apps and Summaries
  • Editor’s Guide

Incidents

  • All Incidents in List Form
  • Flagged Incidents
  • Submission Queue
  • Classifications View
  • Taxonomies

2024 - AI Incident Database

  • Terms of use
  • Privacy Policy
  • Open twitterOpen githubOpen rssOpen facebookOpen linkedin
  • e1b50cd