Skip to Content
logologo
AI Incident Database
Open TwitterOpen RSS FeedOpen FacebookOpen LinkedInOpen GitHub
Open Menu
発見する
投稿する
  • ようこそAIIDへ
  • インシデントを発見
  • 空間ビュー
  • テーブル表示
  • リスト表示
  • 組織
  • 分類法
  • インシデントレポートを投稿
  • 投稿ランキング
  • ブログ
  • AIニュースダイジェスト
  • リスクチェックリスト
  • おまかせ表示
  • サインアップ
閉じる
発見する
投稿する
  • ようこそAIIDへ
  • インシデントを発見
  • 空間ビュー
  • テーブル表示
  • リスト表示
  • 組織
  • 分類法
  • インシデントレポートを投稿
  • 投稿ランキング
  • ブログ
  • AIニュースダイジェスト
  • リスクチェックリスト
  • おまかせ表示
  • サインアップ
閉じる

インシデント 1: Google’s YouTube Kids App Presents Inappropriate Content

概要: YouTube’s content filtering and recommendation algorithms exposed children to disturbing and inappropriate videos.

ツール

新しいレポート新しいレポート新しいレスポンス新しいレスポンス発見する発見する履歴を表示履歴を表示

組織

すべての組織を表示
推定: YouTubeが開発し提供したAIシステムで、Childrenに影響を与えた

インシデントのステータス

インシデントID
1
レポート数
14
インシデント発生日
2015-05-19
エディタ
Sean McGregor
Applied Taxonomies
CSETv0, GMF, CSETv1, MIT

CSETv1 分類法のクラス

分類法の詳細

Incident Number

The number of the incident in the AI Incident Database.
 

1

Special Interest Intangible Harm

An assessment of whether a special interest intangible harm occurred. This assessment does not consider the context of the intangible harm, if an AI was involved, or if there is characterizable class or subgroup of harmed entities. It is also not assessing if an intangible harm occurred. It is only asking if a special interest intangible harm occurred.
 

yes

Date of Incident Year

The year in which the incident occurred. If there are multiple harms or occurrences of the incident, list the earliest. If a precise date is unavailable, but the available sources provide a basis for estimating the year, estimate. Otherwise, leave blank. Enter in the format of YYYY
 

2016

Estimated Date

“Yes” if the data was estimated. “No” otherwise.
 

Yes

Multiple AI Interaction

“Yes” if two or more independently operating AI systems were involved. “No” otherwise.
 

no

Embedded

“Yes” if the AI is embedded in a physical system. “No” if it is not. “Maybe” if it is unclear.
 

no

CSETv0 分類法のクラス

分類法の詳細

Problem Nature

Indicates which, if any, of the following types of AI failure describe the incident: "Specification," i.e. the system's behavior did not align with the true intentions of its designer, operator, etc; "Robustness," i.e. the system operated unsafely because of features or changes in its environment, or in the inputs the system received; "Assurance," i.e. the system could not be adequately monitored or controlled during operation.
 

Unknown/unclear

Physical System

Where relevant, indicates whether the AI system(s) was embedded into or tightly associated with specific types of hardware.
 

Software only

Nature of End User

"Expert" if users with special training or technical expertise were the ones meant to benefit from the AI system(s)’ operation; "Amateur" if the AI systems were primarily meant to benefit the general public or untrained users.
 

Amateur

Public Sector Deployment

"Yes" if the AI system(s) involved in the accident were being used by the public sector or for the administration of public goods (for example, public transportation). "No" if the system(s) were being used in the private sector or for commercial purposes (for example, a ride-sharing company), on the other.
 

No

Data Inputs

A brief description of the data that the AI system(s) used or were trained on.
 

Videos

Lives Lost

Were human lives lost as a result of the incident?
 

No

GMF 分類法のクラス

分類法の詳細

Known AI Goal Snippets

One or more snippets that justify the classification.
 

(Snippet Text: An off-brand Paw Patrol video called "Babies Pretend to Die Suicide" features several disturbing scenarios. The YouTube Kids app filters out most - but not all - of the disturbing videos.

Before any video appears in the YouTube Kids app, it's filtered by algorithms that are supposed to identify appropriate children's content YouTube also has a team of human moderators that review any videos flagged in the main YouTube app by volunteer Contributors (users who flag inappropriate content) or by systems that identify recognizable children's characters in the questionable video. Many of those views came from YouTube's "up next" and "recommended" video section that appears while watching any video. YouTube's algorithms attempt to find videos that you may want to watch based on the video you chose to watch first If you don't pick another video to watch after the current video ends, the "up next" video will automatically play. , Related Classifications: Content Recommendation, Content Search)

MIT 分類法のクラス

Machine-Classified
分類法の詳細

Risk Subdomain

A further 23 subdomains create an accessible and understandable classification of hazards and harms associated with AI
 

1.2. Exposure to toxic content

Risk Domain

The Domain Taxonomy of AI Risks classifies risks into seven AI risk domains: (1) Discrimination & toxicity, (2) Privacy & security, (3) Misinformation, (4) Malicious actors & misuse, (5) Human-computer interaction, (6) Socioeconomic & environmental harms, and (7) AI system safety, failures & limitations.
 
  1. Discrimination and Toxicity

Entity

Which, if any, entity is presented as the main cause of the risk
 

AI

Timing

The stage in the AI lifecycle at which the risk is presented as occurring
 

Post-deployment

Intent

Whether the risk is presented as occurring as an expected or unexpected outcome from pursuing a goal
 

Unintentional

インシデントレポート

レポートタイムライン

+1
Google’s YouTube Kids App Criticized for ‘Inappropriate Content’
Remove YouTube Kids app until it eliminates its inappropriate contentDisturbing YouTube Kids video shows Mickey Mouse with gunThe disturbing YouTube videos that are tricking children+3
YouTube has thousands of disturbing videos targeted at kids: Report
YouTube Kids app is STILL showing disturbing videos+2
YouTube suggested conspiracy videos to children using its Kids app
What parents should know about inappropriate content on YouTubeYouTube Kids Is Nowhere Near as Innocent As It Seems
Google’s YouTube Kids App Criticized for ‘Inappropriate Content’

Google’s YouTube Kids App Criticized for ‘Inappropriate Content’

blogs.wsj.com

Remove YouTube Kids app until it eliminates its inappropriate content

Remove YouTube Kids app until it eliminates its inappropriate content

change.org

Disturbing YouTube Kids video shows Mickey Mouse with gun

Disturbing YouTube Kids video shows Mickey Mouse with gun

today.com

The disturbing YouTube videos that are tricking children

The disturbing YouTube videos that are tricking children

bbc.com

YouTube has thousands of disturbing videos targeted at kids: Report

YouTube has thousands of disturbing videos targeted at kids: Report

businessinsider.com

YouTube says it will crack down on bizarre videos targeting children

YouTube says it will crack down on bizarre videos targeting children

theverge.com

YouTube to crack down on inappropriate content masked as kids’ cartoons

YouTube to crack down on inappropriate content masked as kids’ cartoons

arstechnica.com

YouTube Kids has been a problem since 2015 - why did it take this long to address?

YouTube Kids has been a problem since 2015 - why did it take this long to address?

polygon.com

YouTube Kids app is STILL showing disturbing videos

YouTube Kids app is STILL showing disturbing videos

dailymail.co.uk

YouTube suggested conspiracy videos to children using its Kids app

YouTube suggested conspiracy videos to children using its Kids app

businessinsider.com

Children's YouTube is still churning out blood, suicide and cannibalism

Children's YouTube is still churning out blood, suicide and cannibalism

wired.co.uk

YouTube Kids, Criticized for Content, Introduces New Parental Controls

YouTube Kids, Criticized for Content, Introduces New Parental Controls

nytimes.com

What parents should know about inappropriate content on YouTube

What parents should know about inappropriate content on YouTube

goodmorningamerica.com

YouTube Kids Is Nowhere Near as Innocent As It Seems

YouTube Kids Is Nowhere Near as Innocent As It Seems

studybreaks.com

Google’s YouTube Kids App Criticized for ‘Inappropriate Content’
blogs.wsj.com · 2015

Child and consumer advocacy groups complained to the Federal Trade Commission Tuesday that Google’s new YouTube Kids app contains “inappropriate content,” including explicit sexual language and jokes about pedophilia.

Google launched the ap…

Remove YouTube Kids app until it eliminates its inappropriate content
change.org · 2015

Videos filled with profanity, sexually explicit material, alcohol, smoking, and drug references - this is what parents are finding on Google’s YouTube Kids app. That’s right - its kids app. Now, parents across the country are calling on Go…

Disturbing YouTube Kids video shows Mickey Mouse with gun
today.com · 2016

Update, Nov. 7, 2017: TODAY Parents is resharing this story from 2016 because a new series of inappropriate videos have cropped up on YouTube Kids and are making headlines. While the channel names and characters used may be different, the p…

The disturbing YouTube videos that are tricking children
bbc.com · 2017

Media playback is unsupported on your device

Thousands of videos on YouTube look like versions of popular cartoons but contain disturbing and inappropriate content not suitable for children.

If you're not paying much attention, it might loo…

YouTube has thousands of disturbing videos targeted at kids: Report
businessinsider.com · 2017

An off-brand Paw Patrol video called "Babies Pretend to Die Suicide" features several disturbing scenarios. YouTube

Parents who let their kids watch YouTube unattended might want to pay closer attention to what they're viewing.

YouTube is s…

YouTube says it will crack down on bizarre videos targeting children
theverge.com · 2017

Earlier this week, a report in The New York Times and a blog post on Medium drew a lot of attention to a world of strange and sometimes disturbing videos on YouTube aimed at young children. The genre, which we reported on in February of thi…

YouTube to crack down on inappropriate content masked as kids’ cartoons
arstechnica.com · 2017

Recent news stories and blog posts highlighted the underbelly of YouTube Kids, Google's children-friendly version of the wide world of YouTube. While all content on YouTube Kids is meant to be suitable for children under the age of 13, some…

YouTube Kids has been a problem since 2015 - why did it take this long to address?
polygon.com · 2017

InIn the last few weeks, the world has learned through a number of reports that YouTube is plagued by problems with children’s content. The company has ramped up moderation in recent weeks to fight the wave of inappropriate content, but thi…

YouTube Kids app is STILL showing disturbing videos
dailymail.co.uk · 2018

Google-owned YouTube has apologised again after more disturbing videos surfaced on its YouTube Kids app.

Investigators found several unsuitable videos including one of a burning aeroplane from the cartoon Paw Patrol and footage explaining h…

YouTube suggested conspiracy videos to children using its Kids app
businessinsider.com · 2018

Children were able to watch David Icke's conspiracy videos through YouTube Kids. Flickr/Tyler Merbler

YouTube's app specifically for children is meant to filter out adult content and provide a "world of learning and fun," but Business Insid…

Children's YouTube is still churning out blood, suicide and cannibalism
wired.co.uk · 2018

Video still of a reproduced version of Minnie Mouse, which appeared on the now-suspended Simple Fun channel Simple Fun / WIRED

YouTube videos using child-oriented search terms are evading the company's attempts to control them. In one carto…

YouTube Kids, Criticized for Content, Introduces New Parental Controls
nytimes.com · 2018

YouTube Kids, which has been criticized for inadvertently recommending disturbing videos to children, said Wednesday that it would introduce several ways for parents to limit what can be watched on the popular app.

Beginning this week, pare…

What parents should know about inappropriate content on YouTube
goodmorningamerica.com · 2018

After multiple reports of inappropriate content on YouTube over the past year, “Good Morning America” wanted to take a closer look at the site. How often do kids end up seeing inappropriate content on the video platform? We talked with a gr…

YouTube Kids Is Nowhere Near as Innocent As It Seems
studybreaks.com · 2018

YouTube is both a massive industry and browsing staple that people use to fill their educational and entertainment needs. According to Business Insider, the website accumulates around 1.8 billion logged-in users per month. With its high tra…

バリアント

「バリアント」は既存のAIインシデントと同じ原因要素を共有し、同様な被害を引き起こし、同じ知的システムを含んだインシデントです。バリアントは完全に独立したインシデントとしてインデックスするのではなく、データベースに最初に投稿された同様なインシデントの元にインシデントのバリエーションとして一覧します。インシデントデータベースの他の投稿タイプとは違い、バリアントではインシデントデータベース以外の根拠のレポートは要求されません。詳細についてはこの研究論文を参照してください

よく似たインシデント

テキスト類似度による

Did our AI mess up? Flag the unrelated incidents

Alexa Plays Pornography Instead of Kids Song

Whoops, Alexa Plays Porn Instead of a Kids Song!

Dec 2016 · 16 レポート
Amazon Censors Gay Books

Amazon Censors Gay Books

May 2008 · 24 レポート
Amazon Alexa Responding to Environmental Inputs

Top 5 AI Failures From 2017 Which Prove That ‘Perfect AI’ Is Still A Dream

Dec 2015 · 35 レポート
次のインシデント

よく似たインシデント

テキスト類似度による

Did our AI mess up? Flag the unrelated incidents

Alexa Plays Pornography Instead of Kids Song

Whoops, Alexa Plays Porn Instead of a Kids Song!

Dec 2016 · 16 レポート
Amazon Censors Gay Books

Amazon Censors Gay Books

May 2008 · 24 レポート
Amazon Alexa Responding to Environmental Inputs

Top 5 AI Failures From 2017 Which Prove That ‘Perfect AI’ Is Still A Dream

Dec 2015 · 35 レポート

リサーチ

  • “AIインシデント”の定義
  • “AIインシデントレスポンス”の定義
  • データベースのロードマップ
  • 関連研究
  • 全データベースのダウンロード

プロジェクトとコミュニティ

  • AIIDについて
  • コンタクトとフォロー
  • アプリと要約
  • エディタのためのガイド

インシデント

  • 全インシデントの一覧
  • フラグの立ったインシデント
  • 登録待ち一覧
  • クラスごとの表示
  • 分類法

2024 - AI Incident Database

  • 利用規約
  • プライバシーポリシー
  • Open twitterOpen githubOpen rssOpen facebookOpen linkedin
  • 86fe0f5