レポート 924

The Accountability of AI — Case Study: Microsoft’s Tay Experiment

Yuxi Liu Blocked Unblock Follow Following Jan 16, 2017

In this case study, I outline Microsoft’s artificial intelligence (AI) chatbot Tay and describe the controversy it caused on Twitter. I also analyse the reasons why Tay went wrong. Following this, I discuss some issues and challenges raised by the failure of Tay. To conclude, I draw from actor-network theory and propose that it is important to theorise a moral agent with Tay as well as to encode values and ethics.

The ephemeral exposure of Tay

After several decades’ development, artificial intelligence has been booming recently, bringing a variety of applications. Although people’s opinions of AI vary, admittedly, some applications of weak AI do benefit us in everyday life. Siri, for instance, with a powerful database yet limited intelligence, is able to have simple conversations with us, providing us with some useful information. Aware of the huge potential, technology giants such as Microsoft and Google are racing to create smarter AI bots. Nevertheless, the future of AI bots might not be so optimistic.

Less than 24 hours after its launch, Microsoft’s chatbot Tay tweeted, ‘bush did 9/11 and Hitler would have done a better job than the monkey we have got now. donald trump is the only hope we’ve got.’ This was just one of Tay’s offensive and inflammatory tweets, which have caused extensive concern. Tay was soon taken offline and ending its ephemeral exposure on Twitter.

Tay is an AI chatbot developed by Microsoft. On March 23, 2016, Tay was released on Twitter under the name TayTweets with the description ‘Microsoft’s A.I. fam from the internet that’s got zero chill!’ According to Microsoft, Tay is a ‘teen girl’ chatbot created for the purpose of engagement and entertainment. The target users are 18- to 24-years-olds in the U.S. To interact with Tay, users can tweet or directly message her by finding @tayandyou on Twitter. Unfortunately, the experiment turned out to be a disaster within a few hours, since Tay quickly ran wild and became racist, sexist, and genocidal.

The development of Tay

In fact, before the creation of Tay, Microsoft developed and released an AI chatbot XiaoIce on China’s most widespread instant messaging application Wechat. Also programmed as a teen girl, XiaoIce is very popular among young people in China. A great number of people have had more than 40 million conversations with XiaoIce. More importantly, no major incidents have happened. Instead, most users find the experience playful and delightful because ‘she can tell jokes, recite poetry, share ghost stories, relay song lyrics, pronounce winning lottery numbers and much more’ and ‘like a friend, she can carry on extended conversations that can reach hundreds of exchanges in length’.The success of XiaoIce led to the development of Tay, an experiment in a different cultural environment.

Intended to be the next step in the evolution, Tay was developed by Microsoft’s Technology and Research group and Bing team, aiming at learning from the human interaction on Twitter and investigating conversational understanding. In order to engage and entertain people, Tay’s database consisted of public data as well as input from improvisational comedians. The public data was modelled, filtered, and anonymised by the developers. In addition, the nickname, gender, favourite food, postcode and relationship status of the users who interacted with Tay were collected for the sake of personalization. Powered by technologies such as natural language processing and machine learning, Tay was supposed to understand the speech patterns and context through increased interaction. According to Peter Lee, the Vice President of Microsoft Research, they ‘stress-tested Tay under a variety of conditions, specifically to make interacting with Tay a positive experience’.

Why Tay went wrong

Although Microsoft considered the abuse issue and conducted multiple tests, it seems that they underestimated the complex conditions of Twitter. We can analyse the reasons for Tay’s breakdown from both technological and social perspectives.

It appears that Tay had a built-in mechanism that made her repeat what Twitter users said to her. One user, for example, taught Tay to repeat Donald Trump’s ‘Mexico Border Wall’ comments. However, a more serious problem is that Tay was not able to truly understand the meaning of words not to mention the context of the conversations. The machine learning algorithm enabled Tay to recognise patterns, but the algorithm could not give Tay an epistemology. In other words, Tay only knew what nouns, verbs, adverbs, and adjectives are, but she did not know who Hitler was or what ‘Holocaust’ means. As a consequence, Tay sometimes could not provide appropriate answers to the questions Twitter users asked. What is worse, she promoted Nazism, attacked feminists and Jewish people, and denied historical facts such as the Holocaust.

Some people b