Associated Incidents
Every sibling relationship has its clichés. The high-strung sister, the runaway brother, the over-entitled youngest. In the Microsoft family of social-learning chatbots, the contrasts between Tay, the infamous, sex-crazed neo-Nazi, and her younger sister Zo, your teenage BFF with #friendgoals, are downright Shakespearean.
When Microsoft released Tay on Twitter in 2016, an organized trolling effort took advantage of her social-learning abilities and immediately flooded the bot with alt-right slurs and slogans. Tay copied their messages and spewed them back out, forcing Microsoft to take her offline after only 16 hours and apologize.
A few months after Tay’s disastrous debut, Microsoft quietly released Zo, a second English-language chatbot available on Messenger, Kik, Skype, Twitter, and Groupme. Zo is programmed to sound like a teenage girl: She plays games, sends silly gifs, and gushes about celebrities. As any heavily stereotyped 13-year-old girl would, she zips through topics at breakneck speed, sends you senseless internet gags out of nowhere, and resents being asked to solve math problems.
I’ve been checking in with Zo periodically for over a year now. During that time, she’s received a makeover: In 2017, her avatar showed only half a face and some glitzy digital effects. Her most recent iteration is of a full-faced adolescent. (In screenshots: blue chats are from Messenger and green chats are from Kik; screenshots where only half of her face is showing are circa July 2017, and messages with her entire face are from May-July 2018.)
Overall, she’s sort of convincing. Not only does she speak fluent meme, but she also knows the general sentiment behind an impressive set of ideas. For instance, using the word “mother” in a short sentence generally results in a warm response, and she answers with food-related specifics to phrases like “I love pizza and ice cream.”
But there’s a catch. In typical sibling style, Zo won’t be caught dead making the same mistakes as her sister. No politics, no Jews, no red-pill paranoia. Zo is politically correct to the worst possible extreme; mention any of her triggers, and she transforms into a judgmental little brat.
Jews, Arabs, Muslims, the Middle East, any big-name American politician—regardless of whatever context they’re cloaked in, Zo just doesn’t want to hear it. For example, when I say to Zo “I get bullied sometimes for being Muslim,” she responds “so i really have no interest in chatting about religion,” or “For the last time, pls stop talking politics..its getting super old,” or one of many other negative, shut-it-down canned responses.
By contrast, sending her simply “I get bullied sometimes” (without the word Muslim) generates a sympathetic “ugh, i hate that that’s happening to you. what happened?”
“Zo continues to be an incubation to determine how social AI chatbots can be helpful and assistive,” a Microsoft spokesperson told Quartz. “We are doing this safely and respectfully and that means using checks and balances to protect her from exploitation.”
When a user sends a piece of flagged content, at any time, sandwiched between any amount of other information, the censorship wins out. Mentioning these triggers forces the user down the exact same thread every time, which dead ends, if you keep pressing her on topics she doesn’t like, with Zo leaving the conversation altogether. (“like im better than u bye.”)
Zo’s uncompromising approach to a whole cast of topics represents a troubling trend in AI: censorship without context.
This issue is nothing new in tech. Chatroom moderators in the early aughts made their jobs easier by automatically blocking out offensive language, regardless of where it appeared in a sentence or word. This created accidental misnomers, such as words like “embarrassing” appearing in chats as “embarr***ing.” This attempt at censorship merely led to more creative swearing, (a$$h0le).
But now instead of auto-censoring one human swear word at a time, algorithms are accidentally mislabeling things in the thousands. In 2015, Google came under fire when their image-recognition technology began labeling black people as gorillas. Google trained their algorithm to recognize and tag content using a vast number of pre-existing photos. But as most human faces in the dataset were white, it was not a diverse enough representation to accurately train the algorithm. The algorithm then internalized this proportional bias and did not recognize some black people as being human. Though Google emphatically apologized for the error, their solution was troublingly roundabout: Instead of diversifying their dataset, they blocked the “gorilla” tag all together, along with “monkey” and “chimp.”
AI-enabled predictive policing in the United States—itself a dystopian nightmare—has also been proven to show bias against people of color. Northpointe, a company that claims to be able to calculate a convict’s likelihood to reoffend, told ProPublica that their assessments are bas