Associated Incidents
Intelligence officials have warned that the 2024 presidential contest could face an unprecedented flood of fake news, fueled by AI, from foreign actors. A Journal analysis of videos on TikTok has found it's already happening.
Amid all the general political news and lighthearted election memes on TikTok, the Journal found thousands of videos with political lies and hyperbole. Further analysis led the Journal to identify 91 accounts that pushed these videos from China, Nigeria, Iran and Vietnam---and were tied together in complex ways.
These viral foreign networks are hijacking TikTok's well-honed engagement machine with false and sometimes incendiary claims, and their intent or who's behind them isn't clear. TikTok says some are looking for profits. Cybersecurity experts say such groups often aim to cause chaos.
Whatever the intent, the divisive narratives corrode the country's already acrimonious political discourse at a time when about a third of young Americans turn to TikTok for news.
Fake stories have thrived online since the earliest days of the internet. Initially most of these relied on misleading or doctored text and photos. Now AI and other automation tools have made it trivially easy to splice together clips and write and voice scripts at little cost.
"Anyone with $5 and a credit card can do this," said Jack Stubbs, chief intelligence officer of research firm Graphika.
People can then spread these lies to huge audiences online with the help of addictive engagement algorithms that pick up users' tastes.
TikTok, and other social-media platforms, are struggling to keep up.
TikTok's rules forbid misinformation about elections that the company considers harmful. TikTok has said that it's hired experts, added policies and built tools to try to thwart these players. TikTok and researchers say that bad actors keep changing tactics to bypass the platform's defenses.
The TikTok experience
The Journal created a dozen automated accounts, or bots, operating across the U.S. to understand what content TikTok is pushing to people ahead of the election.
The bots browsed TikTok's "For You" feed, the highly personalized, never-ending stream of content curated by TikTok's algorithm. The Journal programmed some to dwell on videos with common political hashtags. Others lingered only on videos from mainstream news outlets on any topic.
The bots viewed more than 60,000 videos. Many of them featured general political commentary. TikTok served more than 6,000 videos that mentioned Trump in their description to the bots---more than twice the number of videos that mentioned President Biden. That's true even before he dropped out of the presidential election. Broadly, videos about Trump have gotten more engagement on social-media platforms than videos about Biden.
A strange pattern emerged: Videos with obvious falsehoods about Trump, whose AI voices repeated the same odd turns of phrase, led the Journal to begin to piece together the web of 91 accounts.
TikTok served videos from the 91 accounts to nearly all the Journal's bots, sometimes within hours of being set up.
The accounts were prolific. Over one month, they spammed TikTok users with more than 3,000 new videos, many pushing misinformation about Trump. The posts peaked at 152 videos in one day.
TikTok said it had already removed close to half of the 91 accounts before the Journal reached out to the company over the course of reporting. The platform continued to delete accounts after the Journal flagged them. Even then, the Journal's bots kept getting videos from other accounts nearly identical to the ones that had been removed.
"We continually take action against spam accounts," a spokeswoman for TikTok said. She said TikTok "will continue to improve how we fight deceptive behavior on an ongoing basis."
Causing Chaos
Users wouldn't know it, but the accounts were posting from far-flung countries. The account settings and video content tied some of the accounts to China and Vietnam.
After the Journal flagged the accounts to TikTok, the company launched an internal investigation that confirmed they belonged to operations from the two countries, as well as from Nigeria and Iran.
Among the hundreds of Trump videos in one account was an errant video shot out the window of a high-rise building in Tengzhou, China. This account operated in concert with a handful of others, including one called trump_news_today, which posted nearly identical videos within minutes of each other.
TikTok isn't available in China. The company said it hasn't determined if the China accounts belong to a state-backed influence operation. It said it has removed other networks from China targeting U.S. users.
There's no direct evidence that any of the accounts were the result of state-backed operations, but many countries, including China, have outsourced some of their influence campaigns to private firms, which can both obscure government ties as well as their motives. Chaos is sometimes just the point.
"Information operations are usually designed to erode trust in institutions," said John Hultquist, chief analyst at Google Cloud's Mandiant cybersecurity division. "That's what they do best."
'Old Donnie'
TikTok said its data shows the fake accounts identified by the Journal belonged to multiple, distinct operations. The Journal found they also shared similar kinds of content and behavior, which could suggest coordination or copycats.
The account that posted the assassination video, gloriabarton19684, had a first name, last name and a number for its user handle.
The user names for about three dozen other Journal-identified accounts followed a similar pattern. They were connected in other ways, too...
Many had dozens of clones with the same names and similar numbers, but because the clones didn't post about politics, they fell outside the Journal's analysis.
Two accounts, lidakylie19712 and marcialawson19744, posted obsessively about Trump. But their clones---lidakylie19958 and marcialawson1966, which shared the same avatar---posted only about video games.
Many of the TikTok accounts copied scripts from a handful of YouTube channels, but often used different AI-generated voices.
YouTube removed the accounts before being contacted by the Journal.
Several accounts had links to China. Three posted videos recorded in China. They and two others had their language set to Chinese in the TikTok app. One repeatedly posted the same content within 10 minutes of several of the others.
About a third of the accounts remixed partisan commentary pulled from MeidasTouch, an online media network that often posts content critical of Trump. The accounts posted short clips of the MeidasTouch videos and inserted sensational captions.
Brett Meiselas, a co-founder of MeidasTouch, said his firm wasn't aware the accounts were using its content and wouldn't have authorized it.
Many accounts posted videos using the same unconventional phrasing, such as referring to the former president as "Old Donnie."
While some accounts are connected in multiple ways to the others, TikTok and researchers said spam accounts sometimes steal content from each other and mimic each other's behavior, making it difficult to tell whether the accounts are operated by the same network.
"It's a rat's nest," Stubbs said of the Journal's findings.
Note: WSJ analyzed 91 accounts, 21 of which are not shown in the categories highlighted.
TikTok said some of the accounts pushing fake Trump videos were doing it for profit, and many had links to Vietnam. TikTok said these accounts violated its rules against spam and deceptive behavior.
TikTok said the network from Vietnam includes about 650 accounts, some of which have been exploiting its creator rewards program. That program compensates account holders based on views and other metrics.
If TikTok paid the accounts identified by the Journal 10 cents per thousand views, which is at the lower end of the rate TikTok pays creators, those accounts could have made about $10,000---more than an annual middle-class income in Vietnam.
The videos received more than 2.5 million likes and more than 300,000 comments.
Some of the accounts changed their outlook on Trump in a reversal that suggests a search for profit, researchers said. Clickbait creators sometimes flip-flop when they're chasing views for payouts.
1.00
A clip from a pro-Trump video posted by 'gloria_barton19684'
Before targeting Trump, one of the accounts in January shared a negative video about Biden. All of its later videos were critical of Trump or Republicans. And in May, an account appeared named gloria_barton19684, whose account region was set to Nigeria. Unlike gloriabarton19684, which posted the assassination video, and the other Gloria Barton accounts, this one was proudly pro-Trump.
The account gloria_barton19684 harvested content from a much larger pro-Trump account, AmericaFirst617, which didn't respond to a request for comment.
A Trump spokesman didn't respond for comment.
Several other accounts were registered with usernames and handles supportive of Trump, such as "TrumpWin51." A comment from one read "I love Trump 🥰," before posting negative videos about the former president.
Regardless of the accounts' intent or who's behind them, the flip-flopping and outrageous content plays into the hands of state actors taking aim at the U.S. election, experts said.
TikTok said it removed 24 influence operations from its platform in the first six months of 2024. In the first three months of the year, TikTok said, it removed more than 170 million inauthentic or fake accounts from the platform. TikTok works with fact-checkers to verify certain claims on its platform. TikTok removes content and accounts that break its rules; in murkier cases when accounts post information that can't be verified, TikTok sometimes labels the content and stops recommending the content to users.
As the tech that bad actors use to evade moderation evolves, social-media companies are struggling to keep pace. The technology to detect content created with AI isn't as sophisticated as the tech to create it, and often platforms only discover the fake content when their users flag it. And AI can create problems of scale that are difficult for the platforms to overcome. In May, TikTok announced it would start automatically labeling certain AI-generated content.
The Journal attempted to contact all of the TikTok accounts for comment. One responded.
The account, called BBN Official News, didn't answer questions about its many anti-Trump videos.
Instead, the person operating the account suggested it was for sale.
"If you buy this account, please contact me," the person said in a message.
METHODOLOGY
The Journal's systems have captured tens of thousands of TikTok posts over the past several months. Some of these videos were served to the Journal's bots---automated TikTok accounts spread across the country, including in the swing states of Michigan, Georgia, Pennsylvania and Arizona. The bots were programmed to dwell on videos about news and politics. Other videos were captured when Journal reporters used custom-built software to scrape a TikTok account's entire channel.
Journal algorithms analyzed each video automatically, extracting text and image features and the audio transcription with artificial intelligence. The Journal created a dashboard that enabled reporters to sift through and tag videos of interest. Reporters used AI to find more examples of posts that matched these tags. Reporters reviewed the results of any post that was tagged automatically.