Incident 213: Facebook’s Political Ad Detection Reportedly Showed High and Geographically Uneven Error Rates

Description: The performance of Facebook’s political ad detection was revealed by researchers to be imprecise, uneven across countries in errors, and inadequate for preventing systematic violations of political advertising policies.
Alleged: Facebook developed and deployed an AI system, which harmed Facebook users.

Suggested citation format

Lam, Khoa. (2020-07-01) Incident Number 213. in Lam, K. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
213
Report Count
5
Incident Date
2020-07-01
Editors
Khoa Lam

Tools

New ReportNew ReportDiscoverDiscover

Incidents Reports

Major technology companies strive to protect the integrity of political advertising on their platforms by implementing and enforcing self-regulatory policies that impose transparency requirements on political ads. In this paper, we quantify whether Facebook’s current enforcement correctly identifies political ads and ensures compliance by advertisers. In a comprehensive, large-scale analysis of 4.2 million political and 29.6 million non-political ads from 215,030 advertisers, we identify ads correctly detected as political (true positives), ads incorrectly detected (false positives), and ads missed by detection (false negatives). Facebook’s current enforcement appears imprecise: 61% more ads are missed than are detected worldwide, and 55% of U.S. detected ads are in fact non-political. Detection performance is uneven across countries, with some having up to 53 times higher false negative rates among clearly political pages than in the U.S. Moreover, enforcement appears inadequate for preventing systematic violations of political advertising policies: for example, advertisers were able to continue running political ads without disclosing them while they were temporarily prohibited in the U.S. We attribute these flaws to five gaps in Facebook’s current enforcement and transparency implementation, and close with recommendations to improve the security of the online political ad ecosystem.

An Audit of Facebook’s Political Ad Policy Enforcement

In the first known study to quantify the performance of Facebook’s political ad policy enforcement at a large and representative scale, researchers found that when making decisions on how to classify undeclared ads, Facebook often missed political ads while falsely labeling others as political. The political ads that Facebook misses also disappear from its public archive, putting them out of reach for public scrutiny.

Researchers with imec-DistriNet at KU Leuven in Belgium and New York University’s Cybersecurity for Democracy conducted a comprehensive audit of Facebook’s political ad detection and policy enforcement. The researchers examined 33.8 million Facebook ads that ran between July 2020 and February 2021—a timeframe that included elections in both the U.S. and Brazil. This is the first known study to quantify the performance of Facebook’s political ad policy enforcement system at a large and representative scale. The researchers’ review found that:

Globally, Facebook made the wrong decision for 83 percent of those ads that had not been declared as political by their advertisers and that Facebook or the researchers deemed political. Facebook both overcounted and undercounted political ads in this group (Figure 1).

Facebook missed a higher proportion of political ads outside the U.S. The platform had the worst record in Malaysia, where it missed as much as 45 percent of ads from obviously political pages or advertisers. In Macedonia, Argentina, Turkey, Portugal, France, and Serbia, Facebook missed up to one out of four ads from such pages, which were sponsored by candidates or parties. In the U.S., 55 percent of detected political ads were actually overcounts, meaning they did not meet Facebook’s definition for political ads.

Facebook allowed more than 70,000 political ads to run during its moratorium on political ads around the U.S. 2020 elections. These ads were sponsored by pages that ran political ads before the election and continued to run them after the moratorium. In other words, these advertisers stopped declaring these ads as political.

Facebook’s transparency via its Ad Library API was not sufficient to do this analysis. To capture publicly available ads that were not included in the API, the researchers created an alternate pipeline to gather ads from the web portal. Mandated universal ad transparency would alleviate the need for use of alternate methods (Figure 2).

Background

Facebook imposes increased authenticity and transparency requirements for “ads about social issues, elections or politics” in around 120 countries, by requiring advertisers to confirm their identity and location and “declare” who funded the ads, which then appears as a “disclaimer” for the ad. In this analysis, the researchers use the shorthand “political” to refer to this class of ads.

Because not all advertisers self-declare political ads, Facebook conducts additional reviews to ensure all political ads are captured. If they find undeclared political ads, they take these ads down – that is, they remove them from the site. In addition, the platform sometimes creates temporary requirements, such as the moratorium on political ads around the U.S. 2020 elections.

Political Ad Detection in The United States

While Facebook’s detection of political ads in the U.S. performed the best worldwide, at least 10,940 undisclosed electoral ads were never discovered by Facebook. This included ads from a U.S. senator and former presidential primary candidate who spent almost $10 million dollars on ads, running them during the ad ban after the U.S. elections.

In addition, 1,018 advertisers that ran exclusively political ads before the U.S. election ran 71,426 undeclared ads afterwards, despite the platform’s political ad moratorium. This ban applied to new political ads during the week prior to Nov. 3, 2020 and to all political ads between Nov. 4 and March 4, 2021.

The analysis showed that some 55 percent of political ads were overcounts—ads that the platform labeled as political, but did not meet Facebook’s definition. Among these overcounted ads, which were therefore taken down in error, were e-commerce ads and ads with public health messages, including some related to COVID-19.

Worldwide Ad Detection

Worldwide, Facebook misidentified 83 percent of the undeclared ads that Facebook or the researchers deemed political (Figure 1).

62 percent of the ads, or 116,963 ads, were missed by Facebook— that is, they were political but not labeled as such.

Facebook overcounted 40,191 additional ads, or 21 percent, as political when they did not meet its own criteria for political ads.

Facebook had the worst success rate when detecting political ads from Malaysian advertisers, where it missed 45 percent of political ads. Macedonia and Argentina were not far behind, with 37 and 34 percent of political ads missed. Facebook had the best success rate when detecting political ads from advertisers based in the U.S. or New Zealand. (See Tables 1 and 2 for ranking by country.)

Facebook pages that ran obviously political ads resulted in significant exposure worldwide—some 4 billion impressions, or views, before they were caught, if ever. Advertisers spent an estimated $12.2–20.7 million on ads that violated Facebook’s policy but were later detected, and an additional estimated $4.6–9.2 million on violating ads that went undetected.

Impacts of Poor Ad Detection

The study identifies a number of risks that arise from Facebook’s failure to accurately detect and flag political ads:

When Facebook fails to identify advertisers who do not properly declare their ads as political, those who are malicious can spread disinformation, prohibited content or engage in coordinated inauthentic behavior with impunity.

Users who see ads without a political disclaimer may not be aware that their intent is to influence them. They may then be shown more of the same kind of content.

Poor detection also disadvantages well-meaning advertisers, who may be confused about policies or whose ads may be taken down in error.

Missed ads disappear from the Facebook Ad Library after they run, making them inaccessible to researchers.

Data Sources and Methodology

Researchers who study political ads on Facebook use data that the company provides via its Ad Library API, Ad Library Report, and Ad Library web portal. However, this data is insufficient for conducting an analysis like this (Figure 2). For example:

The Ad Library API is missing ads that Facebook has failed to detect as political, and includes ads labeled as political that do not meet the criteria.

The Ad Library Report features inaccurate aggregate spending data because it overcounts some ads and is missing others.

Facebook removes inactive ads from the Ad Library web portal, meaning that if it fails to label an ad as political, and that ad campaign is over, it is no longer available.

The research team thus created a novel large-scale data collection pipeline to archive data from the Ad Library web portal, which lists all active ads for a given advertiser, regardless of whether they are political. The resulting data set is likely the most comprehensive archive of Facebook political ads for the time period, which includes the U.S. and Brazilian elections, since it also contains the ads that Facebook missed.

Recommendations for Better Enforcement of Political Ad Policies

To address the inconsistencies in its ad detection systems, the research team recommends that Facebook:

Closely monitor pages run by candidates, parties, and other obviously political entities, and require all their ads to be declared as political.

Impose appropriate consequences when advertisers violate policies, including taking down pages or preventing them from publishing ads.

Invest in improving performance of ad detection systems globally, including engaging with local governments, regulators and organizations and ensuring systems adapt to local contexts.

Clarify which ads they consider political and clear up the ambiguity associated with “issue ads.”

Given both the risks and Facebook’s reticence to make changes that affect its bottom line, outside actors may need to intervene. Legislators could harmonize definitions of both political and issue ads across platforms to remove ambiguity. They could also set enforcement and transparency requirements that would be overseen by regulators and come with penalties if the requirements are not met. Finally, the team joins calls for mandated universal ad transparency, which would provide independent researchers the data necessary to conduct audits such as this one and allow the public to scrutinize and better understand the political ad ecosystem.

Summary of findings: An audit of Facebook’s political ad policy enforcement

A new study shows that the vast majority of the time Facebook has made an enforcement decision on a political ad after it ran, it’s made the wrong call.

Political advertisers on Facebook are supposed to identify themselves as such. That way, Facebook can log their ads in an archive and, as was the company's policy in 2020, even prevent those ads from running close to an election.

Of course, both accidentally and intentionally, not every political advertiser plays by Facebook’s rules, meaning Facebook often has to decide what is and isn’t a political ad even after the ad runs. Now, a new study by researchers at New York University and KU Leuven in Belgium suggests that the vast majority of the time Facebook has to make these decisions, it makes the wrong call.

The researchers analyzed 189,000 political ads from around the world that Facebook needed to make an enforcement decision on between July 2020 and February 2021. They found that Facebook misidentified a whopping 83% of those ads.

According to the study, 117,000 of those ads clearly fell under Facebook’s definition of a political ad, but went undetected by Facebook. Another 40,000 ads were flagged as political, but the researchers say they clearly were not. For example, some ads run by Sen. Elizabeth Warren and Alaska Rep. Don Young were not marked as political, while ads for a “silky peanut butter pie” recipe and a Ford pickup truck listing were.

The researchers also found Facebook’s enforcement varied wildly depending on where the ads appeared. In the U.S, Facebook overcorrected and ended up mislabeling more ads that weren’t actually political. In Malaysia, the opposite was true, with Facebook letting some 45% of these political ads go unlabeled. That disparity mirrors the company’s struggles to moderate other aspects of its platform in non-English speaking parts of the world.

But perhaps the most glaring issue the researchers found was related to Facebook’s much-debated decision to ban political ads leading up to and immediately following the U.S. election in 2020. According to the study, more than 1,000 advertisers who had run nothing but political ads prior to the ban were able to continue running ads — more than 70,000 of them — during the ban. This, the researchers argue, suggests that some political advertisers stopped self-reporting their ads to evade Facebook’s ban — and Facebook failed to stop them.

These failures, the researchers argue, suggest that Facebook needs a new approach to enforcing this policy that is consistent across borders and includes penalties for advertisers who don’t self-report. Right now, there are some penalties for advertisers that skirt these rules, but the researchers argue they’re not adequately enforced.

“Facebook isn’t a very good cop,” said Laura Edelson, one of the study’s authors and a computer science Ph.D. candidate at NYU. “We’re very fortunate that the vast majority of political advertisers are voluntarily complying with Facebook’s policy here, because when we actually study how good Facebook is at enforcing their own policy, when they have to do the enforcing, as you can see, their accuracy rate is pretty low.”

Protocol presented Meta with a summary of findings from the researchers, none of which the company directly contested. In a statement, a Meta spokesperson said, "The vast majority of political ads they studied were disclosed and labeled just as they should be. In fact, their findings suggest there were potential issues with less than 5 percent of all political ads."

The spokesperson said that Meta offers "more transparency into political advertising than TV, radio or any other digital advertising platform," and said that even overtly political figures need to disclose whether each individual ad is political or not.

If Edelson’s name sounds familiar, it’s because she is part of the NYU team that Facebook cut off from its platform earlier this year. Edelson and her fellow researchers had been scraping ads and ad-targeting data from users who installed their browser plug-in, a method Facebook said violated its terms of service.

But the NYU team had already been working on this project, and it relied not on the browser extension but on Facebook’s own ad library. That library contains an archive of all of the ads Facebook has identified as political, along with a repository of all the other ads running on the platform at any given moment. Because those non-political ads disappear from the library after they’ve stopped running, they’re difficult to study over time. So the NYU researchers built a tool that could scrape the library every 24 hours and store all of that information before ads disappeared. That way, the researchers could see which political ads Facebook had failed to identify.

The researchers scraped the ads' content and metadata for 14 days after they ran so they could observe how quickly Facebook detected political ads that hadn’t been labeled — and if it detected them at all. When Facebook detects unlabeled political ads, it takes those ads down.

Edelson and her fellow researchers then compared this larger trove of all Facebook ads to the smaller political ad archive. To find political ads Facebook missed — which the researchers referred to as “false negatives” — they looked exclusively at pages that are overtly political because they belong to, for instance, a registered politician or political group or self-identified as a political organization on their Facebook pages. They then looked at all the ads those pages ran to see which ones appeared in the political ad archive. The ads that didn’t were considered false negatives.

Edelson argues that means the error rate she and her colleagues found is a “floor” because there may have been lots of other missed political ads that didn’t come from one of these pages. “This is a very conservative estimate,” she said.

This is something that Facebook — and Meta as a whole — will have to grapple with as the U.S. midterm election approaches. The company is simultaneously struggling to adapt to new privacy protections put in place by Apple that make it harder to measure the effectiveness of ads that campaigns use for fundraising and voter turnout efforts. And it recently made changes that prevent advertisers from targeting users based on “sensitive” topics, including their politics, health, religion and more. This, too, will create challenges for political advertisers that have used Facebook to target key demographics for years. All of it ensures that the way political groups and campaigns have used Facebook for campaigning in the past is about to dramatically change.

All that said, the researchers found that more than 4 million political ads that ran during the study period were self-reported by advertisers. That makes the problem they’re describing — the 189,000-plus ads that slipped through the cracks — a small slice of the overall political advertising picture on Facebook. And Facebook has said that in 2020 it rejected 3.3 million ads targeting the U.S. before they ever ran for failing to complete the authorization process for political advertisers.

But the political ads Facebook is missing — and the regular ads it’s removing for no good reason — still matter, Edelson said. “Ads are theoretically where Facebook has the potential to do the best. They have information about who pays for the content. They literally have a profit motive to get this right, and this is what the accuracy rate is."

How political advertisers get away with skirting Facebook’s rules

PARIS (AFP) - Facebook misidentified tens of thousands of advertisements flagged under its political ads policy, according to a study released Thursday (Dec 9), which warned that the failure could lead to political manipulation.

Researchers at Belgium's KU Leuven university and New York University examined 33.8 million Facebook ads that ran on the social media site between July 2020 and February 2021.

"This is the first known study to quantify the performance of Facebook's political ad policy enforcement system at a large and representative scale," the team said in a summary of their findings.

Facebook imposes stricter conditions on paid advertisements that concern "social issues, elections or politics", including posts that promote particular candidates.

Ads labelled as political appear on the site with a disclaimer that explains who paid for them. Ads that are found to be political, when they were not declared as such, are taken down.

But the researchers found that in 189,000 cases when Facebook reviewed an ad to check whether or not it should be treated as political, it was wrong 83 per cent of the time.

These included 117,000 cases when Facebook's detection system failed to flag up ads that should have been treated as political, and 40,000 ads that were mistakenly flagged as political when they were not.

The researchers noted that Facebook's enforcement of the policy relies heavily on detecting keywords in ads under an automated system, although staff also play a role in moderating the content.

The period studied included elections in two large Facebook markets, the United States and Brazil, and the researchers warned that mislabelling the ads created opportunities for manipulation.

"When Facebook fails to identify advertisers who do not properly declare their ads as political, those who are malicious can spread disinformation," they warned.

"Users who see ads without a political disclaimer may not be aware that their intent is to influence them," they added.

The team noted that the US social media giant missed a higher proportion of undeclared political ads outside the US.

"The platform had the worst record in Malaysia, where it missed as much as 45 per cent of ads from obviously political pages or advertisers," the research summary said.

"In Macedonia, Argentina, Turkey, Portugal, France and Serbia, Facebook missed up to one out of four ads from such pages, which were sponsored by candidates or parties." The study follows criticism of Facebook over a recent whistleblower scandal, including accusations that the site's ability to tamp down hate speech and misinformation has been seriously lacking outside the West.

Within the US, Facebook introduced a moratorium on political ads around the 2020 presidential election, following deep controversy over the platform's role leading up to the 2016 election of Mr Donald Trump.

Facebook nonetheless allowed more than 70,000 political ads to run during the 2020 moratorium, the researchers found.

AFP has reached out to Facebook to request a comment on the study.

Facebook misidentified thousands of political ads: Study

Researchers found thousands of cases where advertisers skirted the company's rules without ever being flagged.

In the years since the Cambridge Analytica scandal revealed how easily Facebook’s political ads could be weaponized by bad actors abroad, the social network has made some major overhauls that, in a perfect world, would keep a scandal like that from happening again. It rolled out an ever-growing library of the ads political parties were running on its platform, put up a new authorization process for anyone looking to run political ads in the U.S., and even let users opt out of seeing some of those ads if they chose.

These tools largely rely on advertisers being honest about disclosing their political ties, though—and those folks aren’t known for being honest. Facebook knows this, which is why it also uses a (mostly automated) review process to sniff out political ads that might be trying to skirt these rules.

Now a joint study between researchers at New York University and KU Leuven in Belgium suggests that that this problem might be worse than we thought. The team investigated 189,000 political ads that ran on Facebook between July 2020 and February of this year, and found that Facebook had missed the mark a whopping 83%(!) of the time: 21% were regular ads for products that the platform had misidentified as being political, and the remaining 62% were outright political ads that Facebook just... missed.

If you’re familiar with Facebook’s less-than-stellar reputation when it comes to moderating content written in non-English languages, you won’t be surprised to hear that the lion’s share of these misses came from overseas.

In Malaysia, for example, the researchers found nearly half of the political ads they collected—45%—weren’t labeled, despite being run by blatantly political advertisers or pages. Pages from Macedonia, Argentina, and Turkey, meanwhile, had about a quarter of their political ads fly under Facebook’s radar. The U.S., meanwhile, had something of the opposite problem: about 55% of the political ads that Facebook had flagged weren’t political at all; the examples compiled on the researchers’ site include ads for products like a Ford pickup and a baby cradle being branded as “political.”

Of course, there are some major misses in the U.S., too. The tracking both of these teams were doing coincided with Facebook’s temporary ban on political ads following last year’s presidential election. The researchers found more than 70,000 political ads running wild on the platform during this period, by pages that had openly run nothing but political ads before the ban began—the only thing that changed is that they weren’t labeled anymore. And Facebook just didn’t notice.

We’ve reached out to Facebook about the study and will update here when we hear back.

Facebook's Political Ad Promises Mostly Miss the Mark, Study Shows