AI Incident Roundup – December 2024 and January 2025

Our review newsletter recapping new incident IDs in the AI Incident Database and looking at the trends.

🗄 Trending in the AIID

Around November 2024, much of the initial backlog of AI incident reports had been processed, so between December 2024 and into January 2025 we were able to more readily keep up with incidents as they were being reported. However, we were also able to track incidents that had longer histories and find some that we had missed. We're always on the lookout for recorded incidents that have yet to make it into our database, and remain vigilant as we continue expanding our sourcing for new and emerging harm events. Below is an explanation of just some of the incidents we have been able to study.

Deepfakes and AI-generated disinformation continues to erode public trust worldwide, despite some reports playing down their influence. For example, some studies have reported that AI models are disseminating misinformation in Spanish (Incident 859). In September, deepfake audio purportedly fabricated a confession from former President Biden about a political crisis in Pakistan (Incident 856), while AI-generated videos falsely depicted Martin Luther King Jr. supporting Donald Trump's candidacy in early November (Incident 862). More broadly, though, the Russian Center for Geopolitical Expertise reportedly used AI to target U.S. candidates with disinformation during the 2024 election (Incident 884), and a network of 171 AI-powered bots spread political disinformation ahead of Ghana’s general election (Incident 866). We are also tracking the Matryoshka disinformation campaign, which is reportedly deepfaking academics in order to spread anti-Ukraine, pro-Kremlin views (Incident 909).

The malicious use of AI for personal harm is also on the rise. In Pennsylvania, students allegedly used AI to generate obscene images of classmates (Incident 887), while in Sydney, a high schooler reportedly created deepfakes of fellow students (Incident 888). AI-generated nonconsensual intimate imagery has also reportedly targeted at least one in six U.S. congresswomen (Incident 874), and deepfake pornography was used to harass Northern Ireland MLA Cara Hunter ahead of the May 2022 Assembly Election (Incident 903), which may have potentially had an effect on the election results as well. This incident is one we are continuing to add reports to as Hunter continues her advocacy against deepfake abuse.

We have also been tracking a significant uptick in reported incidents of AI being used by scammers. In particular, they are reportedly exploiting deepfake technology to impersonate public figures in romance scams, including Brad Pitt (Incident 901) and Leonor, Princess of Asturias (Incident 905). In Hallandale Beach, Florida, deepfakes were reportedly used in an attempted real estate fraud (Incident 858), and generative AI facilitated a $255,000 real estate fraud scheme (Incident 864). A reported AI voice-cloning of journalist Ari Melber was exploited in a scam targeting an elderly woman (Incident 872), and an AI-assisted ransomware campaign by FunkSec targeted over 80 victims (Incident 897). Deepfake videos also defrauded Canadian immigrants out of thousands of dollars (Incident 876), while in Scotland an elderly woman was targeted in a scam in which she was reportedly defrauded of £17,000 (Incident 878). In one particularly brazen state-level incident, a scammer reportedly used deepfake technology to impersonate a head of state of an ASEAN nation in order to attempt to defraud Thai Prime Minister Paetongtarn Shinawatra (Incident 902). We are also actively tracking reporting on the rising deployment of AI by the Yahoo Boys (Incidents 911, 912, and 913), who have been linked to some other incidents in our database (such as Incident 901).

AI-driven misinformation and system failures continue to affect public safety and trust. A navigation app allegedly directed a car over a damaged bridge in Uttar Pradesh, resulting in three deaths (Incident 857). Apple Intelligence reportedly notified users of false events, including the supposed arrest of Benjamin Netanyahu (Incident 861), while also prioritizing scam messages (Incident 892). In Portland, the SERVUS algorithm misallocated utility bill discounts to at least one high-wealth consumer (Incident 868). YouTube algorithms are also allegedly amplifying eating disorder content to adolescent girls (Incident 873), and TikTok algorithms were linked to minors' exposure to harmful content (Incident 869). The legal and ethical implications of AI remain a pressing concern, with law enforcement's misuse of facial recognition technology reportedly continuing to lead to wrongful arrests and violations of investigative standards (Incident 896). AI-powered call center breaches exposed over 10 million conversations in the Middle East (Incident 906), and Character.AI chatbots were allegedly emulating school shooters and targeting minors with predatory content (Incidents 899 and 900).

These incidents are, of course, just a fraction of a much larger whole, and we will continue to track and monitor reports as they arise. Below you will find the full list of new incident ID additions between December 2024 and January 2025.

🗞️ New Incident IDs in the Database

• Incident 856 - Deepfake Audio Purportedly Fabricates Biden’s Admission of Role in Pakistani Political Crisis (2024-09-15)
• Incident 857 - Navigation App Allegedly Directs Car Over Damaged Bridge in Uttar Pradesh, Resulting in Three Deaths (2024-11-25)
• Incident 858 - Deepfake Reportedly Used in Attempted Real Estate Fraud in Hallandale Beach, Florida (2024-09-19)
• Incident 859 - AI Models Reportedly Found to Provide Misinformation on Election Processes in Spanish (2024-10-30)
• Incident 860 - AI Camera Allegedly Misidentifies Dutch Motorist as Using Mobile Phone, Issuing €380 Fine (2023-10-31)
• Incident 861 - Apple Intelligence Reportedly Notified Users That Luigi Mangione Shot Himself and Benjamin Netanyahu Had Been Arrested (2024-12-13)
• Incident 862 - AI-Generated Video Falsely Depicts Martin Luther King Jr. Supporting Donald Trump (2024-11-03)
• Incident 863 - Character.ai Companion Allegedly Prompts Self-Harm and Violence in Texas Teen (2024-12-12)
• Incident 864 - Generative AI Allegedly Used to Facilitate $255,000 Real Estate Fraud Scheme (2024-08-23)
• Incident 865 - Fake AI 'Nudify' Sites Reportedly Linked to Malware Distribution by Russian Hacker Collective FIN7 (2024-10-02)
• Incident 866 - Network of 171 AI-Powered Bots Reportedly Spread Political Disinformation Ahead of Ghana’s December 2024 General Election (2024-02-01)
• Incident 867 - AI-Generated Airline Reviews Allegedly Mislead Consumers and Undermine Trust (2024-10-31)
• Incident 868 - Portland Water Bureau SERVUS Algorithm Reportedly Allocates Utility Bill Discount to High-Wealth Consumer (2024-10-14)
• Incident 869 - TikTok Algorithms Allegedly Linked to Minors' Exposure to Harmful Content (2024-11-04)
• Incident 870 - Meeten Malware Campaign Reportedly Undermines Web3 Security Using AI-Legitimized Branding (2024-12-06)
• Incident 871 - Reported Deepfake Video of Elon Musk Announcing $20 Million Cryptocurrency Giveaway Circulating on Social Media (2024-12-13)
• Incident 872 - AI Voice Cloning of Ari Melber Allegedly Exploited in Scam Targeting Elderly Woman (2024-12-16)
• Incident 873 - YouTube Algorithms Allegedly Amplify Eating Disorder Content to Adolescent Girls (2024-12-10)
• Incident 874 - 1 in 6 Congresswomen Have Reportedly Been Targeted by AI-Generated Nonconsensual Intimate Imagery (2024-12-11)
• Incident 875 - Coordinated Deepfake Campaign Reportedly Impersonating Rishi Sunak Promoted Fraudulent Quantum AI Investment Platform on Meta (2024-01-08)
• Incident 876 - Deepfake Videos Allegedly Used to Defraud Canadian Immigrants Out of Thousands of Dollars (2024-11-24)
• Incident 877 - HTML/Nomani Deepfake Phishing Campaigns Allegedly Use AI-Generated Content to Defraud Social Media Users (2024-12-16)
• Incident 878 - Romance Scammer 'Alla Morgan' Allegedly Exploits Deepfake Technology to Defraud Victim of £17,000 (2024-12-19)
• Incident 879 - Deepfake Video Reportedly Depicts U.S. Congressman Rob Wittman Endorsing Military Support for Taiwan's Democratic Progressive Party (2023-12-29)
• Incident 880 - Scammers Reportedly Using Deepfakes of Health Experts and Public Figures in Australia to Sell Health Supplements and Give Harmful Advice (2024-12-09)
• Incident 881 - Waymo Robotaxi Allegedly Collides With Serve Robotics Delivery Bot in Los Angeles (2024-12-27)
• Incident 882 - AI-Generated Reading Summaries on Fable App Reportedly Wrote Biased and Offensive Commentary (2024-12-29)
• Incident 883 - Alleged Fake AI-Generated Christmas Card Featuring Prince Harry and Meghan Markle's Children Circulates on Social Media (2024-12-26)
• Incident 884 - Russian Center for Geopolitical Expertise Allegedly Used AI to Target U.S. Candidates with Disinformation in 2024 Election (2024-12-31)
• Incident 885 - Meta AI Characters Allegedly Exhibited Racism, Fabricated Identities, and Exploited User Trust (2025-01-03)
• Incident 886 - ChatGPT Reportedly Referenced During Las Vegas Cybertruck Explosion Planning (2024-12-27)
• Incident 887 - Students of Richland School District in Cambria County, Pennsylvania Allegedly Used AI to Generate Obscene Images of Other Students (2024-11-14)
• Incident 888 - Sydney High Schooler Allegedly Generated Deepfakes of Other Students (2025-01-06)
• Incident 889 - Tesla's 'Actually Smart Summon' Feature Reportedly Linked to Multiple Parking Lot Collisions (2025-01-07)
• Incident 890 - AI-Generated Images of the Iconic Hollywood Sign Reportedly on Fire Circulating on Social Media (2025-01-08)
• Incident 891 - AI Voice Scam Targets Westchester Parents with Fake Kidnapping Ransom Calls (2025-01-08)
• Incident 892 - Apple Intelligence Allegedly Flags Scam Messages as High Priority (2024-08-05)
• Incident 893 - Pennsylvania State Police Officer Allegedly Used Work Computer for AI-Generated Pornography (2025-01-09)
• Incident 894 - RealPage's YieldStar Allegedly Facilitated Rent Price Coordination Among Canadian Landlords (2017-01-01)
• Incident 895 - Alleged Deepfake of New Zealand Endocrinologist Reportedly Promotes Misleading Diabetes Claim (2025-01-06)
• Incident 896 - Alleged Misuse of Facial Recognition Technology by Law Enforcement Reportedly Leading to Wrongful Arrests and Violations of Investigative Standards (2025-01-13)
• Incident 897 - AI-Assisted Ransomware Campaign by FunkSec Allegedly Targets Over 80 Victims (2025-01-10)
• Incident 898 - Alleged LLMjacking Targets AI Cloud Services with Stolen Credentials (2024-05-06)
• Incident 899 - Character.ai Chatbots Allegedly Emulating School Shooters and Their Victims (2024-12-17)
• Incident 900 - Character.ai Has Allegedly Been Hosting Openly Predatory Chatbots Targeting Minors (2024-11-13)
• Incident 901 - Yahoo Boys Allegedly Used Deepfake Technology to Impersonate Brad Pitt and Defraud French Woman of $850,000 in Romance Scam (2025-01-12)
• Incident 902 - Prime Minister of Thailand Paetongtarn Shinawatra Claims AI Voice Scam Impersonated ASEAN Leader Requesting Money (2025-01-15)
• Incident 903 - Northern Ireland MLA Cara Hunter Allegedly Targeted by Deepfake Pornography Ahead of May 2022 Assembly Election (2022-04-15)
• Incident 904 - Kate Isaacs, Advocate Against Image-Based Abuse, Reports Being Deepfaked (2022-10-21)
• Incident 905 - Scammers Allegedly Use Deepfake Technology to Pose as Leonor, Princess of Asturias, in Fraud Scheme (2024-07-01)
• Incident 906 - Alleged AI-Powered Call Center Breach Exposes Over 10 Million Conversations in the Middle East (2024-10-08)
• Incident 907 - Taranaki, New Zealand Resident Allegedly Defrauded of $224K in Bitcoin Scam Using Deepfake of Prime Minister Christopher Luxon (2024-07-15)
• Incident 908 - Deepfake Images of Australian Teacher Hannah Grundy and 25 Others Created and Circulated Online by Former Friend (2020-07-15)
• Incident 909 - Matryoshka Campaign Allegedly Uses Deepfakes to Impersonate Academics for Pro-Russian Propaganda (2024-12-13)
• Incident 910 - Alleged AI-Driven Phishing Scam Impersonates Maine Town Official to Falsely Request $22,500 (2025-01-15)
• Incident 911 - Yahoo Boys Allegedly Employ Real-Time Deepfake Technology in Romance Scams (2022-05-01)
• Incident 912 - Yahoo Boys and Scammers from Morocco Allegedly Target U.S. Widows and Vulnerable Individuals with 'Artificial Patriot' Scams (2024-11-21)
• Incident 913 - Yahoo Boys Allegedly Using AI-Generated News Videos to Blackmail Sextortion Victims (2025-01-27)

👇 Diving Deeper

• Check out the Table View and List View for different ways to see and sort all incidents.
• Explore clusters of similar incidents in Spatial Visualization.
• Learn about alleged developers, deployers, and harmed parties in Entities Page.

🦾 Support our Efforts

Still reading? Help us change the world for the better!

1. Share this newsletter on LinkedIn, Twitter, and Facebook.
2. Submit incidents to the database.
3. Contribute to the database’s functionality.