Description: In August 2024, the San Francisco City Attorney’s Office filed a first-of-its-kind lawsuit against 16 websites accused of using generative AI to create and distribute nonconsensual pornographic deepfakes, including images of minors. The sites, which had over 200 million visits in six months, invited users to upload real images to generate explicit fakes. By March 2025, 11 of the targeted websites had shut down following legal pressure.
Editor Notes: Timeline notes: The San Francisco City Attorney's office announced its actions against the 16 operators of these sites on August 15, 2024 (the date for this incident ID). By March 2025, reporting on these actions stated that 11 of the sites had been taken down.
Entities
View all entitiesAlleged: Unknown deepfake technology developers developed an AI system deployed by Operators of 16 deepfake pornography websites, which harmed Women depicted in deepfake pornography and Minors depicted in deepfake pornography.
Alleged implicated AI systems: Unknown deepfake apps and Unknown generative AI tools
Incident Stats
Incident ID
1004
Report Count
2
Incident Date
2024-08-15
Editors
Daniel Atherton
Incident Reports
Reports Timeline
AI-generated nonconsensual explicit images of real adults and minors cause immeasurable harm and violate state and federal laws
SAN FRANCISCO (August 15, 2024) --- San Francisco City Attorney David Chiu announced a first-of-its-kind lawsuit…
The San Francisco City Attorney's Office is targeting websites accused of creating and spreading deepfake nudes, taking the sites offline and sending a clear message.
Chief Deputy City Attorney Yvonne Meré told NBC Bay Area the one-of-a-kin…
Variants
A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.
Similar Incidents
Did our AI mess up? Flag the unrelated incidents
Similar Incidents
Did our AI mess up? Flag the unrelated incidents