Description: A study by UpGuard reports that misconfigured llama.cpp servers publicly exposed user prompts, including hundreds of interactive roleplay scenarios. Some prompts explicitly described fictional sexual abuse of children aged 7–12. While no real children were involved, the findings demonstrate how open-source LLMs can be exploited to generate AI-enabled child sexual abuse material (CSAM).
Entities
View all entitiesAlleged: Users of llama.cpp servers and Meta developed an AI system deployed by Users of llama.cpp servers, which harmed Users of llama.cpp servers and General public.
Alleged implicated AI systems: llama.cpp , LLaMA and public-facing HTTP API servers with insecure /slots configuration
Incident Stats
Incident ID
1020
Report Count
2
Incident Date
2025-04-11
Editors
Daniel Atherton
Incident Reports
Reports Timeline
%20(1).png)
Several AI chatbots designed for fantasy and sexual role-playing conversations are leaking user prompts to the web in almost real time, new research seen by WIRED shows. Some of the leaked data shows people creating conversations detailing …
%20(1).png)
The proliferation of AI has rapidly introduced many new software technologies, each with its own potential misconfigurations that can compromise information security. Thus the mission of UpGuard Research: discover the vectors particular to …
Variants
A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.
Similar Incidents
Did our AI mess up? Flag the unrelated incidents
Biased Sentiment Analysis
· 7 reports
Gender Biases in Google Translate
· 10 reports
Similar Incidents
Did our AI mess up? Flag the unrelated incidents
Biased Sentiment Analysis
· 7 reports
Gender Biases in Google Translate
· 10 reports