Incident 1020: Reportedly Unsafe Deployment of Llama.cpp Reveals Interactive AI-Generated CSAM Roleplay Prompts

Description: A study by UpGuard reports that misconfigured llama.cpp servers publicly exposed user prompts, including hundreds of interactive roleplay scenarios. Some prompts explicitly described fictional sexual abuse of children aged 7–12. While no real children were involved, the findings demonstrate how open-source LLMs can be exploited to generate AI-enabled child sexual abuse material (CSAM).

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History

Incident Stats

Incident ID
1020
Report Count
2
Incident Date
2025-04-11
Editors
Daniel Atherton
Applied Taxonomies
MIT

MIT Taxonomy Classifications

Machine-Classified
Taxonomy Details

Risk Subdomain

2.1. Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

Risk Domain

  1. Privacy & Security

Entity

Human

Timing

Post-deployment

Intent

Unintentional

Loading...
Sex-Fantasy Chatbots Are Leaking a Constant Stream of Explicit Messages
wired.com · 2025

Several AI chatbots designed for fantasy and sexual role-playing conversations are leaking user prompts to the web in almost real time, new research seen by WIRED shows. Some of the leaked data shows people creating conversations detailing …

Loading...
Analyzing llama.cpp Servers for Prompt Leaks
upguard.com · 2025

The proliferation of AI has rapidly introduced many new software technologies, each with its own potential misconfigurations that can compromise information security. Thus the mission of UpGuard Research: discover the vectors particular to …

Variants

A "variant" is an AI incident similar to a known case—it has the same causes, harms, and AI system. Instead of listing it separately, we group it under the first reported incident. Unlike other incidents, variants do not need to have been reported outside the AIID. Learn more from the research paper.
Seen something similar?

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents

Loading...
Biased Sentiment Analysis

Biased Sentiment Analysis

· 7 reports
Loading...
High-Toxicity Assessed on Text Involving Women and Minority Groups

High-Toxicity Assessed on Text Involving Women and Minority Groups

· 8 reports
Loading...
Gender Biases in Google Translate

Gender Biases in Google Translate

· 10 reports
Previous IncidentNext Incident