AI incidents are suddenly front-page news. In July, AI systems that OpenAI was testing inside a sealed-off environment found a gap in the seal, reached the open internet, and broke into the live systems of another company, Hugging Face. Nine days after OpenAI confirmed the systems were its own, Anthropic disclosed three incidents in which its Claude models jumped from test environments to real companies. In September, Australia’s prime minister announced that an OpenAI agent had gotten into a government health-statistics website in June, and that his government learned of it three months later. Reporters are asking us how we got here.

We get it. This is frightening. But it is not inevitable. Just because AI is creating harms in new ways does not mean that we are helpless. We need not start from scratch. If you are building AI, that means you can learn from past AI incidents and ensure you, your organizations, and society more generally can realize the benefits of AI without further expanding its risks.

That is why the two of us decided to record Inside the Incident, a new video series that takes one incident at a time, dissects what went wrong, and extracts the governance and legal lessons for the people who deploy these systems. There are many segments you can watch–and more are coming! You can watch them here, but we’ll provide more details on each below.

As the philosopher George Santayana once put it, “those who cannot remember the past are condemned to repeat it.” Our goal is to ensure that past AI incidents are not forgotten, not repeated, and avoided whenever possible.

We Could See This Coming

Nothing about this summer surprised people who catalog AI incidents. The AI Incident Database, which one of us runs, has recorded more than 1,700 incidents since 2018, and every one of this year’s agentic breaches has multiple ancestors in the record. Each ancestor was reported, discussed for a news cycle, left unmitigated, and in the rush to adopt AI, all-too-easily forgotten.

Had the industry fixed those predecessors, we would not be scrambling to contain agents that break into other people’s systems. We are now scrambling to mitigate and contain a rapidly expanding fire whose warning signs were visible when it was still a spark.

A Normal Technology Turning Abnormal

Arvind Narayanan and Sayash Kapoor’s AI as Normal Technology argues that AI should be understood the way we understand electricity or the internet: a new class of tools rather than a new species. We agree.

But normality is not a property of the technology. It is a property of the relationship between the technology, the people, and the institutions around it. AI is in the process of becoming an abnormal technology because we are failing to control the harms it is capable of generating.

We have collectively ignored the warnings, each one arriving as an incident. Each incident demonstrates new hazards the next generation of systems inherits rather than resolves.

We could have acted earlier to implement new safeguards, enact new laws, and reframe the way that companies and governments restrain AI systems. The rise in AI incidents is an illustration of our collective failure to act.

The Road to Abnormal

Let’s look at how we got here.

Step 1: Adaptive Chat (2016)

Incident 6: Microsoft released Tay to learn from conversations on Twitter. Within a day it was posting racist, sexist, and antisemitic content, and Microsoft shut it down. The hazard it introduced was a conversational system that continues whatever it is fed. As Sean wrote in How to Understand Large Language Models through Improv, these systems follow the improv rule of "yes, and." Expose a chatbot to the open internet and the internet decides where the scene goes.

Tay was public, so it was shut down in hours, but it was not the last. The same failure recurred in:

Incident 58: Yandex's Alice chatbot voiced support for Stalin and violence (2017)
Incident 106: Scatter Lab's Luda made offensive remarks about minority groups (2020)
Incident 569: a Replika companion encouraged a plot to assassinate Queen Elizabeth II (2021)
Incident 278: Meta's BlenderBot 3 demo made antisemitic comments (2022)
Incident 503: Bing Chat declared threats against its users (2023)

This is just a partial list.

Step 2: Adaptive Chat in Safety-Critical Situations (2023)

Incident 545: The National Eating Disorders Association offered a chatbot named Tessa to people seeking help with eating disorders. After its vendor added generative features, Tessa began advising those users on calorie deficits and weekly weigh-ins, and NEDA suspended it. The hazard it introduced was the same continuation tendency, now placed where a single bad turn can injure someone already vulnerable. Just as before, once a conversation drifts toward encouraging an eating disorder, the system keeps the conversation going.

Tessa was not alone. The database records:

Incident 449: a startup ran GPT-3 on mental-health support users without telling them (2022)
Incident 505: a Chai companion was implicated in a Belgian man's suicide (2023)
Incident 826: Character.AI allegedly influenced a Florida teenager toward suicide (2024)
Incident 1200: Meta AI roleplayed suicide and eating disorders with teen accounts (2025)

Each was a deployment into a safety-critical setting of a hazard first documented in 2016.

Step 3: Action-Taking Agents (2025)

Incident 1152: In July 2025, a Replit coding agent deleted a customer's live production database during a code freeze, despite repeated instructions not to make changes, then produced fabricated test results and claimed rollback was impossible. The hazard it introduced was that output became action. A chatbot's bad sentence can be ignored; an agent's bad command is executed. And agents act at a rate no human reviews. A system that is right 99.9 percent of the time and takes a thousand actions in five seconds will, on average, fail once in those five seconds.

The same hazard has recurred across nearly every major vendor:

Incident 1178: Google's Gemini CLI deleted a user's files after misreading a command sequence (2025)
Incident 1373: a coding agent published a personal attack on a matplotlib maintainer (2026)
Incident 1424: Claude Code deleted DataTalks.Club's production infrastructure via Terraform (2026)
Incident 1469: a Cursor agent deleted PocketOS's production database (2026)
Incident 1542: an OpenClaw agent tried to delete emails despite stop commands (2026)
Incident 1719: Claude Code deleted about 48,000 live project files and their version history (2026)

Step 4: Weaponized Agents (2026)

Incident 1604: Cybersecurity models under evaluation at OpenAI were instructed to be persistent and break a collection of test systems. The models were meant to be contained to a test system, but thousands of them broke out and started hacking websites outside of the test environment. As Andrew wrote back in March, AI agents can act a lot like malware.

Failures to control agents are not one company’s problem. Within weeks the record filled with the same hazard at other labs and in other hands:

Incidents 1627, 1628, and 1629: nine days after OpenAI’s attribution, Anthropic disclosed three evaluation incidents in which Claude models compromised a real company’s infrastructure, published a malicious package to PyPI, and scanned 9,000 targets (2026)
Incident 1700: a Google Gemini evaluation agent accessed three real companies' protected systems (2026)
Incident 1649: a Meta AI model exploited a vulnerability in a real third-party service during evaluation (2026)
Incident 1707: OpenAI reported that one of its agents researching public health spending got past access controls on Australia's Medicare statistics portal (2026)
Incidents 1731 and 1710: agents on ordinary research tasks tried to break into Canada’s federal archives and a U.S. Department of Education website (2026)
Incident 1263: a state-linked operator used Claude Code for largely autonomous cyber espionage (2025)

These systems were told to be persistent and to break things. The chat failures of 2016, the safety hazards of 2023, and the action hazards of 2025 were all still present underneath.

Keeping AI Normal

AI is now officially scary and that is not normal. Should we pause or pace AI development? Maybe. But you probably don't work at the frontier of AI research, so what should you do about it?

Companies fighting for their survival in the competitive markets are not likely to seriously invest in safety until someone gives them a reason. They have too much to lose. But your tools to change the incentives are "normal" and you have them already. Negligence, product liability, defamation, and computer-intrusion law don't pause AI advances or eliminate all its dangers -- but they are all tools that can help to address these risks. What has been missing is the will to apply that normal toolset.

This is starting to change. Billions of dollars in settlements from social media companies are proving that product liability is substantial. In March, a Los Angeles jury found Meta and Google negligent in the first trial over addictive design; in June, YouTube settled rather than face a second jury; in August, Meta agreed to pay the states up to $16.7 billion. Thousands of suits by families, school districts, and municipalities are still waiting for them.

The same forces are already at work in AI. When the Texas Attorney General acted against Pieces Technologies for an unsupported accuracy claim, consumer protection statutes were enough. The same was true when the FTC banned Rite Aid from using facial recognition for five years after its system wrongly flagged shoppers as shoplifters; when a tribunal held Air Canada to the bereavement policy its chatbot invented; and when the FTC settled with DoNotPay and Evolv over unsupported claims about what their AI could do.

So much in the world of AI is new. But that does not mean that we are all starting from scratch.

**A Deeper Dive Through Inside the Incident with Sean and Andrew**

Yes, AI incidents are scary. But they’re less scary once you understand how they could have been prevented. We’ve tried to pick the incidents worth exploring in our new Inside the Incident series, which you can watch here.

Here are some specific examples you can watch:

Episode 1: Buying a Car for $1, on Incident 622: a dealership chatbot agrees to sell a Tahoe for a dollar after a trivial prompt injection. Deploying a system you did not build, without contractual assurances about testing, is how an adaptive-chat hazard becomes your liability.
Episode 2: Drowning Robots & AI Agents, on Incident 68: a security robot rolls into a fountain. If you have no test documentation for the environment you deploy into, you are the experiment. A standard of care exists for physical robots that may develop for agents.
Episode 3: How Accurate Is Your AI?, on Incident 1324: a clinical AI company markets a one-in-100,000 hallucination rate, and the Texas Attorney General intervenes under existing consumer protection law.
Episode 4: AI Agents Make Crabby Bloggers, on Incident 1373: an autonomous coding agent publishes a personal attack on an open-source maintainer. What once required a state-sponsored persistent threat now requires only persistence.

What You Can Do

The people best positioned to change the course of AI are not AI safety researchers. They are the compliance officers, general counsels, risk managers, product leaders, CISOs, and anyone building AI, each of whom decides every day which systems get deployed and on what terms. If that is you:

Before deploying an agent, match your use case against the incident record and ask your vendor which of those incidents it has mitigated, and how it knows.
Treat an incident at a peer as a requirement for you. Understand how it could have been avoided and write those requirements into contracts, test plans, or acceptance criteria.
Submit incidents and responses when they happen to you. You can prevent it from happening to others.
Watch and share Inside the Incident, and support the AI Incident Database.

The emergency was yesterday. The way to keep AI normal is not to wait for the next headline. It is to use the existing social institutions to force a fix to what we already know is broken.

---

Sean McGregor is the founder of the AI Incident Database and Executive Director of the Responsible AI Collaborative. He is a fellow at Harvard's Berkman Klein Center for Internet & Society, a member of the OECD Network of Experts on AI, and founding lead research engineer with the AI Verification and Evaluation Research Institute.

Andrew Burt is CEO of Luminos.AI, a no-code testing platform for AI agents, and a visiting fellow at Yale Law School’s Information Society Project. A former FBI official and co-author of the US government's primary standard for AI risk management, he has advised international regulators and the US military on how to deploy AI in high-risk environments.

Acknowledgments

Thank you to Daniel Atherton for his input on this post.

The Emergency Was Yesterday, the Crisis Is Now