Executive Summary

After many conversations with those taking inspiration and building from the AI Incident Database, we have notes to facilitate the growth of the incident cataloging effort.

• Corporations can start by integrating their legal and engineering efforts.
• Governments can start with AI security incident reporting.

---

It was late 2018 when I began the work that would become the AI Incident Database. The idea was straightforward: mature industrial sectors like aviation and nuclear power systematically collect their real-world failures to inform safety improvements. Intelligent systems were already causing real-world harms, but there was no collective memory of those failings. Companies were repeating the same mistakes in the design, development, and deployment of AI because no one had built the shared infrastructure to learn from what went wrong. So I set out to build it, first with the help of the Partnership on AI, and later through the organization we founded to steward the work: the Responsible AI Collaborative.

Eight years later, I find myself reflecting on how far we have come and how we are likely at an impasse unless we recognize political realities and call for pragmatic solutions to the AI safety data problem: we need to better integrate corporate legal and engineering departments. To explain what this means, I'll start with history and return to recommendations.

What the AI Incident Database Has Achieved

The AI Incident Database has grown from a scrappy collection of ad hoc reports into the world's standard measure of AI harms in the real world. The Guardian relied on our analysis to report that deepfake fraud has reached "industrial scale." The OECD built its common reporting framework for AI incidents on foundations that we helped lay, and that framework kicked off the ETSI and ISO incident standards processes.

Our research program has matured in parallel. We have published on preventing repeated AI failures (IAAI-21), on indexing AI risks with incidents, issues, and variants, on lessons for editors of AI incidents, and on AI flaw reporting practices, and on scalable incident classification. We have built taxonomies, entity graphs, multilingual reporting infrastructure, and an editorial community that keeps the database current. These outputs have enabled thousands of other researchers, educators, and engineers to build better and safer. We are proud to have built something bringing everyone from labor activists to management consultants to the same space.

These are real accomplishments. But it is far from enough.

Why Incident Data Matters

The core insight behind the AI Incident Database remains as relevant today as it was in 2018. As I wrote in the original paper introducing the AI Incident Database to the world: intelligent systems cause real-world harms without a collective memory of their failings, and as a result, companies repeatedly make the same mistakes. The aviation industry owes much of its extraordinary safety record to the systematic practice of analyzing and archiving past accidents and incidents. Air travel did not become one of the safest forms of transportation by accident; it became safe because the industry committed to learning from every failure.

AI is different from aviation in important respects. The comprehensive nature of "intelligence" means AI incident databases must ingest unforeseen and novel contexts, technologies, and failures. A facial recognition system misidentifying a pedestrian's shirt text as a license plate, an AI companion chatbot allegedly influencing a teenager toward suicide, deepfake face-swapping used to bypass financial authentication — these are profoundly different events with profoundly different causative factors. But the principle is the same. Without tracking incidents, we cannot avoid them. Without a shared record of what has gone wrong, every product team starts from zero.

A Troubling Question

This brings me to something that has been weighing on me. The AI Incident Database has always been premised on the development of a federated ecosystem of incident data. It is why the organization behind the database is the "Responsible AI Collaborative" — the name was chosen to signal that this was never meant to be a solo endeavor. The aspiration was always that multiple parties, including corporations and governments, would collect and share incident data, creating a rich tapestry of information about what goes wrong with AI and why.

Eight years in, that federated ecosystem has not materialized in the way we hoped. The question I keep asking is: why?

Why Corporations Are Not Sharing Incidents

Corporations are, in many ways, the parties best positioned to catalogue AI incidents. They have the domain expertise, the engineering teams, and the operational context to understand the causative factors behind a failure. In a better world, they would be the most prolific contributors to a shared body of incident knowledge.

But cataloguing AI incidents is a harm-centered activity. It is, by definition, the documentation of "bad things that happen with AI." And harms are closely related to tort liability. You can't sue for damages under tort law without finding a harm and product liability law is an extension of tort law.

We saw this play out dramatically in March 2026, when a jury found Meta and Google liable in the landmark social media addiction trial, ordering them to pay damages. Roughly two thousand additional pending lawsuits awaited the outcome of that suit as legal precedent. Product liability law is now extending its reach into the digital world in ways the technology industry long resisted. Courts are treating social media platforms — and potentially, AI systems — as products that can be defective under traditional tort principles. The Character.AI settlement over a teenager's suicide, the growing docket of product liability suits against OpenAI, and the broader doctrinal trajectory all point in the same direction.

In this legal environment, a corporation developing a written record of AI harms may produce records subject to legal discovery. The very documentation that would make AI safer — detailed, causal, and honest accounts of what went wrong — is precisely the kind of material that plaintiffs' attorneys would love to surface in litigation. This creates a perverse incentive structure in which the entities most capable of understanding AI failures are also the most reluctant to document them.

This does not mean corporations are ignoring the problem entirely. Many are collecting what I think of as "incident analogues" (e.g., user complaints, bug reports, flagging model flaws, etc.) that serve some of the same functions without the explicit framing of "harm" or "incident." These analogues are important and underdeveloped in policy. For now, legal staff at technology companies should understand that their technologies are broadly getting safer through time through concerted engineering effort, and incident reporting is the most effective means of measuring and supporting that assertion.

So software product liability may make incident collection an even more challenging task for companies. Maybe the government can help?

Why Governments Are Not Collecting AI Incidents

Governments are natural coordinators of safety data. The NTSB collects aviation incidents. The FDA collects adverse drug events. The Consumer Product Safety Commission (CPSC) collects product safety data. One might expect governments to play a similar role for AI.

The challenge is that the framing of "AI incident" as a "harm event" is far more consistent with a product liability framing than a computer security framing. And this distinction matters enormously for political feasibility.

Governments have extensive experience coordinating security data. The CVE system for cybersecurity vulnerabilities and the responsible disclosure frameworks that the security community has developed over decades. When a government collects and distributes data about security vulnerabilities, it is broadly understood to advance both private and public interests. The political dynamics are manageable.

But collecting data that could increase the liabilities of powerful corporate actors is a much more challenging fight politically. The CPSC's authority to collect and distribute product hazard data was hard-won and remains contested. Software has never been subject to this sort of scrutiny. There is no regulatory tradition of treating software outputs as products whose harms must be systematically tracked by a government body. The EU AI Act and California's SB 53 require certain incident reporting to authorities, but only for the most serious or safety-critical cases, and the infrastructure for collecting, analyzing, and distributing this data at scale does not yet exist. Introduced just last week, H.R. 9333 aims to plug this hole as it explicitly seeks to define and address the full range of incident terms (e.g., vulnerabilities, failure modes, accidents, failures, hazards, catastrophes, misuse, safety incidents, security incidents, adverse events). The bill also seeks to establish a path to federal incident reporting. I expect the terms, thresholds, and implementation to be a challenging political balancing act. I hope we can start in areas of broad consensus (security) and work towards areas of greater contention.

As my co-author Kathrin Grosse and I wrote for the OECD, the cybersecurity and general safety communities have genuinely different priorities, methodologies, and requirements when it comes to incident reporting. Attempting to force security norms into non-security incidents, or product-safety norms into security contexts, creates friction that slows adoption. The square peg of security methodology does not fit neatly into the round hole of common incident reporting, and vice versa.

With these challenges stifling a critically important approach to AI safety, what should we do?

What I Have Learned From Working Across Contexts

Over the past several years, I have been consulting with researchers and governments across six continents. In each of these conversations, it is necessary to think about not only what would ideally exist but also what is achievable given the political context within which AI is operating.

If you are a corporation, my recommendation is to work to collect information about incidents internally and to have in-house legal counsel translate those incidents into new engineering requirements in a way that does not increase corporate liabilities. The knowledge embedded in your incident data is extraordinarily valuable for making your products safer. The challenge is structuring that knowledge flow so that it serves engineering improvement without becoming a litigation liability. This is not an impossible problem, but it requires deliberate institutional design. I am excited to be working with Arcadia Impact on this design problem now.

If you are a government, my recommendation is to focus initially on collecting AI security incidents specifically. The institutional patterns for security data collection are well-established, the political dynamics are more favorable, and the need is urgent as AI proves increasingly vulnerable just as AI-enabled attacks grow more sophisticated. From that foundation, governments can potentially broaden to all AI safety incidents but only after building the institutional capacity to address the broader collection demands. If you are a government focused on cataloging AI security incidents: please ensure you explicitly label the data AI security incidents rather than "AI incidents." Names have power and this subtle change resolves many conflicts between communities of practice. H.R. 9333's focus on definition and voluntary starting points give the capacity to extend security practices to the AI space while designing for a future that covers safety data beyond the security frame.

Conclusion

After eight years of cataloguing AI incidents, the AI Incident Database still constantly refines the processes and criteria upon which it makes ingestion decisions. We have now read more about the bad things associated with AI than perhaps anyone else on earth. But we remain optimistic about what can be done to make the world better.

If we are to successfully navigate into a future with greater augmentation by AI, a balanced attitude is essential. Safety arises from a blameless safety culture. The aviation industry learned this decades ago: the first time a bad thing happens is an opportunity to learn, not an occasion for punishment. It is precisely this principle that makes aviation's incident reporting systems work. People report failures because they know the system is designed to prevent recurrence.

AI needs the same culture. It does not yet have it. But we can build it together by advancing the parallel streams of culture, law, best practices, and database systems.

I hope we will see greater participation in the incident cataloging space, including by governments and corporations. The need has never been more apparent, even as the institutional obstacles remain formidable. Regardless, the AI Incident Database will continue to be the world's first launched and catchall of real world harms associated with AI. And we're excited to work with new collaborators towards a safer digital and real world future. Let's work together to produce an effective ecosystem of blameless incident reporting.

Donate to make a safer world through data.