As AI incidents become more frequent and severe, societies need reliable ways to identify harms early and keep them from becoming crises. Countries therefore need systems that connect what is learned from an incident to an effective response, while carrying those lessons forward into future preparedness. In this guest post, Omer Bilgin and Caio Machado from The Future Society propose what such an AI incident management and crisis prevention pipeline could look like. Their proposal grows out of their flagship report on cross-border AI incident infrastructure and draws on work already underway, including efforts such as the AI Incident Database.

- Sean McGregor, Executive Director, Responsible AI Collaborative
- Daniel Atherton, Lead Editor, AI Incident Database

Background & Context

Recent loss-of-control incidents involving OpenAI, Anthropic, Meta, and the UK AI Security Institute add to existing evidence that incidents with increasingly severe consequences are starting to emerge across companies, evaluators, and jurisdictions. 

These cases have exposed serious sandboxing, oversight and containment issues in the internal testing and evaluation processes of frontier AI companies, going beyond existing issues related to the malicious use of their systems by human actors. Even worse, some companies have demonstrated that they will not always voluntarily disclose information about unfolding incidents to relevant authorities until those incidents become public or clearly cross mandatory incident reporting thresholds, which are generally set quite high in existing legislation.

At The Future Society, we use a multi-stage escalation pathway when thinking about AI-induced risks and harms: hazards refer to circumstances or findings that signal growing risk that real-world harm might be imminent, such as near-misses or experimental findings uncovered by frontier AI companies during internal development about specific AI capabilities or propensities; incidents refer to tangible harms that have occurred but that stay limited in scale, contained without widespread spillover into broader society; and crises refer to a subset of widespread, high-impact incidents that cause severe, large-scale harm or result in systemic disruptions, typically extending across borders or sectors, and that demand coordinated response across multiple institutions and/or jurisdictions. 

Since capabilities keep broadening, performance keeps improving, and agentic deployments across both individuals and organizations keep expanding, both the severity and frequency of AI incidents will likely only increase. If the events that have transpired over the past two months tell us anything, it is that current incident management and crisis prevention practices are fundamentally inadequate, and that future crises are looming.

Recognizing this reality, our recent report, The Case for Cross-Border AI Incident Infrastructure, argues that effective AI governance requires robust institutional, technical, and legal arrangements that can enable governments and organizations to better detect, analyze, prepare for, and act upon AI incidents to prevent them from escalating into widespread crises. However, operationalizing the recommendations laid out in that report requires a foundational, well-defined, mutually recognized understanding of the full AI incident management and crisis prevention (AI-IMCP) pipeline, which current discourse around AI incidents largely lacks. 

To effectively prevent AI crises, policymakers, regulators, and industry leaders must first understand the various processes that make up the AI-IMCP pipeline. The goal of this piece is to introduce, define and clearly lay out the full AI-IMCP pipeline, namely what processes and actors comprise it, to ensure everyone in the AI governance ecosystem is speaking about the same things when it comes to incident management and crisis prevention. Below we offer a figure illustrating the pipeline and in the body of this piece we describe each element.

Figure 1. The full AI-IMCP pipeline. The pipeline comprises five sequential components: crisis prevention and preparedness; monitoring and detection; reporting; response; and post-incident analysis. Analytical insights from post-incident analyses can feed back into crisis prevention, making the pipeline a continuous cycle rather than a linear process. A sixth component, information-sharing, runs continuously across all stages.

The AI Incident Management and Crisis Prevention (AI-IMCP) Pipeline

The AI-IMCP pipeline requires actions from a wide range of actors, distributed across the AI industry value chain and beyond, such that effective AI-IMCP cannot be secured through the actions of a single actor, but rather through a series of processes that span multiple stakeholders and often multiple jurisdictions. 

The full pipeline consists of six interconnected procedural components:

1. Crisis Prevention & Preparedness

Crisis prevention and preparedness refer to the plans and proactive protocols organizations can establish to anticipate and prevent isolated incidents from escalating into crises.

This component covers the design and publication of Serious Incident Prevention Policies (SIPP)—robust policies and pre-determined protocols for preventing and preparing for incidents—by frontier AI developers, downstream deployers or governments, the execution of mandatory tabletop exercises simulating realistic crisis scenarios that stress-test proposed incident response plans, and the utilization of insights derived from the forensic root-cause and supply-chain analysis of past incidents to prevent their future recurrence and escalation into crises.

2. Monitoring & Detection

In the context of industry actors, incident monitoring and detection refers to the continuous, automated systems and mechanisms used to observe AI system behavior, log granular telemetry data, and identify emerging hazards before they escalate. This entails tracking operational metrics—such as tool-call traces, API access records, user interaction logs, latency, and chain-of-thought reasoning—to enable forensic investigation following safety incidents or security breaches.

Because AI behaviors are tightly bound to their deployment context, effective incident monitoring and detection relies on a complex, distributed chain of actors. The stakeholders that can be involved in monitoring telemetry data include frontier AI developers, downstream deployers, AI agent harness developers, AI tool providers, cloud compute providers, MCP server providers, and AI model hosting services. Moreover, many sectoral regulators monitor AI incidents and hazards through a wide array of operational data, ranging from direct security incident reports to aggregated consumer complaints gathered via dedicated reporting portals, mandatory scheduled data submissions, and third-party sources. 

However, it is also important to note that monitoring and detection extend far beyond the technical telemetry collected by industry actors. A diverse ecosystem of non-industry stakeholders—including sectoral regulators, local and federal government authorities, AI Safety Institutes (AISIs) and AISI-equivalents, multilateral bodies, journalists, and civil society organizations—also plays a critical role in detecting AI harms. Their efforts focus less on granular system metrics and more on aggregating real-world impacts. 

For instance, researchers and civil society organizations maintain comprehensive public incident databases and AI vulnerability reporting platforms that continuously monitor and track incidents and hazards surfaced by news media and public disclosures from researchers and companies. Insights from these can then be used by government authorities to inform their governance decisions. As such, robust governance in all other subsequent pipeline components depend on the comprehensiveness and quality of conducted practices in this one.

3. Reporting

Incident reporting entails the formal disclosure of detected incidents, near-misses, or hazards to relevant stakeholders. 

The actors involved in incident reporting span those who disclose incident-relevant information, which can include frontier AI developers, high-risk deployers, whistleblowers, civil society organizations, and news media outlets, and those who receive it, primarily government authorities and relevant external oversight bodies.

Numerous jurisdictions – including in the EU, China, South Korea, and the United States (through state-level legislation in California, New York, and Illinois) – have started legally mandating frontier AI companies to disclose serious incidents in which their models are implicated to specific government authorities within defined timelines. However, despite the existence of such reporting regimes, most incident reporting efforts to date have been limited to either whistleblowers from within frontier AI companies, victims of AI harm, or members of the general public disclosing particular incidents or hazards they have observed to journalists (who then report that information to the public) or government authorities. 

4. Response

Response covers the immediate, concrete actions taken by various stakeholders to mitigate, contain, and recover from an active AI incident or crisis once it has struck. This procedural step involves things like executing predefined incident response plans and containment workflows, initiating system rollbacks, revoking API access, restricting tool use, and establishing clear prioritization and communication hierarchies during fast-moving crises.

Effective incident and crisis response requires coordination among both industry players and regulatory actors. Stakeholders relevant to incident response include corporate leadership, engineering, IT and security teams within AI developing or deploying companies, local and federal law enforcement, state-level, federal and regional emergency response bodies like the California Governor’s Office of Emergency Services (CalOES), the US’s Department of Homeland Security, the EU’s Emergency Response Coordination Centre (ERCC), and national legislative authorities. 

5. Post-Incident Investigation & Analysis

Post-incident investigation and analysis refer to the structured, retrospective examination of AI incidents or crises after they have been contained, aimed at establishing what happened, why it happened, and what must change to prevent their recurrence.

This procedural component’s main purpose is to extract preparedness-relevant insights from past incident records to let various stakeholder groups systematically understand why specific incidents occurred. Where incident response addresses the immediate issue, investigation and analysis convert notable learnings from its management into durable governance knowledge. This can include, for example, a frontier AI company translating raw internal monitoring data about unreleased models involved in a particular training or evaluation run into actionable evidence that can be shared with the broader industry ecosystem as well as with government authorities to inform their decision-making and ensure that harms and vulnerabilities spotted in one deployment environment can act as early warnings for others.

This component covers forensic root-cause analysis across technical, organizational, governance, and ecosystemic dimensions; supply-chain analysis tracing how a failure at one stage of the AI value chain propagated across systems, sectors, and borders; multi-level impact assessment capturing harms at the individual, societal, and systemic levels; and the translation of these findings into policy-relevant insights, such as the identification of scope gaps (where no legal duty currently exists), capacity gaps (where duties exist without the tools to fulfil them), and coordination gaps (where duties lack cross-border reach).

The actors involved in investigation and analysis span frontier AI developers conducting internal post-mortems, national AISI-equivalents and sectoral regulators with the technical expertise and statutory access needed to examine non-public incident data, independent investigative bodies where these exist, and academic researchers and civil society organizations who analyze documented incidents from public databases. Because the evidence required for meaningful analysis is often dispersed across multiple organizations and jurisdictions—as cases like the LAION-5B dataset contamination illustrate, where dataset curation, model training, hosting, and downstream harm each sat under different actors and legal regimes—effective investigation frequently demands cross-organizational and cross-jurisdictional cooperation.

The analytical insights generated at this stage can and should feed directly back into the pipeline's first component (Crisis Prevention & Preparedness): they are the raw material according to which SIPPs should be updated, tabletop exercise scenarios should be designed, and regulatory frameworks should be revised—closing the loop between reactive incident management and proactive, anticipatory governance.

6. Information-Sharing

Information-sharing refers to the continuous exchange of safety-relevant data and knowledge across the AI ecosystem to promote collective incident resilience and crisis preparedness. 

As the broadest procedural component outlined in this piece, information-sharing occurs continuously throughout the entire AI-IMCP pipeline and relies on contributions from every single stakeholder group involved in the global AI governance ecosystem. When stakeholders exchange information with one another, intelligence gathered in one organization or country can inform and protect another.

Because visibility is fragmented across the AI value chain, information-sharing can occur at all levels, covering a wide array of incident-relevant information between different actors. For example, established information-sharing regimes between frontier AI developers already exist in the form of the Frontier Model Forum (FMF). Information-sharing can also occur within the broader AI industry through hazard and flaw reporting platforms like FLARE-AI, the MITRE ATLAS Incident Sharing Initiative, the AI Vulnerability Database (AVID), or through secure, non-public channels such as industry-exclusive Information Sharing and Analysis Centers (ISACs). It can also occur between AISIs and frontier AI developers, downstream AI deployers, sectoral regulators, and regulatory authorities, and between AISI-equivalents, national security and intelligence agencies and other government authorities from different jurisdictions. Furthermore, incident-relevant information constantly flows between AI governance researchers and civil society through public incident databases such as the AIID, the OECD AIM, the MIT AI Incident Tracker, Arcola AI, and the AI Risk Explorer.

Conclusion

Adequate cross-organizational and cross-jurisdictional infrastructure needs to be built across each of the procedural components highlighted in this piece, and understanding the full AI-IMCP pipeline is a key prerequisite for that work. By breaking down incident management and crisis prevention into clear procedural components, we hope this piece can help decision-makers identify where new authorities, international standards, and capacity-building partnerships are needed, and what stakeholder groups they can coordinate or collaborate with to strengthen incident prevention, preparedness, and response efforts.

Improving AI incident management and crisis resilience across borders requires every stakeholder—from frontier AI companies to multilateral institutions—to recognize their specific role within this pipeline. Only by establishing this shared understanding can the international AI governance community advance the recommendations laid out in our report, and thereby better defend societies against AI harms.

- - - 

Omer Bilgin is an Analyst in International AI Governance at The Future Society. His work focuses on monitoring emerging AI hazards and real-world incidents and translating the insights they yield into targeted advocacy, primarily for governments, to promote stronger incident management and crisis prevention practices at both national and international levels. He holds a Master of Studies in Practical Ethics from the University of Oxford and a Bachelor of Arts in Philosophy from University College London.

Caio Vieira Machado is a Senior Associate in International AI Governance at The Future Society and is affiliated with Harvard University’s Berkman Klein Center. His work focuses on international AI governance, AI incidents and safety, platform regulation, algorithmic fairness, and disinformation. He holds doctorates from the University of Oxford and the University of São Paulo, as well as degrees from Oxford, Paris 1 Panthéon-Sorbonne, and USP.