The societal and institutional value of incident data compounds when used to proactively prevent incident recurrence. Past incidents can be linked to present deployments where the associated risk is still actionable, before a foreseeable harm repeats itself. This post documents one approach to that forward-linking problem, developed by AIID sponsor Trustible, an AI governance platform. We share it here as a case study in applied use of the database, illustrating a workable method that can be used across the assurance ecosystem.

The Underlying Problem

Software vulnerability management assumes a fairly stable premise: a system is safe if there are no known vulnerabilities, and unsafe when a weakness could be exploited to impact availability or behavior. Cataloging efforts like the CVE work because the object of concern, a flaw in code, is discrete and exists at rest without requiring circumstances of harm to develop in the real world.

Non-security AI incidents don't fit that premise. A hiring model trained on biased historical data isn't malfunctioning; it's successfully reproducing the statistical patterns present in its training data. A generative model that produces a fabricated citation isn't in error by its own design objective, which is plausibility, not veracity. These systems are not defective "at rest" as a system may be vulnerable to an attacker at all times. They commission harm only when the system meets a use case to which it is unprepared to reliably process.

But how do we know when a system combined with its context risks reproducing past harms?

The Matching Problem

Linking incidents to deployed systems by tool or vendor name is the most obvious approach, and the least reliable. It fails in two directions. A widely used model will accumulate incidents spanning many unrelated applications, most of which are irrelevant to any given deployment. Conversely, a use case may share meaningful risk characteristics with an incident involving an entirely different underlying model. The most revealing unit of comparison is intended use, not tech stack, and neither incident reports nor use case documentation are written with that comparison in mind. Incident narratives describe failure, largely without describing the system's purpose. Use case documentation describes purpose, but embeds it in governance detail (data handling, controls, review status) that adds noise for this specific matching task. With general purpose AI, the use case is sometimes in the mind of a single human who often does not explain their intended use.

There's a further constraint particular to enterprise deployment. Organizations documenting their own AI use cases often need to keep significant portions of that documentation confidential, whether for competitive, legal, or security reasons. Any matching system built for this context has to work from what an organization is willing to disclose into its inventory, rather than assuming full visibility into system internals. That constraint shapes the design as much as the retrieval problem itself.

Method

In their 2023 SafeAI research paper, Nikiforos Pittaras and Sean McGregor introduced "A Taxonomic System for Failure Cause Analysis of Open Source AI Incidents," that included human-determined AI system goals. Generating these annotations was time consuming and speculative. As a small non-profit operating with highly-scaled incident data, we wanted methods that would not be gated on highly-skilled and time-intensive human processes.

Trustible developed an approach that addresses the matching problem at scale by machine generating standardized intended use summaries: for incidents, a plausible statement of purpose that could have been written prior to deployment; for companies cataloging use cases, a purpose statement stripped down to intended use, drawn from what the organization documents in its AI inventory. These summaries are then embedded and compared using a similarity metric, rather than lexical overlap.

This mirrors a known technique from retrieval-augmented generation, Hypothetical Document Embeddings, where a query is recast into the form of a plausible answer before retrieval, on the premise that answers resemble other answers more closely than they resemble the questions that prompted them. Applied here, the technique lets an incident involving one system (say, a code-completion tool producing unlicensed code) surface for a use case built on an entirely different system, provided the underlying intended use, real-time code suggestion within a developer's IDE, is sufficiently similar.

For a deeper technical walkthrough of this method, see Trustible's blog post.

Limitations and Broader Implications

This method, like the database itself, is bounded by what the database contains. Incident coverage skews toward events that attracted media or academic attention, so absence of a matching incident is not evidence of absence of risk. The method also introduces a dependency on generative AI to produce faithful intended-use summaries, which carries its own, separate risk of misrepresentation that any implementer should account for.

Even with those caveats, this case illustrates a pattern for others to follow: incident data becomes more useful when organizations build infrastructure around continuous matching, working within the confidentiality constraints real deployments require, rather than one-time manual review. That distinction, between an incident database as static reference and as a live input to risk assessment, may be the more consequential design question for the field going forward.

---

While Trustible is an AIID sponsor, they did not sponsor this post. The AIID does not accept paid blog articles. Please don't ask. But we do ask that you donate to sustain the AI Incident Database.

Anastassia Kornilova is the Director of Machine Learning and Founding Engineer at Trustible AI, a platform for Responsible AI Governance. In this role, she researches the benefits and risks of AI technologies and helps companies prepare for upcoming AI regulations. Previously, she has worked on large Machine Learning systems across sectors with a focus on Legal NLP.

Sean McGregor is the founder of the AI Incident Database and Executive Director of the Responsible AI Collaborative. He is a fellow at Harvard's Berkman Klein Center for Internet & Society, a member of the OECD Network of Experts on AI, and founding lead research engineer with the AI Verification and Evaluation Research Institute.

Daniel Atherton is lead editor and researcher with the AI Incident Database, where his work focuses on documenting real-world AI harms and improving how incident evidence can inform responsible decision-making. He also teaches through the Department of English at Georgetown University.