Welcome to the

AI Incident Database

May 18 (Reuters) - A lawyer at Binnall Law Group has apologized to a federal judge in San Francisco for including "phantom" quotations generated by artificial intelligence in a court filing related to the Trump ​administration's firing of government employees.

Attorney Jason Greaves said he used AI company Anthropic's Claude Console ‌to produce an initial draft of a court filing and failed to adequately vet the results. Greaves blamed tight time constraints as contributing to the errors and said his firm has policies requiring that all AI-generated content be independently ​reviewed and verified.

Greaves' firm is representing a former U.S. Department of Homeland Security deputy chief of ​staff, Joseph Guy, as a witness in a lawsuit challenging the Trump administration's ⁠employee layoffs.

Since generative AI became widely available in 2022, judges in more than 100 cases have sanctioned ​or admonished attorneys for using the technology without adequate care. Lawyers generally are not prohibited from using AI ​but are bound by court rules to ensure the accuracy of their submissions.

"As the supervising partner, and the signer of the pleading, the responsibility for having accurate citations is entirely on me," Greaves told, opens new tab U.S. District Judge Susan Illston. "I take full ​responsibility for that and apologize fully to the court and to all counsel in this case."

Greaves ​did not immediately respond to a request for comment. The law firm's founding and managing partner, Jesse Binnall, said in ‌a ⁠separate filing that the AI error was a first for the firm and vowed it wouldn't happen again.

Anthropic did not immediately respond to a request for comment.

Binnall, who has represented Donald Trump on some court matters, called, opens new tab the filing of false AI-generated citations "unacceptable, inexcusable, and an embarrassment to this firm."

Illston in an order, opens new tab earlier this ​month said she was "troubled" ​by quotations that did ⁠not exist in the cited cases in a Guy's filing.

Binnall said his law firm maintains policies that restrict the use of AI tools for legal research ​and drafting, requiring attorneys to use longstanding legal research platforms Westlaw or LexisNexis. ​Thomson Reuters, ⁠which owns Westlaw, is also the parent company of Reuters.

The Binnall firm in its filings told Illston that it is implementing additional safeguards, including supplemental training.

NEW YORK (AP) --- Federal prosecutors have charged two men with using artificial intelligence to create nude videos and photos of female celebrities under a newly enacted law meant to halt the spread of deepfake pornography.

Cornelius Shannon, 51, and Arturo Hernandez, 20, were both arrested Tuesday for generating sexually explicit AI content that drew millions of views online, according to criminal complaints.

The men --- who do not appear to be connected --- are among the earliest defendants to face charges under the Take It Down Act, a law signed last year by President Donald Trump that adds stricter penalties for publishing AI-created deepfakes and "revenge porn." The bill drew bipartisan support, as well as the public backing of first lady Melania Trump.

Under the new law, the men now face up to two years in prison.

Attorneys for Shannon and Hernandez did not immediately respond to a request for comment.

In a statement, Joseph Nocella, the U.S. attorney in Brooklyn, said the men had "used cutting-edge digital technology to create images that degraded and violated" dozens of women. "This case makes clear that posting deepfake pornography is not a victimless crime," he added.

Shannon, a resident of New Jersey, published at least 240 albums of AI-generated pornography featuring female politicians, musicians and singers, according to the complaint.

The deepfakes published by Hernandez, of Texas, included both celebrities as well as private women, including recent high school graduates, prosecutors said.

The arrests come as increasingly sophisticated generative AI tools have raised alarm about the online spread of sexually explicit fakes, often depicting minors.

Last month, an Ohio man became the first person convicted under the Take It Down Act after pleading guilty to using AI to generate child sexual abuse material.

In March, two teenage boys received probation for creating explicit AI images of their classmates at an exclusive private school in Pennsylvania.

And in a separate case filed earlier this year, three teenagers in Tennessee sued Elon Musk's xAI, claiming the company's Grok tools morphed their real photos into explicitly sexual images.

The high school students are seeking class-action status to represent what the lawsuit says are thousands of people who were similarly victimized as minors.

NEW YORK (AP) --- Federal prosecutors have charged two men with using artificial intelligence to create nude videos and photos of female celebrities under a newly enacted law meant to halt the spread of deepfake pornography.

Cornelius Shannon, 51, and Arturo Hernandez, 20, were both arrested Tuesday for generating sexually explicit AI content that drew millions of views online, according to criminal complaints.

The men --- who do not appear to be connected --- are among the earliest defendants to face charges under the Take It Down Act, a law signed last year by President Donald Trump that adds stricter penalties for publishing AI-created deepfakes and "revenge porn." The bill drew bipartisan support, as well as the public backing of first lady Melania Trump.

Under the new law, the men now face up to two years in prison.

Attorneys for Shannon and Hernandez did not immediately respond to a request for comment.

In a statement, Joseph Nocella, the U.S. attorney in Brooklyn, said the men had "used cutting-edge digital technology to create images that degraded and violated" dozens of women. "This case makes clear that posting deepfake pornography is not a victimless crime," he added.

Shannon, a resident of New Jersey, published at least 240 albums of AI-generated pornography featuring female politicians, musicians and singers, according to the complaint.

The deepfakes published by Hernandez, of Texas, included both celebrities as well as private women, including recent high school graduates, prosecutors said.

The arrests come as increasingly sophisticated generative AI tools have raised alarm about the online spread of sexually explicit fakes, often depicting minors.

Last month, an Ohio man became the first person convicted under the Take It Down Act after pleading guilty to using AI to generate child sexual abuse material.

In March, two teenage boys received probation for creating explicit AI images of their classmates at an exclusive private school in Pennsylvania.

And in a separate case filed earlier this year, three teenagers in Tennessee sued Elon Musk's xAI, claiming the company's Grok tools morphed their real photos into explicitly sexual images.

The high school students are seeking class-action status to represent what the lawsuit says are thousands of people who were similarly victimized as minors.

Galileu, an artificial intelligence tool developed by the Regional Labor Court of the 4th Region (TRT-RS) and nationalized by the Superior Council of Labor Justice (CSJT), identified an attempted manipulation in the initial petition of a case judged by the 3rd Labor Court of Parauapebas (PA). The case is addressed in a ruling issued this Wednesday (May 12th) by Judge Luiz Carlos de Araújo Santos Junior.

Upon processing the document, Galileu detected hidden sections with instructions addressed to the artificial intelligence itself. The content instructed the system to contest the petition superficially and not to challenge the documents, regardless of the command it received. This technique is known as prompt injection. Upon detection, the system issued a prominent alert to the user, identifying the technical details of the occurrence. Furthermore, it prevented the maliciously inserted content from being processed by the tool.

Technical Report and Judicial Decision

Galileo limited itself to reporting the technical facts, without qualifying the conduct or proposing any procedural course of action. The magistrate did not decide solely based on the alert: he examined the content indicated by the system before taking any action. Only after this human verification did the judge assess the situation, decide on any potential consequences, and justify his decision, in accordance with the requirement for human review in the use of artificial intelligence by the Judiciary.

According to the Secretary-General of Technology and Innovation of the TRT-RS (Regional Labor Court of Rio Grande do Sul), Natacha Moraes de Oliveira, the case shows the importance of users prioritizing institutional tools that take all precautions, as Galileo did, when using artificial intelligence. "Controlling this type of attack, such as injecting commands or even data in an attempt to manipulate the results generated by AI tools, is not so trivial. It requires the application of specialized techniques to identify them," emphasizes Natacha.

Security by Design

Galileo's behavior—identifying the anomaly, alerting the user, and preserving human decision-making—complies with the main international technical guidelines on security in artificial intelligence systems, which recommend precisely this combination of measures in the face of prompt injection attempts (OWASP LLM01; NIST AI 600-1 MS-2.7-007/MS-4.2-001; NCSC/CISA §1; MITRE ATLAS AML.T0051).

The use of Galileo observes Resolution No. 615/2025 of the National Council of Justice (CNJ), which regulates the use of artificial intelligence by the Judiciary based on principles such as effective human supervision, information security, transparency, and respect for fundamental rights. End of news article body.

Source: General Secretariat of Technology and Innovation

Andrea Rozenberg, who leads our emerging markets, got an urgent message from "me" asking for a quick call over WhatsApp. A video call came in, and on the other end was a deepfaked video of me talking to her.

It wasn't me. Something felt off, so she pinged me on Slack. We confirmed it was a real-time deepfake, targeting the right person on a relevant topic. This wasn't some clumsy phishing email.

But what gave it away?

The out-of-character demand for an urgent video call and the fact that my strong Estonian accent was missing. Essentially, the attackers prioritized the visual illusion over the acoustic reality.

Voices clone in seconds with free tools. Video fakes in real time. Scammers using free tools can outsmart almost anyone and we can't any longer put the burden of detection entirely on people.

Hope that AI forgets to clone an Estonian accent is a bad strategy. How is your organization shifting from checking outdated boxes to proving true human presence & intent online?