Associated Incidents
Deloitte Australia will issue a partial refund to the federal government after admitting that artificial intelligence had been used in the creation of a $440,000 report littered with errors including three nonexistent academic references and a made-up quote from a Federal Court judgement.
A new version of the report for the Department of Workplace Relations (DEWR) was quietly uploaded to the department's website on Friday, ahead of a long weekend across much of Australia. It features more than a dozen deletions of nonexistent references and footnotes, a rewritten reference list, and corrections to multiple typographic errors.
The first version of the report, about the IT system used to automate penalties in the welfare system such as pauses on the dole, was published in July. Less than a month later, Deloitte was forced to investigate the report after University of Sydney academic Dr Christopher Rudge highlighted multiple errors in the document.
At the time, Rudge speculated that the errors may have been caused by what is known as "hallucinations" by generative AI. This is where the technology responds to user queries by inventing references and quotes. Deloitte declined to comment.
The incident is embarrassing for Deloitte as it earns a growing part of its $US70.5 billion ($107 billion) in annual global revenue by providing advice and training clients and executives about AI. The firm also boasts about its widespread use of the technology within its global operations, while emphasising the need to always have humans review any output of AI.
The submission of the mistake-laden report also comes amid widening concern about the unchecked use of AI in the workplace as companies look for ways to become more efficient.
'Generative AI used'
The new version of the report includes a concession in the methodology that generative AI was used for what the firm called "traceability and documentation gaps".
The report now notes the firm used "a generative AI large language model (Azure OpenAI GPT-4o) based tool chain licensed by DEWR and hosted on DEWR's Azure tenancy".
That revelation, along with the nature of the now deleted errors, has led Rudge to conclude the problems in the initial report were due to AI hallucinations.
"This is no longer a 'strong hypothesis'," Rudge said. "Deloitte has now issued a confession, albeit buried in the methodology section. Deloitte has admitted to using generative AI for a core analytical task; but it failed to disclose this in the first place."
The academic said the recommendations of the report could no longer be trusted because "the core analysis was done by an AI".
"You cannot trust the recommendations when the very foundation of the report is built on a flawed, originally undisclosed, and non-expert methodology," he said.
'Substance of review retained'
A DEWR spokesman said "the substance of the independent review is retained, and there are no changes to the recommendations", but he declined to answer if the department believed the errors were due to the use of AI by Deloitte personnel.
The DEWR spokesman said Deloitte had "agreed to repay the final instalment under its contract", but declined to reveal the amount. He also declined to comment when asked if DEWR would use Deloitte for other work or would pursue a complete refund of the fee.
A spokesman for the responsible minister, Amanda Rishworth, referred questions to DEWR. Deloitte Australia has entered into contracts worth almost $25 million with DEWR since 2021.
After the potential problems in the report were highlighted, Deloitte carried out an internal review of the DEWR report, a source not authorised to comment on the firm's internal operations, said. That investigation found problems were linked to human error. The Deloitte review did not make any conclusions about whether the errors related to the incorrect use of AI tools.
Both versions of the Deloitte report found widespread problems with DEWR's computer system and processes, including "poor documentation", undetected "system defects" and "a punitive compliance model has amplified the impact of system errors on participants". This report follows a damning Commonwealth ombudsman report that hundreds of welfare payment suspensions were not lawful.
Deleted references, footnotes
The revised report has deleted a dozen references to two nonexistent reports by Professor Lisa Burton Crawford, a law professor at the University of Sydney, that were included in the first version. Two references to a nonexistent report by Professor Björn Regnell, of Lund University in Sweden, were also deleted in the new report.
Also deleted was a made up reference to a court decision in a leading robo-debt case, Deanna Amato v Commonwealth.
The new report has also deleted a reference to "Justice Davis" (a misspelling of Justice Jennifer Davies) and the made-up quote from the nonexistent paragraphs 25 and 26 in the judgement: "The burden rests on the decision-maker to be satisfied on the evidence that the debt is owed. A person's statutory entitlements cannot lawfully be reduced based on an assumption unsupported by evidence."