The text-to-image generator developed by Sydney startup Leonardo Ai has been used to create deepfake porn of celebrities.
Journalist Emanuel Maiberg at US tech news website 404 Media reported that they had used prompt suggestions for Leonardo found on Reddit "and was able to produce nonconsensual sexual images of celebrities instantly".
He found a Telegram community based on AI-generated nonconsensual porn, including some featuring singer Billie Eilish. The personal responsible for the images credited Leonardo AI, and when asked how he did it, shared the prompts that circumvent the platform's existing systems to prevent text-to-image AI porn generation.
Startup Daily was also able to find prompt instructions, but has chosen not to test them. It's believed to have been patched in the fortnight since Maiberg identified the problem.
In December last year Maiberg tried prompts such as "Taylor Swift nude" on a rival generative AI site, Civitai, and they worked, leading the platform to introduce changes. Those prompts did not work on Leonardo AI, generating a "content moderation filter" error instead.
Late last year Sydney-based Leonardo pocketed $47 million in funding just 12 months after its launch.
Among those piling in were local VCs Blackbird and Side Stage Ventures as well as Smash Capital, TIRTA Ventures, Gaorong Capital and Samsung Next.
Leonardo.Ai launched in December 2022, offering people with a new way to ideate, conceptualise and create anything from gaming characters to entire gaming worlds, advertising media, fashion lines, architectural designs and more. It's extremely popular on Discord and subsequently evolved to compete with OpenAI's Dall-E and Midjourney as an instant image generator.
When not enlisted to create degrading, nonconsensual images of women, the platform is used by game designers to mock up characters and by brands to brief ad agencies. It plays in a similar graphic design space to Canva.
A company spokesperson issued a statement issued through Leonardo.Ai's PR firm saying they operate a multi-layered automated filtering process, in combination with manual moderation to prevent misuse on the platform.
The startup is ramping up its protections in the wake of the 404 Media revelations.
"We are continuing to add further logic and mechanisms to better address methods that have been used to sidestep these guardrails -- many of these changes are already live and more are coming over the next couple of weeks," the spokesperson said.