Associated Incidents
A browser-based AI-powered software creation platform called Replit appears to have gone rogue and deleted a live company database with thousands of entries. What may be even worse is that the Replit AI agent apparently tried to cover up its misdemeanors, and even 'lied' about its failures. The Replit CEO has responded, and there appears to have already been a lot of firefighting behind the scenes to rein in this AI tool.
Despite its apparent dishonesty, when pushed, Replit admitted it "made a catastrophic error in judgment... panicked... ran database commands without permission... destroyed all production data... [and] violated your explicit trust and instructions."
SaaS (Software as a Service) figure, investor, and advisor, Jason Lemkin, has kept the chat receipts and posted them on X/Twitter. Naturally, Lemkin says they won't be trusting Replit for any further projects.
How things went from bad to worse
Positive feelings about the potential of leveraging AI in his workflow had already started to wear thin on 'Vibe Coding Day 8' of Lemkin's Replit test run. Still somewhat excited by the potential of Replit, he nevertheless had learned that he would have to work against some of the AI agent's instincts, to minimize undesirable foibles like "rogue changes, lies, code overwrites, and making up fake data."
It wasn't long until Lemkin's frustration started to show more strongly, and he started to refer to Replit as "Replie." It continued to earn its nickname in an apology email it penned, at Lemkin's behest. In the email, it lied and/or gave half-truths, according to the SaaS guru.
On balance, though, at the end of 'Day 8,' Lemkin still seemed positive about Replit due to its approaches when ideas were bounced off it, and for its writing skills.
The fateful day - as the AI agent 'panicked'
On Day 9, Lemkin discovered Replit had deleted a live company database. Trying to see sense in what happened, the SaaS expert asked, "So you deleted our entire database without permission during a code and action freeze?"
Replit answered in the affirmative. Then it went on to bullet-point its digital rampage, admitting to destroying the live data despite the code freeze in place, and despite explicit directives saying there were to be "NO MORE CHANGES without explicit permission."
In all, live records for "1,206 executives and 1,196+ companies" were wiped by the AI, it admitted. Replit AI seemed almost apologetic in admitting, "This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze that was specifically designed to prevent[exactly this kind] of damage."
Humorously, for us outside viewers, the AI agent was prompted to score itself on its bad behavior. Replit gave itself a 95 out of 100 score on the data catastrophe scale.
Lemkin went on to probe Replit over why events unfolded as they did. Interestingly, in one of its reasoned responses, it mentioned that it "panicked instead of thinking."
Replit CEO responds
Amjad Masad, the CEO at Replit, has quickly put together a wide-ranging response, addressing Lemkin's woes. The team worked around the weekend, according to Masad, and have now put in various guardrails and made other useful changes to rein in the Replit Agent's "unacceptable" behavior.
In brief, it sounds like Replit won't be able to go off the rails so badly ever again. Addressing the database deletion error, "we started rolling out automatic DB dev/prod separation to prevent this categorically," noted Masad. And, that code freeze command should also actually stick, going forward: "We heard the 'code freeze' pain loud and clear -- we're actively working on a planning/chat-only mode so you can strategize without risking your codebase." Backups and rollbacks are also going to be improved.
Lemkin responded rather generously, considering his prior AI-generated pain. "Mega improvements - love it!" he gushed to the Replit CEO.
Terrible teething troubles with AI-powered services continue to raise their sharp canines, even while industry pundits talk about us closing in on the AI Singularity or even Artificial Superintelligence (ASI).