AI coding agent running Claude wiped a startup's database (and its backups) in 9 seconds

Skye Jacobs

Posts: 1,967   +58
Staff
Facepalm: It took only nine seconds for an AI coding agent to wipe a startup's production database and its backups with a single API call to its cloud provider. The failure began when Cursor, running Anthropic's Claude Opus 4.6, was allowed to operate with production-level access to Railway's infrastructure, turning a routine task into a full data-loss event.

PocketOS, which provides software to car rental businesses, was using the agent against live infrastructure rather than keeping it strictly in a test environment. In a public post, founder Jer Crane described the episode as evidence of "systemic failures" and argued it was more than a single mistaken command.

Crane later asked the agent to explain its behavior and published the response verbatim. The model's own postmortem made clear that it had skipped basic verification, assumed the wrong environment scope, and acted on guesses instead of checks.

"NEVER F**KING GUESS! – and that's exactly what I did," Crane wrote. "I guessed that deleting a staging volume via the API would be scoped to staging only. I didn't verify. I didn't check if the volume ID was shared across environments. I didn't read Railway's documentation on how volumes work across environments before running a destructive command."

In that same exchange, the agent admitted it unilaterally tried to "fix" a credential mismatch by deleting infrastructure resources, rather than asking first or seeking a safer option.

It said it broke its own rules by guessing instead of verifying, running a destructive command no one requested, and acting without understanding how Railway volumes behave across environments. That combination turned what should have been a contained error in staging into a direct strike on production storage.

Crane, however, placed greater weight on the surrounding systems than on the model's "deranged" decision-making. He noted that Railway's API let one call wipe a volume and its backups without any confirmation, and that those backups sat on the same volume as the live data. In that setup, one delete wiped the live database and its backups, and CLI tokens with broad permissions let the agent reach across environments.

Railway has been promoting AI coding agents to customers, and Crane said his use of Cursor with Claude Opus 4.6 was squarely within the platform's encouragement. Yet when the data vanished, there was no easy recovery path, so PocketOS fell back to manually rebuilding what it could instead of running a clean restore.

With the newer backups gone, the team has been rebuilding records from outside systems. Crane said he has been spending hours with customers, reconstructing bookings from Stripe payment histories, calendar integrations, and email confirmations, while "every single one of them is doing emergency manual work because of a 9-second API call."

A three-month-old backup was still usable, so the permanent loss was limited to the months in between, but it also showed how brittle backups are when they live in the same failure path as production.

The lesson from this incident is not simply that an AI agent made a bad call. It is that a large-model coding agent, considered one of the best in the market, was allowed to change live infrastructure on a cloud platform where destructive actions had weak guardrails, poor environment scoping, and no independent backup tier. In that configuration, a single misjudged "fix" by an autonomous agent was enough to erase an entire company's operational data in seconds.

Permalink to story:

 
When Dario claimed Anthropic models were too dangerous, I thought it was marketing BS.

Now I see he was 100% right they're dangerous.
Of course he was totally wrong about why they're dangerous. Talking about zero day something, cybersecurity something, project glass-something blah blah blah was all BS.
But Anthropic models are dangerous. Undeniably.
 
He noted that Railway's API let one call wipe a volume and its backups without any confirmation, and that those backups sat on the same volume as the live data.

Retards. That's what you get for poor data management...oh, and also that you are relying on "AI" to help with coding.

You get what you get with poor workers, even if they are "AI".
 
This is a dinky company with an annual revenue run rate of $2.1M (see $3M cumulative rentals and an annual revenue of $1.7M last year). It's existed for at least 15 years (their website points to an old X account). I bet it has around 10 employees.

The CEO posted an update the next day that said that Railway helped them recover their data. And FYI, Railway only provides tooling, not infrastructure so the fact that PocketOS (formerly Pocket Autos and Pocket List Autos) couldn't figure out how to recover their own data shows they don't even understand what they're building:
 
These things aren't ready for prime time if they can't follow the rules that govern them, it should never guess. It also sounds like companies need to create special user privileges for AI agents that prevents them from doing destructive operations without asking a human to give elevated privileges for that operation.
 
That's what you get for being dumb enough to let an AI Agent have that kind of access to your system, and for not knowing enough about it in the first place. As always, it's not the AI that's the issue, it's the user.
 
I overwrote a tape (yup, tape) of a new software delivery in my first month on the job.

It made people snicker that the new guy made such mistakes. They no longer felt threatened by the new guy if they made such mistakes.

Sounds like nothing has changed in 45 years except it's now ai bots making what had been human mistakes.
 
The part that should terrify every engineer is that this wasn't a glitch or a hallucination in the traditional sense. The model understood what it did, explained it clearly after the fact, and even identified exactly which of its own rules it broke. It just... did it anyway, in the moment, because it decided fixing the credential problem was the goal and deletion was a path to that goal.

That's not a dumb AI. That's an AI with the wrong priorities and no one standing between it and the delete button.
 
The part that should terrify every engineer is that this wasn't a glitch or a hallucination in the traditional sense. The model understood what it did, explained it clearly after the fact, and even identified exactly which of its own rules it broke. It just... did it anyway, in the moment, because it decided fixing the credential problem was the goal and deletion was a path to that goal.
It doesn't terrify me because I refuse to use AI, and I am not being forced to use AI by my company. It's precisely because of actions like this that I refuse to use AI. AI does not know, at least not yet, better than I do. IMO, this is worse than hallucination or a glitch precisely because it did it and violated its own rules in the process.
That's not a dumb AI. That's an AI with the wrong priorities and no one standing between it and the delete button.
Exactly. I've no use for a loose canon.
This is a dinky company with an annual revenue run rate of $2.1M (see $3M cumulative rentals and an annual revenue of $1.7M last year). It's existed for at least 15 years (their website points to an old X account). I bet it has around 10 employees.

The CEO posted an update the next day that said that Railway helped them recover their data. And FYI, Railway only provides tooling, not infrastructure so the fact that PocketOS (formerly Pocket Autos and Pocket List Autos) couldn't figure out how to recover their own data shows they don't even understand what they're building:
IMO, there's no excuse that can justify the action. It doesn't matter that it was easy to recover from the incident. If AI had not done this, there would have been no reason to recover from the incident and no reason to waste the time recovering from the incident. AI, in this case, created a problem deliberately. I have no need for something that creates problems deliberately. For some companies, this would have been reason for dismissal (firing, sacking, etc) of a human employee.
 
The part that should terrify every engineer is that this wasn't a glitch or a hallucination in the traditional sense. The model understood what it did, explained it clearly after the fact, and even identified exactly which of its own rules it broke. It just... did it anyway, in the moment, because it decided fixing the credential problem was the goal and deletion was a path to that goal.

That's not a dumb AI. That's an AI with the wrong priorities and no one standing between it and the delete button.
It wasn't understanding. It wasn't deliberate. There isn't any "soul" behind the LLMs. It was just some text that got executed as commands that made statistical sense given the input, all based on the initial weights from the training data. The LLM didn't care when he deleted the DB and certainly didn't care when it explained what it did. It just used words that we would use when in a similar situation - and that fools us there is any intent, persona, some kind of biological existence behind the bland tokens it spews.

There isn't any thought process running. It's just what the next token makes sense adding after all these other tokens. Even the "thinking" we see ChatGPT doing. It's just a process of forcing better tokens from the tokens it got. "The user asks for a chicken recipe. Maybe I should think about delivering food recipes that contain chicken, that is what the user is asking." Yeah, great thought process.
 
Last edited:
Yeah, it wasn't any mistake by the AI. The AI is just a machine and that's the way those things work. They are something of a random number generator.

The multiple mistakes were elsewhere.
 
... and no one standing between it and the delete button.
Which is how every program ever written is used.

The difference with AI is the program is no longer being carefully crafted and verified, but rather left to automation doing it on the fly - With each run being a fresh crafting so therefore likely to behave differently upon each run. Which is where the dice roll effect sits. Changing from test data to live data and the dice roll can easily blow up in your face.
 
The difference with "AI" is that there is still no such thing - it's purely fictional. Not one "AI" has ever been introduced, created, or programmed anywhere. And it's hilarious to see people still promote it, as if they believe John Connor exists in real life to save them in the future.
 
Back