Google discloses how they mistakenly erased $135 billion cloud customer account

[zohaibahd]

Facepalm: One of Google Cloud's worst nightmares came true in early May, when an embarrassing snafu completely erased a customer's account and data backups. The unlucky victim was Australian pension fund UniSuper, which manages a staggering $135 billion in assets for over 600,000 members. The pension fund was essentially frozen for two weeks, unable to fully operate while it scrambled to recover from third-party backups.

The incident started on May 2 when UniSuper suddenly lost access to all of its data and services hosted on Google Cloud, including backups. Soon after, a joint statement by the two companies admitted that an "inadvertent misconfiguration" resulted in the deletion, but details were slim. UniSuper was only able to come back online on May 15 after completing a full restoration.

This week, Google detailed exactly what went wrong. Someone at the company accidentally left a parameter blank while provisioning UniSuper's private cloud services using an internal tool. That seemingly small mistake had the catastrophic consequence of marking UniSuper's account for automatic deletion after a fixed term.

Google has put up a TL;DR on the matter:

"During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer's GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again."

Following the blunder, Google notes that the "customer and Google teams worked 24x7 over several days to recover the customer's GCVE Private Cloud, restore the network and security configurations, restore its applications, and recover data to restore full operations."

Google also admitted that no "customer notification" was triggered because this was an inadvertent deletion done through Google's internal tools. The whole incident must have come as a shock for UniSuper.

That said, there was conflicting information about whether UniSuper's backups stored in Google Cloud Storage were actually deleted or not, as Ars Technica points out. Initially, UniSuper claimed it had to rely on third-party backups because its Google backups were gone too. But Google's blog states the cloud backups were unaffected and "instrumental" in the restoration.

To their credit, Google has promised broad "remediation" steps to ensure this can never happen again. They've nuked the problematic internal tool and moved that functionality to customer-controlled interfaces. They've also scrubbed their databases and confirmed no other Google Cloud accounts are improperly configured for deletion.

The company reiterated that robust deletion safeguards exist, including soft deletes, advance notifications, and human approval checks.

It's certainly an alarming event for millions of cloud customers, but Google has emphasized this was an "isolated incident" affecting a single customer. They insist there are no systemic issues that put other Google Cloud customers at risk of spontaneous data vaporization.

Permalink to story:

Google discloses how they mistakenly erased $135 billion cloud customer account

 

[Dimitriid]

Hey good to know that google is at least consistently awful: From gmail to youtube to mega projects counted in the hundreds of billions, it's still an insistence of obtuse-to-the-point-of-madness interfaces and configurations that not even Google is fully aware of, all to just save a buck on having the smallest possible actual humans review anything at all manually: For Google and the whole of Alphabet it must be all automated all of the time and it doesn't matter how much potential problems they cause for any size customers by insisting on this ridiculous policy.

 

[veLa]

“Fired.”

 

[Homerlovesbeer]

“Fired.”

You'd hope so right?

 

[bclaymiles]

"They've also scrubbed their databases and confirmed no other Google Cloud accounts are improperly configured for deletion."

Typically when one "scrubs" a database it means that it is wiped clean - meaning all data deleted from it.

I think you meant to write "scanned".

 

[skittlebrau]

"They've also scrubbed their databases and confirmed no other Google Cloud accounts are improperly configured for deletion."

Typically when one "scrubs" a database it means that it is wiped clean - meaning all data deleted from it.

I think you meant to write "scanned".

Scrubbing in this context is correct though?

https://en.m.wikipedia.org/wiki/Data_scrubbing

 

[bclaymiles]

Scrubbing in this context is correct though?

https://en.m.wikipedia.org/wiki/Data_scrubbing

It's a stretch even in context. They didn't claim it was a failure due to an error in the data, but instead human error. Human errors can't be scrubbed. All they did was search their databases for the particular field in question, determined what dependencies there may have been, then eliminated or gave a default/null value to the field.

But you're right, 'scrub' is more generally used to describe checking the database for inconsistencies; for example in postgresql, it's called 'vacuuming'.

 

[sdms96825]

“Fired.”

probably "Promoted."

 

[nitebird]

Local and cloud for storage, nothing is foul proof but if you loose both at the same time you can rest easy it's the end of the world anyways.

 

[Che Cazzo]

"This week, Google detailed exactly what went wrong. Someone at the company accidentally left a parameter blank..."

"Someone at the company"? Who, the janitor??

You'd expect really smart and experienced professionals dealing with any backup account, but especially with such huge accounts!

People and companies should be reminded that "The Cloud" is just a fancy name for "someone else's server"...where bad things can also happen. Especially when "someone" there can be an ***** working on your backup data!.

It behooves large companies to have a second and even a third backup server!

 

[zamroni111]

Wasn't there any reminder alarm before that deletion schedule?

 

[Endymio]

You'd expect really smart and experienced professionals dealing with any backup account, but especially with such huge accounts!

You've been fooled by a clickbait headline. Unisuper may manage a large dollar amount of assets, but that doesn't translate into either a large company itself nor -- more importantly -- a large account with Google. I doubt their entire cloud database sees more than a few hundred transactions per hour -- trivial, compared some companies with only a million or two in assets.

 

[BadThad]

NEVER fully trust your data with "the cloud"!

 

[mbk34]

Mistakes do happen and at least they've done everything in their power to fix the issue and stop it from happening again. I don't think it was really the fault of the person making the change in the configuration but in the process that will delete an large customer's data without checking that that's the right thing to do.

 

[Grok451]

I'm sure AI will fix this. /s

 

[Gezzer]

"This week, Google detailed exactly what went wrong. Someone at the company accidentally left a parameter blank..."

"Someone at the company"? Who, the janitor??

You'd expect really smart and experienced professionals dealing with any backup account, but especially with such huge accounts!

People and companies should be reminded that "The Cloud" is just a fancy name for "someone else's server"...where bad things can also happen. Especially when "someone" there can be an ***** working on your backup data!.

It behooves large companies to have a second and even a third backup server!

But, but, the cost!?! We need to make shareholders as much as we possibly can so we have forgone spending extra on backup servers and competent employees. But rest assured we are in the process of implementing a new and more fool proof system. Of course to save money it will be AI based...

 

[Gezzer]

NEVER fully trust your data with "the cloud"!

Problem is cloud providers have been pimping their systems as fool proof and cost effective means to store and distribute data for years now. And all it takes is a bean counter believing the BS and eliminating everything but the cloud based storage to save money. I wouldn't be too surprised if this happens more often and the only reason it came to light was the size of the potential loss.