Google data center hit by four successive lightning strikes, some data permanently lost

[Shawn Knight]

There’s an old idiom that suggests lighting never strikes the same place twice. Apparently it’s one that Google never queried as one of its data centers in Belgium recently lost data following four successive lightning strikes courtesy of Mother Nature.

The incident in question took place last week, causing a brief loss of power to storage systems that host capacity for Google Compute Engine (GCE) instances for the region. The search giant said in an incident report that automatic auxiliary systems quickly restored power and although the systems were also on battery backup, some storage systems were attached to battery backups that had experienced extended or repeated battery drain.

Google said manual intervention was required to restore the systems to their normal state and unfortunately, a small amount of data was permanently lost.

The company said it takes full responsibility for the data loss and is using the incident to remind users that GCE instances and persistent disks within a zone exist in a single data center. As such, they are unavoidably vulnerable to data center-scale disasters such as this.

Google said it is also in the process of upgrading to storage hardware that is less susceptible to the power failure mode that triggered the incident.

The unfortunate episode serves as a reminder to back up your important data in an offline location. It’s also not a bad idea to unplug your computer from its power source during a thunderstorm or other dangerous weather event that could induce an electrical surge.

Image courtesy Wikipedia

Permalink to story.

https://www.techspot.com/news/61821-google-data-center-hit-four-successive-lightning-strikes.html

 

[Skidmarksdeluxe]

I'm guess most of those faithless meatheads who signed up for Ashley Madison's service wished that act of God had happened to their data centre/servers before their info was hacked and splashed all over the internet.

 

[stewi0001]

Hmmm I thought this is why lighting rods are put on buildings to divert lighting strikes from doing damage.

 

[mailpup]

Evidently Google doesn't back up their data like we ordinary mortals are supposed to.

Speaking of lighting rods, they disprove the old saying that lightning never strikes the same place twice. They get struck all the time as they are designed to.

 

[stewi0001]

Evidently Google doesn't back up their data like we ordinary mortals are supposed to.

Speaking of lighting rods, they disprove the old saying that lightning never strikes the same place twice. They get struck all the time as they are designed to.

path of least resistance

 

[fimbles]

Evidently Google doesn't back up their data like we ordinary mortals are supposed to.

Speaking of lighting rods, they disprove the old saying that lightning never strikes the same place twice. They get struck all the time as they are designed to.

Lightning rods are not designed to be struck directly or attract lightning, They are simply a way to divert a rogue strike to earth.

Its a very common myth that I only recently found out myself

 

[DelJo63]

Evidently Google doesn't back up their data like we ordinary mortals are supposed to.

No, they DID - - but to a local device in the same facility :sigh: Back in my mainframe days, we took backups to tape and had a rotation system where 1/n was send to offsite storage - - but that was tape (very portable). Today with 100s of GB hd storage, backups are hd->hd and not so portable.

There are products that perform backups which implement:

• active system => local bk ==> across town bk ===> out-of-area bk.

Think of hurricane Andrew which took a swath across the Florida peninsula taking out Holmstead, FA. Both the local bk ==> across town bk sites were destroyed and business continuation planning (as the subject is known) was only left with out-of-area data. It is obvious that DATA in and of itself was insufficient - - the whole data center had to be recreated or 'borrowed'.

 

[yRaz]

What everyone here seems to be missing here is that it wasn't the lighting strike that caused the damage, but that's thanks to the click bait article name

What did happen is that the lighting caused a power outage. In data centers they have a 2 stage power backup system. They have batteries that switch over during immediate powerless (usually these batteries have somewhere between 60-90 seconds of power in them). These batteries are meant to keep everything running while the generators have a chance to kick on and take over.

So what happened at this data center was that a small amount of the battery backups were faulty. This lead to data loss because a small number of systems couldn't access emergency power.

 

[stewi0001]

What everyone here seems to be missing here is that it wasn't the lighting strike that caused the damage, but that's thanks to the click bait article name

What did happen is that the lighting caused a power outage. In data centers they have a 2 stage power backup system. They have batteries that switch over during immediate powerless (usually these batteries have somewhere between 60-90 seconds of power in them). These batteries are meant to keep everything running while the generators have a chance to kick on and take over.

So what happened at this data center was that a small amount of the battery backups were faulty. This lead to data loss because a small number of systems couldn't access emergency power.

No, what everyone is missing is that Yahoo sent Sith Lords to zap Google! ;P

 

[GeforcerFX]

(Tin foil hat) The us government did it with HARRP, proving to all there data is never safe from the us government. (Tin foil hat off)

Well thats unfortunate, you can nver back up enough and spread the data around geographically if safely possible.

 

[IAMTHESTIG]

Guess they don't do offsite backups... Pretty sad considering how big Google is.

 

[tonylukac]

Lightening has truck our house on iroquois lane at least twice, both times shorting out a power switch to a tv or something where you cannot turn the device off. We then got a delta wye installed, for three phase power to power my ibm mainframe computer. Lightening has struck the building many times where I live now, setting off the fire alarm. Edison has installled some of the smart grid.

 

[tonylukac]

When lightening struck the building one time, There was like static charge on your leg clothing.

 

[DelJo63]

Lightening has truck our house on iroquois lane at least twice, both times shorting out a power switch to a tv or something where you cannot turn the device off.

To protect the big ticket items, I have local UPS supplies between the devices and the ac outlets. I use power strips to create enough sockets for the devices at each location - - like @ the HDtv, there's also the wifi router and Internet streaming console. When necessary, power of the UPS then isolates everything attached.

three phase power to power my ibm mainframe computer.

@ home ??? Which model and what does that cost you / month?

 

[tonylukac]

To protect the big ticket items, I have local UPS supplies between the devices and the ac outlets. I use power strips to create enough sockets for the devices at each location - - like @ the HDtv, there's also the wifi router and Internet streaming console. When necessary, power of the UPS then isolates everything attached.


@ home ??? Which model and what does that cost you / month?

I know you worked for ibm. I talked to you about that before. 370 158. Don't remember, but took three humdred amps at five cents a kilowatt hour. Don't know the calculations. Go rid of in in aoubt 1995 when the internet was born.