858TB gone forever: South Korea's data center fire shows the cost of no backups

midian182

Posts: 11,723   +177
Staff member
Facepalm: Plenty of people are guilty of not backing up their data, but one would expect a government facility to have numerous copies of its information. The National Information Resources Service (NIRS) in South Korea, however, recently lost 858TB of government data after a fire broke out in the building – and there were no backups.

On September 26, technicians were relocating lithium-ion batteries in the NIRS data center. Forty minutes later, the batteries exploded, causing a blazing inferno

The fire raged for hours. Thankfully, nobody was killed in the blaze, but it did destroy the server racks where G-Drive was stored.

G-Drive, which stands for Government drive rather than Google, stored data from around 125,000 federal workers – who were allowed 30GB each – and was core to 163 public-facing services, holding everything from import and export certifications to product safety checks, according to reports.

What's stunned South Korea is the revelation that the G-Drive was not backed up. An official told The Chosun that the drive could not be backed up due to its large size. About 858TB of data was lost. Ninety-five other systems in the fire were also destroyed, but they were backed up.

According to the National Information Resources Service, of the 647 systems at the Daejeon headquarters, 62% were backed up daily, but the remaining 38% were backed up once a month. Some systems were last backed up on August 31.

The incident affected government services, including tax filings and workers' emails. Restoring the services has been slower than expected, with less than 20% back online a week after the fire. Several government services are expected to be offline for up to a month.

While some of the G-Drive data has been recovered via partial backups and manual recreations, experts say a "valuable portion" of the data is gone forever.

While nobody died in the fire, there is a tragic element to the story. A 56-year-old recovery specialist, who was working at the NIRS backup site in nearby Sejong, reportedly jumped from a 15th-story window. His phone showed he had been working around the clock during the efforts, suggesting the unrelenting pressure and public scrutiny had become too much. South Korea's leadership expressed its condolences and called for improved working conditions for those dealing with the fire's aftermath.

Permalink to story:

 
I’m pretty sure there’s plenty of very affordable (what, a few thousand dollars a month if that for offsite cloud, tape would be even cheaper) cold-storage options available for something like this. It’s completely inexcusable.
 
It is most unfortunate that the data was not backed up securely off-site, and obviously due to inadequate funding for an IT department or contractor. Critical government services should always have the necessary infrastructure in place.

As for the 56 year-old recovery specialist losing his life - it's sad, tragic and disturbing that such events still happen as a cause and effect of a preventable circumstance.
 
They must have been severely underfunded, no way you wouldn't have such important systems directly replicated to another datacenter.

I feel especially bad for the recovery specialist, we all deal with stress differently, it's sad to see a life lost to something like this.

Edit: Hold up, I had to double take, this was South Korea?! Surely funding wasn't a problem? Surely?
 
"An official told The Chosun that the drive could not be backed up due to its large size"

Just means somebody didn't want to pay for it or put the work into it. 858TB is hardly too big to backup.
 
Nice clickbait!! "62% were backed up daily, but the remaining 38% were backed up once a month. Some systems were last backed up on August 31."

In other news:
"Yo, why no backup?? Are you stupendously crazy??"
"It's too big to backup....."
"Oh, OK then, carry on"
 
The persons responsible for the decisions to not have backups should be tried for the death of the recovery specialist...
 
A little bit disingenuous… it WAS backed up… but some of it was only backed up once a month.

Still inexcusable, but the headline is a bit misleading.
Not really it is a datacenter and the data stored could change significantly within a single month depending on the usage and what is being stored? Our team generates about 60TB of new simulation data a month and it would be a big deal if we lost that data.

Those are snapshots not backups.
 
Interesting. I learned about this fire, last week!

Whats different is that were I read this information, I was insisted that the fire was allowed to burn. Also, that most of the data lost was actually used by the US and its services.
 
Not really it is a datacenter and the data stored could change significantly within a single month depending on the usage and what is being stored? Our team generates about 60TB of new simulation data a month and it would be a big deal if we lost that data.

Those are snapshots not backups.
Of course - I did say it was inexcusable… but the headline implies everything wasn’t backed up… 858tb was just a fraction of their data…
 
The persons responsible for the decisions to not have backups should be tried for the death of the recovery specialist...

It is definitely tragic, but I don't see holding the person that made the decision of insufficient recovery mechanisms responsible for this person's death. I can't help but feel the poor deceased fellow had to have had additional substantial mental health issues.
 
Back