Microsoft 365 outage led to chlorine gas leak at German hotel, poisoning of 15 guests and sending four to hospital

midian182

Posts: 12,069   +184
Staff member
WTF?! The recent global outage of Microsoft 365 services affected millions across the world, but the impact on people varied enormously. Take, for example, the 15 guests at a hotel in Germany who were exposed to a chlorine gas leak as a result of the debacle. That sounds a lot worse than not being able to use Teams.

According to regional publication WDR (via Notebookcheck), the incident happened at the Hotel Ling Bao, part of Phantasialand theme park in Germany. It experienced a malfunction with its dosing system, which pumped too much chlorine into the pool.

The facility uses a maintenance safety system in its sauna area that sends emails warning when chlorine levels are too high. But because of the Microsoft 365 service disruption, the emails never arrived.

Forty first responders were dispatched to the incident. Fifteen people required triage due to respiratory irritation, four were taken to the hospital, and one had to remain under observation due to an injury.

The disruption began on November 12, when users around the world started reporting problems with Outlook, Word, Excel, PowerPoint, Teams, and Copilot. Downdetector received nearly 150,000 complaints, while Microsoft said its servers were experiencing unusually heavy loads. Outlook users couldn't send emails or access their inboxes, and some servers went offline again after being restored.

In Japan, businesses and government agencies were forced to dig out fax machines and landline phones when cloud services stopped working. The country's shared government IT infrastructure was also disrupted. Xbox users struggled to sign in, launch digital games, and synchronize cloud saves.

Microsoft has now revealed that the outage was caused by a software defect introduced in July, which went undetected for more than three months. A surge in server traffic, combined with reduced network capacity, triggered the bug and caused more than 80% of the affected gateway servers to become unresponsive within two hours. The company admitted that its testing had failed to catch the problem.

While software failures can cause inconvenience and financial losses, there are often more serious consequences like this one. In July 2024, a faulty CrowdStrike update caused Windows systems worldwide to crash. Hospitals couldn't book patients, some 911 emergency call centers were affected, and airlines grounded flights.

In one of the strongest examples of software failures destroying lives, bugs in the UK's Horizon accounting system contributed to wrongful convictions of hundreds of Post Office workers. Some were imprisoned, many suffered financial ruin, and the scandal was linked to at least four suicides.

In 2018, Wells Fargo revealed that a software error had incorrectly denied mortgage modifications to around 625 customers who should have qualified for them. The mistake had devastating consequences, with approximately 400 borrowers losing their homes through foreclosure.

Permalink to story:

 
Safety warning emails without a backup or secondary notification (e.g., a buzzer where the danger is) is simply a badly designed system.
Yeah, and if it was smart enough to send an alert, it should have been smart enough to recognize no response and shut off the chlorine (unless a mechanical fault prevented it from shutting it off). It was just a pool after all, not critical infra.
 
Safety warning emails without a backup or secondary notification (e.g., a buzzer where the danger is) is simply a badly designed system.
Indeed this is an atrocious system. What if someone doesn't read the email that comes through? You're telling me that for decades the lives of guests were in danger and only through blind luck did nobody miss the email this whole time?

Whomever designed that system deserves attempted murder charges.
 
Clickbait headline. The failed notification system did not "lead to a chlorine gas leak." It failed to transmit a warning about a leak that happened for other reasons. There's a difference.

And while its convenient for the hotel to try to blame Microsoft for this, it's hard to see how an email to what is likely a busy hotel management inbox was ever suitable for real-time life safety response in the first place.
 
My feeling is they are guilty of something and will point fingers to MS. In general, and I'm just saying and not specfically, there are many companies that don't spend a lot of money on maintenance/preventive maintenance because of cost. But they will spend the money if it's a regulatory mandate. The question is, do they keep up with the preventive maintenance of that mandate?

Regulatory Agencies will inspect their books, documents, systems, everything. Probably will be fined and advised too. The clincher is, inspectors can now visit them unannounced anytime besides their regular inspections. But I don't know what type of laws are in Germany.
 
Back