Businesses around the world hit by 'Blue Screen of Death' following faulty CrowdStrike update

[midian182]

What just happened? The dreaded Blue Screen of Death has been hitting Windows machines across the world as they boot up, impacting banks, airlines, media outlets, food chains, and many other businesses. It's been confirmed that the problem stems from security firm CrowdStrike and an issue with its Falcon Sensor agent. There's also been an apparent separate issue with Microsoft 365 apps and services.

Reports started arriving a few hours ago of PCs and servers going offline and getting stuck in recovery boot loops.

The impact has been on a scale reminiscent of the WannaCry ransomware attack in 2017: UK medical facilities have been unable to book in patients, airports across the globe are dealing with massive waiting times, broadcasters have been unable to air shows, some 911 emergency call centers have been impacted, and bank systems have gone offline.

We're obviously not on air - we're trying ð¤Â@SkyNews Breakfast pic.twitter.com/ZKvVacRgUY

– Jacquie Beltrao (@SkyJacquie) July 19, 2024

Britain's biggest train company has warned passengers to expect severe disruption, and the London Stock Exchange is facing technical issues.

CrowdStrike, used by many businesses for their PC and server security, wrote that it has received widespread reports of BSODs on Windows hosts, occurring on multiple sensor versions.

The company said it has identified the problem and reverted the update that caused the issue.

It's confirmed !! Crowdstrike Issue Guys, they are working on it, in about maybe 45 mins things will be fix #csagent #crowdstrike #BSOD pic.twitter.com/0mkfRbUAF8

– Xaaavier_8613 (@Xaaavier_8613) July 19, 2024

CrowdStrike wrote that the crashes are related to its Falcon Sensor, an agent that the company says "blocks attacks on your systems while capturing and recording activity as it happens to detect threats fast."

The CrowdStrike Reddit forum has posted a workaround that involves booting Windows into Safe Mode or the Windows Recovery Environment, navigating to the CrowdStrike directory, and deleting the "C-00000291*.sys" file before rebooting the machine.

It's worth noting that this solution isn't going to help everyone.

It certainly sounds like today is going to be a tough one for businesses. The outage arrived soon after Microsoft said users may be unable to access various Microsoft 365 apps and services due to a configuration change in a portion of its Azure backend workloads. This caused an interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services.

The US Federal Aviation Administration has just announced that all flights from Delta, United, and American Airlines have been grounded.

Brody Nisbet, CrowdStrike's chief threat hunter, wrote on X that the problem is a faulty channel file, so "not quite an update."

There is a fix of sorts so some devices in between BSODs should pick up the new channel file and remain stable.

P0 incident ongoing.

2/2

– Brody (@brody_n77) July 19, 2024

CrowdStrike's shares have fallen 20% in premarket trading following news of the incident. Microsoft is down 2.5%.

Permalink to story:

Businesses around the world hit by 'Blue Screen of Death' following faulty CrowdStrike update

 

[dangh]

"Brody Nisbet, CrowdStrike's chief threat hunter, wrote on X that the problem is a faulty channel file, so "not quite an update.""

lol. end result is the same however he is going to spin that up. Crazy this 'faulty file' has not been properly tested before the deployment, and this is both crowd strike and microsoft issue.
As well it shows pretty well that setting critical infrastructure on an autoupdate, and as well have no fall back architecture in place preferably on different OS, is asking for a trouble.

 

[brucek]

Holy smokes. For a second I thought this was a joke or exaggerated "windows update broke windows" post, but here's the headline from the Wall Street Journal: "A massive tech outage swept the globe, knocking out operations for banks, media companies and emergency services, and forcing airlines to ground flights."

Congrats to CrowdStrike on becoming a household name.

 

[SilverCider]

Holy smokes. For a second I thought this was a joke or exaggerated "windows update broke windows" post, but here's the headline from the Wall Street Journal: "A massive tech outage swept the globe, knocking out operations for banks, media companies and emergency services, and forcing airlines to ground flights."

Congrats to CrowdStrike on becoming a household name.

We may thank CrowdStrike in time as this will surely ramp up efforts to improve system resilience. It only goes to show how devastating a threat to a single system could be!

 

[themastergoose]

Working in IT, I've long advocated against putting "all eggs in one basket." Despite this, my warnings were dismissed as higher-up's pushed for moving all on-premises servers to the cloud to cut initial costs and the need for in-house management. It's a flawed strategy when your backups reside on the same cloud infrastructure that can go offline. But my expertise seems to fall on deaf ears until it's too late.

Although the issue at hand is not Microsoft's responsibility, depending solely on one company for your business's online operations is not advisable as you've introduced a single point of failure that you have no control over. Yet, time and again, accountants override the recommendations of their hired experts (me). Reap what you sow bean counters...

 

[stewi0001]

Working in IT, I've long advocated against putting "all eggs in one basket." Despite this, my warnings were dismissed as higher-up's pushed for moving all on-premises servers to the cloud to cut initial costs and the need for in-house management. It's a flawed strategy when your backups reside on the same cloud infrastructure that can go offline. But my expertise seems to fall on deaf ears until it's too late.

Although the issue at hand is not Microsoft's responsibility, depending solely on one company for your business's online operations is not advisable as you've introduced a single point of failure that you have no control over. Yet, time and again, accountants override the recommendations of their hired experts (me). Reap what you sow bean counters...

Preaching to the choir son. XD

 

[GodisanAtheist]

Oh Crowdstrike.

Take your bets: Goes out of Business or Too Big to Fail?

 

[themastergoose]

Oh Crowdstrike.

Take your bets: Goes out of Business or Too Big to Fail?

UK media is reporting they had 16% wiped off their share value. Works out to around 14 BILLION dollars for such a small percentage.

I'd say too big to fail unless that share price keeps dropping (which it isn't).

 

[Gars]

Friday update lol
they never learn

 

[Plutoisaplanet]

UK media is reporting they had 16% wiped off their share value. Works out to around 14 BILLION dollars for such a small percentage.

I'd say too big to fail unless that share price keeps dropping (which it isn't).

It would've been funny if it stopped dropping because because it disrupted trading on the NASDAQ, but it only did that to the London Exchange!

 

[zamroni111]

Working in IT, I've long advocated against putting "all eggs in one basket." Despite this, my warnings were dismissed as higher-up's pushed for moving all on-premises servers to the cloud to cut initial costs and the need for in-house management. It's a flawed strategy when your backups reside on the same cloud infrastructure that can go offline. But my expertise seems to fall on deaf ears until it's too late.

Although the issue at hand is not Microsoft's responsibility, depending solely on one company for your business's online operations is not advisable as you've introduced a single point of failure that you have no control over. Yet, time and again, accountants override the recommendations of their hired experts (me). Reap what you sow bean counters...

so whats your alternative solution to crowdstrike matter?
it's endpoint problem not cloud thing

 

[themastergoose]

so whats your alternative solution to crowdstrike matter?
it's endpoint problem not cloud thing

I won't delve into excessive detail due to its complexity.

The TL;DR: We use on-prem endpoint protections that we can fully control (they will not update unless greenlit for example). They work ultra simply by disabling the entire box if anything dodgy is detected, I get an alert (or my team does). We take a look, check it and act accordingly. No AI, and no "letting the machine decide". We have the final say and full control.

Longer explanation:
The primary defence mechanism is Sophos Intercept X, serving as an endpoint protection platform (EPP). It encompasses malware detection, exploit prevention, and ransomware protection. It operates on-premises from a cluster of security gateway servers positioned between the WAN and LAN sides of the server racks. These servers perform the bulk of the processing and are platform-independent. Sophos updates the gateway server software but not the endpoint servers themselves - We control this. Due to physical routing, bypassing the security gateways is not possible neither, so resources are over-provisioned for these servers, and the gateways cannot compromise a server's OS (though they can sandbox and block it from WAN connectivity if anomalies are detected).

However, relying solely on EPP is insufficient (as mentioned earlier regarding the single point of weakness). Additional measures include multiple VLAN's, firewall systems with advanced threat prevention capabilities, adherence to a zero-trust security model, and a rigorous testing and staged update policy for deploying software, drivers, services, or OS patches to production servers.

In essence, it's a synergistic combination of software and hardware solutions, each reinforcing the other, combined with established best practices for robust resilience. Not relying on cloud processing was entirely by design for production servers. It's all under my control.

 

[zamroni111]

I won't delve into excessive detail due to its complexity.

The TL;DR: We use on-prem endpoint protections that we can fully control (they will not update unless greenlit for example). They work ultra simply by disabling the entire box if anything dodgy is detected, I get an alert (or my team does). We take a look, check it and act accordingly. No AI, and no "letting the machine decide". We have the final say and full control.

Longer explanation:
The primary defence mechanism is Sophos Intercept X, serving as an endpoint protection platform (EPP). It encompasses malware detection, exploit prevention, and ransomware protection. It operates on-premises from a cluster of security gateway servers positioned between the WAN and LAN sides of the server racks. These servers perform the bulk of the processing and are platform-independent. Sophos updates the gateway server software but not the endpoint servers themselves - We control this. Due to physical routing, bypassing the security gateways is not possible neither, so resources are over-provisioned for these servers, and the gateways cannot compromise a server's OS (though they can sandbox and block it from WAN connectivity if anomalies are detected).

However, relying solely on EPP is insufficient (as mentioned earlier regarding the single point of weakness). Additional measures include multiple VLAN's, firewall systems with advanced threat prevention capabilities, adherence to a zero-trust security model, and a rigorous testing and staged update policy for deploying software, drivers, services, or OS patches to production servers.

In essence, it's a synergistic combination of software and hardware solutions, each reinforcing the other, combined with established best practices for robust resilience. Not relying on cloud processing was entirely by design for production servers. It's all under my control.

so your suggestion is using another epp that technically its driver level software can go wrong like crowdstrike?

I prefer to simply use free windows defender.
it regularly gives 4 signature updates per day and I set mine to check hourly.
it's free but no popping ads.
and I can set the engine update to 48 hours delay to prevent crowdstrike problem

Create a custom gradual rollout process for Microsoft Defender updates - Microsoft Defender for Endpoint

Learn how to use supported tools to create a custom gradual rollout process for updates

learn.microsoft.com