Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

Did microsoft actually say restart 15 times to fix it

Cromax said:

Did microsoft actually say restart 15 times to fix it

Click to expand...

yes, under certain race conditions, that seemed to be working for people to back on their systems.

In a nutshell, boxes with the bad channel update need to boot into recovery mode, delele c:\windows\system32\drivers\crowdstrike\c-00000291*.sys, and then reboot again.

CrowdStrike issue impacting Windows endpoints causing an error message on a blue screen
Microsoft has identified an issue impacting Windows endpoints that are running the CrowdStrike Falcon agent. These endpoints may encounter an error message on a blue screen and experience a continual restarting state.

We have received reports of successful recovery from some customers attempting multiple restart operations on affected Windows endpoints.

To mitigate this issue, follow these steps:

1. Start Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Restart the device.
5. Recovery of systems requires a Bitlocker key in some cases.

For Windows Virtual Machines running on Azure follow the mitigation steps in Azure status

Additional details from CrowdStrike are available here: Statement on Windows Sensor Update - CrowdStrike Blog

Good thing I checked first before posting. So umm... TPM2 still a "good" idea?

@pparks1 Like I said, I'm a simple thinker and know maintaining such systems is waaay beyond my pay grade. But I wasn't suggesting redundancy for non-essential corporations. My observations were from a standpoint for such outages in instances like 911, hospitals, and the like where people's lives are at risk. One would think emergency measures would be in place for them.

Since it appears you do have experience in IT, I'd like a ask you a IT question so that I can understand. This is not related to the way your company handles their business, just corporate IT in general.

If IT departments maintain windows systems, is it not normal practice to test a Windows update or driver or bios update before that update is pushed to all the end systems within the company? If so, since Crowd Strike affects endpoints just like those updates do, why wouldn't that same precaution be taken when pushing a Crowd Strike(or any other endpoint protection company) update to end systems? What's the difference? Is it just because Crowd Strike hasn't ever screwed up to this extent before and companies never considered something like this could happen? I'm just curious.

This massive problem was caused by an update to software that was already running on the client systems - not malware or a dodgy download...

IT support teams will be putting in some serious overtime this weekend

mecanicogolf said:

My toilet still won't flush, but I guess it's got nothing to do with this.

Click to expand...

That's an easy fix, call Joe the plumber, he doesn't depend on Winders to operate his plunger.

The update must have been automatic as all users of Crowdstrike across the world were affected simultaneously. So it would seem to me the update probably occurred in much the same way as anti-virus definition files are regularly updated (often several times a day).

glasskuter said:

@pparks1 Like I said, I'm a simple thinker and know maintaining such systems is waaay beyond my pay grade. But I wasn't suggesting redundancy for non-essential corporations. My observations were from a standpoint for such outages in instances like 911, hospitals, and the like where people's lives are at risk. One would think emergency measures would be in place for them.

Since it appears you do have experience in IT, I'd like a ask you a IT question so that I can understand. This is not related to the way your company handles their business, just corporate IT in general.

If IT departments maintain windows systems, is it not normal practice to test a Windows update or driver or bios update before that update is pushed to all the end systems within the company? If so, since Crowd Strike affects endpoints just like those updates do, why wouldn't that same precaution be taken when pushing a Crowd Strike(or any other endpoint protection company) update to end systems? What's the difference? Is it just because Crowd Strike hasn't ever screwed up to this extent before and companies never considered something like this could happen? I'm just curious.

Click to expand...

Good question.


Well, you purchase a software like Crowdstrike to protect yourself against the latest and most devastating 0 day exploits. You run the risk of getting pwned by malicious stuff if you sit back and don't get these updates on your system, but then you also run the risk of pwning yourself with a bad update from a software vendor.

For almost everything, we test in nonprod, we roll out slowly to prod and we test and ensure everything is well. Security updates for endpoint protection devices always go much faster and they are coming from the 3rd party vendor, so the app is just watching for a new update to post and then it updates. It's not something that we download on our own and push out when we are ready. Hence reason everybody got hit.

pparks1 said:

It's not something that we download on our own and push out when we are ready.

Click to expand...

But from now on, maybe?

antspants said:

But from now on, maybe?

Click to expand...

No because that's not the way it works.

antspants said:

They immediately took responsibility. They immediately set out to rectify it, all in the same day and first few hours of the incident.
If anyone needs to be held responsible for anything debilitating it should really be company and services system administrators for not having backup systems and processes in place.
Just my opinion of course.

Click to expand...

Crowdstrike is to blame here. Period.

I think a lot of people blaming IT staff for not having backups is not understanding the issue here at all.

We do have backup systems, we do have redundancy. This is all on crowdstrike antivirus for not having good quality control pushing out updates for their system.

Imagine if windows defender, avast, Kaspersky, symantec etc got an update that blue screened your machine as soon as it received it. And then you got stuck in a boot loop.

It doesn't matter that you have a system image backup, or have proper procedures in place. This was not a windows update or something you have control over. Restoring from backups and fixing the problem takes time and you need to touch every machine.

Failures are a part of life with systems, but its not usually everything at once.

This affected every system in the workplace that was on when this crowdstrike SILENT update was pushed. This affected servers, virtual machines, workstations, everything.

I am an admin at a tv station. I cannot choose when the antivirus gets updated. I have wsus and of course patch and approve windows updates. But this was something out of our hands entirely.

Let me be clear, this was not a windows issue. Windows was not the problem at all. It was solely on crowdstrike.

pparks1 said:

People in my organization have been up for past 10 hours dealing with fallout from this debacle. While there are backups, and policies and procedures, it's a huge impact on the business and we are still on the calls and working through the problems. Fortunately, our non-windows portions of our environment were not impacted.....but this still hit between 700 and 1000 servers, and countless workstation from end-users. It certainly wasn't the quiet Friday most people were hoping for.

Click to expand...

Exactly. People thinking we could have done something to prevent this can eat a shoe. I cannot control corporate security policy for their use in crowdstrike or manage updates for it. I was up since 2am working on this, and still not finished. I am taking a break. Believe me I hope crowdstrike fails. They barely did anything. I had to hunt for the fix on my own, which I found on reddit of all places.

I have to boot into safe mode on everything and do this:

This was the fix:

del /f /q "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"

After doing that everything was back to normal. Oh but some systems have bitlocker, so you have to look up the bitlocker recovery key.

This was a ****** nightmare. So please try to understand.

I fully understand this problem was caused by CrowdStrike but the general public does not. Everyone is blaming Microsoft since it was Windows that crashed. I was at Brookshires today and all their gas pumps had a sign on them saying "Out of order due to Microsoft issues". It's sad, really, that MS will get the black eye over this. People know Microsoft. They've never heard of CrowdStrike....and the public doesn't fully understand. No matter how many times Crowdstrike is mentioned along with Windows in the same news report, Windows is all they hear.

glasskuter said:

I fully understand this problem was caused by CrowdStrike but the general public does not. Everyone is blaming Microsoft since it was Windows that crashed. I was at Brookshires today and all their gas pumps had a sign on them saying "Out of order due to Microsoft issues". It's sad, really, that MS will get the black eye over this. People know Microsoft. They've never heard of CrowdStrike....and the public doesn't fully understand. No matter how many times Crowdstrike is mentioned along with Windows in the same news report, Windows is all they hear.

Click to expand...

exactly it's so frustrating. The news started doing what they always do and dont check anything out they just parrot whatever AP enps tells them. Because people are stupid with tech and dont research anything.

We knew it was cloudstrike pretty quickly. But they didn't get the blame until much later in the day.

If I was microsoft I would be quite mad right now. But to be fair, windows should have something in place where if it bluescreens right after an application update is done, it should revert it. But malware.... so yeah.

I'm just so tired.

andrew129260 said:

and you need to touch every machine.

Click to expand...

If IT management doesn't grok the concept of how PXE Server can be used to break out of an infinite reboot loop, then they should all go to pick strawberries IMO.

andrew129260 said:

….. I'm just so tired.

Click to expand...

That kind of stress is a killer! Take an extended break away if possible.

hdmi said:

If IT management doesn't grok the concept of how PXE Server can be used to break out of an infinite reboot loop, then they should all go to pick strawberries IMO.

Click to expand...

Organizations have things set up that are out of your hands man. We don't have pxe anything. I can only change what I can change.

Also our systems are locked down and don't have that option to boot from that. It only allows boot from a locked bitlocker drive. I cannot alter the bios.

And even if we had pxe it would still be wiping the system or restoring from a backup, which you still need to touch each machine to tell it to boot from pxe.

I only have the tools I have.

Also clients with laptops that are not in the office, good luck with that.

We are all free to have opinions, I think I said “in my opinion”
I don’t begrudge anyone of having one.

antspants said:

We are all free to have opinions, I think I said “in my opinion”
I don’t begrudge anyone of having one.

Click to expand...

I understand your opinion I really do. But it's not a perfect world, and I cannot change corporate policy. Many companies are not going to pay to have two different antivirus companies so that this doesn't happen. Or have the team or resources to test each antivirus update. Or if the antivirus even offers a management console to begin with to deny or approve updates.

I know it seems easy to just have backups in place and just have a solution, but that is not always possible. Also not everyone in IT is as smart as a lot of the people in here. I know I am not. I do what I can with what I have. And that is all I can do. I learn as much as I can.

Corporate especially is a whole other animal. Trying to get them to do the right thing or have good procedures is like pulling teeth.

Maybe there is things they could do, but I just don't think its practical enough. stuff happens. And unfortunately many people and organizations it happened today.