Undoubtedly, the CrowdStrike outage will be discussed for a long time in the IT world. Although we cannot fully determine its scope and the consequences it will have on affected companies and users, we can already conclude that this was not just any Windows BSOD (Blue Screen of Death). In fact, thousands of people had their daily lives disrupted, and hundreds of companies had to focus almost exclusively on solving the problems it caused.
In this article, we will review what happened, who was affected, and, if you haven't already managed to do so, how to fix the issue on devices that experienced problems.
UPDATE 22 July: Microsoft has released a CrowdStrike recovery tool designed to help IT teams quickly recover Windows devices affected by the flawed update.
What caused the Windows BSOD?
This Friday, airports, banks, and companies around the world were confronted with the alarming Windows screen that no one wants to see. The Windows BSOD indicated an operating system crash and the interruption of their operations.
What caused the problem? A failure in an update from CrowdStrike, an U.S. cybersecurity company that provides software to thousands of companies worldwide.
The company's CEO, George Kurtz, stated on the social network X that the problem was "identified and isolated" and that they are "deploying a solution." He also dismissed initial claims that it was a security incident or cyberattack.
Everything You Need to Know About Microsoft's Blue Screen of Death (BSOD)
Who is affected by the CrowdStrike incident?
It is difficult to pinpoint the full extent of those affected by this failure. Airports, hospitals, banks, media outlets, government offices, supermarkets, and countless companies from various sectors suffered — and are still suffering — the consequences of this failure.
Although the exact number of affected individuals is unknown — Microsoft Windows is one of the most widely used operating systems in the world — we can mention that Europe and the United States were hit the hardest.
Despite the release of the fix, experts point out that the manual process required to restart each affected terminal will take considerable time and effort. Therefore, it is estimated that the solution will not be quick for everyone.
How to perform a quick CrowdStrike audit with InvGate Asset Management
Before diving into the workaround for the Windows BSOD, it's a good idea to know exactly how many devices are currently running Windows operating systems and how many of those have installed the Falcon Sensor (the agent that is causing the problem).
You can perform a CrowdStrike audit on InvGate Asset Management with just a few clicks, and address the issue with the knowledge needed to solve it rapidly and effectively.
How a CMBD can help?
A Configuration Management Database (CMDB) is an essential tool to map the IT infrastructure. In this particular case, by having one you can easily spot which services are affected by the CrowdStrike incident, and therefore plan for the outage.
What’s the workaround?
The Windows BSOD can be alarming and discouraging. Fortunately, CrowdStrike has provided a workaround involving the deletion of a specific file in Safe Mode or the Windows Recovery Environment to resolve the issue. Here is what you need to know.
Workaround steps for individual hosts
- Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, go to step 2.
- Boot Windows into Safe Mode or the Windows Recovery Environment. Using a wired network (as opposed to WiFi) and Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory and locate the file matching “C-00000291*.sys”. Delete it.
- Boot the host normally. Bitlocker-encrypted hosts may require a recovery key.
Workaround steps for public cloud or similar environment (including virtual)
You have two options according to the CrowdStrike blog.
Option 1:
- Detach the operating system disk volume from the impacted virtual server.
- Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes.
- Attach/mount the volume to a new virtual server.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
- Locate the file matching “C-00000291*.sys”. Delete it.
- Detach the volume from the new virtual server.
- Reattach the fixed volume to the impacted virtual server.
Option 2:
Roll back to a snapshot before 0409 UTC.
Conclusion
The CrowdStrike update incident will undoubtedly be discussed for a long time in the IT world. While the full extent and long-term consequences are still unfolding, one thing is clear: this was not just any Windows BSOD.
Thousands of individuals had their daily lives disrupted, and hundreds of companies had to divert significant resources to address the issues caused by this update. This article has provided a detailed overview of what happened, who was affected, and a step-by-step guide on how to resolve the issue on affected devices.
By following the workaround provided by CrowdStrike, organizations can mitigate the impact of this incident and restore normal operations. It's crucial for IT teams to stay calm and methodical during such crises, ensuring that they address each affected device carefully and efficiently.
While the recovery process may be time-consuming, especially for large-scale environments, these steps offer a clear path to resolution and help maintain the security and stability of IT infrastructures.
