Our statement on the CrowdStrike outage

In the world of cyber security, we live with statements like, “Bad things will happen,” and “It’s not, ‘if’ but ‘when.” Problems will always occur, but it is how we react and handle the crisis that really defines us.

CrowdStrike as a security vendor has, without a doubt, pushed the standard high within cyber security. It has earned the right to be called a leader in the market, and particularly the EDR sector. While the cyber security community is aware of CrowdStrike’s popularity and widespread use, the general public may not have been until the 19th July, when the outage had a global impact. What we need to ask is: was this popularity ill-founded and did we have it wrong?

To answer these questions, we need to understand the issue and why it was allowed to happen. Currently we do not have all the information to make that call, but let’s review what we do know.

Was it a Microsoft or CrowdStrike issue? The press widely reported the outage as a Windows outage and on the face of it laid a large proportion of the blame at Microsoft. While yes, this only affected the Windows OS, it is not that simple. The investigation is still ongoing and we don’t yet know why exactly this occurred. We do know that this was a ‘bad’ driver file published by CrowdStrike which was automatically updated and cause Windows to halt and crash (as any Operating System would and should, to protect itself). This was further compounded because the driver is defined as a ‘boot driver’ which meant that it could not be halted upon a reboot, which then meant that engineers needed to manually remove the ‘bad’ file within a restricted boot and typically required physical access to the machine.

A lot of Windows skeptics will lay blame at Microsoft, asking why their OS would crash in such a way with a 3rd party piece of software. Without getting too technical, in order for CrowdStrike to gain visibility on all activity on the system, it has to tie-in to the OS kernel. This makes it a critical driver and therefore one which can have a much bigger impact on the system if and when it does fail. Therefore the blame needs to be solely at the feet of CrowdStrike, and we need to understand why and how this issue occurred.

Now that I am pointing the finger in their direction, the question I must ask is: have I lost trust in CrowdStrike? 

My approach to answering this has been similar to how I view an organisation during and after a breach. Do they understand how it happened? Could it have been avoided? What have they learnt? And what are they going to do to make sure it doesn’t happen again? They must be honest and transparent; hiding anything will create distrust and as a security vendor that is ‘game over’ for me. As of writing I would rate CrowdStrike high in this area. Their communication has been clear and appears to be open and honest from the initial incident, updates, fix communications and the message put out by their CEO. CrowdStrike are handling this as a mature and strong organisation.

The things which are yet to be answered, and which will be essential for me to remain a promoter of CrowdStrike are as follows. Why and how was this update released? This incident affected all versions of Windows from 7.11 to the current version of 11, and we all saw the widespread impact this had. Why was this not caught with in Quality Assurance checks? How did this code make production and what will be done to make sure this never happens again? For me we need the honest answer to this, and this cannot be ignored or swept under the rug.

What could we do to prevent this? I have heard a number of comments and opinions on this from multiple EDR vendors to stop using Windows completely! Sadly, we are also seeing some vendors and resellers ‘ambulance chasing’ and promoting their alternative solution, claiming it would not happen to them. But we need to be pragmatic when we consider our reactions. The impact was so widespread due to the amount of CrowdStrike customers and protected machines, which shows confidence in their platform. We are also seeing that while there was an immediate drop in share price value the price is still holding strong, which does show commercial confidence.

It is also not as simple as replacing one with another or supporting multiple EDRs across your environment. Your organisation has already invested in this technology. Do you risk moving to a less effective solution? At the operational level how do you ensure that your team can support multiple EDR solutions?

This post asks more questions than it answers and right now it is still too early to give a clear opinion. However, if CrowdStrike continue to handle the issue in the way they have so far then I am confident we will get the answers we need, and SEP2 will continue to be supportive of Crowdstrike as a key vendor and pillar within our customers environments.

-Paul Starr, CEO and Director