After a CrowdStrike outage crashed IT systems around the world, CIOs turned inward and began reviews of their own business continuity strategies to prevent a similar crisis from striking twice.
On July 18, a defective software update in CrowdStrike Falcon Sensor caused Windows computer systems to crash, bricking everything from airlines to banks and hospitals. Part of the problem, CIOs say, was a lack of preparedness and control over vendor-provided software.
“It was a little scary and just shines a light on how vulnerable a lot of companies are,” said Jon Dack, CIO at software company Celonis. That doesn’t mean CIOs are powerless.
Checking the links between IT systems, setting clear expectations with critical software providers and reviewing business continuity plans can help CIOs stop the next CrowdStrike-like interruption before it happens — or respond quickly to any other potential wrench thrown into company systems.
While the CrowdStrike outage was not a hack, it did remind Michael Corrigan, CIO of World Insurance Associates, about supply chain issues associated with the 2020 SolarWinds and 2021 Log4j compromises. The key commonality is that the initial issue was outside the control of impacted users, Corrigan said.
To mitigate the next potential problem — whether via bad actor or bad update — companies should be working to get a better understanding of the connections within their own infrastructure. What might seem like a low risk server, for example, could in fact be “connected to something else that’s connected to the terminals at the airport,” Dack said.
Knowing those links can help technology teams better understand where their risks are, and possibly make changes so that one point of failure doesn’t take down an entire system.
CIOs should be talking with vendors to make sure that they’re following customer protocols instead of pushing changes upon them. “We have the processes in place but if it’s bypassed by a direct update you’re counting on the vendor supplier to have followed a rigorous change control process,” Corrigan said.
Building in resilience
The CrowdStrike outage has CIOs rethinking the so far relentless push for automation and efficacy. “This wasn’t a bad actor thing," said Kevin Rooney, CIO of West Monroe. "It was something that was working as it was supposed to work.”
Instead of automating everything together on the same platform, CIOs can reduce risks by automating processes in phases to improve the response capacity, Rooney said. That means the operationality of a company won’t be taken down by one mistake if it's injected into an automated system.
The CrowdStrike outage also showed the downside of running everything within one IT ecosystem in general, and the importance of diversifying. The outage was attributed to the number of enterprise machines running Windows and CrowdStrike’s endpoint protection platform, Cybersecurity Dive reported. Customers using the Linux or Mac versions of the update were unaffected.
Rethinking business continuity plans
Preparing for the next event doesn’t just mean trying to stop it. It also means having a plan in place should something sneak through.
An incident plan should include how to resolve the issue and also communicate with end users, including customers, about what’s happening, Corrigan said.
World Insurance Associates, which uses Microsoft systems, was impacted by the outage, Corrigan said. Despite the early hour of the outage, a business continuity plan helped the company react quickly.
The company had alternative systems in place already, which included manual workarounds and separate systems on standby that they could spin up in the case of an emergency.
Technology workers reviewed any potential cybersecurity issues, in case bad actors planned to use the global IT problem to their advantage.
The company’s plan also included a contact system for the internal IT team so they could be alerted to the emergency and get to work, and communicate with key stakeholders within the business about how much they might be impacted.
“You pull all your plans to have everyone come together immediately,” Corrigan said.
