When systems fail, businesses have two choices: normalize failure or learn from the problem and mitigate a recurrence.
"Never let a crisis go to waste," said Johnson Controls VP & CIO Diane Schwarz, speaking at the Forbes CIO Summit in April. If a system goes down, businesses have the opportunity to communicate to leaders how critical the technology is.
Since March 2020, half of IT decision-makers say IT downtime has increased, according to data from LogicMonitor released in February. Over six in 10 IT leaders say that an outage has meant lost productivity.
From the cloud to niche SaaS tools, a vendor outage can make a company falter if employees don't have a clear contingency plan. To successfully guide the company through a vendor outage, tech executives need to know what steps to take — and the right questions can steer the business away from a shutdown.
Here are 5 questions to ask after a vendor fails:
Can we find out what happened?
Status pages don't always reflect, immediately or accurately, critical system outages. Some in the tech industry call this the "watermelon effect" — status pages that are green on the outside but red on the inside, said Josh Chessman, senior director analyst at Gartner.
In the wake of an outage "one of the challenges, especially with SaaS vendors or any kind of cloud-based vendor, is that you don't know if you're going to be able to find anything out" about the root cause, said Chessman.
Getting accurate information on what transpired gives leaders the ability to prepare, and assess whether the outage was caused by a long-term structural problem or a short-term hiccup.
Why did the outage happen?
If they know why an outage took place, CIOs can "better understand whether or not you have any type of redundancy built into your process," said Alla Valente, senior analyst at Forrester. "Next time you have a vendor outage: Do you have another provider of similar services that can kick in?"
In the last two years, there's been a movement toward platform consolidation. An outage is an opportunity to gauge how widespread a vendor is used throughout the organization, and what parts of the business the outage impacted.
"For many reasons that strategy makes sense, but there is such a thing as over consolidation, to the point where it creates a concentration risk," Valente said.
When will services come back into operations?
Solving for vendor outages can be difficult because their effect trickles down. If an employee is unable to complete a task due to an outage, that causes disruption for clients. If the makers of a software solution lose their main cloud provider, that can bring down the availability of the product.
When a service is unavailable, as a CIO "you can't stay silent," said Naveen Chhabra, senior analyst at Forrester.
"You have to inform, update the users, your internal users, your partners," Chhabra said. "They are depending on the tech executive to ensure the services are available."
Organizations and partners will ask the tech executives whether a service will stay down for hours or days.
Has this happened before?
In the tech stack, recurring problems need attention. If outages become a habit for providers, then so do the consequences of those outages, which can include failing customers or damaging the employee experience.
"Once is a problem, but if this has occurred previously then it is unforgivable," Andy Wood, VP at Proxima, said in an email. "You would have expected contingency to have been applied following any prior outage from those lessons learnt. Lightening should not strike twice."
Companies should consider incumbent suppliers with a track record of unsolved outages as candidates for replacement, according to Wood.
How well did the organization's contingency (or back-up) plan / disaster mitigation strategy perform?
No matter how thoroughly companies fleshed out their business continuity plans, it's unlikely many contemplated a pandemic scenario. Widespread operational disruption tested those plans last year.
Now, companies are more familiar with risk, and the possibility for sudden issues — such as a vendor outage. Assuming that outages, or disasters, are inevitable, companies need the right strategy and plans in place to weather those challenges, said Ted Wohn, senior manager, change management at Eagle Hill Consulting, in an email.
Planning should also address "how organizations manage their operational risks" and how risk management practices will help them be prepared for any future disruption scenarios, he said.