Editor's note: This article is part of The Water Cooler, a recurring column for technology executives to digest, discuss and debate. Next up: How do you disconnect from work, especially with holidays coming up? Email us here: [email protected]
When a crisis hits, some managers scramble and improvise to put the fire out. Others focus on executing previously drawn courses of action.
For IT executives, preparedness is as crucial as it's ever been — a technology crisis is likely to seriously disrupt a company's ability to operate. A cloud provider outage can mean work halts until the problem is solved. When collaboration tools fail, the ability to collaborate from anywhere is upended. Application woes translate to disappointed clients and missed deadlines.
To address a technology crisis that could affect the business, communication and calmness are essential. But prior to a systems outage or cyberattack, IT execs can help lower potential impact to their organizations by strategizing in peacetime.
Here's how 5 executives communicate and delegate crisis management among the IT ranks.
(The comments below have been lightly edited for length and clarity.)
Harry Wan, head of security at Britive
"Having [a tabletop exercise] attended by the entire executive team ... helps us work out roles and responsibilities proactively making our communication more effective."
Harry Wan
Head of security at Britive
In a previous role, I was a vendor that was part of a larger team responding to repeated and persistent credential stuffing attacks. The weakness I experienced as part of this team and in post-mortems of the event were:
- Lack of expertise in diagnosing the path the attacker was taking to accomplish the attack.
- Lack of authority of the security team to effect proactive protection. We recommended specific firewall changes that were not implemented, and the recommended change would have prevented a subsequent attack.
- Involvement of executives, on both the vendor side and the customer side, only after significant damage was done to the customer and to our relationship with the customer.
At Britive, we've performed a tabletop exercise involving all our top executives with regards to business continuity and threat response, as part of our ongoing SOC certification. Having such an exercise attended by the entire executive team contrasts with my last experience and helps us work out roles and responsibilities proactively making our communication more effective.
Yoni Farin, CTO and co-founder at Coralogix
"We're not discussing the why or how of what happened, we're leaving those parts of the discussions for a post-mortem and not getting distracted by it at this point."
Yoni Farin
CTO and co-founder at Coralogix
When we have an incident and our R&D team needs to shift into crisis management, our most important step is to get all of the team leaders on Discord and sitting in the same "room." It's crucial for us to not rely on asynchronous messaging systems for crisis management.
This is especially important now with COVID-19 and having a global workforce. In this call, we're discussing the immediate actions that need to be taken and split or share the tasks among the teams. Then, each member will activate the relevant people and the call is ongoing until the crisis has been completely resolved.
During the call, we're focusing only on the immediate actions that need to be taken. Crisis management is really about making fast decisions based on the data you have and maintaining a tight feedback loop for the decision making. We're not discussing the why or how of what happened, we're leaving those parts of the discussions for a post-mortem and not getting distracted by it at this point.
Rick Song, CEO and co-founder at Persona
"One important point is to let employees know what's happening, so they can explain it to their customers and adapt to get their jobs done."
Rick Song
CEO and co-founder at Persona
Security threats can be unpredictable and catch you by surprise. Therefore, it's critical for companies to follow a set of protocols to ensure they are ready to respond to any issues. Here's what we recommend for customers undergoing fraud attacks:
- First order of things involves containment. Bottom line is to stop the fraud and prevent further damage. Lock down file exfiltration, for example.
- Next, assign people to identify and patch immediate vulnerabilities where possible. Evaluate the impact on customers and communicate the threat and remediation steps in detail. Transparency is key.
- The next stage is hunting for adjacent threats. Is this an isolated incident or part of a larger attack?
- Lastly, once the situation is under control and proper communication has been established, evaluate current systems and processes. That could lead to changing vendors, reconfiguring or adding security controls or more automation, updating their security posture if needed.
One important point is to let employees know what's happening, so they can explain it to their customers and adapt to get their jobs done.
Peter Marsh, VP of security and compliance at BetterCloud
"When people panic nothing works as it should."
Peter Marsh
VP of security and compliance at BetterCloud
In any crisis, people tend to panic. When people panic nothing works as it should. Individuals may freeze up, not knowing how to proceed, they may focus their attention on the wrong area, or they may make errors that compromise the ability to recover.
The role of a leader in such situations is to provide a source of calm.
The best way to provide calm for your team and yourself is to have a well laid out and well rehearsed plan to follow. Each team member should know the plan and their role in it. It provides them with focus and removes the sense of helplessness that can occur in crisis situations.
Better still, if certain parts of the process can be automated, such as creating a Slack war room or initiating a call bridge, the team will imply that the situation is under control.
To be effective the plan must also be thoroughly tested to identify any issues with the current plan such as the omission of key processes. Those problems can then be addressed so that, when a real crisis hits, your plan and your team can hit the ground running in a confident and effective manner.
Joe Byrne, VP of technology strategy and executive CTO at Cisco AppDynamics
"I've found a best practice for companies is to have existing disaster recovery plans in place."
Joe Byrne
VP of technology strategy and executive CTO at Cisco AppDynamics
Every tech department is different, but something that should always be considered is having a plan and tools in place to ensure that plan is executed correctly, especially because an outage or cyberattack can create chaos.
In my experience as a former engineering leader and while I talk to customers in my current role, I've found a best practice for companies is to have existing disaster recovery plans in place.
However, if an outage occurs, technologists need to quickly assess when the failure occurred and what failed in order to get back up and running.