Running infrastructure at a Fortune 100 firm is akin to managing an amusement park. Lights flash, rides whirl, and distractions emerge left and right. But the goal remains the same: keep operations running smoothly, delegate the fire-fighting and focus on the experience. 

Now imagine the amusement park is spread across the globe. 

In his day job, Evan Kotsovinos, senior vice president and global head of infrastructure at American Express, is responsible for the financial services firm's global infrastructure, which includes mainframes, data centers, data platforms, tech operations, IT service management, crisis management and cloud. 

His "night job" is to run the technology response to the pandemic, Kotsovinos told CIO Dive. 

American Express, No. 67 on the Fortune 500 this year, reports a $79.4 billion market cap and has 65,000 employees operating in more than 40 countries. The company relies on technology cost efficiencies for operating leverage, investing in technology to create digital capabilities and grow revenue. 

When the pandemic hit, in a two-week period, American Express went "100% virtual around the world," Stephen Squeri, CEO of American Express, said during its Q1 2020 earnings call in April. 

To support a large-scale shift to work from home, American Express had to ensure its employees had access to the software and hardware necessary for day-to-day productivity. Normally requirements are centered around the office, but the coronavirus pandemic sent large parts of the workforce home, clipped international freight and locked down supply chains.

American Express isn't alone — the shift to remote work left companies scrambling to get their employees remote-enabled technology and services. A little over half of companies had the technology required to support remote work, Andrew Hewitt, analyst at Forrester, told CIO Dive. It's not just laptops, it's also the monitors, keyboards and network connection. 

An extra layer of complication came from the sheer number of employees at American Express. Deploying large amounts of tech to people around the world requires simplicity, Kotsovinos said. 

When moving a large workforce from the office to home, companies lose control over a large part of the infrastructure — in the office companies have "total control," Kotsovinos said. Connecting over home networks, relying on a combination of office and employee-provided equipment, organizations have less control, challenging the maintenance of a high level of productivity. 

To take its workforce remote, American Express:

• Added capacity, intelligence and telemetry to the network to manage employee experience and productivity. In April, the company logged 38 million meeting minutes, doubling the number over February.

• Worked to understand how employees were feeling, using documentation, surveys, chats and town halls, where about 20,000 people connected.

• Honing applications for remote connection. Normally set to operate in the office, front- and back-end don't connect as well when distributed, Kotsovinos said. 

What's next

The remote transition was accomplished months ago when lockdown orders were put in place. Now, focus turns to fine-tuning the user experience. 

Infrastructure teams can augment the end-to-end experience with data and analytics to anticipate, and quickly navigate, technology hiccups. 

Help desks usually support enterprise needs, but now they're tasked with troubleshooting "these additional personally-owned things" and the enterprise has to rapidly support them, Hewitt said. Problems can arise navigating the sheer volume of service needs. 

To navigate it, some organizations will divert people from more visionary, transformation projects and temporarily pivot them to a support role, Hewitt said. 

American Express had to initially increase service desk capacity as volumes doubled what they normally would be, though they've since normalized, Kotsovinos said. It also set up a "situation room" to monitor the experience of infrastructure up through the application stack, combining data, analytics and operational discipline. 

This evolves into multicloud management, he said. 

American Express is using a cloud-based service that provides telemetry on broadband networks and ISP, Kotsovinos said. Before an employee reports an issue, the company can detect where the end-to-end network experience drops off. 

Several times, American Express has seen a number of employees in a particular location having trouble connecting to a VPN, Kotsovinos said. The company can catch the issue in real time and determine if its an infrastructure issue on their end or an issue with the ISP. American Express can then work with the ISP to resolve the issue. 

Without that anticipation, people struggling to connect would call the service desk, which would take them through remediation steps but would eventually still require escalation. 

The scrutiny on employee connectivity stems from the demand. If technology impacts employees, it trickles throughout the business. 

Technology drives about 30% of employee engagement, according to Hewitt. If workers have a better experience it will increase engagement, which trickles up to having a positive customer impact.

"If you don't deliver a good experience to colleagues serving the customers, the customers are going to suffer," Kotsovinos said.  

Pandemic impacts on transformation

Supporting American Express' remote operation is an underlying modernization pivot to private cloud, which the company began in 2016. The company runs its data centers and mainframes remotely "more than you would imagine," Kotsovinos said. 

The pandemic has not changed the company's transformation efforts in a meaningful way; it's still focused on hybrid cloud, deploying applications in secure containers running on private and public cloud. 

With American Express working remote, there's less focus on the "how" of technology. What's important is the end result: a technology-enabled workforce. 

The more mixed the experience gets — public or private cloud, in person or remote — the less focus on the nuts and bolts of the technology stack. Whether in office or remote, Kotsovinos is accountable for the end-to-end experience and the service. 

"I don't think anybody knows" what office operations will look like long-term, Kotsovinos said, Throughout, regardless of the outcome, he'll remain the "curator of the service."