The federal government is often ridiculed for its reliance on legacy systems. A Government Accountability Office inventory of federal technology released last May found one agency still using floppy disk drives and another employed a computer language developed in the 1950s.

The same report found the federal government spends almost 75% of its $80 billion IT budget on operations and maintenance of legacy systems. Because so much money is spent keeping aging systems running, federal agencies often find it difficult to modernize.

Richard McKinney experienced this challenge first hand when he became CIO of the federal Department of Transportation in 2013. Though he wanted to standardize on Office 365, he was faced with a systems rollout across a huge organization with a decentralized approach to IT. That, combined with worrisome network capacity, created quite the challenge for an incoming CIO.

An enormous undertaking

When McKinney first joined the DOT, he wanted to standardize on Office 365 to allow his agency to better manage email and messaging storage more cheaply. The move would also allow for new capabilities, such as video calls, which the agency could not provide natively in its legacy environment.

But the DOT is an enormous organization. Depending on how they are counted or measured, there are nine or 10 individual administrations that operate under the DOT umbrella, including the Federal Aviation Administration, the Federal Railroad Administration and the National Highway Traffic Safety Administration.

The DOT’s Office of the CIO alone includes approximately 150 federal employees and several hundred contractors.

Beyond managing the migration challenges associated with the scope of the organization, McKinney was also challenged by the fact that, like many federal agencies, IT at the DOT was historically managed in a decentralized manner. In other words, each operating administration navigated IT on its own and via its own budget.

"This decentralized approach to IT rarely works well. The notion that IT done in a decentralized, federated manner is somehow better than an enterprise approach is the Achilles heel of government," said McKinney, in an interview with CIO Dive. 

"The private sector got over this a long time ago, because they drive to a bottom line. Government doesn't face that pressure, so it hobbles along inefficiently," he said. "No one sneaks up on them and steals their business, so they become very complacent."

McKinney realized to make progress he’d need to strengthen the governance process first, so he gathered other department CIOs to begin coordinating a move to Office 365. But when he did so, he encountered another issue: the other CIOs were concerned their networks couldn’t handle the traffic.

Because each operating administration ran its own network, and the interconnection of those networks is what formed the DOT network, there was no oversight to ensure each network was up to par.

So when the CIOs told McKinney they weren’t sure their individual networks could handle migration, he immediately knew what they meant.

"They had been doing their network on the cheap all these years, so fears that their networks couldn’t handle the move to a cloud app like Microsoft 365 were very real," said McKinney. "If we made the move and then had bandwidth problems and unhappy users, we would have cut off our nose to spite our face."

So the DOT engaged a private sector partner to probe its network to determine where it might have bandwidth capacity issues when deploying Microsoft 365. The company conducted a network inventory as well as a "health check" on all devices connected to the network to determine the last time each device was patched, the status of its security settings, etc.

"Because our network was decentralized, we expected to get a laundry list of things we needed to address, and we did," McKinney said.

DOT took a couple of months to clean up the mess, and McKinney said he was starting to feel pretty good about getting the agency’s ducks in a row. But then, the DOT’s private sector partner offered to conduct an "auto detect" scan, which checks network devices and reports back the IP addresses of any piece of equipment those devices are "talking" to via the network. In other words, shadow IT.

About 24 hours later the company reported it had found 200 pieces of network gear on the DOT network that DOT was unaware of.

McKinney called it a "wow moment."

"Your imagination runs wild at that point," he said. "Have foreign governments put equipment on our network? Have we been compromised? You begin to have these horrible thoughts. How could you not know about 200 devices?"

Within a few days, DOT’s contractor physically located all the devices, which brought both relief and additional frustration. "This wasn't an adversary putting equipment on our network," said McKinney. "We had done this to ourselves. This was a self-inflicted gunshot wound."

The cowboy approach

It’s easy to understand how such situations occur, especially at the pace technology tools have evolved over the last few years. When a DOT office needed a switch with bigger capacity, an employee would simply head to Best Buy, purchase a new switch, and daisy-chain it onto the network.

"They just cowboyed their way through their bandwidth problems," said McKinney. "Meanwhile, headquarters didn't know anything about it."

DOT had not been compromised, but only because it was lucky. Few devices in use were the secure, industrial-strength equipment it required. Many of those pieces of equipment had poor security protocols, or were still using factory default passwords.

"All these things had to be eliminated, so we had to go to all the operating administrations and say here is a list of all the things in your office that have to go," said McKinney.

In the end, DOT ended up with a complete and accurate list of all their network assets for the first time, which was important not only for DOT management, but also because the agency had signed an agreement with the Department of Homeland Security allowing them to probe their network for security reasons.

"It was an unintended consequence, but we ended up with a good understanding of our network," said McKinney.

For McKinney, who left the DOT when the Trump administration took office, it was another lesson about the dangers of a decentralized approach to IT and the lack of visibility that can accompany it.

"Before we did this work, there is no way we could have known if our network had been compromised," he said. "Knowing what we had and how it was being used was critical to ensuring we could protect ourselves as well as the first step in modernizing our technology."