Editor's note: This article is part of The Water Cooler, a recurring column for technology executives to digest, discuss and debate. Next up: Would you rather join an organization running on a fully modernized tech stack or an organization still running legacy tech? Email us here.
Deciding to pilot a project requires buy-in from stakeholders across the business and a tech team ready to step up to the challenge.
Taking a risk on a new project is tricky — dedicating the time, resources and money to an endeavor without guaranteed results. But taking a chance that could result in failure is a part of the innovation process behind piloting new projects.
CIOs have struggled to find the balance between operations and piloting new ideas. Three-quarters of CIOs say they struggle to strike the balance between business innovation and operational excellence, according to IDG's State of the CIO report.
As CIOs balance operations and innovation, CIO Dive wondered how tech executives select the next projects to pilot. Here's how 7 professionals approach pilots:
(The comments below have been lightly edited for length and clarity.)
Elizabeth Hoemeke, CIO at One Inc
"If you don't understand the technology, you'll never know how or whether it could potentially be used to solve a critical problem."
Elizabeth Hoemeke
CIO at One Inc
When piloting new technologies, I've come to learn the evaluation process includes three categories: (1) exploratory R&D, (2) solving a specific problem, and (3) evaluating technology to ensure systems, applications and infrastructure are either a "fast follower" or cutting edge.
Take each of these in turn. First, to make true advancements across these three pillars, technology teams require at least one person dedicated in the area of applied research. While there are other ways to generate ideas beyond having a full-time staff — like hosting hack-a-thons and innovation days — it's critical to have someone focused on constantly exploring the use cases for emerging technologies. They must be savvy enough to evaluate and anticipate how new technology could solve problems across your industry, company or even an individual department.
Second, we see the testing phase as critical. If you don't understand the technology, you'll never know how or whether it could potentially be used to solve a critical problem. Quality partners — who knew our requirements and could lean on a wide network of customers and industry experts — were invaluable in driving such a specific research, assessment and implementation requirement.
When asking these questions around pilots, I also like to follow "the next 24" rule by ensuring the team is looking around the corner at what's coming — in the next 24 days, weeks or months. That gives us a shot at implementing these modern technologies when the opportunity, or specific problem to solve for, presents itself.
Finally, the saying, "if it's not broke, don't fix it," is a recipe for disaster. During the pandemic, we witnessed an evolution overnight as many companies realized their IT infrastructure is not as effective as once thought, presenting a potential opportunity for competitors. That is another reason to incorporate pilots as part of your culture: don't let your company or team be lulled to sleep thinking that everything is operating well with no need to change.
Kevin Skibbe, head of engineering at Calendly
"Leverage your networks for input and consider paying for expertise or consulting if you know it will accelerate your path toward finding a solution."
Kevin Skibbe
Head of engineering at Calendly
If you start with the technology instead of the problem you're trying to solve, you're approaching it from the wrong direction.
At Calendly, our decision to pilot technologies always starts with clearly defining the problem or the goal. Incorporating new technologies can be an expensive or risky proposition, you shouldn't do it simply because something is modern or "cool." There has to be a real need driving the investment.
Once a goal or problem is defined, look to the experts. Leverage your networks for input and consider paying for expertise or consulting if you know it will accelerate your path toward finding a solution. We also like to look at the community that's built up around a particular technology and challenge ourselves with questions here at Calendly. For example, if it's open source, how active is it? If we had to hire someone with experience using this technology, how difficult would it be?
Once you've done your research, narrow down a list of two to four technologies and run a bake-off. This type of test will help you ensure the technology will play nicely with your existing approved tech palette. Before proceeding with piloting, confirm the learning curve isn't too high; get educated on the pricing, support, and update models to ensure they'll meet your company's needs; and of course, make sure the new technology will help your organization overcome the challenges in front of you.
Johnathan Hunt, VP of security at GitLab
"Know when to cut your losses and don't try to force a bad hand — the tech will always call your bluff."
Johnathan Hunt
VP of security at GitLab
With technologies like AI becoming commonplace, businesses must take a critical look at whether technologies like this are a nice-to-have or a must-have. When assessing an AI investment, your key performance indicators (KPIs) should be personal and tie back to your long-term plans for success. AI-enabled tools and enhancements are worth considering, but first ask yourself: Is this the right solution for my business? Does this technology exist to solve my problem? Will this investment ultimately add value to my organization?
New technologies alone don't solve problems. AI, for example, has not yet reached the benchmark of solving real world incidents or building on human conversation. However, the technology has proven it can handle simple tasks, which clears up space for workers to complete more complex tasks.
AI/ML and other emerging technologies are not replacements for people. In fact, they may drive up expenses and generate new issues that require more labor. Several companies that have tried implementing AI for automated coding, ordering or customer support have abandoned it after terrible customer experiences or significant brand damage. If that's the case, know when to cut your losses and don't try to force a bad hand — the tech will always call your bluff.
With so much potential to drive efficiency, AI is a great investment (if it's the right fit for you).
George Brasher, general manager and global head of print services and solutions at HP
"The decision-making process to pilot new technologies at my company is always rooted in customer insights."
George Brasher
General manager and global head of print services and solutions at HP
The decision-making process to pilot new technologies at my company is always rooted in customer insights.
By having a deep understanding of how people work, create, and learn, we can design products, services, and solutions that make life better for everyone, everywhere. This process starts by crafting a solution to a specific problem a customer is experiencing. As part of this, we define what challenges or requirements we are observing in the user base, identify the current solutions and how well they address pain points, and outline the proposed timeline to be in market.
This is then reviewed by a broad group — including development, finance, sales, marketing, and support — to add context and solidify the proposal. Following the strategic review, the team begins the design phase of the offering. Once it is working, we see if there is value for others in the marketplace and then start industrializing the solution.
Amir Rapson, CTO and co-founder of vFunction
"Having the answers to these questions is crucial to plan the right course of action, because piloting a technology is just the beginning."
Amir Rapson
CTO and co-founder of vFunction
When approached with a new technology to pilot I ask myself four questions:
- Does it provide value and solves a problem my company is facing?
- Was it built to solve this problem?
- Will it integrate well with my company's current systems?
- And lastly, will the other decision makers be happy using it?
You may say that if this new technology provides substantial value by reducing costs or accelerating time-to-market, then that is all I need.
It is true that if the new tech solves a big problem, (and assuming I couldn't find another technology, better suited), then I will use it even if it wasn't built specifically for this problem, I might be willing to work harder to integrate it, and I may work harder to get more acceptance from the other decision makers. However, having the answers to these questions is crucial to plan the right course of action, because piloting a technology is just the beginning.
Irfan Khan, president of HANA Database and Analytics at SAP
"It's important to consider the long-term sustainability of a technology before it's piloted with customers."
Irfan Khan
President of SAP HANA Database and Analytics
When selecting new technologies to pilot, it's important to have a customer-first approach. New technology often comes along with a lot of promises, but the reality can often be different. Moreover, it can be hard to determine whether something looks great because it solves a real problem or just because it's new and shiny!
When making the decision, it's important to have a high-level evaluation framework; do our customers need/benefit from this new technology? How mature is it? Will adoption result in "first day cost" or "first day value?"
It's also important to understand the commercial implications — can a business case be created to justify the investment and future adoption?
It's important to consider the long-term sustainability of a technology before it's piloted with customers. Innovation is not an atomic event. Underpinning this, there needs to be a vibrant community and eco-system to accelerate innovation scope and subsequent customer adoption, matched by a supply and demand of skilled practitioners in the market. The ultimate aim should be non-disruptive to existing landscapes and improve the overall experience and usage scenarios of the users.
It's equally important to assess how the technology adds value to the current portfolio. It's very clear SAP is an enterprise technology company and we continuously evaluate our offerings against the market to ensure we are able to turn data into actionable insights for our customers. When deciding what new technology to pilot, we also need to have a clear vision of where it fits in our portfolio and how it complements or improves what we already have in our organic plans.
John Wirtz, chief product officer and co-founder of Hudl
"If we think we can roll out a suite of new products organically with adoption primarily driven by word of mouth and referrals, we know we are on the right track."
John Wirtz
chief product officer and co-founder of Hudl
At Hudl, the first key element driving the development of new technologies is the customer. We first need to clearly chart how it unlocks value for our customers. The ability to offer tangible value underpins our decision making in this area and we find that it typically solves lots of key questions right out of the gate. For example, if we think we can roll out a suite of new products organically with adoption primarily driven by word of mouth and referrals, we know we are on the right track. On the flip side, if we feel it is going to require a lot of top-down education and sales pressure, that’s a sign that the strategy is off.
The second is an emphasis on how we will feed high quality inputs — in our case video or data — into that technology or product. Great technology often fails to deliver because of poor inputs. Our customers rely on tools that can offer efficient, flexible and comprehensive results from video and data.
Finally, at the end of the day, we require our technology to be used quickly by athletes on the go, meaning that we need to build every new product with CX and compatibility in mind. Coaches and athletes need to use this technology to inform in-game adjustments that can be quickly communicated to athletes on the field, for example. We can create a quality CX and retain coaches and athletes on staff by continually integrating the feedback of pilot users throughout our R&D process.
