Dive Brief:

• Data professionals are at or above capacity when it comes to handling their organization's data needs, according to a survey released Tuesday by data engineering company Ascend.io. The company polled 400 U.S.-based data professionals in the second quarter of 2021. 

• More than nine in 10 data professionals say they anticipate the number of data pipelines in their company to increase by the end of 2021. More than half say the number of data pipelines will increase by over 50%. 

• One in five data professionals says the infrastructure and systems in place are unable to scale accordingly in response to increased data volume processing needs. At three-quarters of companies surveyed, the need for data products is growing faster than the size of the data team.

Dive Insight:

An ideal digital company runs on data to make decisions, reach more customers and increase revenue. But while companies have gotten better at collecting data over time, quickly getting to the insight is still a challenge.

"For the longest time we've been focused on how to store more data, how do we process more data," said Sean Knapp, founder and CEO at Ascend.io. Given the improvements to existing data infrastructure tools, CIOs can now "turn their attention to the challenges that sit higher up in the stack, that are actually closer to business value."

Even with more advanced technology, data professionals remain under pressure to produce insights, leaving businesses to solve for capacity constraints in their ranks. Training strategies to increase the capabilities of data professionals is one part of the equation.

"Companies should be providing time and resources to allow their current talent to continue and further their education and knowledge in this field," said Irina Sedenko, senior manager of the data and analytics consulting practice at Info-Tech Research, in an email. "As with many other professions, data science continues to change, so continuous learning is a must."

Given the expansion of data inputs to handle, more than one-third of companies are only somewhat confident, or not confident, in their ability to leverage data insights, a Starburst and Red Hat study found.

Increasing the thirst for data management in the organization is the adoption of AI and machine learning systems, two technologies that demand data in order to operate, said Sedenko.

"However, it’s a shortage of quality data," said Sedenko. "Therefore, organizations need a framework for proper data management to know what data they have and ensure the quality of it."

A CIO's role amid the complexity is to help get more leverage from the existing data team, according to Knapp. Automation, no code and low code can assist with the task. 

"Data productivity is the new scale challenge," said Knapp. "We should be turning all of our attention as an industry towards facilitating that."