Dive Brief:
- Ballooning data costs afflict most companies, according to a Wakefield Research survey of 200 DevOps and Site Reliability Engineering professionals commissioned by Edge Delta. The firm also surveyed 300 observability, Kubernetes and CX application professionals for the study.
- Nearly all respondents said their organization encountered data cost overruns at least a few times per year and over half experienced overages or unexpected spending spikes monthly.
- Despite the business value of analytics, organizations often have to purge data. Nearly all respondents admitted their organization discards data to cut costs and more than 4 in 5 said they commonly limit log ingestion for the same purpose. Nearly one-third acknowledged data is discarded randomly by their company.
Dive Insight:
Data is the lifeblood of business, but too much of even a good thing can be toxic.
Nearly 4 in 5 respondents admitted disputes arose within their company over data ingestion and the same number said they’re paying too much for observability solutions even after imposing limitations.
The scope of the problem is reflected in the rate of data estate expansion.
Log data for the average company has expanded by a factor of five in the last three years, the report found. More than one-third of respondents said their company generates up to 1TB of data daily. An additional 37% generate more than 1TB each day.
As mining tools improved, sources proliferated and cloud unlocked a practically bottomless well of storage, enterprises turned to observability solutions and FinOps practices to rationalize usage and reduce cost.
“The area that businesses were most focused was on how to drive more value from data.” Naveen Zutshi, CIO at data software company Databricks, told CIO Dive earlier this year. “Now, they are struggling with operational complexity.”
Data analysts, data engineers and data scientists each use discrete tools and different production pathways, Zutshi said, which can lead to a disconnected data ecosystem. Tagging by function ties ownership to workgroups and gives IT leaders more clarity into spending, he said.
While greater visibility into data operations helps organizations determine who’s using what and how much it costs, observability solutions bring additional headaches. More than 9 in 10 respondents said their leadership team is concerned about rising observability costs and anticipates greater spending scrutiny by their organization over the next year.
Ultimately, solutions require investments, too, and more than 4 in 5 respondents said ROI on observability platforms has not kept pace with spending.
