Avoiding the Spaghetti Architecture
- By Lachlan Colquhoun
- May 29, 2023
For all the commentary and talk about cloud platforms, multicloud and data, a recent global study found that only 63% of the organizations surveyed use cloud services or infrastructure widely in their data architecture and only 34% operate multiple cloud environments.
Given that reality then, it is unsurprising then that when organizations look at their data priorities over the next two years, the single biggest and most common enabler is the wider adoption of cloud platforms.
This is the infrastructure which will deliver what the research by the MIT Technology Review, sponsored by Databricks, found were the three main data priorities. They include the enhancement of data analytics and machine learning, better data management, and expanding the use of enterprise data.
The research project is the result of global interviews conducted with 351 chief data officers, analytics officers, information officers and other senior technology executives, in words the C suite of technology.
Work in progress
The premise for the research is that while data is the priority and widely regarded as the ‘supreme driver of business value’, harnessing it to best advantage is still a work in progress. Many organizations are also lagging behind a comparatively small group of leaders.
The pace of change in data management has been ‘both breathtaking and frustrating’, and while organizations might be ‘spoilt for choice’ by the number of providers in the vast data eco-system, choosing the right ones and optimizing the configuration of the solutions is proving something of a challenge.
Settling on a cloud strategy is also no guarantee of success either. Many organizations have moved to the cloud so fast that they may be simultaneously using on-premises databases, data lakes and data warehouses to handle much of the same data.
“Now I can get insights about our customers, about our supply chain, about how people work that I just couldn’t capture before”
According to the MIT research, just 13% of organizations excel at delivering on their data strategy, and these high achievers are also advanced cloud adopters with 74% running half or more of their data services in the cloud.
This group of ‘high achievers’ deliver measurable business results and are succeeding thanks to their attention to the foundations of sound data management and architecture. It enables them to ‘democratize data’ and leverage value from machine learning.
“It used to be difficult and costly for me to get data about many elements of our customer experience,” Bob Darin, the chief data officer of CVS Health, told the researchers.
“Now I can get insights about our customers, about our supply chain, about how people work that I just couldn’t capture before. We have all the tools to analyze that data at scale, and the cost of those tools is coming down.”
The result is that Darin and his team can develop insights at scale and integrate them as part of patient and customer workflows, creating a more ‘personalized and relevant experience.’
Embracing open standards
Where an organization is in terms of its data success also influences its future priorities. Where 53% of high achievers are focused on advancing their ML use cases, 59% of low achievers put a priority on improved data management.
Underlining the obstacles often posted by legacy, open standards are also the top requirements of future data strategies.
Frustrations ‘abound’ with organizations saddled by legacy, on-premises silos that resist integration, incur high costs, or cause problems from duplication and poor quality.
“If respondents could build a new data architecture for their business, the most critical advantage over the existing architecture would be a greater embrace of open-source standards and open data formats,” the report says.
Not everyone can do that, of course, and continue to operate their business, and so businesses dealing with legacy need to move more slowly and deal with greater complexity.
“Architectures have gotten really complicated, but only because we tend to over-complicate them,” Sol Rashidi, the chief analytics officer at the The Estee Lauder Companies told the researchers.“We do this because we lose sight of what matters most. We too often bring in the latest and greatest in technology and platforms, thinking they will solve the problems.”
“But unless the business is ready to leverage the tools, has the maturity to extract the insights and processes and logic are agreed upon, we’re only adding to the spaghetti architecture,” he added.
Lachlan Colquhoun is the Australia and New Zealand correspondent for CDOTrends and the NextGenConnectivity editor. He remains fascinated with how businesses reinvent themselves through digital technology to solve existing issues and change their entire business models. You can reach him at [email protected].
Image credit: iStockphoto/kuricheva