Why Analytics Is So Hard
- By Paul Mah
- August 20, 2022
Data has become synonymous with modern life. And businesses everywhere are waking up to its incredible potential as they leverage data to become leaders in their industries. But as many organizations quickly realize, data-driven success doesn’t come by wistful thinking or corporate declarations, but by focused, sustained investment and effort.
And even then, some that have made the right hires and purchased the requisite tools soon find their data initiatives stuck in a rut. Why is finding success with analytics so difficult? And what are some strategies that organizations adopt to get ahead?
The challenge with data
In my conversations with chief data officers, analysts, and data scientists, the impediments to building a data-driven culture are varied and many: a lack of talent, the difficulty of implementing change, lackluster support from the rank-and-file, or senior management that see everything as a solution they can purchase rather than an organizational-level transformation.
But take a moment to pause and consider the digital landscape today, and it is quickly evident that making sense of data is hard even when most of these impediments are removed. At the heart of this challenge is the sheer volume of data that must be managed, and the velocity by which they are continually created.
Depending on who you ask, the world produced around five exabytes of data a day last year, and this is expected to rise to a staggering 463 exabytes per day in 2025. Yet many businesses are trying to handle this deluge of data using the same kind of tools and thinking as they did in 2010.
For instance, if your data is still managed and supplied by your IT department, then your more enlightened competitors are probably running circles around you. Instead of ferrying the data around with buckets – such as emails or spreadsheets – they need to start putting the advanced digital plumbing in place to autonomously funnel and clean (more on that later) the data to where it is needed.
Getting visibility into your data
Ever decluttered your storeroom to find expired food products or purchases you didn’t realize you had? Now imagine a storeroom the size of a large warehouse and swap out your regular household items and excess Covid-era toilet rolls with voluminous digital data.
Then clone whatever data you had a random number of times and start working on some copies while leaving others untouched or accessible only by some departments. This is the reality that businesses face in the real world: Haphazard silos of data with no clear source of truth.
These silos of data could stem from disparate solutions such as CRM and financial systems deployed over the years, or standalone databases established to serve long-forgotten needs. Then there are social media feeds, streaming data from newly installed IoT systems, and more.
The very first step is to gain visibility over all these data. Only then can organizations have a hope of organizing it, cleaning it, and setting up the systems to keep it that way. Neglect any aspect at your peril, especially cleaning the data – a recent survey found that data professionals spend a staggering 40 percent of their time evaluating or checking data quality.
Some organizations would have you adopt their cloud-based data platforms to curate and catalog your data. The cloud does offer various distinct advantages for managing data, though it is not the only workable option. When choosing a solution that works, it is worth noting that the use of disparate, non-interoperable data tools might render data lineage unachievable.
A concerted push to use data
Finally, the entire organization must be involved in the data-driven transformation from the onset for it to succeed.
On that front, a recent report in the Harvard Business Review recommends a three-step approach centered on assessing an organization’s existing capabilities, gauging the consensus of employees, and working to fix weaknesses and enhance strengths.
The first step should start with a set of core employees and other internal stakeholders representing a range of management levels and business functions. The idea is to conduct a workshop to accurately assess the organization’s data readiness.
The workshop can also expose hidden areas of disagreement, allowing the next step which is to build consensus among key players and work out where differences in perceptions of capabilities may exist. Only then can the business make a concerted push to use data, fix areas they are weak in, and optimize their performance in areas where there is still room for growth.
On my end, I must say that I fully agree with Harvard Business Review’s summation of the data journey: “Every company is somewhere along an open-ended journey to achieve data and analytics superiority. While there is no final destination… there will always be more to do.”
Paul Mah is the editor of DSAITrends. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose. You can reach him at [email protected].
Image credit: iStockphoto/lzf
Paul Mah
Paul Mah is the editor of DSAITrends, where he report on the latest developments in data science and AI. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose.