The Hidden Problems of Bad Data
- By Paul Mah
- December 04, 2022
Bad data can cause a variety of problems for any organization, resulting in inaccurate analysis, poor customer experience, and poor business decisions. We all know that. But bad data often culminate in other problems that are more insidious – and just as harmful to the organization.
Here are some of the hidden problems of bad data.
It takes up valuable resources
Speaking at a roundtable discussion at the Informatica World Tour held in Singapore last month, Peter Ku, the vice president and chief industry strategist for Financial Services at Informatica, shared an anecdote of a chief executive officer who was insistent that his organization’s data was perfect.
It turned out that the executive’s opinion was informed by the accurate reports and dashboards that he was perusing. Unbeknownst to him, however, a significant amount of effort was devoted to cleaning and correcting errors in the data by a team of financial analysts who spent most of their time on this task.
Beyond the obvious issue of how a lack of data management processes often results in bad data, businesses must awaken to the fact that bad data is often obscured by well-meaning efforts to clean them. In this instance, bad data was invisible to a top executive by the simple fact of employees acting responsibly and endeavoring to produce good work.
In this case, highly qualified employees hired to move the needle on innovation or to increase profitability ended up cleaning data and correcting mundane data mistakes. The solution? Implement appropriate data governance and tools to enforce good data, ensuring that valuable resources are not wasted on correcting bad data.
It erodes trust
We all know that poor data can lead to subpar decisions. But the biggest danger of poor analytics or inaccurate reports is an erosion of trust that puts employees off data-driven decisions. Moreover, employees who choose to ignore data are unlikely to tell you that, which simply amplifies the problem.
Imagine meetings where everyone nods their head over charts and dashboards but continues to base vital business decisions on their “gut feel”. This becomes a self-reinforcing cycle when the deliberate ignoring of data culminates in atrophied data skills, which leads them to further reduce the use of data.
To build a strong foundation that builds trust, start with small pilots that favor data accuracy and leverage it to develop competency with data. Also, some data vendors think that the huge amount of data that businesses currently grapple with simply cannot be managed manually.
According to them, manual data quality management cannot scale to today’s volume, and necessitates the implementation of automated processes for auditing, assessing, and cleaning their data. Of course, these vendors have something to sell you – though I think they might have a point here.
It puts customers off
Ever received a marketing email that got your name wrong or reference information that is years out of date? How did you feel? Unfortunately, it can be next to impossible to fix certain types of “bad” data.
People change jobs and relocate. They get married and their priorities change as kids come along and parents grow old. Their preference might vary over time. How are organizations to know?
While this is true, I would also argue that good data processes along with a strong data culture can go a long way toward ensuring that relevant data gleaned at all customer touchpoints are properly captured and updated.
As I wrote previously, establishing a strong data culture is difficult. It requires embedding data into the very fabric of the organization, and an aspiration to get everybody to see the value in data and proper training, and continually inspiring everyone to leverage data for better decision-making. It is also non-negotiable.
To be clear, it is not a one-time achievement, but a continuous process. And not fixing a poor data culture is akin to helping a hoarder clean up without addressing the underlying issues.
As the pace of transformation picks up around the world, the onus is on business leaders to bring about the necessary focus on data to help their organizations thrive. And as noted by a participant at a recent CDOTrends roundtable, the best time to change was yesterday. If not, there is no time like today.
Paul Mah is the editor of DSAITrends. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose. You can reach him at [email protected].
Image credit: iStockphoto/RomoloTavani
Paul Mah
Paul Mah is the editor of DSAITrends, where he report on the latest developments in data science and AI. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose.