Written by Bjarne Keytsman
Data is the backbone of accurate and confident decision-making, ensuring data quality is non-negotiable. After all, the best algorithms and dashboards are only as good as the data they're built upon. Our observation is that a lot of data residing in organizations is of low quality [1], due to never being important, limitations of legacy systems or other reasons. Meanwhile, dashboarding environments and analytics, even the advanced ones, are becoming more and more commoditized. The opportunity lies in having high quality data to feed your decision making through dashboards or even generative AI applications.
However, isolated data quality projects can lose momentum, often resulting in half-solved issues. To maintain engagement and drive results, adopting a use-case approach anchored by clear business objectives—be it a dashboard, a predictive model, or user-centric reporting is crucial. This not only guarantees value delivery but also fosters collaboration. The journey might be challenging, but with a focus on value from day one and a collaborative approach between business and IT teams, it's worth embarking on.
Consider the journey of one of our clients, who wanted to become more data driven. One of the goals for their digital transformation was to improve the reporting practice by creating a dashboard environment. They wanted to replace their Excel reporting with intuitive and modular dashboards that presented data visually and facilitated data exports. Crucially, these dashboards were designed to be as close to real-time as necessary.
Simple right? However, as we delved deeper into the development phase, we quickly realized that the data did not meet the desired quality standards. Several metrics can measure data quality, including its completeness, accuracy, consistency, validity, timeliness, and integrity. In our journey, we had to work with disparate databases that contradicted each other, encountered information gaps, and faced issues with data needing to be updated. Problems rooted in years of legacy systems that were edited and expanded upon. Those systems eventually ended up doing tasks they were not designed for. Due to data migrations, data fields were mismatched, nuances were lost, and parallel out-of-sync databases led to duplicate or conflicting records. Without solid data governance and management practices, these challenges often snowball, resulting in glaring data quality issues.
We had to fix the foundational issues before we could start building the updated reporting practice. Our initial step was to demonstrate the potential of the dashboards. Using preliminary data from both business and IT teams, we crafted prototype dashboards. While these prototypes weren't exhaustive, they effectively showcased the potential of the applications, sparking user interest and eventually motivate users to address the data challenges.
Once a consensus was reached on the prototype, we transitioned the dashboard to production. This involved integrating it with the company's ERP (Enterprise Resource Planning) or SQL databases. Our responsibility was to bridge the gap between business needs and technical execution, owning the data model and making sure it was both functional and logical.
The production phase created the opportunity for data quality refinement. Generally, discrepancies came from two areas: either the data source was flawed, or the extraction method was incorrect. For example, a status field should have had five options, but due to multiple versions of the same option, there were effectively 50. But we did not always choose to fix the data quality at the source. Instead, we implemented transformations during data extraction. By iterating fast in the build phase and making the new dashboards available to the users, we incentivized the users to report data quality issues when encountered. This iterative approach ensured that issues were addressed progressively, always with an emphasis on delivering value in the dashboards that were actively used.
In addition to these technical adjustments, we also supported establishing a data organization, tailored to the company's needs. This involved structuring a data organization around key domains, cataloguing essential data, and refining the tools and processes used for data extraction, transformation, and loading. Concurrently, we emphasized change management strategies to cultivate a strong culture of data quality throughout the company.
Improving data quality can be a complex task that requires the support of people in the organization. By using a use-case approach that clearly shows the benefits of better data, you can get more people on board. As advanced analytics and generative AI tools become more common, the quality of your data can set you apart. In this changing world, having high-quality data can be your edge.
Want to know more about how you can exploit your data in meaningful ways? Contact Bjarne Keytsman or Koen Vanbrabant.
[1] A concerning 82% of companies are making decisions based on outdated information as highlighted by a 2022 Dimensional Research study (https://www.fivetran.com/blog/new-research-the-high-cost-of-stale-erp-data)
Comments