What do we mean by data quality and why is it critically important for all businesses?
With the rapid growth in digital transformation, data is now central to most, if not all, business decision making, which means that good quality data is a critical requirement to ensuring the long term success of any commercial or public sector organisation.
So what do we mean by data quality? First thing to know is that there isn’t one simple definition that applies to all datasets except to say that it needs to be fit for purpose in terms of the application and the specific operational task it is being used for. But how do you know if your data can be regarded as good quality? Here are some basic objective criteria that you can use to assess your datasets:
- Accuracy – Are all the data elements correct? E.g. are all the customers’ names and addresses spelt correctly?
- Completeness – Do any of the records have missing values or are there any records missing? E.g. have important dates not been added or have some address details been omitted?
- Up-to-date – Has the data been updated within an acceptable timeframe?
- Relevancy – Does the data meet all the requirements needed for the intended purpose?
- Consistency – Are there any mis-matches between different parts of the dataset? E.g. Is the customer data the same in the payment records and the CRM?
The next obvious question is, with potentially hundreds or thousands of records how can you check that your data meets all these criteria and more importantly how can you ensure the integrity of the datasets to prevent issues arising in the future? The answer is to put in place a rigorous set of Data Quality Management policies and procedures that have full corporate buy-in and are based on the principle that “prevention is better than a cure”.
The problem arises particularly for organisations that are coming late to the DQM concept, which can mean that there is a distinct possibility that they would fail on at least one of the criteria. In such cases it is critically important to invest resources into making any essential repairs to the existing datasets to create a baseline of quality data that can be reliably built on going forward. Adding good data to bad is just defeating the objective and wasteful of precious resources.
To help you get started we at ids recommend breaking the task down into three basic step
1. Analysis
First task is to establish what and where the issues are in your current datasets. This can be done using powerful tools like our own iData Data Quality management solution profiling feature that can quickly provide an understanding of what needs to be fixed and how to do it.
2. Improvement
Setting about resolving any issues can be a daunting and time-consuming task but once again there are many specialist software tools available that can be used to automate much of the process. These tools can quickly remove incorrect or duplicated data entries, merge closely related records or modify the values to meet specified rules and standards.
3. Control
Probably the most important step is to put in place a data quality management framework that sets out the policies and procedures that all users and corporate stakeholders are expected to follow. This also includes putting in place an organisational structure that incorporates clearly defined roles and responsibilities needed to ensure standards are maintained in all parts of the business.
To summarise, Data Quality Management is now a fundamental requirement that enables businesses to stay competitive and is needed to underpin successful operations that deliver sustainable long term growth. Getting this aspect of your business right can be a complex procedure that involves juggling many moving parts to ensure you have the right rules and controls in place to maintain data quality standards and avoid any future issues. If this is new to your organisation and you feel you are going need to some expert guidance to support you through the process ids has a team of experienced data quality consultants who would be happy to hear from you.
Do you have any questions, send us a note at: marketing@intelligent-ds.co.uk
