iData

Health Service Case Study

Rapid Implementation of Data Transformation Assurance solution using iData. Learn how our Technical Testing services delivered a repeatable data migration and transformation assurance solution for a Health Service programme in the United Kingdom.

Get In Touch Book An iData Discovery Call

The Challenge:

The Data Migration Test Team was struggling to keep pace with the delivery of new and updated pLDM and volume of data which was being migrated and transformed. We were tasked with implementing a mechanism to assure that the migration and transformation of data had quality gates and assure that the data transformation rules had been thoroughly tested. The goal was to assure the data was available in a correct format and location in the target database.

The health service customer is a central player in the UK health sector and is undergoing an internal transformation to one of its backbone applications. Testing had become inconsistent and caused a bottleneck when faced with the mass of coverage required to deliver confidence to stakeholders that the transformation process was implemented correctly. The Data Migration Test team’s approach had, therefore, become a risk to the project.

This coupled with a strong desire to increase coverage, reduce costs and increase the speed associated with data migration testing they were keen to deliver an efficient execution capability to its internal customers and project stakeholders.

The Test, Release Management and Domain Leads on the programme recognised the importance of completing data migration testing as quickly as possible, but the traditional manual user interface and SQL scripting approach they adopted was becoming a costly bottleneck.

The requirement from the client was to recover the lost time in testing due to unforeseen blockages and speed up the process around data migration and transformation assurance. Given the critical nature of the data being transformed it was imperative that a solution was provided which mitigates the inherent risk associated with migrating sensitive and critical data in large volumes.

IDS have lots of experience in delivering data migration testing projects and so have felt lots of pain as a result. Due to the difficulty associated with assuring transformed data as part of the migration process, we created a solution called iData.

iData is an in-house developed data transformation and migration assurance engine which we use to automate the process of validating the transformation rules are correct and that data exists in its new target database.

iData uses an automation process to generate template scripts that can be easily edited to drive the focus of each test to assure that the transformation process is assured. This addition of templates lends itself to rapid script creation and faster feedback loops for each data migration cycle.

Compounding the inherent challenge of manual testing was the growing complexity of their testing coverage. The team had an overhead of 20-man days testing to complete for each deployment of the migrated data. Coupled with the heavy involved data related validation of results, this exposed the manual testing approach to human error which could be catastrophic for the organisation.

Faced with the increasing cost and complexity of their manual testing approach, the programme benefited from the support of ids to help deliver a leaner, more cost-effective process through our specialist iData migration transformation automated assurance capabilities.

The Solution:

Efficient Automated Data Assurance. Implementing The Solution.

Given that the project had already been subject to significant delays, it was imperative that testing was completed on time, without any reduction in quality and coverage.

Knowing the data migration strategy and design, we decided that the most efficient approach would be to focus on a staged implementation of iData and execute our testing at each cycle of migration. One of the biggest concerns to the project was losing any patient records, and if any were lost during the migration process, at which point the record had been dropped.
We implemented iData and appropriate scripts at each stage. This provided assurance that each transformation had been assumed to be correct and provided an audit trail for any lost data and at which point the data had been dropped.

With a clear understanding of the costs and return on investment opportunities available, we began the process of establishing an automated data migration assurance solution for the execution and re-execution of tests that enable the transformation rules to be exercised every time a Transform and Load took place.

Using iData’s inbuilt learning capabilities, we edited the template scripts in-line with the transformation rules which had been stipulated by the project team. This was over 250 transformation rules that we had to reflect in our scripts.

There had been limited success with testing migrated data previously, but with our background in data migration testing coupled with iData, it was felt that using specialist help to define and implement a standardised approach would prove decisive.

The Outcome:

Eliminating the testing bottleneck, cutting costs and increasing coverage & quality.

iData delivered a Data Migration and Transformation assurance solution

iData enabled the programme to identify and rectify transformation errors across 330,000 records.

iData captured lost records and assisted in locating at which point those records had been lost. iData provided a mechanism to build test assets ahead of time enabling us to be ready whenever the transform and load cycle took place.

Significant time is saved and faster feedback on the state of application quality.

Reporting of the executed was integrated into Jira for tracking, auditing and reporting.

The client now has the benefit of a fully functioning, repeatable migration assurance test pack working as part of a continuous delivery model

Enabling the client’s data teams to Build-Deploy-Test as often as required, resulting in much more robust Export, Transform and Load practices.

Increased confidence in the data being deployed was given to the senior stakeholder teams.

As this testing had previously been completed manually, it could take up to 4 weeks to complete. With iData as a solution, we have calculated that the entire assurance process could be completed in less than 30 minutes and provide 100% coverage across 330,000 records.
If they had chosen to execute the same level of automation using manual techniques the cost and time required would deplete the test budget without getting the required assurance on the quality of the product/application.

Having achieved this, the program has increased the volume of testing as well as improvements in:

Transform & Load reliability Overall Quality Better on-demand reporting metrics

Efficient data transformation testing Improved Testing practices and cohesion with the data team

Through the specialist implementation of IDS’s iData Migration and Transformation assurance solution, the client now benefits from a considerable saving in staff utilisation, freeing up skilled resources to focus on manual and exploratory testing or other areas of concern.

The quality of testing is also greatly improved with the client having greater confidence in the application under test as the possibility of human oversight is eliminated.