iTesting

Central Bank Case Study

Reducing the cost of regression testing by more than 94% using an iTesting interface automation strategy. 

Learn how iTesting delivered a 94% reduction in the monthly cost of regression testing for a Central Bank in the United Kingdom. 

Get In Touch Book An iTesting Discovery Call

Follow us

IDS iTesting Logo

The Challenge: Developing a leaner, more cost-effective testing process

The bank is a central player in the UK banking industry and are undergoing internal changes and increased control around quality assurance practices. This, coupled with a strong desire to reduce costs associated with testing, means they are keen to deliver an efficient execution capable of satisfying its internal customers. 

The Test & Quality & Release Management and Domain Lead Test Automation Manager at the bank, recognised the importance of completing regression testing of the application quickly. However, the traditional manual approach they adopted was becoming a costly bottleneck. 

The requirement from the client was to build an automation process that would scrape data from an Excel exported XBRL file into a 2D table. Then it would use a 2D table setup and pivot tables in Excel, which connects to Cube (Microsoft SQL Analysis Server), to compare the two tables and report any failures accordingly.

Compounding the inherent challenge of manual testing was the growing complexity of their testing coverage. The team had an overhead of 15 man days to complete for each monthly deployment of the application. Coupled with heavily involved data related validation of results exposed the manual testing approach to human error which could be catastrophic for the organisation. 

Faced with the increasing cost and complexity of their legacy testing approach, the bank has requested the support of iTesting to help deliver a leaner, more cost effective process through our specialist Microsoft Visual Studio and Team Foundation automated testing and continuous delivery capabilities.

Tablet computer displaying graphs and bar charts

The Solution: Automated Regression Testing

Counting the cost of manual testing

Working closely with Test management and business unit, the first task was to establish the cost of their UK based regression testing: 

Day rate cost of a single test analyst

£400

Total days to manually execute regression

15

Cost of regression testing

£6,000

Frequency of regression testing (monthly)

1

Total spend (monthly)

£6,000

 

Ideally, you should execute regression tests each time you build and deploy an application. For a typical software application with active development procedures this should be held on a monthly basis.

In the case of the bank, the equivalent costs of their traditional manual testing become even higher:

Cost of regression testing

£6,000

12 iterations of manual regression testing

£72,000

 

With a clear understanding of the costs and return on investment opportunities available, we began the process of establishing an automated solution for the execution and re-execution of tests that will contribute to the overall application lifecycle management (ALM) process. 

There had been limited success with automation previously, but with our background in automated testing it was felt that using specialist help to define and implement a standardised approach would prove decisive. 

Proof of Concept Demonstrates the Benefits

To reduce the risk of failure, iTesting delivered a successful proof of concept to demonstrate the feasibility of the approach using a complete ALM automation platform consisting of Microsoft Visual Studio 2015 along with TFS and Microsoft Test Manager.

Intelligent testing was implemented alongside the development team to develop the correct strategy around storing the automated testing results in Team Foundation Server. 

Once we had a base solution in version control, with our environments set up to be reusable as well as our data set, we began to focus on the build and deploy process. 

As Microsoft Test Manager is in use, we had a requirement to have the new automation solution fully integrated for traceability through the manual test cases, which in turn had links to user stories. 

For each new test case created using Axe, the client can import test documentation into Microsoft Test Manager which then becomes manual test cases. 

Each time regression tests are executed using the new automated test solution, results are imported into Microsoft Test Manager and all test cases are updated automatically with a pass or fail status. 

To assist the bank with test management and reporting we set up a dashboard that had multiple relevant metrics, displayed progress and highlighted risks on VS Online. These charts, graphs and tables are updated automatically and current data is readily available to all management on demand. 

The Outcome: Eliminating the testing bottleneck, cutting costs and increasing quality

 

Our solution was to create a function format using C# which is built together in sequence using Axe.

The report is shown in Excel and a summary of it is available on the Axe report.

This was a fully bespoke implementation, and outside of the normal use of Axe, we also provided a IE dialog which showed the actual progress of the run in a progress bar format.

The bank now has the benefit of a fully functioning, repeatable regression test pack working as part of a continuous delivery model.

Delivery of the solution uses Odin’s Axe Automation tool using the embedded UFT coding functionality across the internal banking system.

Enabling the bank’s development teams to build-deploy-test as often as required, resulting in a much more robust build, better deploy practices, and increased confidence in the application being deployed.

As this testing had previously been completed manually, it would take up to 3 weeks to complete. With this solution, we have calculated that the entire test process could be completed in just 1 day and there would be no need to transfer any knowledge to new resources as the code handles user setup.

If they had chosen to execute the same level of automation using manual techniques the cost would be £72,000. In a like for like comparison they are making an immediate yearly saving of £67,200 using iflex. 

Having achieved ALM, the bank have increased the volume of testing as well as improvements in:

Build & deployment reliability

Overall quality

Better reporting metrics

Automated reports

Through the specialist implementation of a test automation solution, the bank now benefits from a considerable saving in staff utilisation, freeing up skilled resources to focus on manual and exploratory testing or other areas of concern. 

The quality of testing is also greatly improved with the bank having greater confidence in the application under test conditions as the possibility of human oversight is eliminated.

Next Steps: Discuss your data needs with an IDS testing specialist. Contact James Briers to arrange your initial consultation briefing:

James Briers