Adopting an Agile test approach to reduce regression test time

How adopting an Agile test approach reduced regression testing time by 70%

The client: A £1.2 billion online B2B retailer
Industry: Retail
Technologies: Java, Selenium, Cucumber, Gherkin
Ten10 services: Test strategy, automated testing

The client

The client is a global B2B retailer of electrical and electronic components, tools, testing equipment and related items.

The project

The client’s frontend website serves customers throughout the world, across 32 different countries. Historically the client had a robust manual test process in place which was heavily dependent upon contractors. However, the frontend system changes rapidly in response to; new products, customer requirements, and new features, so the client wanted to adopt new ways of working, to allow them to move to an Agile approach and enable software to be released faster and more frequently.

Challenges

Despite an appetite for change, the full value of some of the new techniques were not being realised by the client. An automation regression suite had been created and was running on 1 of the 8 applications, however there was very little confidence in the quality of the information and results produced by the test pack. Specifically the entire pack would produce a single binary pass or fail upon execution. Confidence in the results was so low, that the same tests were then run manually to confirm the results.

The client was facing challenges around the management of new features and fixes into live. Each time a change was made the whole software estate had to be deployed as a single unit, regardless of whether a change had been made to a particular application or not. This increased the volume of testing as all 8 products had to be regression tested following a single code release, irrespective of how trivial the changes to it.

A further consideration was the need to deliver the product suite across 32 different countries. This created a significant regression backlog, either with functionality not being tested or testing taking a very long time.

An example of version control[/caption]

It is also worth noting that there had previously been a number of projects undertaken to modularise the code base, allowing the monolithic code to be broken down into a number of smaller, discrete components that could be tested and released independently. Key restrictions included; the complexity involved in testing all the modularised applications to a level of suitable functional coverage, testing across all the different markets and across all the supported browsers, and the overall length of time it would take to complete. Due to these complexities and associated risks, the historical projects to separate code in modules had previously been unsuccessful. An additional challenge was the client’s 3 week sprint cycle; development was undertaken for nearly 2 weeks with the final week devoted to a full regression test before release. This significantly reduced the effectiveness of the development team by one third and the testing team by two thirds.

The project goals

The first goal of the project was to successfully modularise the code to enable faster releases. After this a primary goal of the engagement with Ten10 was to successfully adopt more of an Agile development methodology; incorporating testing throughout the sprint and allowing the client to spend the full 3 weeks developing and testing new features rather than just 2 weeks out of a three week sprint.

Java Automation pack code sample

The client wanted to be able to see fast feedback when changes were made to fix bugs, so another objective set was to improve the management & reporting and visibility of their defects, as well as to give early feedback of the application’s current state. There was also an overall requirement to significantly reduce costs as well as to improve efficiencies through new testing techniques and approaches.

Modularisation to support Agile testing

Ten10 provided strategic advice on where to focus effort and resource to ensure the initial 6-week project to modularise the code was successful, whilst also deploying a team of 4 testers to work alongside the in-house team of developers. During this period all development was focussed on the modularisation of the code.

Using pre-existing and approved documentation for the test scripts, it was agreed that the initial goal should be expanded to include automation of test scripts for each module.

For three of the eight applications the original specification was followed, however for the other five applications it was identified that additional requirements were needed, so these were scoped and created.

The modularisation project was delivered successfully, on time, with the first application ready to go into full regression testing 3 weeks into the project. 64 critical defects were found from the automated tests prior to go-live. Following go-live only 3 early life support defects were discovered, all outside of scope. Furthermore the discovery of those critical defects was left shifted and all were picked up in the integrated window.

Following successful deployment into live the team then implemented a solution around the automated test pack. The full regression pack is now run 3 times per day, with html generated reports displayed on big screens in the agile delivery team office space. This visual reporting gives the client the fast feedback they require. If there is a failure it is possible to drill down into the details there and then (including screenshots, test code and stack trace) for rapid insight into the issue.

Internationalisation of regression testing

The regression testing automation packs also had to be reprovisioned to ensure they would meet the global requirements of the 32 different countries, with each market running on a number of warranted versions. In order for the packs to work they were refactored so they were not reliant on any labels or names in English.

Additionally Ten10 undertook to remove deprecated CSS on the webpages. In order to ensure this didn’t have any negative impact, a tool was created to compare screenshots pre and post change to check for any rendering errors, and was run across a matrix of browsers and languages using Browserstack. If differences were found above a configurable error threshold, a heatmap was generated which showed the changes at a glance, and which was then sent to a tester for manual checking.

Cost-effective software-developers-in-test

In order to meet the cost-efficiency requirements, Ten10 was able to heavily leverage its Graduate Training Academy to provide the dedicated, Softwaredeveloper-in-test resources required by the client at a competitive onsite cost.

Running since 2013, Ten10’s Graduate Academy takes the brightest and best young talent and trains them in Agile development methodologies, testing, technology, performance, automation and mobile testing. Ten10’s bespoke training, delivered by experienced practitioners who are also certified ILM trainers, ensures our graduate testers are equipped with a balance of theory and effective, pragmatic test techniques. Through this model we were able to ensure the client had the skilled resource required for the project whilst still meeting their cost-efficiency targets.

The results

Ten10 worked with the client to successfully help them transition from a manual, functional regression testing process to an agile delivery methodology, left shifting testing and providing automated regression testing that could be executed with confidence. As a result the full regression testing time has been reduced from 1 week to 1.5 days.

Automated tests are now running 3 times per day continuously – providing constant reporting and feedback on all the applications, including on a variety of browsers across all the regions automatically. The greatly improved integrated and Agile approach to testing has been complemented by the introduction of user-friendly visual reporting, ensuring key stakeholders have access to information in a timely manner.