What is a Regression Test Automation Framework?
How to choose a framework for regression testing?
A Framework For Checking Regression Test Selection Tools
Regression Testing: Strategy Framework, And Best Practices
Factors To Choose The Right Regression Testing Tool
Popular Regression Testing Tools
Cloud Based Regression Testing Benefits
Faster Regression Testing With In-Sprint Automation
Testsigma — Smart Automated Regression Testing Framework
When a team takes up the task to automate their regression test cases, they first need to define what test automation framework they plan to use. A test automation framework defines the structure that will be given to the test cases when they are automated. Advantages of defining a test automation framework are as below:
All the test cases are created using a standard structure
The test cases are easier to create because of a predefined structure that needs to be followed
Test cases are easier to maintain as they are automated following a standard structure
A test automation framework is scalable in the long run which means testers can add more test cases as needed
While preparing a test automation framework, most of the integration related tasks are performed beforehand thus while automating test cases, testers can focus on automating the test cases rather than worrying about integrations
A test automation framework could be prepared from scratch or can be chosen from the ones already available in the market.
Before deciding on what test automation framework to go with, below are a few steps that should be followed, these can vary according to the team and its needs:
1. Figure out the type of application that needs to be automated.
Below are a few examples:
A desktop website
A responsive website
A mobile application
A website and a mobile application
A desktop application
A server-side application
2. Do a review of the test cases that need to be automated.
This is to understand the type of test cases that need automation, this becomes the basis of creation/selection of a test automation framework, below are the types of test cases that could need automation depending on the product and requirements:
Test Cases focused on testing UI
Test Cases for testing APIs
Test Cases focused on testing backend configurations
Test Cases for testing database communication
Test cases for testing workflows and functionality
3. Decide on what approach should be taken for automation, below are a few that are prevalent:
Data-driven test automation framework
Keyword-driven test automation framework
Hybrid test automation framework that uses both data-driven and keyword-driven test automation strategies as and where needed for best results
4. Decide on the execution conditions that need to be met, the most important ones are:
Environments that the test cases will be executed on
Where will the test cases be automated
Where will the test cases be hosted
5. Figure out who will be automating the test cases? This is important to choose the right automation tool:
If the team that is going to automate the test cases knows how to code, an automated testing tool that supports programming like Selenium can be used
If the team that is going to automate the test cases does not know how to code, the automated testing tool chosen should not need heavy coding. Such tools are very easily available in the market. You need to search for scriptless automated testing tools.
6. Decide on the features needed in the framework in addition to the creation and execution of test cases, below are a few that are usually needed:
Easy and efficient reporting of the test execution
Seamless Integration with CI/CD tools
Parallel Execution of automated test cases
Easy to debug
Efficient error reporting
When a framework for automated regression testing is defined or selected, execution of the automated regression testing is it’s most important part. Automated test cases are created on one machine or on the cloud but the execution of these test cases should be possible on multiple environments according to the type of application being tested and the targeted customers.
The automated regression testing tool should support the execution of automated test cases on all needed environments and there should be an option to add more environments for execution if needed.
In the long run, the number of automated test cases is bound to increase and with it the total time taken for execution of the automated test cases as well, so it is important that the automated testing framework supports parallel execution of the automated test cases.
Below are the steps that form a part of the framework for the execution of automated regression testing:
Selection of test cases for execution: This is an important step for the effectiveness of test execution and more so for automated regression testing. Regression testing is done after a change is made in the code of an application. The test cases should be selected based on the change and what areas it could impact. The framework should be created keeping this in mind. For eg, there should be a way to group test cases testing a feature and its impacted areas.
Execution of selected test cases: For an automated testing framework, easy execution of test cases is a basic requirement. But for an effective framework for the execution of automated regression testing - the test cases should be executed independently to ensure that test cases don’t fail to execute because of failure in other test cases.
Execution Report analysis: A test execution report should be sent soon after the test execution is completed and should be the one place where all essential information is provided for understanding the test execution status. This report is not only for the testers and developers to work on but is also for the managers who need to go through the test execution results to decide on the status of next deployment.
Archive of test results: The test reports should be kept in one place and there should be a way to aggregate the results to understand the error causes and frequency. A review of the errors reported and the areas impacted over a period of time can help to understand the areas that need special focus and can be used to plan for rework.
Management of test cycle: A test suite is only useful if it is updated and pruned frequently. This step involves the addition of new test cases in accordance with new changes in the application and removal of test cases that are redundant or no longer useful.
Regression Test Selection is an important step before the execution of regression testing after any code change. Regression Testing is the most effective when only the areas that are impacted by the code change are tested and any other unimpacted areas are not tested.
RTS(Regression Test Selection) tools compute dependencies and affected tests using analysis at different granularity levels. Whereas STARTS and Ekstazi find test dependencies and affected tests at the class level, Clover works at the method level.
Other RTS tools are:
Method-level RTS tools (e.g., Chianti, FaultTracer)
Hybrid class-and Method level RTS techniques like HyRTS
Hybrid class-and-statement level RTS techniques like DejaVOO
Statement-level techniques like Pythia
Module-level techniques like GIB
Tools that capture dependencies across JVM boundaries like RTSLinux
Regression Testing Strategy is formulated depending on a few factors as listed below:
The number of test cases: The number of regression test cases depend on multiple factors, if an application is in its initial stage of development, the number of regression test cases will be on the lower side, while later they could reach a high number. Similarly, if the application in itself is a small application with limited functionalities, the number of test cases will be on the lower side
Type of application: What percentage of test cases can be and should be automated also depends on the type of application. The different types of applications are UI-focused applications, applications with complex backend for different types of configurations, Mobile-only applications, server-side applications.
Testing method: Regression testing method could be manual or automated.
Regression testing approach: The Regression testing approach depends on above factors and can be either complete regression testing or partial regression testing. Partial regression testing is done if impact analysis for the changes in an application is done before starting the regression testing and only the impacted areas are tested. If after a change of a set of changes, all created regression test cases are executed, it is called complete regression testing.
The end goal when constructing a regression test framework is that the test automation should be achieved with minimum possible efforts, time and cost. Keeping these factors in mind, a regression test framework should have below features:
An easy to understand structure for regression test cases and test suites.
The test cases should be kept up to date at all times.
Regression test cases should always be graded according to priority such that execution can be planned according to the time allotted for regression testing.
The regression testing results should be compiled in a report before sending to the stakeholders and should be easy to understand.
All the regression testing reports should be stored in one place and should be accessible for comparison and charting results against the time of execution. This helps in determining weak areas in the code if any.
To achieve the best results from regression testing with minimum cost, having an appropriate strategy and a fitting framework are essential. To round it off, below are few best practices that can help achieve precisely that:
The effectiveness of the regression testing strategy and quality of the results being delivered should be regularly reviewed and any necessary changes should be made to improve the process and in effect, the results.
The regression test suite should be kept up to date at all times. All test cases that render themselves redundant should be regularly removed and any new changes in features/requirements should be regularly included in the regression test suite as test cases.
To save on time, effective impact analysis for every change should be enabled.
The regression test suite should be structured properly for better understanding and execution
Regression Testing should be performed consistently and at regular intervals to avoid any slips during the software development lifecycle.
If there is any area in the code that has a high risk of failure, it should be given special attention during regression testing.
There are few factors that make a tool a good choice for regression testing, few of which are as below:
Test cases can be created easily
Test case maintenance is easy
Complex test cases can be automated as well
To find a gap that exists during the requirements phase.
The tool supports test case execution for the type of application you have.
The structuring for test cases and test suites is easy to understand and maintain
Either the tool should support integration with good reporting tools or should have its own reporting mechanism
The tool supports the execution of the test cases on the supported browsers and devices.
The tool should be able to integrate into your CI/CD pipeline seamlessly
Based on above factors, there are few tools that are very popular as regression testing tools :
Selenium: Free and supports multiple programming languages, one of the most used regression testing tool and has a great presence on public forums to help with any issues. Only works for websites.
Katalon Studio: Free and supports multiple programming languages, and the company offers great support. Works for websites and mobile applications.
Ranorex Studio: Commercial tool, with support for record and as well as support for multiple languages. Works for websites and mobile applications.
TestComplete: Commercial tool, supports scriptless automation, gives testers the ability to create automated tests for Microsoft Windows, Web, Android, and iOS applications.
Testsigma: Commercial tool, supports scriptless automation, AI-Driven Test Automation Ecosystem for Web, Mobile Web, Android & iOS Apps, and API automated testing.
Easy access from anywhere, anytime: When regression test cases are hosted and executed on the cloud, the test cases become accessible to the team from anywhere around the world at any time around the clock. Thus, the limitations pertaining to a location and time is no longer there.
Enhances collaboration and enables better continuous testing: The same repository of test cases can be accessed by multiple people from multiple locations at the same time, thus, it enhances collaboration and also promotes continuous testing.
Scalable use of infrastructure: As the test cases are hosted and executed on the cloud, the infrastructure in use can be increased and decreased according to the state of execution. The infrastructure in use can be increased if the number of test cases to be executed is more, while it can be easily decreased if the number of test cases to be executed is less.
Minimum cost: The pricing only reflects the amount of infrastructure used. Thus, you don’t need to pay for the times when your test cases were not being executed and the infrastructure in use was at its minimum.
A sprint is a specified period of time in an agile methodology. It is during this time that a team works on some predefined work and aims to deliver it in a working condition. The work in an agile methodology is broken into multiple sprints. A usual sprint for software development has development tasks for a few new features and corresponding testing tasks. It may also include bug fixes for some other features.
When new changes are done to the code of an application, regression testing becomes essential. If manual regression testing is performed for all the changes in a sprint, it will take a lot of time and resources. Thus, a solution is needed to make regression testing sprint friendly. The solution is “In-Sprint Automation”.
To implement "In-Sprint Automation" successfully, below are a few steps that should be followed:
When following the agile methodology, tasks are broken into smaller tasks that could be completed in one sprint. Same should be done for the regression testing automation task. Every sprint, the task of automation and execution of regression test cases should be taken up.
Every sprint, it should be ensured that the regression test suite is up to date.
Every sprint, automated regression test suite should be executed to make sure no existing functionality is broken after the changes are done in a sprint.
Testsigma is a cloud based automated testing tool. This is a tool perfect for regression testing because of the features listed below:
Test cases can be automated with the least amount of effort because test cases need to be written in simple English using NLP. Thus, the learning curve is minimal.
No programming language is needed during test case automation. Test cases can be created in record time.
The tool has AI at its core and aims to maintain any failures that happen because of changes in the functionality.
Test cases are hosted and executed on cloud. Thus, no restriction related to location or infrastructure.
A fully loaded lab that has all popular browsers and devices that you might need to execute your test cases on.
Supports parallel execution so even if the test cases are many, the time taken to execute them will not increase as much.
User friendly test and error reports
