In-sprint test automation for regression testing at DevOps speed

“Enter UI Automation.”

UI automation is typically done after all the manual testing is completed – which means at a lag of at least one sprint. While, the Agile and DevOps approach suggests to complete the testing, including automation, at the speed of development.

If not automated, you still need to do regression testing corresponding to all the changes done in the sprint. Sometimes, multiple times. If done manually, this could take a lot of time. And just that is why teams take the risk of ignoring regression testing corresponding to the new features at the end of the cycle.

So, what is an alternate strategy to UI test automation?

An agile-friendly approach to automation which will allow teams to perform In-sprint automation.

Testers need to automate user stories within the sprint alongside development by automating Unit tests, API tests and part of the UI tests but just leaving the UI mapping. Once the UI is close to done, all the participants should be working more closely to complete the UI mapping. This ensures the tests are completely automated along with the development and ready to run through continuous integration.

By the time the code is complete and ready to be committed, there are unit tests, API tests, and a handful of automated UI tests. The feature is all set for automated regression testing and then to move to the production. With this when the feature is built, the new code change has been explored and complete test coverage with the new feature included achieved and had rounds of bug fixing.

Regression testing is now faster and the development flow will be smoother. Instead of manually testing all the regression tests to identify the impact of the new changes, Continuous Integration and testing will give you a clear picture about the state of the product and it can be shipped.

With this the test coverage problem with in the sprint will also be solved. Instead of a person performing similar tests in 5 different web browsers, a script can run in all of the browsers in parallel while a person looks for far more important problems.

This strategy and approach are crucial for teams that want to deliver software faster in Continuous Delivery and DevOps.

Test Automation in Definition of Done (DoD)

As the release cycles get shorter in Agile and DevOps, it becomes necessary to automate the testing so that these new features/enhancements are easily tested as part of regression testing in the upcoming cycles.

Automating every test case of a user story isn’t easy since there is hardly any time between dev complete and test complete. Therefore, most of the time automation tasks are getting spilled over to future sprints since the development is getting dragged till the last moment.

As the dev team adds a new functionality to the software, the automation team might still be working on an old feature and cannot advance at the same speed as development. This delay can also be due to application unavailability in some cases.

This is less productive because the developer has to go back and forth discussing with the automation team. Hence, most of the times, a few activities are pushed to the next sprints.

Also, in short release cycles, there is not enough time to automate and hence the coverage of the new features will be less.

The user story should be implemented, tested and the test cases should be automated to be claimed as “done”.

1. The biggest challenge is finding a tool or a test automation framework that allows for In-sprint test automation, like being able to automate unit, API testing and UI tests for different application types like, Web mobile and native apps and API services.
2. Team members are executing sprint in mini-waterfalls which is not the right approach and cannot do things in parallel.
3. Like we discussed before, Test Automation done with the legacy tools demand not just the automation engineer’s time but that of developer’s and QA’s. During short development cycles, it is difficult to buy their time for automation.
4. The lack of a Unified solution will be a big challenge to automate multiple apps as part of the same project. It takes a lot of time and effort to work on multiple tools to automate different application types.
5. Most of the tools/framework is not flexible enough to enhance or optimize quickly for the new automation requirements or enhancements if there are any.
6. Teams may not have all the skill sets to automate testing within a sprint. Most tools demand a steep learning curve and a deep technical understanding.
7. The right set of tools(with important elements like test environments, continuous integration environment, automated build, servers etc.

1. Better collaboration between dev, automation and QA teams to release faster with no one being left behind to work on the old features. Everyone can be on the same page.
2. You don’t have to spend a significant amount of time testing a new feature to understand its impact on an existing feature towards the end of the cycle.
3. With the new feature being automated along with code complete, there will be good enough coverage for the new features right from the first build.
4. Tests are ready at the early stages of the sprint, so developers can identify test scenarios clearly and try to cover those tests during development.

Checklist for In-sprint test automation and DevOps Implementation

1. Test automation framework that can support end-to-end testing including unit, API and UI testing
2. A framework that supports web, windows, mobile, web services, database testing
3. Right Test automation Strategy and approach by considering the tools and resources that you have chosen
4. Team with the skill set that matches the framework and the chosen tool
5. DevOps infrastructure setup with the right set of tools (i.e. Jenkins, Docker, etc)
6. Build automation is in place and the continuous integration pipeline is ready with continuous testing tools
7. Reliable and scalable test environments to match with your dynamic testing requirements
8. POCs and feasibility reports and analysis that backs the chosen tool and frameworks.

Testsigma’s approach

Here’s a practical guide to how In-Sprint Test Automation is done with Testsigma.

1. Doesn’t depend on the application’s availability: Testsigma allows you to get started even before the application is fully available.
2. Natural language scripts: The reusable test scripts are developed using simple natural language and involve no complex coding.
   Refer Scriptless Test Automation for Agile and DevOps
   
3. Scalable test environments availability: A stable Test environment with valid test data sets to run tests automatically throughout the continuous delivery pipeline.
4. Unified platform to support API, UI end to end testing: A unified, extendable and secure Continuous Testing Platform, Testsigma help testers connect thousands of web, mobile devices with different device configurations, in one go.
   Check outWhy Unified Automated Testing Software for Continuous Testing in Agile and DevOps
5. A framework that supports web, windows, mobile, web services, reports and database validations.
6. Native integrations required for Continuous Delivery and DevOps