Test Execution: A Comprehensive Guide With Examples ￼
Considering the current software size, software testing can be seen as a separate entity from SDLC, where SDLC is referred for development and STLC for testing. When a phase in the software industry becomes so big, no matter which domain it belongs to, it is bound to be divided into multiple stages of its own for better execution and efficiency.
Testing is no exception with popular phases such as test planning and test summary reporting evolving to focus more on theoretics than implementation through test scripts. In between these two ends comes a stage that explores the requirements of the project by running commands and actions on the application through test automation tools and frameworks. This practical exploration and verification stage is called test execution and the aim of this post is to reason its position among its peer stages in the testing lifecycle.
Table Of Contents
- 1 What is test execution?
- 2 Importance of test execution
- 3 Test execution prerequisites
- 4 Activities required for test execution
- 5 Methods to perform test execution
- 6 Test execution states
- 7 Defect lifecycle in test execution
- 8 Test execution priorities
- 9 Reporting in test execution
- 10 How to increase test execution cycles for better testing?
- 11 Test execution in test automation with Testsigma
- 12 Conclusion
- 13 Frequently Asked Questions (FAQs)
What is test execution?
Test execution is a process of executing the test cases on the application to determine its functional and non-functional parameters with respect to the requirements. Test execution is often preceded by test planning and test analysis. However, these are recommended but not mandatory steps, and therefore in this post, we will always assume these steps are considered in the testing cycles.
First, we create a plan that includes script planning, test scripting, pseudo code creation, direction to move forward, etc. This plan is always referred to while moving forward with test cases. Once this is done, the technical part of the plan is analyzed by senior members of the team for approval while non-technical such as flow and requirements of the project can either be analyzed by the same members or business analysts. Then, we can proceed towards writing the test cases after which test execution takes place. This is done on the application ready from the development team for release to the customers.
While it seems that test execution is an automatic process of test execution, it is not the case practically. Test execution has been the sole element of the testing phase in the earlier days when the software was small. We could just write test cases and execute them on the go. With so many stages now surrounding the test execution phase, it needs to adjust accordingly to all of them and that includes preparations, setup, planning, and much more. Discussing all of these elements can help us understand the in-depth relationship between test execution and other testing parts. The end result, however, highlights significant changes in the process which glorifies test execution and makes it an important element in the cycle. These results are discussed in the next section.
Importance of test execution
No testing phase can be completed without test execution. But we can only reap certain benefits when we focus our energy on this process and optimize its individual parts. The major benefits out of all can be defined as follows:
All requirements are evaluated
Focusing on test execution means we are focusing on each requirement associated with the project. This way we can keep track of how many requirements we have covered and how well we have done that.
Quality of the application is improved
Test execution helps us focus on each test case that corresponds to various areas of an application. They can be as large as database operations to as small as page load timings. With proper time given to each test case (especially in case of failure) post-execution, we improve the quality of the application significantly. This, in turn, will boost business growth as well.
Enhanced test data gathering and reporting
With test execution in place, we gather the data about each test when the execution is completed. For instance, which test failed, why it failed, bugs associated with that failure, etc. As we keep releasing the application in subsequent sprints, this data can be used to track down the progress of not only the testing team but also the development team.
Other than these, test execution helps streamline the process and increase the efficiency of the complete testing cycle.
Test execution prerequisites
Now that we know the meaning and importance of test execution in product release and STLC, we can connect the dots with other stages to understand how can we make this process successful and without any hitches. For this, we must take care of certain prerequisites related to the connected stages with respect to test execution.
Knowledge of team members with respect to the project
While this point stands true in any aspect of testing, test execution can fail terribly if test cases are not appropriate. This is often seen when team members do not have a clear understanding of the requirements and start the planning too early. This mistake should be avoided by giving the team members appropriate time with doubt resolutions that can result in proper test planning.
Design should be complete
The importance of test design is reflected in the process that is followed after its completion. One of the immediate processes is test execution which takes references and templates from test designs directly into practical implementation. If the test design is incomplete, we lose this reference link and team members start test execution according to their own respective experience. This, as we can imagine, will vary based on their past work and knowledge. It means that as we go forward, we keep digging holes that will result in heavy monetary losses once they seep into production. Hence, for a smooth test execution, a well-planned and completed test design is required.
Tools and frameworks should be configured
Test execution will require tools and frameworks to perform actions on the applications. These tools and frameworks should be compatible with the test cases and libraries and plugins used. All this configurational setup should be done before test execution. If not, the execution shows failures that may look like test case failures instead and will be documented in the report. A misconfigured tool will also take additional time during corrections done while test execution is in place.
Monitoring systems should be set up
The need for monitoring arises after test execution has taken place. However, these systems take into consideration the behavior of test execution while it is running and post-execution as well. Therefore, it is advised to keep them already setup before test execution, including a small test run to test their performance and efficiency.
Automate post-execution processes
Apart from monitoring, there are other processes that are run post test execution and do not consider the execution state. For instance, test reporting is a process that takes test execution results into consideration and creates a report out of it. This process can be automated using tools that support such features as Testsigma. When all these processes are automated, we can analyze test execution, provide quicker feedback and rectify anything if required.
Working on test execution prerequisites is not an optional but a mandatory task. It helps create a streamlined process and a smooth run of tests. It also helps pinpoint the defects in the system when failures are observed.
Activities required for test execution
Test execution requires certain testing activities for a better test execution to be performed by the testing team and help validate the quality of the application. This job is performed by the QA team and involves testing and defect management tasks divided into the following five low-level activities:
Activity 1 – System integration testing
System integration testing, as the name suggests, involves testing the application with its modules integrated as a unit. It is black box testing and majorly aims at interacting with the application to verify its compliance with the system. QA team tries to find as many bugs as they can with respect to multiple layers involved in the application. These bugs are then documented for the next activity.
Activity 2 – Reporting the defects
The system integration testing activity helps explore a lot of defects in the application, especially if it is big and complex. These defects cannot be just written in a single line and must be documented to keep records and track the status. For this, we use bug-tracking tools or frameworks that can integrate such tools. The bug tracking tool can contain information such as a defect in summary, steps to reproduce, the system on which the bug was observed, expected fix, severity, assignee, etc. They may also contain features such as “commenting” to provide a better chronology of events and we can reflect back on them in the future.
Activity 3 – Mapping the defects
Once the defect is found and documented, it needs to be mapped to all the entities it relates to. This depends on how a team is performing testing and test execution. They can map the defects to the appropriate test cases and/or they can also target specific requirements, test design elements, etc. The main motive of this activity is to keep the records as much detailed as possible. The more data we have for each defect, the more thorough analysis can be done and the cycles become shorter leading to quicker application delivery.
Activity 4 – Re-testing the failed tests
In activity 1, we are more interested in finding the bugs and quickly documenting them for further analysis. Therefore, once a defect is found, we do not try to make too much effort to put it under a microscope. Once we have documented and mapped the defects, we pick them one by one for re-testing. The aim of this activity is to focus on each bug and evaluate the reasons for its failure. This includes changing scenarios, environments, inputs, and a lot more so that we know exactly what went wrong and why. All this goes into the test reports and defect mapping documents.
Activity 5 – Regression testing
We have completed the defect finding and documenting process till activity 4. By this time, all the test cases that have passed are documented, those that have failed have their reasons and scenarios, and there is nothing ongoing in the testing process. But since we made a lot of hustle in the previous four activities, including changes of scenarios and inputs, we need to be sure that the application is as stable as it was when the previous version was released. This is achieved through regression testing.
For more information on this, using a test automation tool, you can navigate to this guide: https://testsigma.com/automated-testing
Once these activities are completed, we can move ahead to perform test execution using different ways.
Methods to perform test execution
Test execution deals with the execution of test cases on the application. However, it is not the process of running the test cases on the application always. When we say whether we have performed test execution or not and the answer is “yes”, we might mean the actual test case run or we might mean performing other types of execution that do not involve individual test cases. These methods are discussed below.
Running test cases directly
The first method is the most direct and obvious method to perform test execution i.e. by running test cases. The test cases can be attached to the application and the “run” action can be executed to run the tests one by one and produce the final result as “pass” or “fail”. After this, if the test cases have been associated with other elements such as reporting, bug tracking tools, management tools, etc, they are updated and verified.
Running test suites
A collection of test cases (preferably logical) is called a test suite. An alternate method to perform test execution is to run test suites rather than test cases. This gives us the flexibility to run only a segment of test cases rather than all. This saves a lot of time if the primary intention was to run a few test cases only. A test execution also facilitates parallelism as the test cases are divided logically and any two test suites can be considered independent of each other. Therefore, we can run multiple test suites together reducing the time of execution.
Record results without execution
A riskier method of performing test execution is to not perform the actual running of test cases but directly document the results with respect to them. This can be achieved by either comparing the test cases with the already executed cases or the default scripts.
Run test cases using various testing types
Test execution can be done in a variety of ways. Manual test execution demands manually executing each test case whereas automated test cases are taken over by a test automation tool. Apart from this, tests can also be executed in regressions, pipelines such as continuous testing, and through APIs. Recently, codeless test automation has also joined this group and is used extensively to perform test execution. The only requirement is that the tool should support codeless executions such as Testsigma.
Test execution can be performed frequently without any manual intervention by scheduling the test runs (cases, suites, or records) at a particular time. All modern tools, frameworks, and pipelines support this feature. Alternatively, testers can also use CRON jobs to set the schedule manually, which can be saved on the server with respect to days and times of the week.
Apart from these options, if as a tester, you come across other methods to perform test execution, let us know in the comment section.
Test execution states
The current state of test execution as a process is determined by one of the many pre-defined states. These states, although exhibit a high-level generalized meaning, the actual meaning is defined by the tester and the organization only. A lot of the post-test execution processes may depend on or get affected by the current state of test execution. For instance, if the test execution state is “Running”, we cannot initiate reporting as of now. The most popular test execution states are as follows:
Passed: A test case is considered in a “passed” state if the actual result matches the expected result. If all the test cases in a suite pass, the suite is in a “passed” state.
Failed: Conversely to the “passed” state, a test case is in a “Failed” state if the expected result does not match the actual result.
In Progress: An “In Progress” state determines the currently ongoing test execution. This is an intermediate state and not an end state like “Passed” or “Failed”.
Incomplete: When a test case fails to execute, it comes into an “Incomplete” state.
Error: An “Error” state determines that the test case has thrown an “Error” that may be due to any reason. For example, a typing error in the script will throw an error (such as “Not an attribute”), and the test execution will end from there.
Paused: When the test execution is “Paused” in the middle of the run deliberately, it comes to the “Paused” state.
Inconclusive: When the result cannot be determined or the final state cannot be determined among the above-listed ones, this state is called an inconclusive state.
Deferred: A test case that has been intentionally not run in this release but kept for execution in later versions is tagged as “Deferred”.
Blocked: When any preconditions are not met for test case execution, the test case is tagged in the “Blocked” state.
Partially Blocked: When the “Blocked” stage is reached during the execution of test cases, it is termed as “Partial Blocked”.
Not Run: If there has been no action taken on the test case in any way, it remains at a stage of “Not Run” which most of the time is a default stage as well.
As a note, organizations need not use so many stages all the time in test execution. They can omit a few among them or add some of their own. It depends on the type of project, team, and the organization’s testing standards.
Defect lifecycle in test execution
One of the main aims of performing test execution is to point out bugs in the system and document them for either exploring them later or forwarding them to the developers. In between these two end stages of finding a bug and closing it for further actions, it goes through multiple cycles or stages in between. These stages help define the status of a bug in test execution through which we can conclude test execution quality as well.
Read more about the defect/bug lifecycle here: https://testsigma.com/blog/bug-life-cycle/
Test execution priorities
The test execution process is actually a process of executing test cases in one way or another. Whatever factors affect the test execution, do affect test cases as well. So when it comes to prioritizing the individual parts of test execution, we are actually prioritizing the type of test cases in their behavior, target, or scripting.
A lot of the test cases come into the backlog category due to a variety of reasons. A simple example is a failure in the current sprint. When a failure occurs, it is not always possible to rectify it in a day or two. If it is highly critical i.e. something that breaks a crucial functionality, it is taken care of. However, if that is not the scenario, the test cases generally turn into bugs that we discussed in the defect lifecycle. They are important to be noted because they can magnify or even catch the eye of the user.
This is just one example of what goes into the backlog. There are lots of scenarios that fail a test and the rectification is delayed. This needs to be of high priority to make sure the application works as expected and test execution is not hindered in the future.
A high risk involved in a test case would mean it corresponds to a functionality that is critical in nature. Any failure in such test cases could stop the release and force everyone to engage in this area. This is not what anyone would want at the last minute of the completion of a sprint. Hence, test execution prioritizes high-risk test cases to be executed first so that we can get the defect status earlier and fix them quickly.
Depending on the application, a few platforms may serve as a higher priority than others. For instance, an application made to be run on servers will almost always be run on a Linux machine because that’s what most of the servers use. In such cases, test execution priority is higher for Linux OS. Hence, we prioritize cases with a certain platform and seek to resolve related issues quickly.
Finally, there are certain requirements that are of higher priority than others. They may expand to many test cases, features, platforms, etc. These requirements may correspond to critical functionalities or are conveyed by the client themselves or by the higher management. Whatever the reason could be, we always prioritize test cases based on requirements as they are important. This becomes a bit of a gray area for other prioritization in test execution sometimes. For instance, we may have a low priority for Windows OS based on our server-based application, but if there is a requirement and we are supporting Windows for that requirement, that needs to be tested well.
With this section, we must also note that test execution priorities are not always mandatory. Practically speaking, they are almost always a concern when the project is huge and test cases are expanding into thousands. The modules are also divided into multiple domains and a lot of tests are executed by different team members at once. In such cases, a test run can take as much as 12 to 15 hours which is not a good sign as we will come to know about critical issues the next day. So, we prioritize test execution to be ensured about certain things first in maybe a couple of hours and then go for a full run. This saves time and provides assurance to the developers and testers that the project release is on the right track.
Reporting in test execution
The last stage of test execution, or any other phase of testing, is reporting. If you would count any one particular area which is focused the most after the testing is completed, it will be reporting. And not only the team members, a report is distributed, stored, archived, and always kept on record for anyone in any team no matter what the team’s work or background is. So it becomes a responsibility to make reports highly intuitive with quick understanding and easy-to-read elements. With the sections and elements listed below, we can make sure that we produce a high-quality report easily.
A unique identifier
Start with a unique identifier for each report. It’s like an employee ID which helps in identifying each employee as we know two employees can have the same name and may even belong to the same place. Similarly, the same product may have a lot of similar test execution reports between two releases, especially when features are not focused. A unique identifier helps keep records with respect to each report and is easier to convey to other people who wish to explore a few of them.
The unique identifiers are generally not understandable for a human and are made for searching and record-keeping purposes. For instance, tr_230_122 can be a unique identifier. A summary accompanies this identifier which is a short description of the report that helps the individual understand the content he can expect from it.
A quick result summary
This section can provide a quick result summary to give a gist about the results an individual can expect. For instance, “The test execution identified three failures and network time exceeding threshold limit by 750 ms.”
Everything executed and fetched data from the results goes into this section. This is the most exhaustive section with a very detailed explanation of each subsection. This section also includes graphs and other pictorial representations to provide a quick glance that will take time reading paragraphs or tables. This section majorly concerns the testers as it can become technical and this is the reason we provide a quick summary and evaluation for non-technical people to understand the gist of this part.
A comprehensive assessment is detailed and only testers can conclude results from it. To make things easier for people from all the domains, results can be evaluated and the evaluation can be described in this section. This section also becomes important because of the various methods we use in test execution. The evaluation should conclude all these runs by giving a high-level view and a low-level view and the reader can pick any section of their liking.
All the activities performed in the test execution cycle are described in this section. These activities can correspond to prerequisites done, what kind of environments were used, what kind of runs were used, why these runs were used, and everything else a tester feels is important.
The execution cycle needs to be approved by the team manager for the release to go through. This is an integral part of a report, especially for higher management who will directly seek this section to get an explanation. Needless to say, a negative approval can stop the release and push the testing cycle back for further analysis.
These sections of test execution reporting should be able to cover all the data required for present and future analysis including data pattern creation. If you feel any other section should accompany the existing ones, let us know in the comment section.
How to increase test execution cycles for better testing?
The integral part of test execution cycles is the actual execution of test cases itself. If any way we can increase the test execution process, we can increase the test execution cycle. There are many benefits of investing time in this process. The most important is we can execute the test quickly and wrap things faster than before. As a result, we can release the product quickly and shorten the testing cycle. In addition, as a complementary result, the cost of the projects can be reduced and the money can be spent on other high-priority things.
One of the best methods to increase test execution cycles for better testing is to adopt cloud technology. A cloud-based infrastructure does not require any local system to set up, execute, and maintain the test execution. So a tester can perform testing from any location and through any system. They just need to have the correct credentials.
Cloud-based technologies are also optimized in their workings. To achieve this on-premise, we might need to spend extra bucks by hiring a separate team that can dedicate their complete time to infrastructure maintenance. Moreover, from time to time, new technologies emerge in the market and we may need to modify the infrastructure accordingly. When we go “cloud-based”, all this is taken care of by the organizations and all we need to do is focus on test case development, test design development, and test execution. The tools can even take care of the procurement of real devices, and creating detailed and enhanced reports automatically.
Test execution in test automation with Testsigma
Lastly, we cannot wrap up the post without talking about the most used method of test execution – test automation. The process of automating the test is an integral part of test execution. It is quite impossible to execute thousands of test cases manually on so many platforms again and again during every release. However, when we turn towards test automation, we don’t turn towards an execution method but a lot of its characteristics that differ around with different tools.
One small example of this characteristic is the support for different testing types. A test automation tool may perform API test execution smoothly but may not have any option for data-driven testing. Similarly, language support plays a major role in test automation tools, third-party support, integration support, and the list goes on and on. A wrong test automation tool choice can spoil the test execution process and can risk the quality of the application. To help you save from scrolling through hundreds of such options, why not start with a cloud-based tool like Testsigma for a better and faster test execution cycle?
Testsigma is a cloud-based test automation tool that believes in a no-code (also known as codeless) test automation system where programming languages do not become a barrier between a tester and his testing skills. Completely based on the cloud, Testsigma is open-source, free to use, and covers all the major test automation types such as data-driven testing, API testing, web testing, mobile app testing, and regression testing without writing any code. It uses English-based language to understand the intent of the test which is easy to use and does not require any programming skills from a tester.
Testsigma also incorporates self-healing mechanisms to automatically heal the test cases based on the changes done to the user interface of the application. From a test execution perspective, it seems to be one of the perfectly fit solutions that take care of test execution cycles, priorities, properties, and reporting without any local resources.
Test execution can be made simple with tools like Testsigma that are unique in their own way but still provide an easy learning curve and comfortable environment to the testers, new or experienced.
In between test design, test scripting, and report analysis is a process of executing the test cases. This process is often considered an atomic action with all the things working in unison. However, when we put this process under the microscope, the different areas we explore from it surprise us with how deep the test execution process goes and its impacts on the application even further ahead in time.
This post focuses on test execution and the components that provide its importance in the software testing lifecycle. This includes preparing for the test execution, activities involved in the process, methods to execute test cases, the journey of a defect from its exploration to rectification with constant updation of states, and priorities, and finally, reporting everything for record and sharing purposes. With this, we end this post and hope that the discussion done in this post will provide a different angle to test execution and help you make it faster and more efficient.
Frequently Asked Questions (FAQs)
How do you write a test case execution?
Test case execution requires documentation from the inception of the test case to its reporting. This includes designing the scripts, writing pseudo code, documenting the characteristics of the test case such as objectives and goals, writing actual scripts, setting up the environment, executing tests, following guidelines, documenting defects, and finally reporting it.
What are the levels of test execution?
The levels of test execution depend on the type of testing we are following. In general, we perform test execution on four levels:
- Unit test execution.
- Integration test execution.
- System test execution.
- Acceptance test execution.
What is test execution with an example?
Test execution is a process of executing the test cases on the application to verify its functional and non-functional properties and requirements. It is an integral part of the testing lifecycle and helps keep the application quality in check and uncover defects in the application at an early stage.