Test Failure Analysis In Test Automation

Testing is one of the key areas in the Software Development Life Cycle(SDLC). As the DevOps and Agile approach has become popular in modern software development, test automation is becoming the way to test. 

One of the major challenges in test automation is test failure analysis. As the failure might have originated from the application or test script itself. Without traces and sufficient logs, it becomes difficult to analyze the test failure.

 In this article let’s discuss analyzing the failures in depth and detail.

What Is Test Failure Analysis?

Test failure analysis is a process to find the root cause of a failed test. Failure analysis helps the team to analyze and fix the failing tests. If failure originated from the application, team needs to raise the defect to the development board to fix the application issues.

The Importance of Test Failure Analysis

Test automation helps to speed up the development and delivery process by providing fast feedback. A greater benefit comes if challenges are handled well. One of most challenging task in is maintenance. 

Two types of test failures.

1. False failures: The application is stable and working as expected however test automation script is failing due to scripting issues such as milliseconds of delay, hiccups in test automation simulation, etc.
2. Expected failures: The application functionality is broken, test automation script is trying to perform an end-user task however due to an application issue it is failing. This is also known as valid failures. Valid failures are considered a defect. Application failure usually occur due to changes in flow, and changes in the DOM tree or It may be also due to some of the functionalities being broken.

Let’s understand important areas of test failure analysis.

• Test failure analysis helps to understand false failures and valid failures
• It helps to understand the performance bottlenecks which are causing the delay in execution.
• You can isolate the slow-running tests to get immediate feedback.
• It helps to tune the core automation framework to streamline the needs.
• It helps to configure the parallel execution based on the script execution time.

Just like how the developer does the debugging, test automation scripts need to be debugged in the same way. However, to debug you need to know where to start and a high level of failure information. Especially if you are using a continuous testing (CI/CD) pipeline you may end up with a large amount of unrelated logs. So only test result logs, screenshots, and good reporting can help you in this scenario.

Critical Features of Test Failure Analysis

As discussed before there is a lot of information required to analyze and fix the test failures. Below are some of the important Features of test failure analysis.

• Understand the failure area: If the test reports are properly configured it should provide the nearest area where the test case is failing.
• Root cause analysis: Once you identify where the test is failing, you need to drill down to why the test failed. In the root cause analysis, you will be identifying the reason for failures.
• Filtering failures: As we discussed before, test cases can fail due to various reasons in the test script or they may be valid failures. Both failures have to be investigated and need to take appropriate action once the analysis has been completed.
• Actionable insights: Step-by-step approach to drill the reason and fix them accordingly
• Test Artifacts: Test artifacts may provide the logs, videos, network information, screenshots, and optionally may also provide the infra details. All these things help to fix the failures.
• Intelligent Dashboards: The perfect dashboard should have a single point of view for all of your test results.  If it is smart it can also provide information related to unstable tests along with supporting information.
• Enterprise automation tool and reports: Not everyone is interested in everything. For example, the tester might be interested in concrete-level information. The stakeholder might be interested in how the application is performing today. So the enterprise dashboard should provide a different view as per the need.

Test failures in test automation

Many think that test automation is “Write once and run forever”. But none of the test automation tools works in that way. If you need reliable test automation results you need to maintain them. You need to fix the failures as and when required and remove the performance bottlenecks.

Test failures are most common in test automation. So before adopting the best-fit automation tool, you need to evaluate it as per the need. Most of the time while evaluating the automation tool reporting feature is not taken into consideration, but this is the most critical feature when it comes to maintenance and failure analysis.

Most of the tools provide HTML, JSON, XML, etc. types of reports. Remember, more than the reporting structure, the information which the report provides matters. If the report provides accurate and sufficient information for failure analysis, maintenance will be easier.

Let’s take an example of the below scenario

Test Case Name: Search and Navigate to the Testsigma page.

• Navigate to google.com
• Search for testsigma
• Click on the first result which is having a link to the Testsigma page.

Once you execute the script, imagine the script is failing while giving the information “Search and Navigate to testsigma page has failed”. If you look at the failure result, there is not much information, as the test case has many steps. If the tool provides minimal information debugging tests will be difficult. If the test automation tool is intelligent it should fail the test case with accurate information such as

• Where has it failed?
• Why has it failed?
• Suggestions to fix the failures

Tools like Testsigma also provide screenshots and videos that help to understand the failures. Makes it easier to analyze and fix them within a few minutes. Another advantage would be even non-technical people will be able to understand and suggest the fix.

How Testsigma Helps in test failure analysis

The time taken to fix the script depends on how accurate the test results are and its information. If the test fails with one line of failure message, the tester needs to spend a lot of time analyzing the root cause. The sooner you find the root cause, the faster you fix the failures. 

Testsigma understands the pain of finding the root cause. So it has tried to make failure analysis very simple by providing the most accurate, detailed information along with supporting screenshots, videos, and logs.

Let’s understand what are the features of Testsigma reporting.

• It provides reports at different levels such as Testplan, Testsuite, Testcase, and Step level.
• Testcase level report contains the most granular level of information such as exact steps where it failed and the reason for failure.
• The test report also provides different environmental information that you have selected before execution. Such as Windows 11 with Chrome, MacOS with Safari, etc.
• It provides step-level screenshots where failures have occurred

• Optionally, you can also download the log files which provide network and connection information thus it helps for further analysis.
• The filtering option helps to filter out test results in multiple ways
• As mentioned before, not everyone is interested in all the reports so it provides high-level reports with charts which will be helpful to stakeholders.
• You can export the test results in PDF, or JSON format and build your dashboard.
• When integrated into CI/CD pipeline it provides reports for each build making continuous testing easier.

Let’s take an example of a scenario and try to analyze it in detail.

Consider we have a scenario

Test Case Name: Search and Navigate to the Testsigma page.

• Navigate to google.com
• Search for testsigma
• Click on the first result which is having a link to the Testsigma page.

After execution we are seeing the failures in the above script, Testsigma reports look like below

Looking at the screenshot, Let’s find the root cause by analyzing the report

• From the top environment information, we understand that it has failed in Windows 11 with Chrome version 114
• There are 6 steps in total, and the test failed in the 5^th step
• From the error message “The element corresponding to the locator type “XPATH” and locator “//span[contains(text(),’Testsigma1′)]” is not found”. It clearly says that the locator is not found when it tries to click.
• When we look at step 5 we are performing the click action on the XPATH locator “//span[contains(text(),’Testsigma1′)]” however it does not exist on the page.
• If you navigate to the suggested documentation page it clearly says how to fix the issue.
• From the screenshot, it is clear that the failure is not related to page loading or delay or application under test (AUT). It seems like an issue with our automation script.
• From the above investigation, we can derive the conclusion
  • Application behaving as expected
  • Page is rendering is fine
  • There is no such issue related to performance
  • The issue is originating from the locator that we have used.
• You have now two solutions to fix it
  • Record the script
  • Correct the locator which is causing the issue
• In the locator we are using XPATH “//span[contains(text(),’Testsigma1′)]” where we are looking for the text “Testsigma1” but the actual locator is “//span[contains(text(),’Testsigma’)]”. 1 was appended at the end which was now corrected.
• To edit the locator Click on TestSteps and then Click on Edit

• Click on the locator and replace it with the correct locator. Optionally you can Record the Element to correct it.

• Click on Update and Rerun the test
• After fixing the locator you can see the Test is passing

I have explained it in detail so it might look lengthy when you look at it but in the real world, these steps won’t even take 5 minutes. One of the reasons is we could able to find all relevant failure information in place. Importantly, we are suggested with documents to fix them, no need to search for a solution. In this way, any complex failure is simplified by Testsigma, which makes it a developer-friendly tool.

Test Automation Tools with Test Failure Analysis

Almost every tool is shipped with test reporting but the accuracy is what matters most here. Some example test reporting tools for failure analysis are.

Testsigma

A no-code automation tool provides an “anybody can automate” feature. This tool comes with the most granular reporting with screenshots, videos, logs, exact steps, reason for failure, and help documents for fixing them.

Selenium IDE

Selenium IDE is open source record and playback tool. The tool also provides reporting however for non-technical people it may be difficult to understand them. As the error messages are more technical and there is no high-level report to share. If you need detailed reporting you need to manually configure it using one of the supported programming languages

Katalon

Katalon provides both record-playback options and code-based automation capability. However, unlike Testsigma it is not a SaaS tool, so the configuration of reports in a Continuous testing environment can be tricky. If you need to use the Katalon you need to download the Katalon studio locally and then configure it as per your need. 

Test failure analysis in automation can be challenging and time-consuming if you don’t have accurate information. Codeless automation tools like Testsigma provide detailed documents and accurate information whenever a test fails. 

While you hunt for the test automation never forgot to have reporting as one of the criteria for evaluation. 

Continuous testing demands speedy delivery, for that the automation script should execute without any performance hiccups. The artifacts should contain detailed reports and logs which can help you to debug and fix them swiftly.

Frequently Asked Questions

What is an example of a failing test case?

Let’s take an example of the below scenario

Test Case Name: Search and Navigate to the Testsigma page.

Steps:

• Navigate to google.com
• Search for testsigma
• Click on the first result which is having a link to the Testsigma page.

Once you execute the script, imagine the script is failing while giving the information “Search and Navigate to testsigma page has failed”.

The failed test case could be mean that either searching for this page did not work or navigating to this page did not work. It would be on further analysis of the test results that we could get to know the isolated cause. Read more here about Failed Testing.