Test optimization techniques in software testing

In 1980, Martin Hellman introduced a concept of time-space tradeoff. This concept states that when we tend to decrease the execution time of an algorithm, we need to increase the space it takes on the system and vice-versa. This endeavor of a programmer in any domain is termed “optimization” and the resulting code is an “optimized” version of the previous one. While optimization has generally been referred to as time reduction, this is not true. Since 1980, we have had exploration in various areas of computer science. As of today, we can work on time, memory, disk space, bandwidth, and even the power consumption of an algorithm. This means if we can bring down the value of any of the resources used for an algorithm, we are optimizing in one way or the other. And the best part is, whatever path we take, it ends up minimizing the costs of the project.

In this post, we take up the topic of optimization in the field of testing. Test cases, like any other algorithm, require optimization to run in minimum time and reduce costs. This need not be in the actual code itself and as we move forward, we will explore the “non-programming” areas that would have a direct impact on the optimization of the overall execution of tests. But first, let’s see what optimization means when it comes to testing.

What is test optimization?

Test optimization is a process of effectively bringing down the time of test execution and ultimately the costs associated with it. However, this should not be done at the cost of accuracy, efficiency, and quality of testing that existed before optimization.

Test optimization is a very generalized term in a very vast space. While optimizations in the code and execution cycles are important, an equally important part is optimizing processes that lead towards it or originate from it. This means, the processes conducted before coding/scripting and after that should also be optimized with a similar zeal.

For instance, assuming a team of testers works on a project and executes the test cases sequentially. Being a small project and only a couple hundred test cases, the team does not face any problems in the process. However, with time, they will definitely need to backtrack and analyze when the time for test execution will multiply as the project increases. A good optimization technique here is to opt for parallelization at this stage itself. This will reduce the time, maintainability, and overall costs associated. As you can see, this test optimization did not involve any code but just the execution strategies adopted by the team.

Why should you optimize your testing cycles?

While the minimizations in time execution and costs are the primary and most visible reasons to invest time in optimization, there are other reasons as well that may snowball in the future and tend to be problematic for the team if not taken care of.

• Test optimization reduces maintenance time which could prove to be a major time investment in the future.
• Helps eliminate redundant and useless code.
• Helps eliminate “illusion” in various parameters in testing. For instance, it may seem like test coverage is high due to extra code but in reality, it might not be the case.
• Optimized processes are always streamlined and organized which increases the team’s working efficiency.
• Optimized tests always provide an increased return on investment to the organization.

These benefits are lucrative enough to motivate us in optimizing current scenarios or always keep this in mind while creating new ones.

Test optimization techniques

Jumping on optimizing our testing phase without a plan can lead us to wander around without any direction. This direction is achieved by learning about the techniques involved and the areas we can explore in our journey of test optimization.

Expand testing in different domains – Shift left

Conventionally, testing has been done as a subsequent process of the development phase. Testers test the recently developed code in various phases such as functional testing, data-driven testing, and regression testing. However, this strategy has a major problem – the overall process becomes hectic and more time-consuming. Since the codebase is large, it will take time to find defects in the code. Then, these will be raised to the developers who will again take up some extra time for the same reasons. This cycle repeats multiple times. What if we could somehow implement testing on a smaller code instead?

The best path to take for this scenario is to involve testing when the code is smaller i.e. at the start of the coding phase. When testing is “shifted left” to the graph that represents the timeline of software development, we get interesting inputs from the testers. 

These inputs can shape the development phase to be more efficient. But more important than that, once testers are incorporated inside the development, they get access to a small code that needs to be tested. Small code means easier debugging and quicker to relay it back to developers which is the ultimate goal for optimization in this case.

Expand testing into sprints – In Sprint

In the earlier technique, we brought down the time of execution by providing a smaller code base to the tester. But what if there is no time left to provide feedback to the developer and test again? For this issue, we turn towards in-sprint automation. In this process, we involve the testers in all our sprints. Earlier, testers used to get the code when the sprints were on the verge of ending. This would put them under pressure to complete everything within time which always used to prove a stretch for the team. And what if there is a bug? Do we have time to repeat everything when only two days are left for code freeze?

In-sprint ensures that testers have a voice when tasks are planned so that we not only plan till the end of development but till the end of release. This gives more time to the testers and eventually a better quality software to the users.

Choose the best tool

The definition of a “best tool” is always shaded grey. There are some tools that are extremely popular and the teams can commit the mistake of choosing the same tool considering it to be the most optimum choice. What if the only feature that tool does not have turns out to be important for one of the requirements?

Choosing the best tool is not about the features it offers but how many features resonate with the team and project requirements. If the tool does not support a language preferred by the majority of your team members, that tool is not perfect. 

Test optimization technique of choosing a tool is extremely important and the team should always brainstorm when it comes to this choice. A good tool can facilitate all the processes required in the future while a bad choice may increase the workload and time to execution.

Testsigma, is a codeless test automation tool that lets you automate your tests for web, mobile, desktop and APIs, from the same place. The tests can be created and executed 10x faster and the learning curve is very low too.

Try to mix real devices

Real device testing makes use of real devices to test applications especially when we deal with native mobile applications. Instead of this being a widely accepted fact, a lot of organizations do not opt for real devices because of multiple constraints:

1. Real devices are expensive which a lot of companies cannot afford.
2. Read devices demand constant maintenance.
3. A company needs to procure multiple devices of the same type but with different configurations (for example different Android versions) to test completely. This will increase the expenses even more.
4. Real devices keep on releasing very frequently from various manufacturers. So it is not a one-time investment but a constant associated cost.

The organizations instead opt for emulators and simulators which have their own disadvantages. The problem is that we might not see a bug on an emulator and pass the application for release. A post-production bug is something that no company can afford. 

To optimize testing with minimum expense, we need to change our strategy to a mix of both. There are some tests that work perfectly with emulators such as GUI tests and testing during the development phase. For functionality testing and verifying integral parameters, we can use real devices via test automation tools like Testsigma that provide a runway for testing areas as well as devices. All the overheads are managed by the platform itself.

Incorporate parallel testing

When teams start small, they do not focus much on strategy but on getting the work done quickly. Since the project is small, everything takes lesser time, and the team faces no blockers in the process. One such strategy is sequential testing i.e. testing one thing, waiting for its completion, and then beginning the next test (or type of test). As the project scales up, this process will take a huge chunk of the team’s time which may result in delaying deadlines very frequently. 

 Parallel testing is an alternate way of executing tests where non-related (independent) tests can be run together as their sequence of execution does not affect the output. 

Parallel testing often requires multiple systems to avoid consuming the same resource and putting a load on test execution time. This is a feature that should be supported by the automation testing tool you have picked for the project. A good example is Selenium Grid.

Organize review sessions

The best way to understand whether all the team members are writing good quality code with appropriate coding standards and have gathered deep knowledge of the product is to review the code they write. The code review process is followed in the development cycle as well. Apart from understanding the team’s knowledge, the code review process identifies critical bugs in the code. Since the review is done by a senior member, they can identify the regressions and if any piece of code could initiate a bug or not.

Code reviews are also a great test optimization technique in software testing to explore more scenarios for the same use case. Again, the experience of a senior reviewer helps brainstorm these scenarios which eventually leads to exposing more bugs in the code and increases test coverage as well. Code review is considered an integral test optimization technique today and in some organizations, it becomes mandatory to review the newly written code by a senior before pushing it for production.

Document everything

As we mentioned in the beginning, test optimization does not necessarily mean we will always look toward the code. A great example of this is documentation and our attitude toward it as a team. 

Documentation is a lengthy process and if we see it in terms of an application’s journey within a sprint, it does not have any real value. This is the reason a lot of the time documentation is either postponed or ignored completely. But if we look beyond the sprint, the process of documentation is highly valuable and saves a lot of bucks for the companies.

Consider a scenario where test cases are written but there is no documentation that documents the tests, their behavior, pattern, and coverage for the application. The tester who wrote those tests left the company and now for the next sprint, we see a couple of failures in those tests. We assign this task to another tester of the team who will first understand what are those tests about, their impact on the source code, whether they are written correctly or not, understand the writing style, and much more. All this time and effort could have been saved if there was a document to read and refer.

Documentation not only provides help for a new member but also a reference for guidelines and standards to follow. If everybody follows strict guidelines, its impact can be seen in the test cases leading them to optimize automatically.

Understand your code

Finally, as a tester, one thing that will take you as a professional and the tests written by you far ahead is to know and understand the code. By this, we don’t mean to understand the logic but each line of code you write to implement that logic. For instance, a simple example of a line of code in a test file can be as follows:

Thread.sleep(1000);

This code will sleep the thread for 1000 milliseconds. But is this the best method to apply here? We don’t know. It depends on the context of the test. You must have heard about the method “wait” which looks like a sibling of “sleep” as it also halts the execution of the test. It can be used as:

obj.wait();

In cases where two or more methods look similar in their working remember that if they were similar, there would not be two of them. Determining which method to use depends completely on the scenario and the underlying logic. If we take the example above, wait and sleep are two object and thread-based techniques to halt the execution of a program. Sleep tells a time till which the execution is halted which in our case is 1000ms. “wait” on the other side, halts until some process wakes it up by “notify”. 

These two use cases will decide which method to use. For time-based synchronization cases, sleep is often preferred while for thread-based synchronization, we prefer to wait due to their low-level workings with the scheduler. As a tester, always be aware of what methods are used and if the scenario is perfectly fit for them. There is no preference or dominance of one method over another in such cases.

These are a few test optimization techniques that can be applied by a tester by keeping in mind the common mistakes and strategies adopted by teams. This list can be expanded further by exploring various other areas affected by a testing team. We invite such techniques in the comment section that have been devised as a result of your experience in the field which will help the community optimize their cycles further.

How to prepare a test case for test optimization?

While optimizing all the processes around testing is necessary, the major impact does come from the test cases you write. 

Focus on functionality rather than line coverage

Line coverage is a tempting thing for a tester and more so for managers to showcase that their team has covered all the lines of code. While this may give an illusion of a better test cycle, it is a lot more complicated than it looks.

One function may have a lot of complex logic to implement functionality and obviously, all the lines will not execute in any one scenario. So we try to create multiple tests and mold our test logic to each scenario possible. This is an extra work with more added time and complexity. A complex test case can eat up more time for a tester in the future than it would while writing the test cases themselves. This will hurt our optimization endeavors.

The best solution is to focus on only two things while creating test scenarios:

• The project requirements.
• The product functionality.

The test cases should cover all the requirements in the best way they can. They should also include the functionalities of each module with multiple scenarios. This little adjustment will provide a good quality application without investing too much time in writing test cases.

Know your methods

As already mentioned in detail in the last segment, understanding each line of code you write as a tester is extremely important. Each method in a programming language’s library comes with a purpose and a use case. Knowing exactly what use case a particular method is perfect for will serve your returns long-term. There is nothing more optimized in a test case than a function matching its use case. 

Always keep an eye on the coverage

100% test coverage is not important but that does not mean we completely ignore this part of test writing. Coverage gives us an idea about what parts of the code we are covering and what we have left due to various reasons. Knowing the coverage of your code means you know the missing pieces which is crucial in testing. However, if we start reading each test case and then check whether it covers a particular code and documents in it, this will take forever. Due to this, we have libraries that do that for us in seconds.

To know more about these libraries, you need to explore the language’s official website or community channels as different languages have different libraries and sometimes there are many coverage libraries for a single language with different features. Python, for example, provides coverage.py on PyPi for easy download and installation.

Prepare test report cautiously

Finally, test optimization can be achieved by focusing on the test case reports and documenting our test cases in them. 

A test case report is a complete analysis of the test cases. It contains the approach, actual test cases, results, and what we concluded from all of that. A simple test case report should contain the following sections:

• Test case ID: A unique ID for each test case that helps in referring them in the future within teams.

• Test case description: A description of the test case that defines what the test case does or test. Sometimes, this is written in “Given, When, Then” format to concisely document the description.

• Set up or pre-conditions: If the test case requires any pre-setup, it needs to be documented here so that in the future when a team member looks at it and re-implement, he is not stuck and wondering why the test cases don’t work.

• Test data: If any data is used for the tests, it needs to be written in this section. If not, the variables required/used are written.

• Execution steps: The path that the test case takes is written in this section.

• Expected result: The result expected from the test case.

• Actual result: The actual result we are getting from test execution.
• Final Status: The final status as “Pass” or “Fail” for the test.

As you can see, a test case report will have a sufficient knowledge base for anyone on the team and beyond for a quick analysis. Apart from it, testers reduce time in maintainability and have a document in hand from where they can pick up test cases directly (or refer) to newer projects.

These four sections if followed will prove to be great test optimization techniques in software testing that will help you long in the future.

Expanding optimization to test automation

Now that we have optimized our test cases and the process to keep a report for the future, we jump on to the optimization process for writing test cases. This is through automation testing.

Do not fall for 100% automation

100% automation means all your test cases are automated. This looks like the best path for optimizing for the future but actually, it is just the opposite. 100% automation is a path filled with complexities. It makes test cases hard to understand and will require large time investments in the future as we revisit them for maintenance or as new team members. As a test optimizing technique in software testing for test automation, automate as many of the tests as possible easily. The complexity can only be judged by the author. If they feel that the test is becoming complex unnecessarily, we can leave it and make a note in the test case report.

Another scenario where automation should not be done is the cases where we need manual interactions. For example, usability testing is better done through manual channels than automation. Such points will help optimize the cycles for the future.

Consider actual data

A test can also be optimized not from the logic but from the data it is fed. Data is what will go through the test and provides an output on the basis of which we bend the logic and finalize a test case. However, when the application is released, a bug is probable to come our way mentioning that one particular input failed as it was not taken care of in the source code. This mismatch happens because we feed artificial data that we created ourselves or by consulting with the developer. Such releases are bound to have loopholes. The best way to optimize the test cases is to feed them practical real-life data taken from actual user inputs. This way we train the tests according to what they will face when an end-user is operating the application.

Run automation on each code push

Testers often wait for the completion of tasks assigned in the current sprint to create new tests and run them on the final build that is slated to be released to the user. This just increases the feedback cycle and is definitely not the optimized way to perform test automation. A better solution is to run the automation code on each code push of each developer before merging. This way any regressions due to the developer’s code can be caught at the beginning and there is no risk of merging faulty code.

Keep frequent code review sessions

In the previous section, we touched on the topic of code review in terms of a senior reviewing the code of a junior. But what about reviewing our own code from time to time to modify it according to newer code (or new standards)?

Revisiting the code is necessary to keep our software in line with the development. With time, new theories, design principles, and efficiency methods are researched for a better testing phase. Along with it, older technologies are deprecated which were new at the time of writing the tests and new libraries and languages come along. Similar things happen with methods included in a library. Revisiting our code helps modify our tests accordingly and lay down a minimal maintenance structure for the future. This process is also called “monitoring of tests” and an optimum frequency is considered between four to six months.

Choose the best test automation tool

Optimization in test automation is not possible without a tool that facilitates the process. Since scripting is done using a tool, a wrong decision at this point is not something that a team would want.

In current times, automation tools have grown more from a tester’s point of view than a “testing” point of view. The best outcome of this is codeless test automation tools like Testsigma. Codeless tools eliminate the time invested in writing test scripts by a tester and the team can put all their focus on optimizations. As a result, a reduced test cycle and project budget accompanies such tools. To understand more on this topic, let’s explore Testsigma as an example in the next section.

How can Testsigma help in better optimization of your automated tests?

Testsigma is a codeless test automation tool and this property itself creates a lot of impact on the optimization of test cases for today and the future. A codeless automation test automation tool does not ask for programming scripts from the tester. Instead, they devise newer, easier, and more comfortable ways to achieve the same results. Testsigma does that by choosing its scripting language as English rather than any programming language. So a test is written in Selenium using Java as:

WebDriver driver = new WebDriver();

driver.get(“https://www.testsigma.com”);

Can be instead written as:

“Navigate to https://www.testsigma.com”

An example of English-based tests using NLP

This provides two very important benefits for the team:

• Focus can be shifted to hiring testers who are good at testing i.e. thinking scenarios and actions for that rather than learning a programming language.
• Maintenance is extremely easy and cheap since you just read the English text and understand what the test is all about.

These two are major pillars when it comes to test optimization techniques in software testing.

Apart from this, the testers also get an advantage of features that serve as a bonus for the team. The major ones out of this list are:

• Self-healing reduces the time to find UI defects and fix them manually.
• Cloud-based infrastructure brings down the time to set up and maintain on-premise infrastructure.
• Lets you automate your tests for web, mobile, desktop and APIs, 10x faster, from the same place
• Support for all the major testing fields helps stick to a single platform that in turn helps gather information from various test cycles quickly and at a single place.
• Support for all the major third-party software helps accomplish most of our testing goals from a single account.

All these benefits contribute to the optimization of software testing cycles.

 While Testsigma is a great tool to start with, if it does not coincide with your interest, always remember to choose a tool that helps bring down the time of the testing cycle. A reduced time will automatically reduce the associated costs today as well as tomorrow.

Creating bug reports for test optimization

The test optimization techniques discussed in this post until now help find a bug earlier in time. Once the bug is found, we must raise the issue to the developer and other team members. Earlier, this was a tedious process and something like this could be expected as an example:

• The tester finds the issue in the latest code.
• He raises the issue with the developer along with the steps to reproduce.
• The developer needs more information and asks the tester.
• The tester then collects more information and sends it to the developer.
• The developer cannot find the reason behind the issue. He raises the issue with other team members.
• The process is repeated as many times as a new member is involved in the bug.

Not only this, in the future if someone can recall that they saw a similar bug, they cannot retrospect it since there is no document or data associated with it. 

A bug report is similar to a test case report but here we document the information related to every bug. This information may contain the following sections:

• BugID – A unique Bug ID for the identification of every bug.

• Bug Heading – A small heading describing the bug.

• Bug description – A description stating the issue in as much detail as possible including steps to reproduce, the system used to test, etc.

• Status – The current status of the bug such as Open, In progress, Dev to QA, etc.

• Reporter – The tester reporting the bug.

• Assignee – The developer to which this bug is assigned.

Also, since developers may reassign the bugs within themselves, this information should also be documented with time in the report along with comments, if any.

Summary

Testing is an important part of software release. While the developer will develop a couple of features in a sprint, a tester will go through all the past and current features to make sure the product is alright. This burdens this phase and hence increases the time of execution, project costs, release time, and team size to cope with the responsibilities. But we can avoid all this hustle if we could optimize our test cases and other related stuff from the start to be prepared for the future.

This post highlights the importance of optimization in software testing and techniques that can help us achieve that. Starting from general optimization, we moved on to the complexities of test case-specific areas and finally concluded with automation-related optimizations. Grasping all these sections can help make optimum use of the resources, and team, and lay a careful plan for the future.

Frequently Asked Questions

What are the benefits of test optimization?

Test optimization brings a lot of benefits for the team and organizations. A few of them are:

• Reduced test execution time.
• Reduced costs of the project.
• Early project release.
• Increases maintainability of the code in the present and future versions.
• Eliminates redundant code.

What are the best test optimization techniques?

Generalizing test optimization techniques for any kind of project produces the following results:

• Incorporating shift-left testing.
• Incorporating in-sprint methodologies.
• Selecting the most appropriate tool (including the test automation tool) for the project.
• Testing on real devices.
• Keeping extensive documentation.
• Conducting code review sessions regularly.