Test Process Improvement | What, When, How & Best Practices

It has been while exploring the scope for improvement, that resulted in the research and inventions in the field of computer science and literally everywhere else too. For example, if we take the example of a hard drive (hardware that impacts everyone who uses a computer), it started with this in 1956:

With many improvements with time, we brought it down to this:

Also remember the memory size difference is huge between the hard disk used in both images. They are now more optimized, faster, have a larger capacity, and most importantly compact and easily portable. But it’s not like we arrived at this point suddenly. Such transformations require constant improvements in each process involved in the development and testing. 

Till today, the basic principles of improvement journey remain the same and with this post, we will try to take a peek into it using test process improvement. 

What is test process improvement?

Test process improvement is exactly what its name suggests i.e. improvement of the test process from various standpoints. It is obvious that whatever process we currently use always has a scope for improvement in it. Also, whatever process we currently use is an improved iteration of its previous version. This makes us stand at a stage from where we can analyze our current arrangement and determine the weak areas for further research into it. In the end, we correct those areas and generate a process to be followed in the future.

Test process improvement is an important phase to create a direct impact on testing and development. An improved process is not only more capable of finding better bugs but also testing the application in ways that uplift its quality and standards. 

Sometimes referred to as TPI, test process improvement can be a continuous or non-continuous process depending on how the team operates. This involves depending on certain models that can help us reap the benefits of the end results. These benefits are important to discuss for further defining our motivation towards doing all this work.

Why to conduct test process improvement?

Test process improvement comes with benefits that span beyond the testing part of an application to other domains as well. The most highlighted benefits of working on test process improvement are:

Testing improvements

When a test process is improved, naturally we see an improvement in the testing phases and the way things are executed.

Good quality software

An improved testing means we are now exploring the application in a more detailed way. This will help expose more bugs which will ultimately improve the quality of the application.

Align testing with other phases

Improvement is not confined to the testing phase. We can witness it in development technologies, and even CI pipelines that get optimized with time. In such a case, if test process improvement remains stagnant and other processes improve with time, the quality of the application will suffer significantly. It is very important to align testing with other phases so that when other processes improve, we can strategize according to those changes as well.

Provide effectiveness in test activities

Streamlining the testing activities and improving each of them individually brings effectiveness to the system which also helps wrap things much faster than before.

Improved costs

Newly improved processes are more effective in getting things done and hence require much fewer resources than before. Sometimes, we may also see an introduction to test automation in processes that were used manually. Codeless test automation is a good example of test process improvement in the field of web automation. Hence, a reduced number of resources may directly affect the number of engineers required in a project.

An example of codeless testing using Testsigma

 All of this when combined results in a notable reduction in costs of the testing phase.

Automate your tests for web, mobile, desktop applications and APIs, 5x faster, with the codeless test automation solution Testsigma

Boosted business growth

All of the processes mentioned help deliver a high-quality application, with minimum costs and in the shortest possible time. 

Test process improvement brings a ton of benefits. When everything you are doing is now done in a better way, results will also turn out to be better than before. However, this is not something that can always be kept in the pipeline. We cannot just focus on improving our testing methods in each sprint. It is too short a time to retrospect and compare the results. For this, we can either plan according to the past cycles or keep an eye on certain hints that we catch during the testing processes. These small signals will help us indicate the need to improve and turn our focus in the right direction.

When to perform test process improvement?

The need for test process improvement does not arise very frequently in any domain. For any particular process, optimized for the current times, we will witness the results for some time. Once that time concludes, we will start observing the impacts of the outdated process we have been using. Keeping an eye on such hints will help us start testing process improvement for further optimizations. The major impacts in such scenarios are as follows:

Significant rise in bugs 

The most visible effect we will witness that indicates a need for test process improvement is the rise of bugs we face. While bugs in an application are a natural and expected element (as a bug-free application raises a whole other lot of questions), their frequency is always monitored by the higher management. A rise in bugs indicates the weak areas of our application have increased and our testing methods are not proving to be effective. As a result, we retrospect and aim for improvement.

Increase in complexity (test management)

Managing the testing phase is a skill that people acquire with years of experience. Every piece has to be aligned perfectly to carry out the testing process in full efficiency. Along with people’s skills, we also need support from individual testing processes as the execution of these elements decides other parameters as well. With no monitoring of increasing complexity, test management can become a cluttered arrangement of unorganized tasks. When such behavior starts to pop up, we may need to improve the testing process. 

Increase in involved resources

When complexity and bugs increase, we need more resources to cope with the deadlines. Sometimes this is a natural phenomenon on a scaling project. But this should reflect the same equation as seen in scaling the application. If the application has not scaled much but we are still struggling with the resources in hand, a call for improvement is required.

Increase in time of testing

All the effects of inefficient testing leads to an increased time of the testing phase. More bugs mean more time required to resolve them. Similarly, fewer resources will take more time to accomplish piled-up work with constantly increasing complexity. When the time of testing starts to increase constantly, we cannot ignore it. It directly impacts project delivery and release cycles are hurt. This can also lead to a bad reputation for the organization. However, all of this can be normalized by analyzing our current testing process and finding the most time-consuming elements. Improving those elements will certainly improve testing time.

Increase in testing costs

An increased time means more billable hours for engineers as well as more days of subscriptions for the tools we use. When we add these additional costs along with overheads, the testing phase constitutes a major percentage of the overall expenses that occurred in the project. This becomes a concerning factor from a financial point of view and invites serious thought towards individual components of testing and improving them for better efficiency.

Newer methods have arrived

Engineering elements are always evolving and engineers always keep an eye on new technologies that can do the same work faster, cheaper, and with minimum resource usage. Whenever we find such technologies that resonate with the work we have been doing, we need to move ahead with test process improvement. Initially, we may need to do additional work in integrating the technology (which includes learning) with our project, but the return on investment dominates this heavily.

We have not retrospected for a long time

Finally, we should perform test process improvement if it has been a long time since we did it last time. Managers and higher management should remind the team about taking a break and spend a couple of days just analyzing their past work, comparing it with what they did a while back, and analyzing the increase in complexity over time. Focusing on just the elements they work on will give us a magnified view of what could be improved for the future. Keeping the testing process in check eliminates the risk of getting stuck when things become a little hard to manage.

These hints are enough to understand that the time has arrived when we need some retrospection. Ignoring these signs will only increase the correction and reviewing costs.

How to implement test process improvement?

Test process improvement is a game of analysis and acting upon our observation. To smoothly accomplish this, we divide the implementation process into four parts:

Diagnose the test improvement

To implement test process improvement, the first thing we need to do is diagnose the situation. This includes first determining why we need to perform test improvement. Here we need to report one or more things discussed in the previous section that led us to believe that our current testing methods are not performing effectively.

Once we report our findings, we need to dig a little deeper into those areas to explore specific findings that can point us to the exact problems. For example, if our concern is cost management, then, we need to document what the costs were earlier, how it has grown over time, graphs depicting a clear view of the growth, what has impacted this high cost in the recent past, any significant change in the resource, etc. Consider this document as the sole evidence of moving further with test process improvement.

Initiate the process/planning phase

The next step is to initiate the process of test improvement which is also termed as planning phase. In this phase, we plan the actions we will take for the process improvement. This includes documenting our objectives (such as budget control with reference to the previous example), goals, processes we need to follow with steps, and coverage of process improvement.

Apart from these sections, we also need to put elements such as risks that can slow down our process or even force us to halt before completion. If any risk is identified, we can put down the expected delay in days or hours depending on the risk.

Finally, at last, we document the final expected date of completion of the test process improvement and the date (or release version) from which we will start seeing the results of our endeavour.

Acting on the plan

With all the blueprints in our hand, we start the actual method of test process improvement. This should follow the guidelines and expectations described during the planning phase and focus only on defects pointed out during the diagnosis phase. 

The senior testers are required to monitor and guide this process through their expertise. They should also ensure that delivery dates are not impacted and the process remains on schedule.

Verify, Report, and Learn

At last, we verify the actions we performed with the results we got and match them with the expectations we set during the planning phase. Next, we report all our results (including metrics) on the report for approval and sign-off from senior testers and higher management. Lastly, we retrospect on the test improvement process from phases 1 to 4 and document our learnings throughout the process. These learnings will help us commit fewer mistakes in every testing phase followed by the improvement phase.

These four stages of test process improvement are easier to execute and if followed correctly, will reverse the defects observed during (or before) the diagnosis phase. In the same line of thought, let’s look at the models of test process improvements and their needs in the process.

Software testing process improvement models

Similar to software development models (such as Agile and Waterfall), we rely on many test process improvement models to follow a set standard and improve our testing process. These models are often divided into “maturities”, a term we will often use in this section, which means a certain stage that progresses toward better arrangement, organization, and completion. For reference, you can easily link this to waterfall steps such as “design”, “development”, etc.

Testing Maturity Model integration (TMMi)

TMMi is a “maturity model” which means it focuses on the maturity levels of each process to effectively implement the improvement in the testing lifecycle. Through TMMi, we can determine the maturity level of the testing process in an organization and improve it to achieve higher maturity. It was introduced in 2005 as a response to the ineffectiveness of the Capability Maturity Model (CMM) in the testing domain. TMMi tries to involve the organization structure and all the testers to progress together in the improvement process.

The benefits of using TMMi models are:

• Covers test-related activities extensively.
• Covers the best practices from the existing models.
• Covers the requirements for current trends and needs.
• Covers all aspects of test quality as described by industry experts.
• Provides a common standard for use.
• Can be applied to all phases of the software development lifecycle.

TMMi is divided into five maturity levels.

Maturity Level 1 – Initial

Maturity level 1 is not defined by the TMMi model because it presumes that all organizations start at this basic level. So, whatever you currently do and whatever process you follow, you can assume it to be of maturity level 1. This includes debugging and chaotic (unorganized) processes.

Maturity Level 2 – Managed

If the organization follows even a basic test approach towards testing and manages the same approach, it comes to maturity level 2. At this level, the organization should have a foundational structure in place that includes test planning, test policy, monitoring, and setting up of a test environment. This, however, depends deeply on the project as a lot of things change when a new project is introduced.

Maturity Level 3 – Defined

Maturity level 3 standardizes the testing process across the organization and expects each project to follow the same process. With this level, testing is now integrated very early into the development making it part of the development cycle. Teams are required to be trained in testing and each member has a specific job for testing i.e. teams are more organized than before.

Maturity level 3 also expects that non-functional testing be planned and executed accordingly with reviews.

Maturity Level 4 – Measured

When the outcomes and parameter values are applied to all the projects to ensure a bug-free application, we reach maturity level 4. The review practices introduced in the maturity level 3 are now more advanced and thorough in nature.

Maturity Level 5 – Optimization

At maturity level 5, the organization has a series of methods set up for optimization of the processes followed in testing up to maturity level 4. Continuous optimization leads to a bug-free application. 

Test Process Improvement (TPI) Next

The Test Process Improvement Next model, also called TPI Next, is the next generation of its predecessor TPI (Test Process Improvement) model. Keeping its maturity and characteristics intact, TPI Next aims to strongly establish its predecessor’s strengths while keeping in focus the business goals of the organization. Since TPI Next aims at the business-related key areas, it can be referred to for activities beyond testing and is a generic model to implement. 

The TPI Next model describes four maturity levels:

Maturity Level 1 – Initial

Initial maturity level has the same meaning in TPI (Next) as in the TMMi model. There is no fixed process, the organization does not follow any rules and the process is chaotic. Here too it is assumed that the organization is already at the maturity level 1 with whatever approach they have been following.

Maturity Level 2 – Controlled

A controlled maturity level includes the stakeholders to streamline the process with minimum structure. The structure is organized and helps in reducing bugs. However, this just represents a controlled state of the process and in no way is optimized. Therefore, we can assume a lot of bugs will still exist if the organization is at maturity level 2.

Maturity Level 3 – Efficient

Maturity level 3 aims at the efficiency of the currently working process in testing. At this level, we make sure we are not spending extra time on the redundant processes that arise when the testing process has certain overlapping stages. We try to streamline these processes and combine each process for a unified testing cycle. 

Maturity Level 4 – Optimizing

In the final level of maturity, we assign coordinators who evaluate the process continuously from time to time. Their evaluation helps optimize the process and improve it for the next project. This includes evaluating the bugs we faced, any delays, or any other type of hindrances faced during the run.

When the maturity levels are integrated with the key areas involved in the test process improvement and the checkpoints that analyze the progress at various intervals, we get the test process matrix. 

Critical Testing Processes (CTP)

As the name goes by, critical testing processes aim at only critical processes of the testing lifecycle. These processes are termed critical as their malfunction can result in a complete breakdown of the application. Once these areas are in focus, the critical testing processes can help ensure that the foundational elements of the application are working and the product can be termed reliable. An example of this is sign-up functionality in mobile app testing.

Critical testing processes are not a maturity model and therefore it does not measure your current state of improvement through stages. Instead, it measures the processes in terms of business values considering both qualitative and quantitative analysis. CTP is extremely flexible and allows the testers to put up the challenges in order of priority (based on importance) themselves without defining any strict rules as with other models.

Systematic Test and Evaluation Process (STEP)

A STEP model approach focuses on testing the requirements systematically, evaluating the results, optimizing accordingly, and then moving forward. The basic idea of the STEP model is to accompany testing along with other parts of SDLC and testing should be involved from the beginning to ensure good quality software. The characteristics of STEP can be defined as:

• STEP focuses on requirement-driven testing. This means test cases reflect the requirements of the project and their success would mean that the requirements are satisfied successfully.

• STEP aims at performing testing first and then starting coding. This is similar to test-driven development (TDD) which is recommended in STEP.

• STEP helps in early defect detection due to the “testing first” approach. It asks the team to start testing even before development based on requirements. Since we know there is no code written for a certain requirement, the test fails initially but passes as we code along. This ensures that the requirements are satisfied, the quality of the application is good, and there have been no compromises on testing (which can exist if the structure of code is known especially when done by developers).

• Since there are no maturity levels or predefined steps in this model, it can be combined with other models as well. In practice, organizations blend STEP with TMMi to get the strengths of both sides. However, there are no such standards and it depends on the team.

Such models are popularly used to organize our testing process and define a set standard to be followed by all people, new and existing. One may find many more of them and settle on a single model (or a combination if allowed) for better efficiency and optimization. 

Conclusion

The test process, like any other process, gets outdated as newer working methods are developed. Even more so we may see this effect when other domains of SDLC are continuously developing and making the testing process a little more challenging day by day. This calls for a monitoring process that keeps an eye on testing and through certain checklists (or hints) can decide whether the current process is efficient or it needs improvement.

This monitoring process is called test process improvement. It combines various strategies that measure the performance of the testing phase through multiple parameters. While an organization is certainly free to make its own rules in this regard, most of them rely on well-researched models that are efficient in discovering loopholes in the system. The story of test process improvement, its benefits, and implementation using four popular models is told in this post with an emphasis on efficiency and optimization. We hope this post helps you organize your current process in a better way and redefine the way the team leads the testing cycles for a better and cost-efficient future.

Frequently Asked Questions

What are test process improvement examples?

Test process improvement can be expected to show its results in any of the multiple areas that affect testing in general. For instance, reduced bugs, reduced involvement of resources, reduced costs, etc. While all of these defects won’t be seen together, it is important to always keep an eye on any of the areas that show a decline in efficiency.