When to STOP testing?

“When to stop testing” is a very interesting topic yet highly asked in many interviews these days. There are several answers to that question too such as: “you stop once you have completed an acceptance criteria”, some say “you never stop as testing is an ongoing task” and some day “it depends!” 

Every company and every engineering team have their own criterias set in terms of when testing should be stopped for a particular product/feature. There are several things that impact this decision such as the entry and exit criterias too, however are these enough? 

Let’s dive deeper and understand what is a good way to stop testing with enough confidence. 

When do we start testing? 

Let’s start this blog by understanding when to really initiate your testing process. As we keep hearing a lot in communities, conferences, webinars, etc shift testing to the left and not to leave this major task later in the SDLC. The reason behind this is that it helps find defects earlier, clear any assumptions/ambiguities, reduces costs, time and effort to rework and retest on features. This then leads to a good quality product. 

We can start testing as early as the requirements stage by understanding what is the ask from this feature:

• What do we already have in place? 
• How much testing will be required?
• If we don’t have anything in place, then what would the system expect?
• Pair with developers and Business analysts to get a clear picture and expectations
• Understand if we need other teams too as this might be a feature that impacts other teams or not
• Be involved with the design team to understand the design prior to testing it
• If you are automating tests, you can follow a TDD process too and won’t have to wait till the feature is fully developed

Once we are clear at this stage, deployment will become much smoother for the entire team. 

When to Stop Testing? 

Till date no manager/director has ever said to me or my team to stop testing or reduce testing. All I hear is are we ready to proceed, are we confident with the amount that has been tested? Generally testing is an ongoing task, we do tend to meet acceptance criterias and move our tasks to completion, however sometimes an extension or enhancement is done to the same feature, that means you have to test it again. Therefore having an open mind about this is always good for all teams. 

The following points can be some guidelines to help you stop the testing tasks:

• Decisions from your product owners, line managers to stop the testing tasks
  • a) Budget or/and resources were allocated to some other high priority project
  •  b) The project was canceled (from the client’s side or the company’s side) 
  • c) The allocated budget is over and no more budget is being allocated.

• Deadlines that need to be met
• No high priority and severity bugs found/expected
• Completion of test case execution
• Completion of functional and code coverage to a certain point
• Completion of the acceptance criteria and exit criteria. It is imperative that we consider multiple factors while designing exit criteria, some of which are mentioned below:
  • 1. Strong buy-in from the software development team. In the end, your job is to discover defects and theirs is to fix.
  • 2. If there are clients who will further test the product, that should also be considered as to what will be an acceptable input quality.
• Good coverage of the functionality/code.
• Number of bugs in the feature or too buggy to deep dive (valid criteria for exiting a testing process is the discovery of a forecasted number of bugs).
• Release deadline has been met

It is vital to pay attention to all finer details and errors in the product of the application. It is not possible to find all the bugs that may exist, but the team management will make decisions on when to release the product anyway. Also remember that if a bug is found in a live production environment that is not solely the testers responsibility, it’s the engineering team’s responsibility. Similarly, if you are retesting a vital bug and get asked “are we ready to release the testing?” it should not be the testers decision to say yes or no, it should always come from the management team. The release might go ahead with some unwanted bugs! See, we haven’t really stopped testing. The question is, can we find these bugs prior to the release, and what is their severity? And the best thing, how to stop the same bug from appearing?

Thereafter if you have met the exit criterias, ship it, but understand that it still has an unlimited amount of not yet discovered bugs.

Checklist 

This checklist can serve as a guide to determine whether or not your team have enough confidence for completing the testing:

• Stop the testing when deadlines like release deadlines or testing deadlines have reached.
• Stop the testing when the test cases have been completed with some prescribed pass percentage.
• Stop the testing when the testing budget comes to its end.
• Stop the testing when the code coverage and functionality requirements come to the desired level.
• Stop the testing when the bug rate drops below a prescribed level.
• Stop the testing when the number of high severity Open Bugs is very low.
• Stop the testing when the period of beta testing/alpha testing is over.

Some useful metrics you could also use :

• Have a dashboard of the percentage of test cases that have passed.
• Have a completion percentage based on the number of test cases executed.
• Have a percentage of failed test cases.

One way to speed up your testing is when you automate your tests. Automation, though, should only be taken up if you see an ROI in the long run at least.

Testsigma is one such tool that comes with features made specifically to help you speed up your testing. It is a codeless platform thus you don’t need to learn to code too. Learn more about it here: testsigma.com

Conclusion

Determining when to stop testing is subjective and a complex matter. I believe testing is a never-ending process in a positive way. I have yet to hear from someone that the software is 100% complete from testing. Criteria once designed are not meant to be carved in stone. It should be reviewed constantly in light of the adherence and the post-release issues which come after we meet a set criterion. Therefore, by exploring factors such as meeting release deadlines, the severity of bugs, management decisions, and establishing effective exit criteria, we have equipped you with the necessary thinking points to make informed choices. While the decision may still require careful consideration and evaluation of specific project circumstances, we are confident that the information shared in this blog will empower you to determine appropriate exit criteria and identify the right time to conclude testing. Remember, success lies in finding the right balance between thorough testing and project constraints. 

Happy testing!