Smoke Testing vs Sanity Testing vs Regression Testing Explained

Introduction

The life of a QA Tester will be considered incomplete if the terms ‘Smoke Testing’, ‘Sanity Testing’ and ‘Regression Testing’ are not infused into it. Though these are some regularly used terms, there are some common misconceptions around them too.

Myths and Misconceptions

Before we begin into the details of smoke testing vs sanity testing vs regression testing and what they actually mean, lets go through some common misconceptions and myths around them:

1. Smoke Testing and Sanity Testing are the same and can be used interchangeably : This is not true and should never be done, why? we shall discuss that in this Article.
2. Sanity Testing is equivalent of acceptance testing: Acceptance Testing is done to ensure that the build meets all the requirements that the client specified before the release while Sanity Testing is done to ensure the product is sane, rational for more detailed testing. These are not and should not be used interchangeably.
3. If I am doing Smoke Testing, I can skip sanity testing: Smoke testing is a very high level testing, as in, this testing is done to ensure this build is suitable to begin any other type of testing. For example, this could be just a test to ensure the build installs and then the Sanity Testing can begin. Though sometimes, the smoke test cases are run together with Sanity Test cases, these should not be confused to be the one and the same thing.
4. Regression testing is not related to Smoke Testing or Sanity Testing: In Theory, Sanity Testing is a subset of regression testing. Some high priority regression test cases constitute sanity test case usually.
5. If I intend to run the full regression test suite – I don’t need to run any smoke test or sanity test: While this may be true in some cases as Sanity Testing is anyways a subset of regression testing, it is still advisable to execute the sanity test cases first and then follow them with the rest of the regression test cases.

In this article, we will try to clear above confusions once and for all. In case, you wanted to read more about Manual Testing, do refer to the link here.

Note: Because we will be using the term ‘Software Build’ many times in the article, lets define it here.

What is a Software Build?

‘Software Build’ is a process of converting source code, into a user application, after multiple revisions and code changes and Software Build creation involves multiple processes like, “Version Control, Code Quality & Compilation” and software build is also the result of this building process. In this article, a build will be referred to as a testable version of the software.

Smoke Testing

Smoke testing is performed in the initial phase of the software development life cycle by the developers to ensure the code module is successfully compiling and the software’s core functionality is working properly. Any other level of testing is done after successfully passing the smoke testing.  

Smoke testing will fail if the required environment is not present, other required libraries are not imported in the code itself, or any major bug issue breaks the software build and does not let it run, etc.

What is Smoke Testing?

Smoke testing is an approach which is usually carried out during the initial development stages of the Software Development Life Cycle(SDLC) to make sure that the core functionalities of a program are working fine without any issues. It is executed before any detailed functional tests are done on the software.

The main intent of smoke testing is not to perform deep testing but to verify that the core or main functionalities of the program or the software are working fine. Smoke testing aims to reject a badly broken build in the initial stage so that the testing team does not waste time in installing & testing the software application.

Smoke testing is also called as Build Verification Test.

Let’s see a simple example where you are given an email application to test. The important functions would be logging in to the email application, composing an email and sending it, right? And, in case the email is not sent, does it make any sense to test other functionalities like drafts, deleted messages, archives, etc? This means you will have to drop the build without further validation. This is called Smoke testing.

The main focus of smoke testing is to test the critical areas and not the complete application.

Advantages of Smoke Testing

Smoke testing is run in the early stages of the software development life cycle, ensuring that software core functionality is working properly. 

Smoke testing saves time and energy by detecting any major issues in software before installing it for the test.  

Disadvantages of Smoke Testing

Where smoke testing ensures the working of software core functionality, it does not conduct in-depth testing. Even after Smoke testing the entire application, essential integration and system testing flaws may surface. Because of this, minor bug issues might go unnoticed in the software.

When to Perform Smoke Testing?

• When developers provide a fresh build to the QA team. A fresh build here means when the build has new changes made by the developers.
• When a new module is added to the existing functionality.

Automation & Smoke Testing:

Usually this is the type of testing that is executed before actual automation test cases can run. For organizations that have continuous testing built in, smoke testing is equivalent to successful installation of the build for running test cases or execution of the first test case. So, this is not a type of testing that is deliberately automated but if test automation is put into place, the test automation can only run successfully once the software has passed smoke testing. Or otherwise, the first test case that executes might fail.

Sanity Testing

Sanity test is the subset of regression testing. it ensures that bug fixes, newly added functionality or any other code change doesn’t affect the stable software, build other functionality or introduce any type of bug issue. It also evaluates the software and ensures whether it’s ready for the next level of testing or not.

What is Sanity Testing?

Sanity testing is a kind of testing performed to check whether a software product is working correctly when a new module or functionality gets implemented to an existing product. Sanity testing is a software testing technique which does a quick evaluation of the quality of the software release to determine whether it is eligible for further rounds of testing or not.

Sanity testing is usually performed after receiving a fairly stable software build or sometimes when a software build might have undergone minor changes in the code or functionality. It decides if end to end testing of a software product shall be carried out further or not.

Sanity testing is also a Surface Level Testing which helps in deciding if the software build is good enough to pass it to the next level of testing.

Advantages of Sanity Testing

Sanity testing ensures that the new functionality added to the software doesn’t cause any bug issues. Other than that, it does a quick evaluation of new software releases to ensure whether it’s ready or not for the next level of testing. It helps developers save time and energy by not passing the broken software for the next level of testing. 

Disadvantages of Sanity testing

Sanity testing does not thoroughly test the software functionality; because of this, minor bug issues cannot be found by sanity testing. This testing only evaluates whether further testing can be performed or not. 

Why should Sanity Testing be done?

• To verify and validate the conformity of newly added functionalities and features in existing code.
• To ensure that the introduced changes do not affect other existing functionalities of the product.
• To decide further testing can be carried forward or not.

When to Perform Sanity Testing?

• Build is received after many regressions or if there is a minor change in the code.
• The build is received after bug fixing.
• Just before the deployment on production.

Automation & Sanity Testing:

Considering, Sanity Testing is considered as a subset of regression testing, these are the test cases that can be automated. A recommended approach is to execute these test cases before running the complete regression test suite. The benefit is that if there are any errors in the sanity test cases, then errors can be reported sooner rather than later.

There are companies that have benefited from automating their sanity test cases. Here is a case study where Freshworks – a feature packed product suite for businesses of all sizes – was able to automate 100% of their sanity test cases in 90 days using Testsigma: Testsigma+Freshworks – A case study

Regression Testing

This testing is concurrently done after unit and integration testing. It is the process of continuous testing to ensure that newly added functionality and features are working correctly without affecting other existing functionality of the software. Regression testing is not just a single test but a concurrent combination of various tests.

What is Regression Testing?

Regression testing is the verification of “bug fixes or any changes in the requirement” and making sure they are not affecting other functionalities of the application. Regression testing is effective on automation and usually performed after some modifications have been made in the software build after requirement changes or bug fixes.

Once Sanity testing of the changed functionality is completed, all the impacted features of the application require complete testing. This is called regression testing.

Whenever bug fixes are done in the existing software, some test scenarios need to be executed, to verify the bug fixes. In addition to these, the QA team also has to check the impacted areas, based on the code changes. In regression testing, all those test scenarios will have to be executed, to take care of related functionalities.

Learn More – Features of Regression Testing

Advantages of Regression Testing

Regression testing is very effective in automation. It ensures that any changes, bug fixes, or newly added features in the software do not reoccur or affect the existing functionalities of the software. 

Disadvantages of Regression Testing

Regression testing requires writing complex test cases, which require prior testing experience, it will be challenging and time-taking for a non-experienced tester to perform the regression testing.

When to perform Regression Testing?

• After Code modification according to the required changes
• After some new features are added to the application
• After some bug fixes are incorporated into the build

Automation & Regression Testing:

Regression test cases are actually the ideal test cases for automaton. Usually, when an organization starts automation, these are the test cases that are automated first. If regression testing is an activity that is taking a lot of time for your testers and the same test cases are repeated multiple times then it is time that you start thinking of automation too.

If you are looking for an automated regression testing tool that can help you get started on your automation journey, then you also need to ensure that you choose the right tool. A tool that can also provide you ROI on the efforts invested. We have the guide that can help you there: 10 Points to Help You Choose the Right Test Automation Tool

Now its time to look into the differences between Smoke Testing VS Sanity Testing VS Regression Testing, check it here,

Differences Between Smoke Testing vs Sanity Testing vs Regression Testing

Smoke Testing	Sanity Testing	Regression Testing
Performed on initial builds	Performed on stable builds	Performed on stable builds
To test the stability of new build	To test the stability of new functionality or code changes in the existing build	To test the functionality of all affected areas after new functionality/code changes in the existing build
Covers end to end basic functionalities	Covers certain modules, in which code changes have been made	Covers detailed testing targeting all the affected areas after new functionalities are added
Executed by testers & sometimes also by developers	Executed by testers	Executed by testers, mostly via automation
A part of basic testing	A part of regression testing	Regression Testing is a super set of Smoke and Sanity Testing
Done usually every time there is a new build	Planned when there is not enough time for in-depth testing	Usually performed, when testers have enough time

Now lets get the key points separately on Smoke testing vs Sanity testing , Sanity testing vs Regression testing and Smoke testing vs Regression testing

Key Points of Smoke Testing vs Sanity Testing

The difference between smoke testing and sanity testing is, developers carry out smoke testing in the initial phase of the software development life cycle. It ensures software core functionality is working correctly. Testers perform sanity testing after the smoke testing to ensure the software works correctly after code changes, bug fixes, or new functionality is added. Evaluate whether the software is ready for the next level of testing.     

• Smoke testing is performed in the initial phase of the software development life cycle. In contrast, sanity testing is a subset of regression testing done after smoke testing. 
• Smoke testing ensures the software core functionality is working correctly, where sanity testing is performed to test if the software is working properly after the new addition of functionality and evaluate whether it’s ready for the next level of testing. 
• The development team carries out smoke testing, but the QA and test teams perform sanity testing.

Key Points of Sanity Testing vs Regression Testing

Sanity testing is the subset of regression testing; it ensures the stability of newly added functionality and whether the software is ready for the next level of testing. 

Regression testing is a combination of various types of testing, and sanity testing is one of them. It ensures that newly added functionality doesn’t affect any existing software functionality and is working correctly.

• Sanity testing is the subset of regression testing, ensuring the newly added functionality’s stability. In contrast, regression testing ensures the software works correctly after adding new features.
• Sanity testing is performed under regression testing, whereas various testing types occur under regression testing.  
• Sanity testing ensures that the software is ready for the next level of testing. In contrast, regression testing is about testing the software functionality in detail to find the possible bug issue.

Key Points of Smoke Testing vs Regression Testing

Smoke testing is carried out in the initial stages of the software development life cycle to ensure the software core functionality is working as expected. Whereas regression testing is carried out after the smoke testing, it ensures that newly added features don’t affect the software’s existing functionality and it’s working correctly.

Summary

• Smoke and Sanity testing help the QA team save time by quickly testing to make sure if an application is working properly or not. Also, it ensures that the product is eligible for further testing. Whereas Regression testing helps enhance the confidence about the software quality after a particular change. Especially, that the code changes are not affecting related areas.

• Smoke Testing is done by both the dev team or by the QA team and can be taken as a subset of rigorous testing. Whereas both Sanity & Regression testing are done only by the QA team. Also, Sanity testing can be considered as a subset of acceptance testing.

• Smoke testing is executed at the initial stage of SDLC, to check the core functionalities of an application. Whereas Sanity & Regression testing are done at the final stage of SDLC, to check the main functionalities of an application.

• As per the requirement of testing & time availability, the QA team may have to execute Sanity, Smoke & Regression tests on their software build. In such cases, Smoke tests are executed first, followed by Sanity Testing & then based on time availability regression testing is planned.

Lets conclude Smoke testing vs Sanity testing vs Regression testing,

In practice, all QA teams need to do Smoke, Sanity and Regression testing. All of these testing types have a pre-defined number of test cases that get executed multiple times. This repetitive execution also makes them an ideal candidate for test automation. When looking for automation, you are recommended to use a tool that provides you ROI on automation from the initial stages. Testsigma is one such tool.

Frequently Asked Questions 

Is Sanity Testing done before Regression Testing?

Yes, Regression testing is carried out after executing sanity tests of any altered functionality. Sanity testing is the subset of regression testing. It ensures bug fixes, newly added functionality or any other code changes don’t cause significant bug issues in the software. It determines whether or not the software is eligible for the next level of testing.

Which comes first, Sanity or Smoke Testing?

Smoke testing is carried out early in the software development life cycle to ensure the software’s core functionality is working correctly. Smoke testing rejects badly broken software in the initial testing stage of software development. 

After the smoke testing, sanity testing is carried out to ensure the newly added functionality in the software is stable. It does not affect the software’s existing functionality. 

Which testing is done manually Sanity or Smoke? 

Smoke testing is done manually without any test automation tool because smoke testing is about testing the software core functionality are working properly. Sanity testing is a subset of regression testing performed using an automation tool which evaluates whether the software is ready for the next level of testing.

Is Sanity Testing part of Regression Testing?

Yes, sanity testing is a subset of regression testing. It ensures any types of changes in the code don’t affect the software’s existing functionality and whether the software is eligible for the next level of testing or not.

Why Sanity is subset of Regression Testing?

Sanity testing is the subset of regression testing because, regression testing is not a testing in itself, it’s the combination of various types of tests. Each time a new functionality is added in the software, regression testing runs and ensure the overall functionality of software is working properly, in this process it is the sanity testing under the hood which ensure the stability of newly added functionality and evaluate software is ready for the next level of testing.

Why is Smoke Testing called Smoke Testing?

The name smoke is somehow adopted from the plumbing community. Initially, people used smoke to check for leakage in the pipeline, then adopted it in the software testing. In the same way, smoke is used to check for leakage, and smoke testing finds major leakage in the software.

What is the Test Life Cycle?

Before the product is released to the market, it goes through multiple testing layers, called the test life cycle. It ensures the software is bug-free and user-friendly. In the testing life cycle, various types of testing take place, for example- unit testing, integration testing, regression testing, etc. 

What are the different Levels of Testing?

Different levels of testing are performed at various stages of the testing life cycle; some of the important ones are the following:

• Unit testing- Unit testing is generally performed by the developer but sometimes can also be performed by testers. In unit testing, smaller units of code, like functions, methods, and classes, are tested in isolation. 
• Integration testing- In integration testing, the behavior of two or more units and linked interfaces between them is being tested. 
• Regression testing- Regression testing is the process of continuous testing; it checks the software is working correctly after adding new features. 
• User acceptance testing- User acceptance testing ensures that all user requirements are covered and fulfilled in the software as per requirement. 

Suggested Reading

Software Testing Strategy

The Different Software Testing Types

Regression Testing: Challenges, Strategies, and Best Practices