What is Canary Testing? A Comprehensive Overview

Canary testing is helpful in checking the new functions and features of an application without impacting the user experience or keeping it to a minimum. Typically, people interchange the terms canary testing, canary releases, and canary deployment, but they differ in reality.

In this blog, we will guide you through canary testing information. We will further talk about these different terms and their meaning for a better understanding.

What is Canary Testing?

Canary testing refers to validating the new feature releases to a limited number of users to reduce risk and keep the system running without any interruption or downtime. It is a powerful and practical testing method. It includes deploying code incrementally to the production environment and testing it there itself.

The name comes from the method used by the coal miners to detect odorless but deadly gas in the mines; the miners carried a canary with them down to the mines, and if the canary died in the cave, the miners knew that a lethal gas is building up to dangerous levels. It helped them to vacate the mines at the right time. Similarly, canary testing highlights the issues early on with minor feature releases (canary releases) and helps avoid significant and more complex system issues.

Why is Canary Testing Effective?

Canary testing is effective for several reasons, making it a valuable approach in software development and deployment:

• Canary testing allows organizations to minimize the risk associated with deploying new features or updates to a large user base. Initially, rolling out changes to a small subset of users can detect and address any issues or bugs before they affect the entire user population.
• Canary deployments provide early feedback on the performance and stability of new changes. If problems arise, they are identified early in the process, making fixing them easier and less costly.
• Canary deployments typically involve gradually increasing the percentage of users who receive the new changes. This gradual rollout allows organizations to monitor system performance and user feedback at each stage closely.
• Canary testing often includes gathering feedback from the canary users. This feedback can be invaluable in identifying issues or usability concerns that automated testing may not catch.
• Successful canary testing builds confidence in the reliability and quality of software releases. Teams can have greater assurance that the new changes will not lead to major disruptions or outages when deployed to the entire user base.

When to Use Canary Testing?

Multiple instances arise where canary testing comes out as the right choice. Testers should perform this testing during the release of software changes, updates, or new features. Canary testing is a strategy employed during deployment to validate these changes before they are rolled out to the entire user base. Use canary testing:

• If there are changes to the underlying infrastructure, such as migrating to a new cloud provider, altering server configurations, or implementing load balancing.
• For security-related patches or updates.
• When optimizing application performance to verify that performance improvements are achieved without introducing new bottlenecks or regressions.
• In A/B testing scenarios to validate the behavior and performance of a new feature or user interface variant against the existing one.
• For mission-critical systems or applications to ensure that changes do not disrupt core functionalities and that any potential issues are detected early.

By default, developers run canary testing before releasing new system versions for their users to assess how they will perform. And before doing so, they unit test the code to prevent deployment and other minor issues. Further, it is always a good practice to run canary testing to see the code in action and with users before deciding if the new version is worth the release. The process is helpful for developers who work on continuous deployment/integration.

Another time when canary testing is the right option is when the test or staging environment does not match the conditions of the production environment. In such situations, running tests with a small group of users will help you detect problems in the live environment.

The timing of canary testing can vary based on the organization’s processes and deployment pipeline. It may occur just before a broader release or as part of a continuous delivery pipeline with incremental deployments. In any case, the goal is to catch issues early and ensure a smooth and reliable software release.

How to Do Canary Testing?

Developers perform canary testing by rolling out a certain feature type or functionality to a small set of users. The purpose is to verify if any issues exist in the new code before releasing it to all the users. Such an approach of limiting the distribution to a group of individuals ensures that the teams can validate and assess the new feature before making it available for everyone. Typically, developers choose 5 to 10% of the total users to execute canary testing.

Then, the output of the release is assessed by the teams; they receive feedback from the users. And developers work on improving the performance or fixing the issues in the code, if any, before making it live for a larger audience.

How Canary Testing Works?

There are several steps to the working procedure of canary testing, which includes deployment and gradual rolling out of the service for the users. Let’s look at all these steps in sequence:

• Plan and segment the users who will receive the new update of the application for testing.
• Deploying the update to this selected group of users using a feature flag toggle or tool is the next step. You can read more about this approach in the feature flag section of the same blog.
• During the canary deployment, extensive monitoring and logging are in place to track various performance metrics, such as response times, error rates, and resource utilization.
• If the canary release behaves as expected and shows stable performance, the deployment is gradually expanded to include a larger portion of the user base or server infrastructure. It can either be automatic or manual, depending on the organization’s processes and tools.
• Engineers and testers closely analyze the data collected during the canary release phase. They compare the performance and user experience of the new version with the old one. If any issues or anomalies are detected, they can roll back the release or make adjustments before continuing the rollout.
• Once the canary release successfully passes all performance and stability checks and demonstrates that it’s ready for full-scale deployment, it is rolled out to all users or servers.

Canary Test Deployment

A new version of the software, often containing new features, bug fixes, or updates, is prepared for release. Instead of deploying it to all users or servers at once, it is initially deployed to a small and carefully selected subset.

Planning

In the planning phase, define clear objectives for the canary release and select the initial user or server subset. Establish monitoring and data collection strategies to track performance metrics.

Implementing

During implementation, employ deployment methods such as feature flags or load balancer configurations to roll out the new version gradually. Automation tools can assist in managing the deployment expansion.

Analyzing

Once fully deployed, analyze performance and user feedback to assess whether objectives were met and issues identified. This data-driven approach forms decisions on whether to proceed, make adjustments, or roll back the release.

Running Canary Tests with Feature Flags

You can run canary testing using feature flags that release the code but don’t enable the features automatically. Developers can remotely switch the feature option on-off for the selected number of users. The feature flag method allows developers to target as small as 1% of users while monitoring metrics like error rates, latency, and business potential. It ensures that there are no negative effects on the users because of the release. Please note that most users will not see the new update as the feature flag will allow only Canary users to access it.

Moreover, you can easily toggle the feature flag and retract the code changes if users do not approve of it or if there are issues; disable the feature flag by turning it off. And once you are happy with the results, you can expose the new feature gradually to your user base. Lastly, remove the feature flag from your code, as you no longer need it.

Advantages of Canary Tests

Testing reduces issues in applications that impact user experience. Canary testing takes the overall idea a little further by updating the changes in the production environment with minimal or no damage to its usability. Here is a list of benefits that make canary testing a worthwhile process:

• It is easy to deploy canary tests. As the new features are small and only for a few users initially, running canary tests is easy and quick.
• You can also roll back the changes easily if your canary tests fail and users disapprove of the changes.
• The complete test process is low maintenance. Developers work on one small function, push it into production, and move on to the next one after analyzing and running the tests.
• Canary tests require no system downtime.
• Most of the canary tests require simple and small infrastructure to run. Everything is done internally, and the cost of fixing the issues is low as well.
• Developers can innovate and experiment with their code. Since the new code impacts only a small group of selected users, coders can play around with their code.
• You can involve users in the beta version of the programs. They will offer the right and timely feedback to fix the problems and move the code to production.

Challenges of Canary Tests

Along with advantages, there come challenges to everything. Here are some of the common drawbacks of canary testing:

• It is likely that the code change will affect users, albeit only a small percentage, if not done properly.
• It is possible that canary testing can be time-consuming and prone to errors without any automation. You might have information about the changes, but analyzing them requires automation. And usually, DevOps engineers work on it to oversee data collection and logs.
• This testing is not suitable for personal, standalone applications on user devices, including PC, smartphones, and laptops.
• If the users have varying versions of the application or the device, canary testing can become complex.

What is a Canary Release?

A canary release is a new version that developers make live for a limited group of users. This fresh code is a canary release; when you complete a feature or a code, you deploy it in one environment. Thereafter, you route the selected users to this canary release and wait for their feedback. Alongside this, you can also monitor the performance of the new release and eliminate any issues/bugs you come across. And because it impacts only a few users, rectifying the problem does not take much time, cost, or effort.

The goal here is to minimize the impact of issues in new code versions and release it for all the users or a wider audience in less time and effort. Canary releases are highly helpful in preventing major downtimes, lost revenue, or low customer satisfaction. It offers fast learning about the performance of a new feature without affecting the users beyond a specific number.

Overview of Canary Releases

We already know that a canary release is a deployment strategy where the new updates go live only for a subset of users. All the selected systems that receive the update are constantly monitored for any issues along with business KPIs. Thereafter, developers and stakeholders can decide to roll out the new feature for the entire customer base as per the results received from the canary test. If there are no adverse defects encountered with the new functionality, such as security or performance, you can gradually release the update to the remaining users.

Canary releases are different for web-based and mobile-based applications. For the former, you can implement a canary release by hosting two versions of your website. Take this example; making a Chrome extension available to only a few of the users to check the outcome before making it live worldwide can be considered a canary release.

For mobile application-based releases, you can make the latest version available to the group of users of your choice. Because you only have one environment (the user’s device), you cannot host two versions and choose which users get the new version. In such cases, a feature flag proves to be useful in releasing and monitoring canary releases. Suppose you wish to update the Search feature in an application to include exact keywords. You can use the feature flag option to do so without the need to roll it out for every app user.

Should You Execute Your Automated Tests Before Canary Testing?

You should execute your automated tests before running canary testing. It ensures that the piece of code you will make live for your selected users is free of bugs and passes the initial quality test. And typically, enterprises already have a process in place for testing their software changes. Many use A/B testing techniques and rely on DevOps to automatically develop, test, and deploy code modifications in the form of builds.

So, after you are done running automated tests on your new code, push it to the production environment for the users. And start the canary testing process after that.

Using a Test Automation Tool for Execution of Automated Tests Before Canary Testing

To run automation tests before moving on to canary testing, use a test automation tool that offers a high degree of control over your code, all the while recording and reporting the outcome to you. The right automation tool should offer you the capability to execute test cases on web and mobile applications because canary testing will include these devices/environments.

Testsigma is the perfect test automation platform that enables testers to run the web. Mobile and API testing without the need for coding. It is a cloud-based, AI-powered, open-source, no-code automation tool that supports all your testing efforts, including cross-browser testing, continuous integration, visual testing, and more. Explore the platform’s features here and register for a free demo to see all of it in action yourself.

From test development and execution on the cloud to AI-driven test automation and test data management, Testsigma does this and more to automate your tests 5X faster.

Start automating your tests for web, mobile, desktop, and APIs, 5x faster with Testsigma

Conclusion

Changes are inevitable, and even in SDLC, the software undergoes multiple changes and updates. How do you make sure that they are working right? You release the new feature to only a certain number of users and assess their feedback. These releases are called canary releases. And canary testing is doing just this. It helps to save downtime and cost in rolling out features to all the users and withdrawing them when an issue occurs.

You run automated tests on the fresh code before making it live for your users and running canary tests. This blog gives you a comprehensive idea about canary testing, it’s working, and its importance.

Frequently Asked Questions

What are the benefits of canary release?

Canary release is beneficial in rolling out important features that must function correctly in the production environment before going to all the users. It is helpful in performing A/B testing without shutting down the software. One another benefit of canary release is that you can easily test the capacity of the large production environment little by little without hampering its uptime.

Why is it called canary testing?

The name of the canary test is based on the old age practice followed by the coal miners. They would take a canary down to the mine, and if the canary died, it indicated that a toxic, poisonous gas was slowly building up in the cave. This way, they would know when to vacate.

Similarly, in canary testing, the code is first given access to a limited number of users. And only after the feedback is positive is it made live, incrementally, for the rest of the users.