What is Canary Testing? A Comprehensive Overview

Canary testing is helpful in checking the new functions and features of an application without impacting the user experience or keeping it to a minimum. Typically, people interchange the terms canary testing, canary releases, and canary deployment, but they differ in reality.

In this blog, we will guide you through canary testing information. We will further talk about these different terms and their meaning for a better understanding.

What is Canary Testing?

Canary testing is a testing technique in which the new features are released to a small part of the user group rather than a full release to all the users at once. This is done to analyze the user’s response to those features and monitor any failures or bugs. Since the impact area is small, the bugs can be confined to a small group and be fixed quickly without any major consequences.

Why should you Perform Canary Testing?

Canary testing brings many benefits to the team of developers and testers working behind the application.

• Minimizes the risk: Canary testing releases the software to a small group of people first. Hence, even if there are high-intensity failures, they are confined to a small population whose impact does not propagate to everyone.

• Detect issues early: Since canary testing precedes the full production release, issues detected in it can be fixed safely before that. It helps the team manage the software comfortably.

• Easier rollbacks: Canary testing pushes the code to a very small part of their user base. This can be achieved with a single server management system where the users can be redirected if they are among those who are testing the application. Hence, the rollbacks would not require too many configuration changes and they are easier to handle.

• Real feedback from real users: There is always a need to attain feedback from real users because they are not technical and, most importantly, do not work with a fixed pattern like a tester would. Feedback from the canary testing is in its rawest form because the user doesn’t know that he is part of the testing group. The team analyses their behavior and figures out the loopholes themselves. This helps the team witness their application through an end-user which is what counts the most in the end.

• Improves overall quality: Canary testing helps detect issues early and eliminate bugs before the software is released to all users. This 360-degree inspection of the application helps improve the quality and release bug-free software to the users.

• Understand real-world scenarios: Feedback generated through canary testing is purely observational apart from the functional errors notified through monitoring systems directly. Such observations are generated through the user’s behavior and pattern which keeps on changing with time. Hence, canary testing becomes a window to analyze how real-world scenarios are executed today and helps shape the application as close to this world as possible.

Apart from these essential benefits, canary testing also helps build deployment trust, incite confidence for release in the team, and stabilizes the user’s trust in the application and the business.

Why is Canary Testing Effective?

Canary testing is effective for several reasons, making it a valuable approach in software development and deployment:

• Canary testing allows organizations to minimize the risk associated with deploying new features or updates to a large user base. Initially, rolling out changes to a small subset of users can detect and address any issues or bugs before they affect the entire user population.
• Canary deployments provide early feedback on the performance and stability of new changes. If problems arise, they are identified early in the process, making fixing them easier and less costly.
• Canary deployments typically involve gradually increasing the percentage of users who receive the new changes. This gradual rollout allows organizations to monitor system performance and user feedback at each stage closely.
• Canary testing often includes gathering feedback from the canary users. This feedback can be invaluable in identifying issues or usability concerns that automated testing may not catch.
• Successful canary testing builds confidence in the reliability and quality of software releases. Teams can have greater assurance that the new changes will not lead to major disruptions or outages when deployed to the entire user base.

When to Use Canary Testing?

Multiple instances arise where canary testing comes out as the right choice. Testers should perform this testing during the release of software changes, updates, or new features. Canary testing is a strategy employed during deployment to validate these changes before they are rolled out to the entire user base. Use canary testing:

• If there are changes to the underlying infrastructure, such as migrating to a new cloud provider, altering server configurations, or implementing load balancing.
• For security-related patches or updates.
• When optimizing application performance to verify that performance improvements are achieved without introducing new bottlenecks or regressions.
• In A/B testing scenarios to validate the behavior and performance of a new feature or user interface variant against the existing one.
• For mission-critical systems or applications to ensure that changes do not disrupt core functionalities and that any potential issues are detected early.

By default, developers run canary testing before releasing new system versions for their users to assess how they will perform. And before doing so, they unit test the code to prevent deployment and other minor issues. Further, it is always a good practice to run canary testing to see the code in action and with users before deciding if the new version is worth the release. The process is helpful for developers who work on continuous deployment/integration.

Another time when canary testing is the right option is when the test or staging environment does not match the conditions of the production environment. In such situations, running tests with a small group of users will help you detect problems in the live environment.

The timing of canary testing can vary based on the organization’s processes and deployment pipeline. It may occur just before a broader release or as part of a continuous delivery pipeline with incremental deployments. In any case, the goal is to catch issues early and ensure a smooth and reliable software release.

Check here – Deployment Testing

How to Do Canary Testing?

Canary testing is done in stages where each stage is carefully planned to provide new functionalities to a select group of users.

• User selection: The first and most crucial step is to select the users that will receive the new updates. This group can be specific to the new features or a random selection. However, it is always advised to carefully select a group that is diverse in all parameters to accumulate the best feedback. Also, user selection has to be run repeatedly each time canary testing is executed as different requirements will shortlist different users.

• Production release setup: In this step, the team prepares the setup for production release. While the app is released for a small part of the audience, it should be prepared and released like a complete full release. This step will also include server reconfigurations or a new small server setup through which the newer features will be redirected to the selected user base.

• Deploy: Once the servers are configured and the setup is ready for release, the team can deploy the binary to the server from where it will be fetched by the users like a regular update not knowing that they are part of the testing process.

• Analyze, observe, and monitor: Now is the time to analyze the user’s behavior and feedback and monitor all the parameters to spot any bugs. This part of canary testing can be seen as equivalent to test execution in functional testing.

• Scale up: If all the metrics and vitals seem fine, start scaling up the canary testing process. This means increasing the testing group and slowly releasing the feature to more people. Then, repeat the analysis phase after each phase of scaling.

• Full release: In this final step, if the team is confident about their features and all past bugs have been eliminated, they should aim for a full release to all their users.

If these steps are followed in the same order, smooth canary testing can be executed easily.

Canary Test Deployment

A new version of the software, often containing new features, bug fixes, or updates, is prepared for release. Instead of deploying it to all users or servers at once, it is initially deployed to a small and carefully selected subset.

Planning

In the planning phase, define clear objectives for the canary release and select the initial user or server subset. Establish monitoring and data collection strategies to track performance metrics.

Implementing

During implementation, employ deployment methods such as feature flags or load balancer configurations to roll out the new version gradually. Automation tools can assist in managing the deployment expansion.

Analyzing

Once fully deployed, analyze performance and user feedback to assess whether objectives were met and issues identified. This data-driven approach forms decisions on whether to proceed, make adjustments, or roll back the release.

Running Canary Tests with Feature Flags

You can run canary testing using feature flags that release the code but don’t enable the features automatically. Developers can remotely switch the feature option on-off for the selected number of users. The feature flag method allows developers to target as small as 1% of users while monitoring metrics like error rates, latency, and business potential. It ensures that there are no negative effects on the users because of the release. Please note that most users will not see the new update as the feature flag will allow only Canary users to access it.

Moreover, you can easily toggle the feature flag and retract the code changes if users do not approve of it or if there are issues; disable the feature flag by turning it off. And once you are happy with the results, you can expose the new feature gradually to your user base. Lastly, remove the feature flag from your code, as you no longer need it.

Advantages of Canary Tests

Testing reduces issues in applications that impact user experience. Canary testing takes the overall idea a little further by updating the changes in the production environment with minimal or no damage to its usability. Here is a list of benefits that make canary testing a worthwhile process:

• It is easy to deploy canary tests. As the new features are small and only for a few users initially, running canary tests is easy and quick.
• You can also roll back the changes easily if your canary tests fail and users disapprove of the changes.
• The complete test process is low maintenance. Developers work on one small function, push it into production, and move on to the next one after analyzing and running the tests.
• Canary tests require no system downtime.
• Most of the canary tests require simple and small infrastructure to run. Everything is done internally, and the cost of fixing the issues is low as well.
• Developers can innovate and experiment with their code. Since the new code impacts only a small group of selected users, coders can play around with their code.
• You can involve users in the beta version of the programs. They will offer the right and timely feedback to fix the problems and move the code to production.

Challenges of Canary Tests

Along with advantages, there come challenges to everything. Here are some of the common drawbacks of canary testing:

• It is likely that the code change will affect users, albeit only a small percentage, if not done properly.
• It is possible that canary testing can be time-consuming and prone to errors without any automation. You might have information about the changes, but analyzing them requires automation. And usually, DevOps engineers work on it to oversee data collection and logs.
• This testing is not suitable for personal, standalone applications on user devices, including PC, smartphones, and laptops.
• If the users have varying versions of the application or the device, canary testing can become complex.

What is a Canary Release?

A canary release is a new version that developers make live for a limited group of users. This fresh code is a canary release; when you complete a feature or a code, you deploy it in one environment. Thereafter, you route the selected users to this canary release and wait for their feedback. Alongside this, you can also monitor the performance of the new release and eliminate any issues/bugs you come across. And because it impacts only a few users, rectifying the problem does not take much time, cost, or effort.

The goal here is to minimize the impact of issues in new code versions and release it for all the users or a wider audience in less time and effort. Canary releases are highly helpful in preventing major downtimes, lost revenue, or low customer satisfaction. It offers fast learning about the performance of a new feature without affecting the users beyond a specific number.

Overview of Canary Releases

We already know that a canary release is a deployment strategy where the new updates go live only for a subset of users. All the selected systems that receive the update are constantly monitored for any issues along with business KPIs. Thereafter, developers and stakeholders can decide to roll out the new feature for the entire customer base as per the results received from the canary test. If there are no adverse defects encountered with the new functionality, such as security or performance, you can gradually release the update to the remaining users.

Canary releases are different for web-based and mobile-based applications. For the former, you can implement a canary release by hosting two versions of your website. Take this example; making a Chrome extension available to only a few of the users to check the outcome before making it live worldwide can be considered a canary release.

For mobile application-based releases, you can make the latest version available to the group of users of your choice. Because you only have one environment (the user’s device), you cannot host two versions and choose which users get the new version. In such cases, a feature flag proves to be useful in releasing and monitoring canary releases. Suppose you wish to update the Search feature in an application to include exact keywords. You can use the feature flag option to do so without the need to roll it out for every app user.

Should You Execute Your Automated Tests Before Canary Testing?

You should execute your automated tests before running canary testing. It ensures that the piece of code you will make live for your selected users is free of bugs and passes the initial quality test. And typically, enterprises already have a process in place for testing their software changes. Many use A/B testing techniques and rely on DevOps to automatically develop, test, and deploy code modifications in the form of builds.

So, after you are done running automated tests on your new code, push it to the production environment for the users. And start the canary testing process after that.

Using a Test Automation Tool for Execution of Automated Tests Before Canary Testing

To run automation tests before moving on to canary testing, use a test automation tool that offers a high degree of control over your code, all the while recording and reporting the outcome to you. The right automation tool should offer you the capability to execute test cases on web and mobile applications because canary testing will include these devices/environments.

Testsigma is the perfect test automation platform that enables testers to run the web. Mobile and API testing without the need for coding. It is a cloud-based, AI-powered, open-source, no-code automation tool that supports all your testing efforts, including cross-browser testing, continuous integration, visual testing, and more. Explore the platform’s features here and register for a free demo to see all of it in action yourself.

From test development and execution on the cloud to AI-driven test automation and test data management, Testsigma does this and more to automate your tests 10x faster.

Conclusion

Changes are inevitable, and even in SDLC, the software undergoes multiple changes and updates. How do you make sure that they are working right? You release the new feature to only a certain number of users and assess their feedback. These releases are called canary releases. And canary testing is doing just this. It helps to save downtime and cost in rolling out features to all the users and withdrawing them when an issue occurs.

You run automated tests on the fresh code before making it live for your users and running canary tests. This blog gives you a comprehensive idea about canary testing, it’s working, and its importance.

Frequently Asked Questions

Is canary testing better than blue-green testing?

Canary testing and blue-green testing, are both used on different occasions and therefore their comparison cannot be done directly. Canary testing is a better choice when the application is big, there are scalability risks, and where there are gradual incremental updates. Blue-green testing is better for simpler applications where server resources are not heavy (as two of them will cost twice) and the updates rolled out are short but critical.

How do we determine when canary testing makes sense?

To make sense of canary testing, it should be only used to its advantage rather than in any situation. Canary testing releases the software to a small group, often starting with a part of release functionalities, and gradually scaling up. If the functionalities are low-risk or the application is small, canary testing will only increase the budget without providing anything in return. Instead, it should be used when there are critical functionalities (those that cannot afford to be used with failure) and the application is big and has a large user base. Only when these conditions are satisfied, canary testing will make sense.

What are the benefits of canary release?

Canary release is beneficial in rolling out important features that must function correctly in the production environment before going to all the users. It is helpful in performing A/B testing without shutting down the software. One another benefit of canary release is that you can easily test the capacity of the large production environment little by little without hampering its uptime.

Why is it called canary testing?

The name of the canary test is based on the old age practice followed by the coal miners. They would take a canary down to the mine, and if the canary died, it indicated that a toxic, poisonous gas was slowly building up in the cave. This way, they would know when to vacate.

Similarly, in canary testing, the code is first given access to a limited number of users. And only after the feedback is positive is it made live, incrementally, for the rest of the users.