Testing In Production: A Complete Guide
When we complete development of certain features and are ready to release this to a live environment so our customers can start using it – we need to ensure it works as intended. Ensuring this is not easy or simple as it means having certain processes and testing in place. One of the processes that comes super handy here is “testing in production”.
Testing in production can be tricky as that is where product is live and customers use it for real. Even then, there are certain teams that like to take this risk of testing in production. They do execute their smoke tests in production.
In this blog we will look at what production testing is all about, the benefits it brings to the table, the drawbacks and the right process to do it.
Table Of Contents
What Is Testing in Production?
Production testing is a software development practice in which new code changes are tested on live user traffic rather than in a staging environment.” Testing in production does not mean to replace other functional and non-functional tests, but complement them.
The question arises in many teams whether its worth performing this type of testing or not. It mainly comes down to risk management for the product going live. It can be quite beneficial for gathering real-world feedback and improving the application, but should only be done with caution to minimise the impact on its users.
What are really the big risks:
- You risk deploying bad code
- You may accidentally leak sensitive data
- It can possibly cause system overload
- You can mess up your tracking and analytics
- You risk releasing a poorly designed product or feature
Automating tests for testing in production is another area that may be worth looking into. Testsigma can support this. Continuously testing an application while moving through the continuous delivery pipeline provides instant visibility into potential failure risks every time there is a new check-in. The added efficiency helps the organizations to deliver error-free software without delay.
Automate your production tests, for web, mobile, desktop and APIs, for better efficiency and effectiveness, with Testsigma
It all comes down to the team and testing strategy of whether this type of testing is something your team wants to achieve or not.
Benefits of testing in production
There are various benefits of testing live in production. If it is done properly with the right process it really does add value. Remember, production testing enables continuous deliveries which is positive as users won’t have to keep waiting longer for each release.
- Find bugs – With this type of testing as you are testing on live, users can discover bugs or any other issues that you might not find.
In some situations there may be issues that won’t be visible or obvious until the users use it and find out. Therefore monitoring the performance of the application and the releases to production would be a good practice as it will help optimize future releases.
- Quality releases– Testing in production is also a form of continuous feedback from users, therefore helps improve features of the application and release them to meet the users needs. This also helps with maintainability, reliability and scalability too.
- Collection of real-world data and fast feedback– Testing in production helps collect real user data and helps understand the engagement of the users with the application released. This is all valuable feedback and really important as well, as it helps optimize the application based on user experience. It will also help with future features that the team may not have thought of before.
In general it also provides the following:
- Better Performance Testing
- Beta Programs With Instant Feedback
- Fast Problem Detection
- Live App Performance Monitoring
- Running Unpredictable Scenarios
- Evaluating User Experience Effectiveness Objectively
- Continuous Testing Feedback
- Deployment Strategies and Methods.
Drawbacks of testing in production
- Responsibility in testing – A this testing is done in production, it is risky and requires to executed more responsibly as compared to other types of testing done on test environments. That said, testing in production will only be effective if the team collaborates fully and does not neglect other types of testing.
- Time for testing needs to be considered carefully – If you are doing load testing on production, it can lead to poor user experiences. Therefore select a time in the day in which it might not impact the users such as maintenance hours. As user experience is tied with profit, a tester not adhering to the recommended timings can lower the business revenue.
- Chaos Monkey – This is when on purpose you add chaos in the production environment, to understand everything works as intended. The things chaos monkey injects in are things like failures and this could damage the web server. Therefore this needs to be done carefully to drive a resilient application.
Test Automation Pre Production vs Post Production
Pre production testing
This is the stage when product development takes place and testing takes place alongside too. You can use great tools to automate your tests too at this stage – some suggested tools are Testsigma, Selenium, Appium, JUnit, Pytest, Robot, Mocha, Jest, etc. You can then use these tools to connect to a live environment and run your automated checks.
Post production
Once the product has successfully been released, that’s when it becomes even more important to monitor its behavior. This will then help improve the product and maintain it. Therefore you could run some automated checks again, on production, that can provide some sense of confidence in the performance of the application. Also simple API tests could help fetch test results data for decision making within your continuous integration and deployment pipeline.
Tools that can help in Production Testing
Testsigma
Testsigma is an no-code test automation platform for automating your end-to-end tests for web, mobile, desktop and APIs. Testsigma is completely cloud-based, thus you are able to author, debug and execute your tests on Testsigma. To execute you can run your tests on devices available on cloud labs or local ones, depending on your choice. It also integrates seamlessly with all popular CICD tools.
DataDog
DataDog is a comprehensive monitoring tool with amazing tracing capabilities. This helps a lot in debugging with a very low overhead.
New Relic
A very strong APM tool, which offers log management, AI ops, monitoring and more.
Lightrun
Lightrun is a powerful production debugger. It enables adding logs, performance metrics and traces to production and staging in real-time, on demand. Lightrun enables developers to securely add instrumentation without having to redeploy or restart.
Prometheus
Prometheus is an open source monitoring solution that includes metrics scraping, querying, visualization and alerting.
Conclusion
Today, majority of software development follows agile methodology. There are certain testing types like functional or non-functional such as CI/CD, A/B testing, Canary/blue/green deployments, usability testing, sanity and smoke testing – which are done in live production environment.
The main responsibility here is to make sure its done in a safe manner and does not impact the real users. The goal is that the testing should provide enough information to improve the product and the user experience. The production testing can be done manually or can be automated too.
Frequently Asked Questions
What is an example of testing in production?
Let’s take the example of social media platform facebook; let’s say they added a new feature in the application. They do some testing in test environments and release it for the public to use. While public is using it, they also run some tests on the same application, this testing will be called production testing.
Is testing in production a good idea?
Testing in production can be risky and should only be done if the testers understand the risks and are able to execute the production tests without impacting the usability and performance of the application.
What are the risks of testing in production?
- Risk of deploying bad code
- Accidentally leaking sensitive data
- Possibility of system overload
- Messing up the analytics
- Risk of releasing product or feature that is not usable
