When we complete development of certain features and are ready to release this to a live environment so our customers can start using it – we need to ensure it works as intended. Ensuring this is not easy or simple as it means having certain processes and testing in place. One of the processes that comes super handy here is “testing in production”.
Testing in production can be tricky as that is where product is live and customers use it for real. Even then, there are certain teams that like to take this risk of testing in production. They do execute their smoke tests in production.
In this blog we will look at what production testing is all about, the benefits it brings to the table, the drawbacks and the right process to do it.
Table Of Contents
- 1 What Is Testing in Production?
- 2 Why Test In Production And Its Importance?
- 3 Benefits of Testing in Production
- 4 Types Of Testing In Production
- 5 Drawbacks of Testing in Production
- 6 Test Automation Pre Production vs Post Production
- 7 How To Perform Production Testing?
- 8 Best Practices for Testing in Production
- 9 Tools that can Help in Production Testing
- 10 Conclusion
- 11 Frequently Asked Questions
What Is Testing in Production?
Production testing is a software development practice in which new code changes are tested on live user traffic rather than in a staging environment.” Testing in production does not mean to replace other functional and non-functional tests, but complement them.
The question arises in many teams whether its worth performing this type of testing or not. It mainly comes down to risk management for the product going live. It can be quite beneficial for gathering real-world feedback and improving the application, but should only be done with caution to minimise the impact on its users.
What are really the big risks:
- You risk deploying bad code
- You may accidentally leak sensitive data
- It can possibly cause system overload
- You can mess up your tracking and analytics
- You risk releasing a poorly designed product or feature
Automating tests for testing in production is another area that may be worth looking into. Testsigma can support this. Continuously testing an application while moving through the continuous delivery pipeline provides instant visibility into potential failure risks every time there is a new check-in. The added efficiency helps the organizations to deliver error-free software without delay.
Automate your production tests, for web, mobile, desktop and APIs, for better efficiency and effectiveness, with Testsigma
It all comes down to the team and testing strategy of whether this type of testing is something your team wants to achieve or not.
Why Test In Production And Its Importance?
Production testing is crucial as it is not possible to find all the bugs in the development and staging environments. Regardless of the efforts of engineering and QA teams, some bugs may still go unnoticed. So, users may encounter software issues even after extensive testing during development.
Also, it is not possible to simulate live and real-world scenarios in a test environment. That is why production testing has become crucial in DevOps and testing these days.
Benefits of Testing in Production
There are various benefits of testing live in production. If it is done properly with the right process it really does add value. Remember, production testing enables continuous deliveries which is positive as users won’t have to keep waiting longer for each release.
- Find bugs – With this type of testing as you are testing on live, users can discover bugs or any other issues that you might not find.
In some situations there may be issues that won’t be visible or obvious until the users use it and find out. Therefore monitoring the performance of the application and the releases to production would be a good practice as it will help optimize future releases.
- Quality releases– Testing in production is also a form of continuous feedback from users, therefore helps improve features of the application and release them to meet the users needs. This also helps with maintainability, reliability and scalability too.
- Collection of real-world data and fast feedback– Testing in production helps collect real user data and helps understand the engagement of the users with the application released. This is all valuable feedback and really important as well, as it helps optimize the application based on user experience. It will also help with future features that the team may not have thought of before.
In general it also provides the following:
- Better Performance Testing
- Beta Programs With Instant Feedback
- Fast Problem Detection
- Live App Performance Monitoring
- Running Unpredictable Scenarios
- Evaluating User Experience Effectiveness Objectively
- Continuous Testing Feedback
- Deployment Strategies and Methods.
Types Of Testing In Production
The following are some of the types of testing in production,
- A/B Testing: Here, the test is performed by showing two different product versions to different users.
- Load Testing: To evaluate how the application is handling normal or expected user loads.
- Canary Releases: This involves displaying a product version to a subset of users and gathering feedback before releasing the software to the public.
- Feature Flagging: This involves using conditional toggles to expose or hide features to users.
- Production User Acceptance Testing: UAT testing in production is performed with real-world scenarios by end-users or a specific group of testers/developers.
- Visual regression testing: To identify any unintended visual changes in the application’s user interface.
- Disaster recovery testing: To validate the application’s disaster recovery plan and its ability to restore operations after a catastrophic event.
- Accessibility testing: To check if the application is accessible to everyone and complies with accessibility standards.
- Performance Testing: To evaluate the application’s responsiveness, speed, and overall performance under different conditions.
- Stress Testing: To evaluate the application’s stability and reliability under extreme or unexpected user loads that are beyond normal operational capacity.
- Integration Testing: To verify the interactions and data flow between different software components or modules.
- Spike Testing: To evaluate the application’s performance under sudden, significant increases in load or activity.
Check here – Canary Testing
Drawbacks of Testing in Production
- Responsibility in testing – A this testing is done in production, it is risky and requires to executed more responsibly as compared to other types of testing done on test environments. That said, testing in production will only be effective if the team collaborates fully and does not neglect other types of testing.
- Time for testing needs to be considered carefully – If you are doing load testing on production, it can lead to poor user experiences. Therefore select a time in the day in which it might not impact the users such as maintenance hours. As user experience is tied with profit, a tester not adhering to the recommended timings can lower the business revenue.
- Chaos Monkey – This is when on purpose you add chaos in the production environment, to understand everything works as intended. The things chaos monkey injects in are things like failures and this could damage the web server. Therefore this needs to be done carefully to drive a resilient application.
Test Automation Pre Production vs Post Production
Pre production testing
This is the stage when product development takes place and testing takes place alongside too. You can use great tools to automate your tests too at this stage – some suggested tools are Testsigma, Selenium, Appium, JUnit, Pytest, Robot, Mocha, Jest, etc. You can then use these tools to connect to a live environment and run your automated checks.
Post production
Once the product has successfully been released, that’s when it becomes even more important to monitor its behavior. This will then help improve the product and maintain it. Therefore you could run some automated checks again, on production, that can provide some sense of confidence in the performance of the application. Also simple API tests could help fetch test results data for decision making within your continuous integration and deployment pipeline.
How To Perform Production Testing?
There are two approaches to production testing, which are discussed in detail below,
1. A/B Testing
A/B testing is one of the most commonly used approaches in production environments. Here, the user gets to see two different product versions and can decide which one is better. This provides valuable feedback to the product teams, helping them choose the right version.
2. Canary Releases
Similar to A/B testing, Canary releases introduce a new product version to a subset of users for close monitoring. The released versions undergo small functionality changes before the entire product is released to the entire user base.
Best Practices for Testing in Production
Testing in production can enhance your testing strategy when done correctly. So, remember to follow these best practices for an effective production testing,
- Use Feature Flags
Feature flagging helps you test new features in the production environment, allowing easy rollback with a ‘kill switch’ if needed. This allows you to evaluate new components in the real world, as you can not entirely mimic production in staging.
- Test In Peak Hours
Be considerate of the timings when you conduct tests. For example, performing load tests during business hours can greatly impact user experience compared to performing the same tests during non-business hours or maintenance.
- Closely Monitor The Tests
Use robust monitoring to track server and database activities. Effective monitoring is crucial for successful testing in production.
- Regression Testing
Automated regression testing during production ensures that any changes do not affect existing functionality.
- Performance Testing
Use performance testing to evaluate the application’s speed, scalability, and reliability to identify potential bottlenecks.
- Stress Testing
Use stress testing to evaluate if the application can handle heavy unexpected user laods.
- Sanity Testing
Perform sanity testing to check that the new software build works as expected.
Tools that can Help in Production Testing
Testsigma
Testsigma is an no-code test automation platform for automating your end-to-end tests for web, mobile, desktop and APIs. Testsigma is completely cloud-based, thus you are able to author, debug and execute your tests on Testsigma. To execute you can run your tests on devices available on cloud labs or local ones, depending on your choice. It also integrates seamlessly with all popular CICD tools.
DataDog
DataDog is a comprehensive monitoring tool with amazing tracing capabilities. This helps a lot in debugging with a very low overhead.
New Relic
A very strong APM tool, which offers log management, AI ops, monitoring and more.
Lightrun
Lightrun is a powerful production debugger. It enables adding logs, performance metrics and traces to production and staging in real-time, on demand. Lightrun enables developers to securely add instrumentation without having to redeploy or restart.
Prometheus
Prometheus is an open source monitoring solution that includes metrics scraping, querying, visualization and alerting.
Conclusion
Today, majority of software development follows agile methodology. There are certain testing types like functional or non-functional such as CI/CD, A/B testing, Canary/blue/green deployments, usability testing, sanity and smoke testing – which are done in live production environment.
The main responsibility here is to make sure its done in a safe manner and does not impact the real users. The goal is that the testing should provide enough information to improve the product and the user experience. The production testing can be done manually or can be automated too.
Frequently Asked Questions
What is an example of testing in production?
Let’s take the example of social media platform facebook; let’s say they added a new feature in the application. They do some testing in test environments and release it for the public to use. While public is using it, they also run some tests on the same application, this testing will be called production testing.
Is testing in production a good idea?
Testing in production can be risky and should only be done if the testers understand the risks and are able to execute the production tests without impacting the usability and performance of the application.
What are the risks of testing in production?
- Risk of deploying bad code
- Accidentally leaking sensitive data
- Possibility of system overload
- Messing up the analytics
- Risk of releasing product or feature that is not usable
