Configuration Testing: How to Perform?

An application doesn’t just run on the code used in its development. Device settings, device operating software, specifications, and hardware, all contribute to a successful and smooth application execution. These variables are termed configurations of a device and their testing is called configuration testing.

Configuration testing opens up possibilities of minimizing production bugs that have no relation to the code logic or business requirements. It focuses on how an application works on various device configurations and ensures that the user experience remains consistent across all these devices. However, this is not an easy feat to achieve. Once the team dips into this area, they are flooded with challenges and complex problems that require knowledge and exposure as provided in this post along with solutions and handling procedures in detail.

What is Configuration Testing?

Configuration testing is a type of software testing that verifies the performance of a system under different configurations of hardware and software. It is important to perform this testing because different configurations can affect the system’s behavior. For example, if the system is configured with different hardware, it may need to be able to run faster and more efficiently.

Here is an analogy to help you understand this testing:

Imagine you are buying a new car. You want to ensure the car performs well, so take it for a test drive. However, you do more than test-drive the car once. You test drive it on different roads, in different weather conditions, and with different types of cargo. This is because you want to ensure the car performs well in all possible configurations.

Configuration testing is similar to this. You are testing the system under different configurations to ensure it performs well in all possible configurations.

Importance of Configuration Testing

In the highly fragmented device world with more unique devices arriving each year, configuration testing becomes more important than ever to deliver a high-quality application.

• Consistent user experience: Configuration testing ensures that the application works consistently across different devices. Hence, the user experience too remains consistent for all users no matter what device they use.

• Minimizes bugs: As much as 15% of overall bugs that arise in an application are due to configuration incompatibility. Once these bugs navigate to production, they are extremely costly to repair and manage. Configuration testing helps minimize such bugs (and overall bug quantity) and ultimately enhance the app’s quality.

• Increase in user outreach: Configuration testing ensures that the application is stable in all types of configurations. Hence, it does not restrict any users from using the application solely based on the configuration of their device which ultimately increases the outreach.

• Saves costs: All the bugs identified at this stage would help reduce the load on developers and testers in maintaining and fixing production issues. The time invested can be used in other productive work which will be the best use of business costs that will also return in the future.

• Build business reputation: Reduction in bugs, increased outreach, and enhanced user experience are all good ingredients in building a business reputation and spreading positive word of mouth among peers. A good reputation invites more users and helps in business growth.

Configuration Testing Example

A company is developing a new software application that customers will use on different operating systems. The company must perform this testing to ensure the application works correctly on all supported operating systems.

Here are some of the things that the company might need to test:

• The application’s ability to install and run on different operating systems.
• The application’s ability to use other features of different operating systems.
• The application can handle various errors and exceptions on other operating systems.

The company could perform manual configuration testing by manually installing the application on different operating systems and testing the application to see if it still works correctly. Or, the company could use an automated configuration testing tool to automatically install the application on different operating systems and then test the application.

By performing configuration testing, the company can help guarantee that the application works correctly on all supported operating systems and is not vulnerable to attack.

Also Read: Testing on Emulators vs Simulators

Pre-requisites for Configuration Testing

Here are the pre-requisites

• Clear understanding of the system: The tester must have a clear account of the system, including its components, interfaces, and dependencies.
• Complete configuration documentation: The tester must have complete configuration documentation, which should include the different configurations that need to be tested and the expected results for each configuration.
• Access to the system: The tester must have access to the system to test it.
• Tools and resources: The tester must have the necessary tools and resources to perform this testing, such as configuration management and test automation tools.
• Time and budget: This testing can be time-consuming and expensive, so the tester must have the necessary time and budget to complete the testing.

Configuration Testing Objectives

Here are the key objectives the team should plan to achieve from configuration testing.

1.  Verify the software system is correctly configured according to the specified requirements and settings.
2. Identify any configuration errors or inconsistencies that may impact the functionality or performance of the system.
3. Ensure all components, modules, and dependencies are correctly installed and integrated.
4. Validate the compatibility of the software with different hardware, operating systems, and network configurations.
5. Test the system’s ability to handle configuration changes or updates without causing disruptions or failures.
6. Assess the security measures implemented in the configuration to protect against unauthorized entry or data breaches.
7. Evaluate the system’s scalability and performance under different configuration scenarios.
8. Confirm that all configuration settings are correctly documented and can be easily replicated or restored.

Also Read: Different Types of Testing

Types of Configuration Testing

Configuration testing is divided into two major types; software and hardware.

Software configuration testing

Software configuration testing targets the software part such as the operating system and its versions. The goal behind this testing type is to ensure that the device software is compatible with the application under test. This can be achieved by installing and uninstalling the application and the target software repeatedly for all the test cases. 

Needless to say, this is a time-heavy job and to save time, the most recommended method is to choose a third-party cloud-based online software like Testsigma which eliminates this repeated job and instead, provides all the devices with unique configuration ready to be used with a couple clicks.

Hardware configuration testing

When the focus of configuration testing is hardware rather than software, it is termed hardware configuration testing. Here, the team doesn’t care much about the operating system version but the type of processor and its generation, for example. It is important to note that hardware configuration testing cannot be performed on virtual devices such as emulators and simulators. This is because the hardware metric is extremely important here, whose perfect values cannot be achieved by mimicking the device.

The best path to optimum hardware configuration testing is to use real devices with real hardware attached to them (either through connection or in-built). Again, choosing online tools is the recommended method to prevent additional costs of procurement and tester’s salaries.

Check here – Compatibility Testing

Configuration Testing Test Cases

An example of a mobile application is the best method to understand configuration testing in more depth.

Consider a mobile application that trades stocks on various stock exchanges. This is a highly sensitive application as a lot of personal data and financials are involved. A small breach into this platform can push the majority of users out to other competitors and land the business into multiple legal suites.

A stock trading application has to work perfectly in all configurations as the transactions are executed in milliseconds. It has to be scalable, reliable, and highly available for operations. Keeping these requirements in mind, one can draw the following configuration testing test cases for this scenario:

OS-specific case:

Check the system under different OS configurations.

Test Case ID	Configuration	Test case steps	Expected values
TC-01	OS: Android 11 Device: Samsung Galaxy S24UI Version: One UI 6.1RAM: 8 GBNetwork: 5G bandwidth	Install the applicationLogin to accountSelect a shareExecute 20 trades for the stockVerify the trade execution	Trades should get executed within 1 ms (or pre-decided threshold) and the purchased stocks should be visible in the account.
TC-02	OS: iOS 18.2.1Device: iPhone 16 UI Version: iOS 18RAM: 8 GBNetwork: 2G	Install the applicationLogin to accountSelect a shareExecute 20 trades for the stockVerify the trade execution	Trades should get executed within 3 seconds (or pre-decided threshold) and the purchased stocks should be visible in the account. If the delay is more than 3 seconds, the amount should be refunded to the user’s account immediately.

Similarly, more OS-based use cases can be applied.

Consistency-based test cases:

Check the consistency of the system.

Test Case ID	Configuration	Test case steps	Expected values
TC-03	OS: Android 11Device: Google Pixel 9RAM: 8 GBNetwork: 4G	Install the applicationLogin to accountSelect a shareNotice the change in stock value in real-time.Verify changes with real-time stock exchange API or web app.	The stock value in real-time should be reflected on the application with a refresh time of 1s and on each manual refresh request from the user.

Configuration testing with consistency tests is extremely important in such a sensitive application that shows the correct price of the stock all the time. Therefore, similar test cases should be followed with different network configurations for a high-quality application.

Load-specific test cases (Scalability):

Check the system performance under load for scalability issues.

Test Case ID	Configuration	Test case steps	Expected values
TC-04	OS: Android 11Device: One Plus 9RAM: 8 GBNetwork: 5G	Simulate an API request (such as buying the same share at the same time) for more than 20000 users.	The API should be executed successfully, with shares credited to each user’s account.

The load can be expanded into various dimensions such as user creation, selling the stocks, millions of users requesting the price of the same share simultaneously, etc.

Security testing:

Check the security of the system.

Test Case ID	Configuration	Test case steps	Expected values
TC-05	OS: iOS 18.2.1Device: iPhone 16 UI Version: iOS 18RAM: 8 GBNetwork: 5G	Simulating attacks on the database Injection simulation Phishing	The infrastructure should be strong enough to resist injection activities, and the application layer should not be breached for phishing activities.

Security is a crucial element especially when such sensitive operations are involved. It should be prioritised in configuration testing, or any other type of testing, if that matters.

Check here – Parallel Testing

How to do Configuration Testing

Here are the steps on how to perform in a step-by-step process:

1. Plan: This involves identifying the configurations that need to be tested and the expected results for each configuration. The tester must also consider the time and budget constraints and the availability of resources.
2. Create test cases: The tester must create test cases verifying the configurations. The test cases should be clear, concise, and repeatable.
3. Execute the test cases: The tester must execute the test cases and record the results. If any defects are found, the tester must report them to the development team.
4. Analyze the results: The tester must analyze the results of the test cases to identify any trends or patterns. This information can be used to improve the test cases or to identify areas where the system is more likely to have defects.
5. Retest the system: If any defects are found, the system must be retested to ensure they have been fixed.
6. Document the results: The results of this testing must be documented so they can be used for future reference.

Configuration Testing Comparison with other Testing

A testing cycle consists of multiple testing phases that target different areas of an application. While configuration testing is a crucial element of this phase, so are functional testing, compatibility testing, cross-browser testing, and performance testing. Making its presence a priority in such a crowded world is no easy task. Hence, a brief comparison with each critical testing phase is important to understand the stance of configuration testing in this crowd.

Configuration testing vs compatibility testing

Compatibility testing tests the application’s compatibility with different devices and platforms. It differs from configuration testing as it does not involve configurations such as network variability and system hardware. All a compatibility testing cycle focuses on is how the application presents on various operating systems, browsers, and screen sizes.

Compatibility testing is a lot easier to conduct and manage as the variables are few and are easier to control as well. However, configuration testing and compatibility testing are often talked about interchangeably which is incorrect and may lead to certain anomalies.

Configuration testing vs Functional testing

Functional testing tests the software functionality and the correctness of its features. The main goal of implementing functional testing is to make sure that all business requirements are complete and that all these requirements work as expected – logically and functionally. However, there is no association of functional testing with different devices and their configurations. 

Generally, functional testing is meant to check the backend logic, and a single device is enough for this work. Configuration testing, however, can be considered an expansion of functional testing. Here, the team tests the functions on multiple devices after being assured that these functionalities work correctly in at least one system. Performing configuration testing after functional testing also ensures that the bug is in device variation and not in the functional logic.

Configuration testing vs. Cross-browser testing

Cross-browser testing, as the name suggests, focuses on verifying the application functionality on different browsers and their versions. Since different browser engines work differently, a lot of the time unique bugs may arise specific to certain browsers based on their engine’s working. Cross-browser testing aims to uncover those bugs. 

While there are certain device labs that you let you do cross browser manually like lambdatest and browserstack. Teams that do cross browser testing repeatedly and need many devices to be tested, go for test automation with tools that support multiple device types and save your test authoring time too – Testsigma is one such tool.

Cross-browser testing differs from configuration testing as hardware and other device specifications are not involved in cross-browser testing. While cross-browser testing is almost always used for user interface bugs, configuration testing focuses on the consistency of performance, compatibility, and scalability of applications on various devices with different configurations.

Configuration testing vs. Performance testing

Performance testing is an important metric when the application is scaling up. It helps analyze the current and future performance of the application based on various factors such as traffic and/or stress (such as low network latency). 

The main goal of performance testing is to ensure that the application never dips down certain threshold values. If the application performance goes below these threshold values – it not only creates a bad reputation for the business but also creates inconsistencies in the database and other areas of the application. 

While configuration testing tests the functionality and behavior of the application, a certain section of it also focuses on the performance of various configurations. However, the performance part here is only focused on nominal performance. For example, if the requirement is to get a response from an API in a maximum of 200 ms and configuration tests find it at 100 ms, the test will pass as the performance is accepted. However, in performance testing, various types of load will be put (such as mimicking 1 million simultaneous API requests), and the response time will be verified to be in bounds again.

These four types of tests are often compared and confused with configuration testing and therefore this section helps clear the air by introducing specific points of differences.

Best Practices in Configuration Testing

To perform configuration testing in the most optimum way, the following best practices should be followed:

• Record a matrix: A matrix is a document that keeps track of all the combinations used in configuration testing. It is essential because when a configurational bug arises, the team can inspect whether that device configuration was tested or not helping them discover the root cause for the issues. Since new devices keep arriving in the market, it is also important to update this matrix regularly.

• Use real devices: Configuration testing involves the device’s hardware which cannot be perfectly mimicked using a simulator or emulator. To achieve the exact parameter values, there is no better option than real devices as they are the same devices used by the end-user. For this, it is recommended to use cloud software that provides real devices rather than procuring physical ones.

• Use automation wherever possible: Automation saves time and efforts of the team that can be invested in other tasks. Configuration testing, unlike UI testing, does not contain actionable inputs and outputs due to which automation testing can be applied to a large section of the suite. Therefore, if your team needs to execute the same set of configuration tests multiple times, it is recommended to use automation and save costs for the team and businesses.

• Do not miss the non-functional requirements: Functional requirements and their impact have a high visibility due to which they are often preferred. Also, if the app doesn’t work functionally, users may never get to experience the non-functional part of the app.

However, non-functional aspects are as important as functional and should never be ignored. Their failure has a large impact on all the users on a very big scale that often costs businesses a fortune to recover. Security breaches and performance failures are two such examples.

•  Push testing left: Configuration testing seems like a job that is performed after the application is developed, and its performance needs to be measured. While this is true, the testers should not just “wait” for the time when they will get the application only to find bugs, and revert to developers, who will then fix and repeat the process. This whole loop takes a lot of time which can be cut short by shifting configuration testing to the left. Testers should initiate with testing the requirements of configuration testing sent to the developers at the start of the sprint when the functionality has not been developed yet. This way, developers will shape the application according to configuration requirements which will minimize the bugs and save time and costs.

Along with these mandatory best practices, the team can also include generic best practices such as ensuring high coverage at every step, optimizing strategies for resource utilization, and making sure that accessibility criteria are also involved in configuration testing steps.

Common Challenges in Configuration Testing and Solutions

Configuration testing invites certain challenges for which it is better to be prepared beforehand and implement solutions to avoid time investment and shorten the testing cycle.

Problem 1: Dynamic and fragmented world.

Explanation: The device and configurational world is highly fragmented and dynamic. There are a lot of options available and a lot of new options keep coming in a very short time frame. Due to this, many strategies and matrices need to be changed which consume a significant portion of the tester’s time in just the maintenance part.

Solution: While no one can change how many devices with new configurations are launched in a year, we can definitely work on how we manage these devices at our end. The best solution is to choose a cloud-based provider that keeps purchasing these devices and puts them up on their infrastructure to be used by third parties like us. Testsigma is a great example of such tools that are feasible and update their device arrays regularly.

Problem 2: User-specific customized configurations.

Explanation: A lot of the time an end-user customizes the device configurations according to their comfort. For instance, some users prefer to operate their mobile devices with a single hand and therefore often keep that setting enabled on the device. This presents a unique set of cases from those who were involved in the configuration testing of this device. Managing all such custom configurations is a challenge a team faces.

Solution: It is impossible to cater to all types of custom configurations for all types of devices. Configuration testing matrices are always loaded to their capacity due to high fragmentation. However, the team can retrieve all the common custom configurations by analyzing the device logs either from manufacturers or from application analytics. The most a team can do is to satisfy all these common configurations and cover the majority of their users. It is also a good idea to propagate these configurations to the developers so that the application can gather logs based on custom configurations and the application supports a majority of them.

Problem 3: Too much to test

Explanation: The number of devices when permuted with the number of configurations generates a large number of configuration matrix units. Given that most companies now follow the agile methodology and there is less time to wrap things up, these many variables become a challenge for the team as none of them can be ignored or avoided.

Solution: Automation testing is a good solution to adopt when the matrix is becoming too large to handle. It can be implemented in the earlier stages and can handle most of the team’s load with great returns on investment in the future.

In addition to these challenges, the team may face certain project-specific or team-specific challenges as well such as not being able to manage scalability, dependency, or coverage numbers easily. Such problems may need specific solutions, however, most of them can be eliminated by opting for a cloud-based test automation tool that provides real devices such as Testsigma.

How to Automate Configuration Testing with Testsigma?

Testsigma is a cloud-based test automation software that fulfills the testing needs of the team with a lot of artificial intelligence that acts as a facilitator. Configuration testing is a complex job with a lot of variables involved. Therefore, a tool with automatic test case generation, English-based automation scripting, and an infrastructure with already set-up real devices saves time and helps minimize the risks of failures that would pop up on physical premises.

Testsigma can be used specifically for configuration testing for:

• Creating test cases tailored to practical configurations. 
• Running test cases on different machines with different specifications. 
• Tracking the outcomes of test cases across the set configurations. 
• Generating thorough reports displaying the results of configuration testing.

Testers can also use Testsigma for automated configuration testing where scripts can be written in English and converted to actions using natural language processing. Once the test plan is developed and test scenarios are crafted after discussion, these test cases can be created just by creating a new Project after free sign-up:

Once done, create test cases in English to construct test cases:

Select real devices to execute these tests on:

That’s it. These few steps are enough to conduct configuration testing on real devices without any programming language bringing down test creation, execution, and maintenance time significantly.

Automate your configuration tests for web, mobile, desktop applications and APIs, 10x faster, with Testsigma.

Summary

Today’s market is flooded with multiple devices that come with specific configurations that make them unique to each other. While it is exciting for an end-user to have so many options to choose from, it can easily become a nightmare for the testing team. Each new device release hints towards an extended configuration matrix signifying the importance and priority of configuration testing in the current market.

Configuration testing ensures that the application works smoothly and consistently across all domains and configurations. For this, it requires tools, techniques, knowledge, and solutions to challenges – all of which were the key sections of this post. With this, in conclusion, we hope that this post glorifies configuration testing and provides a deep knowledge of its importance in the testing cycle. This will help enhance the quality of the application and minimize post-production bugs and maintenance time as well.

Frequently Asked Questions

What would happen if configuration Testing is not done? 

Bugs, errors, and defects may occur. You will face some technical issues like:

• The system may not be able to run on a particular hardware platform.
• The system may not be able to connect to a particular database.
• The system may not be able to handle a particular load.
• The system may be vulnerable to attack.