iOS App Testing Tutorial: Manual & Automation
In the 2018 WWDC talk, Tim Cook said, “We’re also happy to announce that this week we’re going to achieve another huge milestone. The money that developers have earned through the App Store will top $100 billion.” He further announced, “There are now 20 million registered developers on the App Store.” Although, this number might be way out of the ballpark considering that 2016 statistics showed just 2.8 million iOS application developers.
Be it as it may, iOS development has risen over the years, evident by the growth in the applications hosted on the App Store. As of March 2022, the Apple store hosts around 3.7 million non-gaming applications, which has consistently increased.
But why are we talking about statistics of Apple store when the post is about testing the iOS application? This is because the more applications are hosted on the App Store, the more competition you will face. In this crowded market, users have always got a lot of options to choose from unless you have a unique idea that none of the 3.7 million apps have (which is quite rare). So, what would make us prove our mettle in the endless choice – a high-quality and well-tested application? In this iOS testing guide, we will discuss iOS app testing and why it needs a lot of focus before pushing the code to the users. What methods we could choose and what areas to focus on would also be an important part of our discussion.
Table Of Contents
- 1 What do we mean by iOS app testing?
- 2 Why is iOS app testing important?
- 3 What to cover in iOS app testing?
- 4 Strategizing the iOS app testing phase
- 5 Types of iOS app testing
- 6 Testing methods in the iOS application?
- 7 Devices and their impact on iOS app testing
- 8 iOS app testing best practices
- 9 Conclusion
What do we mean by iOS app testing?
When we release an application to the end-user, we give them the best version. We ensure this by performing iOS app testing.
With iOS app testing, we test our iOS application on mobile devices (emulators or real devices, depending on the use case). Here, we pass it through various testing phases to ensure that the final version has minimum or no bugs. These can include manually inspecting the application like an end-user or running an automation framework like Appium or Testsigma.
Finalizing our release version helps ensure the end-user experiences no bugs, and we see an increase in the installation numbers.
Why is iOS app testing important?
The next thing to make sense of is why we are doing this and its importance in the real world. Since there could be hundreds of reasons to analyze here, we can filter them up to the two most important:
Removing bugs seeping into production
One of the main motives of iOS app testing is to stop the bugs from seeping into production. When we need to correct a bug from the production, it may cost us four or five times more than what it would have cost in the testing phase. This number can extend up to 100 times in the worst situations.
Such heavy expenses might shake our budget and shrink our ROI in a limited time frame.
Sustain the user growth/number
In this highly competitive world, nobody wants to use a buggy application. In a 2018 study, it was found that 62% of the users uninstall the application if they face crashes and errors. Also, 94% of users uninstall the application within 30 days of installation. These high numbers show how difficult it is to maintain the user base and keep our application going in the market. In such a scenario, we cannot afford errors; therefore, iOS app testing becomes much more critical.
Apart from these two reasons, there can be numerous other advantages such as preventing crashes, ensuring what we made is what is delivered, etc. However, the motive of this section was to enlighten you about the consequences of releasing a lousy iOS mobile application.
What to cover in iOS app testing?
iOS app testing, similar to other OS’ testing, spans a variety of diverse areas that a tester needs to consider. A few of them are as follows:
- Installation parameters: First, how are the installation parameters working out in your application? Questions such as: Are the logs pointing to correct data? Is the time taken to install the application on a real device near the expected time? – are good to understand at this stage.
- Native functionality: A significant difference between a web app and a mobile application is that the latter can use native mobile device features. Since a user expects native features to work, such as gyrosensor or multi-touch, your application should be thoroughly tested for this.
- CDNs and third-party services: Another critical area to consider in iOS app testing is the accuracy of CDNs and third-party services. For example, if your iOS app uses maps from a third party, the location requested and shown should be the same.
- Notifications: A lot of the time, I have faced issues related to notifications that directly hint at improper testing. For example, this:
Notification plays a vital role as the user is prompted to open the application through it. Its testing is, therefore, of high importance.
- Measure performance metrics: Performance metrics help you understand how your iOS app performs on various parameters. For instance, how the network part is, how are the function calls doing, etc. This will help you optimize your application in a positive direction.
- UI/UX: Needless to say, the application’s user interface and user experience must be top-notch. It will help sustain the user and popularize by word of mouth.
- Battery-related issues: With time, mobile devices have become much more powerful than they were five or seven years ago. The RAM has increased, and the devices can withstand high-end applications (majorly gaming apps) that drain the battery. Therefore, users are now more concerned about their battery life than earlier. As a result, if your iOS app is draining out the battery, there is a high chance that the user will uninstall the app. In this area, we look at the battery drain trend and check whether there are any abnormalities.
These are some of the major areas to cover. However, these areas contain many sub-areas under them that need proper detailing at the time of test execution. Testers are required to blueprint them carefully.
Strategizing the iOS app testing phase
The next thing to cover in our iOS testing guide is strategizing the testing phase and approaching it in a phased manner. Since there are multiple types of iOS app testing (about which we will talk later), this section is kept generic.
Understand the product
A good understanding of the product is required from each person related to the application in any way. It does not matter whether you are a tester, a sales guy, or a delivery manager; each person in the team should know the product inside out. However, the depth of understanding may differ depending on the designation. As a sales guy, you don’t need to know the in-depth technical working, but as a tester, you do.
Understanding the product starts by contacting various teams and people who are associated with it. A developer will tell you the technical aspects of the product; a business analyst will tell you the business aspects, and so on. However, all this has to be done only once when either the app is new, or you are a new member. This step requires patience and a considerable amount of time. Finishing this in a hurry can have an adverse effect on other stages.
Outline business goals
The next step that, surprisingly, I rarely find in the companies or people talking about is the tester understanding the business goals. If a tester does not understand this, his approach toward testing will be just “achieving the functionality accuracy” and not “achieving what is intended from the product.”
For example, referring to my personal experience while filling out a travel insurance form on TATA AIG made me realize the difference in what I mentioned above. While the visa requirements were that the name should be the same in the travel insurance as in the passport, TATA AIG had only two fields – First Name and Last Name. In such a scenario, customers generally put their middle name along with the first name in the same field separated by a space. However, the TATA AIG form did not allow spaces in the name field.
So, here the technical requirements were clear. The name (first or last) should have only one word. This was taken care of. However, the business requirement (the middle name for visa purposes) was unclear. As a tester, if you are focused on just technical requirements, you’d pass this form as they did. However, if you are a tester who understands the business goals and where this form will be used, this bug can be pointed out in seconds, saving the customers’ time.
Now that you understand the business goals and in which direction to move forward, the next thing to look at is test-related objectives. The test-related objectives in iOS app testing will focus on the types of tests that will be prioritized based on the type of application. For instance, for an application booking a train ticket, the UI is not as important as it is for a gaming application. If your gaming application is not on a good engine (which would be visible), you might not be able to grow regularly.
Finalize tools and technologies
The next step is to finalize the tools and technologies used in the iOS app testing process. Since the world is filled with tools that can do a fantastic job in the app testing domain, the decisions certainly boil down to two things – your requirements and your team’s expertise.
So, let’s consider your team has expertise in Appium, but you, as a team leader, decide to go with XCTest instead. No doubt whatever you aim towards can be achieved with XCTest efficiently, but since the team is unaware, you spend a lot of money on training. In addition, less expertise would mean more mistakes and a little slower pace of work. All this will cost you money and time. Therefore, choosing a tool (or tech) based on requirement and the team’s expertise bear fruitful results.
But what would you do if your team had not unanimously voted on a single framework to work on? This scenario could be tricky, as you end up spending a few extra bucks any way you go. The best path to choose here is codeless testing with an online platform.
Codeless iOS app testing with Testsigma
Codeless testing is a futuristic way to test your iOS applications, and platforms like Testsigma help you achieve that free of cost. This type of testing means you do not need expertise in programming but only an eye for observing bugs. If your team cannot settle on one framework, Testsigma is your go-to tool.
To perform iOS app testing on Testsigma, create a free account by signing up.
Next, go to the mobile testing section and upload your binary file.
Choose the cloud platform where you wish to run the mobile application.
Record this session or just type test actions in plain simple English.
Testsigma uses natural language processing to understand your English and perform the test on the application. Do give it a try before moving ahead.
Distribute work to resource
The final stage before the actual testing takes place is strategizing the distribution of work to resources. This requires good observational skills as to which team member does which part best in the testing phase. Hand over specific tasks to people according to their expertise for a fast wrap-up of app testing.
Document and Reports
The final phase in strategizing is documenting everything and creating a finalized report. Documenting everything is considered good practice since you will undoubtedly forget a few things you did before the next release. Also, new team members often require some written document to learn things faster. It also becomes a go-to element when they quickly need to re-visit something in the testing.
The most common mistake that testers make here is documenting only what they think is important and leaving the rest. This can harm the process in the future. There is no “important” or “not important” thing in testing. What might be irrelevant to you could be of some (or high) importance to others. The more you document, the more flaws you see, and the more you optimize. Ultimately, the testing phase becomes stronger, more efficient, and smoother.
Types of iOS app testing
Similar to other application testing, iOS app testing can also be divided into two types:
- Manual testing
- Automated testing
Manual testing for iOS applications
Manual testing is the process of scrutinizing the application manually. Contrary to manual testing of the 90s, a couple of misconceptions do float in the IT industry:
- Manual testing does not use any software.
- Manual testing should be de-prioritized against automation.
These misconceptions degrade the strength of manual testing and its importance in delivering a quality application. Let’s address each of them individually here.
Manual testing does not mean that the tester interacts with the application, keeps a note of everything manually, and repeats the same process. If you are testing manually, you are interacting with the application step by step and not using robotic instructions. But, a manual tester uses tools to identify bugs and achieve things faster. For instance, JMeter is a tool that can be used for load testing, and the manual tester can identify the load the application can take.
Secondly, de-prioritizing manual testing can have fatal effects on the final release. No matter how much you try, you can never imitate a real user through automation, at least in 2022. Even if you are releasing a simple iOS application with basic functionalities, you can still find areas where manual testing can point out a genuine bug.
While automation can work in the specific areas defined through scripts, a human can point out subtle bugs that automation cannot. Also, since you are releasing the app to a human and not a machine, it needs to be dealt with by a human first.
Automated testing for iOS applications
The next part of our iOS testing guide is the automated testing of iOS application. Automated testing seems the best choice for tasks and scenarios requiring repeated actions from the testers.
Such actions lead to errors and significantly more time than a computer would take. For example, data-driven testing is one example where repeatedly, much data is fed into the same field. This is done to make our fields shielded from all the irrelevant data.
Automated testing also saves time and helps deliver the product faster to the next phase. This can extend up to 75% of your time! This is because since a computer is at work, it need not sleep or take tea breaks. You can submit a job for regression testing and call it a day. The system will run the job on the server, and the results will be ready when you log in the next day.
The only demerit of automated testing (apart from the mentioned in the manual testing section) is that it does not give returns so early into its implementation. The time frame varies on various parameters, such as tool used, project size, what you are automating, etc. A simple time-cost graph in this regard may look as follows:
We have published a complete post on understanding the return on investment in automation testing. You can read that out for better analysis.
Testing methods in the iOS application?
Alright! Since we are now aware of the high-level concepts of iOS app testing, let’s dig one step deeper and see what kind of paradigms we use in manual and automated iOS app testing.
The first thing that the user will see is the UI of the application, and therefore it deserves to be discussed before other types of testing.
UI testing is user interface testing. It defines and verifies how the application appears visually and behaves when a user interacts. For instance, are all the elements within the device boundary? Or if I click the empty field, does the keyboard pop up, and the field moves above? All such scenarios come under UI testing.
UI testing is significant in 2022 for your iOS app. If your iOS app has a good UI, it can raise conversion rates to 200%. This could be profitable for the business and application’s future and, therefore, for you. While there can be a long list of UI scenarios that are relevant in UI testing, it is to be kept in mind that both manual and automation should be involved in UI testing. Some excellent tools available on the internet can help you in UI testing, but in the last step, to make sure, hand over your app to a manual tester. Let him analyze the UI and the elements associated with it. It doesn’t take too much time but can ensure a better application.
UI testing tools
The following tools can be used to test the UI of your iOS applications:
After the UI is cleared and you are ensured, the next step is rigorously performing functional testing. In UI testing, a part of functional testing is performed as UI displays the functionality outputs. However, it is limited, so it is better to perform UI testing first and then functional testing.
Functional testing in iOS app testing covers the functionalities of the mobile application and whether they are working correctly in all scenarios. For example, a notification system is one thing that needs rigorous testing. It can increase the count of application openings which, of course, depends on the day of the week:
The rest of the functionalities should also be carefully tested so there are no reports from the end-user.
Functional testing in iOS apps is better if done through automation. There are a lot of smart tools available that can match the expected performance and behavior of the application with the actual one. Manual functional testing can be time-consuming, and there are no such significant differences as we saw in the UI testing section.
Functional testing tools
The following tools can be used for functional testing of the iOS apps:
In a research paper titled Defect Detection Efficiency: Test Case Based vs. Exploratory Testing, authors Juha Itkonen et al. explore the importance of exploratory testing and find some fantastic numbers. On average, as the paper defines, exploratory testing finds 29% more bugs when it comes to noticeable bugs (basic or simple) and 33% more when it comes to complex bugs. This is a huge number as 33% more bugs could be devastating for the future of our application. A number like this cannot be ignored.
Exploratory testing, as its name gives an idea, is a testing method with no set of rules, methods, or protocols. You explore the application and find out if anything is wrong with it. Therefore, a tester can set their own rules and processes on how to approach this testing. It is also apparent here that automated testing cannot be used for exploratory testing. Since you scrutinize and operate the app like a human, a machine cannot bring about the same essence to it.
Our iOS testing guide cannot be completed without focusing on the importance of security and the vulnerability that surrounds the cyber world. In the March of 2021, Wired reported research by Zimperium that 6608 iOS applications were leaking private and sensitive data such as passwords. These applications were found using the public cloud providers such as Amazon’s AWS.
Your application can easily be one in this list of 6608 iOS applications. The major defect was found in the setup configurations and environment, which anybody could do. We cannot discover it during configuration setup, but we can always prevent it by performing security testing.
The aim of security testing is that your application remains secure in all aspects. It should store the user’s personal data securely and transfer it to and fro with security mechanisms such as encrypting it at the source. With security testing in place, we try to pinpoint the bugs that can leak data and report it to the security team. This type of testing is non-negotiable and should be performed manually and through automation.
Security testing tools
The following security testing tools can be used for testing iOS mobile applications:
A/B testing (optional)
A/B testing of an application is creating two slightly different versions of the app with minor modifications and analyzing the end-users response through their actions. The version that gives you a better response is the one you should finalize and roll out for everyone.
This type of testing is majorly done in the web application because of the wider area it serves. For instance, Unveil had their first version of the page as follows:
The last line reads, “real-time feedback & approvals from just $1 per month.” This line was changed as follows in the second version:
The second version resulted in 31% more signups than usual! So, they decided to keep the second page. Similar things can also be done on your iOS application by tweaking little things and checking the conversion rates. However, since mobile apps require installation first (which itself is a heavy task for the user), this testing certainly becomes less prioritized. However, it’s best to know!
Beta testing (optional)
As famous as A/B testing is in the web development world, the same place goes for beta testing in mobile apps (which is not so popular in web dev). Beta testing is where you select a few users and release your iOS mobile application to them. The number of users depends on the scale of the application, the project, and your budget. Some app owners beta test with just 10 users, some do with 50, and some you might find with 200. The number might also go to thousands as done by PUB-G, but that is riskier and cost-heavy.
In this iOS testing guide, we are keeping beta testing as an optional choice because of certain limitations that come along with it. Firstly, beta testers are not always serious in their job. You cannot rely 100% on their feedback. However, if monetary or similar benefits are involved, the feedback rate might jump to 100%, but it doesn’t have to be accurate. Some people might provide feedback in a hurry just for the rewards. Therefore, you might have to find a pattern and settle on the popular choices. This can require a larger pool and cost you more.
Secondly, if you have finished your application and still not releasing it to the user (because of beta testing schedules), you are losing money. This can increase your budget and extend the deadlines. However, beta testing can point out bugs that would have cost you even more if your application had been released to the user.
An alternative to beta testing is usability testing. In usability testing, we hire people and ask them to interact with the application. But in this case, generally, we connect with them while they are in the process. This way, we even talk to them to know their thoughts and why they selected the button or option. It gives us a more detailed idea and better accuracy. But in the end, it all comes down to the team’s choice of which way they wish to proceed.
Devices and their impact on iOS app testing
With time, Apple’s devices have grown substantially in the market.
With so many devices, it is not easier to test applications by uninstalling and reinstalling many times a day. Also, not all companies can purchase these mobile devices due to cost issues (considering an iPhone currently costs at least $990 and you need all types of models in multiple numbers). On top of it, high maintenance costs are also involved, which may impact the budget. Therefore, using devices that are not as costly as real devices but are effective in testing is necessary for iOS app testing. Here, we come up with concepts of emulators and simulators that have advantages and disadvantages.
A simulator simulates an apple device on another device (like a desktop) without purchasing a physical one. A simulator is well understood by game simulators as people often run mobile simulators on their desktops to run mobile games. However, in this case, we run a mobile simulator which means it opens up a mobile device on the desktop, and you can use it as a real device.
The main advantage of using a simulator is that it is free, and you can use a single system to run as many simulators as you want. So you don’t have to spend anything and can own every iOS device.
However, its downsides often dominate its benefits. The most prevalent of it is that simulator gives you a perception of using an iOS device, but in the end, you are using a desktop for it. So, the hardware on which your application is running (desktops) is exceptionally high, and you may get dubious results. Secondly, simulators cannot make you analyze some extremely important things in iOS app testing. These include battery drainage, calling features, cameras, practical interrupts, etc.
The next type of device that gives you a better advantage than simulators are the emulator device. An emulator and a simulator are almost similar in their presentation or method of working on them. The difference among them is that emulators behave more like a real device rather than just providing a simulation. They do this by considering two extra things:
- Providing hardware support similar to a mobile device.
- Providing real device-like features such as call interrupts.
Both of these features bring you closer to the real device, providing a better picture of how your application would work when released to the end-user.
But, it is not that it doesn’t come with a downside. Emulators may be free to use and much better than simulators, but you cannot rely 100% on them. For instance, can you rely on the emulator’s battery drain graph when it does not even have a battery? In addition, even if emulators have developed significantly in the past year, many things are still missing from its pool. For example, you cannot imitate a network bandwidth variation as it would be in the real world.
With that being said, if cost is the issue in your organization or if you are working alone, emulators seem to be the best choice.
The ideal thing to use while testing the mobile app is a real device. A real device is the actual manufactured device by Apple that you need to purchase like an end-user. With a real device in hand, you don’t need to worry about anything, as whatever you see are real parameters. You can also test your iOS app while on the go to mimic practical network situations.
The only demerit with using real devices is the cost associated with it. The more devices you procure, the more expenses you bear. Not only that, but since iOS devices are released frequently, you must keep purchasing them regularly to test your application.
Also, every user might not have the latest iOS version update installed. So, you might have to purchase more of the same device with different versions installed. All this adds up to your expenses which may not give you immediate returns. However, real devices give you the best analysis for your iOS app.
This brief comparison between real devices, emulators, and simulators makes it easier to point out that real devices are the best option. But practically, purchasing real devices is not a solution. So, in such scenarios, we prefer two options:
- Mix emulators with real devices.
- Try online platforms.
Let’s check these options out in brief for your iOS app testing.
Mixing emulators with real devices
When we say mixing emulators with real devices, we separate tasks that we will allot to emulators and real devices. If we think logically with the current standards of emulators, many things that emulators can do are highly accurate. For instance, the user interface you see on an emulator will be precisely the same on a real device, and there is no hardware involvement. The same goes for a few functionalities. This way, we can keep our budget in hand and test our iOS app smartly.
Trying online real-device platforms
The best method to opt for is to reach out to an online cloud platform that maintains the real devices for you. It’s like when you need a house to live in, you don’t have to buy the house but can rent it from someone who has bought it.
A lot of companies purchase and maintain real iOS devices for testing purposes. They have multiple devices with various specifications connected to their cloud infrastructure. All you need is to purchase a plan and start using their devices instantly. It would feel like an emulator, but an actual device is running behind the hood. So, you can be assured that the parameters are correct.
Online cloud platform for real iOS devices comes at a nominal cost. For instance, Testsigma provides a free plan for users with basic needs and access to real devices. Such platforms can substantially lower your expenses, and since you wrap things much faster, they lower the testers’ billable hours.
iOS app testing best practices
iOS or mobile app testing may not work wonders unless you are doing it efficiently. Otherwise, you may spend a lot of time on things that could have been done within a few minutes. In this iOS testing guide, we will discuss a few best practices that will help you along the way:
Use automation wherever possible
Automation testing is the preferable way when things need to wrap up faster. The main idea here is to keep automation prioritized but remember not to compromise. This means that automation may not be able to fulfill everything (100% of things) all the time. It may depend on the testing type and your end goals. So, use automation in the areas where you can. In other areas, prefer manual testing. A mix of both is the best path to travel.
Don’t wait till the end of development (in-sprint)
One of the common development practices currently adopted by the industry is to test while the application is in the development stage. Previously, testers used to wait for the completion of development. This way, the feedback loop becomes longer, and it takes more time to complete the application. The process of testing while in development is called in-sprint test automation.
In-sprint automation helps QA get involved with the sprint meetings and understand the application well from its core. The deep understanding they get can then be used by testing more minor corner things of the application. As a result, the iOS app quality is upgraded.
Data-driven testing can save you
Data-driven testing is the best way to ensure your data validations work perfectly on your input fields. In data-driven testing, we input a variety of data (sometimes numbering more than 500) and match the expected value with the actual value. For example, a simple “first name” field can have values such as:
- Harish – Actual Pass, Expected Pass
- 1Harish – Actual Pass, Expected Fail
- 123 – Actual Pass, Expected Fail
- Harish123 – Actual Fail, Expected Fail
In the above four cases, the second and third tests failed, which shows our input validations do not work correctly.
Since the data amount is enormous here, it is better to perform data-driven testing by automating your iOS application. You can refer to this detailed guide if you wish to learn more on this topic with demonstrations.
Use real devices as much as possible
The final best practice I would like to mention in this iOS testing guide is to make use of real devices as much as possible. Since we discussed the merits and demerits of the devices available to us, we can easily conclude here the role of an online cloud platform that provides real devices. In summary, the more real devices you use, the better your analysis will be, and so is the quality of your iOS app.
In this long iOS testing guide, we hopped on to some exciting topics and methods phase-wise. Everything we discussed will be used when you start iOS app testing in your project. To summarize, let’s start with the goals we achieved in this post.
First and foremost, application testing has a lot of importance because of the competition in the market. If you are developing an application, there is an extremely high chance that another iOS app already exists with similar business goals. So, a small bug can force the user to press that “uninstall” button, which nobody in the team would want.
We start with the areas we can cover in our iOS app testing journeys, such as battery drainage and performance metrics. But all these areas cannot be covered within time and efficiently if we do not strategize our iOS app testing. So, we always jump onto the points that need to be tightened up when we are testing. Once we are done with it, we decide the type of testing we want. Should we go manual or automated? While 100% of both is risky, practically, we try to maintain a mix of both with areas defined in that particular section.
Setting up our manual and automation infrastructure would now require us to divide types of testing into these two sections. Once we are all finished with planning and setup, we choose the type of device to work on. In my experience, online platforms seem to be the best for iOS app testing with quick ROIs.
Following this detailed iOS guide with the best practices, I hope you can initiate, plan and execute your iOS app testing better than before. For any suggestions and queries, let us know in the comment section. Thank you for giving this post your valuable time.