Testing is crucial. Once you’ve built your mobile app, i.e., everything is ready from the development end of things. A sigh of relief as all your hard work now comes to fruition. Push out into the world and the world throws back bugs and user complaint reports. How could this have been avoided? Surely it could have been avoided?
That’s when you run to your QA and testers only to find the most understaffed and underfunded team under pressure when your disappointment adds to the user complaints.
After a few days of being live on every app store, It hits you. There are major issues and the people have been less than kind.
What went wrong? What did we miss in our endless meetings? These are often the issues that product management, QA, and CTO’s have to deal with.
QA needs to be scaled and at the heart of your development efforts right from the start. Once you do this, there is only one question to be solved. How do you conduct mobile testing at scale?
End-to-End Testing on real iOS, Android Devices & Browsers
End-to-End Testing on real iOS, Android Devices & Browsers
Testing on a quagmire of fragmented devices
The one thing that has caused the most difficulties for those of us in QA, is the fragmented nature of the devices, operating systems, and ecosystems of mobile devices. Testing on different devices becomes an absolute mess. Variations across different models, operating system updates, and other download applications all influence the behavior of a device. While much of this talk is often thrown at Android, Apple and Windows devices also vary enough to have their own fragmentation issues.
The only way to get around this, unfortunately, is by having a system available to you which can accommodate your testing needs based on your audience. Many companies opt to test on the top 10-15 devices for instance and the last few years of operating system versions. But, even this small amount of testing can result in hundreds of variations which means your QA team gets overwhelmed. Some companies opt for the use of virtual devices at this stage to attempt to address the sheer volume required. Unfortunately, manpower and bandwidth are still required to set up the testing, devices, and options.
Additionally, virtualized testing does not address contextual factors found when real people use a device in the wild. So, unless you are testing an application that has been designed for use under laboratory conditions, you’ll end up missing out on the exact kind of bugs that got you into that mess, to begin with.
How to keep scaling under faster delivery times
Scaling the efforts of testing becomes a bigger problem as you move towards faster delivery times and mechanisms. Most software development firms have embraced the move to faster development initiatives moving them towards a state of continuous delivery. Nowhere else is this more clear than in the mobile space. With companies such as Amazon releasing hourly, Facebook multiple times a day, and Google running hot releases, you need to ensure that testing the mobile apps that you develop does not become the bottleneck.
You can do this by ensuring that the entire QA process is a feedback loop that delivers in sync and in tandem with your development processes. Make sure that you have enough resources dedicated to your team or that you use a crowdsourced testing platform to fill in the gaps. The more eyes and ears you can get on your mobile app during the testing and development phase, the quicker you can resolve issues and prevent them from turning into app store review nightmares.
Dealing with never ending bugs
Of course, none of the advice already provided will mean much if a bug pops up when the app is released globally. The problem is, you cannot account for the extra issues that pop up in a global app launch.
If your plan is to release globally, you need some kind of mobile app usability testing where you release. The most important initial phase of testing will be functional exploratory testing to allow testers to explore the app and find bugs through creative means. Later on down the path, you can switch the focus a little into creating and executing test cases. Whatever process you elect to use, the only way you are going to get a successful global app is via testing with the actual people in the markets you hope to capture.
None of the advice and suggestions provided here is going to provide a bulletproof mobile app release without any bugs. What the advice and suggestions will do, is at least lower the acceptable level of risk for you and your organization when you release or update your apps.
In fact, while the suggestions presented here focus on the initial release of your mobile app, the testing can and should be part of your ongoing release plans and TestGrid can help you with your mobile app testing. Every time you build a new feature, launch into a new market, or develop for a different OS, exploratory testing via professional testing toolkits like TestGrid should be the bedrock of your plan.
CTO at TestGrid
