How to gain confidence while testing an app whose DAU is nearly ~20 million?
A few years back, It was a tough decision for me while making an app release whose daily active users are in millions. All our test cases were passing and we were done with multiple regressions runs. Still, this question was arising in our mind before every release night, Is everything will work for all 20 mn users or not?
During 2013, I was taking care of the Nimbuzz messenger app for Android and iOS Platforms, Just like Skype, Nimbuzz messenger allows you to make calls to mobile phones. In this article, I am going to cover all those aspects and gaps which QAs generally missed before releasing with an app whose current user base is in millions.
Metrics which need to be gathered before any release
Whether its iOS or Android platform, we need to gather all below metrics in order to make any customer-centric decisions
- All the compatible test devices where DAU(Daily Active Users) falls are covered during testing or not? For example, your app has ~5K daily active users in Google Phone or iPhone 7, these devices must be included in app testing activity
- All the OSes where the most number of active users are upgraded is tested or not? This data can be easily configured while login to the app play store console.
- Monitor all the reported crash and ANRs which is reported and try to find the root cause before any release, For android app check for android vitals and for iOS check for the reported crashes
- Target your audience according to their last firebase activities and clicks, If you are trying to release a messenger app please have a look to how many daily active users are visited and clicked chat screen and figure out the useful data for testing, Basically core functionalities of an app
Always have handy useful test cases which others are not doing
- Try test cases where you have to make intentional app crashes and check whether it’s getting reported to monitoring tool like fabric/firebase or not, It will help you to get the app stability reports after releases
- Have a few cases related to security stuff as well to verify the sensitive information in logs like auth token, certs, domains, other sensitive information
- Have a few automated test cases using some useful tool like testprojects in order to validate upgradations from older APK/IPA files
At last, after having this data decide whether you are getting a good quality score or not. If yes, please go ahead with the releases and monitor the reviews and crashes while making incremental rollouts.
Please leave your questions/comments/feedback below.
Happy Learning 🙂
Connect to me on LinkedIn (Click here)
On a mission to contribute to the Software Testing Community in all possible ways.
