> 3) How do you determine which devices and OS versions we should test on?
> This should generally be an easy question for the candidate. Good candidates will point to app analytics as the best measure, looking for the most used devices for their particular app. Another good answer would be to check the app reviews where people might have complained about specific issues happening on their devices. Less creative candidates might simply suggest “looking at the top devices” on the market
Looking at top devices is also incredibly important, and it seems foolish to dismiss that strategy.
What if your website doesn't work on iPhones, and so you have no iPhone users? What if, because people can't even sign up on an iPhone, they don't bother complaining about specific features not working?
Surely it makes more sense to test based on your target market, rather than the subset of people who are self-selecting to use your product?
Ideally you should be looking at both usage and market data. If your target market contains lots of iPhone users, but everyone's on Android, this points to a potential problem.
As an Android developer, if hiring for an Android QA position I'd be thrilled to interview a QA Manager or QA Person who was specifically emphatic about testing on a wide array of Samsung devices.
Not because they are among the "top devices" (though they usually are for each subcategory), but because it shows they have enough experience to have seen first-hand that a surprising amount of device and/or OS version specific issues that Android apps run into occur on Samsung/TouchWiz devices.
The purpose of QA is to serve the existing user-base base first, therefore it makes sense to find what that user-base is first; you're missing the point.
> 3) How do you determine which devices and OS versions we should test on?
> This should generally be an easy question for the candidate. Good candidates will point to app analytics as the best measure, looking for the most used devices for their particular app. Another good answer would be to check the app reviews where people might have complained about specific issues happening on their devices. Less creative candidates might simply suggest “looking at the top devices” on the market
Looking at top devices is also incredibly important, and it seems foolish to dismiss that strategy.
What if your website doesn't work on iPhones, and so you have no iPhone users? What if, because people can't even sign up on an iPhone, they don't bother complaining about specific features not working?
Surely it makes more sense to test based on your target market, rather than the subset of people who are self-selecting to use your product?
Ideally you should be looking at both usage and market data. If your target market contains lots of iPhone users, but everyone's on Android, this points to a potential problem.