iPhone OS Testing Best Practices

Question

Not so long ago iPhone development was quite simple, only a few OS versions and even less devices.

Now however, there are 2 major OS versions and 5 different devices to consider.

As a company about to release several applications testing has been become more and more of an issue.

What are the best ways to test all combinations, do I need to acquire every generation of iPhone and iPod Touch? Are there any gotchas with specific hardware/OS combinations I might need to code around?

I guess my question is, "What's the minimum amount of testing required to cover all the bases?"

Answer 1

I don't have much experience with the iPhone itself, but in general this might be a good application of pairwise testing. In practice, you can get 90% coverage with a small fraction of the testing of an exhaustive test pass. Then later if you find certain configurations have gotchas you can add them to your set of configurations and still not need to do an exhaustive pass.

Answer 2

I am not sure if its a best practice. But I have heard many people using ibetatest for exactly what you are looking for. Lots of enthusiastic beta testers out there. You shouldn't have any issues on 95% of your code running on different versions, and something like ibetatest should catch the remaining 5%.

Answer 3

Get all the combinations is the short answer.

I started out with just the iPod, but it's 100MHz faster than the previous iPhone model. My app's performance sensitive, so I'm gonna have to get an old iPhone. I can hardly ask my beta testers (volunteers?) to run the GL performance tool.

Answer 4

You can consider outsourcing to a company like ThePortingLab.com that basically specializes in porting and testing onto mobile devices. They have their own internal Test Plan cycles but you can get specific test cases added on to it. Also, have a look at their learn iphone programming online courses to see if there is any interest or help that you can get from them.

Answer 5

In my experience, you won't have much compatibility trouble between iPod/iPhone. There are other gotchas to be aware of:

• The devices run at different speeds. iPhone v1 and iPhone 3G run at 412MHz; iPod Touch runs at 532MHz, and the new 3GS runs at 600MHz. This can have a big impact on performance and even functionality if you're getting fancy.
• There's a huge performance difference between EDGE/3G/WiFi networks; often the differences are counter-intuitive. EDGE can often have better latency (time to first byte) than 3G, while 3G has 10x better bandwidth. You'll want to test your app under all three conditions.
• Are you using Core Location? iPhone v1 and iPod Touch do not have GPS.
• Are you using the camera? The iPod Touch does not have a camera.
• Is your app compatible with jailbroken phones? A lot of people have done it, and if your app crashes on them, they will blame you, not the Dev Team hackers, and this will be reflected in your App Store ratings. Note especially that background apps can use up memory that you might have thought would be available exclusively for your app. Leave yourself some overhead.

So, what do you need to buy? If you're a serious developer, yes, you should have all 5 devices available. But do you need to test every build on all 5 devices? Does every one of your developers need 5 devices each? No.

One developer can probably test everything that matters with one iPhone 3G and an iPod Touch. Toss in a 3GS and your coverage is probably nearly perfect. (Note that development on 3GS is much nicer just because the CPU is faster, so your apps deploy more quickly.)

At work we have one device per developer, but they're a mix of 3G and iPod devices (and, today, one 3GS).

Answer 6

I'd suggest a couple things:

• you probably will need a range of devices for your own internal testing. As has already been pointed out, there are various differences between each generation of iPod Touch/iPhone

• you may want to look into using crowd-sourcing to supplement your internal testing. This potentially allows you to have a bigger audience to test your apps and any cost of using crowd-sourcing is potentially offset by the fact that you can spend less on devices and potentially less on internal testing staff and at the same time, potentially producing a better product. One crowd-sourced test group not mentioned above is uTest.com