I was an avid TestFlight user since 2011. TestFlight made it easy to distribute beta builds to testers. It helped make provisioning with device ids relatively painless, especially with its ability to re-provision a beta with new devices. It was still a bit of a pain though, when users got new devices, there was a bit of a dance that had to be performed, getting the device id, adding that in developer.apple.com, then adding the new device to a provisioning profile, and regenerating the profile. And of course there was the yearly 100 device limitation with no ability to remove old devices during the year.
When TestFlight was purchased by Apple and stopped accepting new apps, I switched to Crashlytics Beta (I was already using Crashlytics for crash tracking). It’s been a good replacement for TestFlight.
With the re-release of TestFlight from Apple, I had some hope of simplifying the process, but as always with developer tools from Apple, there are real tradeoffs in what you get. So here is a list of pros and cons with Apple’s TestFlight that I’ve found so far:
– No more worrying about devices! This is the biggest advantage of TestFlight. When you add a user (either an Internal or External tester), you simply add the email address of their Apple ID, and they will be able to install beta builds on their devices (up to 10 per tester).
– It is nice to be able to do testing with Internal testers first (a new submitted build shows up automatically for internal testers), then send out the build to external testers.
– Beta builds appear with an orange dot (and will automatically expire after 30 days), so it’s easy to tell that it’s a beta
– With the addition of Groups, it’s easier to manage external testers across multiple products (though this could still be improved)
– Once you’ve submitted an app for initial review for external testing, subsequent betas can be readied for external review immediately after entering build information.
– Users can only install the latest beta build (they can’t install old builds from the history of builds). This is another big problem if you end up sending out a beta that causes bigger problems than it fixes (not uncommon during betas).
– You can only send out 2 beta builds per day for external beta testing! This is a big limitation when you’re trying to send out quick changes to fix an odd bug that can’t be reproduced in-house
– You must submit the app for review for external beta testing. For the initial submission, it took a few days to get it approved.
– After you submit the build, you then need to go to itunesconnect and submit it for external beta testing, filling in the ‘what to test’, and answering a few questions about changes to the build. I found the process a little cumbersome – would be nice if that info could be filled in during submission of the build (would be nice to have a separate Submit Beta button in XCode)
– Still waiting on any kind of good crash reporting (I won’t be switching from Crashlytics anytime soon)
TestFlight is definitely a major step forward for beta testing of apps for iOS, but there are some difficult tradeoffs right now. I’ll probably stick with it so I don’t have to worry about managing devices and provisioning, but I’m certainly hoping for continued improvement.