TestFlight keeps getting better

Just found out today that TestFlight improved their web site this week. They split the web site functionality in four sections: Dashboard, Builds, People and Support. I love the new organization: it is easier to grasp and work with.

TestFlightApp is getting better and cooler!

A while ago, I wrote a complete review of @TestFlightApp, a web service allowing iOS developers to help them distribute adhoc builds of their apps for testing purposes (TestFlightApp - a complete review). That's kind of incomplete description of this wonderfully designed web service.

Today, I want to highlight a few upcoming features on TestFlightApp:

• see who is really testing your apps, how much effort they put into;
• see application usage patterns with checkpoints;
• ask for feedback the users running your application, directly from the device;
• get easy access to application's crash logs, complete with symbolication.

All of this will come under a new SDK on which I spent some time to test.

Who is really testing your ahdoc builds?

Sometimes you want to go out and ask for others to test your iOS apps for a specific feature or just to get general feedbacks. So you ask a bunch of people to submit their device's UDID in order to create an adhoc build for them. But, out of 20 or 30 beta testers, how many are really doing what you expect them to do? Before TestFlightApp's new SDK, you would judge this by the amount of feedback you were getting via emails. Now, with the new SDK, you'll see exactly who is really testing your apps, thanks to the new Sessions tab.

As you can see, you'll see who is the most active of your beta tester, total spent in your app, how many sessions, average time spent per session, etc. This is really useful to help manage your team of beta testers.

See application usage patterns with checkpoints

Now you can embed checkpoints within your app. After adding the TestFlightApp SDK to your project, simply add calls like this at strategic workflow points within your application:

[TestFlight passCheckpoint:@"CHECKPOINT_NAME"];

This way, you'll be able to see usage patterns and eventually be able to identify user interaction problems. This is really cool. If you already use Flurry analytics library in your shipping apps, you'll see that this is really similar. What is cool here is the ability to see all your stuff in one place on the web while having complete control on the beta management process. Plus, TestFlightApp web site is way cooler than Flurry's. No doubt about this.

Checkpoints view

As you will see in the next section, checkpoint are critical to the next SDK's feature: asking for user feedback. 

Asking beta testers for direct feedback

The new SDK allows every developers to ask for direct feedback from within their application. This one is really cool. Let say you have designed a new welcome tutorial and you want the testers to give their impressions. To accomplish this, you have to have defined checkpoints in your application. Then, on the TestFlightApp management page, you assign a question to this checkpoint. The next time the user hit this specific checkpoint from within the app, the TestFlighApp SDK will bring up the question to the user. 

You can ask for feedback with Yes or No answers

Free form feedback can also be submitted

On the backend, TestFlightApp let the developer see all the answers in a nice dashboard view. 

Access to application's crash logs, easily
As developers, we all know how tricky it is to get user's crash logs. But what if the whole process was automated from start to finish? This is exactly what the SDK does for you. If your application crashes while being tester, open the next restart, the integrated TestFlightApp SDK will try to recover the logs and send them to the web service. Completely symbolicated.

This alone is worth the small time it takes to integrate the SDK into your app (less than five minutes). This is so cool that I asked the support team if they would allow developers to include the SDK into shipping applications on the App Store. Their answer was very revealing: they are away of other developers who do just that. I think I will try that with one of my app to see how it goes.

XCode Project Settings required for crash reporting

As a side note, the SDK is taking into account the network availability so if the network is missing, pending sessions, crash reports and the like are waiting for the next opportunity.

All in all, I really like how TestFlightApp is evolving. If you are not already registered with them, you should take the time to open an account (free!) and see for yourself.

Conclusions on my first open beta under TestFlightService

I've been using TestFlightApp service for 10 days now for an open beta of Ultimate Password Manager 3.0 that should come out in the next few days. Here are a few interesting facts.

• More than 40 peoples signed up for the open beta. 
• 12 builds have been seeded to beta testers. 
• I've received many feedbacks. Lot of them demonstrated very engaged and professional iOS developers. 
• The service is invaluable to help build better software.
• About a third of all testers actually installed the application on their registered device.

Following is a graph of beta tester involvements with my open beta.

As you can see, this is a small portion of testers who sends feedback. What can I say about this? This could be because the application is mature and nearly ready to go on the App Store and there is not too many issues to report. The amount of feedback is not really important, what is important is the quality of the issue reported. I was lucky to have a few testers with very interesting comments about the general design of the application and I took the feedback very seriously. The end result is an improved application to release to the App Store.

For a complete review read my blog post "TestFlightApp, a complete review".

TestFlightApp: a complete review

In this blog post, I want to review something new, something cool for all iOS developers. I want to review a new service that may change the iOS development landscape quite a bit because it is quite... magic.

If I could name what is the most frustrating and time consuming aspect of building iPhone apps after the actual development time would be the management of the beta phase. What are the issues regarding beta testing iOS apps right now?

1. finding beta testers to test your iOS app can be hard beyond iOS developer own network
2. managing the provisioning profiles to include beta testers UDID of their devices
3. keeping in touch with all beta testers and provide them new builds
4. managing beta testers feedback

Currently Apple's offering doesn't address any of these problems. To help developers overcome these shortcoming, a new service went live in open beta form and is called TestFlight available at http://www.testflightapp.com.

TestFlight's magic comes with beta distribution of your apps over the air. Here is how the service work. After registering yourself as a team leader and registering your own devices for testing, you build a network of beta testers by sending invites or by advertising a link to you application opened for testing. Registered users along with their registered devices get a chance to download you app, over the air simply by logging in to the web service from their device. No need to send them an .ipa file with the provisioning profile. iTunes is not needed. Something Apple should have done itself.

So you first create a team. You can create a team in order to work for different clients or for different jobs. A team is a group of people with one or many devices.  Inside a team, you can create distribution list. These will allow you to target a group of testers. You use them to send notifications of new uploaded builds.

When uploading a new build, you may choose to advertise it to a few testers, not all by using distribution lists that you create

Next, you make sure that all registered user devices are defined on the Provisioning portal at developer.apple.com. You download your new profile and install it on your Mac. You create a new build of your application with the "Build and Archive" command. Then, from Organizer, you share this build as a file (and make sure you sign it with the right provisioning profile just downloaded). Finally, you upload your build to the TestFlightApp web site. 

You upload the build created from Xcode's Organizer

Beta tester will receive an email asking them to install the new build like shown below.

Installing a beta build - from start to finish

What I love about TestFlightApp

• The web site design is simply gorgeous. Really. I'm really impressed.
• The service is simple to use because the web site is very well designed.
• The service helps me create a network of beta testers and build some awareness of my iOS app I'm currently working on. You can invite member of the press to test a nearly ready for market app very effectively.

What TestFlight cannot do

• You still need to add each device on the provisioning portal at developer.apple.com.
• You still have to generate a new provisioning profile before building a new Ad Hoc release.
• You still need to build and archive a new version of your application for distribution and sign it with the new provisioning profile.

A few more things to notes

• after uploading a new build, users must visit the web site from their device in order to view available build of every applications.
• each beta tester needs to register with TestFlightApp and register at least one device with the service. We cannot distribute app to anonymous beta tester.
• we can use an URL to share with peers to advertise a beta campaign. Those who follow this link will get registered as recruits. 

What is still missing or area open for improvements.

TestFlightApp is still in beta but is surprisingly well done, stable and complete. But, I see a lot of potential for this service like:

• having a place to discuss found bugs. Right now, the only way is by email directly to the developer;
• being able to remove failed uploads of apps;
• being able to know which teammate downloaded a specific build, we only have a general download counter;
• visiting the site from an iPad after being authenticated doesn't allow viewing the dashboard; you only get the install apps view;
• developer profile could be created with testing abilities, area of interest, how much effort that will be invested in beta testing (user interface, design, type of application);
• analytics that we can include in beta build that would interact with TestFlight services and provide automated feedback from the beta testers.

All in all, TestFlighApp maybe one of the most useful tool to come to the market in 2011 for iOS developers (well, maybe Xcode 4 will be more useful). So, if you are a developer, go and register yourself with the service... One more thing: it is free. How cool is that!

Beta testers needed for UPM 3.0 (through TestFlightApp)

Using the new and really cool TestFlight service, I want to try an open beta of the next release of Ultimate Password Manager. Version 3.0 comes with a lot of new features and updates. If you are an iOS developer, take this opportunity to test this service and my application too!

Enter the beta with TestFlight here: http://testflightapp.com/join/e44da85044a1f3083b14a6182e13f6a1-MTQxMg/

Password Safe in Ultimate Password Manager 3.0

Change log:

[NEW] : iOS 4.0 Fast application switching support.
[NEW] : Search in password safe.
[NEW] : Alphabetic index in password safe.
[NEW] : Local notifications for expired password entries.
[NEW] : Use "Pull To Refresh" to get a new set of generated passwords.
[NEW] : Filter password safe entries with the new action menu instead of the segmented bar.
[NEW] : Restore mode now accessible from the Dropbox sync view.
[NEW] : You can now set a password to never expire by tapping a button instead of using the slider.
[NEW] : Show the password for a safe entry without editing the entry.
[UPDATED] : Full support for Retina Display.
[UPDATED] : Application's icon.
[UPDATED] : Application's name on home screen.
[UPDATED] : Optimized network connectivity detection.
[UPDATED] : Passwords tab is now called Generate.
[UPDATED] : Rules are now set from within Generate.
[UPDATED] : Profiles are now called Policies.
[UPDATED] : Application's initialization routines.
[UPDATED] : Safe entry's description and note fields are now much bigger.
[UPDATED] : Application's settings logical grouping.
[UPDATED] : All application feedback messages are now dependant of the Show Live feedback in Settings.
[UPDATED] : Password Safe design and look.
[UPDATED] : Dropbox status now includes the account name.
[UPDATED] : The period for expiring passwords is now 7 days instead of 3.
[UPDATED] : Application icon.
[UPDATED] : Tabbar icon.
[UPDATED] : Icons in Settings, Password types and Categories
[UPDATED] : Redone the Quick Startup Guide.
[FIXED] : Date displayed are now in local time (instead of GMT).
[FIXED] : Data corruption could happen in rare cases.