Testing iPhone apps effectively with release builds

Question

We have an app that has crashed on armv6 iOS devices coming from the App Store. armv7 iOS devices ran it just fine. When the app was built and tested as debug, it ran just fine on both armv6 and armv7. In the logs, I was getting EXC_BAD_INSTRUCTION when it tried to construct an object from a library. The crash appeared to be a linking error on the release build since I have several static libraries from three20. At first I thought it was an iOS version problem, but its now looking like a "fat binary" issue.

Is ad-hoc the best way to have a simulated App Store for testing? What is the best way to test a release build on an device? What would be the best way to test the linking with the different devices on a release build?

Answer 1

I use ANT to create releases, utilizing the xcodebuild command line utility. The ant target looks like this:

 <target name="build-adhoc">
        <echo>Running XCODE compiler</echo>
        <exec executable="xcodebuild" failonerror="true" vmlauncher="false" dir="${connect_src}">
            <arg value="build"/>
            <arg value="-target" />
            <arg value="myappname" />
            <arg value="-configuration"/>
            <arg value="AdHoc"/>
            <arg value="SYMROOT=${export_app}-${version.number}/AdHoc" />
            <env key="USER_HEADER_SEARCH_PATHS" value="/tmp/build/trunk/Libraries/somesource/**\ /tmp/build/trunk/Libraries/somemoresource/**"/>
        </exec>
        <echo>xcode build complete</echo>
    </target>

Create a configuration for your ad hoc release and make sure it uses an ad hoc ready provisioning profile. Or, you can create a normal, non-ad hoc configuration that uses your Developer provisioning profile. If you do the former, you can email the product, along with your provisioning profile, to testers. If you do the latter, you can use the iPhone Config Utility to install on any device in the profile.

Also, we have the ANT target setup with dependencies on doing a fresh checkout from source control and doing the build from there. Finally, we do a build right after that using a real App Store-ready configuration so we can test the first build and ship the second to Apple, with some level of certainty that the code behavior will be the same.

Here is a summary of our ANT targets in order of processing:

1. set env vars needed for builds
2. clear checkout and build directories
3. checkout HEAD from source control
4. modify plists to remove debug and elements only for internal use (using PLISTBUDDY)
5. Increment build version (using agvtool)
6. Run unit tests (this needs the sdk setting set to iphonesimulator4.x and a special target -- see iPhone Developers Guide for unit testing)
7. Do internal build (using either dev profile or ad hoc profile)
8. Do distribution build (using App Store distribution certificate)
9. Checkin plist files that have the new build version
10. Do a tag in source control indicating the build
11. Report build statistics (build version, location, etc.)

NOTE: make sure you set the SYMROOT env variable in the xcodebuild command syntax so your build ends up in the proper directory. We've had problems copying App Store builds after the fact.

Here is the full ANT script used to create multiple builds, one for testing and one for distribution. For example: "ant distribution" will create one build using the dev profile and one using the App Store distribution profile.

This script is "cleaned," so it only serves as a model from which to start:

<?xml version="1.0" ?>

<project basedir="." default="xcode-build" name="Temp">
    <property file="build.properties" />
    <property name = "CONFIGURATION_INTERNAL" value = "InternalRelease" />
    <property name = "CONFIGURATION_DISTRIBUTION" value = "Distribution" />
    <property name = "CONFIGURATION_ADHOC" value = "InternalAdHoc" />
    <property name="cvsroot" value=":pserver:${username}:${password}@${cvsurl}"/>
    <tstamp>
      <format property="TODAY"
              pattern="MM-dd-yyyy"
              locale="en,US"/>
    </tstamp>


    <target name="init">
        <echo message="deleting old directories" />
        <delete dir="${check_out_location}"/>
        <mkdir dir="${check_out_location}"/>
    </target>


    <target name="set-source-trees" depends="init">
    <echo message="exporting source tree variables" />
        <echo message="${somesourcedir}" />
        <exec executable="/bin/bash" os=" Mac OS X">
            <arg value="-c" />
            <arg value="${export_src_trees}"/>
            <arg value="${anothersourcedir}"/>
        </exec>
        <exec executable="/bin/bash" os=" Mac OS X">
            <arg value="-c" />
            <arg value="${export_src_trees}"/>
            <arg value="${yetanothersourcedir}"/>
        </exec>
    </target>


    <target name="cvs-login" depends="init" description="CVS Login">
        <echo>Login CVS</echo>
        <cvs cvsroot=":pserver:${username}:${password}@${cvsurl}" command="login" />
    </target>

    <target name="checkout" depends="cvs-login" description="Check out source from CVS">
        <echo message="check out from CVS ...." />
        <echo message="${check_out_location}" />
        <cvs cvsroot="${cvsroot}"  command=" -Q checkout -P -d${project_trunk} ${project_repository_root}/${project_trunk} " dest="${check_out_location}" compression="true" />
        <echo message="...check out from CVS done" />
    </target>


     <target name="strip-settings" depends="checkout" description="Remove elements from the Settings.bundle that we don't want in the distribution">
        <echo message="Removing Settings not valid for distribution"/>

        <exec executable="/usr/libexec/PlistBuddy" failonerror="TRUE" dir="${app_src}/Resources">
            <arg value="-c"/>
            <arg value="Delete :PreferenceSpecifiers:3"/>
            <arg value="Settings.bundle/Root.plist"/>
        </exec>

        <exec executable="/usr/libexec/PlistBuddy" failonerror="TRUE" dir="${app_src}/Resources">
            <arg value="-c"/>
            <arg value="Delete :PreferenceSpecifiers:3"/>
            <arg value="Settings.bundle/Root.plist"/>
        </exec>
    </target>


    <target name="build-version" depends="checkout">

        <property name = "LOGLEVEL" value = "DEBUG" />

       <!-- GET THE NEXT VERSION NUMBER (major and minor) -->
        <exec executable="/tmp/build/trunk/Build/version.sh" failonerror="TRUE" outputproperty="version.number" dir="${app_src}"></exec>

        <echo message="Increment Build Number"/>

        <exec executable="agvtool" failonerror="TRUE" dir="${app_src}">
            <arg value="new-version"/>
            <arg value="-all"/>
            <arg value="${version.number}"/>
        </exec>

        <!-- GET THE MINOR portion of the version number for later use -->
        <exec executable="/tmp/build/trunk/Build/minor.sh" failonerror="TRUE" outputproperty="version.minor" dir="${app_src}"></exec>

        <!-- SET the version number as reference in the Settings.bundle -->
        <exec executable="/usr/libexec/PlistBuddy" failonerror="TRUE" dir="${app_src}">
            <arg value="-c"/>
            <arg value="Set :PreferenceSpecifiers:1:DefaultValue ${version.number}"/>
            <arg value="./Resources/Settings.bundle/Root.plist"/>
        </exec>

        <echo message="New build number=${version.number}"/>

        <!-- SET the log level - NOTE, ANT vars are immutable, if LOGLEVEL was previously set, it cannot be overridden -->
        <exec executable="/usr/libexec/PlistBuddy" failonerror="TRUE" dir="${app_src}">
            <arg value="-c"/>
            <arg value="Set LogLevel.Default ${LOGLEVEL}"/>
            <arg value="./Resources/SharedConfig.plist"/>
        </exec>
        <echo message="Log level set to ${LOGLEVEL}"/>

    </target>


     <target name="encrypt" depends="build-version">
        <!-- SOME ENCRYPTION OF SENSITIVE DATA -->
    </target>


    <target name="build-internal" depends="test-lib1, test-lib2, test-app, encrypt">
        <echo>Running XCODE compiler</echo>
        <exec executable="${xcode_builder}" failonerror="true" vmlauncher="false" dir="${app_src}">
            <arg value="clean"/>
            <arg value="install"/>    
            <arg value="-target" />
            <arg value="MyApp" />
            <arg value="-configuration"/>
            <arg value="${CONFIGURATION_INTERNAL}"/>
            <arg value="SYMROOT=${build_release}" />
            <env key="USER_HEADER_SEARCH_PATHS" value="/tmp/build/trunk/Libraries/somesource/**\ /tmp/build/trunk/Libraries/someothersource/**"/>
        </exec>
        <echo>xcode build complete</echo>
    </target>

     <target name="test-lib1" depends="checkout">
        <echo>Running XCODE compiler</echo>
        <exec executable="${xcode_builder}" failonerror="true" vmlauncher="false" dir="${somelib_dir}">
            <arg value="clean"/>
            <arg value="build"/>
            <arg value="-target" />
            <arg value="Lib1UnitTests"/>
            <arg value="-sdk"/>
            <arg value="iphonesimulator4.1"/>
            <arg value="SYMROOT=${build_release}" />
        </exec>
        <echo>xcode build complete</echo>
    </target>

     <target name="test-lib2" depends="checkout">
        <echo>Running XCODE compiler</echo>
        <exec executable="${xcode_builder}" failonerror="true" vmlauncher="false" dir="${somelib2_src_dir}">
            <arg value="clean"/>
            <arg value="build"/>
            <arg value="-target" />
            <arg value="Lib2UnitTests"/>
            <arg value="-sdk"/>
            <arg value="iphonesimulator4.1"/>
            <arg value="SYMROOT=${build_release}" />
            <env key="USER_HEADER_SEARCH_PATHS" value="/tmp/build/trunk/Libraries/Lib1source/**"/>
        </exec>
        <echo>xcode build complete</echo>
    </target>

      <target name="test-app" depends="checkout">
        <echo>Running XCODE compiler</echo>
        <exec executable="${xcode_builder}" failonerror="true" vmlauncher="false" dir="${app_src}">
            <arg value="clean"/>
            <arg value="build"/>
            <arg value="-target" />
            <arg value="AppUnitTests"/>
            <arg value="-sdk"/>
            <arg value="iphonesimulator4.1"/>
            <arg value="SYMROOT=${build_release}" />
            <env key="USER_HEADER_SEARCH_PATHS" value="/tmp/build/trunk/Libraries/Lib1/**\ /tmp/build/trunk/Libraries/Lib2/**"/>
        </exec>
        <echo>xcode build complete</echo>
    </target>

     <target name="build-distribution" depends="internal-release, test-lib1, test-lib2, test-app">
        <echo>Running XCODE compiler</echo>
        <exec executable="${xcode_builder}" failonerror="true" vmlauncher="false" dir="${app_src}">
            <arg value="build"/>
            <arg value="-target" />
            <arg value="MyApp" />
            <arg value="-configuration"/>
            <arg value="${CONFIGURATION_DISTRIBUTION}"/>
            <arg value="SYMROOT=${export_app}-${version.number}/Distribution" />
            <env key="USER_HEADER_SEARCH_PATHS" value="/tmp/build/trunk/Libraries/Lib1/**\ /tmp/build/trunk/Libraries/Lib2/**"/>
        </exec>
        <echo>xcode build complete</echo>
    </target>

    <target name="build-adhoc" depends="internal-release, test-lib1, test-lib2, test-app">
        <echo>Running XCODE compiler</echo>
        <exec executable="${xcode_builder}" failonerror="true" vmlauncher="false" dir="${app_src}">
            <arg value="build"/>
            <arg value="-target" />
            <arg value="MyApp" />
            <arg value="-configuration"/>
            <arg value="${CONFIGURATION_ADHOC}"/>
            <arg value="SYMROOT=${export_app}-${version.number}/AdHoc" />
            <env key="USER_HEADER_SEARCH_PATHS" value="/tmp/build/trunk/Libraries/Lib1/**\ /tmp/build/trunk/Libraries/Lib2/**"/>
        </exec>
        <echo>xcode build complete</echo>
    </target>


    <target name="checkin" depends="build-internal" description="Commit source to CVS" >
        <echo message="Committing to CVS ...." />
        <echo message="${check_out_location}" />

         <cvs cvsroot="${cvsroot}"  command="commit -m 'Commit of internal release build ${version.number}' MyApp-Info.plist" dest="${check_out_location}/trunk/Applications/MyApp" />
        <echo message="...commit done" />
    </target>


     <target name="tag-release" depends="checkin" description="Tag source in CVS" >
        <echo message="Tagging source ... Release-${version.minor}" />
        <echo message="${check_out_location}" />
        <cvs cvsroot="${cvsroot}"  command="tag Release-${version.minor}" dest="${check_out_location}"/>
        <echo message="...tag done" />
    </target>


     <target name="internal-release" depends="tag-release" description="Tag source in CVS" >
        <echo message="Creating Internal release ..." />
        <echo message="Deploying files to . . .${export_app}" />
        <copy todir="${export_app}-${version.number}">
            <fileset dir="${prod_dir_internal}"/>
        </copy>

        <echo message="Internal release ${version.number} complete." />
    </target>


     <target name="override-default-env"  description="Setup Env for Distribution" >
                <property name = "LOGLEVEL" value = "WARN" />
     </target>


     <target name="distribution" depends="override-default-env, strip-settings, build-distribution" description="Create Distribution" >

        <echo message="Creating Distribution ..." />
        <echo message="Deploying files to . . .${export_app}-${version.number}/Distribution" />
        <echo message="Distribution ${version.number} complete." />

    </target>

     <target name="adhoc" depends="override-default-env, strip-settings, build-adhoc" description="Create Ad Hoc Distribution" >

        <echo message="Creating ad hoc distribution ..." />
        <echo message="Deploying files to . . .${export_app}-${version.number}/AdHoc" />
        <echo message="Ad hoc distribution ${version.number} complete." />

    </target>


</project>

Hope this helps.

Answer 2

I can't find a reference to the iOS compatibility testing labs but I heard this is presently available. If I can find this resource, I'll update my response.

In the mean time, you can find UI automation testing advice here: http://stackoverflow.com/questions/402389/automated-testing-for-iphone

Unfortunately, you'll have to find a 3g iPhone to verify arm6 compatibility. I expect you should be able to find one easy enough just by asking everyone you know who has a new iPhone. They probably have their old phone sitting in a drawer like I do. I use mine for testing.

If you can justify a sophisticated solution, you could permanently attach the old iPhone to a mac mini and drive the UI using the above referenced UI testing frameworks using Hudson or CruiseControl. This would be the most reliable and least time consuming approach if you can make that up front investment.

Answer 3

You can change the code signature authority on a built app using the codesign command-line tool. After building your distribution target, the app file will have a "distribution" authority; you need to change it to development authority. After the change, you can then install the build on a development device and test away.

To see a verbose dump of the app signature:

$ codesign -d -vv MyApp.app/MyApp

To change the codesign authority:

$ codesign -f -s "My iPhone Developer Name" -vv MyApp.app/MyApp

You might need to create an environment variable to grab the correct version of codesign_allocate:

$ export CODESIGN_ALLOCATE=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/codesign_allocate

I found these directions on Craig Hockenberry's blog, with lots of additional information on this process: http://furbo.org/2008/11/12/the-final-test/