Madefire Press

Blog

Keeping sane with a build system

Posted by Dan ,
general | Permalink | 7 Comments »

Any software developer can tell you what a nightmare it can be to make sure two consecutive builds are the same. A lot of pulled hair and late nights have gone before us as a testament to having a sane and uniform build system that is responsible for churning out builds the same way each time. As a small team we have been slowly improving on this aspect of things. Here’s how we’ve added a bit of sanity to the process of developing our iOS app at Madefire.

Jenkins

There are a lot of continuous integration and build systems out there. We’ve decided to go with Jenkins as our system. While we have the iOS app we also have other facets of our technology stack that need testing, including server side and web tools. Jenkins helps us make sure we’re not locked in to just a single platform for everything, since it will run on OS X, Linux, and Windows if needed. The ability for Jenkins to have a master and many slaves that we can add as needed is also a big help. If we ever find ourselves in a place where our current builds are backed up we will just need to add another machine to the pool.

One of the absolute greatest strengths of Jenkins is the plugin system. Many people have published their hard work in extending Jenkins as plugins. The plugins we use all the time are:

Without these we’d have to write our own scripts to handle quite a lot of functionality.

What We Build

We have a slew of build jobs on Jenkins, one for the web services, one for authoring scripts, and four for our iOS builds. For the rest of this post we’re going to focus on the iOS builds we have:

  • • Dev — every time we push to master on Github this starts a build, mostly as a sanity check to make sure nothing breaks long term.
  • • AppStore — when we’re ready for the App Store submission process we build here and then use the result to test and submit.
  • • Testflight — when we need an AdHoc build to share with our registered test devices we start this build and it pushes the final product to Testflight on success.
  • • Enterprise — we also have an Enterprise iOS account for all of our test devices so that we don’t take too many device slots on our regular portal. Like the Testflight build, this pushes the product to Testflight on success.

The Build Machine

For iOS development we’re limited to a Mac as our build machine. For our purposes we put a headless Mac mini (don’t forget to turn on screen sharing so you can get back to it) in the office and let it do its thing. We got the base model with 2GB of RAM. If we need more we can expand later, but for now this is more than enough machine for this use. As usual, we installed Xcode on it and made sure git was usable. We created a builder user whose only job is to run Jenkins as a client and build things.

The world of iOS development requires code signing in order to get an app onto a device. In order to make this work, we exported our certificates and profiles, including the distribution certificates, and installed them for the builder user. We haven’t yet got a system to keep the build machine up to date with provisioning profiles, but on the list to look at is cupertino from @mattt.

Configuring Xcode Projects

We want to build repeatably without worrying about the configuration of our local machines every time. We did this via a combination of Xcode Build Configurations and Schemes. Providing a Scheme per build type allows you to have pre- and post-build commands that are tailored to each type of build (e.g., changing push configurations based on the build type). We also use build configurations to tailor the build settings to each build type. For example, for each build configuration we can have a specific code signing parameter setup so that we don’t have to worry about which profile and certificate will get used. Other things that can be changed per Build Configuration (and with preprocessing the Info.plist file) are the display name of the app and bundle identifier.

Xcode Build Configurations
Xcode Build Configs for the Project

Xcode Code Signing Parameters
Xcode Code Signing Parameters for the Target

Once we had the Build Configs defined we created a Scheme for each one. This step may be overkill for your project. You may be able to get away with a Dev, AdHoc, and AppStore Scheme trio and then just flip Build Configurations to get what you need. Once everything is setup in Xcode and you can build it and get the results you expect, it’s time to make some Jenkins jobs.

This is Xcode. Jenkins Knows Xcode.

As mentioned above, we have a Dev build that builds every time we push to the master branch on Github. We don’t need nearly as much churn as that for our AppStore, Testflight, and Enterprise builds. In fact, since two-thirds of those builds are uploading to Testflight, for these we want to only start builds manually to avoid too many versions being too confusing on Testflight. For the App Store we only want that when we’re ready to submit, never any other time. In the Xcode plugin for Jenkins there are many fields to consider. From our Xcode configuration work above we have the needed values for Target, Xcode Schema File, and Configuration. The next step is that we want to change the Marketing version, or in Info.plist parlance the CFShortBundleVersionString. Originally, we were going to change the Technical Version or CFBundleVersion but Apple requires that to be a number and we use it for content version compatibility checks. In our Info.plist we will leave the value for CFBundleVersion alone at whatever our latest release to the App Store version is. All builds that are for non-engineer use should come from Jenkins.

For the Marketing version field we put a prefix like "TF-" for Testflight and "E-" for Enterprise and the Jenkins variable for the build number. When we look at a build we’ll be able to know which build Scheme and Configuration was used and when it was built via Jenkins. This is set at build time and doesn’t need to be commited to our repo every time we want to build. For our App Store version we have a small script that runs before the build to increment the CFBundleVersion/Technical version, commit that back to the repository, and push it to the origin. This allows us to only have the build number increment when Jenkins builds an App Store build, which is manually triggered.

Further down in the Xcode plugin options is Build IPA?. We do indeed want an IPA generated for us so we can upload it to Testflight or store it on S3 for later retrieval. In addition the plugin will zip up the dSYM and upload it so your crashes can be symbolicated. The Embedded Profile setting will need to point to a copy of the provisioning profile. You’ll have to have your profiles available to Xcode to code sign. In addition we keep a copy in the builder users home directory (or at least a symlink to the Xcode location) for easy access.

A word of warning on the Xcode Plugin distributed by default for Jenkins: there is a known bug that only surfaces if you don’t set the Technical version manually. It is noted in the Github pull request #9 for the Xcode plugin. Once we’re done with our release cycle we’ll be looking into contributing effort to help get a new release of the plugin out, but until then we’ve built a version of the Xcode plugin that contains the patch from the pull request. It is available in the downloads of the Madefire fork of the Xcode plugin.

Previously, we had installed the needed code signing certificates in our Keychain. Thankfully, the Xcode plugin can unlock your keychain for you. We tick the Unlock Keychain? option and then fill in Keychain path and Keychain password. The default login keychain location is ${HOME}/Library/Keychains/login.keychain. The password is the same as the builder user’s login password. Now Xcode can code sign your builds until heat death of the universe (extra-ordinary conditions excepted).

Artifacts For Digital Archeology

At the bottom of the Jenkins job configuration page there is a button that says Add post-build action. We have three post-build actions:Upload artifacts to S3, Upload to Testflight, and Git Publisher.

S3

The S3 Plugin is a little different for most Jenkins plugins in that it actually has a global configuration in the main Jenkens management screen. We configured one S3 profile to upload to and then returned to the jobs. For each one, we selected the configured profile and told it what files to upload with the Source entry. This field must be an Ant style glob, so we put something like **build/Product NameConfiguration-*.*. That will grab both the IPA and dSYM zip file. The Destination bucket we grabbed from the S3 interface. We decided for now to configure only one bucket for all builds to go into. We can change that later if we need to.

Testflight

For Testflight uploads the configuration is fairly straightforward. The API Token is your Testflight user’s personal key (we’ll make a builder user on Testflight so these automated builds don’t all look like they’re coming from one person) and the Team Token is key for the team where builds will be uploaded. For the IPA File path we choose ${WORKSPACE}/build/Product NameConfigurationMarketing version.ipa. This is different from the default output by the Xcode plugin (see the note about the fix above) that would be ${WORKSPACE}/build/Product NameConfigurationTechnical version.ipa. Having the same technical version until we run an AppStore build means we would potentially have a lot of overlapping files named the same thing, but containing different builds. To rectify this, we rename from the Technical version file name to the Marketing version file name in a shell script after the Xcode build has occurred. Please remember that each Xcode project will be different and the values will need to be customized to it, those are not variables in Jenkins.

Our Build output directory is set to ${WORKSPACE}/build so that the build products are local to the Jenkins workspace and easily accessible. The dSYM file is named similarly to the IPA as ${WORKSPACE}/build/Product NameConfigurationMarketing version-dSYM.zip (and we have to rename it the same as the IPA above). For the Build Notes we put a fairly generic string of "Uploaded from Jenkins. (Technical version ${BUILD_ID})". The ${BUILD_ID} variable puts the date and time just in case. That message will help us find uploaded builds and when we’re ready to send that Testflight build out we’ll change the build notes.

Tagging

We decided that we wanted to keep track of where every build for Testflight, Enterprise, and AppStore was in our tree. The Git plugin has a post build step called Git Publisher. With it we can tag the git repo with a tag like "REL_Technical version". That tag is then pushed back to our Github repo and we can step back through the tree if we need to. We worried about too many tags, but since we’re manually building these we shouldn’t have too many of them to worry about. If we were to do this tagging step for builds kicked off when code is pushed, it would likely be just noise, but for just our distributed builds it will be a bigger help down the road when we need to look at the state of the tree for a build.

Final Thoughts

Now we have builds that are generated by a stable non-developers’ machine. They are uploaded to S3 and Testflight automatically for us, removing another manual step from the process (for the AppStore builds we skip the Testflight upload). For all Jenkins jobs we have it email our engineering email list if there is a failure, and again when it returns to healthy. It’s almost so easy that we can forget it’s there and just let it work for us, as computers are supposed to do.

There are a few things that would make Jenkins configuration easier. In order to get all of this working we created a single job and made changes to the configuration until it worked. We then created new jobs that copied the working job and changed the few parameters that needed to be unique. It would be great to have common configuration things like Github and Testflight that could be centrally managed for all the redundant parts. The only per-job configuration would be items like filenames of artifacts that are specific to a job. We’d also like to see more variables from the Token Macro plugin. Getting the built product’s version string (e.g., 1.0) anywhere would be great. Possibly the Xcode plugin can be modified to export those values like the Git plugin does for the repository. That is something we’ll have to investigate later. Overall, Jenkins is a great tool to use. Anyone from small to large companies should be doing continuous integration and using a tool like Jenkins to have consistent builds for release. It will save you time down the road, and maybe save your bacon in a pinch.

Tags: , ,

7 Responses to “Keeping sane with a build system”

Matt Baker says:

We are trying to implement the same kind of build system using Jenkins. I was wondering how you are submitting your apps to the app store. Are you saving the .xcarchive? Are you still using the Application Loader application or if you are using Xcode itself to submit the app? Any help would be great.

Thanks,
Matt

    Dan says:

    Hi Matt,

    I haven’t yet gotten us into building xcarchives, but it’s something I’ve been interested in. Right now we just use the Jenkins Xcode plugin to build an IPA and zip the dSYM for us. We then upload that to a bucket on S3. Since our builds that are uploaded to S3 are triggered manually there aren’t too many of them, but you could have it upload for every automated build too. Once we have that IPA and dSYM we can pull them locally and then use the Application Loader.app that comes with Xcode (via the Xcode > Open Developer Tool > Application Loader menu item) to upload the IPA file to Apple.

Arshad says:

Hi Dan,

Great writeup. This would have been helpful back when I set this up! A couple of additional details that might be helpful to you and your readers, and a couple of questions for things that were hard for us to deal with and I figured out some solutions.

Q1: You mention that you have an app store build for testing and submitting to the app store. Do you actually have a way to test that build? In my experience I can’t test the one signed for the app store, and that’s caused some issues. If I make a debug version of it, I can test it, but that’s the best I can do.

Q2: You have both enterprise and app store building from the same machine. This has caused issues for us because we have our certificates both with the same organization name, and XCode then complains that “Certificate identity ‘xxx’ appears more than once in the keychain. The codesign tool requires there only be one.” Did you run into this problem?

I solved the latter somewhat by creating two separate Keychain Files and putting the enterprise certs in one and the App Store certs in the other. Jenkins has a place where you can specify the keychain to use, and it uses exclusively that one, and that works great.

XCode without Jenkins does not work as well, as it seems to like to mysteriously find and install the cert, or sometimes an old cert, in my login keychain, even when I’ve deleted it. I haven’t found a way to explicitly tell Xcode to use a specific keychain.

I also check in the keychains.

One thing we also do to help setup on new machines is that once a machine is properly set up with certs, keys, provisioning profiles, I go to the Organizer in XCode, and select my Team, and hit “Export”. I then save the resulting file where other machines can get it. This is a one time hit for everyone else whenever we change certs or provisioning profiles, but it’s no so bad.

There’s no way to automate the import easily because of the password… I did manage to make an applescript that can send the correct keypresses, but I don’t recommend that.

Instead, I check in the provisioning profiles, and also have a pre-build script that copies them into ~/Library/MobileDevice/Provisioning Profiles/ before each build, so others don’t even have to think about it.

    Dan says:

    Hi Arshad,

    Thanks for the great comment. You’ve given great explanations about things I haven’t had to deal with as we only have one build machine at this time, but I appreciate your clear writing so that I can follow along if we get to that point.

    As for your questions:

    A1: We test the exact binary that we send to Apple using the Ad Hoc provisioning profile. I wrote about this a few years ago on my personal blog. Basically you always have three profiles for every App Store app. Development, Distribution, and Ad Hoc. If you build an Xcode Archive you can resign the app with the Ad Hoc profile to make an IPA that can be installed directly or via a service like TestFlight or HockeyApp. The great thing about the Ad Hoc profile is that if it’s installed on a device and you put the App Store build on the device it will see the Ad Hoc profile and work. Make sure that if you regenerate the either the Ad Hoc or App Store profiles that you regenerate both at the same time to be safe.

    A2: Our organization name for Enterprise is slightly different from the normal App Store organization (missing a comma). That helps, but what we also have because of App Store limitations is a different bundle identifier for our Enterprise version (as an aside, that also allows us to have the store and Enterprise builds on a single device at the same time). So, if we have com.example.StellarApp for the store that becomes com.example.StellarApp.enterprise. Since only one distribution certificate will match that and our organization names are different we don’t run into any issues. Your second keychain method is how I handled this issue at a previous job, so you’ve got that solved.

    As for Xcode finding old certs and re-installing them, I can explain that too (based on my understanding of it). When you make a certificate using KeyChain Access it will be set to expire in a year. If you re-use the same certificate signing request the next year both certificates will have the same private key but different public keys. When Xcode sees an old profile that was generated it has the public keys in there and matches them to the private key that is already in the keychain. When it sees a private key without a public key it re-inserts the public key, in effect recreating your old expired certificate for you. Very helpful, I know.

    The solution for me, and it’s a bit more work, is that when you make a new certificate you use a new Certificate Signing Request. I also regenerate any old profiles that used the old certificate since you can’t use them any more anyway.

    I hope that helps!

    Dan

David Welch says:

This is an awesome post, with great comments! Thanks to everyone who shared.

I moved jobs about a year ago and we’ve had a very messy iOS Jenkins box that barely gets the job done. Recently the Mac Mini started acting up (about 6 years old now) so I’m in the process of building new machine and this has helped me out tremendously, so thank you!!

One question though: has anyone found a way to submit an app directly to the AppStore for review from the Jenkins build process (or even just a script)? A friend from another company mentioned he was close to having it solved, but I haven’t had a chance to hear how.

    Dan says:

    Hi David,

    I’m glad you found the post useful. I know newer Jenkins Xcode plugin versions can make an archive for you to drop into Xcode to submit with, but I don’t know of an automated way to submit at this point. The only way to submit is either via Xcode or the Application Loader.app. The AppleScript support in Xcode is currently weak to non-existant. I’m not sure about Application Loader, but that could be an option.

    Dan

Leave a Reply

 

Get the RSS feed

Sign-up for our newsletter