How to run your iPhone test on AWS Device Farm

Amazon introduced AWS Device Farm some time ago and later on added iOS support. This service is great if you need to test on a specific device, but don't have one handy as you can remotely test your app.

The other huge benefit is that you can run your automated tests on real devices with a mix of OS version. Unfortunately, this entails a fairly complex process, so here is a walkthrough.

What you'll need

- Xcode 8

- a project you want to play with


Getting Started

If you are creating your project from scratch for this, make sure that you select a team when you create the project. This will ensure that all the code signing works correctly. Also, check the "Include UI Tests" check box.

Remember to set the team and to select to "Include UI Tests"

Create your UI tests

If you were not creating the project from scratch, you'll need to create a target to contain your UI tests. Else skip this section.

Choose your project in the project navigator on the left, then hit the "+" at the bottom of the screen.

In the dialog that opens, select "iOS UI Testing Bundle" and hit "Next".

Double check that all the settings for your test bundle are to your liking and click "Finish".


Build your XC UI Test bundle for tests

Here you'll be introduced to a feature of Xcode I did not know about until recently. You can build for specific purpose and in our case we want to build the test bundle for testing. This will generate an app to be upload to AWS Device Farm later on. 

Note that AWS Device Farm requires the test bundle to be build for iOS device and not for the simulator as the tests will be ran on actual devices. If you miss this step, you'll get a cryptic error that the file you uploaded was invalid... With that in mind, in the device selector next to the scheme selector, select "Generic iOS Device".

Then go to Product > Build For > Testing to actually build the runner.

Once your build is complete, you need to find your runner app. 

  1. Head over to your Derived Data folder (~/Library/Developer/Xcode/DerivedData)
  2. Open the folder that looks like <Name of your project><random letters>
  3. Dig down following path: Build/Products/Debug-iphoneos/
  4. Find the file named <Project Name> 

That test runner is what AWS device farm needs, but for some reason they need it as a .IPA file. Here is how to do it:

  1. Create a folder and name it "Payload" (this is case sensitive)
  2. Copy your test runner to the Payload folder you just created
  3. Right click on the Payload folder and click "Compress Payload"
  4. Rename the file you just create to "Payload.ipa"

Keep this file for later:-)

Archive the app

Next step is to archive the app, this should be familiar to you, nothing special here.

  1. Select Product > Archive.
  2. Once the archive is complete, open the Organizer (Window > Organizer)
  3. Select the archive you just created and hit "Export"
Screen Shot 2016-09-21 at 11.49.19.png

Select "Save for Development Deployment", hit next. Select to export one app for all supported devices, hit next. And finally hit "Next" one more time. 

A few minutes later, Xcode will ask you where to save the IPA, select a handy location as you'll need it soon.

UPload to AWS device farm

We're done playing with Xcode, everything else is happening on AWS. So head over to AWS device farm and create a new project.


On the next screen you'll hit "Create a new run".

For the rest of the process, AWS was kind enough to number the steps here so i'll use those step numbers too.

In the page that opens, for step 1, you'll have to upload your application IPA.

  1. Click on the Apple/Android button to indicate you're doing mobile app tests and not web site tests
  2. Hit upload and choose the application IPA you exported when you archived your application
  3. Enter a name for the run and hit "Next step"

In step 2, you'll upload your test runner app (Payload.ipa).

  1. Select the type of test you'd like to run on the device (in our case XCTest UI)
  2. Hit upload
  3. Choose the Payload.ipa file you created earlier
  4. Hit "Next step"

On the next screen you'll likely get an error. XCTest UI requires iOS9 or above and the default devices AWS offers still run iOS 8... So you'll need to create a new device pool (a list of device you want to test on).

  1. Hit "Create a new device pool"
  2. Select iOS9 devices
  3. Name the pool (i.e.: iOS9 devices)
  4. Hit Save.


Then when you are back on the "Select devices" screen, select the pool you just created.

Step 4 allows you some extra options that I won't cover today, so just go ahead and click "Review and start run".

That's it, now you have to wait for your tests to run. There is no end of tests notification, so you'll have to periodically check the page... According to Amazon, they are working on a notification mechanism.



What do you think?

Will you test your app on more devices now that you can do it all in the cloud without having to have the actual device? In another post i'll also talk about what it takes to do this from the command line or a continuous integration system like

Feel free to ask questions in the comments, I'll do my best to answer those.