Test Your App On Any Device You Want, Without The Hardware From the Command Line

In a past blog post, I ran you through the steps necessary to run your XC UI Test on AWS device farm. Today, I'll show you how to run those via the command line.

The best part about being able to run those tests via the command line is that once you can do it via a script you can make it part of your automated build (but that will be for another blog post).

So what do you need:

  • Xcode 8
  • a project to play with a UI test target
  • An account with AWS

The steps we'll go through

  • Create a project
  • Create your UI tests
  • Build your UI test bundle
  • Archive the app
  • Create a project on AWS Device Farm
  • Create a device pool on AWS Device Farm
  • Install the AWS CLI
  • Configure the AWS CLI
  • Uploading the test runner and app to AWS Device Farm
  • Scheduling the test on AWS Device Farm

Getting Started

If you are creating your project from scratch, make sure that you select a team when you create the project. This will ensure that all the code signing works correctly. Also, check the "Include UI Tests" check box.

Remember to set the team and to select to "Include UI Tests"

Create your UI tests

If you were not creating the project from scratch, you'll need to create a target to contain your UI tests. Else skip this section.

Choose your project in the project navigator on the left, then hit the "+" at the bottom of the screen.

In the dialog that opens, select "iOS UI Testing Bundle" and hit "Next".

Double check that all the settings for your test bundle are to your liking and click "Finish".

Build your test bundle

Here you'll be introduced to a feature of Xcode I did not know about until recently. You can build for specific purpose and in our case we want to build the test bundle for testing. This will generate an app to be upload to AWS Device Farm later on. 

Note that AWS Device Farm requires the test bundle to be build for iOS device and not for the simulator as the tests will be ran on actual devices. If you miss this step, you'll get a cryptic error that the file you uploaded was invalid... 

First here is the command line, I'll explain it below (remember to replace the appropriate string with your workspace and scheme name).

xcodebuild -workspace "aws device farm blog post.xcworkspace" \
 -scheme "aws device farm blog \
 -destination "generic/platform=iOS" \
 -derivedDataPath "./build" \ 
 -configuration "Debug" \
 build-for-testing \ 

So what am I doing here: 

  1. -workspace: the workspace to build with the option
  2. -scheme: the scheme to build, get it from your Xcode list of schemes in your project
  3. -destination: sets to build for devices and not for the simulator
  4. -derivedDataPath: specifies a local derived data path so you don't have to go dig out your build products
  5. -configuration: specifies which configuration to use to build, this is optional, but specifying it ensures that you know what gets built
  6. build-for-testing: the magic command that will build your test runner
  7. CODE_SIGN_IDENTITY="" and CODE_SIGNING_REQUIRED are just a way I found to workaround any kind of code signing issue your project might have

Once your build is complete, you'll have the test runner that AWS device farm needs, but for some reason they need it as an .IPA file. So here are the steps to create that .IPA file:

# create the payload directory
mkdir -p ./Payload

#copy the test runner in the payload directory
cp -R "./build/Build/Products/Debug-iphoneos/aws device farm blog postUITests-Runner.app" ./Payload

# zip the folder
# this command is important to get right so that archive only contains relative path
zip -r ./Payload.ipa ./Payload

./payload.ipa is the IPA that contains your test bundle and that will be uploaded to AWS Device Farm in a few steps.

Archive the app

Next step is to archive the app, this should be familiar to you, nothing special here. But as a refresher:

xcodebuild clean archive -archivePath build/AWS \ 
 -scheme "aws device farm blog post"
  • -archivePath tells Xcode where to save the archive, choose a handy location
  • -scheme indicate which scheme to build

Export the APP

Same here nothing special, but here are the details:

xcodebuild -exportArchive \
 -exportFormat ipa \
 -archivePath "build/AWS.xcarchive" \
 -exportPath "build/AWS.ipa" \
 -exportProvisioningProfile "ProvisioningProfileName"
  • -exportFormat: specifies what kind of output you are looking for and here we need an IPA
  • -archivePath: indicates which archive to export, so this should be the archive you generated in the previous section
  • -exportPath: where is the .IPA file is going to be saved
  • -exportProvisioningProfile: specify which provisioning profile to use

At this point the app .IPA is at build/AWS.ipa

Create an AWS project

This could be done with the command line, but I don't see the point as you do this once and forget about it. So simply log in to AWS device Farm and click on "Create Project", give it a good name and you're good to go.

Create a device pool

Same here, you could do it via the command line, but given that this is the kind of thing you do once and forget, you might as well do it via the UI.

  1. click on the gear icon at the top right of the page
  2. click on the device pools tab
  3. Hit "Create a new device pool"
  4. Select iOS9 devices (only iOS9 devices allow to UI test to run)
  5. Name the pool (i.e.: iOS9 devices)
  6. Hit Save.

download & install AWS CLI

AWS has a handy command line tool, which you can easily install with brew:

brew install awscli

Configuring the AWS CLI

At this point we're almost ready to go and start testing on AWS Device farm, but before we can move on, we need to get some information from AWS Device Farm. Namely, we need a couple of ARNs. ARNs are the way AWS uniquely identify entities in their system. So we need the ARN of the project and the ARN of the device pool we just created.

The only way I found to get those is via the command line.

So let's dive right in. The AWS command line needs a few pieces of information to run correctly. I recommend setting those as environment variable.

export AWS_ACCESS_KEY_ID=xxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxx
export AWS_DEFAULT_REGION=us-west-2
  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY: you need to get those from your AWS account, they allow to identify you to AWS
  • AWS_DEFAULT_REGION: my understanding is that AWS Device Farm only works in one region right now, so this should stay constant

Getting a few more pieces of information

Now we're ready to get the project ARN:

aws devicefarm list-projects

You should get a response like below

"projects": [
"name": "my great project", 
"arn": "arn:aws:devicefarm:us-west-2:111111111111:project:xxxxxxxx-yyyy-zzzz-aaaa-2637f3a428bc", 
"created": 1474027979.191

You project ARN is contained in the arn value: "arn:aws:devicefarm:us-west-2:111111111111:project:xxxxxxxx-yyyy-zzzz-aaaa-2637f3a428bc" you'll need it a little later. 

Now on to the device pool ARN:

aws devicefarm list-device-pools --arn arn:aws:devicefarm:us-west-2:111111111111:project:xxxxxxxx-yyyy-zzzz-aaaa-2637f3a428bc

Again you'll get a JSON response

"devicePools": [
"rules": [
"operator": "IN", 
"attribute": "ARN", 
"value": "[...]"
"name": "Top Devices", 
"arn": "arn:aws:devicefarm:us-west-2::devicepool:082d10e5-d7d7-48a5-ba5c-b33d66efa1f5", 
"description": "Top devices"
"rules": [
"operator": "IN", 
"attribute": "ARN", 
"value": "[...]"
"name": "iOS test pool", 
"arn": "arn:aws:devicefarm:us-west-2:077687060716:devicepool:xxxxxx-yyyy-zzzz-aaaa-2637f3a428bc/a9b95f6c-xxxx-yyyy-aaaa-38781f632e97"

By default AWS creates a "Top devices" pool, you can ignore that one. You'll find your device pool ARN in that response in the arn field. Write down the ARN.

Uploading the app and the test bundle

Before we can run our tests, we need to upload the application and the test bundle. An upload actually consists of two steps:

  1. Creating an upload with the AWS CLI
  2. Doing the actual upload

So here is how you upload the test bundle:

#create an upload
aws devicefarm create-upload --name testBundle.ipa \
 --project-arn <your project ARN>
  • --name: name you want to give to the upload. It's not that important, just make sure its extension is .ipa
  • --type: indicates the type of test your are uploading, in our case XCTEST_UI_TEST_PACKAGE
  • --project-arn is the arn we retrieved a few step above

Again, you'll get a json blob back, copy the ARN of the upload for later use and copy the url, you'll need it in the next step.

#actually uploading the file
curl -T <path to the IPA containing your test bundle> <url parameter from the last step>)

This will take a little bit of time depending on your connection speed and the size of your app. 

Once it is done, you can repeat the process with the application IPA

#upload the app IPA
aws devicefarm create-upload --name app.ipa \
 --type IOS_APP \ 
 --project-arn <project ARN>
  • --name: name you want to give to the upload. It's not that important, just make sure its extension is .ipa
  • --type: indicates the type of test your are uploading, in our case IOS_APP
  • --project-arn is the ARN we retrieved a few step above

Again, you'll get a json blob back, copy the ARN of the upload for later use and copy the url, you'll need it in the next step.

#actually uploading the app file
curl -T <path to your app IPA> <url parameter from the last step>)

Running the tests

Once both those upload are completed, you can move on to the last step... Yes, we are at the point where we can start those damn tests.

 aws devicefarm schedule-run \
 --project-arn <your project ARN> \
 --app-arn <the ARN of the app upload> \
 --device-pool-arn <the ARN of your device pool> \
 --name "<something recognizable, I use the commit message>" \
 --test '{"type":"XCTEST_UI","testPackageArn":"'<your test bundle ARN'"}'


  • --name: when you go back to AWS device farm site, this is what will allow you to differentiate your various test runs, so give it something meaningful, I like to give the commit message of the commit I'm testing
  • --test: a JSON blob containing the type of test (here: XCTEST_UI) and testPackageArn which is the test bundle ARN you got when you uploaded your test package

The command will return almost immediately. All the command did was schedule the run, you'll have to go to AWS Device Farm site to see the results. Or you could poll and wait for the run to have completed. But that is for another day.

Obviously the bug benefit of doing this via the command line is to be able to automate this process as part of your build automation. This will allow to run your tests on physical device before you do a release automatically, which is pretty cool.


If you liked what you just read, don't forget to sign up for my newsletter and to share it with your friends.