Set Up Continuous Testing with Appium, AWS CodeCommit, Jenkins, and AWS Device Farm

AWS Device Farm is an app testing service that enables you to test your native, hybrid, and web apps on real Android and iOS devices hosted in the AWS Cloud. A test report containing high-level results, low-level logs, pixel-to-pixel screenshots, and performance data is updated as tests are completed.

The purpose of this walkthrough is to show how AWS Device Farm can be integrated into a continuous testing process to expose application risks with each build, enabling the development team to prevent problems from progressing to the next stage of the software development lifecycle. In this walkthrough, we will be leveraging the Appium automation framework for Android (Java based), AWS CodeCommit as our source control service, a Jenkins server for continuous builds, and AWS Device Farm to test against real mobile devices.

Pre-requisites

Below are the pre-requisites for getting started:

Eclipse IDE for Java
EGit, Eclipse tooling for Git (in Eclipse, copy and paste http://download.eclipse.org/egit/updates in the Help –> Install new Software dialog)
M2Eclipse, provides tight integration for Apache Maven into the IDE (in Eclipse, copy and paste http://download.eclipse.org/technology/m2e/releases in the Help –> Install new Software dialog)

Configure your Appium Maven project

In this walkthrough, we will be configuring a Java based Appium Maven project. If you don’t have an existing Appium Maven project in Eclipse, you can use the following sample project available on GitHub. Simply Download the project zip, extract it to a local directory, and import into Eclipse (File –> Import –> Maven –> Existing Maven Project).

Before going to the next step, make sure you can get a clean Maven build within Eclipse. To create a Maven build, simply right click on pom.xml and select Run As –> Maven build.

In the Edit Configuration dialog, specify “clean install” as Goals and click Run.

A zip-with-dependencies file will be generated in the target sub-folder if the build is successful:

Set up AWS CodeCommit and configure EGit

Create an AWS CodeCommit Repository

First create an AWS CodeCommit repository using the AWS CodeCommit Console or AWS CLI. Note the SSH URL of your repository as this will be needed in a following step:

Configure SSH Authentication for your CodeCommit repository

We will be using SSH credentials for accessing CodeCommit from EGit. Please use the following walkthroughs for setting up SSH credentials on Windows or OS X environments (you may skip Step 2 and Step 4, but make sure to test your SSH credentials in Step 3 before proceeding further).

Create a local clone of your AWS CodeCommit repository

The next step is to create a local clone of your AWS CodeCommit repository to be used with your Appium project. In Eclipse, open the Git Repositories view (Window –> Show View –> Other –> Git –> Git Repositories) and click on Clone a Git repository link. In the Clone Git Repository dialog box, select Clone URI option and click Next:

Paste the AWS CodeCommit SSH URL as the URI. The Host and Repository path fields will be auto-populated. Select ssh as the Protocol and click Next.

A warning message will appear on the next screen stating that the Source Git Repository is empty. Click Next to continue. On the next dialog, configure the local storage for the repository and click Finish.

Now right click on your Appium Maven project and select Team –> Share Project.

In the Repository dropdown, select the repository you have just cloned and press Finish.

Right click again on your Appium Maven project and select Team –> Commit. Add a commit message, select all files and click Commit.

Right click one more time on your Maven project and select Team –> Push branch ‘master’ to push the branch to your AWS CodeCommit repository. Click Next and Finish on the following two dialogs. A Push Results dialog will confirm a succesful push to the CodeCommit repository.

You may also login to AWS CodeCommit console to confirm that the repo has been updated:

Configure Jenkins

Install Jenkins server

If you don’t already have a Jenkins server, you can install one very quickly and easily on an Amazon Linux AMI using the following instructions:

If setting up a new Jenkins serer, create an Amazon EC2 service role and attach a managed policy AWSCodeCommitPowerUser to it. Jenkins will use the permissions granted by the IAM role to access the CodeCommit repositories.
Launch an Amazon Linux AMI and login to your instance.
Update your repositories: sudo yum update
Get Jenkins repository: sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
Get Jenkins repository key: sudo rpm –import http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
Install Jenkins server: sudo yum install jenkins
Start Jenkins server: sudo service jenkins start
Make sure the Jenkins server starts up automatically: sudo chkconfig jenkins on
Open your browser and navigate to http://<Server-IP>:8080. You should see the Jenkins dashboard.
Secure Jenkins server by selecting “Manage Jenkins” on the Jenkins dashboard and click on “Configure Global Security”. Check “Enable Security” and select the desired Access Control and Authorization.

Install Jenkins plugins

Login to Jenkins
Navigate to Manage Jenkins, Manage Plugins, and then choose the following plugins: aws-device-farm, Git plugin, and Copy To Slave Plugin and click Save.
Navigate to Jenkins Dashboard and select Manage Jenkins and then Configure System. Scroll down to Maven section and click Add Maven. Enter any name for the Maven installation and click Save to automatically configure the Maven installation.
Restart Jenkins

Configure the AWS Device Farm Jenkins plugin

Once your Jenkins server is up and running, install the AWS Device Farm Jenkins Plugin following these steps. You will then create an IAM user and Policy with DeviceFarmAll permissions that will be used by the plugin to submit Runs to the AWS Device Farm service.

Configure access to your CodeCommit respository

SSH to the Jenkins server.Install Git: sudo yum install git
If you are using Amazon Linux AMI for your Jenkins server the AWS CLI is already pre-installed. If you are not using Amazon Linux AMI, install AWS CLI.
Configure the AWS CLI for the jenkins user.

   cd ~jenkins
   sudo -u jenkins aws configure

Specify the AWS Access Key ID and AWS Secret Access Key for a user that has permissions to the CodeCommit repository you created earlier. If you are using an EC2 service role, leave the AWS Access Key ID and AWS Secret Access Key blank. Jenkins will use the permissions granted by the IAM role to access the CodeCommit repositories.

   AWS Access Key ID [None]: paste the Access Key ID, and then press Enter
   AWS Secret Access Key [None]: paste the Secret Access Key, and then press Enter
   Default region name [None]: type us-east-1, and then press Enter
   Default output format [None]: type json, and then press Enter

Next, configure Git on the Jenkins server to use IAM credentials and an HTTP path to access the repositories hosted by AWS CodeCommit.

   sudo -u jenkins git config --global credential.helper '!aws codecommit credential-helper $@'
   sudo -u jenkins git config --global credential.useHttpPath true
   sudo -u jenkins git config --global user.email "me@mycompany.com"
   sudo -u jenkins git config --global user.name "MyJenkinsServer"

Restart Jenkins server

Configure your Jenkins project with CodeCommit and Device Farm

From the Jenkins home page, select New Item. Type a project name and select a Maven project and click OK:

In the Source Code Management section, select Git and paste the HTTP endpoint of your CodeCommit repository.

You can get the HTTP endpoint of your CodeCommit repository in the AWS CodeCommit console:

For the Build Trigger, select Poll SCM and specify a schedule, for instance H/05 * * * *

In the Build Environment section, select Copy files into the job’s workspace before building and specify the name of the Android application file such as “app-debug.apk.” This file needs to be uploaded to $JENKINS_HOME/userContent directory on the Jenkins server before the build is triggered. Alternatively, you can include the application file within your Appium Maven project within Eclipse, depending on your development and testing workflow.

In the Post Steps section, select Run Tests on AWS Device Farm under the Add post-build action dropdown:

In the Run Tests on AWS Device Farm section, specify the following parameters:

Project: select the name of the Device Farm Project.
Device Pool: select a default Top Devices pool or another device pool that you have created previously.
Application: specify the name of the Android application file such as “app-debug.apk” (either included in your Appium Maven Project or copied to the workspace as specified in the Build Environment section above)
Choose test to run: select Appium Java TestNG and specify target/zip-with-dependencies.zip which is the Maven build output file containing all of the tests.

Click Save to save your Jenkins project configuration.

Trigger an automatic test run on Device Farm

Make an update to your Appium Maven Project in Eclipse, commit the changes and push the branch to AWS CodeCommit repository. After a few minutes you should see a Jenkins build in progress:

You can also navigate to Console Output to confirm a successful Maven build and a scheduled Device Farm run:

Return to the build status page and click on the Full Report link to automatically login to the AWS Device Farm console to view detailed statistics for the run and artifacts including detailed logs and screenshots.

Conclusion

In this tutorial we demonstrated how AWS Device Farm can be integrated into your Continuous Testing strategy. First, we showed how an Appium Maven project can be pushed to AWS CodeCommit repository. Then, by leveraging Jenkins, we configured a project that automatically triggers a Maven build and submits Appium tests to Device Farm for testing against hundreds of real mobile devices. If you have questions, please let us know in the comments section or in our developer forum.