Front-End Web & Mobile
Get started with the AWS Device Farm CLI and Calabash Part 2: Retrieving Reports and Artifacts
In Part 1 of the walkthrough, we went through the process of creating an AWS Device Farm run for Calabash-Android test scripts. In the second and final part of the series, we will go through the CLI commands for obtaining the status of the run and to retrieve all of the artifacts associated with a run including log files and screenshots.
Device Farm Run Components
Before we begin, let’s review the hierarchical structure of a Device Farm run:
- RUN – A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a specific set of devices. A run produces a report that contains information about the test execution. A run contains one or more jobs.
- JOB – A job is a request to test a single app against a single device. A job contains one or more suites.
- SUITE – A suite is a hierarchical organization of tests in a test package. A suite contains one or more tests.
- TEST – A test is an individual test within a suite. This is the most granular entity in Device Farm’s data hierarchy.
Retrieving Reports and Artifacts
Using Device Farm CLI, we can obtain the status and result counters for each level of the run hierarchy including run, job, suite, and test. As you’ll see, the data available for each level of the hierarchy is very similar, with the scope of the data becoming more granular as you navigate deeper in the hierarchy.
Step 1: Retrieve the Status and Result Counters for a Device Farm Run
Use the get-run CLI command to retrieve the status and result counters of the run.
[ec2-user]$ aws devicefarm get-run –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917
Output:
{
“run”: {
“status”: “COMPLETED”,
“name”: “firstCLIRun”,
“created”: 1443331615.166,
“totalJobs”: 101,
“completedJobs”: 101,
“platform”: “ANDROID_APP”,
“result”: “ERRORED”,
“billingMethod”: “METERED”,
“type”: “CALABASH”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917”,
“counters”: {
“skipped”: 37,
“warned”: 16,
“failed”: 46,
“stopped”: 0,
“passed”: 514,
“errored”: 66,
“total”: 679
}
}
}
The above Run has a total of 101 jobs corresponding to 101 unique devices, all of which have completed. This Run also has 679 individual tests with 46 failures and 66 errors.
Step 2: Retrieve the Status and Counters for all Jobs within a Run:
Use the list-jobs CLI command to list all jobs within a run along with job-level status and result counters.
[ec2-user]$ aws devicefarm list-jobs –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917
Output (partial):
{
“jobs”: [
{
“status”: “COMPLETED”,
“name”: “ASUS Nexus 7 – 2nd Gen (WiFi)”,
“created”: 1443331616.586,
“result”: “PASSED”,
“device”: {
“formFactor”: “TABLET”,
“name”: “ASUS Nexus 7 – 2nd Gen (WiFi)”,
“resolution”: {
“width”: 1200,
“height”: 1920
},
“image”: “NA”,
“platform”: “ANDROID”,
“heapSize”: 0,
“memory”: 34359738368,
“model”: “Nexus 7 – 2nd Gen”,
“os”: “4.3.1”,
“cpu”: {
“frequency”: “MHz”,
“architecture”: “ARMv7”,
“clock”: 1512.0
},
“arn”: “arn:aws:devicefarm:us-west-2::device:208FE7EE973042EA97DEC2EEF31CD10A”,
“manufacturer”: “ASUS”
},
“message”: “Successful test lifecycle of Setup Test”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066”,
“counters”: {
“skipped”: 0,
“warned”: 0,
“failed”: 0,
“stopped”: 0,
“passed”: 7,
“errored”: 0,
“total”: 7
}
},
{
“status”: “COMPLETED”,
“name”: “Amazon Kindle Fire HDX 7 (WiFi)”,
“created”: 1443331615.56,
“result”: “FAILED”,
“device”: {
“formFactor”: “TABLET”,
“name”: “Amazon Kindle Fire HDX 7 (WiFi)”,
“resolution”: {
“width”: 1920,
“height”: 1200
},
“image”: “NA”,
“platform”: “ANDROID”,
“heapSize”: 0,
“memory”: 17179869184,
“model”: “Kindle Fire HDX 7”,
“os”: “4.4.3”,
“cpu”: {
“frequency”: “MHz”,
“architecture”: “ARMv7”,
“clock”: 2150.0
},
“arn”: “arn:aws:devicefarm:us-west-2::device:BC44B6802F134918BDAB6FB4F38C37CC”,
“manufacturer”: “Amazon”
},
“message”: “I see the text “In Theaters” failed: Timeout waiting for elements: * {text CONTAINS[c] ‘In Theaters’} (Calabash::Android::WaitHelpers::WaitError)”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00018”,
“counters”: {
“skipped”: 0,
“warned”: 0,
“failed”: 5,
“stopped”: 0,
“passed”: 2,
“errored”: 0,
“total”: 7
}
}
Note the hierarchical nature of the ARN. The ARN type is “job” and the job ID is appended to the Run ARN: arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066
Step 3: Retrieve the Status and Result Counters for all Suites within a Job:
Use the list-suites CLI command to list all Suites within a job along with suite-level status and result counters.
[ec2-user]$ aws devicefarm list-suites –arn arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066
Output (partial)
“suites”: [
{
“status”: “COMPLETED”,
“name”: “Setup Suite”,
“created”: 1443331656.183,
“result”: “PASSED”,
“message”: “Successful test lifecycle of Setup Test”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000”,
“counters”: {
“skipped”: 0,
“warned”: 0,
“failed”: 0,
“stopped”: 0,
“passed”: 1,
“errored”: 0,
“total”: 1
}
},
{
“status”: “COMPLETED”,
“name”: “STARmeter”,
“created”: 1443331656.221,
“result”: “PASSED”,
“message”: “Given I see the text “In Theaters” passed”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00001”,
“counters”: {
“skipped”: 0,
“warned”: 0,
“failed”: 0,
“stopped”: 0,
“passed”: 1,
“errored”: 0,
“total”: 1
}
}
The returned ARN type is “suite” and the suite ID is appended to the end: arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006
Step 4: Retrieving the Status and Result Counters for all Tests within a Suite
Use the list-tests CLI command to list all tests within a suite along with test-level status and result counters.
[ec2-user]$ aws devicefarm list-tests –arn arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006
Output:
{
“tests”: [
{
“status”: “COMPLETED”,
“name”: “Teardown Test”,
“created”: 1443331656.424,
“result”: “PASSED”,
“message”: “Successful test lifecycle of Teardown Test”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000”,
“counters”: {
“skipped”: 0,
“warned”: 0,
“failed”: 0,
“stopped”: 0,
“passed”: 1,
“errored”: 0,
“total”: 1
}
}
]
}
The returned ARN type is “test” and the Test ID is appended to the end of the ARN“: “arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000
Step 5: Retrieving the Artifacts for a Run, Job, Suite or Test
Using the list-artifacts CLI command, you can retrieve all of the artifacts associated with any of the levels of the run by specifying the corresponding ARN. For example:
- List artifacts for the run by specifying the run ARN:
[ec2-user]$ aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917 –type FILE
- List artifacts for the job by specifying the job ARN:
[ec2-user]$ aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066 –type FILE
- List artifacts for the suite by specifying the suite ARN:
[ec2-user]$ aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000 –type FILE
- List artifacts for the individual test by specifying the test ARN:
[ec2-user]$ aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000 –type FILE
Note that there are three types of artifacts available: FILE, LOG and SCREENSHOT.
The output of the list-artifacts CLI command will include a pre-signed S3 URL that you can use to securely download the artifact:
{
“artifacts”: [
{
“url”: “https://prod-us-west-2-results.s3-us-west-2.amazonaws.com/705582597265/2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000/00000/3d811922-64d2-4d0c-8195-ffb715300836.logcat?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1446165402&Signature=D02sc66m01XXYtSJV0hVGK4l9aA%3D“,
“extension”: “logcat”,
“name”: “Logcat”,
“arn”: “arn:aws:devicefarm:us-west-2:705582597265:artifact:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000/00000/00000”
}
]
}
Conclusion
In this two-part series we have demonstrated the use of the AWS CLI to submit a Calabash-Android test script to Device Farm for testing against real mobile devices. We have also demonstrated the use of the CLI to retrieve the status of the test Run as well as to retrieve the reports and artifacts for your test Run. If you have questions, please let us know in the comments section or in our developer forum.