XCTest UI Testing Hints and Tips

As I’m writing this, I’m watching fastlane’s snapshot tool generate 900 snapshots (3 app variants, 6 devices, 5 screenshots per device, 10 languages) slowly in the background while I work on other things.  If you haven’t looked at fastlane yet, you should (https://github.com/fastlane/fastlane), it’s an incredible timesaver in distributing beta builds, managing metadata and screenshots for the app store, and much more.

Fastlane’s snapshot tool was recently modified to use the new XCTest UI Testing (replacing the older Instruments Javascript based UI Testing).  In the long run, this is a great thing.  The new UI Testing has a lot of promise, but it it’s still not documented and has a number of issues that can make it frustrating to use. I wrote this blog based on some issues that I struggled with and resolved in creating the screenshot code for snapshot.  I’m hoping others will find this useful.

In my own research to resolve problems, I found the following sites and articles extremely helpful in learning more about the UI Testing framework:

masilotti.com – Multiple articles and basic documentation of UI Test classes

UI Testing Gotchas at Big Nerd Ranch

UI Testing in Xcode at Infinity Learning

XCode 7 UI Testing, a first look at mokacoding

I would recommend reading these sources to get a better overall understanding, since this blog focuses on some specific issues and solutions rather than documenting the basics of UI Testing.

Adding Test Files To Your App

The first thing I needed for my screenshots was to be able to add a number of test files, including photos and videos that needed to be in the Photos app on the simulator.  In the previous version of snapshot, I could copy the files as part of the Snapfile script, but that isn’t possible now.  So, for this version, I added a Run Script action in the UI Test target‘s Build Phases that copies the files into the app itself:

cp -R "${HOME}/Dropbox/LumaTouch/Product Assets/PinnacleStudio/Screenshot-Projects/SnapshotFiles/" "${BUILT_PRODUCTS_DIR}/PinnacleStudio.app/SnapshotFiles"

When the app is built for the simulator, it contains code that looks for the SnapshotFiles folder, and if found, it copies some of its data into the App’s documents folder, and for photos and videos, calls PHAssetLibrary to add the photos and videos to the Photos app.  The code looks for the SnapshotFiles folder like this:

NSURL *folderUrl = [[NSBundle mainBundle] bundleURL];
folderUrl = [folderUrl URLByAppendingPathComponent:@"SnapshotFiles"];
if ([[NSFileManager defaultManager] fileExistsAtPath:folderUrl.path])
{
  ...
}

Accessibility To Identify Elements

The best way to reference UI elements in UI Testing is through accessibility values.  Elements like StaticText (UILabel) elements can automatically be referenced by the text value of the label, which effectively becomes the accessibilityLabel value of the element like:

let labelElement = app.staticTexts["Phone Number"]

However, this is not very useful if your app is localized and the label changes based on the language.  I prefer to set the accessibilityIdentifier for each UI element in the app and use that to reference elements when testing.  The accessibilityIdentifier is an internal identifier and is not localized.  You can set this in Interface Builder, or in code, like this:

phoneLabel.accessibilityIdentifier = "phone-label"

Then, to access in your test, simply use the accessibilityIdentifier, like this:

XCTAssert(app.labels["phone-label"].title == "Phone Number")

Waiting for UI Elements to Appear or Disappear

One of the issues with adding my test files for screenshots is that it may take some time for the app to copy the files and add the media to Photos.  Originally I put a sleep(30) as a guess in my test code, but there is a better way.  I added code in the app to create a hidden button once it was finished setting up the test files.  I created a number of helper functions (code for these helper functions is at the bottom of this post) to wait for elements to exist, or be hittable, or to disappear again.  These were simply added as extensions to XCTestCase in my test code.

  • waitForHittable(element: XCUIElement, waitSeconds: Double) – Wait for an element to exist and be hittable (visible and onscreen)
  • waitForNotHittable(element: XCUIElement, waitSeconds: Double) – Wait until an element is no longer hittable (might still exist, but is either not visible or not onscreen)
  • waitForExists(element: XCUIElement, waitSeconds: Double) – Wait for an element to exist (does not need to be visible or onscreen)

All of these functions take advantage of the waitForExpectationsWithTimeout method in XCTestCase and make it easy to wait for certain things to happen within an allotted time.  So, in my case, I have the following code in my test to wait for the test files to be finished:

waitForExists(app.otherElements["snapshotReady"], waitSeconds: 60)

As soon as the “snapshotReady” view is created and added as a subview in my app’s code (it’s hidden, and created with a rect that’s offscreen, so it’s never seen on the UI), the test continues.  If the processing of the snapshot test files fails, then after 60 seconds, the test will fail.

In another point in my test code, I need to start a render process, which brings up a progress view with a cancel button.  In my old test code, I just put in a long sleep() to make sure it could finish, but with the wait helper functions, I can wait until the progress view appears, then disappears again to continue:

app.buttons["edit-render"].tap()
waitForHittable(app.buttons["export-process-cancel"], waitSeconds: 5)
waitForNotHittable(app.buttons["export-process-cancel"], waitSeconds: 320)

Debugging UI Elements When Testing

The best way to see what is going on is to set a breakpoint in your test code, then output a complete list of the UI Elements that are onscreen.  You can do this in lldb with the following command:

po print(app.debugDescription)

This assumes that app is assigned to XCUIApplication(), which is normally the case. This will print a well-formatted description of the app element and all of its descendants (if you don’t use print(), you’ll get a very ugly output).  You can use this on any XCUIElement(), for example:

po print(app.buttons["edit-render"])

Shortest Path To An Element

When you’re recording tests, XCode might record something like this when tapping on a tableView cell (assuming you’ve set accessibility identifiers or labels on the table and cells):

app.tables["address-table"].cells["address-3"].tap()

When you’re writing your own test code, if you know that the accessibilityIdentifier for the cells are unique in the UI, you can shortcut this with:

app.cells["address-3"].tap()

This is because the cells call simply queries all descendants that are Cell elements, and as long as it finds only one that matches the identifier, that will be the element returned.

Duplicate Identifiers in TableView Cells

When running one of my tests, I kept getting a failure Multiple Matches Found when I tried to tap() a cell in my table.  After a great deal of experimenting I figured out the problem. I was setting the accessibilityIdentifier in cellForIndexPath, but I was never clearing that identifier, and somehow the cached cell, was being found in the query. To solve this problem I added code to clear the accessibilityIdentifier in tableView:didEndDisplayingCell.

Failed To Find Matching Element

A number of times when recording, I got the following message added as a comment in my test code:

Failed to find matching element please file bug (bugreport.apple.com) and provide output from Console.app

I tried many different changes of accessibilityIdentifiers and different paths to the element, but nothing helped, so I filed a Radar with Apple.  It was a duplicate, so obviously I wasn’t the only one.  In playing with XCode 7.2 beta 2, it seems to be better about this, but I was still having a problem when calling tap() on cells in some tables.  It wouldn’t fail, but the cell was not being selected.  After a lot of playing around, I finally found the cause of the problem:

Tap Was A Long Press

The table that was failing had a long press gesture recognizer (0.175 seconds) attached (for drag-and-drop), and what was happening is that the tap() call was being picked up as a long press, so the cell wasn’t being selected.  To solve this problem, I changed the tap() to a pressForDuration, like this:

 app.tables["text-album-5-0"].cells["cell-0-1"].pressForDuration(0.05);

How To Do An Arbitrary Touch or Drag

Without documentation, it’s easy to miss how to do an arbitrary touch or drag in the UI (from/to a particular point on the screen).  For many situations it’s not needed, you can just use the pressForDuration:thenDragToElement call on an element, but I had some situations that weren’t that straightforward.

The solution is XCUICoordinate().  You create an XCUICoordinate() object by calling coordinateWithNormalizedOffset() on any XCUIElement (including XCUIApplication).  The CGVector is a value normalized from 0.0 to 1.0 to represent the element’s rect (you can provide values outside of the rect with negative values and values above 1.0).

The XCUICoordinate() object holds a reference to the XCUIElement() it was retrieved from. It is important to note that when you perform an action with XCUICoordinate() it will calculate the screen point at that moment, so if the element the coordinate references has moved since you first grabbed the coordinate, the tap will be relative to the element’s current location.

To perform a drag with an XCUICoordinate, you can call pressForDuration:thenDragToCoordinate.

You can get the second coordinate the same way you got the first one (by creating it from any element), or you can call coordinateWithOffset(offsetVector: CGVector) on an XCUICoordinate to return a new coordinate offset from the first coordinate (the offsetVector in this case is in absolute points rather than a normalized value).  Here is a complete example of using XCUICoordinate to do a drag:

let startCoord = app.tables["text-album-0-1"].coordinateWithNormalizedOffset(CGVector(dx: 0.5, dy: 0.01));

let endCoord = startCoord.coordinateWithOffset(CGVector(dx: 0.0, dy: -100));

startCoord.pressForDuration(0.05, thenDragToCoordinate: endCoord)

This starts the touch in the text-album-0-1 album halfway across the album, and 1% distant from top, then drags 100 points upward.

Conditional Code for iPad and iPhone

I needed some different code depending on whether the test was running on iPhone or iPad, but didn’t want to write separate tests.  You can determine the size class of the app’s window to provide this type of condition:

if (app.windows.elementBoundByIndex(0).horizontalSizeClass == .Compact || app.windows.elementBoundByIndex(0).verticalSizeClass == .Compact)

More To Come

That’s all of my hints and tips for now.  I’ll create a new blog as I find more hints and tips.  I’m re-creating a suite of hundreds of UI Tests done with OCUnit() back in 2011.  I always find examples the best way to learn something, so I’ve also included a complete example of a test doing a screenshot with fastlanes/snapshot.

Screenshot Test Source Code

func testScreenshot01() {
    let app = XCUIApplication()
    
    XCUIDevice().orientation = .LandscapeRight;

    waitForExists(app.otherElements["snapshotReady"], waitSeconds: 60)

    if (app.windows.elementBoundByIndex(0).horizontalSizeClass == .Regular && appl.windows.elementBoundByIndex(0).verticalSizeClass == .Regular)
        {
            // iPad Screenshot

            // Select and tap again to edit the project
            app.tables["proj-table"].cells["proj-cell-3"].tap();
            app.tables["proj-table"].cells["proj-cell-3"].tap();

            // Select the Albums segment, and select album 4
            app.segmentedControls["album-segments"].buttons.elementBoundByIndex(1).tap()
            sleep(1)

            app.tables["text-album-0-1"].cells["cell-0-4"].pressForDuration(0.05)

           // Render and wait for completion
            waitForHittable(app.buttons["edit-render"], waitSeconds: 5)
            app.buttons["edit-render"].tap()
            waitForHittable(app.buttons["export-process-cancel"], waitSeconds: 5)
            waitForNotHittable(app.buttons["export-process-cancel"], waitSeconds: 320)

            
            // Tap on clip 4 in the storyboard
            app.otherElements["storyClip-4"].tap()

            // Open the audio mixer
            app.buttons["comp-mixer"].tap();            
 
            // Select a video clip for the source viewer
            app.collectionViews["thumbnail-album-0-1-4"].cells["cell-0-8"].images["thumbnail"].pressForDuration(0.05)             
        }
        else
        {
            // iPhone Screenshot

            // Drag upward in the project table to make sure project is visible
            let coord = app.tables["proj-table"].coordinateWithNormalizedOffset(CGVector(dx: 0.5, dy: 0.5))

            let endCoord = coord.coordinateWithOffset(CGVector(dx: 0.0, dy: -50))

            coord.pressForDuration(0.05, thenDragToCoordinate: endCoord)

            sleep(1)
            // Select and edit the project
            app.tables["proj-table"].cells["proj-cell-3"].tap();
            app.tables["proj-table"].cells["proj-cell-3"].tap();

            // Tap breadcrumb to go back to Moments/Albums select
            app.buttons["album-breadcrumb"].tap()

            // Tap on Albums
            app.cells["cell-0-1"].pressForDuration(0.05)
            sleep(1)

            // Open album
            app.tables["text-album-0-1"].cells["cell-0-4"].pressForDuration(0.05);

            // Render and wait for completion
            waitForHittable(app.buttons["edit-render"], waitSeconds: 5)
            app.buttons["edit-render"].tap();
            waitForHittable(app.buttons["export-process-cancel"], waitSeconds: 5)
            waitForNotHittable(app.buttons["export-process-cancel"], waitSeconds: 320)

            // Tap on clips to get to clip 4
            app.otherElements["storyClip-1"].tap();
            sleep(1)
            app.otherElements["storyClip-2"].tap();
            sleep(1)
            app.otherElements["storyClip-3"].tap();
            sleep(1)
            app.otherElements["storyClip-4"].tap();
            sleep(1)

            // Switch to timeline only view
            app.buttons["storytime-button"].tap();

            // Tap in timeline ruler to de-select clip
            app.otherElements["Ruler"].coordinateWithNormalizedOffset(CGVector(dx: 0.1, dy: 0.05)).tap()

            // Open toolbox, tap audio mixer, close toolbox
            app.buttons["toolbox-button"].tap()
            app.buttons["comp-mixer"].tap()
            app.buttons["toolbox-button"].tap()
        }

        // Take the fastlane/snapshot 
        snapshot("Shot01")        

        // Get back to projects page
        app.buttons["audiomix-close"].tap()
        app.buttons["album-projects"].tap()
    }

Helper Functions Source Code

extension XCTestCase {
    func waitForHittable(element: XCUIElement, waitSeconds: Double, file: String = __FILE__, line: UInt = __LINE__) {
        let existsPredicate = NSPredicate(format: "hittable == true")
        expectationForPredicate(existsPredicate, evaluatedWithObject: element, handler: nil)        

        waitForExpectationsWithTimeout(waitSeconds) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after \(waitSeconds) seconds."
                self.recordFailureWithDescription(message,
                        inFile: file, atLine: line, expected: true)
            }
        }
    }

    func waitForNotHittable(element: XCUIElement, waitSeconds: Double, file: String = __FILE__, line: UInt = __LINE__) {
        let existsPredicate = NSPredicate(format: "hittable == false")
        expectationForPredicate(existsPredicate, evaluatedWithObject: element, handler: nil)

        waitForExpectationsWithTimeout(waitSeconds) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after \(waitSeconds) seconds."
                self.recordFailureWithDescription(message,
                    inFile: file, atLine: line, expected: true)
            }
        }
    }




    func waitForExists(element: XCUIElement, waitSeconds: Double, file: String = __FILE__, line: UInt = __LINE__) {
        let existsPredicate = NSPredicate(format: "exists == true")
        expectationForPredicate(existsPredicate, evaluatedWithObject: element, handler: nil)

        waitForExpectationsWithTimeout(waitSeconds) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after \(waitSeconds) seconds."
                self.recordFailureWithDescription(message,
                    inFile: file, atLine: line, expected: true)
            }
        }
    }

}

4 thoughts on “XCTest UI Testing Hints and Tips

  1. Thank you so much for the full example. I am just learning UI Testing and your information was very useful.

  2. Pingback: 使用 Xcode 執行 UI 自動化測試 – Part 2 | 馬克的學習筆記

  3. That will auto-insert the file name and line number that the function is called from, so when it logs an error you can tell where in your code the problem is coming from.

Leave a Reply

Your email address will not be published. Required fields are marked *