Replay Random XCTest Failures on a Cloud Mac

A nightly CI job occasionally reports an XCTest failure that cannot be reproduced locally: inputs come from a random factory, tests run in a random order, and the pipeline automatically retries after a failure. The second run turns green, but by then there is no way to determine which data caused the first failure or which tests ran before it. The solution is not to add more retries, but to turn each run into an experiment that can be replayed in full.

Define the replay contract first

Randomized tests are affected by at least three categories of variables: random business data, test execution order, and state left behind by the simulator and process. Recording only SEED=42 is not enough. After changes to the commit, test inventory, or locale, the same seed may take a different execution path.

At the start of each job, generate a run manifest and require all later steps to read from that manifest. At a minimum, preserve the following fields:

“The same test passed on the next run” is not a reproduction. You can determine whether the defect still exists only when the inputs, order, and environment are all identical.

Archive the run manifest with the test artifacts. Do not leave it only in console logs that may be truncated.

Give random data a stable seed

Swift’s system random source is appropriate for production code, but it cannot replay values from a specified seed. Inject the random number generator into the data factory so tests can use a seedable implementation while production code continues to use the system implementation.

struct SplitMix64: RandomNumberGenerator {
    private var state: UInt64

    init(seed: UInt64) {
        state = seed
    }

    mutating func next() -> UInt64 {
        state &+= 0x9E3779B97F4A7C15
        var value = state
        value = (value ^ (value >> 30)) &* 0xBF58476D1CE4E5B9
        value = (value ^ (value >> 27)) &* 0x94D049BB133111EB
        return value ^ (value >> 31)
    }
}

func makeQuantity<R: RandomNumberGenerator>(using rng: inout R) -> Int {
    Int.random(in: 1...20, using: &rng)
}

The test entry point should read the seed from an environment variable and fail immediately when it is missing instead of silently falling back to the system random source:

let rawSeed = ProcessInfo.processInfo.environment["TEST_SEED"]
let seed = rawSeed.flatMap(UInt64.init)
XCTAssertNotNil(seed, "TEST_SEED must be provided")

Every random factory in the same job should follow an explicit derivation rule. For example, derive a sub-seed from the primary seed and a stable test identifier. This prevents one test from changing the data sequence for the entire suite merely by consuming one additional random value.

Pin the test inventory and execution order

Test order should not depend on filesystem traversal order. First, maintain a text manifest with one fully qualified identifier per line, such as AppTests/LoginTests/testExpiredSession. Then sort the tests deterministically using a digest of the seed and identifier:

import hashlib
import os

seed = os.environ["TEST_SEED"]
with open("ci/tests.txt", encoding="utf-8") as source:
    tests = [line.strip() for line in source if line.strip()]

def key(name):
    value = f"{seed}:{name}".encode()
    return hashlib.sha256(value).hexdigest()

for test in sorted(tests, key=key):
    print(test)

This provides coverage across different orders while allowing any node to reconstruct the same list. Before execution, save the sorted output as ordered-tests.txt, then build the xcodebuild arguments one test at a time:

set -euo pipefail

mkdir -p artifacts
python3 ci/order_tests.py > artifacts/ordered-tests.txt

args=()
while IFS= read -r test_id; do
  args+=("-only-testing:${test_id}")
done < artifacts/ordered-tests.txt

xcodebuild test \
  -workspace App.xcworkspace \
  -scheme App \
  -destination "id=${SIMULATOR_UDID}" \
  -resultBundlePath "artifacts/test-${TEST_SEED}.xcresult" \
  "${args[@]}"

Do not enable a second random-ordering mechanism that cannot be recorded. Otherwise, the order in the manifest and the order actually executed will diverge again.

Isolate simulator state without erasing evidence

Use a dedicated simulator for replays so manual debugging and CI do not share a device. You may shut down and erase the specified device before testing begins, but first confirm that its UDID belongs to the current job. Never perform indiscriminate cleanup on shared devices.

xcrun simctl shutdown "$SIMULATOR_UDID" 2>/dev/null || true
xcrun simctl erase "$SIMULATOR_UDID"
xcrun simctl boot "$SIMULATOR_UDID"
xcrun simctl bootstatus "$SIMULATOR_UDID" -b

The test entry point should also set the locale, language, time zone, and launch arguments explicitly. If network responses, the current time, or UUIDs affect assertions, inject fixed implementations through protocols instead of hoping the simulator happens to be in the same state.

If an earlier test appears to be contaminating the failing test, do not begin by moving every test into a fresh simulator. First replay the original order and confirm the failure. Then use binary search to shorten the preceding test list and identify the smallest contamination chain. This distinguishes failures triggered by random data from state leaked by earlier tests.

Preserve the complete first-failure state

A failed job should retain at least the xcresult, ordered test list, run manifest, standard output, and code commit. It should also record xcodebuild -version, sw_vers, and the simulator runtime. For sensitive environment variables, record only the variable name and whether it was present, never its actual value.

Automatic retries can help classify the problem, but only after the first failure has been archived. Results should be divided into three categories:

Stable failure under the same contract

If the test consistently fails with the same commit, seed, order, and simulator state, investigate deterministic business logic and assertions first.

Failure after changing the order

If the data remains fixed but changing the preceding tests changes the result, focus on singletons, notification observers, databases, caches, and global configuration that was not restored.

Continued drift under the same contract

Check again whether asynchronous work has actually finished, whether the code reads system time, whether network responses remain unfixed, and whether the tests call any random source that was not injected.

The ultimate goal is not to make a red build disappear through retries. It is to attach an executable replay command to every failure. Once that is in place, an intermittent issue stops being a probabilistic event and becomes a set of inputs that can be minimized, compared, and fixed.

Frequently asked questions