Add a script to run iOS test app on AWS Device Farm (#110202)

This adds a script to test PyTorch on actual iOS devices on AWS Device Farm. The test could take quite a long time pending for the devices to become available, so the steps are done manually and documented in `ios/TestApp/README.md`.

### Testing

1. TestApp itself runs fine on my local iPhone 13 and on [device farm](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs/d2653ca8-8ee2-44dd-b15e-0402f9ab0aca).  I can see the benchmark results output at the console log.
```
BUILD_LITE_INTERPRETER=1 USE_PYTORCH_METAL=1 USE_COREML_DELEGATE=1 IOS_PLATFORM=OS IOS_ARCH=arm64 ./scripts/build_ios.sh

pushd ios/TestApp/benchmark
ruby setup.rb --lite 1 -t 9HKVT38N77 --benchmark
popd

ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "OS"
```

2. Trying to run TestAppTests https://github.com/pytorch/pytorch/blob/main/ios/TestApp/TestAppTests/TestLiteInterpreter.mm on my local iPhone ends up with this error `Logic Testing Unavailable. Logic Testing on iOS devices is not supported. You can run logic tests on the Simulator`.  I update the xcode project to reuse TestApp as the host application.
```
ruby setup.rb --lite 1 -t 9HKVT38N77
```

3.. Trying [another round of testing on device farm](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs/18dbd69d-8608-46d8-a868-bd05b69375db)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/110202
Approved by: https://github.com/kit1980

.github/workflows/_ios-build-test.yml

@@ -130,28 +130,16 @@ jobs:
           export TCLLIBPATH="/usr/local/lib"
           ${CONDA_RUN} scripts/build_ios.sh
 
-      - name: Build TestApp
+      - name: Prepare the test models
-        if: matrix.ios_platform == 'SIMULATOR'
-        timeout-minutes: 15
-        run: |
-          # Run the ruby build script
-          if ! [ -x "$(command -v xcodebuild)" ]; then
-            echo 'Error: xcodebuild is not installed.'
-            exit 1
-          fi
-          ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "${IOS_PLATFORM}"
-
-      - name: Run simulator tests
-        if: matrix.ios_platform == 'SIMULATOR'
         shell: bash
+        working-directory: ${{ github.workspace }}/ios/TestApp/benchmark
         run: |
           set -eux
           # shellcheck disable=SC1091
           # Use the pytorch nightly build to generate models
           ${CONDA_RUN} pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
 
-          # Generate models for differnet backends
+          # Generate models for different backends
-          pushd "${GITHUB_WORKSPACE}/ios/TestApp/benchmark"
           mkdir -p ../models
           # NB: Both of the following scripts only export models with lite interpreter
           if [ "${USE_COREML_DELEGATE}" == 1 ]; then
@@ -172,9 +160,28 @@ jobs:
             echo "Setting up the TestApp for Full JIT"
             ruby setup.rb
           fi
-          popd
 
-          pushd "${GITHUB_WORKSPACE}/ios/TestApp"
+      - name: Build TestApp
+        if: matrix.ios_platform == 'SIMULATOR'
+        timeout-minutes: 15
+        shell: bash
+        run: |
+          set -eux
+
+          # Run the ruby build script
+          if ! [ -x "$(command -v xcodebuild)" ]; then
+            echo 'Error: xcodebuild is not installed.'
+            exit 1
+          fi
+          ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "${IOS_PLATFORM}"
+
+      - name: Run simulator tests
+        if: matrix.ios_platform == 'SIMULATOR'
+        shell: bash
+        working-directory: ${{ github.workspace }}/ios/TestApp
+        run: |
+          set -eux
+
           # Instruments -s -devices
           if [ "${BUILD_LITE_INTERPRETER}" == 1 ]; then
             if [ "${USE_COREML_DELEGATE}" == 1 ]; then
@@ -185,7 +192,6 @@ jobs:
           else
             bundle exec fastlane scan --only_testing TestAppTests/TestAppTests/testFullJIT
           fi
-          popd
 
       - name: Dump simulator tests on failure
         if: failure() && matrix.ios_platform == 'SIMULATOR'

ios/TestApp/README.md

@@ -44,6 +44,88 @@ python test/mobile/model_test/gen_test_model.py ios-test
 cd ios/TestApp/benchmark; python coreml_backend.py
 ```
 
+## Run test on AWS Device Farm
+The test app and its test suite could also be run on actual devices via
+AWS Device Farm.
+
+1. The following steps could only be done on MacOS with Xcode installed.
+   I'm using Xcode 15.0 on MacOS M1 arm64
+
+2. Checkout PyTorch repo including all submodules
+
+3. Build PyTorch for iOS devices, not for simulator
+```
+export BUILD_LITE_INTERPRETER=1
+export USE_PYTORCH_METAL=1
+export USE_COREML_DELEGATE=1
+export IOS_PLATFORM=OS
+export IOS_ARCH=arm64
+
+./scripts/build_ios.sh
+```
+
+4. Build the test app locally
+```
+# Use the pytorch nightly build to generate models
+pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
+
+# Generate models for differnet backends
+pushd ios/TestApp/benchmark
+mkdir -p ../models
+
+# This requires numpy==1.23.1
+python coreml_backend.py
+
+# NB: Also need to set the team ID with -t if you are running this locally. This
+# command setups an app that could be used to launch TestAppTests on device. On
+# the other hand, adding the --benchmark flag to build the one that runs benchmark
+# instead.
+ruby setup.rb --lite 1
+popd
+
+# Build the TestApp and its TestAppTests
+ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "OS"
+```
+
+5. Prepare the artifacts
+https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest.html
+
+```
+export DEST_DIR="Payload"
+
+pushd ios/TestApp/build/Release-iphoneos
+mkdir "${DEST_DIR}"
+
+cp -r TestApp.app "${DEST_DIR}"
+# TestApp.ipa is just a zip file with a payload subdirectory
+zip -vr TestApp.ipa "${DEST_DIR}"
+
+pushd TestApp.app/PlugIns
+# Also zip the TestAppTests.xctest test suite
+zip -vr TestAppTests.xctest.zip TestAppTests.xctest
+popd
+
+cp TestApp.app/PlugIns/TestAppTests.xctest.zip .
+popd
+```
+
+6. Upload the artifacts to AWS Device Farm and run the tests
+```
+export PYTORCH_ARN="arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0"
+
+pushd ios/TestApp
+# AWS Device Farm is only available on us-west-2
+AWS_DEFAULT_REGION=us-west-2 python run_on_aws_devicefarm.py \
+  --project-arn "${PYTORCH_ARN}" \
+  --app-file build/Release-iphoneos/TestApp.ipa \
+  --xctest-file build/Release-iphoneos/TestAppTests.xctest.zip \
+  --name-prefix PyTorch
+popd
+```
+
+7. The script will continue polling for the outcome. A visual output of
+   the test results could be view on AWS Device Farm console for [PyTorch project](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs)
+
 ## Debug Test Failures
 Make sure all models are generated. See https://github.com/pytorch/pytorch/tree/master/test/mobile/model_test to learn more.
 

ios/TestApp/TestApp.xcodeproj/project.pbxproj

@@ -151,12 +151,13 @@
             attributes = {
                 LastUpgradeCheck = 1030;
                 TargetAttributes = {
-                        A06D4CAF232F0DB200763E16 = {
+                    A06D4CAF232F0DB200763E16 = {
-                                CreatedOnToolsVersion = 10.3;
+                        CreatedOnToolsVersion = 10.3;
-                        };
+                    };
-                        A0EA3AFE237FCB08007CEA34 = {
+                    A0EA3AFE237FCB08007CEA34 = {
-                                CreatedOnToolsVersion = 11.2.1;
+                        CreatedOnToolsVersion = 11.2.1;
-                        };
+                        TestTargetID = A06D4CAF232F0DB200763E16;
+                    };
                 };
             };
             buildConfigurationList = A06D4CAB232F0DB200763E16 /* Build configuration list for PBXProject "TestApp" */;
@@ -289,8 +290,8 @@
                 GCC_NO_COMMON_BLOCKS = YES;
                 GCC_OPTIMIZATION_LEVEL = 0;
                 GCC_PREPROCESSOR_DEFINITIONS = (
-                        "DEBUG=1",
+                    "DEBUG=1",
-                        "$(inherited)",
+                    "$(inherited)",
                 );
                 GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
                 GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
@@ -363,13 +364,13 @@
             isa = XCBuildConfiguration;
             buildSettings = {
                 ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
-                CODE_SIGN_STYLE = Manual;
+                CODE_SIGN_STYLE = Automatic;
                 DEVELOPMENT_TEAM = "";
                 ENABLE_BITCODE = NO;
                 INFOPLIST_FILE = TestApp/Info.plist;
                 LD_RUNPATH_SEARCH_PATHS = (
-                        "$(inherited)",
+                    "$(inherited)",
-                        "@executable_path/Frameworks",
+                    "@executable_path/Frameworks",
                 );
                 PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestApp;
                 PRODUCT_NAME = "$(TARGET_NAME)";
@@ -382,13 +383,13 @@
             isa = XCBuildConfiguration;
             buildSettings = {
                 ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
-                CODE_SIGN_STYLE = Manual;
+                CODE_SIGN_STYLE = Automatic;
                 DEVELOPMENT_TEAM = "";
                 ENABLE_BITCODE = NO;
                 INFOPLIST_FILE = TestApp/Info.plist;
                 LD_RUNPATH_SEARCH_PATHS = (
-                        "$(inherited)",
+                    "$(inherited)",
-                        "@executable_path/Frameworks",
+                    "@executable_path/Frameworks",
                 );
                 PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestApp;
                 PRODUCT_NAME = "$(TARGET_NAME)";
@@ -400,38 +401,42 @@
         A0EA3B06237FCB08007CEA34 /* Debug */ = {
             isa = XCBuildConfiguration;
             buildSettings = {
+                BUNDLE_LOADER = "$(TEST_HOST)";
                 CODE_SIGN_IDENTITY = "iPhone Developer";
-                CODE_SIGN_STYLE = Manual;
+                CODE_SIGN_STYLE = Automatic;
                 DEVELOPMENT_TEAM = "";
                 INFOPLIST_FILE = TestAppTests/Info.plist;
                 IPHONEOS_DEPLOYMENT_TARGET = 12.4;
                 LD_RUNPATH_SEARCH_PATHS = (
-                        "$(inherited)",
+                    "$(inherited)",
-                        "@executable_path/Frameworks",
+                    "@executable_path/Frameworks",
-                        "@loader_path/Frameworks",
+                    "@loader_path/Frameworks",
                 );
                 PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestAppTests;
                 PRODUCT_NAME = "$(TARGET_NAME)";
                 PROVISIONING_PROFILE_SPECIFIER = "";
                 "PROVISIONING_PROFILE_SPECIFIER[sdk=macosx*]" = "";
                 TARGETED_DEVICE_FAMILY = "1,2";
+                TEST_HOST = "$(BUILT_PRODUCTS_DIR)/TestApp.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/TestApp";
             };
             name = Debug;
         };
         A0EA3B07237FCB08007CEA34 /* Release */ = {
             isa = XCBuildConfiguration;
             buildSettings = {
+                BUNDLE_LOADER = "$(TEST_HOST)";
                 CODE_SIGN_STYLE = Automatic;
                 INFOPLIST_FILE = TestAppTests/Info.plist;
                 IPHONEOS_DEPLOYMENT_TARGET = 12.4;
                 LD_RUNPATH_SEARCH_PATHS = (
-                        "$(inherited)",
+                    "$(inherited)",
-                        "@executable_path/Frameworks",
+                    "@executable_path/Frameworks",
-                        "@loader_path/Frameworks",
+                    "@loader_path/Frameworks",
                 );
                 PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestAppTests;
                 PRODUCT_NAME = "$(TARGET_NAME)";
                 TARGETED_DEVICE_FAMILY = "1,2";
+                TEST_HOST = "$(BUILT_PRODUCTS_DIR)/TestApp.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/TestApp";
             };
             name = Release;
         };

ios/TestApp/TestApp/ViewController.mm

@@ -31,10 +31,12 @@
     NSLog(@"Parse config.json failed!");
     return;
   }
-
+// NB: When running tests on device, we need an empty app to launch the tests
+#ifdef RUN_BENCHMARK
   [Benchmark setup:config];
   [self runBenchmark];
 #endif
+#endif
 }
 
 #ifdef BUILD_LITE_INTERPRETER

ios/TestApp/benchmark/setup.rb

@@ -11,6 +11,9 @@ option_parser = OptionParser.new do |opts|
  opts.on('-l', '--lite ', 'use lite interpreter') { |value|
     options[:lite] = value
  }
+ opts.on('-b', '--benchmark', 'build app to run benchmark') { |value|
+    options[:benchmark] = value
+ }
 end.parse!
 puts options.inspect
 
@@ -26,6 +29,7 @@ end
 puts "Setting up TestApp.xcodeproj..."
 project = Xcodeproj::Project.open(xcodeproj_path)
 targets = project.targets
+test_target = targets.last
 header_search_path      = ['$(inherited)', "#{install_path}/include"]
 libraries_search_path   = ['$(inherited)', "#{install_path}/lib"]
 other_linker_flags      = ['$(inherited)', "-all_load"]
@@ -41,9 +45,12 @@ targets.each do |target|
         else
             config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] = ['$(inherited)']
         end
+        if (options[:benchmark])
+            config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'].append("RUN_BENCHMARK")
+        end
         dev_team_id = options[:team_id]
         if dev_team_id
-            config.build_settings['DEVELOPMENT_TEAM']   = dev_team_id
+            config.build_settings['DEVELOPMENT_TEAM'] = dev_team_id
         end
     end
 end
@@ -107,6 +114,11 @@ puts "Linking static libraries..."
 libs = ['libc10.a', 'libclog.a', 'libpthreadpool.a', 'libXNNPACK.a', 'libeigen_blas.a', 'libcpuinfo.a', 'libpytorch_qnnpack.a', 'libtorch_cpu.a', 'libtorch.a']
 frameworks = ['CoreML', 'Metal', 'MetalPerformanceShaders', 'Accelerate', 'UIKit']
 targets.each do |target|
+    # NB: All these libraries and frameworks have already been linked by TestApp, adding them
+    # again onto the test target will cause the app to crash on actual devices
+    if (target == test_target)
+        next
+    end
     target.frameworks_build_phases.clear
     for lib in libs do
         path = "#{install_path}/lib/#{lib}"

ios/TestApp/run_on_aws_devicefarm.py

@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+
+import datetime
+import os
+import random
+import string
+import sys
+import time
+import warnings
+from typing import Any
+
+import boto3
+import requests
+
+POLLING_DELAY_IN_SECOND = 5
+MAX_UPLOAD_WAIT_IN_SECOND = 600
+
+# NB: This is the curated top devices from AWS. We could create our own device
+# pool if we want to
+DEFAULT_DEVICE_POOL_ARN = (
+    "arn:aws:devicefarm:us-west-2::devicepool:082d10e5-d7d7-48a5-ba5c-b33d66efa1f5"
+)
+
+
+def parse_args() -> Any:
+    from argparse import ArgumentParser
+
+    parser = ArgumentParser("Run iOS tests on AWS Device Farm")
+    parser.add_argument(
+        "--project-arn", type=str, required=True, help="the ARN of the project on AWS"
+    )
+    parser.add_argument(
+        "--app-file", type=str, required=True, help="the iOS ipa app archive"
+    )
+    parser.add_argument(
+        "--xctest-file",
+        type=str,
+        required=True,
+        help="the XCTest suite to run",
+    )
+    parser.add_argument(
+        "--name-prefix",
+        type=str,
+        required=True,
+        help="the name prefix of this test run",
+    )
+    parser.add_argument(
+        "--device-pool-arn",
+        type=str,
+        default=DEFAULT_DEVICE_POOL_ARN,
+        help="the name of the device pool to test on",
+    )
+
+    return parser.parse_args()
+
+
+def upload_file(
+    client: Any,
+    project_arn: str,
+    prefix: str,
+    filename: str,
+    filetype: str,
+    mime: str = "application/octet-stream",
+):
+    """
+    Upload the app file and XCTest suite to AWS
+    """
+    r = client.create_upload(
+        projectArn=project_arn,
+        name=f"{prefix}_{os.path.basename(filename)}",
+        type=filetype,
+        contentType=mime,
+    )
+    upload_name = r["upload"]["name"]
+    upload_arn = r["upload"]["arn"]
+    upload_url = r["upload"]["url"]
+
+    with open(filename, "rb") as file_stream:
+        print(f"Uploading {filename} to Device Farm as {upload_name}...")
+        r = requests.put(upload_url, data=file_stream, headers={"content-type": mime})
+        if not r.ok:
+            raise Exception(f"Couldn't upload {filename}: {r.reason}")
+
+    start_time = datetime.datetime.now()
+    # Polling AWS till the uploaded file is ready
+    while True:
+        waiting_time = datetime.datetime.now() - start_time
+        if waiting_time > datetime.timedelta(seconds=MAX_UPLOAD_WAIT_IN_SECOND):
+            raise Exception(
+                f"Uploading {filename} is taking longer than {MAX_WAIT_IN_SECOND} seconds, terminating..."
+            )
+
+        r = client.get_upload(arn=upload_arn)
+        status = r["upload"].get("status", "")
+
+        print(f"{filename} is in state {status} after {waiting_time}")
+
+        if status == "FAILED":
+            raise Exception(f"Couldn't upload {filename}: {r}")
+        if status == "SUCCEEDED":
+            break
+
+        time.sleep(POLLING_DELAY_IN_SECOND)
+
+    return upload_arn
+
+
+def main() -> None:
+    args = parse_args()
+
+    client = boto3.client("devicefarm")
+    unique_prefix = f"{args.name_prefix}-{datetime.date.today().isoformat()}-{''.join(random.sample(string.ascii_letters, 8))}"
+
+    # Upload the test app
+    appfile_arn = upload_file(
+        client=client,
+        project_arn=args.project_arn,
+        prefix=unique_prefix,
+        filename=args.app_file,
+        filetype="IOS_APP",
+    )
+    print(f"Uploaded app: {appfile_arn}")
+    # Upload the XCTest suite
+    xctest_arn = upload_file(
+        client=client,
+        project_arn=args.project_arn,
+        prefix=unique_prefix,
+        filename=args.xctest_file,
+        filetype="XCTEST_TEST_PACKAGE",
+    )
+    print(f"Uploaded XCTest: {xctest_arn}")
+
+    # Schedule the test
+    r = client.schedule_run(
+        projectArn=args.project_arn,
+        name=unique_prefix,
+        appArn=appfile_arn,
+        devicePoolArn=args.device_pool_arn,
+        test={"type": "XCTEST", "testPackageArn": xctest_arn},
+    )
+    run_arn = r["run"]["arn"]
+
+    start_time = datetime.datetime.now()
+    print(f"Run {unique_prefix} is scheduled as {run_arn}:")
+
+    state = "UNKNOWN"
+    result = ""
+    try:
+        while True:
+            r = client.get_run(arn=run_arn)
+            state = r["run"]["status"]
+
+            if state == "COMPLETED":
+                result = r["run"]["result"]
+                break
+
+            waiting_time = datetime.datetime.now() - start_time
+            print(
+                f"Run {unique_prefix} in state {state} after {datetime.datetime.now() - start_time}"
+            )
+            time.sleep(30)
+    except Exception as error:
+        warnings.warn(f"Failed to run {unique_prefix}: {error}")
+        sys.exit(1)
+
+    if not result or result == "FAILED":
+        print(f"Run {unique_prefix} failed, exiting...")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

scripts/xcode_build.rb

@@ -73,5 +73,4 @@ else
     raise "unsupported platform #{options[:platform]}"
 end
 
-# run xcodebuild
+exec "xcodebuild clean build -project #{xcodeproj_path} -alltargets -sdk #{sdk} -configuration Release -arch #{arch}"
-exec "xcodebuild clean build -project #{xcodeproj_path} -target #{target.name} -sdk #{sdk} -configuration Release -arch #{arch}"