frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 21:14:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add a script to run iOS test app on AWS Device Farm (#110202)

Browse Source 
This adds a script to test PyTorch on actual iOS devices on AWS Device Farm. The test could take quite a long time pending for the devices to become available, so the steps are done manually and documented in `ios/TestApp/README.md`.

### Testing

1. TestApp itself runs fine on my local iPhone 13 and on [device farm](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs/d2653ca8-8ee2-44dd-b15e-0402f9ab0aca).  I can see the benchmark results output at the console log.
```
BUILD_LITE_INTERPRETER=1 USE_PYTORCH_METAL=1 USE_COREML_DELEGATE=1 IOS_PLATFORM=OS IOS_ARCH=arm64 ./scripts/build_ios.sh

pushd ios/TestApp/benchmark
ruby setup.rb --lite 1 -t 9HKVT38N77 --benchmark
popd

ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "OS"
```

2. Trying to run TestAppTests https://github.com/pytorch/pytorch/blob/main/ios/TestApp/TestAppTests/TestLiteInterpreter.mm on my local iPhone ends up with this error `Logic Testing Unavailable. Logic Testing on iOS devices is not supported. You can run logic tests on the Simulator`.  I update the xcode project to reuse TestApp as the host application.
```
ruby setup.rb --lite 1 -t 9HKVT38N77
```

3.. Trying [another round of testing on device farm](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs/18dbd69d-8608-46d8-a868-bd05b69375db)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/110202
Approved by: https://github.com/kit1980
This commit is contained in:
Huy Do
2023-10-06 08:23:13 +00:00
committed by PyTorch MergeBot
parent 7d98549ca9
commit 65afa760a6
7 changed files with 321 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
.github/workflows/_ios-build-test.yml vendored
View File

@ -130,28 +130,16 @@ jobs:
export TCLLIBPATH="/usr/local/lib"
${CONDA_RUN} scripts/build_ios.sh
- name: Build TestApp
if: matrix.ios_platform == 'SIMULATOR'
timeout-minutes: 15
run: |
# Run the ruby build script
if ! [ -x "$(command -v xcodebuild)" ]; then
echo 'Error: xcodebuild is not installed.'
exit 1
fi
ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "${IOS_PLATFORM}"
- name: Run simulator tests
if: matrix.ios_platform == 'SIMULATOR'
- name: Prepare the test models
shell: bash
working-directory: ${{ github.workspace }}/ios/TestApp/benchmark
run: |
set -eux
# shellcheck disable=SC1091
# Use the pytorch nightly build to generate models
${CONDA_RUN} pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
# Generate models for differnet backends
pushd "${GITHUB_WORKSPACE}/ios/TestApp/benchmark"
# Generate models for different backends
mkdir -p ../models
# NB: Both of the following scripts only export models with lite interpreter
if [ "${USE_COREML_DELEGATE}" == 1 ]; then
@ -172,9 +160,28 @@ jobs:
echo "Setting up the TestApp for Full JIT"
ruby setup.rb
fi
popd
pushd "${GITHUB_WORKSPACE}/ios/TestApp"
- name: Build TestApp
if: matrix.ios_platform == 'SIMULATOR'
timeout-minutes: 15
shell: bash
run: |
set -eux
# Run the ruby build script
if ! [ -x "$(command -v xcodebuild)" ]; then
echo 'Error: xcodebuild is not installed.'
exit 1
fi
ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "${IOS_PLATFORM}"
- name: Run simulator tests
if: matrix.ios_platform == 'SIMULATOR'
shell: bash
working-directory: ${{ github.workspace }}/ios/TestApp
run: |
set -eux
# Instruments -s -devices
if [ "${BUILD_LITE_INTERPRETER}" == 1 ]; then
if [ "${USE_COREML_DELEGATE}" == 1 ]; then
@ -185,7 +192,6 @@ jobs:
else
bundle exec fastlane scan --only_testing TestAppTests/TestAppTests/testFullJIT
fi
popd
- name: Dump simulator tests on failure
if: failure() && matrix.ios_platform == 'SIMULATOR'

82
ios/TestApp/README.md
View File

@ -44,6 +44,88 @@ python test/mobile/model_test/gen_test_model.py ios-test
cd ios/TestApp/benchmark; python coreml_backend.py
```
## Run test on AWS Device Farm
The test app and its test suite could also be run on actual devices via
AWS Device Farm.
1. The following steps could only be done on MacOS with Xcode installed.
I'm using Xcode 15.0 on MacOS M1 arm64
2. Checkout PyTorch repo including all submodules
3. Build PyTorch for iOS devices, not for simulator
```
export BUILD_LITE_INTERPRETER=1
export USE_PYTORCH_METAL=1
export USE_COREML_DELEGATE=1
export IOS_PLATFORM=OS
export IOS_ARCH=arm64
./scripts/build_ios.sh
```
4. Build the test app locally
```
# Use the pytorch nightly build to generate models
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
# Generate models for differnet backends
pushd ios/TestApp/benchmark
mkdir -p ../models
# This requires numpy==1.23.1
python coreml_backend.py
# NB: Also need to set the team ID with -t if you are running this locally. This
# command setups an app that could be used to launch TestAppTests on device. On
# the other hand, adding the --benchmark flag to build the one that runs benchmark
# instead.
ruby setup.rb --lite 1
popd
# Build the TestApp and its TestAppTests
ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "OS"
```
5. Prepare the artifacts
https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest.html
```
export DEST_DIR="Payload"
pushd ios/TestApp/build/Release-iphoneos
mkdir "${DEST_DIR}"
cp -r TestApp.app "${DEST_DIR}"
# TestApp.ipa is just a zip file with a payload subdirectory
zip -vr TestApp.ipa "${DEST_DIR}"
pushd TestApp.app/PlugIns
# Also zip the TestAppTests.xctest test suite
zip -vr TestAppTests.xctest.zip TestAppTests.xctest
popd
cp TestApp.app/PlugIns/TestAppTests.xctest.zip .
popd
```
6. Upload the artifacts to AWS Device Farm and run the tests
```
export PYTORCH_ARN="arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0"
pushd ios/TestApp
# AWS Device Farm is only available on us-west-2
AWS_DEFAULT_REGION=us-west-2 python run_on_aws_devicefarm.py \
--project-arn "${PYTORCH_ARN}" \
--app-file build/Release-iphoneos/TestApp.ipa \
--xctest-file build/Release-iphoneos/TestAppTests.xctest.zip \
--name-prefix PyTorch
popd
```
7. The script will continue polling for the outcome. A visual output of
the test results could be view on AWS Device Farm console for [PyTorch project](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs)
## Debug Test Failures
Make sure all models are generated. See https://github.com/pytorch/pytorch/tree/master/test/mobile/model_test to learn more.

47
ios/TestApp/TestApp.xcodeproj/project.pbxproj
View File

@ -151,12 +151,13 @@
attributes = {
LastUpgradeCheck = 1030;
TargetAttributes = {
A06D4CAF232F0DB200763E16 = {
CreatedOnToolsVersion = 10.3;
};
A0EA3AFE237FCB08007CEA34 = {
CreatedOnToolsVersion = 11.2.1;
};
A06D4CAF232F0DB200763E16 = {
CreatedOnToolsVersion = 10.3;
};
A0EA3AFE237FCB08007CEA34 = {
CreatedOnToolsVersion = 11.2.1;
TestTargetID = A06D4CAF232F0DB200763E16;
};
};
};
buildConfigurationList = A06D4CAB232F0DB200763E16 /* Build configuration list for PBXProject "TestApp" */;
@ -289,8 +290,8 @@
GCC_NO_COMMON_BLOCKS = YES;
GCC_OPTIMIZATION_LEVEL = 0;
GCC_PREPROCESSOR_DEFINITIONS = (
"DEBUG=1",
"$(inherited)",
"DEBUG=1",
"$(inherited)",
);
GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
@ -363,13 +364,13 @@
isa = XCBuildConfiguration;
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
CODE_SIGN_STYLE = Manual;
CODE_SIGN_STYLE = Automatic;
DEVELOPMENT_TEAM = "";
ENABLE_BITCODE = NO;
INFOPLIST_FILE = TestApp/Info.plist;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"$(inherited)",
"@executable_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestApp;
PRODUCT_NAME = "$(TARGET_NAME)";
@ -382,13 +383,13 @@
isa = XCBuildConfiguration;
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
CODE_SIGN_STYLE = Manual;
CODE_SIGN_STYLE = Automatic;
DEVELOPMENT_TEAM = "";
ENABLE_BITCODE = NO;
INFOPLIST_FILE = TestApp/Info.plist;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"$(inherited)",
"@executable_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestApp;
PRODUCT_NAME = "$(TARGET_NAME)";
@ -400,38 +401,42 @@
A0EA3B06237FCB08007CEA34 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
CODE_SIGN_IDENTITY = "iPhone Developer";
CODE_SIGN_STYLE = Manual;
CODE_SIGN_STYLE = Automatic;
DEVELOPMENT_TEAM = "";
INFOPLIST_FILE = TestAppTests/Info.plist;
IPHONEOS_DEPLOYMENT_TARGET = 12.4;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestAppTests;
PRODUCT_NAME = "$(TARGET_NAME)";
PROVISIONING_PROFILE_SPECIFIER = "";
"PROVISIONING_PROFILE_SPECIFIER[sdk=macosx*]" = "";
TARGETED_DEVICE_FAMILY = "1,2";
TEST_HOST = "$(BUILT_PRODUCTS_DIR)/TestApp.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/TestApp";
};
name = Debug;
};
A0EA3B07237FCB08007CEA34 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
CODE_SIGN_STYLE = Automatic;
INFOPLIST_FILE = TestAppTests/Info.plist;
IPHONEOS_DEPLOYMENT_TARGET = 12.4;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = com.pytorch.ios.TestAppTests;
PRODUCT_NAME = "$(TARGET_NAME)";
TARGETED_DEVICE_FAMILY = "1,2";
TEST_HOST = "$(BUILT_PRODUCTS_DIR)/TestApp.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/TestApp";
};
name = Release;
};

4
ios/TestApp/TestApp/ViewController.mm
View File

@ -31,10 +31,12 @@
NSLog(@"Parse config.json failed!");
return;
}
// NB: When running tests on device, we need an empty app to launch the tests
#ifdef RUN_BENCHMARK
[Benchmark setup:config];
[self runBenchmark];
#endif
#endif
}
#ifdef BUILD_LITE_INTERPRETER

14
ios/TestApp/benchmark/setup.rb
View File

@ -11,6 +11,9 @@ option_parser = OptionParser.new do |opts|
opts.on('-l', '--lite ', 'use lite interpreter') { |value|
options[:lite] = value
}
opts.on('-b', '--benchmark', 'build app to run benchmark') { |value|
options[:benchmark] = value
}
end.parse!
puts options.inspect
@ -26,6 +29,7 @@ end
puts "Setting up TestApp.xcodeproj..."
project = Xcodeproj::Project.open(xcodeproj_path)
targets = project.targets
test_target = targets.last
header_search_path = ['$(inherited)', "#{install_path}/include"]
libraries_search_path = ['$(inherited)', "#{install_path}/lib"]
other_linker_flags = ['$(inherited)', "-all_load"]
@ -41,9 +45,12 @@ targets.each do |target|
else
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] = ['$(inherited)']
end
if (options[:benchmark])
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'].append("RUN_BENCHMARK")
end
dev_team_id = options[:team_id]
if dev_team_id
config.build_settings['DEVELOPMENT_TEAM'] = dev_team_id
config.build_settings['DEVELOPMENT_TEAM'] = dev_team_id
end
end
end
@ -107,6 +114,11 @@ puts "Linking static libraries..."
libs = ['libc10.a', 'libclog.a', 'libpthreadpool.a', 'libXNNPACK.a', 'libeigen_blas.a', 'libcpuinfo.a', 'libpytorch_qnnpack.a', 'libtorch_cpu.a', 'libtorch.a']
frameworks = ['CoreML', 'Metal', 'MetalPerformanceShaders', 'Accelerate', 'UIKit']
targets.each do |target|
# NB: All these libraries and frameworks have already been linked by TestApp, adding them
# again onto the test target will cause the app to crash on actual devices
if (target == test_target)
next
end
target.frameworks_build_phases.clear
for lib in libs do
path = "#{install_path}/lib/#{lib}"

172
ios/TestApp/run_on_aws_devicefarm.py Executable file
View File

@ -0,0 +1,172 @@
#!/usr/bin/env python3
import datetime
import os
import random
import string
import sys
import time
import warnings
from typing import Any
import boto3
import requests
POLLING_DELAY_IN_SECOND = 5
MAX_UPLOAD_WAIT_IN_SECOND = 600
# NB: This is the curated top devices from AWS. We could create our own device
# pool if we want to
DEFAULT_DEVICE_POOL_ARN = (
"arn:aws:devicefarm:us-west-2::devicepool:082d10e5-d7d7-48a5-ba5c-b33d66efa1f5"
)
def parse_args() -> Any:
from argparse import ArgumentParser
parser = ArgumentParser("Run iOS tests on AWS Device Farm")
parser.add_argument(
"--project-arn", type=str, required=True, help="the ARN of the project on AWS"
)
parser.add_argument(
"--app-file", type=str, required=True, help="the iOS ipa app archive"
)
parser.add_argument(
"--xctest-file",
type=str,
required=True,
help="the XCTest suite to run",
)
parser.add_argument(
"--name-prefix",
type=str,
required=True,
help="the name prefix of this test run",
)
parser.add_argument(
"--device-pool-arn",
type=str,
default=DEFAULT_DEVICE_POOL_ARN,
help="the name of the device pool to test on",
)
return parser.parse_args()
def upload_file(
client: Any,
project_arn: str,
prefix: str,
filename: str,
filetype: str,
mime: str = "application/octet-stream",
):
"""
Upload the app file and XCTest suite to AWS
"""
r = client.create_upload(
projectArn=project_arn,
name=f"{prefix}_{os.path.basename(filename)}",
type=filetype,
contentType=mime,
)
upload_name = r["upload"]["name"]
upload_arn = r["upload"]["arn"]
upload_url = r["upload"]["url"]
with open(filename, "rb") as file_stream:
print(f"Uploading {filename} to Device Farm as {upload_name}...")
r = requests.put(upload_url, data=file_stream, headers={"content-type": mime})
if not r.ok:
raise Exception(f"Couldn't upload {filename}: {r.reason}")
start_time = datetime.datetime.now()
# Polling AWS till the uploaded file is ready
while True:
waiting_time = datetime.datetime.now() - start_time
if waiting_time > datetime.timedelta(seconds=MAX_UPLOAD_WAIT_IN_SECOND):
raise Exception(
f"Uploading {filename} is taking longer than {MAX_WAIT_IN_SECOND} seconds, terminating..."
)
r = client.get_upload(arn=upload_arn)
status = r["upload"].get("status", "")
print(f"{filename} is in state {status} after {waiting_time}")
if status == "FAILED":
raise Exception(f"Couldn't upload {filename}: {r}")
if status == "SUCCEEDED":
break
time.sleep(POLLING_DELAY_IN_SECOND)
return upload_arn
def main() -> None:
args = parse_args()
client = boto3.client("devicefarm")
unique_prefix = f"{args.name_prefix}-{datetime.date.today().isoformat()}-{''.join(random.sample(string.ascii_letters, 8))}"
# Upload the test app
appfile_arn = upload_file(
client=client,
project_arn=args.project_arn,
prefix=unique_prefix,
filename=args.app_file,
filetype="IOS_APP",
)
print(f"Uploaded app: {appfile_arn}")
# Upload the XCTest suite
xctest_arn = upload_file(
client=client,
project_arn=args.project_arn,
prefix=unique_prefix,
filename=args.xctest_file,
filetype="XCTEST_TEST_PACKAGE",
)
print(f"Uploaded XCTest: {xctest_arn}")
# Schedule the test
r = client.schedule_run(
projectArn=args.project_arn,
name=unique_prefix,
appArn=appfile_arn,
devicePoolArn=args.device_pool_arn,
test={"type": "XCTEST", "testPackageArn": xctest_arn},
)
run_arn = r["run"]["arn"]
start_time = datetime.datetime.now()
print(f"Run {unique_prefix} is scheduled as {run_arn}:")
state = "UNKNOWN"
result = ""
try:
while True:
r = client.get_run(arn=run_arn)
state = r["run"]["status"]
if state == "COMPLETED":
result = r["run"]["result"]
break
waiting_time = datetime.datetime.now() - start_time
print(
f"Run {unique_prefix} in state {state} after {datetime.datetime.now() - start_time}"
)
time.sleep(30)
except Exception as error:
warnings.warn(f"Failed to run {unique_prefix}: {error}")
sys.exit(1)
if not result or result == "FAILED":
print(f"Run {unique_prefix} failed, exiting...")
sys.exit(1)
if __name__ == "__main__":
main()

3
scripts/xcode_build.rb
View File

@ -73,5 +73,4 @@ else
raise "unsupported platform #{options[:platform]}"
end
# run xcodebuild
exec "xcodebuild clean build -project #{xcodeproj_path} -target #{target.name} -sdk #{sdk} -configuration Release -arch #{arch}"
exec "xcodebuild clean build -project #{xcodeproj_path} -alltargets -sdk #{sdk} -configuration Release -arch #{arch}"