Automation scripts for perf testing (#28622)

Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28622

### Summary

As discussed in #28405 , this is the third PR.  The`bootstrap.sh` script is mainly for those who want to do perf on iOS but don't want to touch XCode or any iOS code.  But it does require you have valid iOS dev credentials installed on your machine. (You can easily acquire those stuff from any experienced iOS developers. Takes only 5 mins to setup )

 All you need to do is run

```shell
./bootstrap -t ${TEAM_ID} -p ${PROFILE}
```

The testing app will be automatically installed on your device. The log of the benchmark function will be displayed on the screen.

### Test plan

Don't break any CI jobs unless they're flaky.

Test Plan: Imported from OSS

Differential Revision: D18156178

Pulled By: xta0

fbshipit-source-id: cd7ba8d87bf26db885262888b9d6a5fd072309d1

ios/TestApp/README.md

@@ -14,11 +14,11 @@ This will pull the latest version of `LibTorch` from Cocoapods.
 
 ### Circle CI and Fastlane
 
-The TestApp is currenly being used as a dummy app by Circle CI for nightly jobs. The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. This is where Fastlane came to rescue. [Fastlane](https://fastlane.tools/) is a trendy automation tool for building and managing iOS applications. It also works seamlessly with Circl CI. We are going to leverage the `import_certificate` action which can install developer certificates on CI machines. See `Fastfile` for more details.
+The TestApp is currently being used as a dummy app by Circle CI for nightly jobs. The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. This is where Fastlane came to rescue. [Fastlane](https://fastlane.tools/) is a trendy automation tool for building and managing iOS applications. It also works seamlessly with Circle CI. We are going to leverage the `import_certificate` action, which can install developer certificates on CI machines. See `Fastfile` for more details.
 
 ### Benchmark
 
-The benchmark folder contains two scripts that help you setup the benchmark project. The `setup.rb` does the heavy-lifting jobs of setting up the XCode project, whereas the `trace_model.py` is a Python script that you can tweak to generate your model for benchmarking. Simpy follow the steps below to setup the project
+The benchmark folder contains two scripts that help you setup the benchmark project. The `setup.rb` does the heavy-lifting jobs of setting up the XCode project, whereas the `trace_model.py` is a Python script that you can tweak to generate your model for benchmarking. Simply follow the steps below to setup the project
 
 1. In the PyTorch root directory, run `BUILD_PYTORCH_MOBILE=1 IOS_ARCH=arm64 ./scripts/build_ios.sh` to generate the custom build from **Master** branch
 2. Navigate to the `benchmark` folder, run `python trace_model.py` to get your model generated.
@@ -27,4 +27,23 @@ The benchmark folder contains two scripts that help you setup the benchmark proj
 
 The benchmark code is written in C++, see `benchmark.mm` for more details.
 
+### `bootstrap.sh`
+
+For those who want to do perf testing but don't want touch XCode, `bootstrap.sh` is the right tool for you. It'll automatically build and install the app on your device. That being said, it does require you to have 
+
+1. A valid iOS dev certificate installed on your local machine. 
+2. A valid provisioning profile for code signing
+3. A valid team identifier
+
+To run the script, simply type the command below and make sure your phone is unlocked and connected via USB. 
+
+```shell
+./bootstrap -t ${TEAM_ID} -p ${PROVISIONING_PROFILE}
+```
+
+The benchmark log will be displayed on the screen.
+
+> Note This requires ios-deploy to be installed. Please have a look at [ios-deploy](https://github.com/ios-control/ios-deploy). To quickly install it, use `npm -g i ios-deploy`
+
+
 

ios/TestApp/TestApp/Base.lproj/Main.storyboard

@@ -15,10 +15,28 @@
                     <view key="view" contentMode="scaleToFill" id="8bC-Xf-vdC">
                         <rect key="frame" x="0.0" y="0.0" width="414" height="896"/>
                         <autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
+                        <subviews>
+                            <textView clipsSubviews="YES" multipleTouchEnabled="YES" contentMode="scaleToFill" textAlignment="natural" translatesAutoresizingMaskIntoConstraints="NO" id="3QV-4Z-f2v">
+                                <rect key="frame" x="20" y="108" width="374" height="734"/>
+                                <color key="backgroundColor" systemColor="systemBackgroundColor" cocoaTouchSystemColor="whiteColor"/>
+                                <color key="textColor" systemColor="labelColor" cocoaTouchSystemColor="darkTextColor"/>
+                                <fontDescription key="fontDescription" type="system" pointSize="14"/>
+                                <textInputTraits key="textInputTraits" autocapitalizationType="sentences"/>
+                            </textView>
+                        </subviews>
                         <color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
+                        <constraints>
+                            <constraint firstItem="3QV-4Z-f2v" firstAttribute="leading" secondItem="6Tk-OE-BBY" secondAttribute="leading" constant="20" id="LmH-nY-m0w"/>
+                            <constraint firstItem="3QV-4Z-f2v" firstAttribute="top" secondItem="6Tk-OE-BBY" secondAttribute="top" constant="20" id="Qdn-Ua-oPp"/>
+                            <constraint firstItem="6Tk-OE-BBY" firstAttribute="bottom" secondItem="3QV-4Z-f2v" secondAttribute="bottom" constant="20" id="RGf-vE-yDP"/>
+                            <constraint firstItem="6Tk-OE-BBY" firstAttribute="trailing" secondItem="3QV-4Z-f2v" secondAttribute="trailing" constant="20" id="rb2-GC-nMV"/>
+                        </constraints>
                         <viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
                     </view>
                     <navigationItem key="navigationItem" title="speed_benchmark_torch" id="zRt-2x-Qpi"/>
+                    <connections>
+                        <outlet property="textView" destination="3QV-4Z-f2v" id="Z1X-Bu-UDB"/>
+                    </connections>
                 </viewController>
                 <placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>
             </objects>

ios/TestApp/TestApp/Benchmark.h

@@ -4,7 +4,8 @@ NS_ASSUME_NONNULL_BEGIN
 
 @interface Benchmark : NSObject
 
-+ (void)benchmarkWithModel:(NSString* )modelPath;
++ (BOOL)setup:(NSDictionary* )config;
++ (NSString* )run;
 
 @end
 

ios/TestApp/TestApp/Benchmark.mm

@@ -1,7 +1,6 @@
 #import "Benchmark.h"
 #include <string>
 #include <vector>
-
 #include "ATen/ATen.h"
 #include "caffe2/core/timer.h"
 #include "caffe2/utils/string_utils.h"
@@ -9,27 +8,44 @@
 #include "torch/csrc/jit/import.h"
 #include "torch/script.h"
 
-C10_DEFINE_string(model, "", "The given torch script model to benchmark.");
+static std::string model = "model.pt";
-C10_DEFINE_string(input_dims, "1,3,224,224",
+static std::string input_dims = "1,3,224,224";
-                  "Alternate to input_files, if all inputs are simple "
+static std::string input_type = "float";
-                  "float TensorCPUs, specify the dimension using comma "
+static BOOL print_output = false;
-                  "separated numbers. If multiple input needed, use "
+static int warmup = 10;
-                  "semicolon to separate the dimension of different "
+static int iter = 10;
-                  "tensors.");
-C10_DEFINE_string(input_type, "float", "Input type (uint8_t/float)");
-C10_DEFINE_bool(print_output, false, "Whether to print output with all one input tensor.");
-C10_DEFINE_int(warmup, 0, "The number of iterations to warm up.");
-C10_DEFINE_int(iter, 10, "The number of iterations to run.");
 
 @implementation Benchmark
 
-+ (void)benchmarkWithModel:(NSString*)modelPath {
++ (BOOL)setup:(NSDictionary*)config {
-  FLAGS_model = std::string(modelPath.UTF8String);
+  NSString* modelPath = [[NSBundle mainBundle] pathForResource:@"model" ofType:@"pt"];
-  CAFFE_ENFORCE_GE(FLAGS_input_dims.size(), 0, "Input dims must be specified.");
+  if (![[NSFileManager defaultManager] fileExistsAtPath:modelPath]) {
-  CAFFE_ENFORCE_GE(FLAGS_input_type.size(), 0, "Input type must be specified.");
+    NSLog(@"model.pt doesn't exist!");
+    return NO;
+  }
+  model = std::string(modelPath.UTF8String);
+  input_dims = std::string(((NSString*)config[@"input_dims"]).UTF8String);
+  input_type = std::string(((NSString*)config[@"input_type"]).UTF8String);
+  warmup = ((NSNumber*)config[@"warmup"]).intValue;
+  iter = ((NSNumber*)config[@"iter"]).intValue;
+  print_output = ((NSNumber*)config[@"print_output"]).boolValue;
+  return YES;
+}
 
-  std::vector<std::string> input_dims_list = caffe2::split(';', FLAGS_input_dims);
++ (NSString*)run {
-  std::vector<std::string> input_type_list = caffe2::split(';', FLAGS_input_type);
+  std::vector<std::string> logs;
+#define UI_LOG(fmt, ...)                                          \
+  {                                                               \
+    NSString* log = [NSString stringWithFormat:fmt, __VA_ARGS__]; \
+    NSLog(@"%@", log);                                            \
+    logs.push_back(log.UTF8String);                               \
+  }
+
+  CAFFE_ENFORCE_GE(input_dims.size(), 0, "Input dims must be specified.");
+  CAFFE_ENFORCE_GE(input_type.size(), 0, "Input type must be specified.");
+
+  std::vector<std::string> input_dims_list = caffe2::split(';', input_dims);
+  std::vector<std::string> input_type_list = caffe2::split(';', input_type);
   CAFFE_ENFORCE_EQ(input_dims_list.size(), input_type_list.size(),
                    "Input dims and type should have the same number of items.");
 
@@ -54,33 +70,38 @@ C10_DEFINE_int(iter, 10, "The number of iterations to run.");
     at::globalContext().setQEngine(at::QEngine::QNNPACK);
   }
   torch::autograd::AutoGradMode guard(false);
-  auto module = torch::jit::load(FLAGS_model);
+  auto module = torch::jit::load(model);
 
   at::AutoNonVariableTypeMode non_var_type_mode(true);
   module.eval();
-  if (FLAGS_print_output) {
+  if (print_output) {
     std::cout << module.forward(inputs) << std::endl;
   }
-
+  UI_LOG(@"Start benchmarking...", nil);
-  std::cout << "Starting benchmark." << std::endl;
+  UI_LOG(@"Running warmup runs", nil);
-  std::cout << "Running warmup runs." << std::endl;
+  CAFFE_ENFORCE(warmup >= 0, "Number of warm up runs should be non negative, provided ", warmup,
-  CAFFE_ENFORCE(FLAGS_warmup >= 0, "Number of warm up runs should be non negative, provided ",
+                ".");
-                FLAGS_warmup, ".");
+  for (int i = 0; i < warmup; ++i) {
-  for (int i = 0; i < FLAGS_warmup; ++i) {
     module.forward(inputs);
   }
-
+  UI_LOG(@"Main runs", nil);
-  std::cout << "Main runs." << std::endl;
+  CAFFE_ENFORCE(iter >= 0, "Number of main runs should be non negative, provided ", iter, ".");
-  CAFFE_ENFORCE(FLAGS_iter >= 0, "Number of main runs should be non negative, provided ",
-                FLAGS_iter, ".");
   caffe2::Timer timer;
   auto millis = timer.MilliSeconds();
-  for (int i = 0; i < FLAGS_iter; ++i) {
+  for (int i = 0; i < iter; ++i) {
     module.forward(inputs);
   }
   millis = timer.MilliSeconds();
-  std::cout << "Main run finished. Milliseconds per iter: " << millis / FLAGS_iter
+  UI_LOG(@"Main run finished. Milliseconds per iter: %.3f", millis / iter, nil);
-            << ". Iters per second: " << 1000.0 * FLAGS_iter / millis << std::endl;
+  UI_LOG(@"Iters per second: : %.3f", 1000.0 * iter / millis, nil);
+  UI_LOG(@"Done.", nil);
+
+  NSString* results = @"";
+  for (auto& msg : logs) {
+    results = [results stringByAppendingString:[NSString stringWithUTF8String:msg.c_str()]];
+    results = [results stringByAppendingString:@"\n"];
+  }
+  return results;
 }
 
 @end

ios/TestApp/TestApp/ViewController.mm

@@ -3,6 +3,7 @@
 #import "Benchmark.h"
 
 @interface ViewController ()
+@property(weak, nonatomic) IBOutlet UITextView* textView;
 
 @end
 
@@ -11,12 +12,25 @@
 - (void)viewDidLoad {
   [super viewDidLoad];
 
+  NSError* err;
+  NSData* configData = [NSData
+      dataWithContentsOfFile:[[NSBundle mainBundle] pathForResource:@"config" ofType:@"json"]];
+  NSDictionary* config = [NSJSONSerialization JSONObjectWithData:configData
+                                                         options:NSJSONReadingAllowFragments
+                                                           error:&err];
+  if (err) {
+    NSLog(@"Parse config.json failed!");
+    return;
+  }
+
   dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
-    NSString* modelPath = [[NSBundle mainBundle] pathForResource:@"model" ofType:@"pt"];
+    if ([Benchmark setup:config]) {
-    if ([[NSFileManager defaultManager] fileExistsAtPath:modelPath]) {
+      NSString* text = [Benchmark run];
-      [Benchmark benchmarkWithModel:modelPath];
+      dispatch_async(dispatch_get_main_queue(), ^{
+        self.textView.text = text;
+      });
     } else {
-      NSLog(@"model doesn't exist!");
+      NSLog(@"Setup benchmark config failed!");
     }
   });
 }

ios/TestApp/benchmark/config.json

@@ -0,0 +1,7 @@
+{
+    "input_dims": "1,3,224,224",
+    "input_type": "float",
+    "warmup": 10,
+    "iter": 10,
+    "print_output": false
+}

ios/TestApp/benchmark/setup.rb

@@ -1,6 +1,14 @@
-
 require 'xcodeproj'
 require 'fileutils'
+require 'optparse'
+
+options = {}
+option_parser = OptionParser.new do |opts|
+ opts.banner = 'Script for setting up TestApp.xcodeproj'
+ opts.on('-t', '--team_id ', 'developemnt team ID') { |value|
+    options[:team_id] = value
+ }
+end.parse!
 
 puts "Current directory: #{Dir.pwd}"
 
@@ -25,24 +33,33 @@ target.build_configurations.each do |config|
     config.build_settings['LIBRARY_SEARCH_PATHS']   = libraries_search_path
     config.build_settings['OTHER_LDFLAGS']          = other_linker_flags
     config.build_settings['ENABLE_BITCODE']         = 'No'
+    dev_team_id = options[:team_id]
+    if dev_team_id
+        config.build_settings['DEVELOPMENT_TEAM']   = dev_team_id
+    end
 end
 
 puts "Installing the testing model..."
 model_path = File.expand_path("./model.pt")
 if not File.exist?(model_path)
-   raise "no model can be found!"
+   raise "model.pt can't be found!"
 end
+config_path = File.expand_path("./config.json")
+if not File.exist?(config_path)
+    raise "config.json can't be found!"
+ end
 group = project.main_group.find_subpath(File.join('TestApp'),true)
 group.set_source_tree('SOURCE_ROOT')
 group.files.each do |file|
-    if file.name.to_s.end_with?(".pt")
+    if (file.name.to_s.end_with?(".pt") || file.name == "config.json")
-        puts "Found old model, remove it"
         group.remove_reference(file)
         target.resources_build_phase.remove_file_reference(file)
     end
 end
 model_file_ref = group.new_reference(model_path)
+config_file_ref = group.new_reference(config_path)
 target.resources_build_phase.add_file_reference(model_file_ref, true)
+target.resources_build_phase.add_file_reference(config_file_ref, true)
 
 puts "Linking static libraries..."
 target.frameworks_build_phases.clear

ios/TestApp/bootstrap.sh

@@ -0,0 +1,83 @@
+#!/bin/bash
+set -e
+
+echo "Current Dir: $(pwd)"
+if [[ "$OSTYPE" != *"darwin"* ]];then
+    error "Current OS Type is not MacOS"
+    sleep 1
+    exit 1
+fi
+BIN_NAME=$(basename "$0")
+help () {
+  echo "Usage: $BIN_NAME <options>"
+  echo
+  echo "Options:"
+  echo "   -t           Team Identifier"
+  echo "   -p           Name of the Provisioning Profile"
+}
+bootstrap() {
+    echo "starting"
+    PROJ_ROOT=$(pwd)
+    BENCHMARK_DIR="${PROJ_ROOT}/benchmark"
+    XCODE_PROJ_PATH="./TestApp.xcodeproj"
+    XCODE_TARGET="TestApp"
+    XCODE_BUILD="./build"
+    if [ -d ${XCODE_BUILD} ]; then
+        echo "found the old XCode build, remove it"
+        rm -rf ${XCODE_BUILD}
+    fi 
+    cd ${BENCHMARK_DIR}
+    echo "Generating model"
+    python trace_model.py
+    ruby setup.rb -t ${TEAM_ID}
+    cd ..
+    #run xcodebuild
+    if ! [ -x "$(command -v xcodebuild)" ]; then
+        echo 'Error: xcodebuild is not installed.'
+        exit 1
+    fi 
+    echo "Running xcodebuild"
+    xcodebuild clean build  -project ${XCODE_PROJ_PATH}  \
+                            -target ${XCODE_TARGET}  \
+                            -sdk iphoneos \
+                            -configuration Debug \
+                            PROVISIONING_PROFILE_SPECIFIER=${PROFILE}
+    #install TestApp
+    if ! [ -x "$(command -v ios-deploy)" ]; then
+        echo 'Error: ios-deploy is not installed.'
+        exit 1
+    fi 
+    echo "installing, make sure your phone is unlocked"
+    ios-deploy --bundle "${XCODE_BUILD}/Debug-iphoneos/${XCODE_TARGET}.app"
+    echo "Done."
+}
+while [[ $# -gt 1 ]]
+do
+option="$1"
+value="$2"
+case $option in 
+    "" | "-h" | "--help")
+    help
+    exit 0
+    ;;
+    "-t" | "--team")
+    TEAM_ID="${value}"
+    shift
+    ;;
+    "-p"|"--profile")
+    PROFILE="${value}"
+    shift
+    ;;
+    *)
+    echo "unknown options" >& 2 
+    help
+    exit 1
+    ;;
+esac
+shift 
+done
+
+echo TEAM_ID = "${TEAM_ID}"
+echo PROFILE = "${PROFILE}"
+
+bootstrap