.circleci: Remove quotes from --git-dir

git doesn't handle the escapes correctly so let's just not put them
altogether.

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

.bazelrc

@@ -1,3 +0,0 @@
-build --copt=--std=c++14
-build --copt=-I.
-build --copt=-isystem --copt bazel-out/k8-fastbuild/bin

.bazelversion

@@ -1 +0,0 @@
-3.1.0

.circleci/README.md

@@ -31,7 +31,7 @@ Usage
 1. Make changes to these scripts.
 2. Run the `regenerate.sh` script in this directory and commit the script changes and the resulting change to `config.yml`.
 
-You'll see a build failure on GitHub if the scripts don't agree with the checked-in version.
+You'll see a build failure on TravisCI if the scripts don't agree with the checked-in version.
 
 
 Motivation
@@ -55,7 +55,7 @@ Future direction
 See comment [here](https://github.com/pytorch/pytorch/pull/17323#pullrequestreview-206945747):
 
 In contrast with a full recursive tree traversal of configuration dimensions,
-> in the future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
+> in the future future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
 
 ----------------
 ----------------
@@ -71,9 +71,9 @@ A **binary configuration** is a collection of
 * release or nightly
     * releases are stable, nightlies are beta and built every night
 * python version
-    * linux: 3.5m, 3.6m 3.7m (mu is wide unicode or something like that. It usually doesn't matter but you should know that it exists)
-    * macos: 3.6, 3.7, 3.8
-    * windows: 3.6, 3.7, 3.8
+    * linux: 2.7m, 2.7mu, 3.5m, 3.6m 3.7m (mu is wide unicode or something like that. It usually doesn't matter but you should know that it exists)
+    * macos: 2.7, 3.5, 3.6, 3.7
+    * windows: 3.5, 3.6, 3.7
 * cpu version
     * cpu, cuda 9.0, cuda 10.0
     * The supported cuda versions occasionally change
@@ -90,7 +90,7 @@ The binaries are built in CircleCI. There are nightly binaries built every night
 
 We have 3 types of binary packages
 
-* pip packages - nightlies are stored on s3 (pip install -f \<a s3 url\>). releases are stored in a pip repo (pip install torch) (ask Soumith about this)
+* pip packages - nightlies are stored on s3 (pip install -f <a s3 url>). releases are stored in a pip repo (pip install torch) (ask Soumith about this)
 * conda packages - nightlies and releases are both stored in a conda repo. Nighty packages have a '_nightly' suffix
 * libtorch packages - these are zips of all the c++ libraries, header files, and sometimes dependencies. These are c++ only
     * shared with dependencies (the only supported option for Windows)
@@ -104,16 +104,16 @@ All binaries are built in CircleCI workflows except Windows. There are checked-i
 
 Some quick vocab:
 
-* A \**workflow** is a CircleCI concept; it is a DAG of '**jobs**'. ctrl-f 'workflows' on https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml to see the workflows.
+* A\**workflow** is a CircleCI concept; it is a DAG of '**jobs**'. ctrl-f 'workflows' on\https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml to see the workflows.
 * **jobs** are a sequence of '**steps**'
-* **steps** are usually just a bash script or a builtin CircleCI command. *All steps run in new environments, environment variables declared in one script DO NOT persist to following steps*
+* **steps** are usually just a bash script or a builtin CircleCI command.* All steps run in new environments, environment variables declared in one script DO NOT persist to following steps*
 * CircleCI has a **workspace**, which is essentially a cache between steps of the *same job* in which you can store artifacts between steps.
 
 ## How are the workflows structured?
 
 The nightly binaries have 3 workflows. We have one job (actually 3 jobs:  build, test, and upload) per binary configuration
 
-1. binary_builds
+1. binarybuilds
     1. every day midnight EST
     2. linux: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/linux-binary-build-defaults.yml
     3. macos: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/macos-binary-build-defaults.yml
@@ -144,7 +144,7 @@ The nightly binaries have 3 workflows. We have one job (actually 3 jobs:  build,
 
 ## How are the jobs structured?
 
-The jobs are in https://github.com/pytorch/pytorch/tree/master/.circleci/verbatim-sources. Jobs are made of multiple steps. There are some shared steps used by all the binaries/smokes. Steps of these jobs are all delegated to scripts in https://github.com/pytorch/pytorch/tree/master/.circleci/scripts .
+The jobs are in https://github.com/pytorch/pytorch/tree/master/.circleci/verbatim-sources . Jobs are made of multiple steps. There are some shared steps used by all the binaries/smokes. Steps of these jobs are all delegated to scripts in https://github.com/pytorch/pytorch/tree/master/.circleci/scripts .
 
 * Linux jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/linux-binary-build-defaults.yml
     * binary_linux_build.sh
@@ -178,7 +178,8 @@ CircleCI creates a  final yaml file by inlining every <<* segment, so if we were
 So, CircleCI has several executor types: macos, machine, and docker are the ones we use. The 'machine' executor gives you two cores on some linux vm. The 'docker' executor gives you considerably more cores (nproc was 32 instead of 2 back when I tried in February). Since the dockers are faster, we try to run everything that we can in dockers. Thus
 
 * linux build jobs use the docker executor. Running them on the docker executor was at least 2x faster than running them on the machine executor
-* linux test jobs use the machine executor in order for them to properly interface with GPUs since docker executors cannot execute with attached GPUs
+* linux test jobs use the machine executor and spin up their own docker. Why this nonsense? It's cause we run nvidia-docker for our GPU tests; any code that calls into the CUDA runtime needs to be run on nvidia-docker. To run a nvidia-docker you need to install some nvidia packages on the host machine and then call docker with the '—runtime nvidia' argument. CircleCI doesn't support this, so we have to do it ourself.
+    * This is not just a mere inconvenience. **This blocks all of our linux tests from using more than 2 cores.** But there is nothing that we can do about it, but wait for a fix on circleci's side. Right now, we only run some smoke tests (some simple imports) on the binaries, but this also affects non-binary test jobs.
 * linux upload jobs use the machine executor. The upload jobs are so short that it doesn't really matter what they use
 * linux smoke test jobs use the machine executor for the same reason as the linux test jobs
 
@@ -204,7 +205,7 @@ TODO: fill in stuff
 
 ## Overview
 
-The code that runs the binaries lives in two places, in the normal [github.com/pytorch/pytorch](http://github.com/pytorch/pytorch), but also in [github.com/pytorch/builder](http://github.com/pytorch/builder), which is a repo that defines how all the binaries are built. The relevant code is
+The code that runs the binaries lives in two places, in the normal [github.com/pytorch/pytorch](http://github.com/pytorch/pytorch), but also in [github.com/pytorch/builder](http://github.com/pytorch/builder) , which is a repo that defines how all the binaries are built. The relevant code is
 
 
 ```
@@ -260,7 +261,7 @@ Linux, MacOS and Windows use the same code flow for the conda builds.
 Conda packages are built with conda-build, see https://conda.io/projects/conda-build/en/latest/resources/commands/conda-build.html
 
 Basically, you pass `conda build` a build folder (pytorch-nightly/ above) that contains a build script and a meta.yaml. The meta.yaml specifies in what python environment to build the package in, and what dependencies the resulting package should have, and the build script gets called in the env to build the thing.
-tl;dr on conda-build is
+tldr; on conda-build is
 
 1. Creates a brand new conda environment, based off of deps in the meta.yaml
     1. Note that environment variables do not get passed into this build env unless they are specified in the meta.yaml
@@ -270,7 +271,7 @@ tl;dr on conda-build is
 4. Runs some simple import tests (if specified in the meta.yaml)
 5. Saves the finished package as a tarball
 
-The build.sh we use is essentially a wrapper around `python setup.py build`, but it also manually copies in some of our dependent libraries into the resulting tarball and messes with some rpaths.
+The build.sh we use is essentially a wrapper around ```python setup.py build``` , but it also manually copies in some of our dependent libraries into the resulting tarball and messes with some rpaths.
 
 The entrypoint file `builder/conda/build_conda.sh` is complicated because
 
@@ -355,15 +356,15 @@ The Dockerfiles are available in pytorch/builder, but there is no circleci job o
 
 # How to manually rebuild the binaries
 
-tl;dr make a PR that looks like https://github.com/pytorch/pytorch/pull/21159
+tldr; make a PR that looks like https://github.com/pytorch/pytorch/pull/21159
 
 Sometimes we want to push a change to master and then rebuild all of today's binaries after that change. As of May 30, 2019 there isn't a way to manually run a workflow in the UI. You can manually re-run a workflow, but it will use the exact same git commits as the first run and will not include any changes. So we have to make a PR and then force circleci to run the binary workflow instead of the normal tests. The above PR is an example of how to do this; essentially you copy-paste the binarybuilds workflow steps into the default workflow steps. If you need to point the builder repo to a different commit then you'd need to change https://github.com/pytorch/pytorch/blob/master/.circleci/scripts/binary_checkout.sh#L42-L45 to checkout what you want.
 
 ## How to test changes to the binaries via .circleci
 
-Writing PRs that test the binaries is annoying, since the default circleci jobs that run on PRs are not the jobs that you want to run. Likely, changes to the binaries will touch something under .circleci/ and require that .circleci/config.yml be regenerated (.circleci/config.yml controls all .circleci behavior, and is generated using `.circleci/regenerate.sh` in python 3.7). But you also need to manually hardcode the binary jobs that you want to test into the .circleci/config.yml workflow, so you should actually make at least two commits, one for your changes and one to temporarily hardcode jobs. See https://github.com/pytorch/pytorch/pull/22928 as an example of how to do this.
+Writing PRs that test the binaries is annoying, since the default circleci jobs that run on PRs are not the jobs that you want to run. Likely, changes to the binaries will touch something under .circleci/ and require that .circleci/config.yml be regenerated (.circleci/config.yml controls all .circleci behavior, and is generated using ```.circleci/regenerate.sh``` in python 3.7). But you also need to manually hardcode the binary jobs that you want to test into the .circleci/config.yml workflow, so you should actually make at least two commits, one for your changes and one to temporarily hardcode jobs. See https://github.com/pytorch/pytorch/pull/22928 as an example of how to do this.
 
-```sh
+```
 # Make your changes
 touch .circleci/verbatim-sources/nightly-binary-build-defaults.yml
 
@@ -408,7 +409,7 @@ The advantage of this flow is that you can make new changes to the base commit a
 
 You can build Linux binaries locally easily using docker.
 
-```sh
+```
 # Run the docker
 # Use the correct docker image, pytorch/conda-cuda used here as an example
 #
@@ -418,6 +419,8 @@ You can build Linux binaries locally easily using docker.
 #    in the docker container then you will see path/to/foo/baz on your local
 #    machine. You could also clone the pytorch and builder repos in the docker.
 #
+# If you're building a CUDA binary then use `nvidia-docker run` instead, see below.
+#
 # If you know how, add ccache as a volume too and speed up everything
 docker run \
     -v your/pytorch/repo:/pytorch \
@@ -441,7 +444,9 @@ export DESIRED_CUDA=cpu
 
 **Building CUDA binaries on docker**
 
-You can build CUDA binaries on CPU only machines, but you can only run CUDA binaries on CUDA machines. This means that you can build a CUDA binary on a docker on your laptop if you so choose (though it’s gonna take a long time).
+To build a CUDA binary you need to use `nvidia-docker run` instead of just `docker run` (or you can manually pass `--runtime=nvidia`). This adds some needed libraries and things to build CUDA stuff.
+
+You can build CUDA binaries on CPU only machines, but you can only run CUDA binaries on CUDA machines. This means that you can build a CUDA binary on a docker on your laptop if you so choose (though it’s gonna take a loong time).
 
 For Facebook employees, ask about beefy machines that have docker support and use those instead of your laptop; it will be 5x as fast.
 
@@ -451,7 +456,7 @@ There’s no easy way to generate reproducible hermetic MacOS environments. If y
 
 But if you want to try, then I’d recommend
 
-```sh
+```
 # Create a new terminal
 # Clear your LD_LIBRARY_PATH and trim as much out of your PATH as you
 # know how to do
@@ -461,7 +466,7 @@ But if you want to try, then I’d recommend
 # Always install miniconda 3, even if building for Python <3
 new_conda="~/my_new_conda"
 conda_sh="$new_conda/install_miniconda.sh"
-curl -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+curl -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
 chmod +x "$conda_sh"
 "$conda_sh" -b -p "$MINICONDA_ROOT"
 rm -f "$conda_sh"

.circleci/cimodel/data/binary_build_data.py

@@ -5,6 +5,9 @@ for "smoketest" builds.
 Each subclass of ConfigNode represents a layer of the configuration hierarchy.
 These tree nodes encapsulate the logic for whether a branch of the hierarchy
 should be "pruned".
+
+In addition to generating config.yml content, the tree is also traversed
+to produce a visualization of config dimensions.
 """
 
 from collections import OrderedDict
@@ -25,52 +28,34 @@ DEPS_INCLUSION_DIMENSIONS = [
 ]
 
 
-def get_processor_arch_name(gpu_version):
-    return "cpu" if not gpu_version else (
-        "cu" + gpu_version.strip("cuda") if gpu_version.startswith("cuda") else gpu_version
-    )
+def get_processor_arch_name(cuda_version):
+    return "cpu" if not cuda_version else "cu" + cuda_version
+
 
 LINUX_PACKAGE_VARIANTS = OrderedDict(
     manywheel=[
+        "2.7m",
+        "2.7mu",
+        "3.5m",
         "3.6m",
         "3.7m",
         "3.8m",
-        "3.9m"
     ],
     conda=dimensions.STANDARD_PYTHON_VERSIONS,
     libtorch=[
-        "3.7m",
+        "2.7m",
     ],
 )
 
 CONFIG_TREE_DATA = OrderedDict(
-    linux=(dimensions.GPU_VERSIONS, LINUX_PACKAGE_VARIANTS),
+    linux=(dimensions.CUDA_VERSIONS, LINUX_PACKAGE_VARIANTS),
     macos=([None], OrderedDict(
         wheel=dimensions.STANDARD_PYTHON_VERSIONS,
         conda=dimensions.STANDARD_PYTHON_VERSIONS,
         libtorch=[
-            "3.7",
+            "2.7",
         ],
     )),
-    macos_arm64=([None], OrderedDict(
-        wheel=[
-            "3.8",
-        ],
-        conda=[
-            "3.8",
-        ],
-    )),
-    # Skip CUDA-9.2 builds on Windows
-    windows=(
-        [v for v in dimensions.GPU_VERSIONS if v not in ['cuda92'] + dimensions.ROCM_VERSION_LABELS],
-        OrderedDict(
-            wheel=dimensions.STANDARD_PYTHON_VERSIONS,
-            conda=dimensions.STANDARD_PYTHON_VERSIONS,
-            libtorch=[
-                "3.7",
-            ],
-        )
-    ),
 )
 
 # GCC config variants:
@@ -89,11 +74,6 @@ LINUX_GCC_CONFIG_VARIANTS = OrderedDict(
     ],
 )
 
-WINDOWS_LIBTORCH_CONFIG_VARIANTS = [
-    "debug",
-    "release",
-]
-
 
 class TopLevelNode(ConfigNode):
     def __init__(self, node_name, config_tree_data, smoke):
@@ -107,12 +87,12 @@ class TopLevelNode(ConfigNode):
 
 
 class OSConfigNode(ConfigNode):
-    def __init__(self, parent, os_name, gpu_versions, py_tree):
+    def __init__(self, parent, os_name, cuda_versions, py_tree):
         super(OSConfigNode, self).__init__(parent, os_name)
 
         self.py_tree = py_tree
         self.props["os_name"] = os_name
-        self.props["gpu_versions"] = gpu_versions
+        self.props["cuda_versions"] = cuda_versions
 
     def get_children(self):
         return [PackageFormatConfigNode(self, k, v) for k, v in self.py_tree.items()]
@@ -128,10 +108,8 @@ class PackageFormatConfigNode(ConfigNode):
     def get_children(self):
         if self.find_prop("os_name") == "linux":
             return [LinuxGccConfigNode(self, v) for v in LINUX_GCC_CONFIG_VARIANTS[self.find_prop("package_format")]]
-        elif self.find_prop("os_name") == "windows" and self.find_prop("package_format") == "libtorch":
-            return [WindowsLibtorchConfigNode(self, v) for v in WINDOWS_LIBTORCH_CONFIG_VARIANTS]
         else:
-            return [ArchConfigNode(self, v) for v in self.find_prop("gpu_versions")]
+            return [ArchConfigNode(self, v) for v in self.find_prop("cuda_versions")]
 
 
 class LinuxGccConfigNode(ConfigNode):
@@ -141,39 +119,21 @@ class LinuxGccConfigNode(ConfigNode):
         self.props["gcc_config_variant"] = gcc_config_variant
 
     def get_children(self):
-        gpu_versions = self.find_prop("gpu_versions")
+        cuda_versions = self.find_prop("cuda_versions")
 
         # XXX devtoolset7 on CUDA 9.0 is temporarily disabled
         # see https://github.com/pytorch/pytorch/issues/20066
         if self.find_prop("gcc_config_variant") == 'devtoolset7':
-            gpu_versions = filter(lambda x: x != "cuda_90", gpu_versions)
+            cuda_versions = filter(lambda x: x != "90", cuda_versions)
 
-        # XXX disabling conda rocm build since docker images are not there
-        if self.find_prop("package_format") == 'conda':
-            gpu_versions = filter(lambda x: x not in dimensions.ROCM_VERSION_LABELS, gpu_versions)
-
-        # XXX libtorch rocm build  is temporarily disabled
-        if self.find_prop("package_format") == 'libtorch':
-            gpu_versions = filter(lambda x: x not in dimensions.ROCM_VERSION_LABELS, gpu_versions)
-
-        return [ArchConfigNode(self, v) for v in gpu_versions]
-
-
-class WindowsLibtorchConfigNode(ConfigNode):
-    def __init__(self, parent, libtorch_config_variant):
-        super(WindowsLibtorchConfigNode, self).__init__(parent, "LIBTORCH_CONFIG_VARIANT=" + str(libtorch_config_variant))
-
-        self.props["libtorch_config_variant"] = libtorch_config_variant
-
-    def get_children(self):
-        return [ArchConfigNode(self, v) for v in self.find_prop("gpu_versions")]
+        return [ArchConfigNode(self, v) for v in cuda_versions]
 
 
 class ArchConfigNode(ConfigNode):
-    def __init__(self, parent, gpu):
-        super(ArchConfigNode, self).__init__(parent, get_processor_arch_name(gpu))
+    def __init__(self, parent, cu):
+        super(ArchConfigNode, self).__init__(parent, get_processor_arch_name(cu))
 
-        self.props["gpu"] = gpu
+        self.props["cu"] = cu
 
     def get_children(self):
         return [PyVersionConfigNode(self, v) for v in self.find_prop("python_versions")]
@@ -186,6 +146,8 @@ class PyVersionConfigNode(ConfigNode):
         self.props["pyver"] = pyver
 
     def get_children(self):
+
+        smoke = self.find_prop("smoke")
         package_format = self.find_prop("package_format")
         os_name = self.find_prop("os_name")
 

.circleci/cimodel/data/binary_build_definitions.py

@@ -1,28 +1,25 @@
 from collections import OrderedDict
 
-import cimodel.data.simple.util.branch_filters as branch_filters
 import cimodel.data.binary_build_data as binary_build_data
 import cimodel.lib.conf_tree as conf_tree
 import cimodel.lib.miniutils as miniutils
 
+
 class Conf(object):
-    def __init__(self, os, gpu_version, pydistro, parms, smoke, libtorch_variant, gcc_config_variant, libtorch_config_variant):
+    def __init__(self, os, cuda_version, pydistro, parms, smoke, libtorch_variant, gcc_config_variant):
 
         self.os = os
-        self.gpu_version = gpu_version
+        self.cuda_version = cuda_version
         self.pydistro = pydistro
         self.parms = parms
         self.smoke = smoke
         self.libtorch_variant = libtorch_variant
         self.gcc_config_variant = gcc_config_variant
-        self.libtorch_config_variant = libtorch_config_variant
 
     def gen_build_env_parms(self):
-        elems = [self.pydistro] + self.parms + [binary_build_data.get_processor_arch_name(self.gpu_version)]
+        elems = [self.pydistro] + self.parms + [binary_build_data.get_processor_arch_name(self.cuda_version)]
         if self.gcc_config_variant is not None:
             elems.append(str(self.gcc_config_variant))
-        if self.libtorch_config_variant is not None:
-            elems.append(str(self.libtorch_config_variant))
         return elems
 
     def gen_docker_image(self):
@@ -37,12 +34,9 @@ class Conf(object):
         docker_distro_prefix = miniutils.override(self.pydistro, docker_word_substitution)
 
         # The cpu nightlies are built on the pytorch/manylinux-cuda102 docker image
-        # TODO cuda images should consolidate into tag-base images similar to rocm
-        alt_docker_suffix = "cuda102" if not self.gpu_version else (
-            "rocm:" + self.gpu_version.strip("rocm") if self.gpu_version.startswith("rocm") else self.gpu_version)
-        docker_distro_suffix = alt_docker_suffix if self.pydistro != "conda" else (
-            "cuda" if alt_docker_suffix.startswith("cuda") else "rocm")
-        return miniutils.quote("pytorch/" + docker_distro_prefix + "-" + docker_distro_suffix)
+        alt_docker_suffix = self.cuda_version or "102"
+        docker_distro_suffix = "" if self.pydistro == "conda" else alt_docker_suffix
+        return miniutils.quote("pytorch/" + docker_distro_prefix + "-cuda" + docker_distro_suffix)
 
     def get_name_prefix(self):
         return "smoke" if self.smoke else "binary"
@@ -67,82 +61,48 @@ class Conf(object):
         job_def = OrderedDict()
         job_def["name"] = self.gen_build_name(phase, nightly)
         job_def["build_environment"] = miniutils.quote(" ".join(self.gen_build_env_parms()))
+        job_def["requires"] = ["setup"]
         if self.smoke:
-            job_def["requires"] = [
-                "update_s3_htmls",
-            ]
-            job_def["filters"] = branch_filters.gen_filter_dict(
-                branches_list=["postnightly"],
-            )
+            job_def["requires"].append("update_s3_htmls_for_nightlies")
+            job_def["requires"].append("update_s3_htmls_for_nightlies_devtoolset7")
+            job_def["filters"] = {"branches": {"only": "postnightly"}}
         else:
-            filter_branch = r"/.*/"
-            job_def["filters"] = branch_filters.gen_filter_dict(
-                branches_list=[filter_branch],
-                tags_list=[branch_filters.RC_PATTERN],
-            )
+            job_def["filters"] = {
+                "branches": {
+                    "only": "nightly"
+                },
+                # Will run on tags like v1.5.0-rc1, etc.
+                "tags": {
+                    # Using a raw string here to avoid having to escape
+                    # anything
+                    "only": r"/v[0-9]+(\.[0-9]+)*-rc[0-9]+/"
+                }
+            }
         if self.libtorch_variant:
             job_def["libtorch_variant"] = miniutils.quote(self.libtorch_variant)
         if phase == "test":
             if not self.smoke:
-                job_def["requires"] = [self.gen_build_name("build", nightly)]
-            if not (self.smoke and self.os == "macos") and self.os != "windows":
+                job_def["requires"].append(self.gen_build_name("build", nightly))
+            if not (self.smoke and self.os == "macos"):
                 job_def["docker_image"] = self.gen_docker_image()
 
-            # fix this. only works on cuda not rocm
-            if self.os != "windows" and self.gpu_version:
+            if self.cuda_version:
                 job_def["use_cuda_docker_runtime"] = miniutils.quote("1")
         else:
             if self.os == "linux" and phase != "upload":
                 job_def["docker_image"] = self.gen_docker_image()
 
         if phase == "test":
-            if self.gpu_version:
-                if self.os == "windows":
-                    job_def["executor"] = "windows-with-nvidia-gpu"
-                else:
-                    job_def["resource_class"] = "gpu.medium"
+            if self.cuda_version:
+                job_def["resource_class"] = "gpu.medium"
+        if phase == "upload":
+            job_def["context"] = "org-member"
+            job_def["requires"] = ["setup", self.gen_build_name(upload_phase_dependency, nightly)]
 
         os_name = miniutils.override(self.os, {"macos": "mac"})
         job_name = "_".join([self.get_name_prefix(), os_name, phase])
         return {job_name : job_def}
 
-    def gen_upload_job(self, phase, requires_dependency):
-        """Generate binary_upload job for configuration
-
-        Output looks similar to:
-
-      - binary_upload:
-          name: binary_linux_manywheel_3_7m_cu92_devtoolset7_nightly_upload
-          context: org-member
-          requires: binary_linux_manywheel_3_7m_cu92_devtoolset7_nightly_test
-          filters:
-            branches:
-              only:
-                - nightly
-            tags:
-              only: /v[0-9]+(\\.[0-9]+)*-rc[0-9]+/
-          package_type: manywheel
-          upload_subfolder: cu92
-        """
-        return {
-            "binary_upload": OrderedDict({
-                "name": self.gen_build_name(phase, nightly=True),
-                "context": "org-member",
-                "requires": [self.gen_build_name(
-                    requires_dependency,
-                    nightly=True
-                )],
-                "filters": branch_filters.gen_filter_dict(
-                    branches_list=["nightly"],
-                    tags_list=[branch_filters.RC_PATTERN],
-                ),
-                "package_type": self.pydistro,
-                "upload_subfolder": binary_build_data.get_processor_arch_name(
-                    self.gpu_version,
-                ),
-            })
-        }
-
 def get_root(smoke, name):
 
     return binary_build_data.TopLevelNode(
@@ -161,47 +121,35 @@ def gen_build_env_list(smoke):
     for c in config_list:
         conf = Conf(
             c.find_prop("os_name"),
-            c.find_prop("gpu"),
+            c.find_prop("cu"),
             c.find_prop("package_format"),
             [c.find_prop("pyver")],
-            c.find_prop("smoke") and not (c.find_prop("os_name") == "macos_arm64"),  # don't test arm64
+            c.find_prop("smoke"),
             c.find_prop("libtorch_variant"),
             c.find_prop("gcc_config_variant"),
-            c.find_prop("libtorch_config_variant"),
         )
         newlist.append(conf)
 
     return newlist
 
-def predicate_exclude_macos(config):
-    return config.os == "linux" or config.os == "windows"
+
+def predicate_exclude_nonlinux_and_libtorch(config):
+    return config.os == "linux"
+
 
 def get_nightly_uploads():
     configs = gen_build_env_list(False)
     mylist = []
     for conf in configs:
-        phase_dependency = "test" if predicate_exclude_macos(conf) else "build"
-        mylist.append(conf.gen_upload_job("upload", phase_dependency))
+        phase_dependency = "test" if predicate_exclude_nonlinux_and_libtorch(conf) else "build"
+        mylist.append(conf.gen_workflow_job("upload", phase_dependency, nightly=True))
 
     return mylist
 
-def get_post_upload_jobs():
-    return [
-        {
-            "update_s3_htmls": {
-                "name": "update_s3_htmls",
-                "context": "org-member",
-                "filters": branch_filters.gen_filter_dict(
-                    branches_list=["postnightly"],
-                ),
-            },
-        },
-    ]
-
 def get_nightly_tests():
 
     configs = gen_build_env_list(False)
-    filtered_configs = filter(predicate_exclude_macos, configs)
+    filtered_configs = filter(predicate_exclude_nonlinux_and_libtorch, configs)
 
     tests = []
     for conf_options in filtered_configs:
@@ -216,9 +164,7 @@ def get_jobs(toplevel_key, smoke):
     configs = gen_build_env_list(smoke)
     phase = "build" if toplevel_key == "binarybuilds" else "test"
     for build_config in configs:
-        # don't test for macos_arm64 as it's cross compiled
-        if phase != "test" or build_config.os != "macos_arm64":
-            jobs_list.append(build_config.gen_workflow_job(phase, nightly=True))
+        jobs_list.append(build_config.gen_workflow_job(phase, nightly=True))
 
     return jobs_list
 

.circleci/cimodel/data/caffe2_build_data.py

@@ -0,0 +1,92 @@
+from cimodel.lib.conf_tree import ConfigNode, XImportant
+from cimodel.lib.conf_tree import Ver
+
+
+CONFIG_TREE_DATA = [
+    (Ver("ubuntu", "16.04"), [
+        ([Ver("gcc", "5")], [XImportant("onnx_py2")]),
+        ([Ver("clang", "7")], [XImportant("onnx_main_py3.6"),
+                               XImportant("onnx_ort1_py3.6"),
+                               XImportant("onnx_ort2_py3.6")]),
+    ]),
+]
+
+
+class TreeConfigNode(ConfigNode):
+    def __init__(self, parent, node_name, subtree):
+        super(TreeConfigNode, self).__init__(parent, self.modify_label(node_name))
+        self.subtree = subtree
+        self.init2(node_name)
+
+    # noinspection PyMethodMayBeStatic
+    def modify_label(self, label):
+        return str(label)
+
+    def init2(self, node_name):
+        pass
+
+    def get_children(self):
+        return [self.child_constructor()(self, k, v) for (k, v) in self.subtree]
+
+    def is_build_only(self):
+        if str(self.find_prop("language_version")) == "onnx_main_py3.6" or \
+                str(self.find_prop("language_version")) == "onnx_ort1_py3.6" or \
+                str(self.find_prop("language_version")) == "onnx_ort2_py3.6":
+            return False
+        return set(str(c) for c in self.find_prop("compiler_version")).intersection({
+            "clang3.8",
+            "clang3.9",
+            "clang7",
+            "android",
+        }) or self.find_prop("distro_version").name == "macos"
+
+    def is_test_only(self):
+        if str(self.find_prop("language_version")) == "onnx_ort1_py3.6" or \
+                str(self.find_prop("language_version")) == "onnx_ort2_py3.6":
+            return True
+        return False
+
+
+class TopLevelNode(TreeConfigNode):
+    def __init__(self, node_name, subtree):
+        super(TopLevelNode, self).__init__(None, node_name, subtree)
+
+    # noinspection PyMethodMayBeStatic
+    def child_constructor(self):
+        return DistroConfigNode
+
+
+class DistroConfigNode(TreeConfigNode):
+    def init2(self, node_name):
+        self.props["distro_version"] = node_name
+
+    # noinspection PyMethodMayBeStatic
+    def child_constructor(self):
+        return CompilerConfigNode
+
+
+class CompilerConfigNode(TreeConfigNode):
+    def init2(self, node_name):
+        self.props["compiler_version"] = node_name
+
+    # noinspection PyMethodMayBeStatic
+    def child_constructor(self):
+        return LanguageConfigNode
+
+
+class LanguageConfigNode(TreeConfigNode):
+    def init2(self, node_name):
+        self.props["language_version"] = node_name
+        self.props["build_only"] = self.is_build_only()
+        self.props["test_only"] = self.is_test_only()
+
+    def child_constructor(self):
+        return ImportantConfigNode
+
+
+class ImportantConfigNode(TreeConfigNode):
+    def init2(self, node_name):
+        self.props["important"] = True
+
+    def get_children(self):
+        return []

.circleci/cimodel/data/caffe2_build_definitions.py

@@ -0,0 +1,177 @@
+from collections import OrderedDict
+
+import cimodel.data.dimensions as dimensions
+import cimodel.lib.conf_tree as conf_tree
+from cimodel.lib.conf_tree import Ver
+import cimodel.lib.miniutils as miniutils
+from cimodel.data.caffe2_build_data import CONFIG_TREE_DATA, TopLevelNode
+
+
+from dataclasses import dataclass
+
+
+DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/caffe2/"
+
+DOCKER_IMAGE_VERSION = "345"
+
+
+@dataclass
+class Conf:
+    language: str
+    distro: Ver
+    # There could be multiple compiler versions configured (e.g. nvcc
+    # for gpu files and host compiler (gcc/clang) for cpu files)
+    compilers: [Ver]
+    build_only: bool
+    test_only: bool
+    is_important: bool
+
+    @property
+    def compiler_names(self):
+        return [c.name for c in self.compilers]
+
+    # TODO: Eventually we can probably just remove the cudnn7 everywhere.
+    def get_cudnn_insertion(self):
+
+        omit = self.language == "onnx_py2" \
+            or self.language == "onnx_main_py3.6" \
+            or self.language == "onnx_ort1_py3.6" \
+            or self.language == "onnx_ort2_py3.6" \
+            or set(self.compiler_names).intersection({"android", "mkl", "clang"}) \
+            or str(self.distro) in ["ubuntu14.04", "macos10.13"]
+
+        return [] if omit else ["cudnn7"]
+
+    def get_build_name_root_parts(self):
+        return [
+            "caffe2",
+            self.language,
+        ] + self.get_build_name_middle_parts()
+
+    def get_build_name_middle_parts(self):
+        return [str(c) for c in self.compilers] + self.get_cudnn_insertion() + [str(self.distro)]
+
+    def construct_phase_name(self, phase):
+        root_parts = self.get_build_name_root_parts()
+
+        build_name_substitutions = {
+            "onnx_ort1_py3.6": "onnx_main_py3.6",
+            "onnx_ort2_py3.6": "onnx_main_py3.6",
+        }
+        if phase == "build":
+            root_parts = [miniutils.override(r, build_name_substitutions) for r in root_parts]
+        return "_".join(root_parts + [phase]).replace(".", "_")
+
+    def get_platform(self):
+        platform = self.distro.name
+        if self.distro.name != "macos":
+            platform = "linux"
+        return platform
+
+    def gen_docker_image(self):
+
+        lang_substitutions = {
+            "onnx_py2": "py2",
+            "onnx_main_py3.6": "py3.6",
+            "onnx_ort1_py3.6": "py3.6",
+            "onnx_ort2_py3.6": "py3.6",
+            "cmake": "py2",
+        }
+
+        lang = miniutils.override(self.language, lang_substitutions)
+        parts = [lang] + self.get_build_name_middle_parts()
+        return miniutils.quote(DOCKER_IMAGE_PATH_BASE + "-".join(parts) + ":" + str(DOCKER_IMAGE_VERSION))
+
+    def gen_workflow_params(self, phase):
+        parameters = OrderedDict()
+        lang_substitutions = {
+            "onnx_py2": "onnx-py2",
+            "onnx_main_py3.6": "onnx-main-py3.6",
+            "onnx_ort1_py3.6": "onnx-ort1-py3.6",
+            "onnx_ort2_py3.6": "onnx-ort2-py3.6",
+        }
+
+        lang = miniutils.override(self.language, lang_substitutions)
+
+        parts = [
+            "caffe2",
+            lang,
+        ] + self.get_build_name_middle_parts() + [phase]
+
+        build_env_name = "-".join(parts)
+        parameters["build_environment"] = miniutils.quote(build_env_name)
+        if "ios" in self.compiler_names:
+            parameters["build_ios"] = miniutils.quote("1")
+        if phase == "test":
+            # TODO cuda should not be considered a compiler
+            if "cuda" in self.compiler_names:
+                parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
+
+        if self.distro.name != "macos":
+            parameters["docker_image"] = self.gen_docker_image()
+            if self.build_only:
+                parameters["build_only"] = miniutils.quote("1")
+        if phase == "test":
+            resource_class = "large" if "cuda" not in self.compiler_names else "gpu.medium"
+            parameters["resource_class"] = resource_class
+
+        return parameters
+
+    def gen_workflow_job(self, phase):
+        job_def = OrderedDict()
+        job_def["name"] = self.construct_phase_name(phase)
+        job_def["requires"] = ["setup"]
+
+        if phase == "test":
+            job_def["requires"].append(self.construct_phase_name("build"))
+            job_name = "caffe2_" + self.get_platform() + "_test"
+        else:
+            job_name = "caffe2_" + self.get_platform() + "_build"
+
+        if not self.is_important:
+            job_def["filters"] = {"branches": {"only": ["master", r"/ci-all\/.*/"]}}
+        job_def.update(self.gen_workflow_params(phase))
+        return {job_name : job_def}
+
+
+def get_root():
+    return TopLevelNode("Caffe2 Builds", CONFIG_TREE_DATA)
+
+
+def instantiate_configs():
+
+    config_list = []
+
+    root = get_root()
+    found_configs = conf_tree.dfs(root)
+    for fc in found_configs:
+        c = Conf(
+            language=fc.find_prop("language_version"),
+            distro=fc.find_prop("distro_version"),
+            compilers=fc.find_prop("compiler_version"),
+            build_only=fc.find_prop("build_only"),
+            test_only=fc.find_prop("test_only"),
+            is_important=fc.find_prop("important"),
+        )
+
+        config_list.append(c)
+
+    return config_list
+
+
+def get_workflow_jobs():
+
+    configs = instantiate_configs()
+
+    x = []
+    for conf_options in configs:
+        phases = ["build"]
+        if not conf_options.build_only:
+            phases = dimensions.PHASES
+        if conf_options.test_only:
+            phases = ["test"]
+
+        for phase in phases:
+            x.append(conf_options.gen_workflow_job(phase))
+
+    return x

.circleci/cimodel/data/dimensions.py

@@ -1,23 +1,16 @@
 PHASES = ["build", "test"]
 
 CUDA_VERSIONS = [
+    None,  # cpu build
+    "92",
     "101",
     "102",
-    "111",
 ]
 
-ROCM_VERSIONS = [
-    "3.10",
-    "4.0.1",
-]
-
-ROCM_VERSION_LABELS = ["rocm" + v for v in ROCM_VERSIONS]
-
-GPU_VERSIONS = [None] + ["cuda" + v for v in CUDA_VERSIONS] + ROCM_VERSION_LABELS
-
 STANDARD_PYTHON_VERSIONS = [
+    "2.7",
+    "3.5",
     "3.6",
     "3.7",
-    "3.8",
-    "3.9"
+    "3.8"
 ]

.circleci/cimodel/data/pytorch_build_data.py

@@ -3,13 +3,16 @@ from cimodel.lib.conf_tree import ConfigNode, X, XImportant
 
 CONFIG_TREE_DATA = [
     ("xenial", [
+        (None, [
+            X("3.5"),
+            X("nightly"),
+        ]),
         ("gcc", [
             ("5.4", [  # All this subtree rebases to master and then build
+                XImportant("3.6"),
                 ("3.6", [
-                    ("important", [X(True)]),
                     ("parallel_tbb", [X(True)]),
                     ("parallel_native", [X(True)]),
-                    ("pure_torch", [X(True)]),
                 ]),
             ]),
             # TODO: bring back libtorch test
@@ -17,86 +20,42 @@ CONFIG_TREE_DATA = [
         ]),
         ("clang", [
             ("5", [
-                ("3.6", [
-                    ("asan", [
-                        (True, [
-                            ("shard_test", [XImportant(True)]),
-                        ]),
-                    ]),
-                ]),
+                XImportant("3.6"),  # This is actually the ASAN build
             ]),
             ("7", [
                 ("3.6", [
-                    ("onnx", [XImportant(True)]),
+                    ("xla", [XImportant(True)]),
                 ]),
             ]),
         ]),
         ("cuda", [
-            ("9.2", [
-                ("3.6", [
-                    X(True),
-                    ("cuda_gcc_override", [
-                        ("gcc5.4", [
-                            ('build_only', [XImportant(True)]),
-                        ]),
-                    ]),
-                ])
-            ]),
-            ("10.1", [
-                ("3.6", [
-                    ('build_only', [X(True)]),
-                ]),
+            ("9", [
+                # Note there are magic strings here
+                # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L21
+                # and
+                # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L143
+                # and
+                # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L153
+                # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453144)
+                X("3.6"),
             ]),
+            ("9.2", [X("3.6")]),
+            ("10.1", [X("3.6")]),
             ("10.2", [
-                ("3.6", [
-                    ("shard_test", [XImportant(True)]),
-                    ("libtorch", [
-                        (True, [
-                            ('build_only', [X(True)]),
-                        ]),
-                    ]),
-                ]),
-            ]),
-            ("11.1", [
-                ("3.8", [
-                    X(True),
-                    ("libtorch", [
-                        (True, [
-                            ('build_only', [XImportant(True)]),
-                        ]),
-                    ]),
-                ]),
-            ]),
-        ]),
-    ]),
-    ("bionic", [
-        ("clang", [
-            ("9", [
                 XImportant("3.6"),
-            ]),
-            ("9", [
                 ("3.6", [
-                    ("xla", [XImportant(True)]),
-                    ("vulkan", [XImportant(True)]),
+                    ("libtorch", [XImportant(True)])
                 ]),
             ]),
         ]),
-        ("gcc", [
-            ("9", [
-                ("3.8", [
-                    ("coverage", [
-                        (True, [
-                            ("shard_test", [XImportant(True)]),
-                        ]),
-                    ]),
-                ]),
-            ]),
-        ]),
-        ("rocm", [
-            ("3.9", [
+        ("android", [
+            ("r19c", [
                 ("3.6", [
-                    ('build_only', [XImportant(True)]),
-                ]),
+                    ("android_abi", [XImportant("x86_32")]),
+                    ("android_abi", [X("x86_64")]),
+                    ("android_abi", [X("arm-v7a")]),
+                    ("android_abi", [X("arm-v8a")]),
+                ])
             ]),
         ]),
     ]),
@@ -142,7 +101,6 @@ class DistroConfigNode(TreeConfigNode):
 
         next_nodes = {
             "xenial": XenialCompilerConfigNode,
-            "bionic": BionicCompilerConfigNode,
         }
         return next_nodes[distro]
 
@@ -165,34 +123,16 @@ class ExperimentalFeatureConfigNode(TreeConfigNode):
         experimental_feature = self.find_prop("experimental_feature")
 
         next_nodes = {
-            "asan": AsanConfigNode,
             "xla": XlaConfigNode,
-            "vulkan": VulkanConfigNode,
             "parallel_tbb": ParallelTBBConfigNode,
             "parallel_native": ParallelNativeConfigNode,
-            "onnx": ONNXConfigNode,
             "libtorch": LibTorchConfigNode,
             "important": ImportantConfigNode,
-            "build_only": BuildOnlyConfigNode,
-            "shard_test": ShardTestConfigNode,
-            "cuda_gcc_override": CudaGccOverrideConfigNode,
-            "coverage": CoverageConfigNode,
-            "pure_torch": PureTorchConfigNode,
+            "android_abi": AndroidAbiConfigNode,
         }
         return next_nodes[experimental_feature]
 
 
-class PureTorchConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "PURE_TORCH=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_pure_torch"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
 class XlaConfigNode(TreeConfigNode):
     def modify_label(self, label):
         return "XLA=" + str(label)
@@ -203,40 +143,6 @@ class XlaConfigNode(TreeConfigNode):
     def child_constructor(self):
         return ImportantConfigNode
 
-
-class AsanConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "Asan=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_asan"] = node_name
-
-    def child_constructor(self):
-        return ExperimentalFeatureConfigNode
-
-
-class ONNXConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "Onnx=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_onnx"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class VulkanConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "Vulkan=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_vulkan"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
 class ParallelTBBConfigNode(TreeConfigNode):
     def modify_label(self, label):
         return "PARALLELTBB=" + str(label)
@@ -247,7 +153,6 @@ class ParallelTBBConfigNode(TreeConfigNode):
     def child_constructor(self):
         return ImportantConfigNode
 
-
 class ParallelNativeConfigNode(TreeConfigNode):
     def modify_label(self, label):
         return "PARALLELNATIVE=" + str(label)
@@ -258,7 +163,6 @@ class ParallelNativeConfigNode(TreeConfigNode):
     def child_constructor(self):
         return ImportantConfigNode
 
-
 class LibTorchConfigNode(TreeConfigNode):
     def modify_label(self, label):
         return "BUILD_TEST_LIBTORCH=" + str(label)
@@ -267,41 +171,16 @@ class LibTorchConfigNode(TreeConfigNode):
         self.props["is_libtorch"] = node_name
 
     def child_constructor(self):
-        return ExperimentalFeatureConfigNode
+        return ImportantConfigNode
 
+class AndroidAbiConfigNode(TreeConfigNode):
 
-class CudaGccOverrideConfigNode(TreeConfigNode):
     def init2(self, node_name):
-        self.props["cuda_gcc_override"] = node_name
-
-    def child_constructor(self):
-        return ExperimentalFeatureConfigNode
-
-
-class BuildOnlyConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["build_only"] = node_name
-
-    def child_constructor(self):
-        return ExperimentalFeatureConfigNode
-
-
-class ShardTestConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["shard_test"] = node_name
+        self.props["android_abi"] = node_name
 
     def child_constructor(self):
         return ImportantConfigNode
 
-
-class CoverageConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["is_coverage"] = node_name
-
-    def child_constructor(self):
-        return ExperimentalFeatureConfigNode
-
-
 class ImportantConfigNode(TreeConfigNode):
     def modify_label(self, label):
         return "IMPORTANT=" + str(label)
@@ -314,6 +193,7 @@ class ImportantConfigNode(TreeConfigNode):
 
 
 class XenialCompilerConfigNode(TreeConfigNode):
+
     def modify_label(self, label):
         return label or "<unspecified>"
 
@@ -326,19 +206,6 @@ class XenialCompilerConfigNode(TreeConfigNode):
         return XenialCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
 
 
-class BionicCompilerConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return label or "<unspecified>"
-
-    def init2(self, node_name):
-        self.props["compiler_name"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-
-        return BionicCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
-
-
 class XenialCompilerVersionConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["compiler_version"] = node_name
@@ -346,12 +213,3 @@ class XenialCompilerVersionConfigNode(TreeConfigNode):
     # noinspection PyMethodMayBeStatic
     def child_constructor(self):
         return PyVerConfigNode
-
-
-class BionicCompilerVersionConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["compiler_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return PyVerConfigNode

.circleci/cimodel/data/pytorch_build_definitions.py

@@ -1,13 +1,19 @@
 from collections import OrderedDict
-from dataclasses import dataclass, field
-from typing import List, Optional
 
+from cimodel.data.pytorch_build_data import TopLevelNode, CONFIG_TREE_DATA
 import cimodel.data.dimensions as dimensions
 import cimodel.lib.conf_tree as conf_tree
 import cimodel.lib.miniutils as miniutils
-from cimodel.data.pytorch_build_data import CONFIG_TREE_DATA, TopLevelNode
-from cimodel.data.simple.util.branch_filters import gen_filter_dict, RC_PATTERN
-from cimodel.data.simple.util.docker_constants import gen_docker_image
+
+from dataclasses import dataclass, field
+from typing import List, Optional
+
+
+DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/"
+
+# ARE YOU EDITING THIS NUMBER?  MAKE SURE YOU READ THE GUIDANCE AT THE
+# TOP OF .circleci/config.yml
+DOCKER_IMAGE_VERSION = "f990c76a-a798-42bb-852f-5be5006f8026"
 
 
 @dataclass
@@ -17,25 +23,18 @@ class Conf:
     parms_list_ignored_for_docker_image: Optional[List[str]] = None
     pyver: Optional[str] = None
     cuda_version: Optional[str] = None
-    rocm_version: Optional[str] = None
     # TODO expand this to cover all the USE_* that we want to test for
     #  tesnrorrt, leveldb, lmdb, redis, opencv, mkldnn, ideep, etc.
     # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453608)
     is_xla: bool = False
-    is_vulkan: bool = False
-    is_pure_torch: bool = False
     restrict_phases: Optional[List[str]] = None
     gpu_resource: Optional[str] = None
     dependent_tests: List = field(default_factory=list)
-    parent_build: Optional["Conf"] = None
+    parent_build: Optional['Conf'] = None
     is_libtorch: bool = False
     is_important: bool = False
     parallel_backend: Optional[str] = None
 
-    @staticmethod
-    def is_test_phase(phase):
-        return "test" in phase
-
     # TODO: Eliminate the special casing for docker paths
     # In the short term, we *will* need to support special casing as docker images are merged for caffe2 and pytorch
     def get_parms(self, for_docker):
@@ -47,47 +46,31 @@ class Conf:
         leading.append("pytorch")
         if self.is_xla and not for_docker:
             leading.append("xla")
-        if self.is_vulkan and not for_docker:
-            leading.append("vulkan")
         if self.is_libtorch and not for_docker:
             leading.append("libtorch")
-        if self.is_pure_torch and not for_docker:
-            leading.append("pure_torch")
         if self.parallel_backend is not None and not for_docker:
             leading.append(self.parallel_backend)
 
         cuda_parms = []
         if self.cuda_version:
-            cudnn = "cudnn8" if self.cuda_version.startswith("11.") else "cudnn7"
-            cuda_parms.extend(["cuda" + self.cuda_version, cudnn])
-        if self.rocm_version:
-            cuda_parms.extend([f"rocm{self.rocm_version}"])
+            cuda_parms.extend(["cuda" + self.cuda_version, "cudnn7"])
         result = leading + ["linux", self.distro] + cuda_parms + self.parms
-        if not for_docker and self.parms_list_ignored_for_docker_image is not None:
+        if (not for_docker and self.parms_list_ignored_for_docker_image is not None):
             result = result + self.parms_list_ignored_for_docker_image
         return result
 
     def gen_docker_image_path(self):
-        parms_source = self.parent_build or self
-        base_build_env_name = "-".join(parms_source.get_parms(True))
-        image_name, _ = gen_docker_image(base_build_env_name)
-        return miniutils.quote(image_name)
 
-    def gen_docker_image_requires(self):
         parms_source = self.parent_build or self
         base_build_env_name = "-".join(parms_source.get_parms(True))
-        _, requires = gen_docker_image(base_build_env_name)
-        return miniutils.quote(requires)
+
+        return miniutils.quote(DOCKER_IMAGE_PATH_BASE + base_build_env_name + ":" + str(DOCKER_IMAGE_VERSION))
 
     def get_build_job_name_pieces(self, build_or_test):
         return self.get_parms(False) + [build_or_test]
 
     def gen_build_name(self, build_or_test):
-        return (
-            ("_".join(map(str, self.get_build_job_name_pieces(build_or_test))))
-            .replace(".", "_")
-            .replace("-", "_")
-        )
+        return ("_".join(map(str, self.get_build_job_name_pieces(build_or_test)))).replace(".", "_").replace("-", "_")
 
     def get_dependents(self):
         return self.dependent_tests or []
@@ -99,26 +82,22 @@ class Conf:
         build_env_name = "-".join(map(str, build_job_name_pieces))
         parameters["build_environment"] = miniutils.quote(build_env_name)
         parameters["docker_image"] = self.gen_docker_image_path()
-        if Conf.is_test_phase(phase) and self.gpu_resource:
+        if phase == "test" and self.gpu_resource:
             parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
-        if Conf.is_test_phase(phase):
+        if phase == "test":
             resource_class = "large"
             if self.gpu_resource:
                 resource_class = "gpu." + self.gpu_resource
-            if self.rocm_version is not None:
-                resource_class = "pytorch/amd-gpu"
             parameters["resource_class"] = resource_class
-        if phase == "build" and self.rocm_version is not None:
-            parameters["resource_class"] = "xlarge"
-        if hasattr(self, 'filters'):
-            parameters['filters'] = self.filters
         return parameters
 
     def gen_workflow_job(self, phase):
+        # All jobs require the setup job
         job_def = OrderedDict()
         job_def["name"] = self.gen_build_name(phase)
+        job_def["requires"] = ["setup"]
 
-        if Conf.is_test_phase(phase):
+        if phase == "test":
 
             # TODO When merging the caffe2 and pytorch jobs, it might be convenient for a while to make a
             #  caffe2 test job dependent on a pytorch build job. This way we could quickly dedup the repeated
@@ -126,63 +105,43 @@ class Conf:
             #  pytorch build job (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259452641)
 
             dependency_build = self.parent_build or self
-            job_def["requires"] = [dependency_build.gen_build_name("build")]
+            job_def["requires"].append(dependency_build.gen_build_name("build"))
             job_name = "pytorch_linux_test"
         else:
             job_name = "pytorch_linux_build"
-            job_def["requires"] = [self.gen_docker_image_requires()]
+
 
         if not self.is_important:
-            job_def["filters"] = gen_filter_dict()
+            # If you update this, update
+            # caffe2_build_definitions.py too
+            job_def["filters"] = {"branches": {"only": ["master", r"/ci-all\/.*/"]}}
         job_def.update(self.gen_workflow_params(phase))
 
-        return {job_name: job_def}
+        return {job_name : job_def}
 
 
 # TODO This is a hack to special case some configs just for the workflow list
 class HiddenConf(object):
-    def __init__(self, name, parent_build=None, filters=None):
+    def __init__(self, name, parent_build=None):
         self.name = name
         self.parent_build = parent_build
-        self.filters = filters
 
     def gen_workflow_job(self, phase):
-        return {
-            self.gen_build_name(phase): {
-                "requires": [self.parent_build.gen_build_name("build")],
-                "filters": self.filters,
-            }
-        }
+        return {self.gen_build_name(phase): {"requires": [self.parent_build.gen_build_name("build")]}}
 
     def gen_build_name(self, _):
         return self.name
 
-class DocPushConf(object):
-    def __init__(self, name, parent_build=None, branch="master"):
-        self.name = name
-        self.parent_build = parent_build
-        self.branch = branch
-
-    def gen_workflow_job(self, phase):
-        return {
-            "pytorch_doc_push": {
-                "name": self.name,
-                "branch": self.branch,
-                "requires": [self.parent_build],
-                "context": "org-member",
-                "filters": gen_filter_dict(branches_list=["nightly"],
-                                           tags_list=RC_PATTERN)
-            }
-        }
 
 # TODO Convert these to graph nodes
 def gen_dependent_configs(xenial_parent_config):
 
     extra_parms = [
         (["multigpu"], "large"),
-        (["nogpu", "NO_AVX2"], None),
-        (["nogpu", "NO_AVX"], None),
+        (["NO_AVX2"], "medium"),
+        (["NO_AVX", "NO_AVX2"], "medium"),
         (["slow"], "medium"),
+        (["nogpu"], None),
     ]
 
     configs = []
@@ -191,60 +150,24 @@ def gen_dependent_configs(xenial_parent_config):
         c = Conf(
             xenial_parent_config.distro,
             ["py3"] + parms,
-            pyver=xenial_parent_config.pyver,
+            pyver="3.6",
             cuda_version=xenial_parent_config.cuda_version,
             restrict_phases=["test"],
             gpu_resource=gpu,
             parent_build=xenial_parent_config,
-            is_important=False,
+            is_important=xenial_parent_config.is_important,
         )
 
         configs.append(c)
 
     return configs
 
-
 def gen_docs_configs(xenial_parent_config):
     configs = []
 
-    configs.append(
-        HiddenConf(
-            "pytorch_python_doc_build",
-            parent_build=xenial_parent_config,
-            filters=gen_filter_dict(branches_list=r"/.*/",
-                                    tags_list=RC_PATTERN),
-        )
-    )
-    configs.append(
-        DocPushConf(
-            "pytorch_python_doc_push",
-            parent_build="pytorch_python_doc_build",
-            branch="site",
-        )
-    )
+    for x in ["pytorch_python_doc_push", "pytorch_cpp_doc_push"]:
+        configs.append(HiddenConf(x, parent_build=xenial_parent_config))
 
-    configs.append(
-        HiddenConf(
-            "pytorch_cpp_doc_build",
-            parent_build=xenial_parent_config,
-            filters=gen_filter_dict(branches_list=r"/.*/",
-                                    tags_list=RC_PATTERN),
-        )
-    )
-    configs.append(
-        DocPushConf(
-            "pytorch_cpp_doc_push",
-            parent_build="pytorch_cpp_doc_build",
-            branch="master",
-        )
-    )
-
-    configs.append(
-        HiddenConf(
-            "pytorch_doc_test",
-            parent_build=xenial_parent_config
-        )
-    )
     return configs
 
 
@@ -264,18 +187,13 @@ def instantiate_configs():
 
     root = get_root()
     found_configs = conf_tree.dfs(root)
+    restrict_phases = None
     for fc in found_configs:
 
-        restrict_phases = None
         distro_name = fc.find_prop("distro_name")
         compiler_name = fc.find_prop("compiler_name")
         compiler_version = fc.find_prop("compiler_version")
         is_xla = fc.find_prop("is_xla") or False
-        is_asan = fc.find_prop("is_asan") or False
-        is_coverage = fc.find_prop("is_coverage") or False
-        is_onnx = fc.find_prop("is_onnx") or False
-        is_pure_torch = fc.find_prop("is_pure_torch") or False
-        is_vulkan = fc.find_prop("is_vulkan") or False
         parms_list_ignored_for_docker_image = []
 
         python_version = None
@@ -286,14 +204,9 @@ def instantiate_configs():
             parms_list = ["py" + fc.find_prop("pyver")]
 
         cuda_version = None
-        rocm_version = None
         if compiler_name == "cuda":
             cuda_version = fc.find_prop("compiler_version")
 
-        elif compiler_name == "rocm":
-            rocm_version = fc.find_prop("compiler_version")
-            restrict_phases = ["build", "test1", "test2", "caffe2_test"]
-
         elif compiler_name == "android":
             android_ndk_version = fc.find_prop("compiler_version")
             # TODO: do we need clang to compile host binaries like protoc?
@@ -307,36 +220,19 @@ def instantiate_configs():
             gcc_version = compiler_name + (fc.find_prop("compiler_version") or "")
             parms_list.append(gcc_version)
 
-        if is_asan:
-            parms_list.append("asan")
-            python_version = fc.find_prop("pyver")
-            parms_list[0] = fc.find_prop("abbreviated_pyver")
+            # TODO: This is a nasty special case
+            if compiler_name == "clang" and not is_xla:
+                parms_list.append("asan")
+                python_version = fc.find_prop("pyver")
+                parms_list[0] = fc.find_prop("abbreviated_pyver")
 
-        if is_coverage:
-            parms_list_ignored_for_docker_image.append("coverage")
-            python_version = fc.find_prop("pyver")
-
-        if is_onnx:
-            parms_list.append("onnx")
-            python_version = fc.find_prop("pyver")
-            parms_list[0] = fc.find_prop("abbreviated_pyver")
-            restrict_phases = ["build", "ort_test1", "ort_test2"]
-
-        if cuda_version:
-            cuda_gcc_version = fc.find_prop("cuda_gcc_override") or "gcc7"
-            parms_list.append(cuda_gcc_version)
+        if cuda_version in ["9.2", "10", "10.1", "10.2"]:
+            # TODO The gcc version is orthogonal to CUDA version?
+            parms_list.append("gcc7")
 
         is_libtorch = fc.find_prop("is_libtorch") or False
         is_important = fc.find_prop("is_important") or False
         parallel_backend = fc.find_prop("parallel_backend") or None
-        build_only = fc.find_prop("build_only") or False
-        shard_test = fc.find_prop("shard_test") or False
-        # TODO: fix pure_torch python test packaging issue.
-        if shard_test:
-            restrict_phases = ["build"] if restrict_phases is None else restrict_phases
-            restrict_phases.extend(["test1", "test2"])
-        if build_only or is_pure_torch:
-            restrict_phases = ["build"]
 
         gpu_resource = None
         if cuda_version and cuda_version != "10":
@@ -348,10 +244,7 @@ def instantiate_configs():
             parms_list_ignored_for_docker_image,
             python_version,
             cuda_version,
-            rocm_version,
             is_xla,
-            is_vulkan,
-            is_pure_torch,
             restrict_phases,
             gpu_resource,
             is_libtorch=is_libtorch,
@@ -361,33 +254,20 @@ def instantiate_configs():
 
         # run docs builds on "pytorch-linux-xenial-py3.6-gcc5.4". Docs builds
         # should run on a CPU-only build that runs on all PRs.
-        # XXX should this be updated to a more modern build? Projects are
-        #     beginning to drop python3.6
-        if (
-            distro_name == "xenial"
-            and fc.find_prop("pyver") == "3.6"
-            and cuda_version is None
-            and parallel_backend is None
-            and not is_vulkan
-            and not is_pure_torch
-            and compiler_name == "gcc"
-            and fc.find_prop("compiler_version") == "5.4"
-        ):
-            c.filters = gen_filter_dict(branches_list=r"/.*/",
-                                        tags_list=RC_PATTERN)
+        if distro_name == 'xenial' and fc.find_prop("pyver") == '3.6' \
+                and cuda_version is None \
+                and parallel_backend is None \
+                and compiler_name == 'gcc' \
+                and fc.find_prop('compiler_version') == '5.4':
             c.dependent_tests = gen_docs_configs(c)
 
-        if cuda_version == "10.2" and python_version == "3.6" and not is_libtorch:
+        if cuda_version == "10.1" and python_version == "3.6" and not is_libtorch:
             c.dependent_tests = gen_dependent_configs(c)
 
-        if (
-            compiler_name == "gcc"
-            and compiler_version == "5.4"
-            and not is_libtorch
-            and not is_vulkan
-            and not is_pure_torch
-            and parallel_backend is None
-        ):
+        if (compiler_name == "gcc"
+                and compiler_version == "5.4"
+                and not is_libtorch
+                and parallel_backend is None):
             bc_breaking_check = Conf(
                 "backward-compatibility-check",
                 [],
@@ -416,7 +296,7 @@ def get_workflow_jobs():
         for phase in phases:
 
             # TODO why does this not have a test?
-            if Conf.is_test_phase(phase) and conf_options.cuda_version == "10":
+            if phase == "test" and conf_options.cuda_version == "10":
                 continue
 
             x.append(conf_options.gen_workflow_job(phase))

.circleci/cimodel/data/simple/anaconda_prune_defintions.py

@@ -1,28 +0,0 @@
-from collections import OrderedDict
-
-from cimodel.data.simple.util.branch_filters import gen_filter_dict
-from cimodel.lib.miniutils import quote
-
-
-CHANNELS_TO_PRUNE = ["pytorch-nightly", "pytorch-test"]
-PACKAGES_TO_PRUNE = "pytorch torchvision torchaudio torchtext ignite torchcsprng"
-
-
-def gen_workflow_job(channel: str):
-    return OrderedDict(
-        {
-            "anaconda_prune": OrderedDict(
-                {
-                    "name": f"anaconda-prune-{channel}",
-                    "context": quote("org-member"),
-                    "packages": quote(PACKAGES_TO_PRUNE),
-                    "channel": channel,
-                    "filters": gen_filter_dict(branches_list=["postnightly"]),
-                }
-            )
-        }
-    )
-
-
-def get_workflow_jobs():
-    return [gen_workflow_job(channel) for channel in CHANNELS_TO_PRUNE]

.circleci/cimodel/data/simple/android_definitions.py

@@ -1,105 +0,0 @@
-import cimodel.data.simple.util.branch_filters as branch_filters
-from cimodel.data.simple.util.docker_constants import (
-    DOCKER_IMAGE_NDK, DOCKER_REQUIREMENT_NDK
-)
-
-
-class AndroidJob:
-    def __init__(self,
-                 variant,
-                 template_name,
-                 is_master_only=True):
-
-        self.variant = variant
-        self.template_name = template_name
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-
-        base_name_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "android",
-            "ndk",
-            "r19c",
-        ] + self.variant + [
-            "build",
-        ]
-
-        full_job_name = "_".join(base_name_parts)
-        build_env_name = "-".join(base_name_parts)
-
-        props_dict = {
-            "name": full_job_name,
-            "build_environment": "\"{}\"".format(build_env_name),
-            "docker_image": "\"{}\"".format(DOCKER_IMAGE_NDK),
-            "requires": [DOCKER_REQUIREMENT_NDK]
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = branch_filters.gen_filter_dict(branch_filters.NON_PR_BRANCH_LIST)
-
-        return [{self.template_name: props_dict}]
-
-
-class AndroidGradleJob:
-    def __init__(self,
-                 job_name,
-                 template_name,
-                 dependencies,
-                 is_master_only=True,
-                 is_pr_only=False):
-
-        self.job_name = job_name
-        self.template_name = template_name
-        self.dependencies = dependencies
-        self.is_master_only = is_master_only
-        self.is_pr_only = is_pr_only
-
-    def gen_tree(self):
-
-        props_dict = {
-            "name": self.job_name,
-            "requires": self.dependencies,
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = branch_filters.gen_filter_dict(branch_filters.NON_PR_BRANCH_LIST)
-        elif self.is_pr_only:
-            props_dict["filters"] = branch_filters.gen_filter_dict(branch_filters.PR_BRANCH_LIST)
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    AndroidJob(["x86_32"], "pytorch_linux_build", is_master_only=False),
-    AndroidJob(["x86_64"], "pytorch_linux_build"),
-    AndroidJob(["arm", "v7a"], "pytorch_linux_build"),
-    AndroidJob(["arm", "v8a"], "pytorch_linux_build"),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32",
-        "pytorch_android_gradle_build-x86_32",
-        ["pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build"],
-        is_master_only=False,
-        is_pr_only=True),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single",
-        "pytorch_android_gradle_custom_build_single",
-        [DOCKER_REQUIREMENT_NDK],
-        is_master_only=False,
-        is_pr_only=True),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build",
-        "pytorch_android_gradle_build",
-        ["pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build"]),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/bazel_definitions.py

@@ -1,69 +0,0 @@
-from cimodel.data.simple.util.docker_constants import (
-    DOCKER_IMAGE_GCC7,
-    DOCKER_REQUIREMENT_GCC7
-)
-
-
-def gen_job_name(phase):
-    job_name_parts = [
-        "pytorch",
-        "bazel",
-        phase,
-    ]
-
-    return "_".join(job_name_parts)
-
-
-class BazelJob:
-    def __init__(self, phase, extra_props=None):
-        self.phase = phase
-        self.extra_props = extra_props or {}
-
-    def gen_tree(self):
-
-        template_parts = [
-            "pytorch",
-            "linux",
-            "bazel",
-            self.phase,
-        ]
-
-        build_env_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3.6",
-            "gcc7",
-            "bazel",
-            self.phase,
-        ]
-
-        full_job_name = gen_job_name(self.phase)
-        build_env_name = "-".join(build_env_parts)
-
-        extra_requires = (
-            [gen_job_name("build")] if self.phase == "test" else
-            [DOCKER_REQUIREMENT_GCC7]
-        )
-
-        props_dict = {
-            "build_environment": build_env_name,
-            "docker_image": DOCKER_IMAGE_GCC7,
-            "name": full_job_name,
-            "requires": extra_requires,
-        }
-
-        props_dict.update(self.extra_props)
-
-        template_name = "_".join(template_parts)
-        return [{template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    BazelJob("build", {"resource_class": "large"}),
-    BazelJob("test"),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/binary_smoketest.py

@@ -1,193 +0,0 @@
-"""
-TODO: Refactor circleci/cimodel/data/binary_build_data.py to generate this file
-       instead of doing one offs here
- Binary builds (subset, to smoke test that they'll work)
-
- NB: If you modify this file, you need to also modify
- the binary_and_smoke_tests_on_pr variable in
- pytorch-ci-hud to adjust the allowed build list
- at https://github.com/ezyang/pytorch-ci-hud/blob/master/src/BuildHistoryDisplay.js
-
- Note:
- This binary build is currently broken, see https://github_com/pytorch/pytorch/issues/16710
- - binary_linux_conda_3_6_cu90_devtoolset7_build
- - binary_linux_conda_3_6_cu90_devtoolset7_test
-
- TODO
- we should test a libtorch cuda build, but they take too long
- - binary_linux_libtorch_3_6m_cu90_devtoolset7_static-without-deps_build
-"""
-
-import cimodel.lib.miniutils as miniutils
-import cimodel.data.simple.util.branch_filters
-
-
-class SmoketestJob:
-    def __init__(self,
-                 template_name,
-                 build_env_parts,
-                 docker_image,
-                 job_name,
-                 is_master_only=False,
-                 requires=None,
-                 has_libtorch_variant=False,
-                 extra_props=None):
-
-        self.template_name = template_name
-        self.build_env_parts = build_env_parts
-        self.docker_image = docker_image
-        self.job_name = job_name
-        self.is_master_only = is_master_only
-        self.requires = requires or []
-        self.has_libtorch_variant = has_libtorch_variant
-        self.extra_props = extra_props or {}
-
-    def gen_tree(self):
-
-        props_dict = {
-            "build_environment": " ".join(self.build_env_parts),
-            "name": self.job_name,
-            "requires": self.requires,
-        }
-
-        if self.docker_image:
-            props_dict["docker_image"] = self.docker_image
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        if self.has_libtorch_variant:
-            props_dict["libtorch_variant"] = "shared-with-deps"
-
-        props_dict.update(self.extra_props)
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    SmoketestJob(
-        "binary_linux_build",
-        ["manywheel", "3.7m", "cu102", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_manywheel_3_7m_cu102_devtoolset7_build",
-        is_master_only=True,
-    ),
-    SmoketestJob(
-        "binary_linux_build",
-        ["libtorch", "3.7m", "cpu", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build",
-        is_master_only=False,
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_linux_build",
-        ["libtorch", "3.7m", "cpu", "gcc5.4_cxx11-abi"],
-        "pytorch/pytorch-binary-docker-image-ubuntu16.04:latest",
-        "binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build",
-        is_master_only=False,
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_mac_build",
-        ["wheel", "3.7", "cpu"],
-        None,
-        "binary_macos_wheel_3_7_cpu_build",
-        is_master_only=True,
-    ),
-    # This job has an average run time of 3 hours o.O
-    # Now only running this on master to reduce overhead
-    SmoketestJob(
-        "binary_mac_build",
-        ["libtorch", "3.7", "cpu"],
-        None,
-        "binary_macos_libtorch_3_7_cpu_build",
-        is_master_only=True,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["libtorch", "3.7", "cpu", "debug"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_debug_build",
-        is_master_only=False,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["libtorch", "3.7", "cpu", "release"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_release_build",
-        is_master_only=False,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["wheel", "3.7", "cu102"],
-        None,
-        "binary_windows_wheel_3_7_cu102_build",
-        is_master_only=True,
-    ),
-
-    SmoketestJob(
-        "binary_windows_test",
-        ["libtorch", "3.7", "cpu", "debug"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_debug_test",
-        is_master_only=False,
-        requires=["binary_windows_libtorch_3_7_cpu_debug_build"],
-    ),
-    SmoketestJob(
-        "binary_windows_test",
-        ["libtorch", "3.7", "cpu", "release"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_release_test",
-        is_master_only=False,
-        requires=["binary_windows_libtorch_3_7_cpu_release_build"],
-    ),
-    SmoketestJob(
-        "binary_windows_test",
-        ["wheel", "3.7", "cu102"],
-        None,
-        "binary_windows_wheel_3_7_cu102_test",
-        is_master_only=True,
-        requires=["binary_windows_wheel_3_7_cu102_build"],
-        extra_props={
-            "executor": "windows-with-nvidia-gpu",
-        },
-    ),
-
-
-
-    SmoketestJob(
-        "binary_linux_test",
-        ["manywheel", "3.7m", "cu102", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_manywheel_3_7m_cu102_devtoolset7_test",
-        is_master_only=True,
-        requires=["binary_linux_manywheel_3_7m_cu102_devtoolset7_build"],
-        extra_props={
-            "resource_class": "gpu.medium",
-            "use_cuda_docker_runtime": miniutils.quote((str(1))),
-        },
-    ),
-    SmoketestJob(
-        "binary_linux_test",
-        ["libtorch", "3.7m", "cpu", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_test",
-        is_master_only=False,
-        requires=["binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build"],
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_linux_test",
-        ["libtorch", "3.7m", "cpu", "gcc5.4_cxx11-abi"],
-        "pytorch/pytorch-binary-docker-image-ubuntu16.04:latest",
-        "binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_test",
-        is_master_only=False,
-        requires=["binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build"],
-        has_libtorch_variant=True,
-    ),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/docker_definitions.py

@@ -1,55 +0,0 @@
-from collections import OrderedDict
-
-from cimodel.lib.miniutils import quote
-from cimodel.data.simple.util.branch_filters import gen_filter_dict, RC_PATTERN
-
-
-# TODO: make this generated from a matrix rather than just a static list
-IMAGE_NAMES = [
-    "pytorch-linux-bionic-cuda11.1-cudnn8-py3.6-gcc9",
-    "pytorch-linux-bionic-cuda11.1-cudnn8-py3.8-gcc9",
-    "pytorch-linux-bionic-cuda11.0-cudnn8-py3.6-gcc9",
-    "pytorch-linux-bionic-cuda11.0-cudnn8-py3.8-gcc9",
-    "pytorch-linux-bionic-cuda10.2-cudnn7-py3.8-gcc9",
-    "pytorch-linux-bionic-py3.6-clang9",
-    "pytorch-linux-bionic-cuda10.2-cudnn7-py3.6-clang9",
-    "pytorch-linux-bionic-py3.8-gcc9",
-    "pytorch-linux-xenial-cuda10-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda10.1-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda11.0-cudnn8-py3-gcc7",
-    "pytorch-linux-xenial-cuda11.1-cudnn8-py3-gcc7",
-    "pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc5.4",
-    "pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-py3-clang5-android-ndk-r19c",
-    "pytorch-linux-xenial-py3-clang5-asan",
-    "pytorch-linux-xenial-py3-clang7-onnx",
-    "pytorch-linux-xenial-py3.8",
-    "pytorch-linux-xenial-py3.6-clang7",
-    "pytorch-linux-xenial-py3.6-gcc5.4",  # this one is used in doc builds
-    "pytorch-linux-xenial-py3.6-gcc7.2",
-    "pytorch-linux-xenial-py3.6-gcc7",
-    "pytorch-linux-bionic-rocm3.9-py3.6",
-    "pytorch-linux-bionic-rocm3.10-py3.6",
-]
-
-
-def get_workflow_jobs():
-    """Generates a list of docker image build definitions"""
-    ret = []
-    for image_name in IMAGE_NAMES:
-        parameters = OrderedDict({
-            "name": quote(f"docker-{image_name}"),
-            "image_name": quote(image_name),
-        })
-        if image_name == "pytorch-linux-xenial-py3.6-gcc5.4":
-            # pushing documentation on tags requires CircleCI to also
-            # build all the dependencies on tags, including this docker image
-            parameters['filters'] = gen_filter_dict(branches_list=r"/.*/",
-                                                    tags_list=RC_PATTERN)
-        ret.append(OrderedDict(
-            {
-                "docker_build_job": parameters
-            }
-        ))
-    return ret

.circleci/cimodel/data/simple/ge_config_tests.py

@@ -1,78 +0,0 @@
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.versions import MultiPartVersion, CudaVersion
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_BASIC, DOCKER_IMAGE_CUDA_10_2
-
-
-class GeConfigTestJob:
-    def __init__(self,
-                 py_version,
-                 gcc_version,
-                 cuda_version,
-                 variant_parts,
-                 extra_requires,
-                 use_cuda_docker=False,
-                 build_env_override=None):
-
-        self.py_version = py_version
-        self.gcc_version = gcc_version
-        self.cuda_version = cuda_version
-        self.variant_parts = variant_parts
-        self.extra_requires = extra_requires
-        self.use_cuda_docker = use_cuda_docker
-        self.build_env_override = build_env_override
-
-    def get_all_parts(self, with_dots):
-
-        maybe_py_version = self.py_version.render_dots_or_parts(with_dots) if self.py_version else []
-        maybe_gcc_version = self.gcc_version.render_dots_or_parts(with_dots) if self.gcc_version else []
-        maybe_cuda_version = self.cuda_version.render_dots_or_parts(with_dots) if self.cuda_version else []
-
-        common_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-        ] + maybe_cuda_version + maybe_py_version + maybe_gcc_version
-
-        return common_parts + self.variant_parts
-
-    def gen_tree(self):
-
-        resource_class = "gpu.medium" if self.use_cuda_docker else "large"
-        docker_image = DOCKER_IMAGE_CUDA_10_2 if self.use_cuda_docker else DOCKER_IMAGE_BASIC
-        full_name = "_".join(self.get_all_parts(False))
-        build_env = self.build_env_override or "-".join(self.get_all_parts(True))
-
-        props_dict = {
-            "name": full_name,
-            "build_environment": build_env,
-            "requires": self.extra_requires,
-            "resource_class": resource_class,
-            "docker_image": docker_image,
-        }
-
-        if self.use_cuda_docker:
-            props_dict["use_cuda_docker_runtime"] = miniutils.quote(str(1))
-
-        return [{"pytorch_linux_test": props_dict}]
-
-
-WORKFLOW_DATA = [
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["jit_legacy", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"]),
-    GeConfigTestJob(
-        None,
-        None,
-        CudaVersion(10, 2),
-        ["cudnn7", "py3", "jit_legacy", "test"],
-        ["pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_build"],
-        use_cuda_docker=True,
-    ),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/ios_definitions.py

@@ -1,72 +0,0 @@
-from cimodel.data.simple.util.versions import MultiPartVersion
-import cimodel.lib.miniutils as miniutils
-
-XCODE_VERSION = MultiPartVersion([12, 0, 0])
-
-
-class ArchVariant:
-    def __init__(self, name, custom_build_name=""):
-        self.name = name
-        self.custom_build_name = custom_build_name
-
-    def render(self):
-        extra_parts = [self.custom_build_name] if len(self.custom_build_name) > 0 else []
-        return "_".join([self.name] + extra_parts)
-
-
-def get_platform(arch_variant_name):
-    return "SIMULATOR" if arch_variant_name == "x86_64" else "OS"
-
-
-class IOSJob:
-    def __init__(self, xcode_version, arch_variant, is_org_member_context=True, extra_props=None):
-        self.xcode_version = xcode_version
-        self.arch_variant = arch_variant
-        self.is_org_member_context = is_org_member_context
-        self.extra_props = extra_props
-
-    def gen_name_parts(self, with_version_dots):
-
-        version_parts = self.xcode_version.render_dots_or_parts(with_version_dots)
-        build_variant_suffix = "_".join([self.arch_variant.render(), "build"])
-
-        return [
-            "pytorch",
-            "ios",
-        ] + version_parts + [
-            build_variant_suffix,
-        ]
-
-    def gen_job_name(self):
-        return "_".join(self.gen_name_parts(False))
-
-    def gen_tree(self):
-
-        platform_name = get_platform(self.arch_variant.name)
-
-        props_dict = {
-            "build_environment": "-".join(self.gen_name_parts(True)),
-            "ios_arch": self.arch_variant.name,
-            "ios_platform": platform_name,
-            "name": self.gen_job_name(),
-        }
-
-        if self.is_org_member_context:
-            props_dict["context"] = "org-member"
-
-        if self.extra_props:
-            props_dict.update(self.extra_props)
-
-        return [{"pytorch_ios_build": props_dict}]
-
-
-WORKFLOW_DATA = [
-    IOSJob(XCODE_VERSION, ArchVariant("x86_64"), is_org_member_context=False),
-    IOSJob(XCODE_VERSION, ArchVariant("arm64")),
-    IOSJob(XCODE_VERSION, ArchVariant("arm64", "metal"), extra_props={"use_metal": miniutils.quote(str(int(True)))}),
-    IOSJob(XCODE_VERSION, ArchVariant("arm64", "custom"), extra_props={"op_list": "mobilenetv2.yaml"}),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/macos_definitions.py

@@ -1,28 +0,0 @@
-class MacOsJob:
-    def __init__(self, os_version, is_test=False):
-        self.os_version = os_version
-        self.is_test = is_test
-
-    def gen_tree(self):
-        non_phase_parts = ["pytorch", "macos", self.os_version, "py3"]
-
-        phase_name = "test" if self.is_test else "build"
-
-        full_job_name = "_".join(non_phase_parts + [phase_name])
-
-        test_build_dependency = "_".join(non_phase_parts + ["build"])
-        extra_dependencies = [test_build_dependency] if self.is_test else []
-        job_dependencies = extra_dependencies
-
-        # Yes we name the job after itself, it needs a non-empty value in here
-        # for the YAML output to work.
-        props_dict = {"requires": job_dependencies, "name": full_job_name}
-
-        return [{full_job_name: props_dict}]
-
-
-WORKFLOW_DATA = [MacOsJob("10_13"), MacOsJob("10_13", True)]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/mobile_definitions.py

@@ -1,80 +0,0 @@
-"""
-PyTorch Mobile PR builds (use linux host toolchain + mobile build options)
-"""
-
-import cimodel.lib.miniutils as miniutils
-import cimodel.data.simple.util.branch_filters
-from cimodel.data.simple.util.docker_constants import (
-    DOCKER_IMAGE_ASAN,
-    DOCKER_REQUIREMENT_ASAN,
-    DOCKER_IMAGE_NDK,
-    DOCKER_REQUIREMENT_NDK
-)
-
-
-class MobileJob:
-    def __init__(
-            self,
-            docker_image,
-            docker_requires,
-            variant_parts,
-            is_master_only=False):
-        self.docker_image = docker_image
-        self.docker_requires = docker_requires
-        self.variant_parts = variant_parts
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-        non_phase_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "mobile",
-        ] + self.variant_parts
-
-        full_job_name = "_".join(non_phase_parts)
-        build_env_name = "-".join(non_phase_parts)
-
-        props_dict = {
-            "build_environment": build_env_name,
-            "build_only": miniutils.quote(str(int(True))),
-            "docker_image": self.docker_image,
-            "requires": self.docker_requires,
-            "name": full_job_name,
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{"pytorch_linux_build": props_dict}]
-
-
-WORKFLOW_DATA = [
-    MobileJob(
-        DOCKER_IMAGE_ASAN,
-        [DOCKER_REQUIREMENT_ASAN],
-        ["build"]
-    ),
-
-    # Use LLVM-DEV toolchain in android-ndk-r19c docker image
-    MobileJob(
-        DOCKER_IMAGE_NDK,
-        [DOCKER_REQUIREMENT_NDK],
-        ["custom", "build", "dynamic"]
-    ),
-
-    # Use LLVM-DEV toolchain in android-ndk-r19c docker image
-    # Most of this CI is already covered by "mobile-custom-build-dynamic" job
-    MobileJob(
-        DOCKER_IMAGE_NDK,
-        [DOCKER_REQUIREMENT_NDK],
-        ["code", "analysis"],
-        True
-    ),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/nightly_android.py

@@ -1,77 +0,0 @@
-from cimodel.data.simple.util.docker_constants import (
-    DOCKER_IMAGE_NDK,
-    DOCKER_REQUIREMENT_NDK
-)
-
-
-class AndroidNightlyJob:
-    def __init__(self,
-                 variant,
-                 template_name,
-                 extra_props=None,
-                 with_docker=True,
-                 requires=None,
-                 no_build_suffix=False):
-
-        self.variant = variant
-        self.template_name = template_name
-        self.extra_props = extra_props or {}
-        self.with_docker = with_docker
-        self.requires = requires
-        self.no_build_suffix = no_build_suffix
-
-    def gen_tree(self):
-
-        base_name_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "android",
-            "ndk",
-            "r19c",
-        ] + self.variant
-
-        build_suffix = [] if self.no_build_suffix else ["build"]
-        full_job_name = "_".join(["nightly"] + base_name_parts + build_suffix)
-        build_env_name = "-".join(base_name_parts)
-
-        props_dict = {
-            "name": full_job_name,
-            "requires": self.requires,
-            "filters": {"branches": {"only": "nightly"}},
-        }
-
-        props_dict.update(self.extra_props)
-
-        if self.with_docker:
-            props_dict["docker_image"] = DOCKER_IMAGE_NDK
-            props_dict["build_environment"] = build_env_name
-
-        return [{self.template_name: props_dict}]
-
-BASE_REQUIRES = [DOCKER_REQUIREMENT_NDK]
-
-WORKFLOW_DATA = [
-    AndroidNightlyJob(["x86_32"], "pytorch_linux_build", requires=BASE_REQUIRES),
-    AndroidNightlyJob(["x86_64"], "pytorch_linux_build", requires=BASE_REQUIRES),
-    AndroidNightlyJob(["arm", "v7a"], "pytorch_linux_build", requires=BASE_REQUIRES),
-    AndroidNightlyJob(["arm", "v8a"], "pytorch_linux_build", requires=BASE_REQUIRES),
-    AndroidNightlyJob(["android_gradle"], "pytorch_android_gradle_build",
-                      with_docker=False,
-                      requires=[
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build"]),
-    AndroidNightlyJob(["x86_32_android_publish_snapshot"], "pytorch_android_publish_snapshot",
-                      extra_props={"context": "org-member"},
-                      with_docker=False,
-                      requires=["nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_android_gradle_build"],
-                      no_build_suffix=True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/nightly_ios.py

@@ -1,68 +0,0 @@
-import cimodel.data.simple.ios_definitions as ios_definitions
-
-
-class IOSNightlyJob:
-    def __init__(self,
-                 variant,
-                 is_upload=False):
-
-        self.variant = variant
-        self.is_upload = is_upload
-
-    def get_phase_name(self):
-        return "upload" if self.is_upload else "build"
-
-    def get_common_name_pieces(self, with_version_dots):
-
-        extra_name_suffix = [self.get_phase_name()] if self.is_upload else []
-
-        common_name_pieces = [
-            "ios",
-        ] + ios_definitions.XCODE_VERSION.render_dots_or_parts(with_version_dots) + [
-            "nightly",
-            self.variant,
-            "build",
-        ] + extra_name_suffix
-
-        return common_name_pieces
-
-    def gen_job_name(self):
-        return "_".join(["pytorch"] + self.get_common_name_pieces(False))
-
-    def gen_tree(self):
-        extra_requires = [x.gen_job_name() for x in BUILD_CONFIGS] if self.is_upload else []
-
-        props_dict = {
-            "build_environment": "-".join(["libtorch"] + self.get_common_name_pieces(True)),
-            "requires": extra_requires,
-            "context": "org-member",
-            "filters": {"branches": {"only": "nightly"}},
-        }
-
-        if not self.is_upload:
-            props_dict["ios_arch"] = self.variant
-            props_dict["ios_platform"] = ios_definitions.get_platform(self.variant)
-            props_dict["name"] = self.gen_job_name()
-
-        template_name = "_".join([
-            "binary",
-            "ios",
-            self.get_phase_name(),
-        ])
-
-        return [{template_name: props_dict}]
-
-
-BUILD_CONFIGS = [
-    IOSNightlyJob("x86_64"),
-    IOSNightlyJob("arm64"),
-]
-
-
-WORKFLOW_DATA = BUILD_CONFIGS + [
-    IOSNightlyJob("binary", is_upload=True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/util/branch_filters.py

@@ -1,27 +0,0 @@
-NON_PR_BRANCH_LIST = [
-    "master",
-    r"/ci-all\/.*/",
-    r"/release\/.*/",
-]
-
-PR_BRANCH_LIST = [
-    r"/gh\/.*\/head/",
-    r"/pull\/.*/",
-]
-
-RC_PATTERN = r"/v[0-9]+(\.[0-9]+)*-rc[0-9]+/"
-
-def gen_filter_dict(
-        branches_list=NON_PR_BRANCH_LIST,
-        tags_list=None
-):
-    """Generates a filter dictionary for use with CircleCI's job filter"""
-    filter_dict = {
-        "branches": {
-            "only": branches_list,
-        },
-    }
-
-    if tags_list is not None:
-        filter_dict["tags"] = {"only": tags_list}
-    return filter_dict

.circleci/cimodel/data/simple/util/docker_constants.py

@@ -1,33 +0,0 @@
-AWS_DOCKER_HOST = "308535385114.dkr.ecr.us-east-1.amazonaws.com"
-
-def gen_docker_image(container_type):
-    return (
-        "/".join([AWS_DOCKER_HOST, "pytorch", container_type]),
-        f"docker-{container_type}",
-    )
-
-def gen_docker_image_requires(image_name):
-    return [f"docker-{image_name}"]
-
-
-DOCKER_IMAGE_BASIC, DOCKER_REQUIREMENT_BASE = gen_docker_image(
-    "pytorch-linux-xenial-py3.6-gcc5.4"
-)
-
-DOCKER_IMAGE_CUDA_10_2, DOCKER_REQUIREMENT_CUDA_10_2 = gen_docker_image(
-    "pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7"
-)
-
-DOCKER_IMAGE_GCC7, DOCKER_REQUIREMENT_GCC7 = gen_docker_image(
-    "pytorch-linux-xenial-py3.6-gcc7"
-)
-
-
-def gen_mobile_docker(specifier):
-    container_type = "pytorch-linux-xenial-py3-clang5-" + specifier
-    return gen_docker_image(container_type)
-
-
-DOCKER_IMAGE_ASAN, DOCKER_REQUIREMENT_ASAN = gen_mobile_docker("asan")
-
-DOCKER_IMAGE_NDK, DOCKER_REQUIREMENT_NDK = gen_mobile_docker("android-ndk-r19c")

.circleci/cimodel/data/simple/util/versions.py

@@ -1,34 +0,0 @@
-class MultiPartVersion:
-    def __init__(self, parts, prefix=""):
-        self.parts = parts
-        self.prefix = prefix
-
-    def prefixed_parts(self):
-        """
-        Prepends the first element of the version list
-        with the prefix string.
-        """
-        if self.parts:
-            return [self.prefix + str(self.parts[0])] + [str(part) for part in self.parts[1:]]
-        else:
-            return [self.prefix]
-
-    def render_dots(self):
-        return ".".join(self.prefixed_parts())
-
-    def render_dots_or_parts(self, with_dots):
-        if with_dots:
-            return [self.render_dots()]
-        else:
-            return self.prefixed_parts()
-
-
-class CudaVersion(MultiPartVersion):
-    def __init__(self, major, minor):
-        self.major = major
-        self.minor = minor
-
-        super().__init__([self.major, self.minor], "cuda")
-
-    def __str__(self):
-        return f"{self.major}.{self.minor}"

.circleci/cimodel/data/windows_build_definitions.py

@@ -1,148 +0,0 @@
-import cimodel.data.simple.util.branch_filters
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.versions import CudaVersion
-
-
-class WindowsJob:
-    def __init__(
-        self,
-        test_index,
-        vscode_spec,
-        cuda_version,
-        force_on_cpu=False,
-        master_only_pred=lambda job: job.vscode_spec.year != 2019,
-    ):
-        self.test_index = test_index
-        self.vscode_spec = vscode_spec
-        self.cuda_version = cuda_version
-        self.force_on_cpu = force_on_cpu
-        self.master_only_pred = master_only_pred
-
-    def gen_tree(self):
-
-        base_phase = "build" if self.test_index is None else "test"
-        numbered_phase = (
-            base_phase if self.test_index is None else base_phase + str(self.test_index)
-        )
-
-        key_name = "_".join(["pytorch", "windows", base_phase])
-
-        cpu_forcing_name_parts = ["on", "cpu"] if self.force_on_cpu else []
-
-        target_arch = self.cuda_version.render_dots() if self.cuda_version else "cpu"
-
-        base_name_parts = [
-            "pytorch",
-            "windows",
-            self.vscode_spec.render(),
-            "py36",
-            target_arch,
-        ]
-
-        prerequisite_jobs = []
-        if base_phase == "test":
-            prerequisite_jobs.append("_".join(base_name_parts + ["build"]))
-
-        if self.cuda_version:
-            self.cudnn_version = 8 if self.cuda_version.major == 11 else 7
-
-        arch_env_elements = (
-            ["cuda" + str(self.cuda_version.major), "cudnn" + str(self.cudnn_version)]
-            if self.cuda_version
-            else ["cpu"]
-        )
-
-        build_environment_string = "-".join(
-            ["pytorch", "win"]
-            + self.vscode_spec.get_elements()
-            + arch_env_elements
-            + ["py3"]
-        )
-
-        is_running_on_cuda = bool(self.cuda_version) and not self.force_on_cpu
-
-        props_dict = {
-            "build_environment": build_environment_string,
-            "python_version": miniutils.quote("3.6"),
-            "vc_version": miniutils.quote(self.vscode_spec.dotted_version()),
-            "vc_year": miniutils.quote(str(self.vscode_spec.year)),
-            "vc_product": self.vscode_spec.get_product(),
-            "use_cuda": miniutils.quote(str(int(is_running_on_cuda))),
-            "requires": prerequisite_jobs,
-        }
-
-        if self.master_only_pred(self):
-            props_dict[
-                "filters"
-            ] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        name_parts = base_name_parts + cpu_forcing_name_parts + [numbered_phase]
-
-        if base_phase == "test":
-            test_name = "-".join(["pytorch", "windows", numbered_phase])
-            props_dict["test_name"] = test_name
-
-            if is_running_on_cuda:
-                props_dict["executor"] = "windows-with-nvidia-gpu"
-
-        props_dict["cuda_version"] = (
-            miniutils.quote(str(self.cuda_version))
-            if self.cuda_version
-            else "cpu"
-        )
-
-        props_dict["name"] = "_".join(name_parts)
-
-        return [{key_name: props_dict}]
-
-
-class VcSpec:
-    def __init__(self, year, version_elements=None, hide_version=False):
-        self.year = year
-        self.version_elements = version_elements or []
-        self.hide_version = hide_version
-
-    def get_elements(self):
-        if self.hide_version:
-            return [self.prefixed_year()]
-        return [self.prefixed_year()] + self.version_elements
-
-    def get_product(self):
-        return "Community" if self.year == 2019 else "BuildTools"
-
-    def dotted_version(self):
-        return ".".join(self.version_elements)
-
-    def prefixed_year(self):
-        return "vs" + str(self.year)
-
-    def render(self):
-        return "_".join(self.get_elements())
-
-def FalsePred(_):
-    return False
-
-def TruePred(_):
-    return True
-
-_VC2019 = VcSpec(2019)
-
-WORKFLOW_DATA = [
-    # VS2019 CUDA-10.1
-    WindowsJob(None, _VC2019, CudaVersion(10, 1)),
-    WindowsJob(1, _VC2019, CudaVersion(10, 1)),
-    WindowsJob(2, _VC2019, CudaVersion(10, 1)),
-    # VS2019 CUDA-11.1
-    WindowsJob(None, _VC2019, CudaVersion(11, 1)),
-    WindowsJob(1, _VC2019, CudaVersion(11, 1), master_only_pred=TruePred),
-    WindowsJob(2, _VC2019, CudaVersion(11, 1), master_only_pred=TruePred),
-    # VS2019 CPU-only
-    WindowsJob(None, _VC2019, None),
-    WindowsJob(1, _VC2019, None, master_only_pred=TruePred),
-    WindowsJob(2, _VC2019, None, master_only_pred=TruePred),
-    WindowsJob(1, _VC2019, CudaVersion(10, 1), force_on_cpu=True, master_only_pred=TruePred),
-]
-
-
-def get_windows_workflows():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/lib/miniyaml.py

@@ -1,7 +1,5 @@
 from collections import OrderedDict
 
-import cimodel.lib.miniutils as miniutils
-
 
 LIST_MARKER = "- "
 INDENTATION_WIDTH = 2
@@ -31,8 +29,7 @@ def render(fh, data, depth, is_list_member=False):
             tuples.sort()
 
         for i, (k, v) in enumerate(tuples):
-            if not v:
-                continue
+
             # If this dict is itself a list member, the first key gets prefixed with a list marker
             list_marker_prefix = LIST_MARKER if is_list_member and not i else ""
 
@@ -46,7 +43,5 @@ def render(fh, data, depth, is_list_member=False):
             render(fh, v, depth, True)
 
     else:
-        # use empty quotes to denote an empty string value instead of blank space
-        modified_data = miniutils.quote(data) if data == "" else data
         list_member_prefix = indentation + LIST_MARKER if is_list_member else ""
-        fh.write(list_member_prefix + str(modified_data) + "\n")
+        fh.write(list_member_prefix + str(data) + "\n")

.circleci/cimodel/lib/visualization.py

@@ -0,0 +1,84 @@
+"""
+This module encapsulates dependencies on pygraphviz
+"""
+
+import colorsys
+
+import cimodel.lib.conf_tree as conf_tree
+
+
+def rgb2hex(rgb_tuple):
+    def to_hex(f):
+        return "%02x" % int(f * 255)
+
+    return "#" + "".join(map(to_hex, list(rgb_tuple)))
+
+
+def handle_missing_graphviz(f):
+    """
+    If the user has not installed pygraphviz, this causes
+    calls to the draw() method of the returned object to do nothing.
+    """
+    try:
+        import pygraphviz  # noqa: F401
+        return f
+
+    except ModuleNotFoundError:
+
+        class FakeGraph:
+            def draw(self, *args, **kwargs):
+                pass
+
+        return lambda _: FakeGraph()
+
+
+@handle_missing_graphviz
+def generate_graph(toplevel_config_node):
+    """
+    Traverses the graph once first just to find the max depth
+    """
+
+    config_list = conf_tree.dfs(toplevel_config_node)
+
+    max_depth = 0
+    for config in config_list:
+        max_depth = max(max_depth, config.get_depth())
+
+    # color the nodes using the max depth
+
+    from pygraphviz import AGraph
+    dot = AGraph()
+
+    def node_discovery_callback(node, sibling_index, sibling_count):
+        depth = node.get_depth()
+
+        sat_min, sat_max = 0.1, 0.6
+        sat_range = sat_max - sat_min
+
+        saturation_fraction = sibling_index / float(sibling_count - 1) if sibling_count > 1 else 1
+        saturation = sat_min + sat_range * saturation_fraction
+
+        # TODO Use a hash of the node label to determine the color
+        hue = depth / float(max_depth + 1)
+
+        rgb_tuple = colorsys.hsv_to_rgb(hue, saturation, 1)
+
+        this_node_key = node.get_node_key()
+
+        dot.add_node(
+            this_node_key,
+            label=node.get_label(),
+            style="filled",
+            # fillcolor=hex_color + ":orange",
+            fillcolor=rgb2hex(rgb_tuple),
+            penwidth=3,
+            color=rgb2hex(colorsys.hsv_to_rgb(hue, saturation, 0.9))
+        )
+
+    def child_callback(node, child):
+        this_node_key = node.get_node_key()
+        child_node_key = child.get_node_key()
+        dot.add_edge((this_node_key, child_node_key))
+
+    conf_tree.dfs_recurse(toplevel_config_node, lambda x: None, node_discovery_callback, child_callback)
+    return dot

.circleci/codegen_validation/compare_normalized_yaml.sh

@@ -1,17 +0,0 @@
-#!/bin/bash -xe
-
-
-YAML_FILENAME=verbatim-sources/workflows-pytorch-ge-config-tests.yml
-DIFF_TOOL=meld
-
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-pushd ..
-
-
-$DIFF_TOOL $YAML_FILENAME <(./codegen_validation/normalize_yaml_fragment.py < $YAML_FILENAME)
-
-
-popd

.circleci/codegen_validation/normalize_yaml_fragment.py

@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-
-import os
-import sys
-import yaml
-
-# Need to import modules that lie on an upward-relative path
-sys.path.append(os.path.join(sys.path[0], '..'))
-
-import cimodel.lib.miniyaml as miniyaml
-
-
-def regurgitate(depth, use_pyyaml_formatter=False):
-    data = yaml.safe_load(sys.stdin)
-
-    if use_pyyaml_formatter:
-        output = yaml.dump(data, sort_keys=True)
-        sys.stdout.write(output)
-    else:
-        miniyaml.render(sys.stdout, data, depth)
-
-
-if __name__ == "__main__":
-    regurgitate(3)

.circleci/codegen_validation/overwrite_with_normalized.sh

@@ -1,15 +0,0 @@
-#!/bin/bash -xe
-
-YAML_FILENAME=$1
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-pushd ..
-
-TEMP_FILENAME=$(mktemp)
-
-cat $YAML_FILENAME | ./codegen_validation/normalize_yaml_fragment.py > $TEMP_FILENAME
-mv $TEMP_FILENAME $YAML_FILENAME
-
-popd

.circleci/docker/build.sh

@@ -10,64 +10,21 @@ if [ -z "${image}" ]; then
   exit 1
 fi
 
-function extract_version_from_image_name() {
-  eval export $2=$(echo "${image}" | perl -n -e"/$1(\d+(\.\d+)?(\.\d+)?)/ && print \$1")
-  if [ "x${!2}" = x ]; then
-    echo "variable '$2' not correctly parsed from image='$image'"
-    exit 1
-  fi
-}
+# TODO: Generalize
+OS="ubuntu"
+DOCKERFILE="${OS}/Dockerfile"
+if [[ "$image" == *-cuda* ]]; then
+  DOCKERFILE="${OS}-cuda/Dockerfile"
+fi
 
-function extract_all_from_image_name() {
-  # parts $image into array, splitting on '-'
-  keep_IFS="$IFS"
-  IFS="-"
-  declare -a parts=($image)
-  IFS="$keep_IFS"
-  unset keep_IFS
-
-  for part in "${parts[@]}"; do
-    name=$(echo "${part}" | perl -n -e"/([a-zA-Z]+)\d+(\.\d+)?(\.\d+)?/ && print \$1")
-    vername="${name^^}_VERSION"
-    # "py" is the odd one out, needs this special case
-    if [ "x${name}" = xpy ]; then
-      vername=ANACONDA_PYTHON_VERSION
-    fi
-    # skip non-conforming fields such as "pytorch", "linux" or "xenial" without version string
-    if [ -n "${name}" ]; then
-      extract_version_from_image_name "${name}" "${vername}"
-    fi
-  done
-}
-
-if [[ "$image" == *-xenial* ]]; then
+if [[ "$image" == *-trusty* ]]; then
+  UBUNTU_VERSION=14.04
+elif [[ "$image" == *-xenial* ]]; then
   UBUNTU_VERSION=16.04
 elif [[ "$image" == *-artful* ]]; then
   UBUNTU_VERSION=17.10
 elif [[ "$image" == *-bionic* ]]; then
   UBUNTU_VERSION=18.04
-elif [[ "$image" == *-focal* ]]; then
-  UBUNTU_VERSION=20.04
-elif [[ "$image" == *ubuntu* ]]; then
-  extract_version_from_image_name ubuntu UBUNTU_VERSION
-elif [[ "$image" == *centos* ]]; then
-  extract_version_from_image_name centos CENTOS_VERSION
-fi
-
-if [ -n "${UBUNTU_VERSION}" ]; then
-  OS="ubuntu"
-elif [ -n "${CENTOS_VERSION}" ]; then
-  OS="centos"
-else
-  echo "Unable to derive operating system base..."
-  exit 1
-fi
-
-DOCKERFILE="${OS}/Dockerfile"
-if [[ "$image" == *cuda* ]]; then
-  DOCKERFILE="${OS}-cuda/Dockerfile"
-elif [[ "$image" == *rocm* ]]; then
-  DOCKERFILE="${OS}-rocm/Dockerfile"
 fi
 
 TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/14.04/x86_64"
@@ -76,11 +33,43 @@ TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/u
 # configuration, so we hardcode everything here rather than do it
 # from scratch
 case "$image" in
-  pytorch-linux-xenial-py3.8)
-    ANACONDA_PYTHON_VERSION=3.8
+  pytorch-linux-bionic-clang9-thrift-llvmdev)
+    CLANG_VERSION=9
+    THRIFT=yes
+    LLVMDEV=yes
+    PROTOBUF=yes
+    ;;
+  pytorch-linux-xenial-py2.7.9)
+    TRAVIS_PYTHON_VERSION=2.7.9
     GCC_VERSION=7
     # Do not install PROTOBUF, DB, and VISION as a test
     ;;
+  pytorch-linux-xenial-py2.7)
+    TRAVIS_PYTHON_VERSION=2.7
+    GCC_VERSION=7
+    PROTOBUF=yes
+    DB=yes
+    VISION=yes
+    ;;
+  pytorch-linux-xenial-py3.5)
+    TRAVIS_PYTHON_VERSION=3.5
+    GCC_VERSION=7
+    # Do not install PROTOBUF, DB, and VISION as a test
+    ;;
+  pytorch-linux-xenial-py3.8)
+    # TODO: This is a hack, get rid of this as soon as you get rid of the travis downloads
+    TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/16.04/x86_64"
+    TRAVIS_PYTHON_VERSION=3.8
+    GCC_VERSION=7
+    # Do not install PROTOBUF, DB, and VISION as a test
+    ;;
+  pytorch-linux-xenial-py3.6-gcc4.8)
+    ANACONDA_PYTHON_VERSION=3.6
+    GCC_VERSION=4.8
+    PROTOBUF=yes
+    DB=yes
+    VISION=yes
+    ;;
   pytorch-linux-xenial-py3.6-gcc5.4)
     ANACONDA_PYTHON_VERSION=3.6
     GCC_VERSION=5
@@ -101,11 +90,25 @@ case "$image" in
     DB=yes
     VISION=yes
     ;;
-  pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc5.4)
-    CUDA_VERSION=9.2
+  pytorch-linux-xenial-pynightly)
+    TRAVIS_PYTHON_VERSION=nightly
+    GCC_VERSION=7
+    PROTOBUF=yes
+    DB=yes
+    VISION=yes
+    ;;
+  pytorch-linux-xenial-cuda9-cudnn7-py2)
+    CUDA_VERSION=9.0
+    CUDNN_VERSION=7
+    ANACONDA_PYTHON_VERSION=2.7
+    PROTOBUF=yes
+    DB=yes
+    VISION=yes
+    ;;
+  pytorch-linux-xenial-cuda9-cudnn7-py3)
+    CUDA_VERSION=9.0
     CUDNN_VERSION=7
     ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=5
     PROTOBUF=yes
     DB=yes
     VISION=yes
@@ -148,26 +151,6 @@ case "$image" in
     VISION=yes
     KATEX=yes
     ;;
-  pytorch-linux-xenial-cuda11.0-cudnn8-py3-gcc7)
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-cuda11.1-cudnn8-py3-gcc7)
-    CUDA_VERSION=11.1
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
   pytorch-linux-xenial-py3-clang5-asan)
     ANACONDA_PYTHON_VERSION=3.6
     CLANG_VERSION=5.0
@@ -175,13 +158,6 @@ case "$image" in
     DB=yes
     VISION=yes
     ;;
-  pytorch-linux-xenial-py3-clang7-onnx)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
   pytorch-linux-xenial-py3-clang5-android-ndk-r19c)
     ANACONDA_PYTHON_VERSION=3.6
     CLANG_VERSION=5.0
@@ -200,126 +176,6 @@ case "$image" in
     DB=yes
     VISION=yes
     ;;
-  pytorch-linux-bionic-py3.6-clang9)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    VULKAN_SDK_VERSION=1.2.148.0
-    SWIFTSHADER=yes
-    ;;
-  pytorch-linux-bionic-py3.8-gcc9)
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda10.2-cudnn7-py3.6-clang9)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda10.2-cudnn7-py3.8-gcc9)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda11.0-cudnn8-py3.6-gcc9)
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-bionic-cuda11.0-cudnn8-py3.8-gcc9)
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-bionic-cuda11.1-cudnn8-py3.6-gcc9)
-    CUDA_VERSION=11.1
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-bionic-cuda11.1-cudnn8-py3.8-gcc9)
-    CUDA_VERSION=11.1
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-bionic-rocm3.9-py3.6)
-    ANACONDA_PYTHON_VERSION=3.6
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ROCM_VERSION=3.9
-    ;;
-  pytorch-linux-bionic-rocm3.10-py3.6)
-    ANACONDA_PYTHON_VERSION=3.6
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ROCM_VERSION=3.10
-    ;;
-  *)
-    # Catch-all for builds that are not hardcoded.
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    echo "image '$image' did not match an existing build configuration"
-    if [[ "$image" == *py* ]]; then
-      extract_version_from_image_name py ANACONDA_PYTHON_VERSION
-    fi
-    if [[ "$image" == *cuda* ]]; then
-      extract_version_from_image_name cuda CUDA_VERSION
-      extract_version_from_image_name cudnn CUDNN_VERSION
-    fi
-    if [[ "$image" == *rocm* ]]; then
-      extract_version_from_image_name rocm ROCM_VERSION
-    fi
-    if [[ "$image" == *gcc* ]]; then
-      extract_version_from_image_name gcc GCC_VERSION
-    fi
-    if [[ "$image" == *clang* ]]; then
-      extract_version_from_image_name clang CLANG_VERSION
-    fi
-    if [[ "$image" == *devtoolset* ]]; then
-      extract_version_from_image_name devtoolset DEVTOOLSET_VERSION
-    fi
-    if [[ "$image" == *glibc* ]]; then
-      extract_version_from_image_name glibc GLIBC_VERSION
-    fi
-    if [[ "$image" == *cmake* ]]; then
-      extract_version_from_image_name cmake CMAKE_VERSION
-    fi
-  ;;
 esac
 
 # Set Jenkins UID and GID if running Jenkins
@@ -331,11 +187,8 @@ fi
 tmp_tag="tmp-$(cat /dev/urandom | tr -dc 'a-z' | fold -w 32 | head -n 1)"
 
 # Build image
-# TODO: build-arg THRIFT is not turned on for any image, remove it once we confirm
-# it's no longer needed.
 docker build \
        --no-cache \
-       --progress=plain \
        --build-arg "TRAVIS_DL_URL_PREFIX=${TRAVIS_DL_URL_PREFIX}" \
        --build-arg "BUILD_ENVIRONMENT=${image}" \
        --build-arg "PROTOBUF=${PROTOBUF:-}" \
@@ -348,36 +201,23 @@ docker build \
        --build-arg "JENKINS_UID=${JENKINS_UID:-}" \
        --build-arg "JENKINS_GID=${JENKINS_GID:-}" \
        --build-arg "UBUNTU_VERSION=${UBUNTU_VERSION}" \
-       --build-arg "CENTOS_VERSION=${CENTOS_VERSION}" \
-       --build-arg "DEVTOOLSET_VERSION=${DEVTOOLSET_VERSION}" \
-       --build-arg "GLIBC_VERSION=${GLIBC_VERSION}" \
        --build-arg "CLANG_VERSION=${CLANG_VERSION}" \
        --build-arg "ANACONDA_PYTHON_VERSION=${ANACONDA_PYTHON_VERSION}" \
+       --build-arg "TRAVIS_PYTHON_VERSION=${TRAVIS_PYTHON_VERSION}" \
        --build-arg "GCC_VERSION=${GCC_VERSION}" \
        --build-arg "CUDA_VERSION=${CUDA_VERSION}" \
        --build-arg "CUDNN_VERSION=${CUDNN_VERSION}" \
        --build-arg "ANDROID=${ANDROID}" \
        --build-arg "ANDROID_NDK=${ANDROID_NDK_VERSION}" \
        --build-arg "GRADLE_VERSION=${GRADLE_VERSION}" \
-       --build-arg "VULKAN_SDK_VERSION=${VULKAN_SDK_VERSION}" \
-       --build-arg "SWIFTSHADER=${SWIFTSHADER}" \
        --build-arg "CMAKE_VERSION=${CMAKE_VERSION:-}" \
        --build-arg "NINJA_VERSION=${NINJA_VERSION:-}" \
        --build-arg "KATEX=${KATEX:-}" \
-       --build-arg "ROCM_VERSION=${ROCM_VERSION:-}" \
        -f $(dirname ${DOCKERFILE})/Dockerfile \
        -t "$tmp_tag" \
        "$@" \
        .
 
-# NVIDIA dockers for RC releases use tag names like `11.0-cudnn8-devel-ubuntu18.04-rc`,
-# for this case we will set UBUNTU_VERSION to `18.04-rc` so that the Dockerfile could
-# find the correct image. As a result, here we have to replace the
-#   "$UBUNTU_VERSION" == "18.04-rc"
-# with
-#   "$UBUNTU_VERSION" == "18.04"
-UBUNTU_VERSION=$(echo ${UBUNTU_VERSION} | sed 's/-rc$//')
-
 function drun() {
   docker run --rm "$tmp_tag" $*
 }
@@ -395,6 +235,19 @@ if [[ "$OS" == "ubuntu" ]]; then
   fi
 fi
 
+if [ -n "$TRAVIS_PYTHON_VERSION" ]; then
+  if [[ "$TRAVIS_PYTHON_VERSION" != nightly ]]; then
+    if !(drun python --version 2>&1 | grep -qF "Python $TRAVIS_PYTHON_VERSION"); then
+      echo "TRAVIS_PYTHON_VERSION=$TRAVIS_PYTHON_VERSION, but:"
+      drun python --version
+      exit 1
+    fi
+  else
+    echo "Please manually check nightly is OK:"
+    drun python --version
+  fi
+fi
+
 if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
   if !(drun python --version 2>&1 | grep -qF "Python $ANACONDA_PYTHON_VERSION"); then
     echo "ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION, but:"

.circleci/docker/build_docker.sh

@@ -13,7 +13,7 @@ retry () {
 
 #until we find a way to reliably reuse previous build, this last_tag is not in use
 # last_tag="$(( CIRCLE_BUILD_NUM - 1 ))"
-tag="${DOCKER_TAG}"
+tag="${CIRCLE_WORKFLOW_ID}"
 
 
 registry="308535385114.dkr.ecr.us-east-1.amazonaws.com"

.circleci/docker/centos-rocm/Dockerfile

@@ -1,92 +0,0 @@
-ARG CENTOS_VERSION
-
-FROM centos:${CENTOS_VERSION}
-
-ARG CENTOS_VERSION
-
-# Install required packages to build Caffe2
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install devtoolset
-ARG DEVTOOLSET_VERSION
-ADD ./common/install_devtoolset.sh install_devtoolset.sh
-RUN bash ./install_devtoolset.sh && rm install_devtoolset.sh
-ENV BASH_ENV "/etc/profile"
-
-# (optional) Install non-default glibc version
-ARG GLIBC_VERSION
-ADD ./common/install_glibc.sh install_glibc.sh
-RUN if [ -n "${GLIBC_VERSION}" ]; then bash ./install_glibc.sh; fi
-RUN rm install_glibc.sh
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install conda and other packages (e.g., numpy, coverage, pytest)
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# Install rocm
-ARG ROCM_VERSION
-ADD ./common/install_rocm.sh install_rocm.sh
-RUN bash ./install_rocm.sh
-RUN rm install_rocm.sh
-ENV PATH /opt/rocm/bin:$PATH
-ENV PATH /opt/rocm/hcc/bin:$PATH
-ENV PATH /opt/rocm/hip/bin:$PATH
-ENV PATH /opt/rocm/opencl/bin:$PATH
-ENV PATH /opt/rocm/llvm/bin:$PATH
-ENV LANG en_US.utf8
-ENV LC_ALL en_US.utf8
-
-# (optional) Install non-default CMake version
-ARG CMAKE_VERSION
-ADD ./common/install_cmake.sh install_cmake.sh
-RUN if [ -n "${CMAKE_VERSION}" ]; then bash ./install_cmake.sh; fi
-RUN rm install_cmake.sh
-
-# (optional) Install non-default Ninja version
-ARG NINJA_VERSION
-ADD ./common/install_ninja.sh install_ninja.sh
-RUN if [ -n "${NINJA_VERSION}" ]; then bash ./install_ninja.sh; fi
-RUN rm install_ninja.sh
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-USER jenkins
-CMD ["bash"]

.circleci/docker/common/install_android.sh

@@ -4,15 +4,13 @@ set -ex
 
 [ -n "${ANDROID_NDK}" ]
 
-_https_amazon_aws=https://ossci-android.s3.amazonaws.com
-
 apt-get update
 apt-get install -y --no-install-recommends autotools-dev autoconf unzip
 apt-get autoclean && apt-get clean
 rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
 pushd /tmp
-curl -Os --retry 3 $_https_amazon_aws/android-ndk-${ANDROID_NDK}-linux-x86_64.zip
+curl -Os --retry 3 https://dl.google.com/android/repository/android-ndk-${ANDROID_NDK}-linux-x86_64.zip
 popd
 _ndk_dir=/opt/ndk
 mkdir -p "$_ndk_dir"
@@ -47,22 +45,43 @@ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
 # Installing android sdk
 # https://github.com/circleci/circleci-images/blob/staging/android/Dockerfile.m4
 
-_tmp_sdk_zip=/tmp/android-sdk-linux.zip
+_sdk_version=sdk-tools-linux-3859397.zip
 _android_home=/opt/android/sdk
 
 rm -rf $_android_home
 sudo mkdir -p $_android_home
-curl --silent --show-error --location --fail --retry 3 --output /tmp/android-sdk-linux.zip $_https_amazon_aws/android-sdk-linux-tools3859397-build-tools2803-2902-platforms28-29.zip
-sudo unzip -q $_tmp_sdk_zip -d $_android_home
-rm $_tmp_sdk_zip
+curl --silent --show-error --location --fail --retry 3 --output /tmp/$_sdk_version https://dl.google.com/android/repository/$_sdk_version
+sudo unzip -q /tmp/$_sdk_version -d $_android_home
+rm /tmp/$_sdk_version
 
 sudo chmod -R 777 $_android_home
 
 export ANDROID_HOME=$_android_home
 export ADB_INSTALL_TIMEOUT=120
 
-export PATH="${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools:${PATH}"
+export PATH="${ANDROID_HOME}/emulator:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools:${PATH}"
 echo "PATH:${PATH}"
+alias sdkmanager="$ANDROID_HOME/tools/bin/sdkmanager"
+
+sudo mkdir ~/.android && sudo echo '### User Sources for Android SDK Manager' > ~/.android/repositories.cfg
+sudo chmod -R 777 ~/.android
+
+yes | sdkmanager --licenses
+yes | sdkmanager --update
+
+sdkmanager \
+  "tools" \
+  "platform-tools" \
+  "emulator"
+
+sdkmanager \
+  "build-tools;28.0.3" \
+  "build-tools;29.0.2"
+
+sdkmanager \
+  "platforms;android-28" \
+  "platforms;android-29"
+sdkmanager --list
 
 # Installing Gradle
 echo "GRADLE_VERSION:${GRADLE_VERSION}"
@@ -70,7 +89,8 @@ _gradle_home=/opt/gradle
 sudo rm -rf $gradle_home
 sudo mkdir -p $_gradle_home
 
-curl --silent --output /tmp/gradle.zip --retry 3 $_https_amazon_aws/gradle-${GRADLE_VERSION}-bin.zip
+wget --no-verbose --output-document=/tmp/gradle.zip \
+"https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip"
 
 sudo unzip -q /tmp/gradle.zip -d $_gradle_home
 rm /tmp/gradle.zip

.circleci/docker/common/install_base.sh

@@ -2,121 +2,74 @@
 
 set -ex
 
-install_ubuntu() {
-  # NVIDIA dockers for RC releases use tag names like `11.0-cudnn8-devel-ubuntu18.04-rc`,
-  # for this case we will set UBUNTU_VERSION to `18.04-rc` so that the Dockerfile could
-  # find the correct image. As a result, here we have to check for
-  #   "$UBUNTU_VERSION" == "18.04"*
-  # instead of
-  #   "$UBUNTU_VERSION" == "18.04"
-  if [[ "$UBUNTU_VERSION" == "18.04"* ]]; then
-    cmake3="cmake=3.10*"
-  else
-    cmake3="cmake=3.5*"
-  fi
+if [[ "$UBUNTU_VERSION" == "14.04" ]]; then
+  # cmake 2 is too old
+  cmake3=cmake3
+else
+  cmake3=cmake
+fi
 
-  # Install common dependencies
-  apt-get update
-  # TODO: Some of these may not be necessary
-  ccache_deps="asciidoc docbook-xml docbook-xsl xsltproc"
-  numpy_deps="gfortran"
-  apt-get install -y --no-install-recommends \
-    $ccache_deps \
-    $numpy_deps \
-    ${cmake3} \
-    apt-transport-https \
-    autoconf \
-    automake \
-    build-essential \
-    ca-certificates \
-    curl \
-    git \
-    libatlas-base-dev \
-    libc6-dbg \
-    libiomp-dev \
-    libyaml-dev \
-    libz-dev \
-    libjpeg-dev \
-    libasound2-dev \
-    libsndfile-dev \
-    software-properties-common \
-    sudo \
-    wget \
-    vim
+if [[ "$UBUNTU_VERSION" == "18.04" ]]; then
+  cmake3="cmake=3.10*"
+else
+  cmake3="${cmake3}=3.5*"
+fi
 
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Need EPEL for many packages we depend on.
-  # See http://fedoraproject.org/wiki/EPEL
-  yum --enablerepo=extras install -y epel-release
-
-  ccache_deps="asciidoc docbook-dtds docbook-style-xsl libxslt"
-  numpy_deps="gcc-gfortran"
-  # Note: protobuf-c-{compiler,devel} on CentOS are too old to be used
-  # for Caffe2. That said, we still install them to make sure the build
-  # system opts to build/use protoc and libprotobuf from third-party.
-  yum install -y \
-    $ccache_deps \
-    $numpy_deps \
-    autoconf \
-    automake \
-    bzip2 \
-    cmake \
-    cmake3 \
-    curl \
-    gcc \
-    gcc-c++ \
-    gflags-devel \
-    git \
-    glibc-devel \
-    glibc-headers \
-    glog-devel \
-    hiredis-devel \
-    libstdc++-devel \
-    make \
-    opencv-devel \
-    sudo \
-    wget \
-    vim
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install base packages depending on the base OS
-ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
-case "$ID" in
-  ubuntu)
-    install_ubuntu
-    ;;
-  centos)
-    install_centos
-    ;;
-  *)
-    echo "Unable to determine OS..."
-    exit 1
-    ;;
-esac
+# Install common dependencies
+apt-get update
+# TODO: Some of these may not be necessary
+# TODO: libiomp also gets installed by conda, aka there's a conflict
+ccache_deps="asciidoc docbook-xml docbook-xsl xsltproc"
+numpy_deps="gfortran"
+apt-get install -y --no-install-recommends \
+  $ccache_deps \
+  $numpy_deps \
+  ${cmake3} \
+  apt-transport-https \
+  autoconf \
+  automake \
+  build-essential \
+  ca-certificates \
+  curl \
+  git \
+  libatlas-base-dev \
+  libc6-dbg \
+  libiomp-dev \
+  libyaml-dev \
+  libz-dev \
+  libjpeg-dev \
+  libasound2-dev \
+  libsndfile-dev \
+  python \
+  python-dev \
+  python-setuptools \
+  python-wheel \
+  software-properties-common \
+  sudo \
+  wget \
+  vim
 
 # Install Valgrind separately since the apt-get version is too old.
 mkdir valgrind_build && cd valgrind_build
-VALGRIND_VERSION=3.16.1
-if ! wget http://valgrind.org/downloads/valgrind-${VALGRIND_VERSION}.tar.bz2
+if ! wget http://valgrind.org/downloads/valgrind-3.14.0.tar.bz2
 then
-  wget https://sourceware.org/ftp/valgrind/valgrind-${VALGRIND_VERSION}.tar.bz2
+  wget https://sourceware.org/ftp/valgrind/valgrind-3.14.0.tar.bz2
 fi
-tar -xjf valgrind-${VALGRIND_VERSION}.tar.bz2
-cd valgrind-${VALGRIND_VERSION}
+tar -xjf valgrind-3.14.0.tar.bz2
+cd valgrind-3.14.0
 ./configure --prefix=/usr/local
-make -j 4
+make
 sudo make install
 cd ../../
 rm -rf valgrind_build
 alias valgrind="/usr/local/bin/valgrind"
+
+# TODO: THIS IS A HACK!!!
+# distributed nccl(2) tests are a bit busted, see https://github.com/pytorch/pytorch/issues/5877
+if dpkg -s libnccl-dev; then
+  apt-get remove -y libnccl-dev libnccl2 --allow-change-held-packages
+fi
+
+# Cleanup package manager
+apt-get autoclean && apt-get clean
+rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

.circleci/docker/common/install_cache.sh

@@ -2,51 +2,17 @@
 
 set -ex
 
-install_ubuntu() {
-  echo "Preparing to build sccache from source"
-  apt-get update
-  apt-get install -y cargo pkg-config libssl-dev
-  echo "Checking out sccache repo"
-  git clone https://github.com/pytorch/sccache
-  cd sccache
-  echo "Building sccache"
-  cargo build --release
-  cp target/release/sccache /opt/cache/bin
-  echo "Cleaning up"
-  cd ..
-  rm -rf sccache
-  apt-get remove -y cargo rustc
-  apt-get autoclean && apt-get clean
-}
-
-install_binary() {
-  echo "Downloading sccache binary from S3 repo"
-  curl --retry 3 https://s3.amazonaws.com/ossci-linux/sccache -o /opt/cache/bin/sccache
-}
-
 mkdir -p /opt/cache/bin
 mkdir -p /opt/cache/lib
 sed -e 's|PATH="\(.*\)"|PATH="/opt/cache/bin:\1"|g' -i /etc/environment
 export PATH="/opt/cache/bin:$PATH"
 
 # Setup compiler cache
-if [ -n "$ROCM_VERSION" ]; then
-  curl --retry 3 http://repo.radeon.com/misc/.sccache_amd/sccache -o /opt/cache/bin/sccache
-else
-  ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
-  case "$ID" in
-    ubuntu)
-      install_ubuntu
-      ;;
-    *)
-      install_binary
-      ;;
-  esac
-fi
+curl --retry 3 https://s3.amazonaws.com/ossci-linux/sccache -o /opt/cache/bin/sccache
 chmod a+x /opt/cache/bin/sccache
 
 function write_sccache_stub() {
-  printf "#!/bin/sh\nif [ \$(ps -p \$PPID -o comm=) != sccache ]; then\n  exec sccache $(which $1) \"\$@\"\nelse\n  exec $(which $1) \"\$@\"\nfi" > "/opt/cache/bin/$1"
+  printf "#!/bin/sh\nexec sccache $(which $1) \$*" > "/opt/cache/bin/$1"
   chmod a+x "/opt/cache/bin/$1"
 }
 
@@ -54,12 +20,8 @@ write_sccache_stub cc
 write_sccache_stub c++
 write_sccache_stub gcc
 write_sccache_stub g++
-
-# NOTE: See specific ROCM_VERSION case below.
-if [ "x$ROCM_VERSION" = x ]; then
-  write_sccache_stub clang
-  write_sccache_stub clang++
-fi
+write_sccache_stub clang
+write_sccache_stub clang++
 
 if [ -n "$CUDA_VERSION" ]; then
   # TODO: This is a workaround for the fact that PyTorch's FindCUDA
@@ -68,50 +30,6 @@ if [ -n "$CUDA_VERSION" ]; then
   # where CUDA is installed.  Instead, we install an nvcc symlink outside
   # of the PATH, and set CUDA_NVCC_EXECUTABLE so that we make use of it.
 
-  write_sccache_stub nvcc
-  mv /opt/cache/bin/nvcc /opt/cache/lib/
-fi
-
-if [ -n "$ROCM_VERSION" ]; then
-  # ROCm compiler is hcc or clang. However, it is commonly invoked via hipcc wrapper.
-  # hipcc will call either hcc or clang using an absolute path starting with /opt/rocm,
-  # causing the /opt/cache/bin to be skipped. We must create the sccache wrappers
-  # directly under /opt/rocm while also preserving the original compiler names.
-  # Note symlinks will chain as follows: [hcc or clang++] -> clang -> clang-??
-  # Final link in symlink chain must point back to original directory.
-
-  # Original compiler is moved one directory deeper. Wrapper replaces it.
-  function write_sccache_stub_rocm() {
-    OLDCOMP=$1
-    COMPNAME=$(basename $OLDCOMP)
-    TOPDIR=$(dirname $OLDCOMP)
-    WRAPPED="$TOPDIR/original/$COMPNAME"
-    mv "$OLDCOMP" "$WRAPPED"
-    printf "#!/bin/sh\nexec sccache $WRAPPED \"\$@\"" > "$OLDCOMP"
-    chmod a+x "$OLDCOMP"
-  }
-
-  if [[ -e "/opt/rocm/hcc/bin/hcc" ]]; then
-    # ROCm 3.3 or earlier.
-    mkdir /opt/rocm/hcc/bin/original
-    write_sccache_stub_rocm /opt/rocm/hcc/bin/hcc
-    write_sccache_stub_rocm /opt/rocm/hcc/bin/clang
-    write_sccache_stub_rocm /opt/rocm/hcc/bin/clang++
-    # Fix last link in symlink chain, clang points to versioned clang in prior dir
-    pushd /opt/rocm/hcc/bin/original
-    ln -s ../$(readlink clang)
-    popd
-  elif [[ -e "/opt/rocm/llvm/bin/clang" ]]; then
-    # ROCm 3.5 and beyond.
-    mkdir /opt/rocm/llvm/bin/original
-    write_sccache_stub_rocm /opt/rocm/llvm/bin/clang
-    write_sccache_stub_rocm /opt/rocm/llvm/bin/clang++
-    # Fix last link in symlink chain, clang points to versioned clang in prior dir
-    pushd /opt/rocm/llvm/bin/original
-    ln -s ../$(readlink clang)
-    popd
-  else
-    echo "Cannot find ROCm compiler."
-    exit 1
-  fi
+  printf "#!/bin/sh\nexec sccache $(which nvcc) \"\$@\"" > /opt/cache/lib/nvcc
+  chmod a+x /opt/cache/lib/nvcc
 fi

.circleci/docker/common/install_conda.sh

@@ -4,7 +4,7 @@ set -ex
 
 # Optionally install conda
 if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
-  BASE_URL="https://repo.anaconda.com/miniconda"
+  BASE_URL="https://repo.continuum.io/miniconda"
 
   MAJOR_PYTHON_VERSION=$(echo "$ANACONDA_PYTHON_VERSION" | cut -d . -f 1)
 
@@ -24,20 +24,13 @@ if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
   mkdir /opt/conda
   chown jenkins:jenkins /opt/conda
 
-  # Work around bug where devtoolset replaces sudo and breaks it.
-  if [ -n "$DEVTOOLSET_VERSION" ]; then
-    SUDO=/bin/sudo
-  else
-    SUDO=sudo
-  fi
-
   as_jenkins() {
     # NB: unsetting the environment variables works around a conda bug
     # https://github.com/conda/conda/issues/6576
     # NB: Pass on PATH and LD_LIBRARY_PATH to sudo invocation
     # NB: This must be run from a directory that jenkins has access to,
     # works around https://github.com/conda/conda-package-handling/pull/34
-    $SUDO -H -u jenkins env -u SUDO_UID -u SUDO_GID -u SUDO_COMMAND -u SUDO_USER env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
+    sudo -H -u jenkins env -u SUDO_UID -u SUDO_GID -u SUDO_COMMAND -u SUDO_USER env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
   }
 
   pushd /tmp
@@ -56,10 +49,10 @@ if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
   pushd /opt/conda
 
   # Track latest conda update
-  as_jenkins conda update -y -n base conda
+  as_jenkins conda update -n base conda
 
   # Install correct Python version
-  as_jenkins conda install -y python="$ANACONDA_PYTHON_VERSION"
+  as_jenkins conda install python="$ANACONDA_PYTHON_VERSION"
 
   conda_install() {
     # Ensure that the install command don't upgrade/downgrade Python
@@ -71,59 +64,29 @@ if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
   # Install PyTorch conda deps, as per https://github.com/pytorch/pytorch README
   # DO NOT install cmake here as it would install a version newer than 3.5, but
   # we want to pin to version 3.5.
-  if [ "$ANACONDA_PYTHON_VERSION" = "3.8" ]; then
-    # Install llvm-8 as it is required to compile llvmlite-0.30.0 from source
-    conda_install numpy=1.18.5 pyyaml mkl mkl-include setuptools cffi future six llvmdev=8.0.0
-  elif [ "$ANACONDA_PYTHON_VERSION" = "3.7" ]; then
-    # DO NOT install dataclasses if installing python-3.7, since its part of python-3.7 core packages
-    conda_install numpy=1.18.5 pyyaml mkl mkl-include setuptools cffi future six typing_extensions
-  else
-    conda_install numpy=1.18.5 pyyaml mkl mkl-include setuptools cffi future six dataclasses typing_extensions
-  fi
-  if [[ "$CUDA_VERSION" == 9.2* ]]; then
+  conda_install numpy pyyaml mkl mkl-include setuptools cffi typing future six
+  if [[ "$CUDA_VERSION" == 9.0* ]]; then
+    conda_install magma-cuda90 -c pytorch
+  elif [[ "$CUDA_VERSION" == 9.1* ]]; then
+    conda_install magma-cuda91 -c pytorch
+  elif [[ "$CUDA_VERSION" == 9.2* ]]; then
     conda_install magma-cuda92 -c pytorch
   elif [[ "$CUDA_VERSION" == 10.0* ]]; then
     conda_install magma-cuda100 -c pytorch
   elif [[ "$CUDA_VERSION" == 10.1* ]]; then
     conda_install magma-cuda101 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.2* ]]; then
-    conda_install magma-cuda102 -c pytorch
-  elif [[ "$CUDA_VERSION" == 11.0* ]]; then
-    conda_install magma-cuda110 -c pytorch
-  elif [[ "$CUDA_VERSION" == 11.1* ]]; then
-    conda_install magma-cuda111 -c pytorch
-  elif [[ "$CUDA_VERSION" == 11.2* ]]; then
-    conda_install magma-cuda112 -c pytorch
   fi
 
   # TODO: This isn't working atm
   conda_install nnpack -c killeent
 
-  # Install some other packages, including those needed for Python test reporting
+  # Install some other packages
   # TODO: Why is scipy pinned
-  # Pin MyPy version because new errors are likely to appear with each release
-  # Pin hypothesis to avoid flakiness: https://github.com/pytorch/pytorch/issues/31136
-  as_jenkins pip install --progress-bar off pytest \
-    scipy==1.1.0 \
-    scikit-image \
-    librosa>=0.6.2 \
-    psutil \
-    numba \
-    llvmlite \
-    unittest-xml-reporting \
-    boto3==1.16.34 \
-    coverage \
-    hypothesis==4.53.2 \
-    mypy==0.770 \
-    tb-nightly
-
-  # Update scikit-learn to a python-3.8 compatible version
-  if [[ $(python -c "import sys; print(int(sys.version_info >= (3, 8)))") == "1" ]]; then
-    as_jenkins pip install --progress-bar off -U scikit-learn
-  else
-    # Pinned scikit-learn due to https://github.com/scikit-learn/scikit-learn/issues/14485 (affects gcc 5.5 only)
-    as_jenkins pip install --progress-bar off scikit-learn==0.20.3
-  fi
+  # numba & llvmlite is pinned because of https://github.com/numba/numba/issues/4368
+  # scikit-learn is pinned because of
+  # https://github.com/scikit-learn/scikit-learn/issues/14485 (affects gcc 5.5
+  # only)
+  as_jenkins pip install --progress-bar off pytest scipy==1.1.0 scikit-learn==0.20.3 scikit-image librosa>=0.6.2 psutil numba==0.46.0 llvmlite==0.30.0
 
   popd
 fi

.circleci/docker/common/install_db.sh

@@ -51,16 +51,11 @@ install_centos() {
 }
 
 # Install base packages depending on the base OS
-ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
-case "$ID" in
-  ubuntu)
-    install_ubuntu
-    ;;
-  centos)
-    install_centos
-    ;;
-  *)
-    echo "Unable to determine OS..."
-    exit 1
-    ;;
-esac
+if [ -f /etc/lsb-release ]; then
+  install_ubuntu
+elif [ -f /etc/os-release ]; then
+  install_centos
+else
+  echo "Unable to determine OS..."
+  exit 1
+fi

.circleci/docker/common/install_devtoolset.sh

@@ -1,10 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "$DEVTOOLSET_VERSION" ]
-
-yum install -y centos-release-scl
-yum install -y devtoolset-$DEVTOOLSET_VERSION
-
-echo "source scl_source enable devtoolset-$DEVTOOLSET_VERSION" > "/etc/profile.d/devtoolset-$DEVTOOLSET_VERSION.sh"

.circleci/docker/common/install_gcc.sh

@@ -7,15 +7,10 @@ if [ -n "$GCC_VERSION" ]; then
   # Need the official toolchain repo to get alternate packages
   add-apt-repository ppa:ubuntu-toolchain-r/test
   apt-get update
-  if [ "$UBUNTU_VERSION" = "16.04" -a "$GCC_VERSION" = "5" ]; then
-    apt-get install -y g++-5=5.4.0-6ubuntu1~16.04.12
-  else
-    apt-get install -y g++-$GCC_VERSION
-  fi
+  apt-get install -y g++-$GCC_VERSION
 
   update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-"$GCC_VERSION" 50
   update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-"$GCC_VERSION" 50
-  update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-"$GCC_VERSION" 50
 
   # Cleanup package manager
   apt-get autoclean && apt-get clean

.circleci/docker/common/install_glibc.sh

@@ -1,34 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "$GLIBC_VERSION" ]
-if [[ -n "$CENTOS_VERSION" ]]; then
-  [ -n "$DEVTOOLSET_VERSION" ]
-fi
-
-yum install -y wget sed
-
-mkdir -p /packages && cd /packages
-wget -q http://ftp.gnu.org/gnu/glibc/glibc-$GLIBC_VERSION.tar.gz
-tar xzf glibc-$GLIBC_VERSION.tar.gz
-if [[ "$GLIBC_VERSION" == "2.26" ]]; then
-  cd glibc-$GLIBC_VERSION
-  sed -i 's/$name ne "nss_test1"/$name ne "nss_test1" \&\& $name ne "nss_test2"/' scripts/test-installation.pl
-  cd ..
-fi
-mkdir -p glibc-$GLIBC_VERSION-build && cd glibc-$GLIBC_VERSION-build
-
-if [[ -n "$CENTOS_VERSION" ]]; then
-  export PATH=/opt/rh/devtoolset-$DEVTOOLSET_VERSION/root/usr/bin:$PATH
-fi
-
-../glibc-$GLIBC_VERSION/configure --prefix=/usr CFLAGS='-Wno-stringop-truncation -Wno-format-overflow -Wno-restrict -Wno-format-truncation -g -O2'
-make -j$(nproc)
-make install
-
-# Cleanup
-rm -rf /packages
-rm -rf /var/cache/yum/*
-rm -rf /var/lib/rpm/__db.*
-yum clean all

.circleci/docker/common/install_lcov.sh

@@ -1,8 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-git clone --branch v1.15 https://github.com/linux-test-project/lcov.git
-pushd lcov
-sudo make install   # will be installed in /usr/local/bin/lcov
-popd

.circleci/docker/common/install_nccl.sh

@@ -1,4 +0,0 @@
-#!/bin/bash
-
-sudo apt-get -qq update
-sudo apt-get -qq install --allow-downgrades --allow-change-held-packages libnccl-dev=2.5.6-1+cuda10.1 libnccl2=2.5.6-1+cuda10.1

.circleci/docker/common/install_openmpi.sh

@@ -1,4 +0,0 @@
-#!/bin/bash
-
-sudo apt-get update
-sudo apt-get install -y --allow-downgrades --allow-change-held-packages openmpi-bin libopenmpi-dev

.circleci/docker/common/install_protobuf.sh

@@ -46,16 +46,11 @@ install_centos() {
 }
 
 # Install base packages depending on the base OS
-ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
-case "$ID" in
-  ubuntu)
-    install_ubuntu
-    ;;
-  centos)
-    install_centos
-    ;;
-  *)
-    echo "Unable to determine OS..."
-    exit 1
-    ;;
-esac
+if [ -f /etc/lsb-release ]; then
+  install_ubuntu
+elif [ -f /etc/os-release ]; then
+  install_centos
+else
+  echo "Unable to determine OS..."
+  exit 1
+fi

.circleci/docker/common/install_rocm.sh

@@ -1,111 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-install_magma() {
-    # "install" hipMAGMA into /opt/rocm/magma by copying after build
-    git clone https://bitbucket.org/icl/magma.git -b hipMAGMA
-    pushd magma
-    cp make.inc-examples/make.inc.hip-mkl-gcc make.inc
-    echo 'LIBDIR += -L$(MKLROOT)/lib' >> make.inc
-    echo 'LIB += -Wl,--enable-new-dtags -Wl,--rpath,/opt/rocm/lib -Wl,--rpath,$(MKLROOT)/lib -Wl,--rpath,/opt/rocm/magma/lib' >> make.inc
-    echo 'DEVCCFLAGS += --amdgpu-target=gfx803 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908' >> make.inc
-    export PATH="${PATH}:/opt/rocm/bin"
-    make -f make.gen.hipMAGMA -j $(nproc)
-    make lib/libmagma.so -j $(nproc) MKLROOT=/opt/conda
-    make testing/testing_dgemm -j $(nproc) MKLROOT=/opt/conda
-    popd
-    mv magma /opt/rocm
-}
-
-install_ubuntu() {
-    apt-get update
-    if [[ $UBUNTU_VERSION == 18.04 ]]; then
-      # gpg-agent is not available by default on 18.04
-      apt-get install -y --no-install-recommends gpg-agent
-    fi
-    apt-get install -y kmod
-    apt-get install -y wget
-
-    # Need the libc++1 and libc++abi1 libraries to allow torch._C to load at runtime
-    apt-get install -y libc++1
-    apt-get install -y libc++abi1
-
-    # Add rocm repository
-    wget -qO - http://repo.radeon.com/rocm/rocm.gpg.key | apt-key add -
-    echo "deb [arch=amd64] http://repo.radeon.com/rocm/apt/${ROCM_VERSION} xenial main" > /etc/apt/sources.list.d/rocm.list
-    apt-get update --allow-insecure-repositories
-
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
-                   rocm-dev \
-                   rocm-utils \
-                   rocm-libs \
-                   rccl \
-                   rocprofiler-dev \
-                   roctracer-dev
-
-    # precompiled miopen kernels added in ROCm 3.5; search for all unversioned packages
-    # if search fails it will abort this script; use true to avoid case where search fails
-    MIOPENKERNELS=$(apt-cache search --names-only miopenkernels | awk '{print $1}' | grep -F -v . || true)
-    if [[ "x${MIOPENKERNELS}" = x ]]; then
-      echo "miopenkernels package not available"
-    else
-      DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated ${MIOPENKERNELS}
-    fi
-
-    install_magma
-
-    # Cleanup
-    apt-get autoclean && apt-get clean
-    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-
-  yum update -y
-  yum install -y kmod
-  yum install -y wget
-  yum install -y openblas-devel
-
-  yum install -y epel-release
-  yum install -y dkms kernel-headers-`uname -r` kernel-devel-`uname -r`
-
-  echo "[ROCm]" > /etc/yum.repos.d/rocm.repo
-  echo "name=ROCm" >> /etc/yum.repos.d/rocm.repo
-  echo "baseurl=http://repo.radeon.com/rocm/yum/${ROCM_VERSION}" >> /etc/yum.repos.d/rocm.repo
-  echo "enabled=1" >> /etc/yum.repos.d/rocm.repo
-  echo "gpgcheck=0" >> /etc/yum.repos.d/rocm.repo
-
-  yum update -y
-
-  yum install -y \
-                   rocm-dev \
-                   rocm-utils \
-                   rocm-libs \
-                   rccl \
-                   rocprofiler-dev \
-                   roctracer-dev
-
-  install_magma
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install Python packages depending on the base OS
-ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
-case "$ID" in
-  ubuntu)
-    install_ubuntu
-    ;;
-  centos)
-    install_centos
-    ;;
-  *)
-    echo "Unable to determine OS..."
-    exit 1
-    ;;
-esac

.circleci/docker/common/install_swiftshader.sh

@@ -1,24 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "${SWIFTSHADER}" ]
-
-retry () {
-    $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-}
-
-_https_amazon_aws=https://ossci-android.s3.amazonaws.com
-
-# SwiftShader
-_swiftshader_dir=/var/lib/jenkins/swiftshader
-_swiftshader_file_targz=swiftshader-abe07b943-prebuilt.tar.gz
-mkdir -p $_swiftshader_dir
-_tmp_swiftshader_targz="/tmp/${_swiftshader_file_targz}"
-
-curl --silent --show-error --location --fail --retry 3 \
-  --output "${_tmp_swiftshader_targz}" "$_https_amazon_aws/${_swiftshader_file_targz}"
-
-tar -C "${_swiftshader_dir}" -xzf "${_tmp_swiftshader_targz}"
-
-export VK_ICD_FILENAMES="${_swiftshader_dir}/build/Linux/vk_swiftshader_icd.json"

.circleci/docker/common/install_travis_python.sh

@@ -0,0 +1,97 @@
+#!/bin/bash
+
+set -ex
+
+as_jenkins() {
+  # NB: Preserve PATH and LD_LIBRARY_PATH changes
+  sudo -H -u jenkins env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
+}
+
+if [ -n "$TRAVIS_PYTHON_VERSION" ]; then
+
+  mkdir -p /opt/python
+  chown jenkins:jenkins /opt/python
+
+  # Download Python binary from Travis
+  pushd tmp
+  as_jenkins wget --quiet ${TRAVIS_DL_URL_PREFIX}/python-$TRAVIS_PYTHON_VERSION.tar.bz2
+  # NB: The tarball also comes with /home/travis virtualenv that we
+  # don't care about.  (Maybe we should, but we've worked around the
+  # "how do I install to python" issue by making this entire directory
+  # user-writable "lol")
+  # NB: Relative ordering of opt/python and flags matters
+  as_jenkins tar xjf python-$TRAVIS_PYTHON_VERSION.tar.bz2 --strip-components=2 --directory /opt/python opt/python
+  popd
+
+  echo "/opt/python/$TRAVIS_PYTHON_VERSION/lib" > /etc/ld.so.conf.d/travis-python.conf
+  ldconfig
+  sed -e 's|PATH="\(.*\)"|PATH="/opt/python/'"$TRAVIS_PYTHON_VERSION"'/bin:\1"|g' -i /etc/environment
+  export PATH="/opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH"
+
+  python --version
+  pip --version
+
+  # Install pip from source.
+  # The python-pip package on Ubuntu Trusty is old
+  # and upon install numpy doesn't use the binary
+  # distribution, and fails to compile it from source.
+  pushd tmp
+  as_jenkins curl -L -O https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz
+  as_jenkins tar zxf pip-9.0.1.tar.gz
+  pushd pip-9.0.1
+  as_jenkins python setup.py install
+  popd
+  rm -rf pip-9.0.1*
+  popd
+
+  # Install pip packages
+  as_jenkins pip install --upgrade pip
+
+  pip --version
+
+  if [[ "$TRAVIS_PYTHON_VERSION" == nightly ]]; then
+      # These two packages have broken Cythonizations uploaded
+      # to PyPi, see:
+      #
+      #  - https://github.com/numpy/numpy/issues/10500
+      #  - https://github.com/yaml/pyyaml/issues/117
+      #
+      # Furthermore, the released version of Cython does not
+      # have these issues fixed.
+      #
+      # While we are waiting on fixes for these, we build
+      # from Git for now.  Feel free to delete this conditional
+      # branch if things start working again (you may need
+      # to do this if these packages regress on Git HEAD.)
+      as_jenkins pip install git+https://github.com/cython/cython.git
+      as_jenkins pip install git+https://github.com/numpy/numpy.git
+      as_jenkins pip install git+https://github.com/yaml/pyyaml.git
+  else
+      as_jenkins pip install numpy pyyaml
+  fi
+
+  as_jenkins pip install \
+      future \
+      hypothesis \
+      protobuf \
+      pytest \
+      pillow \
+      typing
+
+  as_jenkins pip install mkl mkl-devel
+
+  # SciPy does not support Python 3.7 or Python 2.7.9
+  if [[ "$TRAVIS_PYTHON_VERSION" != nightly ]] && [[ "$TRAVIS_PYTHON_VERSION" != "2.7.9" ]]; then
+      as_jenkins pip install scipy==1.1.0 scikit-image librosa>=0.6.2
+  fi
+
+  # Install psutil for dataloader tests
+  as_jenkins pip install psutil
+
+  # Install dill for serialization tests
+  as_jenkins pip install "dill>=0.3.1"
+
+  # Cleanup package manager
+  apt-get autoclean && apt-get clean
+  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+fi

.circleci/docker/common/install_vision.sh

@@ -47,16 +47,11 @@ install_centos() {
 }
 
 # Install base packages depending on the base OS
-ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
-case "$ID" in
-  ubuntu)
-    install_ubuntu
-    ;;
-  centos)
-    install_centos
-    ;;
-  *)
-    echo "Unable to determine OS..."
-    exit 1
-    ;;
-esac
+if [ -f /etc/lsb-release ]; then
+  install_ubuntu
+elif [ -f /etc/os-release ]; then
+  install_centos
+else
+  echo "Unable to determine OS..."
+  exit 1
+fi

.circleci/docker/common/install_vulkan_sdk.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "${VULKAN_SDK_VERSION}" ]
-
-retry () {
-    $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-}
-
-_https_amazon_aws=https://ossci-android.s3.amazonaws.com
-
-_vulkansdk_dir=/var/lib/jenkins/vulkansdk
-mkdir -p $_vulkansdk_dir
-_tmp_vulkansdk_targz=/tmp/vulkansdk.tar.gz
-curl --silent --show-error --location --fail --retry 3 \
-  --output "$_tmp_vulkansdk_targz" "$_https_amazon_aws/vulkansdk-linux-x86_64-${VULKAN_SDK_VERSION}.tar.gz"
-
-tar -C "$_vulkansdk_dir" -xzf "$_tmp_vulkansdk_targz" --strip-components 1
-
-export VULKAN_SDK="$_vulkansdk_dir/"
-
-rm "$_tmp_vulkansdk_targz"

.circleci/docker/ubuntu-cuda/Dockerfile

@@ -24,7 +24,7 @@ ARG KATEX
 ADD ./common/install_katex.sh install_katex.sh
 RUN bash ./install_katex.sh && rm install_katex.sh
 
-# Install conda and other packages (e.g., numpy, coverage, pytest)
+# Install conda
 ENV PATH /opt/conda/bin:$PATH
 ARG ANACONDA_PYTHON_VERSION
 ADD ./common/install_conda.sh install_conda.sh
@@ -35,10 +35,11 @@ ARG GCC_VERSION
 ADD ./common/install_gcc.sh install_gcc.sh
 RUN bash ./install_gcc.sh && rm install_gcc.sh
 
-# Install clang
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
+# Install non-standard Python versions (via Travis binaries)
+ARG TRAVIS_PYTHON_VERSION
+ENV PATH /opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH
+ADD ./common/install_travis_python.sh install_travis_python.sh
+RUN bash ./install_travis_python.sh && rm install_travis_python.sh
 
 # (optional) Install protobuf for ONNX
 ARG PROTOBUF
@@ -72,16 +73,6 @@ ADD ./common/install_jni.sh install_jni.sh
 ADD ./java/jni.h jni.h
 RUN bash ./install_jni.sh && rm install_jni.sh
 
-# Install NCCL for when CUDA is version 10.1
-ADD ./common/install_nccl.sh install_nccl.sh
-RUN if [ "${CUDA_VERSION}" = 10.1 ]; then bash ./install_nccl.sh; fi
-RUN rm install_nccl.sh
-
-# Install Open MPI for CUDA
-ADD ./common/install_openmpi.sh install_openmpi.sh
-RUN if [ -n "${CUDA_VERSION}" ]; then bash install_openmpi.sh; fi
-RUN rm install_openmpi.sh
-
 # Include BUILD_ENVIRONMENT environment variable in image
 ARG BUILD_ENVIRONMENT
 ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
@@ -90,8 +81,5 @@ ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
 ENV TORCH_CUDA_ARCH_LIST Maxwell
 ENV TORCH_NVCC_FLAGS "-Xfatbin -compress-all"
 
-# Install LLVM dev version (Defined in the pytorch/builder github repository)
-COPY --from=pytorch/llvm:9.0.1 /opt/llvm /opt/llvm
-
 USER jenkins
 CMD ["bash"]

.circleci/docker/ubuntu-rocm/.gitignore

@@ -1 +0,0 @@
-*.sh

.circleci/docker/ubuntu-rocm/Dockerfile

@@ -1,87 +0,0 @@
-ARG UBUNTU_VERSION
-
-FROM ubuntu:${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install clang
-ARG LLVMDEV
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install conda and other packages (e.g., numpy, coverage, pytest)
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# Install rocm
-ARG ROCM_VERSION
-ADD ./common/install_rocm.sh install_rocm.sh
-RUN bash ./install_rocm.sh
-RUN rm install_rocm.sh
-ENV PATH /opt/rocm/bin:$PATH
-ENV PATH /opt/rocm/hcc/bin:$PATH
-ENV PATH /opt/rocm/hip/bin:$PATH
-ENV PATH /opt/rocm/opencl/bin:$PATH
-ENV PATH /opt/rocm/llvm/bin:$PATH
-ENV MAGMA_HOME /opt/rocm/magma
-ENV LANG C.UTF-8
-ENV LC_ALL C.UTF-8
-
-# (optional) Install non-default CMake version
-ARG CMAKE_VERSION
-ADD ./common/install_cmake.sh install_cmake.sh
-RUN if [ -n "${CMAKE_VERSION}" ]; then bash ./install_cmake.sh; fi
-RUN rm install_cmake.sh
-
-# (optional) Install non-default Ninja version
-ARG NINJA_VERSION
-ADD ./common/install_ninja.sh install_ninja.sh
-RUN if [ -n "${NINJA_VERSION}" ]; then bash ./install_ninja.sh; fi
-RUN rm install_ninja.sh
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-USER jenkins
-CMD ["bash"]

.circleci/docker/ubuntu/Dockerfile

@@ -33,7 +33,7 @@ ARG KATEX
 ADD ./common/install_katex.sh install_katex.sh
 RUN bash ./install_katex.sh && rm install_katex.sh
 
-# Install conda and other packages (e.g., numpy, coverage, pytest)
+# Install conda
 ENV PATH /opt/conda/bin:$PATH
 ARG ANACONDA_PYTHON_VERSION
 ADD ./common/install_conda.sh install_conda.sh
@@ -44,9 +44,12 @@ ARG GCC_VERSION
 ADD ./common/install_gcc.sh install_gcc.sh
 RUN bash ./install_gcc.sh && rm install_gcc.sh
 
-# Install lcov for C++ code coverage
-ADD ./common/install_lcov.sh install_lcov.sh
-RUN  bash ./install_lcov.sh && rm install_lcov.sh
+# Install non-standard Python versions (via Travis binaries)
+ARG TRAVIS_PYTHON_VERSION
+ARG TRAVIS_DL_URL_PREFIX
+ENV PATH /opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH
+ADD ./common/install_travis_python.sh install_travis_python.sh
+RUN bash ./install_travis_python.sh && rm install_travis_python.sh
 
 # (optional) Install protobuf for ONNX
 ARG PROTOBUF
@@ -82,18 +85,6 @@ RUN rm AndroidManifest.xml
 RUN rm build.gradle
 ENV INSTALLED_ANDROID ${ANDROID}
 
-# (optional) Install Vulkan SDK
-ARG VULKAN_SDK_VERSION
-ADD ./common/install_vulkan_sdk.sh install_vulkan_sdk.sh
-RUN if [ -n "${VULKAN_SDK_VERSION}" ]; then bash ./install_vulkan_sdk.sh; fi
-RUN rm install_vulkan_sdk.sh
-
-# (optional) Install swiftshader
-ARG SWIFTSHADER
-ADD ./common/install_swiftshader.sh install_swiftshader.sh
-RUN if [ -n "${SWIFTSHADER}" ]; then bash ./install_swiftshader.sh; fi
-RUN rm install_swiftshader.sh
-
 # (optional) Install non-default CMake version
 ARG CMAKE_VERSION
 ADD ./common/install_cmake.sh install_cmake.sh
@@ -120,8 +111,5 @@ RUN bash ./install_jni.sh && rm install_jni.sh
 ARG BUILD_ENVIRONMENT
 ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
 
-# Install LLVM dev version (Defined in the pytorch/builder github repository)
-COPY --from=pytorch/llvm:9.0.1 /opt/llvm /opt/llvm
-
 USER jenkins
 CMD ["bash"]

.circleci/ecr_gc_docker/Dockerfile

@@ -1,6 +1,6 @@
 FROM ubuntu:16.04
 
-RUN apt-get update && apt-get install -y python-pip git && rm -rf /var/lib/apt/lists/* /var/log/dpkg.log
+RUN apt-get update && apt-get install -y python-pip && rm -rf /var/lib/apt/lists/* /var/log/dpkg.log
 
 ADD requirements.txt /requirements.txt
 

.circleci/ecr_gc_docker/gc.py

@@ -5,7 +5,6 @@ import datetime
 import boto3
 import pytz
 import sys
-import re
 
 
 def save_to_s3(project, data):
@@ -88,9 +87,6 @@ parser = argparse.ArgumentParser(description="Delete old Docker tags from regist
 parser.add_argument(
     "--dry-run", action="store_true", help="Dry run; print tags that would be deleted"
 )
-parser.add_argument(
-    "--debug", action="store_true", help="Debug, print ignored / saved tags"
-)
 parser.add_argument(
     "--keep-stable-days",
     type=int,
@@ -150,14 +146,6 @@ def chunks(chunkable, n):
     for i in range(0, len(chunkable), n):
         yield chunkable[i : i + n]
 
-SHA_PATTERN = re.compile(r'^[0-9a-f]{40}$')
-def looks_like_git_sha(tag):
-    """Returns a boolean to check if a tag looks like a git sha
-
-    For reference a sha1 is 40 characters with only 0-9a-f and contains no
-    "-" characters
-    """
-    return re.match(SHA_PATTERN, tag) is not None
 
 stable_window_tags = []
 for repo in repos(client):
@@ -167,48 +155,48 @@ for repo in repos(client):
 
     # Keep list of image digests to delete for this repository
     digest_to_delete = []
+    print(repositoryName)
 
     for image in images(client, repo):
         tags = image.get("imageTags")
         if not isinstance(tags, (list,)) or len(tags) == 0:
             continue
+
+        tag = tags[0]
         created = image["imagePushedAt"]
         age = now - created
-        for tag in tags:
-            if any([
-                    looks_like_git_sha(tag),
-                    tag.isdigit(),
-                    tag.count("-") == 4,  # TODO: Remove, this no longer applies as tags are now built using a SHA1
-                    tag in ignore_tags]):
-                window = stable_window
-                if tag in ignore_tags:
-                    stable_window_tags.append((repositoryName, tag, "", age, created))
-                elif age < window:
-                    stable_window_tags.append((repositoryName, tag, window, age, created))
-            else:
-                window = unstable_window
-
-            if tag in ignore_tags or age < window:
-                if args.debug:
-                    print("Ignoring {}:{} (age: {})".format(repositoryName, tag, age))
-                break
+        # new images build on circle ci use workflow ID as tag, which has 4 "-"
+        if tag.isdigit() or tag.count("-") == 4 or tag in ignore_tags:
+            window = stable_window
+            if tag in ignore_tags:
+                stable_window_tags.append((repositoryName, tag, "", age, created))
+            elif age < window:
+                stable_window_tags.append((repositoryName, tag, window, age, created))
         else:
-            for tag in tags:
-                print("{}Deleting {}:{} (age: {})".format("(dry run) " if args.dry_run else "", repositoryName, tag, age))
-            digest_to_delete.append(image["imageDigest"])
-    if args.dry_run:
-        if args.debug:
-            print("Skipping actual deletion, moving on...")
-    else:
-        # Issue batch delete for all images to delete for this repository
-        # Note that as of 2018-07-25, the maximum number of images you can
-        # delete in a single batch is 100, so chunk our list into batches of
-        # 100
-        for c in chunks(digest_to_delete, 100):
-            client.batch_delete_image(
-                registryId="308535385114",
-                repositoryName=repositoryName,
-                imageIds=[{"imageDigest": digest} for digest in c],
-            )
+            window = unstable_window
 
-        save_to_s3(args.filter_prefix, stable_window_tags)
+        if tag in ignore_tags:
+            print("Ignoring tag {} (age: {})".format(tag, age))
+            continue
+        if age < window:
+            print("Not deleting manifest for tag {} (age: {})".format(tag, age))
+            continue
+
+        if args.dry_run:
+            print("(dry run) Deleting manifest for tag {} (age: {})".format(tag, age))
+        else:
+            print("Deleting manifest for tag {} (age: {})".format(tag, age))
+            digest_to_delete.append(image["imageDigest"])
+
+    # Issue batch delete for all images to delete for this repository
+    # Note that as of 2018-07-25, the maximum number of images you can
+    # delete in a single batch is 100, so chunk our list into batches of
+    # 100
+    for c in chunks(digest_to_delete, 100):
+        client.batch_delete_image(
+            registryId="308535385114",
+            repositoryName=repositoryName,
+            imageIds=[{"imageDigest": digest} for digest in c],
+        )
+
+    save_to_s3(args.filter_prefix, stable_window_tags)

.circleci/generate_config_yml.py

@@ -6,24 +6,13 @@ Please see README.md in this directory for details.
 """
 
 import os
-import shutil
 import sys
-from collections import namedtuple
+import shutil
+from collections import namedtuple, OrderedDict
 
-import cimodel.data.binary_build_definitions as binary_build_definitions
 import cimodel.data.pytorch_build_definitions as pytorch_build_definitions
-import cimodel.data.simple.android_definitions
-import cimodel.data.simple.bazel_definitions
-import cimodel.data.simple.binary_smoketest
-import cimodel.data.simple.docker_definitions
-import cimodel.data.simple.ge_config_tests
-import cimodel.data.simple.ios_definitions
-import cimodel.data.simple.macos_definitions
-import cimodel.data.simple.mobile_definitions
-import cimodel.data.simple.nightly_android
-import cimodel.data.simple.nightly_ios
-import cimodel.data.simple.anaconda_prune_defintions
-import cimodel.data.windows_build_definitions as windows_build_definitions
+import cimodel.data.binary_build_definitions as binary_build_definitions
+import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
 import cimodel.lib.miniutils as miniutils
 import cimodel.lib.miniyaml as miniyaml
 
@@ -32,7 +21,6 @@ class File(object):
     """
     Verbatim copy the contents of a file into config.yml
     """
-
     def __init__(self, filename):
         self.filename = filename
 
@@ -41,7 +29,7 @@ class File(object):
             shutil.copyfileobj(fh, output_filehandle)
 
 
-class FunctionGen(namedtuple("FunctionGen", "function depth")):
+class FunctionGen(namedtuple('FunctionGen', 'function depth')):
     __slots__ = ()
 
 
@@ -51,14 +39,15 @@ class Treegen(FunctionGen):
     """
 
     def write(self, output_filehandle):
-        miniyaml.render(output_filehandle, self.function(), self.depth)
+        build_dict = OrderedDict()
+        self.function(build_dict)
+        miniyaml.render(output_filehandle, build_dict, self.depth)
 
 
 class Listgen(FunctionGen):
     """
     Insert the content of a YAML list into config.yml
     """
-
     def write(self, output_filehandle):
         miniyaml.render(output_filehandle, self.function(), self.depth)
 
@@ -68,6 +57,7 @@ def horizontal_rule():
 
 
 class Header(object):
+
     def __init__(self, title, summary=None):
         self.title = title
         self.summary_lines = summary or []
@@ -81,66 +71,48 @@ class Header(object):
             output_filehandle.write(line + "\n")
 
 
-def gen_build_workflows_tree():
-    build_workflows_functions = [
-        cimodel.data.simple.docker_definitions.get_workflow_jobs,
-        pytorch_build_definitions.get_workflow_jobs,
-        cimodel.data.simple.macos_definitions.get_workflow_jobs,
-        cimodel.data.simple.android_definitions.get_workflow_jobs,
-        cimodel.data.simple.ios_definitions.get_workflow_jobs,
-        cimodel.data.simple.mobile_definitions.get_workflow_jobs,
-        cimodel.data.simple.ge_config_tests.get_workflow_jobs,
-        cimodel.data.simple.bazel_definitions.get_workflow_jobs,
-        cimodel.data.simple.binary_smoketest.get_workflow_jobs,
-        cimodel.data.simple.nightly_ios.get_workflow_jobs,
-        cimodel.data.simple.nightly_android.get_workflow_jobs,
-        cimodel.data.simple.anaconda_prune_defintions.get_workflow_jobs,
-        windows_build_definitions.get_windows_workflows,
-        binary_build_definitions.get_post_upload_jobs,
-        binary_build_definitions.get_binary_smoke_test_jobs,
-    ]
-
-    binary_build_functions = [
-        binary_build_definitions.get_binary_build_jobs,
-        binary_build_definitions.get_nightly_tests,
-        binary_build_definitions.get_nightly_uploads,
-    ]
-
-    return {
-        "workflows": {
-            "binary_builds": {
-                "when": r"<< pipeline.parameters.run_binary_tests >>",
-                "jobs": [f() for f in binary_build_functions],
-            },
-            "build": {
-                "when": r"<< pipeline.parameters.run_build >>",
-                "jobs": [f() for f in build_workflows_functions]
-            },
-        }
-    }
-
-
 # Order of this list matters to the generated config.yml.
 YAML_SOURCES = [
     File("header-section.yml"),
     File("commands.yml"),
     File("nightly-binary-build-defaults.yml"),
     Header("Build parameters"),
-    File("build-parameters/pytorch-build-params.yml"),
-    File("build-parameters/binary-build-params.yml"),
-    File("build-parameters/promote-build-params.yml"),
+    File("pytorch-build-params.yml"),
+    File("caffe2-build-params.yml"),
+    File("binary-build-params.yml"),
     Header("Job specs"),
-    File("job-specs/pytorch-job-specs.yml"),
-    File("job-specs/binary-job-specs.yml"),
-    File("job-specs/job-specs-custom.yml"),
-    File("job-specs/job-specs-promote.yml"),
-    File("job-specs/binary_update_htmls.yml"),
-    File("job-specs/binary-build-tests.yml"),
-    File("job-specs/docker_jobs.yml"),
-    Header("Workflows"),
-    Treegen(gen_build_workflows_tree, 0),
-    File("workflows/workflows-ecr-gc.yml"),
-    File("workflows/workflows-promote.yml"),
+    File("pytorch-job-specs.yml"),
+    File("caffe2-job-specs.yml"),
+    File("binary-job-specs.yml"),
+    File("job-specs-setup.yml"),
+    File("job-specs-custom.yml"),
+    File("binary_update_htmls.yml"),
+    File("binary-build-tests.yml"),
+    File("docker_jobs.yml"),
+    File("workflows.yml"),
+
+    File("workflows-setup-job.yml"),
+    File("windows-build-test.yml"),
+    Listgen(pytorch_build_definitions.get_workflow_jobs, 3),
+    File("workflows-pytorch-macos-builds.yml"),
+    File("workflows-pytorch-android-gradle-build.yml"),
+    File("workflows-pytorch-ios-builds.yml"),
+    File("workflows-pytorch-mobile-builds.yml"),
+    File("workflows-pytorch-ge-config-tests.yml"),
+    Listgen(caffe2_build_definitions.get_workflow_jobs, 3),
+    File("workflows-binary-builds-smoke-subset.yml"),
+    Listgen(binary_build_definitions.get_binary_smoke_test_jobs, 3),
+    Listgen(binary_build_definitions.get_binary_build_jobs, 3),
+    File("workflows-nightly-ios-binary-builds.yml"),
+    File("workflows-nightly-android-binary-builds.yml"),
+
+    Header("Nightly tests"),
+    Listgen(binary_build_definitions.get_nightly_tests, 3),
+    File("workflows-nightly-uploads-header.yml"),
+    Listgen(binary_build_definitions.get_nightly_uploads, 3),
+    File("workflows-s3-html.yml"),
+    File("workflows-docker-builder.yml"),
+    File("workflows-ecr-gc.yml"),
 ]
 
 

.circleci/scripts/binary_checkout.sh

@@ -10,11 +10,6 @@ retry () {
 if [[ "$(uname)" == Darwin ]]; then
   # macos executor (builds and tests)
   workdir="/Users/distiller/project"
-elif [[ "$OSTYPE" == "msys" ]]; then
-  # windows executor (builds and tests)
-  rm -rf /c/w
-  ln -s "/c/Users/circleci/project" /c/w
-  workdir="/c/w"
 elif [[ -d "/home/circleci/project" ]]; then
   # machine executor (binary tests)
   workdir="/home/circleci/project"
@@ -24,19 +19,8 @@ else
 fi
 
 # It is very important that this stays in sync with binary_populate_env.sh
-if [[ "$OSTYPE" == "msys" ]]; then
-  # We need to make the paths as short as possible on Windows
-  export PYTORCH_ROOT="$workdir/p"
-  export BUILDER_ROOT="$workdir/b"
-else
-  export PYTORCH_ROOT="$workdir/pytorch"
-  export BUILDER_ROOT="$workdir/builder"
-fi
-
-# Try to extract PR number from branch if not already set
-if [[ -z "${CIRCLE_PR_NUMBER:-}" ]]; then
-  CIRCLE_PR_NUMBER="$(echo ${CIRCLE_BRANCH} | sed -E -n 's/pull\/([0-9]*).*/\1/p')"
-fi
+export PYTORCH_ROOT="$workdir/pytorch"
+export BUILDER_ROOT="$workdir/builder"
 
 # Clone the Pytorch branch
 retry git clone https://github.com/pytorch/pytorch.git "$PYTORCH_ROOT"

.circleci/scripts/binary_install_miniconda.sh

@@ -31,9 +31,9 @@ fi
 
 conda_sh="$workdir/install_miniconda.sh"
 if [[ "$(uname)" == Darwin ]]; then
-  curl --retry 3 -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+  curl --retry 3 -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
 else
-  curl --retry 3 -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+  curl --retry 3 -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
 fi
 chmod +x "$conda_sh"
 "$conda_sh" -b -p "$MINICONDA_ROOT"

.circleci/scripts/binary_ios_build.sh

@@ -15,8 +15,7 @@ export PATH="~/anaconda/bin:${PATH}"
 source ~/anaconda/bin/activate
 
 # Install dependencies
-conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi requests --yes
-conda install -c conda-forge valgrind --yes
+conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing requests --yes
 export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
 
 # sync submodules

.circleci/scripts/binary_ios_test.sh

@@ -13,7 +13,7 @@ base64 --decode cert.txt -o Certificates.p12
 rm cert.txt
 bundle exec fastlane install_cert
 # install the provisioning profile
-PROFILE=PyTorch_CI_2021.mobileprovision
+PROFILE=TestApp_CI.mobileprovision
 PROVISIONING_PROFILES=~/Library/MobileDevice/Provisioning\ Profiles
 mkdir -pv "${PROVISIONING_PROFILES}"
 cd "${PROVISIONING_PROFILES}"
@@ -25,5 +25,5 @@ if ! [ -x "$(command -v xcodebuild)" ]; then
     echo 'Error: xcodebuild is not installed.'
     exit 1
 fi 
-PROFILE=PyTorch_CI_2021
+PROFILE=TestApp_CI
 ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM} -c ${PROFILE} -t ${IOS_DEV_TEAM_ID}

.circleci/scripts/binary_ios_upload.sh

@@ -14,7 +14,7 @@ mkdir -p ${ZIP_DIR}/src
 cp -R ${ARTIFACTS_DIR}/arm64/include ${ZIP_DIR}/install/
 # build a FAT bianry
 cd ${ZIP_DIR}/install/lib
-target_libs=(libc10.a libclog.a libcpuinfo.a libeigen_blas.a libpthreadpool.a libpytorch_qnnpack.a libtorch_cpu.a libtorch.a libXNNPACK.a)
+target_libs=(libc10.a libclog.a libcpuinfo.a libeigen_blas.a libpytorch_qnnpack.a libtorch_cpu.a libtorch.a libXNNPACK.a)
 for lib in ${target_libs[*]}
 do
     if [ -f "${ARTIFACTS_DIR}/x86_64/lib/${lib}" ] && [ -f "${ARTIFACTS_DIR}/arm64/lib/${lib}" ]; then
@@ -22,6 +22,8 @@ do
         lipo -create "${libs[@]}" -o ${ZIP_DIR}/install/lib/${lib}
     fi
 done
+# for nnpack, we only support arm64 build
+cp ${ARTIFACTS_DIR}/arm64/lib/libnnpack.a ./
 lipo -i ${ZIP_DIR}/install/lib/*.a
 # copy the umbrella header and license
 cp ${PROJ_ROOT}/ios/LibTorch.h ${ZIP_DIR}/src/
@@ -34,13 +36,7 @@ touch version.txt
 echo $(date +%s) > version.txt
 zip -r ${ZIPFILE} install src version.txt LICENSE
 # upload to aws
-# Install conda then 'conda install' awscli
-curl --retry 3 -o ~/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
-chmod +x ~/conda.sh
-/bin/bash ~/conda.sh -b -p ~/anaconda
-export PATH="~/anaconda/bin:${PATH}"
-source ~/anaconda/bin/activate
-conda install -c conda-forge awscli --yes
+brew install awscli
 set +x
 export AWS_ACCESS_KEY_ID=${AWS_S3_ACCESS_KEY_FOR_PYTORCH_BINARY_UPLOAD}
 export AWS_SECRET_ACCESS_KEY=${AWS_S3_ACCESS_SECRET_FOR_PYTORCH_BINARY_UPLOAD}

.circleci/scripts/binary_linux_build.sh

@@ -5,22 +5,26 @@ set -eux -o pipefail
 source /env
 
 # Defaults here so they can be changed in one place
-export MAX_JOBS=${MAX_JOBS:-$(( $(nproc) - 2 ))}
-
-if [[ "${DESIRED_CUDA}" == "cu111" ]]; then
-  export BUILD_SPLIT_CUDA="ON"
-fi
+export MAX_JOBS=12
 
 # Parse the parameters
 if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
   build_script='conda/build_pytorch.sh'
 elif [[ "$DESIRED_CUDA" == cpu ]]; then
   build_script='manywheel/build_cpu.sh'
-elif [[ "$DESIRED_CUDA" == *"rocm"* ]]; then
-  build_script='manywheel/build_rocm.sh'
 else
   build_script='manywheel/build.sh'
 fi
 
+# We want to call unbuffer, which calls tclsh which finds the expect
+# package. The expect was installed by yum into /usr/bin so we want to
+# find /usr/bin/tclsh, but this is shadowed by /opt/conda/bin/tclsh in
+# the conda docker images, so we prepend it to the path here.
+if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
+  mkdir /just_tclsh_bin
+  ln -s /usr/bin/tclsh /just_tclsh_bin/tclsh
+  export PATH=/just_tclsh_bin:$PATH
+fi
+
 # Build the package
-SKIP_ALL_TESTS=1 "/builder/$build_script"
+SKIP_ALL_TESTS=1 unbuffer "/builder/$build_script" | ts

.circleci/scripts/binary_linux_test.sh

@@ -5,17 +5,14 @@ cat >/home/circleci/project/ci_test_script.sh <<EOL
 # =================== The following code will be executed inside Docker container ===================
 set -eux -o pipefail
 
-python_nodot="\$(echo $DESIRED_PYTHON | tr -d m.u)"
-
 # Set up Python
 if [[ "$PACKAGE_TYPE" == conda ]]; then
-  # There was a bug that was introduced in conda-package-handling >= 1.6.1 that makes archives
-  # above a certain size fail out when attempting to extract
-  # see: https://github.com/conda/conda-package-handling/issues/71
-  conda install -y conda-package-handling=1.6.0
   retry conda create -qyn testenv python="$DESIRED_PYTHON"
   source activate testenv >/dev/null
+elif [[ "$DESIRED_PYTHON" == 2.7mu ]]; then
+  export PATH="/opt/python/cp27-cp27mu/bin:\$PATH"
 elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
+  python_nodot="\$(echo $DESIRED_PYTHON | tr -d m.u)"
   python_path="/opt/python/cp\$python_nodot-cp\${python_nodot}"
   # Prior to Python 3.8 paths were suffixed with an 'm'
   if [[ -d  "\${python_path}/bin" ]]; then
@@ -25,19 +22,6 @@ elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
   fi
 fi
 
-EXTRA_CONDA_FLAGS=""
-NUMPY_PIN=""
-if [[ "\$python_nodot" = *39* ]]; then
-  EXTRA_CONDA_FLAGS="-c=conda-forge"
-  # There's an issue with conda channel priority where it'll randomly pick 1.19 over 1.20
-  # we set a lower boundary here just to be safe
-  NUMPY_PIN=">=1.20"
-fi
-
-if [[ "$DESIRED_CUDA" == "cu112" ]]; then
-  EXTRA_CONDA_FLAGS="-c=conda-forge"
-fi
-
 # Install the package
 # These network calls should not have 'retry's because they are installing
 # locally and aren't actually network calls
@@ -46,37 +30,23 @@ fi
 #   conda build scripts themselves. These should really be consolidated
 pkg="/final_pkgs/\$(ls /final_pkgs)"
 if [[ "$PACKAGE_TYPE" == conda ]]; then
-  (
-    # For some reason conda likes to re-activate the conda environment when attempting this install
-    # which means that a deactivate is run and some variables might not exist when that happens,
-    # namely CONDA_MKL_INTERFACE_LAYER_BACKUP from libblas so let's just ignore unbound variables when
-    # it comes to the conda installation commands
-    set +u
-    retry conda install \${EXTRA_CONDA_FLAGS} -yq \
-      "numpy\${NUMPY_PIN}" \
-      future \
-      mkl>=2018 \
-      ninja \
-      dataclasses \
-      typing-extensions \
-      defaults::protobuf \
-      six
-    if [[ "$DESIRED_CUDA" == 'cpu' ]]; then
-      retry conda install -c pytorch -y cpuonly
+  conda install -y "\$pkg" --offline
+  if [[ "$DESIRED_CUDA" == 'cpu' ]]; then
+    retry conda install -y cpuonly -c pytorch
+  fi
+  retry conda install -yq future numpy protobuf six
+  if [[ "$DESIRED_CUDA" != 'cpu' ]]; then
+    # DESIRED_CUDA is in format cu90 or cu102
+    if [[ "${#DESIRED_CUDA}" == 4 ]]; then
+      cu_ver="${DESIRED_CUDA:2:1}.${DESIRED_CUDA:3}"
     else
-      # DESIRED_CUDA is in format cu90 or cu102
-      if [[ "${#DESIRED_CUDA}" == 4 ]]; then
-        cu_ver="${DESIRED_CUDA:2:1}.${DESIRED_CUDA:3}"
-      else
-        cu_ver="${DESIRED_CUDA:2:2}.${DESIRED_CUDA:4}"
-      fi
-      retry conda install \${EXTRA_CONDA_FLAGS} -yq -c nvidia -c pytorch "cudatoolkit=\${cu_ver}"
+      cu_ver="${DESIRED_CUDA:2:2}.${DESIRED_CUDA:4}"
     fi
-    conda install \${EXTRA_CONDA_FLAGS} -y "\$pkg" --offline
-  )
+    retry conda install -yq -c pytorch "cudatoolkit=\${cu_ver}"
+  fi
 elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
   pip install "\$pkg"
-  retry pip install -q future numpy protobuf typing-extensions six
+  retry pip install -q future numpy protobuf six
 fi
 if [[ "$PACKAGE_TYPE" == libtorch ]]; then
   pkg="\$(ls /final_pkgs/*-latest.zip)"
@@ -86,7 +56,6 @@ fi
 
 # Test the package
 /builder/check_binary.sh
-
 # =================== The above code will be executed inside Docker container ===================
 EOL
 echo

.circleci/scripts/binary_linux_upload.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Do NOT set -x
+source /home/circleci/project/env
+set -eu -o pipefail
+set +x
+declare -x "AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
+declare -x "AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
+
+#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
+# DO NOT TURN -x ON BEFORE THIS LINE
+#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
+set -eux -o pipefail
+export PATH="$MINICONDA_ROOT/bin:$PATH"
+
+# This gets set in binary_populate_env.sh, but lets have a sane default just in case
+PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly}
+# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
+#       The only difference is the trailing slash
+# Strip trailing slashes if there
+CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
+
+# Upload the package to the final location
+pushd /home/circleci/project/final_pkgs
+if [[ "$PACKAGE_TYPE" == conda ]]; then
+  retry conda install -yq anaconda-client
+  anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload  "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
+elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
+  retry pip install -q awscli
+  s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
+  for pkg in $(ls); do
+    retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
+  done
+else
+  retry pip install -q awscli
+  s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
+  retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
+fi

.circleci/scripts/binary_macos_test.sh

@@ -20,9 +20,9 @@ if [[ "$PACKAGE_TYPE" == libtorch ]]; then
   unzip "$pkg" -d /tmp
   cd /tmp/libtorch
 elif [[ "$PACKAGE_TYPE" == conda ]]; then
-  conda install -y "$pkg"
+  conda install -y "$pkg" --offline
 else
-  pip install "$pkg" -v
+  pip install "$pkg" --no-index --no-dependencies -v
 fi
 
 # Test

.circleci/scripts/binary_macos_upload.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Do NOT set -x
+set -eu -o pipefail
+set +x
+export AWS_ACCESS_KEY_ID="${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
+export AWS_SECRET_ACCESS_KEY="${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
+
+#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
+# DO NOT TURN -x ON BEFORE THIS LINE
+#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
+set -eux -o pipefail
+
+source "/Users/distiller/project/env"
+export "PATH=$workdir/miniconda/bin:$PATH"
+
+# This gets set in binary_populate_env.sh, but lets have a sane default just in case
+PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly}
+# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
+#       The only difference is the trailing slash
+# Strip trailing slashes if there
+CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
+
+pushd "$workdir/final_pkgs"
+if [[ "$PACKAGE_TYPE" == conda ]]; then
+  retry conda install -yq anaconda-client
+  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
+elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
+  retry pip install -q awscli
+  s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
+  for pkg in $(ls); do
+    retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
+  done
+else
+  retry pip install -q awscli
+  s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
+  retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
+fi

.circleci/scripts/binary_populate_env.sh

@@ -2,31 +2,13 @@
 set -eux -o pipefail
 export TZ=UTC
 
-tagged_version() {
-  # Grabs version from either the env variable CIRCLE_TAG
-  # or the pytorch git described version
-  if [[ "$OSTYPE" == "msys" ]]; then
-    GIT_DESCRIBE="git --git-dir ${workdir}/p/.git describe"
-  else
-    GIT_DESCRIBE="git --git-dir ${workdir}/pytorch/.git describe"
-  fi
-  if [[ -n "${CIRCLE_TAG:-}" ]]; then
-    echo "${CIRCLE_TAG}"
-  elif ${GIT_DESCRIBE} --exact --tags >/dev/null; then
-    ${GIT_DESCRIBE} --tags
-  else
-    return 1
-  fi
-}
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 
 # We need to write an envfile to persist these variables to following
 # steps, but the location of the envfile depends on the circleci executor
 if [[ "$(uname)" == Darwin ]]; then
   # macos executor (builds and tests)
   workdir="/Users/distiller/project"
-elif [[ "$OSTYPE" == "msys" ]]; then
-  # windows executor (builds and tests)
-  workdir="/c/w"
 elif [[ -d "/home/circleci/project" ]]; then
   # machine executor (binary tests)
   workdir="/home/circleci/project"
@@ -43,15 +25,7 @@ configs=($BUILD_ENVIRONMENT)
 export PACKAGE_TYPE="${configs[0]}"
 export DESIRED_PYTHON="${configs[1]}"
 export DESIRED_CUDA="${configs[2]}"
-if [[ "${BUILD_FOR_SYSTEM:-}" == "windows" ]]; then
-  export DESIRED_DEVTOOLSET=""
-  export LIBTORCH_CONFIG="${configs[3]:-}"
-  if [[ "$LIBTORCH_CONFIG" == 'debug' ]]; then
-    export DEBUG=1
-  fi
-else
-  export DESIRED_DEVTOOLSET="${configs[3]:-}"
-fi
+export DESIRED_DEVTOOLSET="${configs[3]:-}"
 if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then
   export BUILD_PYTHONLESS=1
 fi
@@ -73,19 +47,20 @@ PIP_UPLOAD_FOLDER='nightly/'
 # We put this here so that OVERRIDE_PACKAGE_VERSION below can read from it
 export DATE="$(date -u +%Y%m%d)"
 #TODO: We should be pulling semver version from the base version.txt
-BASE_BUILD_VERSION="1.8.0.dev$DATE"
+BASE_BUILD_VERSION="1.5.0.dev$DATE"
 # Change BASE_BUILD_VERSION to git tag when on a git tag
 # Use 'git -C' to make doubly sure we're in the correct directory for checking
 # the git tag
-if tagged_version >/dev/null; then
+GIT_DESCRIBE="git --git-dir ${workdir}/pytorch/.git describe"
+if ${GIT_DESCRIBE} --tags --exact >/dev/null; then
   # Switch upload folder to 'test/' if we are on a tag
   PIP_UPLOAD_FOLDER='test/'
   # Grab git tag, remove prefixed v and remove everything after -
-  # Used to clean up tags that are for release candidates like v1.6.0-rc1
-  # Turns tag v1.6.0-rc1 -> v1.6.0
-  BASE_BUILD_VERSION="$(tagged_version | sed -e 's/^v//' -e 's/-.*$//')"
+  # Used to clean up tags that are for release candidates like v1.5.0-rc1
+  # Turns tag v1.5.0-rc1 -> v1.5.0
+  BASE_BUILD_VERSION="$(${GIT_DESCRIBE} --tags | sed -e 's/^v//' -e 's/-.*$//')"
 fi
-if [[ "$(uname)" == 'Darwin' ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
+if [[ "$(uname)" == 'Darwin' ]] || [[ "$DESIRED_CUDA" == "cu101" ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
   export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}"
 else
   export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}+$DESIRED_CUDA"
@@ -100,14 +75,8 @@ if [[ "$PACKAGE_TYPE" == libtorch ]]; then
   POSSIBLE_JAVA_HOMES+=(/usr/local)
   POSSIBLE_JAVA_HOMES+=(/usr/lib/jvm/java-8-openjdk-amd64)
   POSSIBLE_JAVA_HOMES+=(/Library/Java/JavaVirtualMachines/*.jdk/Contents/Home)
-  # Add the Windows-specific JNI path
-  POSSIBLE_JAVA_HOMES+=("$PWD/.circleci/windows-jni/")
   for JH in "${POSSIBLE_JAVA_HOMES[@]}" ; do
     if [[ -e "$JH/include/jni.h" ]] ; then
-      # Skip if we're not on Windows but haven't found a JAVA_HOME
-      if [[ "$JH" == "$PWD/.circleci/windows-jni/" && "$OSTYPE" != "msys" ]] ; then
-        break
-      fi
       echo "Found jni.h under $JH"
       JAVA_HOME="$JH"
       BUILD_JNI=ON
@@ -130,13 +99,9 @@ export DESIRED_CUDA="$DESIRED_CUDA"
 export LIBTORCH_VARIANT="${LIBTORCH_VARIANT:-}"
 export BUILD_PYTHONLESS="${BUILD_PYTHONLESS:-}"
 export DESIRED_DEVTOOLSET="$DESIRED_DEVTOOLSET"
-if [[ "${BUILD_FOR_SYSTEM:-}" == "windows" ]]; then
-  export LIBTORCH_CONFIG="${LIBTORCH_CONFIG:-}"
-  export DEBUG="${DEBUG:-}"
-fi
 
 export DATE="$DATE"
-export NIGHTLIES_DATE_PREAMBLE=1.8.0.dev
+export NIGHTLIES_DATE_PREAMBLE=1.5.0.dev
 export PYTORCH_BUILD_VERSION="$PYTORCH_BUILD_VERSION"
 export PYTORCH_BUILD_NUMBER="$PYTORCH_BUILD_NUMBER"
 export OVERRIDE_PACKAGE_VERSION="$PYTORCH_BUILD_VERSION"
@@ -153,13 +118,8 @@ export DOCKER_IMAGE="$DOCKER_IMAGE"
 
 export workdir="$workdir"
 export MAC_PACKAGE_WORK_DIR="$workdir"
-if [[ "$OSTYPE" == "msys" ]]; then
-  export PYTORCH_ROOT="$workdir/p"
-  export BUILDER_ROOT="$workdir/b"
-else
-  export PYTORCH_ROOT="$workdir/pytorch"
-  export BUILDER_ROOT="$workdir/builder"
-fi
+export PYTORCH_ROOT="$workdir/pytorch"
+export BUILDER_ROOT="$workdir/builder"
 export MINICONDA_ROOT="$workdir/miniconda"
 export PYTORCH_FINAL_PACKAGE_DIR="$workdir/final_pkgs"
 
@@ -167,7 +127,6 @@ export CIRCLE_TAG="${CIRCLE_TAG:-}"
 export CIRCLE_SHA1="$CIRCLE_SHA1"
 export CIRCLE_PR_NUMBER="${CIRCLE_PR_NUMBER:-}"
 export CIRCLE_BRANCH="$CIRCLE_BRANCH"
-export CIRCLE_WORKFLOW_ID="$CIRCLE_WORKFLOW_ID"
 # =================== The above code will be executed inside Docker container ===================
 EOL
 

.circleci/scripts/binary_run_in_docker.sh

@@ -19,7 +19,7 @@ chmod +x /home/circleci/project/ci_test_script.sh
 VOLUME_MOUNTS="-v /home/circleci/project/:/circleci_stuff -v /home/circleci/project/final_pkgs:/final_pkgs -v ${PYTORCH_ROOT}:/pytorch -v ${BUILDER_ROOT}:/builder"
 # Run the docker
 if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --gpus all ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
+  export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --runtime=nvidia ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
 else
   export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
 fi

.circleci/scripts/binary_upload.sh

@@ -1,98 +0,0 @@
-#!/usr/bin/env bash
-
-set -euo pipefail
-
-PACKAGE_TYPE=${PACKAGE_TYPE:-conda}
-
-PKG_DIR=${PKG_DIR:-/tmp/workspace/final_pkgs}
-
-# Designates whether to submit as a release candidate or a nightly build
-# Value should be `test` when uploading release candidates
-# currently set within `designate_upload_channel`
-UPLOAD_CHANNEL=${UPLOAD_CHANNEL:-nightly}
-# Designates what subfolder to put packages into
-UPLOAD_SUBFOLDER=${UPLOAD_SUBFOLDER:-cpu}
-UPLOAD_BUCKET="s3://pytorch"
-BACKUP_BUCKET="s3://pytorch-backup"
-
-DRY_RUN=${DRY_RUN:-enabled}
-# Don't actually do work unless explicit
-ANACONDA="true anaconda"
-AWS_S3_CP="aws s3 cp --dryrun"
-if [[ "${DRY_RUN}" = "disabled" ]]; then
-  ANACONDA="anaconda"
-  AWS_S3_CP="aws s3 cp"
-fi
-
-do_backup() {
-  local backup_dir
-  backup_dir=$1
-  (
-    pushd /tmp/workspace
-    set -x
-    ${AWS_S3_CP} --recursive . "${BACKUP_BUCKET}/${CIRCLE_TAG}/${backup_dir}/"
-  )
-}
-
-conda_upload() {
-  (
-    set -x
-    ${ANACONDA} \
-      upload  \
-      ${PKG_DIR}/*.tar.bz2 \
-      -u "pytorch-${UPLOAD_CHANNEL}" \
-      --label main \
-      --no-progress \
-      --force
-  )
-}
-
-s3_upload() {
-  local extension
-  local pkg_type
-  extension="$1"
-  pkg_type="$2"
-  s3_dir="${UPLOAD_BUCKET}/${pkg_type}/${UPLOAD_CHANNEL}/${UPLOAD_SUBFOLDER}/"
-  (
-    for pkg in ${PKG_DIR}/*.${extension}; do
-      (
-        set -x
-        ${AWS_S3_CP} --no-progress --acl public-read "${pkg}" "${s3_dir}"
-      )
-    done
-  )
-}
-
-case "${PACKAGE_TYPE}" in
-  conda)
-    conda_upload
-    # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-    # Because there's no actual conda command to read this
-    subdir=$(\
-      tar -xOf ${PKG_DIR}/*.bz2 info/index.json \
-        | grep subdir  \
-        | cut -d ':' -f2 \
-        | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//' \
-    )
-    BACKUP_DIR="conda/${subdir}"
-    ;;
-  libtorch)
-    s3_upload "zip" "libtorch"
-    BACKUP_DIR="libtorch/${UPLOAD_CHANNEL}/${UPLOAD_SUBFOLDER}"
-    ;;
-  # wheel can either refer to wheel/manywheel
-  *wheel)
-    s3_upload "whl" "whl"
-    BACKUP_DIR="whl/${UPLOAD_CHANNEL}/${UPLOAD_SUBFOLDER}"
-    ;;
-  *)
-    echo "ERROR: unknown package type: ${PACKAGE_TYPE}"
-    exit 1
-    ;;
-esac
-
-# CIRCLE_TAG is defined by upstream circleci,
-# this can be changed to recognize tagged versions
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  do_backup "${BACKUP_DIR}"
-fi

.circleci/scripts/binary_windows_build.sh

@@ -1,45 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/c/w/env"
-mkdir -p "$PYTORCH_FINAL_PACKAGE_DIR"
-
-export CUDA_VERSION="${DESIRED_CUDA/cu/}"
-export USE_SCCACHE=1
-export SCCACHE_BUCKET=ossci-compiler-cache-windows
-export NIGHTLIES_PYTORCH_ROOT="$PYTORCH_ROOT"
-
-if [[ "$CUDA_VERSION" == "92" || "$CUDA_VERSION" == "100" ]]; then
-  export VC_YEAR=2017
-else
-  export VC_YEAR=2019
-fi
-
-if [[ "${DESIRED_CUDA}" == "cu111" ]]; then
-  export BUILD_SPLIT_CUDA="ON"
-fi
-
-set +x
-export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}
-export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}
-set -x
-
-if [[ "$CIRCLECI" == 'true' && -d "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages\\_Instances" ]]; then
-  mv "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages\\_Instances" .
-  rm -rf "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-  mkdir -p "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-  mv _Instances "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-fi
-
-echo "Free space on filesystem before build:"
-df -h
-
-pushd "$BUILDER_ROOT"
-if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
-  ./windows/internal/build_conda.bat
-elif [[ "$PACKAGE_TYPE" == 'wheel' || "$PACKAGE_TYPE" == 'libtorch' ]]; then
-  ./windows/internal/build_wheels.bat
-fi
-
-echo "Free space on filesystem after build:"
-df -h

.circleci/scripts/binary_windows_test.sh

@@ -1,19 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/c/w/env"
-
-export CUDA_VERSION="${DESIRED_CUDA/cu/}"
-export VC_YEAR=2017
-
-if [[ "$CUDA_VERSION" == "92" || "$CUDA_VERSION" == "100" ]]; then
-  export VC_YEAR=2017
-else
-  export VC_YEAR=2019
-fi
-
-pushd "$BUILDER_ROOT"
-
-./windows/internal/smoke_test.bat
-
-popd

.circleci/scripts/build_android_gradle.sh

@@ -1,11 +1,7 @@
 #!/usr/bin/env bash
 set -eux -o pipefail
 
-env
-echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
-
 export ANDROID_NDK_HOME=/opt/ndk
-export ANDROID_NDK=/opt/ndk
 export ANDROID_HOME=/opt/android/sdk
 
 # Must be in sync with GRADLE_VERSION in docker image for android
@@ -14,31 +10,6 @@ export GRADLE_VERSION=4.10.3
 export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
 export GRADLE_PATH=$GRADLE_HOME/bin/gradle
 
-# touch gradle cache files to prevent expiration
-while IFS= read -r -d '' file
-do
-  touch "$file" || true
-done < <(find /var/lib/jenkins/.gradle -type f -print0)
-
-export GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
-rm -f $GRADLE_LOCAL_PROPERTIES
-echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-echo "cmake.dir=/usr/local" >> $GRADLE_LOCAL_PROPERTIES
-
-retry () {
-  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-}
-
-# Run custom build script
-if [[ "${BUILD_ENVIRONMENT}" == *-gradle-custom-build* ]]; then
-  # Install torch & torchvision - used to download & dump used ops from test model.
-  retry pip install torch torchvision --progress-bar off
-
-  exec "$(dirname "${BASH_SOURCE[0]}")/../../android/build_test_app_custom.sh" armeabi-v7a
-fi
-
-# Run default build
 BUILD_ANDROID_INCLUDE_DIR_x86=~/workspace/build_android/install/include
 BUILD_ANDROID_LIB_DIR_x86=~/workspace/build_android/install/lib
 
@@ -73,6 +44,9 @@ ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v8a} ${JNI_INCLUDE_DIR}/arm64-v8a
 ln -s ${BUILD_ANDROID_LIB_DIR_arm_v8a} ${JNI_LIBS_DIR}/arm64-v8a
 fi
 
+env
+echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
+
 GRADLE_PARAMS="-p android assembleRelease --debug --stacktrace"
 if [[ "${BUILD_ENVIRONMENT}" == *-gradle-build-only-x86_32* ]]; then
     GRADLE_PARAMS+=" -PABI_FILTERS=x86"
@@ -82,6 +56,20 @@ if [ -n "{GRADLE_OFFLINE:-}" ]; then
     GRADLE_PARAMS+=" --offline"
 fi
 
+# touch gradle cache files to prevent expiration
+while IFS= read -r -d '' file
+do
+  touch "$file" || true
+done < <(find /var/lib/jenkins/.gradle -type f -print0)
+
+env
+
+export GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
+rm -f $GRADLE_LOCAL_PROPERTIES
+echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
+echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
+echo "cmake.dir=/usr/local" >> $GRADLE_LOCAL_PROPERTIES
+
 $GRADLE_PATH $GRADLE_PARAMS
 
 find . -type f -name "*.a" -exec ls -lh {} \;

.circleci/scripts/cpp_doc_push_script.sh

@@ -30,7 +30,13 @@ if [ "$version" == "master" ]; then
   is_master_doc=true
 fi
 
-echo "install_path: $install_path  version: $version"
+# Argument 3: (optional) If present, we will NOT do any pushing. Used for testing.
+dry_run=false
+if [ "$3" != "" ]; then
+  dry_run=true
+fi
+
+echo "install_path: $install_path  version: $version  dry_run: $dry_run"
 
 # ======================== Building PyTorch C++ API Docs ========================
 
@@ -47,22 +53,31 @@ sudo apt-get -y install doxygen
 # Generate ATen files
 pushd "${pt_checkout}"
 pip install -r requirements.txt
-time python -m tools.codegen.gen \
+time python aten/src/ATen/gen.py \
   -s aten/src/ATen \
-  -d build/aten/src/ATen
+  -d build/aten/src/ATen \
+  aten/src/ATen/Declarations.cwrap \
+  aten/src/THCUNN/generic/THCUNN.h \
+  aten/src/ATen/nn.yaml \
+  aten/src/ATen/native/native_functions.yaml
 
 # Copy some required files
+cp aten/src/ATen/common_with_cwrap.py tools/shared/cwrap_common.py
 cp torch/_utils_internal.py tools/shared
 
 # Generate PyTorch files
 time python tools/setup_helpers/generate_code.py \
   --declarations-path build/aten/src/ATen/Declarations.yaml \
-  --native-functions-path aten/src/ATen/native/native_functions.yaml \
   --nn-path aten/src/
 
 # Build the docs
 pushd docs/cpp
-pip install -r requirements.txt
+pip install breathe>=4.13.0 bs4 lxml six
+pip install --no-cache-dir -e "git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme"
+pip install exhale>=0.2.1
+pip install sphinx>=2.0
+# Uncomment once it is fixed
+# pip install -r requirements.txt
 time make VERBOSE=1 html -j
 
 popd
@@ -88,8 +103,24 @@ git status
 git config user.email "soumith+bot@pytorch.org"
 git config user.name "pytorchbot"
 # If there aren't changes, don't make a commit; push is no-op
-git commit -m "Generate C++ docs from pytorch/pytorch@$CIRCLE_SHA1" || true
+git commit -m "Automatic sync on $(date)" || true
 git status
 
+if [ "$dry_run" = false ]; then
+  echo "Pushing to https://github.com/pytorch/cppdocs"
+  set +x
+/usr/bin/expect <<DONE
+  spawn git push -u origin master
+  expect "Username*"
+  send "pytorchbot\n"
+  expect "Password*"
+  send "$::env(GITHUB_PYTORCHBOT_TOKEN)\n"
+  expect eof
+DONE
+  set -x
+else
+  echo "Skipping push due to dry_run"
+fi
+
 popd
 # =================== The above code **should** be executed inside Docker container ===================

.circleci/scripts/driver_update.bat

@@ -1,8 +0,0 @@
-set "DRIVER_DOWNLOAD_LINK=https://s3.amazonaws.com/ossci-windows/452.39-data-center-tesla-desktop-win10-64bit-international.exe"
-curl --retry 3 -kL %DRIVER_DOWNLOAD_LINK% --output 452.39-data-center-tesla-desktop-win10-64bit-international.exe
-if errorlevel 1 exit /b 1
-
-start /wait 452.39-data-center-tesla-desktop-win10-64bit-international.exe -s -noreboot
-if errorlevel 1 exit /b 1
-
-del 452.39-data-center-tesla-desktop-win10-64bit-international.exe || ver > NUL

.circleci/scripts/python_doc_push_script.sh

@@ -7,8 +7,6 @@ sudo apt-get -y install expect-dev
 # This is where the local pytorch install in the docker image is located
 pt_checkout="/var/lib/jenkins/workspace"
 
-source "$pt_checkout/.jenkins/pytorch/common_utils.sh"
-
 echo "python_doc_push_script.sh: Invoked with $*"
 
 set -ex
@@ -40,30 +38,15 @@ echo "error: python_doc_push_script.sh: branch (arg3) not specified"
   exit 1
 fi
 
-echo "install_path: $install_path  version: $version"
+# Argument 4: (optional) If present, we will NOT do any pushing. Used for testing.
+dry_run=false
+if [ "$4" != "" ]; then
+  dry_run=true
+fi
 
+echo "install_path: $install_path  version: $version  dry_run: $dry_run"
 
-build_docs () {
-  set +e
-  set -o pipefail
-  make $1 2>&1 | tee /tmp/docs_build.txt
-  code=$?
-  if [ $code -ne 0 ]; then
-    set +x
-    echo =========================
-    grep "WARNING:" /tmp/docs_build.txt
-    echo =========================
-    echo Docs build failed. If the failure is not clear, scan back in the log
-    echo for any WARNINGS or for the line "build finished with problems"
-    echo "(tried to echo the WARNINGS above the ==== line)"
-    echo =========================
-  fi
-  set -ex
-  return $code
-}
-
-
-git clone https://github.com/pytorch/pytorch.github.io -b $branch --depth 1
+git clone https://github.com/pytorch/pytorch.github.io -b $branch
 pushd pytorch.github.io
 
 export LC_ALL=C
@@ -71,38 +54,26 @@ export PATH=/opt/conda/bin:$PATH
 
 rm -rf pytorch || true
 
+# Install TensorBoard in python 3 so torch.utils.tensorboard classes render
+pip install -q https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py3-none-any.whl
+
 # Get all the documentation sources, put them in one place
 pushd "$pt_checkout"
+git clone https://github.com/pytorch/vision
+pushd vision
+conda install -q pillow
+time python setup.py install
+popd
 pushd docs
+rm -rf source/torchvision
+cp -a ../vision/docs/source source/torchvision
 
 # Build the docs
-pip -q install -r requirements.txt
+pip -q install -r requirements.txt || true
 if [ "$is_master_doc" = true ]; then
-  build_docs html
-  [ $? -eq 0 ] || exit $?
-  make coverage
-  # Now we have the coverage report, we need to make sure it is empty.
-  # Count the number of lines in the file and turn that number into a variable
-  # $lines. The `cut -f1 ...` is to only parse the number, not the filename
-  # Skip the report header by subtracting 2: the header will be output even if
-  # there are no undocumented items.
-  #
-  # Also: see docs/source/conf.py for "coverage_ignore*" items, which should
-  # be documented then removed from there.
-  lines=$(wc -l build/coverage/python.txt 2>/dev/null |cut -f1 -d' ')
-  undocumented=$(($lines - 2))
-  if [ $undocumented -lt 0 ]; then
-    echo coverage output not found
-    exit 1
-  elif [ $undocumented -gt 0 ]; then
-    echo undocumented objects found:
-    cat build/coverage/python.txt
-    exit 1
-  fi
+  make html
 else
-  # skip coverage, format for stable or tags
-  build_docs html-stable
-  [ $? -eq 0 ] || exit $?
+  make html-stable
 fi
 
 # Move them into the docs repo
@@ -111,6 +82,14 @@ popd
 git rm -rf "$install_path" || true
 mv "$pt_checkout/docs/build/html" "$install_path"
 
+# Add the version handler by search and replace.
+# XXX: Consider moving this to the docs Makefile or site build
+if [ "$is_master_doc" = true ]; then
+  find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>\1 \&#x25BC</a>@g"
+else
+  find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>$version \&#x25BC</a>@g"
+fi
+
 # Prevent Google from indexing $install_path/_modules. This folder contains
 # generated source files.
 # NB: the following only works on gnu sed. The sed shipped with mac os is different.
@@ -122,8 +101,24 @@ git status
 git config user.email "soumith+bot@pytorch.org"
 git config user.name "pytorchbot"
 # If there aren't changes, don't make a commit; push is no-op
-git commit -m "Generate Python docs from pytorch/pytorch@$CIRCLE_SHA1" || true
+git commit -m "auto-generating sphinx docs" || true
 git status
 
+if [ "$dry_run" = false ]; then
+  echo "Pushing to pytorch.github.io:$branch"
+  set +x
+/usr/bin/expect <<DONE
+  spawn git push origin $branch
+  expect "Username*"
+  send "pytorchbot\n"
+  expect "Password*"
+  send "$::env(GITHUB_PYTORCHBOT_TOKEN)\n"
+  expect eof
+DONE
+  set -x
+else
+  echo "Skipping push due to dry_run"
+fi
+
 popd
 # =================== The above code **should** be executed inside Docker container ===================

.circleci/scripts/setup_ci_environment.sh

@@ -1,90 +1,81 @@
 #!/usr/bin/env bash
 set -ex -o pipefail
 
+# Set up NVIDIA docker repo
+curl -s -L --retry 3 https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
+echo "deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
+echo "deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
+echo "deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
+
 # Remove unnecessary sources
 sudo rm -f /etc/apt/sources.list.d/google-chrome.list
 sudo rm -f /etc/apt/heroku.list
 sudo rm -f /etc/apt/openjdk-r-ubuntu-ppa-xenial.list
 sudo rm -f /etc/apt/partner.list
 
-retry () {
-  $*  || $* || $* || $* || $*
-}
-
-# Method adapted from here: https://askubuntu.com/questions/875213/apt-get-to-retry-downloading
-# (with use of tee to avoid permissions problems)
-# This is better than retrying the whole apt-get command
-echo "APT::Acquire::Retries \"3\";" | sudo tee /etc/apt/apt.conf.d/80-retries
-
-retry sudo apt-get update -qq
-retry sudo apt-get -y install \
+sudo apt-get -y update
+sudo apt-get -y remove linux-image-generic linux-headers-generic linux-generic docker-ce
+# WARNING: Docker version is hardcoded here; you must update the
+# version number below for docker-ce and nvidia-docker2 to get newer
+# versions of Docker.  We hardcode these numbers because we kept
+# getting broken CI when Docker would update their docker version,
+# and nvidia-docker2 would be out of date for a day until they
+# released a newer version of their package.
+#
+# How to figure out what the correct versions of these packages are?
+# My preferred method is to start a Docker instance of the correct
+# Ubuntu version (e.g., docker run -it ubuntu:16.04) and then ask
+# apt what the packages you need are.  Note that the CircleCI image
+# comes with Docker.
+sudo apt-get -y install \
+  linux-headers-$(uname -r) \
+  linux-image-generic \
   moreutils \
+  docker-ce=5:18.09.4~3-0~ubuntu-xenial \
+  nvidia-container-runtime=2.0.0+docker18.09.4-1 \
+  nvidia-docker2=2.0.3+docker18.09.4-1 \
   expect-dev
 
-echo "== DOCKER VERSION =="
-docker version
+sudo pkill -SIGHUP dockerd
+
+retry () {
+    $*  || $* || $* || $* || $*
+}
 
 retry sudo pip -q install awscli==1.16.35
 
 if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  DRIVER_FN="NVIDIA-Linux-x86_64-460.39.run"
+  DRIVER_FN="NVIDIA-Linux-x86_64-440.59.run"
   wget "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"
   sudo /bin/bash "$DRIVER_FN" -s --no-drm || (sudo cat /var/log/nvidia-installer.log && false)
   nvidia-smi
-
-  # Taken directly from https://github.com/NVIDIA/nvidia-docker
-  # Add the package repositories
-  distribution=$(. /etc/os-release;echo "$ID$VERSION_ID")
-  curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
-  curl -s -L "https://nvidia.github.io/nvidia-docker/${distribution}/nvidia-docker.list" | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
-
-  sudo apt-get update -qq
-  # Necessary to get the `--gpus` flag to function within docker
-  sudo apt-get install -y nvidia-container-toolkit
-  sudo systemctl restart docker
-else
-  # Explicitly remove nvidia docker apt repositories if not building for cuda
-  sudo rm -rf /etc/apt/sources.list.d/nvidia-docker.list
 fi
 
-add_to_env_file() {
-  local content
-  content=$1
-  # BASH_ENV should be set by CircleCI
-  echo "${content}" >> "${BASH_ENV:-/tmp/env}"
-}
-
-add_to_env_file "IN_CI=1"
-add_to_env_file "COMMIT_SOURCE=${CIRCLE_BRANCH:-}"
-add_to_env_file "BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}"
-add_to_env_file "CIRCLE_PULL_REQUEST=${CIRCLE_PULL_REQUEST}"
-
-
 if [[ "${BUILD_ENVIRONMENT}" == *-build ]]; then
-  add_to_env_file "SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2"
-
-  SCCACHE_MAX_JOBS=$(( $(nproc) - 1 ))
-  MEMORY_LIMIT_MAX_JOBS=8  # the "large" resource class on CircleCI has 32 CPU cores, if we use all of them we'll OOM
-  MAX_JOBS=$(( ${SCCACHE_MAX_JOBS} > ${MEMORY_LIMIT_MAX_JOBS} ? ${MEMORY_LIMIT_MAX_JOBS} : ${SCCACHE_MAX_JOBS} ))
-  add_to_env_file "MAX_JOBS=${MAX_JOBS}"
-
+  echo "declare -x IN_CIRCLECI=1" > /home/circleci/project/env
+  echo "declare -x COMMIT_SOURCE=${CIRCLE_BRANCH:-}" >> /home/circleci/project/env
+  echo "declare -x SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2" >> /home/circleci/project/env
   if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-    add_to_env_file "TORCH_CUDA_ARCH_LIST=5.2"
+    echo "declare -x TORCH_CUDA_ARCH_LIST=5.2" >> /home/circleci/project/env
   fi
+  export SCCACHE_MAX_JOBS=`expr $(nproc) - 1`
+  export MEMORY_LIMIT_MAX_JOBS=8  # the "large" resource class on CircleCI has 32 CPU cores, if we use all of them we'll OOM
+  export MAX_JOBS=$(( ${SCCACHE_MAX_JOBS} > ${MEMORY_LIMIT_MAX_JOBS} ? ${MEMORY_LIMIT_MAX_JOBS} : ${SCCACHE_MAX_JOBS} ))
+  echo "declare -x MAX_JOBS=${MAX_JOBS}" >> /home/circleci/project/env
 
   if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
     # This IAM user allows write access to S3 bucket for sccache & bazels3cache
     set +x
-    add_to_env_file "XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}"
-    add_to_env_file "AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}"
-    add_to_env_file "AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}"
+    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
+    echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
+    echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
     set -x
   else
     # This IAM user allows write access to S3 bucket for sccache
     set +x
-    add_to_env_file "XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}"
-    add_to_env_file "AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}"
-    add_to_env_file "AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}"
+    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
+    echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
+    echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
     set -x
   fi
 fi
@@ -93,5 +84,5 @@ fi
 set +x
 export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_ECR_READ_WRITE_V4:-}
 export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_ECR_READ_WRITE_V4:-}
-eval "$(aws ecr get-login --region us-east-1 --no-include-email)"
+eval $(aws ecr get-login --region us-east-1 --no-include-email)
 set -x

.circleci/scripts/setup_linux_system_environment.sh

@@ -33,7 +33,7 @@ systemctl list-units --all | cat
 sudo pkill apt-get || true
 
 # For even better luck, purge unattended-upgrades
-sudo apt-get purge -y unattended-upgrades || true
+sudo apt-get purge -y unattended-upgrades
 
 cat /etc/apt/sources.list
 

.circleci/scripts/upload_binary_size_to_scuba.py

@@ -3,11 +3,9 @@ import json
 import logging
 import os
 import os.path
-import pathlib
 import re
 import sys
 import time
-import zipfile
 
 import requests
 
@@ -41,18 +39,16 @@ def build_message(size):
             "build_num": os.environ.get("CIRCLE_BUILD_NUM"),
             "sha1": os.environ.get("CIRCLE_SHA1"),
             "branch": os.environ.get("CIRCLE_BRANCH"),
-            "workflow_id": os.environ.get("CIRCLE_WORKFLOW_ID"),
         },
         "int": {
             "time": int(time.time()),
             "size": size,
             "commit_time": int(os.environ.get("COMMIT_TIME", "0")),
-            "run_duration": int(time.time() - os.path.getmtime(os.path.realpath(__file__))),
         },
     }
 
 
-def send_message(messages):
+def send_message(message):
     access_token = os.environ.get("SCRIBE_GRAPHQL_ACCESS_TOKEN")
     if not access_token:
         raise ValueError("Can't find access token from environment variable")
@@ -68,7 +64,6 @@ def send_message(messages):
                         "message": json.dumps(message),
                         "line_escape": False,
                     }
-                    for message in messages
                 ]
             ),
         },
@@ -77,59 +72,6 @@ def send_message(messages):
     r.raise_for_status()
 
 
-def report_android_sizes(file_dir):
-    def gen_sizes():
-        # we should only expect one file, if no, something is wrong
-        aar_files = list(pathlib.Path(file_dir).rglob("pytorch_android-*.aar"))
-        if len(aar_files) != 1:
-            logging.exception(f"error getting aar files from: {file_dir} / {aar_files}")
-            return
-
-        aar_file = aar_files[0]
-        zf = zipfile.ZipFile(aar_file)
-        for info in zf.infolist():
-            # Scan ".so" libs in `jni` folder. Examples:
-            # jni/arm64-v8a/libfbjni.so
-            # jni/arm64-v8a/libpytorch_jni.so
-            m = re.match(r"^jni/([^/]+)/(.*\.so)$", info.filename)
-            if not m:
-                continue
-            arch, lib = m.groups()
-            # report per architecture library size
-            yield [arch, lib, info.compress_size, info.file_size]
-
-        # report whole package size
-        yield ["aar", aar_file.name, os.stat(aar_file).st_size, 0]
-
-    def gen_messages():
-        android_build_type = os.environ.get("ANDROID_BUILD_TYPE")
-        for arch, lib, comp_size, uncomp_size in gen_sizes():
-            print(android_build_type, arch, lib, comp_size, uncomp_size)
-            yield {
-                "normal": {
-                    "os": "android",
-                    # TODO: create dedicated columns
-                    "pkg_type": "{}/{}/{}".format(android_build_type, arch, lib),
-                    "cu_ver": "",  # dummy value for derived field `build_name`
-                    "py_ver": "",  # dummy value for derived field `build_name`
-                    "pr": os.environ.get("CIRCLE_PR_NUMBER"),
-                    "build_num": os.environ.get("CIRCLE_BUILD_NUM"),
-                    "sha1": os.environ.get("CIRCLE_SHA1"),
-                    "branch": os.environ.get("CIRCLE_BRANCH"),
-                    "workflow_id": os.environ.get("CIRCLE_WORKFLOW_ID"),
-                },
-                "int": {
-                    "time": int(time.time()),
-                    "commit_time": int(os.environ.get("COMMIT_TIME", "0")),
-                    "run_duration": int(time.time() - os.path.getmtime(os.path.realpath(__file__))),
-                    "size": comp_size,
-                    "raw_size": uncomp_size,
-                },
-            }
-
-    send_message(list(gen_messages()))
-
-
 if __name__ == "__main__":
     file_dir = os.environ.get(
         "PYTORCH_FINAL_PACKAGE_DIR", "/home/circleci/project/final_pkgs"
@@ -137,13 +79,9 @@ if __name__ == "__main__":
     if len(sys.argv) == 2:
         file_dir = sys.argv[1]
     print("checking dir: " + file_dir)
-
-    if "-android" in os.environ.get("BUILD_ENVIRONMENT", ""):
-        report_android_sizes(file_dir)
-    else:
-        size = get_size(file_dir)
-        if size != 0:
-            try:
-                send_message([build_message(size)])
-            except:
-                logging.exception("can't send message")
+    size = get_size(file_dir)
+    if size != 0:
+        try:
+            send_message(build_message(size))
+        except:
+            logging.exception("can't send message")

.circleci/scripts/vs_install.ps1

@@ -1,7 +1,6 @@
 $VS_DOWNLOAD_LINK = "https://aka.ms/vs/15/release/vs_buildtools.exe"
-$COLLECT_DOWNLOAD_LINK = "https://aka.ms/vscollect.exe"
 $VS_INSTALL_ARGS = @("--nocache","--quiet","--wait", "--add Microsoft.VisualStudio.Workload.VCTools",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Tools.14.13",
+                                                     "--add Microsoft.VisualStudio.Component.VC.Tools.14.11",
                                                      "--add Microsoft.Component.MSBuild",
                                                      "--add Microsoft.VisualStudio.Component.Roslyn.Compiler",
                                                      "--add Microsoft.VisualStudio.Component.TextTemplating",
@@ -22,13 +21,5 @@ Remove-Item -Path vs_installer.exe -Force
 $exitCode = $process.ExitCode
 if (($exitCode -ne 0) -and ($exitCode -ne 3010)) {
     echo "VS 2017 installer exited with code $exitCode, which should be one of [0, 3010]."
-    curl.exe --retry 3 -kL $COLLECT_DOWNLOAD_LINK --output Collect.exe
-    if ($LASTEXITCODE -ne 0) {
-        echo "Download of the VS Collect tool failed."
-        exit 1
-    }
-    Start-Process "${PWD}\Collect.exe" -NoNewWindow -Wait -PassThru
-    New-Item -Path "C:\w\build-results" -ItemType "directory" -Force
-    Copy-Item -Path "C:\Users\circleci\AppData\Local\Temp\vslogs.zip" -Destination "C:\w\build-results\"
     exit 1
 }

.circleci/scripts/vs_install_cmath.ps1

@@ -1,5 +0,0 @@
-$CMATH_DOWNLOAD_LINK = "https://raw.githubusercontent.com/microsoft/STL/12c684bba78f9b032050526abdebf14f58ca26a3/stl/inc/cmath"
-$VC14_28_INSTALL_PATH="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29910\include"
-
-curl.exe --retry 3 -kL $CMATH_DOWNLOAD_LINK --output "$home\cmath"
-Move-Item -Path "$home\cmath" -Destination "$VC14_28_INSTALL_PATH" -Force

.circleci/scripts/windows_cuda_install.sh

@@ -1,61 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-cuda_major_version=${CUDA_VERSION%.*}
-
-if [[ "$cuda_major_version" == "10" ]]; then
-    cuda_installer_name="cuda_10.1.243_426.00_win10"
-    msbuild_project_dir="CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions"
-    cuda_install_packages="nvcc_10.1 cuobjdump_10.1 nvprune_10.1 cupti_10.1 cublas_10.1 cublas_dev_10.1 cudart_10.1 cufft_10.1 cufft_dev_10.1 curand_10.1 curand_dev_10.1 cusolver_10.1 cusolver_dev_10.1 cusparse_10.1 cusparse_dev_10.1 nvgraph_10.1 nvgraph_dev_10.1 npp_10.1 npp_dev_10.1 nvrtc_10.1 nvrtc_dev_10.1 nvml_dev_10.1"
-elif [[ "$cuda_major_version" == "11" ]]; then
-    cuda_installer_name="cuda_11.1.0_456.43_win10"
-    msbuild_project_dir="visual_studio_integration/CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions"
-    cuda_install_packages="nvcc_11.1 cuobjdump_11.1 nvprune_11.1 nvprof_11.1 cupti_11.1 cublas_11.1 cublas_dev_11.1 cudart_11.1 cufft_11.1 cufft_dev_11.1 curand_11.1 curand_dev_11.1 cusolver_11.1 cusolver_dev_11.1 cusparse_11.1 cusparse_dev_11.1 npp_11.1 npp_dev_11.1 nvrtc_11.1 nvrtc_dev_11.1 nvml_dev_11.1"
-else
-    echo "CUDA_VERSION $CUDA_VERSION is not supported yet"
-    exit 1
-fi
-
-if [[ "$cuda_major_version" == "11" && "${JOB_EXECUTOR}" == "windows-with-nvidia-gpu" ]]; then
-    cuda_install_packages="${cuda_install_packages} Display.Driver"
-fi
-
-cuda_installer_link="https://ossci-windows.s3.amazonaws.com/${cuda_installer_name}.exe"
-
-curl --retry 3 -kLO $cuda_installer_link
-7z x ${cuda_installer_name}.exe -o${cuda_installer_name}
-cd ${cuda_installer_name}
-mkdir cuda_install_logs
-
-set +e
-
-./setup.exe -s ${cuda_install_packages} -loglevel:6 -log:"$(pwd -W)/cuda_install_logs"
-
-set -e
-
-if [[ "${VC_YEAR}" == "2017" ]]; then
-    cp -r ${msbuild_project_dir}/* "C:/Program Files (x86)/Microsoft Visual Studio/2017/${VC_PRODUCT}/Common7/IDE/VC/VCTargets/BuildCustomizations/"
-else
-    cp -r ${msbuild_project_dir}/* "C:/Program Files (x86)/Microsoft Visual Studio/2019/${VC_PRODUCT}/MSBuild/Microsoft/VC/v160/BuildCustomizations/"
-fi
-
-if ! ls "/c/Program Files/NVIDIA Corporation/NvToolsExt/bin/x64/nvToolsExt64_1.dll"
-then
-    curl --retry 3 -kLO https://ossci-windows.s3.amazonaws.com/NvToolsExt.7z
-    7z x NvToolsExt.7z -oNvToolsExt
-    mkdir -p "C:/Program Files/NVIDIA Corporation/NvToolsExt"
-    cp -r NvToolsExt/* "C:/Program Files/NVIDIA Corporation/NvToolsExt/"
-    export NVTOOLSEXT_PATH="C:\\Program Files\\NVIDIA Corporation\\NvToolsExt\\"
-fi
-
-if ! ls "/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v${CUDA_VERSION}/bin/nvcc.exe"
-then
-    echo "CUDA installation failed"
-    mkdir -p /c/w/build-results
-    7z a "c:\\w\\build-results\\cuda_install_logs.7z" cuda_install_logs
-    exit 1
-fi
-
-cd ..
-rm -rf ./${cuda_installer_name}
-rm -f ./${cuda_installer_name}.exe

.circleci/scripts/windows_cudnn_install.sh

@@ -1,21 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-cuda_major_version=${CUDA_VERSION%.*}
-
-if [[ "$cuda_major_version" == "10" ]]; then
-    cudnn_installer_name="cudnn-${CUDA_VERSION}-windows10-x64-v7.6.4.38"
-elif [[ "$cuda_major_version" == "11" ]]; then
-    cudnn_installer_name="cudnn-${CUDA_VERSION}-windows-x64-v8.0.5.39"
-else
-    echo "CUDNN for CUDA_VERSION $CUDA_VERSION is not supported yet"
-    exit 1
-fi
-
-cudnn_installer_link="https://ossci-windows.s3.amazonaws.com/${cudnn_installer_name}.zip"
-
-curl --retry 3 -O $cudnn_installer_link
-7z x ${cudnn_installer_name}.zip -ocudnn
-cp -r cudnn/cuda/* "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v${CUDA_VERSION}/"
-rm -rf cudnn
-rm -f ${cudnn_installer_name}.zip

.circleci/validate-docker-version.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
+import cimodel.data.pytorch_build_definitions as pytorch_build_definitions
+from yaml import load
+
+try:
+    from yaml import CLoader as Loader
+except ImportError:
+    from yaml import Loader
+
+
+def load_config(filename=".circleci/config.yml"):
+    with open(filename, "r") as fh:
+        return load("".join(fh.readlines()), Loader)
+
+
+def load_tags_for_projects(workflow_config):
+    return {
+        v["ecr_gc_job"]["project"]: v["ecr_gc_job"]["tags_to_keep"]
+        for v in workflow_config["workflows"]["ecr_gc"]["jobs"]
+        if isinstance(v, dict) and "ecr_gc_job" in v
+    }
+
+
+def check_version(job, tags, expected_version):
+    valid_versions = tags[job].split(",")
+    if expected_version not in valid_versions:
+        raise RuntimeError(
+            "We configured {} to use Docker version {}; but this "
+            "version is not configured in job ecr_gc_job_for_{}.  Non-deployed versions will be "
+            "garbage collected two weeks after they are created.  DO NOT LAND "
+            "THIS TO MASTER without also updating ossci-job-dsl with this version."
+            "\n\nDeployed versions: {}".format(job, expected_version, job, tags[job])
+        )
+
+
+def validate_docker_version():
+    tags = load_tags_for_projects(load_config())
+    check_version("pytorch", tags, pytorch_build_definitions.DOCKER_IMAGE_VERSION)
+    check_version("caffe2", tags, caffe2_build_definitions.DOCKER_IMAGE_VERSION)
+
+
+if __name__ == "__main__":
+    validate_docker_version()

.circleci/verbatim-sources/binary-build-params.yml

@@ -52,15 +52,3 @@ binary_mac_params: &binary_mac_params
   environment:
     BUILD_ENVIRONMENT: << parameters.build_environment >>
 
-binary_windows_params: &binary_windows_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    executor:
-      type: string
-      default: "windows-xlarge-cpu-with-nvidia-cuda"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    BUILD_FOR_SYSTEM: windows
-    JOB_EXECUTOR: <<parameters.executor>>

.circleci/verbatim-sources/binary-build-tests.yml

@@ -1,14 +1,14 @@
 
 # There is currently no testing for libtorch TODO
-#  binary_linux_libtorch_3.6m_cpu_test:
+#  binary_linux_libtorch_2.7m_cpu_test:
 #    environment:
-#      BUILD_ENVIRONMENT: "libtorch 3.6m cpu"
+#      BUILD_ENVIRONMENT: "libtorch 2.7m cpu"
 #    resource_class: gpu.medium
 #    <<: *binary_linux_test
 #
-#  binary_linux_libtorch_3.6m_cu90_test:
+#  binary_linux_libtorch_2.7m_cu90_test:
 #    environment:
-#      BUILD_ENVIRONMENT: "libtorch 3.6m cu90"
+#      BUILD_ENVIRONMENT: "libtorch 2.7m cu90"
 #    resource_class: gpu.medium
 #    <<: *binary_linux_test
 #

.circleci/verbatim-sources/binary-job-specs.yml

@@ -1,42 +1,60 @@
   binary_linux_build:
     <<: *binary_linux_build_params
     steps:
-    - checkout
-    - calculate_docker_image_tag
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_scripts
     - run:
         <<: *binary_checkout
     - run:
         <<: *binary_populate_env
+    - run:
+        name: Install unbuffer and ts
+        command: |
+            set -eux -o pipefail
+            source /env
+            OS_NAME=`awk -F= '/^NAME/{print $2}' /etc/os-release`
+            if [[ "$OS_NAME" == *"CentOS Linux"* ]]; then
+              retry yum -q -y install epel-release
+              retry yum -q -y install expect moreutils
+            elif [[ "$OS_NAME" == *"Ubuntu"* ]]; then
+              retry apt-get update
+              retry apt-get -y install expect moreutils
+              retry conda install -y -c eumetsat expect
+              retry conda install -y cmake
+            fi
+    - run:
+        name: Update compiler to devtoolset7
+        command: |
+            set -eux -o pipefail
+            source /env
+            if [[ "$DESIRED_DEVTOOLSET" == 'devtoolset7' ]]; then
+              source "/builder/update_compiler.sh"
+
+              # Env variables are not persisted into the next step
+              echo "export PATH=$PATH" >> /env
+              echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> /env
+            else
+              echo "Not updating compiler"
+            fi
     - run:
         name: Build
         no_output_timeout: "1h"
         command: |
             source "/pytorch/.circleci/scripts/binary_linux_build.sh"
-            # Preserve build log
-            if [ -f /pytorch/build/.ninja_log ]; then
-              cp /pytorch/build/.ninja_log /final_pkgs
-            fi
-    - run:
-        name: Output binary sizes
-        no_output_timeout: "1m"
-        command: |
-            ls -lah /final_pkgs
     - run:
         name: save binary size
         no_output_timeout: "5m"
         command: |
             source /env
             cd /pytorch && export COMMIT_TIME=$(git log --max-count=1 --format=%ct || echo 0)
-            python3 -mpip install requests && \
+            pip3 install requests && \
             SCRIBE_GRAPHQL_ACCESS_TOKEN=${SCRIBE_GRAPHQL_ACCESS_TOKEN} \
             python3 /pytorch/.circleci/scripts/upload_binary_size_to_scuba.py || exit 0
+
     - persist_to_workspace:
         root: /
         paths: final_pkgs
 
-    - store_artifacts:
-        path: /final_pkgs
-
     # This should really just be another step of the binary_linux_build job above.
     # This isn't possible right now b/c the build job uses the docker executor
     # (otherwise they'd be really really slow) but this one uses the macine
@@ -45,10 +63,11 @@
   binary_linux_test:
     <<: *binary_linux_test_upload_params
     machine:
-        image: ubuntu-1604:202007-01
+        image: ubuntu-1604:201903-01
     steps:
     # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
-    - checkout
+    - attach_scripts
+    # TODO: We shouldn't attach the workspace multiple times
     - attach_workspace:
         at: /home/circleci/project
     - setup_linux_system_environment
@@ -60,45 +79,29 @@
     - run:
         name: Prepare test code
         no_output_timeout: "1h"
-        command: .circleci/scripts/binary_linux_test.sh
+        command: ~/workspace/.circleci/scripts/binary_linux_test.sh
     - run:
         <<: *binary_run_in_docker
 
-  binary_upload:
-    parameters:
-      package_type:
-        type: string
-        description: "What type of package we are uploading (eg. wheel, libtorch, conda)"
-        default: "wheel"
-      upload_subfolder:
-        type: string
-        description: "What subfolder to put our package into (eg. cpu, cudaX.Y, etc.)"
-        default: "cpu"
-    docker:
-      - image: continuumio/miniconda3
-    environment:
-      - DRY_RUN: disabled
-      - PACKAGE_TYPE: "<< parameters.package_type >>"
-      - UPLOAD_SUBFOLDER: "<< parameters.upload_subfolder >>"
+  binary_linux_upload:
+    <<: *binary_linux_test_upload_params
+    machine:
+        image: ubuntu-1604:201903-01
     steps:
-      - attach_workspace:
-          at: /tmp/workspace
-      - checkout
-      - designate_upload_channel
-      - run:
-          name: Install dependencies
-          no_output_timeout: "1h"
-          command: |
-            conda install -yq anaconda-client
-            pip install -q awscli
-      - run:
-          name: Do upload
-          no_output_timeout: "1h"
-          command: |
-            AWS_ACCESS_KEY_ID="${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}" \
-              AWS_SECRET_ACCESS_KEY="${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}" \
-              ANACONDA_API_TOKEN="${CONDA_PYTORCHBOT_TOKEN}" \
-              .circleci/scripts/binary_upload.sh
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_scripts
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - attach_workspace:
+        at: /home/circleci/project
+    - run:
+        <<: *binary_populate_env
+    - run:
+        <<: *binary_install_miniconda
+    - run:
+        name: Upload
+        no_output_timeout: "1h"
+        command: ~/workspace/.circleci/scripts/binary_linux_upload.sh
 
   # Nighlty build smoke tests defaults
   # These are the second-round smoke tests. These make sure that the binaries are
@@ -108,10 +111,12 @@
   smoke_linux_test:
     <<: *binary_linux_test_upload_params
     machine:
-      image: ubuntu-1604:202007-01
+      image: ubuntu-1604:201903-01
     steps:
-    - checkout
-    - calculate_docker_image_tag
+    - attach_workspace:
+        at: ~/workspace
+    - attach_workspace:
+        at: /home/circleci/project
     - setup_linux_system_environment
     - setup_ci_environment
     - run:
@@ -135,9 +140,12 @@
   smoke_mac_test:
     <<: *binary_linux_test_upload_params
     macos:
-      xcode: "12.0"
+      xcode: "9.4.1"
     steps:
-      - checkout
+      - attach_workspace:
+          at: ~/workspace
+      - attach_workspace: # TODO - we can `cp` from ~/workspace
+          at: /Users/distiller/project
       - run:
           <<: *binary_checkout
       - run:
@@ -160,10 +168,10 @@
   binary_mac_build:
     <<: *binary_mac_params
     macos:
-      xcode: "12.0"
+      xcode: "9.4.1"
     steps:
     # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
-    - checkout
+    - attach_scripts
     - run:
         <<: *binary_checkout
     - run:
@@ -174,7 +182,7 @@
 
     - run:
         name: Build
-        no_output_timeout: "90m"
+        no_output_timeout: "1h"
         command: |
           # Do not set -u here; there is some problem with CircleCI
           # variable expansion with PROMPT_COMMAND
@@ -198,16 +206,13 @@
         root: /Users/distiller/project
         paths: final_pkgs
 
-    - store_artifacts:
-        path: /Users/distiller/project/final_pkgs
-
-  binary_macos_arm64_build:
+  binary_mac_upload: &binary_mac_upload
     <<: *binary_mac_params
     macos:
-      xcode: "12.3.0"
+      xcode: "9.4.1"
     steps:
     # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
-    - checkout
+    - attach_scripts
     - run:
         <<: *binary_checkout
     - run:
@@ -215,34 +220,24 @@
     - brew_update
     - run:
         <<: *binary_install_miniconda
-
+    - attach_workspace: # TODO - we can `cp` from ~/workspace
+        at: /Users/distiller/project
     - run:
-        name: Build
-        no_output_timeout: "90m"
+        name: Upload
+        no_output_timeout: "10m"
         command: |
-          # Do not set -u here; there is some problem with CircleCI
-          # variable expansion with PROMPT_COMMAND
-          set -ex -o pipefail
-          export CROSS_COMPILE_ARM64=1
-          script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_build.sh"
+          script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_upload.sh"
           cat "$script"
           source "$script"
 
-    - persist_to_workspace:
-        root: /Users/distiller/project
-        paths: final_pkgs
-
-    - store_artifacts:
-        path: /Users/distiller/project/final_pkgs
-
-
   binary_ios_build:
     <<: *pytorch_ios_params
     macos:
-      xcode: "12.0"
+      xcode: "11.2.1"
     steps:
     - attach_workspace:
         at: ~/workspace
+    - attach_scripts
     - checkout
     - run_brew_for_ios_build
     - run:
@@ -266,10 +261,11 @@
   binary_ios_upload:
     <<: *pytorch_ios_params
     macos:
-      xcode: "12.0"
+      xcode: "11.2.1"
     steps:
     - attach_workspace:
         at: ~/workspace
+    - attach_scripts
     - checkout
     - run_brew_for_ios_build
     - run:
@@ -279,120 +275,3 @@
           script="/Users/distiller/project/.circleci/scripts/binary_ios_upload.sh"
           cat "$script"
           source "$script"
-
-  binary_windows_build:
-    <<: *binary_windows_params
-    parameters:
-      build_environment:
-        type: string
-        default: ""
-      executor:
-        type: string
-        default: "windows-xlarge-cpu-with-nvidia-cuda"
-    executor: <<parameters.executor>>
-    steps:
-    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
-    - checkout
-    - run:
-        name: _HACK_ Install CUDA compatible cmath
-        no_output_timeout: 1m
-        command: |
-            powershell .circleci/scripts/vs_install_cmath.ps1
-    - run:
-        <<: *binary_checkout
-    - run:
-        <<: *binary_populate_env
-    - run:
-        name: Build
-        no_output_timeout: "1h"
-        command: |
-          set -eux -o pipefail
-          script="/c/w/p/.circleci/scripts/binary_windows_build.sh"
-          cat "$script"
-          source "$script"
-    - persist_to_workspace:
-        root: "C:/w"
-        paths: final_pkgs
-
-  binary_windows_test:
-    <<: *binary_windows_params
-    parameters:
-      build_environment:
-        type: string
-        default: ""
-      executor:
-        type: string
-        default: "windows-medium-cpu-with-nvidia-cuda"
-    executor: <<parameters.executor>>
-    steps:
-    - checkout
-    - attach_workspace:
-        at: c:/users/circleci/project
-    - run:
-        <<: *binary_checkout
-    - run:
-        <<: *binary_populate_env
-    - run:
-        name: Test
-        no_output_timeout: "1h"
-        command: |
-          set -eux -o pipefail
-          script="/c/w/p/.circleci/scripts/binary_windows_test.sh"
-          cat "$script"
-          source "$script"
-
-  smoke_windows_test:
-    <<: *binary_windows_params
-    parameters:
-      build_environment:
-        type: string
-        default: ""
-      executor:
-        type: string
-        default: "windows-medium-cpu-with-nvidia-cuda"
-    executor: <<parameters.executor>>
-    steps:
-    - checkout
-    - run:
-        <<: *binary_checkout
-    - run:
-        <<: *binary_populate_env
-    - run:
-        name: Test
-        no_output_timeout: "1h"
-        command: |
-          set -eux -o pipefail
-          export TEST_NIGHTLY_PACKAGE=1
-          script="/c/w/p/.circleci/scripts/binary_windows_test.sh"
-          cat "$script"
-          source "$script"
-
-  anaconda_prune:
-    parameters:
-      packages:
-        type: string
-        description: "What packages are we pruning? (quoted, space-separated string. eg. 'pytorch', 'torchvision torchaudio', etc.)"
-        default: "pytorch"
-      channel:
-        type: string
-        description: "What channel are we pruning? (eq. pytorch-nightly)"
-        default: "pytorch-nightly"
-    docker:
-      - image: continuumio/miniconda3
-    environment:
-      - PACKAGES: "<< parameters.packages >>"
-      - CHANNEL: "<< parameters.channel >>"
-    steps:
-      - checkout
-      - run:
-          name: Install dependencies
-          no_output_timeout: "1h"
-          command: |
-            conda install -yq anaconda-client
-      - run:
-          name: Prune packages
-          no_output_timeout: "1h"
-          command: |
-              ANACONDA_API_TOKEN="${CONDA_PYTORCHBOT_TOKEN}" \
-              scripts/release/anaconda-prune/run.sh
-

.circleci/verbatim-sources/binary_update_htmls.yml

@@ -8,10 +8,10 @@
   # then install the one with the most recent version.
   update_s3_htmls: &update_s3_htmls
     machine:
-      image: ubuntu-1604:202007-01
-    resource_class: medium
+      image: ubuntu-1604:201903-01
     steps:
-    - checkout
+    - attach_workspace:
+        at: ~/workspace
     - setup_linux_system_environment
     - run:
         <<: *binary_checkout
@@ -28,15 +28,6 @@
     # make sure it has the same upload folder as the job it's attached to. This
     # function is idempotent, so it won't hurt anything; it's just a little
     # unnescessary"
-    - run:
-        name: define PIP_UPLOAD_FOLDER
-        command: |
-          our_upload_folder=nightly/
-          # On tags upload to test instead
-          if [[ -n "${CIRCLE_TAG}" ]]; then
-            our_upload_folder=test/
-          fi
-          echo "export PIP_UPLOAD_FOLDER=${our_upload_folder}" >> ${BASH_ENV}
     - run:
         name: Update s3 htmls
         no_output_timeout: "1h"
@@ -51,3 +42,55 @@
           }
           retry pip install awscli==1.6
           "/home/circleci/project/builder/cron/update_s3_htmls.sh"
+
+  # Update s3 htmls for the nightlies
+  update_s3_htmls_for_nightlies:
+    environment:
+      PIP_UPLOAD_FOLDER: "nightly/"
+    <<: *update_s3_htmls
+
+  # Update s3 htmls for the nightlies for devtoolset7
+  update_s3_htmls_for_nightlies_devtoolset7:
+    environment:
+      PIP_UPLOAD_FOLDER: "nightly/devtoolset7/"
+    <<: *update_s3_htmls
+
+
+  # upload_binary_logs job
+  # The builder hud at pytorch.org/builder shows the sizes of all the binaries
+  # over time. It gets this info from html files stored in S3, which this job
+  # populates every day.
+  upload_binary_sizes: &upload_binary_sizes
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_workspace:
+        at: ~/workspace
+    - setup_linux_system_environment
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_install_miniconda
+    - run:
+        name: Upload binary sizes
+        no_output_timeout: "1h"
+        command: |
+          set +x
+          echo "declare -x \"AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}\"" > /home/circleci/project/env
+          echo "declare -x \"AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}\"" >> /home/circleci/project/env
+          export DATE="$(date -u +%Y_%m_%d)"
+          retry () {
+              $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
+          }
+          source /home/circleci/project/env
+          set -eux -o pipefail
+
+          # This is hardcoded to match binary_install_miniconda.sh
+          export PATH="/home/circleci/project/miniconda/bin:$PATH"
+          # Not any awscli will work. Most won't. This one will work
+          retry conda create -qyn aws36 python=3.6
+          source activate aws36
+          pip install awscli==1.16.46
+
+          "/home/circleci/project/builder/cron/upload_binary_sizes.sh"
+

.circleci/verbatim-sources/build-parameters/promote-build-params.yml

@@ -1,14 +0,0 @@
-
-promote_common: &promote_common
-  docker:
-    - image: pytorch/release
-  parameters:
-    package_name:
-      description: "package name to promote"
-      type: string
-      default: ""
-  environment:
-    PACKAGE_NAME: << parameters.package_name >>
-    ANACONDA_API_TOKEN: ${CONDA_PYTORCHBOT_TOKEN}
-    AWS_ACCESS_KEY_ID: ${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}
-    AWS_SECRET_ACCESS_KEY: ${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}

.circleci/verbatim-sources/caffe2-build-params.yml

@@ -0,0 +1,28 @@
+caffe2_params: &caffe2_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+    build_ios:
+      type: string
+      default: ""
+    docker_image:
+      type: string
+      default: ""
+    use_cuda_docker_runtime:
+      type: string
+      default: ""
+    build_only:
+      type: string
+      default: ""
+    resource_class:
+      type: string
+      default: "large"
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+    BUILD_IOS: << parameters.build_ios >>
+    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
+    DOCKER_IMAGE: << parameters.docker_image >>
+    BUILD_ONLY: << parameters.build_only >>
+  resource_class: << parameters.resource_class >>
+

.circleci/verbatim-sources/caffe2-job-specs.yml

@@ -0,0 +1,200 @@
+  caffe2_linux_build:
+    <<: *caffe2_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_scripts
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: Build
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          cat >/home/circleci/project/ci_build_script.sh \<<EOL
+          # =================== The following code will be executed inside Docker container ===================
+          set -ex
+          export BUILD_ENVIRONMENT="$BUILD_ENVIRONMENT"
+
+          # Reinitialize submodules
+          git submodule sync && git submodule update -q --init --recursive
+
+          # conda must be added to the path for Anaconda builds (this location must be
+          # the same as that in install_anaconda.sh used to build the docker image)
+          if [[ "${BUILD_ENVIRONMENT}" == conda* ]]; then
+            export PATH=/opt/conda/bin:$PATH
+            sudo chown -R jenkins:jenkins '/opt/conda'
+          fi
+
+          # Build
+          ./.jenkins/caffe2/build.sh
+
+          # Show sccache stats if it is running
+          if pgrep sccache > /dev/null; then
+            sccache --show-stats
+          fi
+          # =================== The above code will be executed inside Docker container ===================
+          EOL
+          chmod +x /home/circleci/project/ci_build_script.sh
+
+          echo "DOCKER_IMAGE: "${DOCKER_IMAGE}
+          time docker pull ${DOCKER_IMAGE} >/dev/null
+          export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${DOCKER_IMAGE})
+          docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_build_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Push intermediate Docker image for next phase to use
+          if [ -z "${BUILD_ONLY}" ]; then
+            if [[ "$BUILD_ENVIRONMENT" == *cmake* ]]; then
+              export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-cmake-${CIRCLE_SHA1}
+            else
+              export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+            fi
+            docker commit "$id" ${COMMIT_DOCKER_IMAGE}
+            time docker push ${COMMIT_DOCKER_IMAGE}
+          fi
+
+  caffe2_linux_test:
+    <<: *caffe2_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_scripts
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        name: Test
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          # TODO: merge this into Caffe2 test.sh
+          cat >/home/circleci/project/ci_test_script.sh \<<EOL
+          # =================== The following code will be executed inside Docker container ===================
+          set -ex
+
+          export BUILD_ENVIRONMENT="$BUILD_ENVIRONMENT"
+
+          # libdc1394 (dependency of OpenCV) expects /dev/raw1394 to exist...
+          sudo ln /dev/null /dev/raw1394
+
+          # conda must be added to the path for Anaconda builds (this location must be
+          # the same as that in install_anaconda.sh used to build the docker image)
+          if [[ "${BUILD_ENVIRONMENT}" == conda* ]]; then
+            export PATH=/opt/conda/bin:$PATH
+          fi
+
+          # Upgrade SSL module to avoid old SSL warnings
+          pip -q install --user --upgrade pyOpenSSL ndg-httpsclient pyasn1
+
+          pip -q install --user -b /tmp/pip_install_onnx "file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx"
+
+          # Build
+          ./.jenkins/caffe2/test.sh
+
+          # Remove benign core dumps.
+          # These are tests for signal handling (including SIGABRT).
+          rm -f ./crash/core.fatal_signal_as.*
+          rm -f ./crash/core.logging_test.*
+          # =================== The above code will be executed inside Docker container ===================
+          EOL
+          chmod +x /home/circleci/project/ci_test_script.sh
+
+          if [[ "$BUILD_ENVIRONMENT" == *cmake* ]]; then
+            export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-cmake-${CIRCLE_SHA1}
+          else
+            export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          fi
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
+            export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+          else
+            export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+          fi
+          docker cp /home/circleci/project/. "$id:/var/lib/jenkins/workspace"
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_test_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+  caffe2_macos_build:
+    <<: *caffe2_params
+    macos:
+      xcode: "9.4.1"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - attach_scripts
+      - checkout
+      - run_brew_for_macos_build
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+
+            export IN_CIRCLECI=1
+
+            brew install cmake
+
+            # Reinitialize submodules
+            git submodule sync && git submodule update -q --init --recursive
+
+            # Reinitialize path (see man page for path_helper(8))
+            eval `/usr/libexec/path_helper -s`
+
+            export PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:$PATH
+
+            # Install Anaconda if we need to
+            if [ -n "${CAFFE2_USE_ANACONDA}" ]; then
+              rm -rf ${TMPDIR}/anaconda
+              curl --retry 3 -o ${TMPDIR}/conda.sh https://repo.continuum.io/miniconda/Miniconda${ANACONDA_VERSION}-latest-MacOSX-x86_64.sh
+              chmod +x ${TMPDIR}/conda.sh
+              /bin/bash ${TMPDIR}/conda.sh -b -p ${TMPDIR}/anaconda
+              rm -f ${TMPDIR}/conda.sh
+              export PATH="${TMPDIR}/anaconda/bin:${PATH}"
+              source ${TMPDIR}/anaconda/bin/activate
+            fi
+
+            pip -q install numpy
+
+            # Install sccache
+            sudo curl --retry 3 https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
+            sudo chmod +x /usr/local/bin/sccache
+            export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
+
+            # This IAM user allows write access to S3 bucket for sccache
+            set +x
+            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            set -x
+
+            export SCCACHE_BIN=${PWD}/sccache_bin
+            mkdir -p ${SCCACHE_BIN}
+            if which sccache > /dev/null; then
+              printf "#!/bin/sh\nexec sccache $(which clang++) \$*" > "${SCCACHE_BIN}/clang++"
+              chmod a+x "${SCCACHE_BIN}/clang++"
+
+              printf "#!/bin/sh\nexec sccache $(which clang) \$*" > "${SCCACHE_BIN}/clang"
+              chmod a+x "${SCCACHE_BIN}/clang"
+
+              export PATH="${SCCACHE_BIN}:$PATH"
+            fi
+
+            # Build
+            if [ "${BUILD_IOS:-0}" -eq 1 ]; then
+              unbuffer scripts/build_ios.sh 2>&1 | ts
+            elif [ -n "${CAFFE2_USE_ANACONDA}" ]; then
+              # All conda build logic should be in scripts/build_anaconda.sh
+              unbuffer scripts/build_anaconda.sh 2>&1 | ts
+            else
+              unbuffer scripts/build_local.sh 2>&1 | ts
+            fi
+
+            # Show sccache stats if it is running
+            if which sccache > /dev/null; then
+              sccache --show-stats
+            fi

.circleci/verbatim-sources/commands.yml

@@ -1,26 +1,14 @@
 commands:
-
-  calculate_docker_image_tag:
-    description: "Calculates the docker image tag"
+  # NB: This command must be run as the first command in a job. It
+  # attaches the workspace at ~/workspace; this workspace is generated
+  # by the setup job. Note that ~/workspace is not the default working
+  # directory (that's ~/project).
+  attach_scripts:
+    description: "Attach the scripts that power everything else"
     steps:
-      - run:
-          name: "Calculate docker image hash"
-          command: |
-            DOCKER_TAG=$(git rev-parse HEAD:.circleci/docker)
-            echo "DOCKER_TAG=${DOCKER_TAG}" >> "${BASH_ENV}"
-
-  designate_upload_channel:
-    description: "inserts the correct upload channel into ${BASH_ENV}"
-    steps:
-      - run:
-          name: adding UPLOAD_CHANNEL to BASH_ENV
-          command: |
-            our_upload_channel=nightly
-            # On tags upload to test instead
-            if [[ -n "${CIRCLE_TAG}" ]]; then
-              our_upload_channel=test
-            fi
-            echo "export UPLOAD_CHANNEL=${our_upload_channel}" >> ${BASH_ENV}
+      - attach_workspace:
+          name: Attaching workspace
+          at: ~/workspace
 
   # This system setup script is meant to run before the CI-related scripts, e.g.,
   # installing Git client, checking out code, setting up CI env, and
@@ -30,14 +18,14 @@ commands:
       - run:
           name: Set Up System Environment
           no_output_timeout: "1h"
-          command: .circleci/scripts/setup_linux_system_environment.sh
+          command: ~/workspace/.circleci/scripts/setup_linux_system_environment.sh
 
   setup_ci_environment:
     steps:
       - run:
           name: Set Up CI Environment After attach_workspace
           no_output_timeout: "1h"
-          command: .circleci/scripts/setup_ci_environment.sh
+          command: ~/workspace/.circleci/scripts/setup_ci_environment.sh
 
   brew_update:
     description: "Update Homebrew and install base formulae"
@@ -96,79 +84,3 @@ commands:
       - brew_update
       - brew_install:
           formulae: libtool
-
-  optional_merge_target_branch:
-    steps:
-      - run:
-          name: (Optional) Merge target branch
-          no_output_timeout: "10m"
-          command: |
-            if [[ -n "$CIRCLE_PULL_REQUEST" && "$CIRCLE_BRANCH" != "nightly" ]]; then
-              PR_NUM=$(basename $CIRCLE_PULL_REQUEST)
-              CIRCLE_PR_BASE_BRANCH=$(curl -s https://api.github.com/repos/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/pulls/$PR_NUM | jq -r '.base.ref')
-              if [[ "${BUILD_ENVIRONMENT}" == *"xla"* || "${BUILD_ENVIRONMENT}" == *"gcc5"* ]] ; then
-                set -x
-                git config --global user.email "circleci.ossci@gmail.com"
-                git config --global user.name "CircleCI"
-                git config remote.origin.url https://github.com/pytorch/pytorch.git
-                git config --add remote.origin.fetch +refs/heads/release/1.8:refs/remotes/origin/release/1.8
-                git fetch --tags --progress https://github.com/pytorch/pytorch.git +refs/heads/release/1.8:refs/remotes/origin/release/1.8 --depth=100 --quiet
-                # PRs generated from ghstack has format CIRCLE_PR_BASE_BRANCH=gh/xxx/1234/base
-                if [[ "${CIRCLE_PR_BASE_BRANCH}" == "gh/"* ]]; then
-                  CIRCLE_PR_BASE_BRANCH=release/1.8
-                fi
-                export GIT_MERGE_TARGET=`git log -n 1 --pretty=format:"%H" origin/$CIRCLE_PR_BASE_BRANCH`
-                echo "GIT_MERGE_TARGET: " ${GIT_MERGE_TARGET}
-                export GIT_COMMIT=${CIRCLE_SHA1}
-                echo "GIT_COMMIT: " ${GIT_COMMIT}
-                git checkout -f ${GIT_COMMIT}
-                git reset --hard ${GIT_COMMIT}
-                git merge --allow-unrelated-histories --no-edit --no-ff ${GIT_MERGE_TARGET}
-                echo "Merged $CIRCLE_PR_BASE_BRANCH branch before building in environment $BUILD_ENVIRONMENT"
-                set +x
-              else
-                echo "No need to merge with $CIRCLE_PR_BASE_BRANCH, skipping..."
-              fi
-            else
-              echo "This is not a pull request, skipping..."
-            fi
-
-  upload_binary_size_for_android_build:
-    description: "Upload binary size data for Android build"
-    parameters:
-      build_type:
-        type: string
-        default: ""
-      artifacts:
-        type: string
-        default: ""
-    steps:
-      - run:
-          name: "Binary Size - Install Dependencies"
-          no_output_timeout: "5m"
-          command: |
-            retry () {
-              $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-            }
-            retry pip3 install requests
-      - run:
-          name: "Binary Size - Untar Artifacts"
-          no_output_timeout: "5m"
-          command: |
-            # The artifact file is created inside docker container, which contains the result binaries.
-            # Now unpackage it into the project folder. The subsequent script will scan project folder
-            # to locate result binaries and report their sizes.
-            # If artifact file is not provided it assumes that the project folder has been mounted in
-            # the docker during build and already contains the result binaries, so this step can be skipped.
-            export ARTIFACTS="<< parameters.artifacts >>"
-            if [ -n "${ARTIFACTS}" ]; then
-              tar xf "${ARTIFACTS}" -C ~/project
-            fi
-      - run:
-          name: "Binary Size - Upload << parameters.build_type >>"
-          no_output_timeout: "5m"
-          command: |
-            cd ~/project
-            export ANDROID_BUILD_TYPE="<< parameters.build_type >>"
-            export COMMIT_TIME=$(git log --max-count=1 --format=%ct || echo 0)
-            python3 .circleci/scripts/upload_binary_size_to_scuba.py android

.circleci/verbatim-sources/docker_jobs.yml

@@ -4,44 +4,12 @@
           type: string
           default: ""
       machine:
-        image: ubuntu-1604:202007-01
+        image: ubuntu-1604:201903-01
       resource_class: large
       environment:
         IMAGE_NAME: << parameters.image_name >>
-        # Enable 'docker manifest'
-        DOCKER_CLI_EXPERIMENTAL: "enabled"
-        DOCKER_BUILDKIT: 1
       steps:
         - checkout
-        - calculate_docker_image_tag
-        - run:
-            name: Check if image should be built
-            command: |
-              set +x
-              export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_DOCKER_BUILDER_V1}
-              export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1}
-              eval $(aws ecr get-login --no-include-email --region us-east-1)
-              set -x
-              # Check if image already exists, if it does then skip building it
-              if docker manifest inspect "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/${IMAGE_NAME}:${DOCKER_TAG}"; then
-                circleci-agent step halt
-                # circleci-agent step halt doesn't actually halt the step so we need to
-                # explicitly exit the step here ourselves before it causes too much trouble
-                exit 0
-              fi
-              # Covers the case where a previous tag doesn't exist for the tree
-              # this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly
-              if ! git rev-parse "$(git merge-base HEAD << pipeline.git.base_revision >>):.circleci/docker"; then
-                echo "Directory '.circleci/docker' not found in tree << pipeline.git.base_revision >>, you should probably rebase onto a more recent commit"
-                exit 1
-              fi
-              PREVIOUS_DOCKER_TAG=$(git rev-parse "$(git merge-base HEAD << pipeline.git.base_revision >>):.circleci/docker")
-              # If no image exists but the hash is the same as the previous hash then we should error out here
-              if [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then
-                echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
-                echo "       contact the PyTorch team to restore the original images"
-                exit 1
-              fi
         - run:
             name: build_docker_image_<< parameters.image_name >>
             no_output_timeout: "1h"
@@ -53,7 +21,7 @@
               cd .circleci/docker && ./build_docker.sh
   docker_for_ecr_gc_build_job:
       machine:
-        image: ubuntu-1604:202007-01
+        image: ubuntu-1604:201903-01
       steps:
         - checkout
         - run:
@@ -77,7 +45,6 @@
           type: string
       environment:
         PROJECT: << parameters.project >>
-        # TODO: Remove legacy image tags once we feel comfortable with new docker image tags
         IMAGE_TAG: << parameters.tags_to_keep >>
       docker:
         - image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/gc/ecr
@@ -86,17 +53,6 @@
             aws_secret_access_key: ${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1}
 
       steps:
-        - checkout
-        - run:
-            # NOTE: see 'docker_build_job' for how these tags actually get built
-            name: dynamically generate tags to keep
-            no_output_timeout: "1h"
-            command: |
-              GENERATED_IMAGE_TAG=$(\
-                git log --oneline --pretty='%H' .circleci/docker \
-                  | xargs -I '{}' git rev-parse '{}:.circleci/docker' \
-                  | paste -sd "," -)
-              echo "export GENERATED_IMAGE_TAG='${GENERATED_IMAGE_TAG}'" >> ${BASH_ENV}
         - run:
             name: garbage collecting for ecr images
             no_output_timeout: "1h"
@@ -105,4 +61,24 @@
               export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_DOCKER_BUILDER_V1}
               export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1}
               set -x
-              /usr/bin/gc.py --filter-prefix ${PROJECT}  --ignore-tags "${IMAGE_TAG},${GENERATED_IMAGE_TAG}"
+              /usr/bin/gc.py --filter-prefix ${PROJECT}  --ignore-tags ${IMAGE_TAG}
+
+  docker_hub_index_job:
+      docker:
+        - image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/gc/ecr
+          aws_auth:
+            aws_access_key_id: ${CIRCLECI_AWS_ACCESS_KEY_FOR_DOCKER_BUILDER_V1}
+            aws_secret_access_key: ${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1}
+
+      steps:
+        - run:
+            name: garbage collecting for ecr images
+            no_output_timeout: "1h"
+            command: |
+              set +x
+              export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_DOCKER_BUILDER_V1}
+              export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1}
+              export DOCKER_HUB_USERNAME=${CIRCLECI_DOCKER_HUB_USERNAME}
+              export DOCKER_HUB_PASSWORD=${CIRCLECI_DOCKER_HUB_PASSWORD}
+              set -x
+              /usr/bin/docker_hub.py

.circleci/verbatim-sources/header-section.yml

@@ -7,14 +7,6 @@
 
 version: 2.1
 
-parameters:
-  run_binary_tests:
-    type: boolean
-    default: false
-  run_build:
-    type: boolean
-    default: true
-
 docker_config_defaults: &docker_config_defaults
   user: jenkins
   aws_auth:
@@ -29,14 +21,9 @@ executors:
       image: windows-server-2019-nvidia:stable
       shell: bash.exe
 
-  windows-xlarge-cpu-with-nvidia-cuda:
+  windows-cpu-with-nvidia-cuda:
     machine:
+      # we will change to CPU host when it's ready
       resource_class: windows.xlarge
       image: windows-server-2019-vs2019:stable
       shell: bash.exe
-
-  windows-medium-cpu-with-nvidia-cuda:
-    machine:
-      resource_class: windows.medium
-      image: windows-server-2019-vs2019:stable
-      shell: bash.exe

.circleci/verbatim-sources/job-specs-custom.yml

@@ -0,0 +1,442 @@
+  pytorch_python_doc_push:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-python-doc-push
+      # TODO: stop hardcoding this
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.6-gcc5.4:f990c76a-a798-42bb-852f-5be5006f8026"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_scripts
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        name: Doc Build and Push
+        no_output_timeout: "1h"
+        command: |
+          set -ex
+          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+
+          # master branch docs push
+          if [[ "${CIRCLE_BRANCH}" == "master" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/master master site") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # stable release docs push. Due to some circleci limitations, we keep
+          # an eternal PR open for merging v1.2.0 -> master for this job.
+          # XXX: The following code is only run on the v1.2.0 branch, which might
+          # not be exactly the same as what you see here.
+          elif [[ "${CIRCLE_BRANCH}" == "v1.2.0" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/stable 1.2.0 site dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # For open PRs: Do a dry_run of the docs build, don't push build
+          else
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/master master site dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          fi
+
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Save the docs build so we can debug any problems
+          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
+          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
+          time docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
+
+  pytorch_cpp_doc_push:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-cpp-doc-push
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.6-gcc5.4:f990c76a-a798-42bb-852f-5be5006f8026"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_scripts
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        name: Doc Build and Push
+        no_output_timeout: "1h"
+        command: |
+          set -ex
+          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+
+          # master branch docs push
+          if [[ "${CIRCLE_BRANCH}" == "master" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/master master") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # stable release docs push. Due to some circleci limitations, we keep
+          # an eternal PR open (#16502) for merging v1.0.1 -> master for this job.
+          # XXX: The following code is only run on the v1.0.1 branch, which might
+          # not be exactly the same as what you see here.
+          elif [[ "${CIRCLE_BRANCH}" == "v1.0.1" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/stable 1.0.1") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # For open PRs: Do a dry_run of the docs build, don't push build
+          else
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/master master dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          fi
+
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Save the docs build so we can debug any problems
+          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
+          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
+          time docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
+
+  pytorch_macos_10_13_py3_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
+    macos:
+      xcode: "9.4.1"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - attach_scripts
+      - checkout
+      - run_brew_for_macos_build
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+            export IN_CIRCLECI=1
+
+            # Install sccache
+            sudo curl --retry 3 https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
+            sudo chmod +x /usr/local/bin/sccache
+            export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
+
+            # This IAM user allows write access to S3 bucket for sccache
+            set +x
+            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            set -x
+
+            chmod a+x .jenkins/pytorch/macos-build.sh
+            unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
+
+            # copy with -a to preserve relative structure (e.g., symlinks), and be recursive
+            cp -a ~/project ~/workspace
+
+      - persist_to_workspace:
+          root: ~/workspace
+          paths:
+            - miniconda3
+            - project
+
+  pytorch_macos_10_13_py3_test:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
+    macos:
+      xcode: "9.4.1"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      # This workspace also carries binaries from the build job
+      - attach_scripts
+      - run_brew_for_macos_build
+      - run:
+          name: Test
+          no_output_timeout: "1h"
+          command: |
+            set -e
+            export IN_CIRCLECI=1
+
+            # copy with -a to preserve relative structure (e.g., symlinks), and be recursive
+            cp -a ~/workspace/project/. ~/project
+
+            chmod a+x .jenkins/pytorch/macos-test.sh
+            unbuffer .jenkins/pytorch/macos-test.sh 2>&1 | ts
+      - store_test_results:
+          path: test/test-reports
+
+  pytorch_android_gradle_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:f990c76a-a798-42bb-852f-5be5006f8026"
+      PYTHON_VERSION: "3.6"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_scripts
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: pytorch android gradle build
+        no_output_timeout: "1h"
+        command: |
+          set -eux
+          docker_image_commit=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+
+          docker_image_libtorch_android_x86_32=${docker_image_commit}-android-x86_32
+          docker_image_libtorch_android_x86_64=${docker_image_commit}-android-x86_64
+          docker_image_libtorch_android_arm_v7a=${docker_image_commit}-android-arm-v7a
+          docker_image_libtorch_android_arm_v8a=${docker_image_commit}-android-arm-v8a
+
+          echo "docker_image_commit: "${docker_image_commit}
+          echo "docker_image_libtorch_android_x86_32: "${docker_image_libtorch_android_x86_32}
+          echo "docker_image_libtorch_android_x86_64: "${docker_image_libtorch_android_x86_64}
+          echo "docker_image_libtorch_android_arm_v7a: "${docker_image_libtorch_android_arm_v7a}
+          echo "docker_image_libtorch_android_arm_v8a: "${docker_image_libtorch_android_arm_v8a}
+
+          # x86_32
+          time docker pull ${docker_image_libtorch_android_x86_32} >/dev/null
+          export id_x86_32=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # arm-v7a
+          time docker pull ${docker_image_libtorch_android_arm_v7a} >/dev/null
+          export id_arm_v7a=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_arm_v7a})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_arm_v7a" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir ~/workspace/build_android_install_arm_v7a
+          docker cp $id_arm_v7a:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_arm_v7a
+
+          # x86_64
+          time docker pull ${docker_image_libtorch_android_x86_64} >/dev/null
+          export id_x86_64=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_64})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_x86_64" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir ~/workspace/build_android_install_x86_64
+          docker cp $id_x86_64:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_x86_64
+
+          # arm-v8a
+          time docker pull ${docker_image_libtorch_android_arm_v8a} >/dev/null
+          export id_arm_v8a=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_arm_v8a})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_arm_v8a" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir ~/workspace/build_android_install_arm_v8a
+          docker cp $id_arm_v8a:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_arm_v8a
+
+          docker cp ~/workspace/build_android_install_arm_v7a $id_x86_32:/var/lib/jenkins/workspace/build_android_install_arm_v7a
+          docker cp ~/workspace/build_android_install_x86_64 $id_x86_32:/var/lib/jenkins/workspace/build_android_install_x86_64
+          docker cp ~/workspace/build_android_install_arm_v8a $id_x86_32:/var/lib/jenkins/workspace/build_android_install_arm_v8a
+
+          # run gradle buildRelease
+          export COMMAND='((echo "source ./workspace/env" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GRADLE_OFFLINE=1" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir -p ~/workspace/build_android_artifacts
+          docker cp $id_x86_32:/var/lib/jenkins/workspace/android/artifacts.tgz ~/workspace/build_android_artifacts/
+
+          output_image=$docker_image_libtorch_android_x86_32-gradle
+          docker commit "$id_x86_32" ${output_image}
+          time docker push ${output_image}
+    - store_artifacts:
+        path: ~/workspace/build_android_artifacts/artifacts.tgz
+        destination: artifacts.tgz
+
+  pytorch_android_publish_snapshot:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-publish-snapshot
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:f990c76a-a798-42bb-852f-5be5006f8026"
+      PYTHON_VERSION: "3.6"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_scripts
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: pytorch android gradle build
+        no_output_timeout: "1h"
+        command: |
+          set -eux
+          docker_image_commit=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+
+          docker_image_libtorch_android_x86_32_gradle=${docker_image_commit}-android-x86_32-gradle
+
+          echo "docker_image_commit: "${docker_image_commit}
+          echo "docker_image_libtorch_android_x86_32_gradle: "${docker_image_libtorch_android_x86_32_gradle}
+
+          # x86_32
+          time docker pull ${docker_image_libtorch_android_x86_32_gradle} >/dev/null
+          export id_x86_32=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32_gradle})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export SONATYPE_NEXUS_USERNAME=${SONATYPE_NEXUS_USERNAME}" && echo "export SONATYPE_NEXUS_PASSWORD=${SONATYPE_NEXUS_PASSWORD}" && echo "export ANDROID_SIGN_KEY=${ANDROID_SIGN_KEY}" && echo "export ANDROID_SIGN_PASS=${ANDROID_SIGN_PASS}" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/publish_android_snapshot.sh") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          output_image=${docker_image_libtorch_android_x86_32_gradle}-publish-snapshot
+          docker commit "$id_x86_32" ${output_image}
+          time docker push ${output_image}
+
+  pytorch_android_gradle_build-x86_32:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-only-x86_32
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:f990c76a-a798-42bb-852f-5be5006f8026"
+      PYTHON_VERSION: "3.6"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_scripts
+    - run:
+        name: filter out not PR runs
+        no_output_timeout: "5m"
+        command: |
+          echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
+          if [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
+            circleci step halt
+          fi
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: pytorch android gradle build only x86_32 (for PR)
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          docker_image_libtorch_android_x86_32=${DOCKER_IMAGE}-${CIRCLE_SHA1}-android-x86_32
+          echo "docker_image_libtorch_android_x86_32: "${docker_image_libtorch_android_x86_32}
+
+          # x86
+          time docker pull ${docker_image_libtorch_android_x86_32} >/dev/null
+          export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GRADLE_OFFLINE=1" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir -p ~/workspace/build_android_x86_32_artifacts
+          docker cp $id:/var/lib/jenkins/workspace/android/artifacts.tgz ~/workspace/build_android_x86_32_artifacts/
+
+          output_image=${docker_image_libtorch_android_x86_32}-gradle
+          docker commit "$id" ${output_image}
+          time docker push ${output_image}
+    - store_artifacts:
+        path: ~/workspace/build_android_x86_32_artifacts/artifacts.tgz
+        destination: artifacts.tgz
+
+  pytorch_ios_build:
+    <<: *pytorch_ios_params
+    macos:
+      xcode: "11.2.1"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - attach_scripts
+      - checkout
+      - run_brew_for_ios_build
+      - run:
+          name: Run Fastlane
+          no_output_timeout: "1h"
+          command: |
+            set -e
+            PROJ_ROOT=/Users/distiller/project
+            cd ${PROJ_ROOT}/ios/TestApp
+            # install fastlane
+            sudo gem install bundler && bundle install
+            # install certificates
+            echo ${IOS_CERT_KEY} >> cert.txt
+            base64 --decode cert.txt -o Certificates.p12
+            rm cert.txt
+            bundle exec fastlane install_cert
+            # install the provisioning profile
+            PROFILE=TestApp_CI.mobileprovision
+            PROVISIONING_PROFILES=~/Library/MobileDevice/Provisioning\ Profiles
+            mkdir -pv "${PROVISIONING_PROFILES}"
+            cd "${PROVISIONING_PROFILES}"
+            echo ${IOS_SIGN_KEY} >> cert.txt
+            base64 --decode cert.txt -o ${PROFILE}
+            rm cert.txt
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+            export IN_CIRCLECI=1
+            WORKSPACE=/Users/distiller/workspace
+            PROJ_ROOT=/Users/distiller/project
+            export TCLLIBPATH="/usr/local/lib"
+
+            # Install conda
+            curl --retry 3 -o ~/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+            chmod +x ~/conda.sh
+            /bin/bash ~/conda.sh -b -p ~/anaconda
+            export PATH="~/anaconda/bin:${PATH}"
+            source ~/anaconda/bin/activate
+
+            # Install dependencies
+            retry () {
+                $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
+            }
+
+            retry conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing requests --yes
+
+            # sync submodules
+            cd ${PROJ_ROOT}
+            git submodule sync
+            git submodule update --init --recursive
+
+            # export
+            export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
+
+            # run build script
+            chmod a+x ${PROJ_ROOT}/scripts/build_ios.sh
+            echo "IOS_ARCH: ${IOS_ARCH}"
+            echo "IOS_PLATFORM: ${IOS_PLATFORM}"
+
+            #check the custom build flag
+            echo "SELECTED_OP_LIST: ${SELECTED_OP_LIST}"
+            if [ -n "${SELECTED_OP_LIST}" ]; then
+                export SELECTED_OP_LIST="${PROJ_ROOT}/ios/TestApp/custom_build/${SELECTED_OP_LIST}"
+            fi
+            export IOS_ARCH=${IOS_ARCH}
+            export IOS_PLATFORM=${IOS_PLATFORM}
+            unbuffer ${PROJ_ROOT}/scripts/build_ios.sh 2>&1 | ts
+      - run:
+          name: Run Build Test
+          no_output_timeout: "30m"
+          command: |
+            set -e
+            PROJ_ROOT=/Users/distiller/project
+            PROFILE=TestApp_CI
+            # run the ruby build script
+            if ! [ -x "$(command -v xcodebuild)" ]; then
+              echo 'Error: xcodebuild is not installed.'
+              exit 1
+            fi
+            echo ${IOS_DEV_TEAM_ID}
+            if [ ${IOS_PLATFORM} != "SIMULATOR" ]; then
+              ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM} -c ${PROFILE} -t ${IOS_DEV_TEAM_ID}
+            else
+              ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM}
+            fi
+            if ! [ "$?" -eq "0" ]; then
+              echo 'xcodebuild failed!'
+              exit 1
+            fi
+      - run:
+          name: Run Simulator Tests
+          no_output_timeout: "2h"
+          command: |
+            set -e
+            if [ ${IOS_PLATFORM} != "SIMULATOR" ]; then
+              echo "not SIMULATOR build, skip it."
+              exit 0
+            fi
+            WORKSPACE=/Users/distiller/workspace
+            PROJ_ROOT=/Users/distiller/project
+            source ~/anaconda/bin/activate
+            pip install torch torchvision --progress-bar off
+            #run unit test
+            cd ${PROJ_ROOT}/ios/TestApp/benchmark
+            python trace_model.py
+            ruby setup.rb
+            cd ${PROJ_ROOT}/ios/TestApp
+            instruments -s -devices
+            fastlane scan

.circleci/verbatim-sources/job-specs-setup.yml

@@ -27,3 +27,4 @@
       - persist_to_workspace:
           root: .
           paths: .circleci/scripts
+

.circleci/verbatim-sources/job-specs/job-specs-custom.yml

@@ -1,601 +0,0 @@
-  pytorch_doc_push:
-    resource_class: medium
-    machine:
-      image: ubuntu-1604:202007-01
-    parameters:
-      branch:
-        type: string
-        default: "master"
-    steps:
-    - attach_workspace:
-        at: /tmp/workspace
-    - run:
-        name: Generate netrc
-        command: |
-          # set credentials for https pushing
-          cat > ~/.netrc \<<DONE
-            machine github.com
-            login pytorchbot
-            password ${GITHUB_PYTORCHBOT_TOKEN}
-          DONE
-    - run:
-        name: Docs push
-        command: |
-          pushd /tmp/workspace
-          git push -u origin "<< parameters.branch >>"
-
-  pytorch_python_doc_build:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-python-doc-push
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.6-gcc5.4"
-    resource_class: large
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: Doc Build and Push
-        no_output_timeout: "1h"
-        command: |
-          set -ex
-          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-          tag=${CIRCLE_TAG:1:5}
-          target=${tag:-master}
-          echo "building for ${target}"
-          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-          export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && '"export CIRCLE_SHA1='$CIRCLE_SHA1'"' && . ./.circleci/scripts/python_doc_push_script.sh docs/'$target' '$target' site") | docker exec -u jenkins -i "$id" bash) 2>&1'
-
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_artifacts
-          docker cp $id:/var/lib/jenkins/workspace/pytorch.github.io/docs/master ~/workspace/build_artifacts
-          docker cp $id:/var/lib/jenkins/workspace/pytorch.github.io /tmp/workspace
-
-          # Save the docs build so we can debug any problems
-          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
-          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
-          time docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
-    - persist_to_workspace:
-        root: /tmp/workspace
-        paths:
-          - .
-    - store_artifacts:
-        path: ~/workspace/build_artifacts/master
-        destination: docs
-
-  pytorch_cpp_doc_build:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-cpp-doc-push
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.6-gcc5.4"
-    resource_class: large
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: Doc Build and Push
-        no_output_timeout: "1h"
-        command: |
-          set -ex
-          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-          tag=${CIRCLE_TAG:1:5}
-          target=${tag:-master}
-          echo "building for ${target}"
-          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-          export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && '"export CIRCLE_SHA1='$CIRCLE_SHA1'"' && . ./.circleci/scripts/cpp_doc_push_script.sh docs/"$target" master") | docker exec -u jenkins -i "$id" bash) 2>&1'
-
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_artifacts
-          docker cp $id:/var/lib/jenkins/workspace/cppdocs/ /tmp/workspace
-
-          # Save the docs build so we can debug any problems
-          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
-          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
-          time docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
-
-    - persist_to_workspace:
-        root: /tmp/workspace
-        paths:
-          - .
-
-  pytorch_macos_10_13_py3_build:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
-    macos:
-      xcode: "12.0"
-    steps:
-      - checkout
-      - run_brew_for_macos_build
-      - run:
-          name: Build
-          no_output_timeout: "1h"
-          command: |
-            set -e
-            export IN_CI=1
-
-            # Install sccache
-            sudo curl --retry 3 https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
-            sudo chmod +x /usr/local/bin/sccache
-            export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
-
-            # This IAM user allows write access to S3 bucket for sccache
-            set +x
-            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
-            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
-            set -x
-
-            chmod a+x .jenkins/pytorch/macos-build.sh
-            unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
-
-      - persist_to_workspace:
-          root: /Users/distiller/workspace/
-          paths:
-            - miniconda3
-
-  pytorch_macos_10_13_py3_test:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
-    macos:
-      xcode: "12.0"
-    steps:
-      - checkout
-      - attach_workspace:
-          at: ~/workspace
-      - run_brew_for_macos_build
-      - run:
-          name: Test
-          no_output_timeout: "1h"
-          command: |
-            set -e
-            export IN_CI=1
-
-            chmod a+x .jenkins/pytorch/macos-test.sh
-            unbuffer .jenkins/pytorch/macos-test.sh 2>&1 | ts
-      - store_test_results:
-          path: test/test-reports
-
-  pytorch_android_gradle_build:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c"
-      PYTHON_VERSION: "3.6"
-    resource_class: large
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: pytorch android gradle build
-        no_output_timeout: "1h"
-        command: |
-          set -eux
-          docker_image_commit=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-
-          docker_image_libtorch_android_x86_32=${docker_image_commit}-android-x86_32
-          docker_image_libtorch_android_x86_64=${docker_image_commit}-android-x86_64
-          docker_image_libtorch_android_arm_v7a=${docker_image_commit}-android-arm-v7a
-          docker_image_libtorch_android_arm_v8a=${docker_image_commit}-android-arm-v8a
-
-          echo "docker_image_commit: "${docker_image_commit}
-          echo "docker_image_libtorch_android_x86_32: "${docker_image_libtorch_android_x86_32}
-          echo "docker_image_libtorch_android_x86_64: "${docker_image_libtorch_android_x86_64}
-          echo "docker_image_libtorch_android_arm_v7a: "${docker_image_libtorch_android_arm_v7a}
-          echo "docker_image_libtorch_android_arm_v8a: "${docker_image_libtorch_android_arm_v8a}
-
-          # x86_32
-          time docker pull ${docker_image_libtorch_android_x86_32} >/dev/null
-          export id_x86_32=$(docker run --env-file "${BASH_ENV}" -e GRADLE_OFFLINE=1 --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          # arm-v7a
-          time docker pull ${docker_image_libtorch_android_arm_v7a} >/dev/null
-          export id_arm_v7a=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_arm_v7a})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_arm_v7a" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_android_install_arm_v7a
-          docker cp $id_arm_v7a:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_arm_v7a
-
-          # x86_64
-          time docker pull ${docker_image_libtorch_android_x86_64} >/dev/null
-          export id_x86_64=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_64})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_x86_64" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_android_install_x86_64
-          docker cp $id_x86_64:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_x86_64
-
-          # arm-v8a
-          time docker pull ${docker_image_libtorch_android_arm_v8a} >/dev/null
-          export id_arm_v8a=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_arm_v8a})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_arm_v8a" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_android_install_arm_v8a
-          docker cp $id_arm_v8a:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_arm_v8a
-
-          docker cp ~/workspace/build_android_install_arm_v7a $id_x86_32:/var/lib/jenkins/workspace/build_android_install_arm_v7a
-          docker cp ~/workspace/build_android_install_x86_64 $id_x86_32:/var/lib/jenkins/workspace/build_android_install_x86_64
-          docker cp ~/workspace/build_android_install_arm_v8a $id_x86_32:/var/lib/jenkins/workspace/build_android_install_arm_v8a
-
-          # run gradle buildRelease
-          export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_android_artifacts
-          docker cp $id_x86_32:/var/lib/jenkins/workspace/android/artifacts.tgz ~/workspace/build_android_artifacts/
-
-          output_image=$docker_image_libtorch_android_x86_32-gradle
-          docker commit "$id_x86_32" ${output_image}
-          time docker push ${output_image}
-    - upload_binary_size_for_android_build:
-        build_type: prebuilt
-        artifacts: /home/circleci/workspace/build_android_artifacts/artifacts.tgz
-    - store_artifacts:
-        path: ~/workspace/build_android_artifacts/artifacts.tgz
-        destination: artifacts.tgz
-
-  pytorch_android_publish_snapshot:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-publish-snapshot
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c"
-      PYTHON_VERSION: "3.6"
-    resource_class: large
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: pytorch android gradle build
-        no_output_timeout: "1h"
-        command: |
-          set -eux
-          docker_image_commit=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-
-          docker_image_libtorch_android_x86_32_gradle=${docker_image_commit}-android-x86_32-gradle
-
-          echo "docker_image_commit: "${docker_image_commit}
-          echo "docker_image_libtorch_android_x86_32_gradle: "${docker_image_libtorch_android_x86_32_gradle}
-
-          # x86_32
-          time docker pull ${docker_image_libtorch_android_x86_32_gradle} >/dev/null
-          export id_x86_32=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32_gradle})
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export SONATYPE_NEXUS_USERNAME=${SONATYPE_NEXUS_USERNAME}" && echo "export SONATYPE_NEXUS_PASSWORD=${SONATYPE_NEXUS_PASSWORD}" && echo "export ANDROID_SIGN_KEY=${ANDROID_SIGN_KEY}" && echo "export ANDROID_SIGN_PASS=${ANDROID_SIGN_PASS}" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/publish_android_snapshot.sh") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          output_image=${docker_image_libtorch_android_x86_32_gradle}-publish-snapshot
-          docker commit "$id_x86_32" ${output_image}
-          time docker push ${output_image}
-
-  pytorch_android_gradle_build-x86_32:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-only-x86_32
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c"
-      PYTHON_VERSION: "3.6"
-    resource_class: large
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - checkout
-    - setup_ci_environment
-    - run:
-        name: pytorch android gradle build only x86_32 (for PR)
-        no_output_timeout: "1h"
-        command: |
-          set -e
-          docker_image_libtorch_android_x86_32=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}-android-x86_32
-          echo "docker_image_libtorch_android_x86_32: "${docker_image_libtorch_android_x86_32}
-
-          # x86
-          time docker pull ${docker_image_libtorch_android_x86_32} >/dev/null
-          export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32})
-
-          export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GRADLE_OFFLINE=1" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          mkdir -p ~/workspace/build_android_x86_32_artifacts
-          docker cp $id:/var/lib/jenkins/workspace/android/artifacts.tgz ~/workspace/build_android_x86_32_artifacts/
-
-          output_image=${docker_image_libtorch_android_x86_32}-gradle
-          docker commit "$id" ${output_image}
-          time docker push ${output_image}
-    - upload_binary_size_for_android_build:
-        build_type: prebuilt-single
-        artifacts: /home/circleci/workspace/build_android_x86_32_artifacts/artifacts.tgz
-    - store_artifacts:
-        path: ~/workspace/build_android_x86_32_artifacts/artifacts.tgz
-        destination: artifacts.tgz
-
-  pytorch_android_gradle_custom_build_single:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c"
-      PYTHON_VERSION: "3.6"
-    resource_class: large
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - checkout
-    - calculate_docker_image_tag
-    - setup_ci_environment
-    - run:
-        name: pytorch android gradle custom build single architecture (for PR)
-        no_output_timeout: "1h"
-        command: |
-          set -e
-          # Unlike other gradle jobs, it's not worth building libtorch in a separate CI job and share via docker, because:
-          # 1) Not shareable: it's custom selective build, which is different from default libtorch mobile build;
-          # 2) Not parallelizable by architecture: it only builds libtorch for one architecture;
-
-          echo "DOCKER_IMAGE: ${DOCKER_IMAGE}:${DOCKER_TAG}"
-          time docker pull ${DOCKER_IMAGE}:${DOCKER_TAG} >/dev/null
-
-          git submodule sync && git submodule update -q --init --recursive
-          VOLUME_MOUNTS="-v /home/circleci/project/:/var/lib/jenkins/workspace"
-          export id=$(docker run --env-file "${BASH_ENV}" ${VOLUME_MOUNTS} --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${DOCKER_IMAGE}:${DOCKER_TAG})
-
-          export COMMAND='((echo "export GRADLE_OFFLINE=1" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          # Skip docker push as this job is purely for size analysis purpose.
-          # Result binaries are already in `/home/circleci/project/` as it's mounted instead of copied.
-
-    - upload_binary_size_for_android_build:
-        build_type: custom-build-single
-
-  pytorch_ios_build:
-    <<: *pytorch_ios_params
-    macos:
-      xcode: "12.0"
-    steps:
-      - checkout
-      - run_brew_for_ios_build
-      - run:
-          name: Run Fastlane
-          no_output_timeout: "1h"
-          command: |
-            set -e
-            PROJ_ROOT=/Users/distiller/project
-            cd ${PROJ_ROOT}/ios/TestApp
-            # install fastlane
-            sudo gem install bundler && bundle install
-            # install certificates
-            echo ${IOS_CERT_KEY} >> cert.txt
-            base64 --decode cert.txt -o Certificates.p12
-            rm cert.txt
-            bundle exec fastlane install_cert
-            # install the provisioning profile
-            PROFILE=PyTorch_CI_2021.mobileprovision
-            PROVISIONING_PROFILES=~/Library/MobileDevice/Provisioning\ Profiles
-            mkdir -pv "${PROVISIONING_PROFILES}"
-            cd "${PROVISIONING_PROFILES}"
-            echo ${IOS_SIGN_KEY} >> cert.txt
-            base64 --decode cert.txt -o ${PROFILE}
-            rm cert.txt
-      - run:
-          name: Build
-          no_output_timeout: "1h"
-          command: |
-            set -e
-            export IN_CI=1
-            WORKSPACE=/Users/distiller/workspace
-            PROJ_ROOT=/Users/distiller/project
-            export TCLLIBPATH="/usr/local/lib"
-
-            # Install conda
-            curl --retry 3 -o ~/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
-            chmod +x ~/conda.sh
-            /bin/bash ~/conda.sh -b -p ~/anaconda
-            export PATH="~/anaconda/bin:${PATH}"
-            source ~/anaconda/bin/activate
-
-            # Install dependencies
-            retry () {
-                $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-            }
-
-            retry conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi requests typing_extensions --yes
-
-            # sync submodules
-            cd ${PROJ_ROOT}
-            git submodule sync
-            git submodule update --init --recursive
-
-            # export
-            export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
-
-            # run build script
-            chmod a+x ${PROJ_ROOT}/scripts/build_ios.sh
-            echo "IOS_ARCH: ${IOS_ARCH}"
-            echo "IOS_PLATFORM: ${IOS_PLATFORM}"
-            echo "USE_PYTORCH_METAL": "${USE_METAL}"
-
-            #check the custom build flag
-            echo "SELECTED_OP_LIST: ${SELECTED_OP_LIST}"
-            if [ -n "${SELECTED_OP_LIST}" ]; then
-                export SELECTED_OP_LIST="${PROJ_ROOT}/ios/TestApp/custom_build/${SELECTED_OP_LIST}"
-            fi
-            export IOS_ARCH=${IOS_ARCH}
-            export IOS_PLATFORM=${IOS_PLATFORM}
-            if [ ${IOS_PLATFORM} != "SIMULATOR" ]; then
-              export USE_PYTORCH_METAL=${USE_METAL}
-            fi
-            unbuffer ${PROJ_ROOT}/scripts/build_ios.sh 2>&1 | ts
-      - run:
-          name: Run Build Test
-          no_output_timeout: "30m"
-          command: |
-            set -e
-            PROJ_ROOT=/Users/distiller/project
-            PROFILE=PyTorch_CI_2021
-            # run the ruby build script
-            if ! [ -x "$(command -v xcodebuild)" ]; then
-              echo 'Error: xcodebuild is not installed.'
-              exit 1
-            fi
-            echo ${IOS_DEV_TEAM_ID}
-            if [ ${IOS_PLATFORM} != "SIMULATOR" ]; then
-              ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM} -c ${PROFILE} -t ${IOS_DEV_TEAM_ID}
-            else
-              ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM}
-            fi
-            if ! [ "$?" -eq "0" ]; then
-              echo 'xcodebuild failed!'
-              exit 1
-            fi
-      - run:
-          name: Run Simulator Tests
-          no_output_timeout: "2h"
-          command: |
-            set -e
-            if [ ${IOS_PLATFORM} != "SIMULATOR" ]; then
-              echo "not SIMULATOR build, skip it."
-              exit 0
-            fi
-            WORKSPACE=/Users/distiller/workspace
-            PROJ_ROOT=/Users/distiller/project
-            source ~/anaconda/bin/activate
-            pip install torch torchvision --progress-bar off
-            #run unit test
-            cd ${PROJ_ROOT}/ios/TestApp/benchmark
-            python trace_model.py
-            ruby setup.rb
-            cd ${PROJ_ROOT}/ios/TestApp
-            instruments -s -devices
-            fastlane scan
-  pytorch_linux_bazel_build:
-    <<: *pytorch_params
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: Bazel Build
-        no_output_timeout: "1h"
-        command: |
-          set -e
-          # Pull Docker image and run build
-          echo "DOCKER_IMAGE: "${DOCKER_IMAGE}:${DOCKER_TAG}
-          time docker pull ${DOCKER_IMAGE}:${DOCKER_TAG} >/dev/null
-          export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${DOCKER_IMAGE}:${DOCKER_TAG})
-
-          echo "Do NOT merge master branch into $CIRCLE_BRANCH in environment $BUILD_ENVIRONMENT"
-
-          git submodule sync && git submodule update -q --init --recursive
-
-          docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/build.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          # Push intermediate Docker image for next phase to use
-          if [ -z "${BUILD_ONLY}" ]; then
-            # Augment our output image name with bazel to avoid collisions
-            output_image=${DOCKER_IMAGE}:${DOCKER_TAG}-bazel-${CIRCLE_SHA1}
-            export COMMIT_DOCKER_IMAGE=$output_image
-            docker commit "$id" ${COMMIT_DOCKER_IMAGE}
-            time docker push ${COMMIT_DOCKER_IMAGE}
-          fi
-
-  pytorch_linux_bazel_test:
-    <<: *pytorch_params
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: Test
-        no_output_timeout: "90m"
-        command: |
-          set -e
-          output_image=${DOCKER_IMAGE}:${DOCKER_TAG}-bazel-${CIRCLE_SHA1}
-          export COMMIT_DOCKER_IMAGE=$output_image
-          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-
-          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-
-          if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
-            export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --gpus all -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-          else
-            export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-          fi
-
-          retrieve_test_reports() {
-            echo "retrieving test reports"
-            docker cp -L $id:/var/lib/jenkins/workspace/bazel-testlogs ./ || echo 'No test reports found!'
-          }
-          trap "retrieve_test_reports" ERR
-
-          if [[ ${BUILD_ENVIRONMENT} == *"multigpu"* ]]; then
-            export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/multigpu-test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-          else
-            export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-          fi
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          retrieve_test_reports
-          docker stats --all --no-stream
-    - store_test_results:
-        path: bazel-testlogs
-
-  pytorch_doc_test:
-    environment:
-      BUILD_ENVIRONMENT: pytorch-doc-test
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.6-gcc5.4"
-    resource_class: medium
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: Doc test
-        no_output_timeout: "30m"
-        command: |
-          set -ex
-          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-          export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-          export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && . ./.jenkins/pytorch/docs-test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts

.circleci/verbatim-sources/job-specs/job-specs-promote.yml

@@ -1,18 +0,0 @@
-
-  promote_s3:
-    <<: *promote_common
-    steps:
-      - checkout
-      - run:
-          name: Running promote script
-          command: |
-            scripts/release/promote/wheel_to_s3.sh
-
-  promote_conda:
-    <<: *promote_common
-    steps:
-      - checkout
-      - run:
-          name: Running promote script
-          command: |
-            scripts/release/promote/conda_to_conda.sh

.circleci/verbatim-sources/job-specs/pytorch-job-specs.yml

@@ -1,355 +0,0 @@
-jobs:
-  pytorch_linux_build:
-    <<: *pytorch_params
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - optional_merge_target_branch
-    - setup_ci_environment
-    - run:
-        name: Build
-        no_output_timeout: "1h"
-        command: |
-          set -e
-          if [[ "${DOCKER_IMAGE}" == *rocm3.9* ]]; then
-            export DOCKER_TAG="f3d89a32912f62815e4feaeed47e564e887dffd6"
-          fi
-          if [[ ${BUILD_ENVIRONMENT} == *"pure_torch"* ]]; then
-            echo 'BUILD_CAFFE2=OFF' >> "${BASH_ENV}"
-          fi
-          if [[ ${BUILD_ENVIRONMENT} == *"paralleltbb"* ]]; then
-            echo 'ATEN_THREADING=TBB' >> "${BASH_ENV}"
-            echo 'USE_TBB=1' >> "${BASH_ENV}"
-          elif [[ ${BUILD_ENVIRONMENT} == *"parallelnative"* ]]; then
-            echo 'ATEN_THREADING=NATIVE' >> "${BASH_ENV}"
-          fi
-          echo "Parallel backend flags: "${PARALLEL_FLAGS}
-          # Pull Docker image and run build
-          echo "DOCKER_IMAGE: "${DOCKER_IMAGE}:${DOCKER_TAG}
-          time docker pull ${DOCKER_IMAGE}:${DOCKER_TAG} >/dev/null
-          export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -d -w /var/lib/jenkins ${DOCKER_IMAGE}:${DOCKER_TAG})
-
-          git submodule sync && git submodule update -q --init --recursive
-
-          docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
-
-          export COMMAND='((echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/build.sh && find ${BUILD_ROOT} -type f -name "*.a" -or -name "*.o" -delete") | docker exec -u jenkins -i "$id" bash) 2>&1'
-
-          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-          # Copy dist folder back
-          docker cp $id:/var/lib/jenkins/workspace/dist /home/circleci/project/. || echo "Dist folder not found"
-
-          # Push intermediate Docker image for next phase to use
-          if [ -z "${BUILD_ONLY}" ]; then
-            # Note [Special build images]
-            # The xla build uses the same docker image as
-            # pytorch_linux_bionic_py3_6_clang9_build. In the push step, we have to
-            # distinguish between them so the test can pick up the correct image.
-            output_image=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-            if [[ ${BUILD_ENVIRONMENT} == *"xla"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-xla
-            elif [[ ${BUILD_ENVIRONMENT} == *"libtorch"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-libtorch
-            elif [[ ${BUILD_ENVIRONMENT} == *"paralleltbb"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-paralleltbb
-            elif [[ ${BUILD_ENVIRONMENT} == *"parallelnative"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-parallelnative
-            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-x86_64"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-android-x86_64
-            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-arm-v7a"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-android-arm-v7a
-            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-arm-v8a"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-android-arm-v8a
-            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-x86_32"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-android-x86_32
-            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-vulkan-x86_32"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-android-vulkan-x86_32
-            elif [[ ${BUILD_ENVIRONMENT} == *"vulkan-linux"* ]]; then
-              export COMMIT_DOCKER_IMAGE=$output_image-vulkan
-            else
-              export COMMIT_DOCKER_IMAGE=$output_image
-            fi
-            docker commit "$id" ${COMMIT_DOCKER_IMAGE}
-            time docker push ${COMMIT_DOCKER_IMAGE}
-          fi
-    - store_artifacts:
-        path: /home/circleci/project/dist
-
-  pytorch_linux_test:
-    <<: *pytorch_params
-    machine:
-      image: ubuntu-1604:202007-01
-    steps:
-    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
-    - checkout
-    - calculate_docker_image_tag
-    - setup_linux_system_environment
-    - setup_ci_environment
-    - run:
-        name: Download Docker image
-        no_output_timeout: "90m"
-        command: |
-          set -e
-          export PYTHONUNBUFFERED=1
-          if [[ "${DOCKER_IMAGE}" == *rocm3.9* ]]; then
-            export DOCKER_TAG="f3d89a32912f62815e4feaeed47e564e887dffd6"
-          fi
-          # See Note [Special build images]
-          output_image=${DOCKER_IMAGE}:${DOCKER_TAG}-${CIRCLE_SHA1}
-          if [[ ${BUILD_ENVIRONMENT} == *"xla"* ]]; then
-            export COMMIT_DOCKER_IMAGE=$output_image-xla
-          elif [[ ${BUILD_ENVIRONMENT} == *"libtorch"* ]]; then
-            export COMMIT_DOCKER_IMAGE=$output_image-libtorch
-          elif [[ ${BUILD_ENVIRONMENT} == *"paralleltbb"* ]]; then
-            export COMMIT_DOCKER_IMAGE=$output_image-paralleltbb
-          elif [[ ${BUILD_ENVIRONMENT} == *"parallelnative"* ]]; then
-            export COMMIT_DOCKER_IMAGE=$output_image-parallelnative
-          elif [[ ${BUILD_ENVIRONMENT} == *"vulkan-linux"* ]]; then
-            export COMMIT_DOCKER_IMAGE=$output_image-vulkan
-          else
-            export COMMIT_DOCKER_IMAGE=$output_image
-          fi
-          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-
-          if [[ ${BUILD_ENVIRONMENT} == *"paralleltbb"* ]]; then
-            echo 'ATEN_THREADING=TBB' >> "${BASH_ENV}"
-            echo 'USE_TBB=1' >> "${BASH_ENV}"
-          elif [[ ${BUILD_ENVIRONMENT} == *"parallelnative"* ]]; then
-            echo 'ATEN_THREADING=NATIVE' >> "${BASH_ENV}"
-          fi
-          echo "Parallel backend flags: "${PARALLEL_FLAGS}
-
-          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-
-          # TODO: Make this less painful
-          if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
-            export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --gpus all --shm-size=2g -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-          elif [[ ${BUILD_ENVIRONMENT} == *"rocm"* ]]; then
-            hostname
-            export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --shm-size=8g --ipc=host --device /dev/kfd --device /dev/dri --group-add video -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-          else
-            export id=$(docker run --env-file "${BASH_ENV}" --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --shm-size=1g --ipc=host -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-          fi
-          echo "id=${id}" >> "${BASH_ENV}"
-
-    - run:
-        name: Check for no AVX instruction by default
-        no_output_timeout: "20m"
-        command: |
-          set -e
-          is_vanilla_build() {
-            if [ "${BUILD_ENVIRONMENT}" == "pytorch-linux-bionic-py3.6-clang9-test" ]; then
-              return 0
-            fi
-            if [ "${BUILD_ENVIRONMENT}" == "pytorch-linux-xenial-py3.6-gcc5.4-test" ]; then
-              return 0
-            fi
-            return 1
-          }
-
-          if is_vanilla_build; then
-            echo "apt-get update && apt-get install -y qemu-user gdb" | docker exec -u root -i "$id" bash
-            echo "cd workspace/build; qemu-x86_64 -g 2345 -cpu Broadwell -E ATEN_CPU_CAPABILITY=default ./bin/basic --gtest_filter=BasicTest.BasicTestCPU & gdb ./bin/basic -ex 'set pagination off' -ex 'target remote :2345' -ex 'continue' -ex 'bt' -ex='set confirm off' -ex 'quit \$_isvoid(\$_exitcode)'" | docker exec -u jenkins -i "$id" bash
-          else
-            echo "Skipping for ${BUILD_ENVIRONMENT}"
-          fi
-    - run:
-        name: Run tests
-        no_output_timeout: "90m"
-        command: |
-          set -e
-
-          cat >docker_commands.sh \<<EOL
-          # =================== The following code will be executed inside Docker container ===================
-          set -ex
-          export SCRIBE_GRAPHQL_ACCESS_TOKEN="${SCRIBE_GRAPHQL_ACCESS_TOKEN}"
-          ${PARALLEL_FLAGS}
-          cd workspace
-          EOL
-          if [[ ${BUILD_ENVIRONMENT} == *"multigpu"* ]]; then
-            echo ".jenkins/pytorch/multigpu-test.sh" >> docker_commands.sh
-          elif [[ ${BUILD_ENVIRONMENT} == *onnx* ]]; then
-            echo "pip install click mock tabulate networkx==2.0" >> docker_commands.sh
-            echo "pip -q install --user \"file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx\"" >> docker_commands.sh
-            echo ".jenkins/caffe2/test.sh" >> docker_commands.sh
-          else
-            echo ".jenkins/pytorch/test.sh" >> docker_commands.sh
-          fi
-          echo "(cat docker_commands.sh | docker exec -u jenkins -i "$id" bash) 2>&1" > command.sh
-          unbuffer bash command.sh | ts
-    - run:
-        name: Report results
-        no_output_timeout: "5m"
-        command: |
-          set -e
-          docker stats --all --no-stream
-
-          cat >docker_commands.sh \<<EOL
-          # =================== The following code will be executed inside Docker container ===================
-          set -ex
-          export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}
-          export SCRIBE_GRAPHQL_ACCESS_TOKEN="${SCRIBE_GRAPHQL_ACCESS_TOKEN}"
-          export CIRCLE_TAG="${CIRCLE_TAG:-}"
-          export CIRCLE_SHA1="$CIRCLE_SHA1"
-          export CIRCLE_PR_NUMBER="${CIRCLE_PR_NUMBER:-}"
-          export CIRCLE_BRANCH="$CIRCLE_BRANCH"
-          export CIRCLE_JOB="$CIRCLE_JOB"
-          export CIRCLE_WORKFLOW_ID="$CIRCLE_WORKFLOW_ID"
-          cd workspace
-          python test/print_test_stats.py --upload-to-s3 test
-          EOL
-          echo "(cat docker_commands.sh | docker exec -u jenkins -i "$id" bash) 2>&1" > command.sh
-          unbuffer bash command.sh | ts
-
-          echo "Retrieving test reports"
-          docker cp $id:/var/lib/jenkins/workspace/test/test-reports ./ || echo 'No test reports found!'
-          if [[ ${BUILD_ENVIRONMENT} == *"coverage"* ]]; then
-              echo "Retrieving C++ coverage report"
-              docker cp $id:/var/lib/jenkins/workspace/build/coverage.info ./test
-          fi
-          if [[ ${BUILD_ENVIRONMENT} == *"coverage"* || ${BUILD_ENVIRONMENT} == *"onnx"* ]]; then
-              echo "Retrieving Python coverage report"
-              docker cp $id:/var/lib/jenkins/workspace/test/.coverage ./test
-              docker cp $id:/var/lib/jenkins/workspace/test/coverage.xml ./test
-              python3 -mpip install codecov
-              python3 -mcodecov
-          fi
-        when: always
-    - store_test_results:
-        path: test-reports
-
-  pytorch_windows_build:
-    <<: *pytorch_windows_params
-    parameters:
-      executor:
-        type: string
-        default: "windows-xlarge-cpu-with-nvidia-cuda"
-      build_environment:
-        type: string
-        default: ""
-      test_name:
-        type: string
-        default: ""
-      cuda_version:
-        type: string
-        default: "10.1"
-      python_version:
-        type: string
-        default: "3.6"
-      vc_version:
-        type: string
-        default: "14.16"
-      vc_year:
-        type: string
-        default: "2019"
-      vc_product:
-        type: string
-        default: "BuildTools"
-      use_cuda:
-        type: string
-        default: ""
-    executor: <<parameters.executor>>
-    steps:
-      - checkout
-      - run:
-          name: _HACK_ Install CUDA compatible cmath
-          no_output_timeout: 1m
-          command: |
-              powershell .circleci/scripts/vs_install_cmath.ps1
-      - run:
-          name: Install Cuda
-          no_output_timeout: 30m
-          command: |
-            if [[ "${USE_CUDA}" == "1" ]]; then
-              .circleci/scripts/windows_cuda_install.sh
-            fi
-      - run:
-          name: Install Cudnn
-          command : |
-            if [[ "${USE_CUDA}" == "1" ]]; then
-              .circleci/scripts/windows_cudnn_install.sh
-            fi
-      - run:
-          name: Build
-          no_output_timeout: "90m"
-          command: |
-            set -e
-            set +x
-            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_WIN_BUILD_V1}
-            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_WIN_BUILD_V1}
-            set -x
-            .jenkins/pytorch/win-build.sh
-      - persist_to_workspace:
-          root: "C:/w"
-          paths: build-results
-      - store_artifacts:
-          path: C:/w/build-results
-
-  pytorch_windows_test:
-    <<: *pytorch_windows_params
-    parameters:
-      executor:
-        type: string
-        default: "windows-medium-cpu-with-nvidia-cuda"
-      build_environment:
-        type: string
-        default: ""
-      test_name:
-        type: string
-        default: ""
-      cuda_version:
-        type: string
-        default: "10.1"
-      python_version:
-        type: string
-        default: "3.6"
-      vc_version:
-        type: string
-        default: "14.16"
-      vc_year:
-        type: string
-        default: "2019"
-      vc_product:
-        type: string
-        default: "BuildTools"
-      use_cuda:
-        type: string
-        default: ""
-    executor: <<parameters.executor>>
-    steps:
-      - checkout
-      - attach_workspace:
-          at: c:/users/circleci/workspace
-      - run:
-          name: Install Cuda
-          no_output_timeout: 30m
-          command: |
-            if [[ "${CUDA_VERSION}" != "cpu" ]]; then
-              if [[ "${CUDA_VERSION}" != "10" || "${JOB_EXECUTOR}" != "windows-with-nvidia-gpu" ]]; then
-                .circleci/scripts/windows_cuda_install.sh
-              fi
-            fi
-      - run:
-          name: Install Cudnn
-          command : |
-            if [[ "${CUDA_VERSION}" != "cpu" ]]; then
-              .circleci/scripts/windows_cudnn_install.sh
-            fi
-      - run:
-          name: Test
-          no_output_timeout: "30m"
-          command: |
-            set -e
-            export IN_CI=1
-            set +x
-            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_WIN_BUILD_V1}
-            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_WIN_BUILD_V1}
-            set -x
-            .jenkins/pytorch/win-test.sh
-      - store_test_results:
-          path: test/test-reports