Update onnx.rst (#40605)

Correcting the link (current 404s)

.bazelrc

@@ -1,3 +0,0 @@
-build --copt=--std=c++14
-build --copt=-I.
-build --copt=-isystem --copt bazel-out/k8-fastbuild/bin

.bazelversion

@@ -1 +0,0 @@
-3.1.0

.circleci/README.md

@@ -71,9 +71,8 @@ A **binary configuration** is a collection of
 * release or nightly
     * releases are stable, nightlies are beta and built every night
 * python version
-    * linux: 3.5m, 3.6m 3.7m (mu is wide unicode or something like that. It usually doesn't matter but you should know that it exists)
-    * macos: 3.6, 3.7, 3.8
-    * windows: 3.6, 3.7, 3.8
+    * linux: 2.7m, 2.7mu, 3.5m, 3.6m 3.7m (mu is wide unicode or something like that. It usually doesn't matter but you should know that it exists)
+    * macos and windows: 2.7, 3.5, 3.6, 3.7
 * cpu version
     * cpu, cuda 9.0, cuda 10.0
     * The supported cuda versions occasionally change
@@ -93,12 +92,12 @@ We have 3 types of binary packages
 * pip packages - nightlies are stored on s3 (pip install -f <a s3 url>). releases are stored in a pip repo (pip install torch) (ask Soumith about this)
 * conda packages - nightlies and releases are both stored in a conda repo. Nighty packages have a '_nightly' suffix
 * libtorch packages - these are zips of all the c++ libraries, header files, and sometimes dependencies. These are c++ only
-    * shared with dependencies (the only supported option for Windows)
+    * shared with dependencies
     * static with dependencies
     * shared without dependencies
     * static without dependencies
 
-All binaries are built in CircleCI workflows except Windows. There are checked-in workflows (committed into the .circleci/config.yml) to build the nightlies every night. Releases are built by manually pushing a PR that builds the suite of release binaries (overwrite the config.yml to build the release)
+All binaries are built in CircleCI workflows. There are checked-in workflows (committed into the .circleci/config.yml) to build the nightlies every night. Releases are built by manually pushing a PR that builds the suite of release binaries (overwrite the config.yml to build the release)
 
 # CircleCI structure of the binaries
 
@@ -189,18 +188,6 @@ binary_run_in_docker.sh is a way to share the docker start-up code between the b
 
 We want all the nightly binary jobs to run on the exact same git commit, so we wrote our own checkout logic to ensure that the same commit was always picked. Later circleci changed that to use a single pytorch checkout and persist it through the workspace (they did this because our config file was too big, so they wanted to take a lot of the setup code into scripts, but the scripts needed the code repo to exist to be called, so they added a prereq step called 'setup' to checkout the code and persist the needed scripts to the workspace). The changes to the binary jobs were not properly tested, so they all broke from missing pytorch code no longer existing. We hotfixed the problem by adding the pytorch checkout back to binary_checkout, so now there's two checkouts of pytorch on the binary jobs. This problem still needs to be fixed, but it takes careful tracing of which code is being called where.
 
-# Azure Pipelines structure of the binaries
-
-TODO: fill in stuff
-
-## How are the workflows structured?
-
-TODO: fill in stuff
-
-## How are the jobs structured?
-
-TODO: fill in stuff
-
 # Code structure of the binaries (circleci agnostic)
 
 ## Overview
@@ -248,15 +235,13 @@ pytorch/builder
   - build_common.sh           # Actual build script that ^^ call into
 - wheel/
   - build_wheel.sh            # Entrypoint for wheel builds
-- windows/
-  - build_pytorch.bat         # Entrypoint for wheel builds on Windows
 ```
 
 Every type of package has an entrypoint build script that handles the all the important logic.
 
 ## Conda
 
-Linux, MacOS and Windows use the same code flow for the conda builds.
+Both Linux and MacOS use the same code flow for the conda builds.
 
 Conda packages are built with conda-build, see https://conda.io/projects/conda-build/en/latest/resources/commands/conda-build.html
 
@@ -275,7 +260,7 @@ The build.sh we use is essentially a wrapper around ```python setup.py build```
 
 The entrypoint file `builder/conda/build_conda.sh` is complicated because
 
-* It works for Linux, MacOS and Windows
+* It works for both Linux and MacOS
     * The mac builds used to create their own environments, since they all used to be on the same machine. There’s now a lot of extra logic to handle conda envs. This extra machinery could be removed
 * It used to handle testing too, which adds more logic messing with python environments too. This extra machinery could be removed.
 
@@ -287,7 +272,7 @@ Manywheels are pip packages for linux distros. Note that these manywheels are no
 
 The entrypoint file `builder/manywheel/build_common.sh` is really really complicated because
 
-* This used to handle building for several different python versions at the same time. The loops have been removed, but there's still unnecessary folders and movements here and there.
+* This used to handle building for several different python versions at the same time. The loops have been removed, but there's still unneccessary folders and movements here and there.
     * The script is never used this way anymore. This extra machinery could be removed.
 * This used to handle testing the pip packages too. This is why there’s testing code at the end that messes with python installations and stuff
     * The script is never used this way anymore. This extra machinery could be removed.
@@ -306,19 +291,6 @@ The entrypoint file `builder/wheel/build_wheel.sh` is complicated because
 
 Note that the MacOS Python wheels are still built in conda environments. Some of the dependencies present during build also come from conda.
 
-## Windows Wheels (Windows pip and libtorch packages)
-
-The entrypoint file `builder/windows/build_pytorch.bat` is complicated because
-
-* This used to handle building for several different python versions at the same time. This is why there are loops everywhere
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This used to handle testing the pip packages too. This is why there’s testing code at the end that messes with python installations and stuff
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This also builds libtorch packages
-    * This should really be separate. libtorch packages are c++ only and have no python. They should not share infra with all the python specific stuff in this file.
-
-Note that the Windows Python wheels are still built in conda environments. Some of the dependencies present during build also come from conda.
-
 ## General notes
 
 ### Note on run_tests.sh, smoke_test.sh, and check_binary.sh
@@ -332,7 +304,7 @@ Note that the Windows Python wheels are still built in conda environments. Some
 
 Libtorch packages are built in the wheel build scripts: manywheel/build_*.sh for linux and build_wheel.sh for mac. There are several things wrong with this
 
-* It’s confusing. Most of those scripts deal with python specifics.
+* It’s confusinig. Most of those scripts deal with python specifics.
 * The extra conditionals everywhere severely complicate the wheel build scripts
 * The process for building libtorch is different from the official instructions (a plain call to cmake, or a call to a script)
 
@@ -340,12 +312,12 @@ Libtorch packages are built in the wheel build scripts: manywheel/build_*.sh for
 
 All linux builds occur in docker images. The docker images are
 
-* pytorch/conda-cuda
+* soumith/conda-cuda
     * Has ALL CUDA versions installed. The script pytorch/builder/conda/switch_cuda_version.sh sets /usr/local/cuda to a symlink to e.g. /usr/local/cuda-10.0 to enable different CUDA builds
     * Also used for cpu builds
-* pytorch/manylinux-cuda90
-* pytorch/manylinux-cuda92
-* pytorch/manylinux-cuda100
+* soumith/manylinux-cuda90
+* soumith/manylinux-cuda92
+* soumith/manylinux-cuda100
     * Also used for cpu builds
 
 The Dockerfiles are available in pytorch/builder, but there is no circleci job or script to build these docker images, and they cannot be run locally (unless you have the correct local packages/paths). Only Soumith can build them right now.
@@ -411,7 +383,7 @@ You can build Linux binaries locally easily using docker.
 
 ```
 # Run the docker
-# Use the correct docker image, pytorch/conda-cuda used here as an example
+# Use the correct docker image, soumith/conda-cuda used here as an example
 #
 # -v path/to/foo:path/to/bar makes path/to/foo on your local machine (the
 #    machine that you're running the command on) accessible to the docker
@@ -426,7 +398,7 @@ docker run \
     -v your/pytorch/repo:/pytorch \
     -v your/builder/repo:/builder \
     -v where/you/want/packages/to/appear:/final_pkgs \
-    -it pytorch/conda-cuda /bin/bash
+    -it soumith/conda-cuda /bin/bash
 
 # Export whatever variables are important to you. All variables that you'd
 # possibly need are in .circleci/scripts/binary_populate_env.sh
@@ -466,7 +438,7 @@ But if you want to try, then I’d recommend
 # Always install miniconda 3, even if building for Python <3
 new_conda="~/my_new_conda"
 conda_sh="$new_conda/install_miniconda.sh"
-curl -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+curl -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
 chmod +x "$conda_sh"
 "$conda_sh" -b -p "$MINICONDA_ROOT"
 rm -f "$conda_sh"
@@ -497,8 +469,8 @@ N.B. installing a brand new miniconda is important. This has to do with how cond
     1. if you installed `foo` in `new_env`, then `path/to/conda_root/envs/new_env/bin/foo` will get called, as expected.
     2. But if you forgot to installed `foo` in `new_env` but happened to previously install it in your root conda env (called ‘base’), then unix/linux will still find `path/to/conda_root/bin/foo` . This is dangerous, since `foo` can be a different version than you want; `foo` can even be for an incompatible python version!
 
-Newer conda versions and proper python hygiene can prevent this, but just install a new miniconda to be safe.
+Newer conda versions and proper python hygeine can prevent this, but just install a new miniconda to be safe.
 
 ### Windows
 
-TODO: fill in
+Maybe @peterjc123 can fill this section in.

.circleci/cimodel/data/binary_build_data.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 """
 This module models the tree of configuration variants
 for "smoketest" builds.
@@ -5,6 +7,9 @@ for "smoketest" builds.
 Each subclass of ConfigNode represents a layer of the configuration hierarchy.
 These tree nodes encapsulate the logic for whether a branch of the hierarchy
 should be "pruned".
+
+In addition to generating config.yml content, the tree is also traversed
+to produce a visualization of config dimensions.
 """
 
 from collections import OrderedDict
@@ -31,13 +36,15 @@ def get_processor_arch_name(cuda_version):
 
 LINUX_PACKAGE_VARIANTS = OrderedDict(
     manywheel=[
+        "2.7m",
+        "2.7mu",
+        "3.5m",
         "3.6m",
         "3.7m",
-        "3.8m",
     ],
-    conda=dimensions.STANDARD_PYTHON_VERSIONS,
+    conda=dimensions.CONDA_PYTHON_VERSIONS,
     libtorch=[
-        "3.7m",
+        "2.7m",
     ],
 )
 
@@ -45,39 +52,26 @@ CONFIG_TREE_DATA = OrderedDict(
     linux=(dimensions.CUDA_VERSIONS, LINUX_PACKAGE_VARIANTS),
     macos=([None], OrderedDict(
         wheel=dimensions.STANDARD_PYTHON_VERSIONS,
-        conda=dimensions.STANDARD_PYTHON_VERSIONS,
+        conda=dimensions.CONDA_PYTHON_VERSIONS,
         libtorch=[
-            "3.7",
-        ],
-    )),
-    windows=(dimensions.CUDA_VERSIONS, OrderedDict(
-        wheel=dimensions.STANDARD_PYTHON_VERSIONS,
-        conda=dimensions.STANDARD_PYTHON_VERSIONS,
-        libtorch=[
-            "3.7",
+            "2.7",
         ],
     )),
 )
 
-# GCC config variants:
-#
-# All the nightlies (except libtorch with new gcc ABI) are built with devtoolset7,
-# which can only build with old gcc ABI. It is better than devtoolset3
-# because it understands avx512, which is needed for good fbgemm performance.
-#
-# Libtorch with new gcc ABI is built with gcc 5.4 on Ubuntu 16.04.
-LINUX_GCC_CONFIG_VARIANTS = OrderedDict(
-    manywheel=['devtoolset7'],
-    conda=['devtoolset7'],
-    libtorch=[
-        "devtoolset7",
-        "gcc5.4_cxx11-abi",
-    ],
-)
 
-WINDOWS_LIBTORCH_CONFIG_VARIANTS = [
-    "debug",
-    "release",
+# Why is this an option?
+# All the nightlies used to be devtoolset3 and built with the old gcc ABI. We
+# added a devtoolset7 option so that we could build nightlies with the new gcc
+# ABI. That didn't work since devtoolset7 can't build with the new gcc ABI. But
+# then we set devtoolset7 to be the default anyways, since devtoolset7
+# understands avx512, which is needed for good fbgemm performance.
+# This should be removed. The base dockers should just be upgraded to
+# devtoolset7 so we don't have to reinstall this in every build job.
+# The same machinery that this uses, though, should be retooled for a different
+# compiler toolchain that can build with the new gcc ABI.
+DEVTOOLSET_VERSIONS = [
+    7,
 ]
 
 
@@ -101,7 +95,13 @@ class OSConfigNode(ConfigNode):
         self.props["cuda_versions"] = cuda_versions
 
     def get_children(self):
-        return [PackageFormatConfigNode(self, k, v) for k, v in self.py_tree.items()]
+        packaging_variants = [PackageFormatConfigNode(self, k, v) for k, v in self.py_tree.items()]
+
+        if self.find_prop("smoke"):
+            filtered_packaging_variants = list(filter(lambda x: x.get_label() != "libtorch", packaging_variants))
+            return filtered_packaging_variants
+        else:
+            return packaging_variants
 
 
 class PackageFormatConfigNode(ConfigNode):
@@ -113,40 +113,28 @@ class PackageFormatConfigNode(ConfigNode):
 
     def get_children(self):
         if self.find_prop("os_name") == "linux":
-            return [LinuxGccConfigNode(self, v) for v in LINUX_GCC_CONFIG_VARIANTS[self.find_prop("package_format")]]
-        elif self.find_prop("os_name") == "windows" and self.find_prop("package_format") == "libtorch":
-            return [WindowsLibtorchConfigNode(self, v) for v in WINDOWS_LIBTORCH_CONFIG_VARIANTS]
+            return [LinuxGccConfigNode(self, v) for v in DEVTOOLSET_VERSIONS]
         else:
             return [ArchConfigNode(self, v) for v in self.find_prop("cuda_versions")]
 
 
 class LinuxGccConfigNode(ConfigNode):
-    def __init__(self, parent, gcc_config_variant):
-        super(LinuxGccConfigNode, self).__init__(parent, "GCC_CONFIG_VARIANT=" + str(gcc_config_variant))
+    def __init__(self, parent, devtoolset_version):
+        super(LinuxGccConfigNode, self).__init__(parent, "DEVTOOLSET=" + str(devtoolset_version))
 
-        self.props["gcc_config_variant"] = gcc_config_variant
+        self.props["devtoolset_version"] = devtoolset_version
 
     def get_children(self):
         cuda_versions = self.find_prop("cuda_versions")
 
         # XXX devtoolset7 on CUDA 9.0 is temporarily disabled
         # see https://github.com/pytorch/pytorch/issues/20066
-        if self.find_prop("gcc_config_variant") == 'devtoolset7':
+        if self.find_prop("devtoolset_version") == 7:
             cuda_versions = filter(lambda x: x != "90", cuda_versions)
 
         return [ArchConfigNode(self, v) for v in cuda_versions]
 
 
-class WindowsLibtorchConfigNode(ConfigNode):
-    def __init__(self, parent, libtorch_config_variant):
-        super(WindowsLibtorchConfigNode, self).__init__(parent, "LIBTORCH_CONFIG_VARIANT=" + str(libtorch_config_variant))
-
-        self.props["libtorch_config_variant"] = libtorch_config_variant
-
-    def get_children(self):
-        return [ArchConfigNode(self, v) for v in self.find_prop("cuda_versions")]
-
-
 class ArchConfigNode(ConfigNode):
     def __init__(self, parent, cu):
         super(ArchConfigNode, self).__init__(parent, get_processor_arch_name(cu))
@@ -164,6 +152,8 @@ class PyVersionConfigNode(ConfigNode):
         self.props["pyver"] = pyver
 
     def get_children(self):
+
+        smoke = self.find_prop("smoke")
         package_format = self.find_prop("package_format")
         os_name = self.find_prop("os_name")
 

.circleci/cimodel/data/binary_build_definitions.py

@@ -1,12 +1,15 @@
+#!/usr/bin/env python3
+
 from collections import OrderedDict
 
-import cimodel.data.simple.util.branch_filters as branch_filters
 import cimodel.data.binary_build_data as binary_build_data
 import cimodel.lib.conf_tree as conf_tree
 import cimodel.lib.miniutils as miniutils
+import cimodel.lib.visualization as visualization
+
 
 class Conf(object):
-    def __init__(self, os, cuda_version, pydistro, parms, smoke, libtorch_variant, gcc_config_variant, libtorch_config_variant):
+    def __init__(self, os, cuda_version, pydistro, parms, smoke, libtorch_variant, devtoolset_version):
 
         self.os = os
         self.cuda_version = cuda_version
@@ -14,20 +17,15 @@ class Conf(object):
         self.parms = parms
         self.smoke = smoke
         self.libtorch_variant = libtorch_variant
-        self.gcc_config_variant = gcc_config_variant
-        self.libtorch_config_variant = libtorch_config_variant
+        self.devtoolset_version = devtoolset_version
 
     def gen_build_env_parms(self):
         elems = [self.pydistro] + self.parms + [binary_build_data.get_processor_arch_name(self.cuda_version)]
-        if self.gcc_config_variant is not None:
-            elems.append(str(self.gcc_config_variant))
-        if self.libtorch_config_variant is not None:
-            elems.append(str(self.libtorch_config_variant))
+        if self.devtoolset_version is not None:
+            elems.append("devtoolset" + str(self.devtoolset_version))
         return elems
 
     def gen_docker_image(self):
-        if self.gcc_config_variant == 'gcc5.4_cxx11-abi':
-            return miniutils.quote("pytorch/pytorch-binary-docker-image-ubuntu16.04:latest")
 
         docker_word_substitution = {
             "manywheel": "manylinux",
@@ -36,80 +34,56 @@ class Conf(object):
 
         docker_distro_prefix = miniutils.override(self.pydistro, docker_word_substitution)
 
-        # The cpu nightlies are built on the pytorch/manylinux-cuda102 docker image
-        alt_docker_suffix = self.cuda_version or "102"
+        # The cpu nightlies are built on the soumith/manylinux-cuda100 docker image
+        alt_docker_suffix = self.cuda_version or "100"
         docker_distro_suffix = "" if self.pydistro == "conda" else alt_docker_suffix
-        return miniutils.quote("pytorch/" + docker_distro_prefix + "-cuda" + docker_distro_suffix)
+        return miniutils.quote("soumith/" + docker_distro_prefix + "-cuda" + docker_distro_suffix)
 
     def get_name_prefix(self):
         return "smoke" if self.smoke else "binary"
 
-    def gen_build_name(self, build_or_test, nightly):
+    def gen_build_name(self, build_or_test):
 
         parts = [self.get_name_prefix(), self.os] + self.gen_build_env_parms()
 
-        if nightly:
-            parts.append("nightly")
-
         if self.libtorch_variant:
             parts.append(self.libtorch_variant)
 
         if not self.smoke:
             parts.append(build_or_test)
 
-        joined = "_".join(parts)
-        return joined.replace(".", "_")
+        return "_".join(parts)
+
+    def gen_yaml_tree(self, build_or_test):
+
+        env_tuples = [("BUILD_ENVIRONMENT", miniutils.quote(" ".join(self.gen_build_env_parms())))]
 
-    def gen_workflow_job(self, phase, upload_phase_dependency=None, nightly=False):
-        job_def = OrderedDict()
-        job_def["name"] = self.gen_build_name(phase, nightly)
-        job_def["build_environment"] = miniutils.quote(" ".join(self.gen_build_env_parms()))
-        if self.smoke:
-            job_def["requires"] = [
-                "update_s3_htmls",
-            ]
-            job_def["filters"] = branch_filters.gen_filter_dict(
-                branches_list=["nightly"],
-                tags_list=[branch_filters.RC_PATTERN],
-            )
-        else:
-            if phase in ["upload"]:
-                filter_branch = "nightly"
-            else:
-                filter_branch = r"/.*/"
-            job_def["filters"] = branch_filters.gen_filter_dict(
-                branches_list=[filter_branch],
-                tags_list=[branch_filters.RC_PATTERN],
-            )
         if self.libtorch_variant:
-            job_def["libtorch_variant"] = miniutils.quote(self.libtorch_variant)
-        if phase == "test":
-            if not self.smoke:
-                job_def["requires"] = [self.gen_build_name("build", nightly)]
-            if not (self.smoke and self.os == "macos") and self.os != "windows":
-                job_def["docker_image"] = self.gen_docker_image()
-
-            if self.os != "windows" and self.cuda_version:
-                job_def["use_cuda_docker_runtime"] = miniutils.quote("1")
-        else:
-            if self.os == "linux" and phase != "upload":
-                job_def["docker_image"] = self.gen_docker_image()
-
-        if phase == "test":
-            if self.cuda_version:
-                if self.os == "windows":
-                    job_def["executor"] = "windows-with-nvidia-gpu"
-                else:
-                    job_def["resource_class"] = "gpu.medium"
-        if phase == "upload":
-            job_def["context"] = "org-member"
-            job_def["requires"] = [
-                self.gen_build_name(upload_phase_dependency, nightly)
-            ]
+            env_tuples.append(("LIBTORCH_VARIANT", miniutils.quote(self.libtorch_variant)))
 
         os_name = miniutils.override(self.os, {"macos": "mac"})
-        job_name = "_".join([self.get_name_prefix(), os_name, phase])
-        return {job_name : job_def}
+        d = {"<<": "*" + "_".join([self.get_name_prefix(), os_name, build_or_test])}
+
+        if build_or_test == "test":
+
+            if not (self.smoke and self.os == "macos"):
+                env_tuples.append(("DOCKER_IMAGE", self.gen_docker_image()))
+
+            if self.cuda_version:
+                env_tuples.append(("USE_CUDA_DOCKER_RUNTIME", miniutils.quote("1")))
+
+        else:
+            if self.os == "linux" and build_or_test != "upload":
+                d["docker"] = [{"image": self.gen_docker_image()}]
+
+        d["environment"] = OrderedDict(env_tuples)
+
+        if build_or_test == "test":
+            if self.cuda_version:
+                d["resource_class"] = "gpu.medium"
+
+        return d
+
 
 def get_root(smoke, name):
 
@@ -134,77 +108,106 @@ def gen_build_env_list(smoke):
             [c.find_prop("pyver")],
             c.find_prop("smoke"),
             c.find_prop("libtorch_variant"),
-            c.find_prop("gcc_config_variant"),
-            c.find_prop("libtorch_config_variant"),
+            c.find_prop("devtoolset_version"),
         )
         newlist.append(conf)
 
     return newlist
 
-def predicate_exclude_macos(config):
-    return config.os == "linux" or config.os == "windows"
 
-def get_nightly_uploads():
-    configs = gen_build_env_list(False)
-    mylist = []
-    for conf in configs:
-        phase_dependency = "test" if predicate_exclude_macos(conf) else "build"
-        mylist.append(conf.gen_workflow_job("upload", phase_dependency, nightly=True))
+def predicate_exclude_nonlinux_and_libtorch(config):
+    return config.os == "linux" and (config.smoke or config.pydistro != "libtorch")
 
-    return mylist
 
-def get_post_upload_jobs():
-    """Generate jobs to update HTML indices and report binary sizes"""
-    configs = gen_build_env_list(False)
-    common_job_def = {
-        "context": "org-member",
-        "filters": branch_filters.gen_filter_dict(
-            branches_list=["nightly"],
-            tags_list=[branch_filters.RC_PATTERN],
-        ),
-        "requires": [],
-    }
-    for conf in configs:
-        upload_job_name = conf.gen_build_name(
-            build_or_test="upload",
-            nightly=True
-        )
-        common_job_def["requires"].append(upload_job_name)
-    return [
-        {
-            "update_s3_htmls": {
-                "name": "update_s3_htmls",
-                **common_job_def,
-            },
-        },
-    ]
+def add_build_entries(jobs_dict, phase, smoke, filter_predicate=lambda x: True):
+
+    configs = gen_build_env_list(smoke)
+    for conf_options in filter(filter_predicate, configs):
+        jobs_dict[conf_options.gen_build_name(phase)] = conf_options.gen_yaml_tree(phase)
+
+
+def add_binary_build_specs(jobs_dict):
+    add_build_entries(jobs_dict, "build", False)
+
+
+def add_binary_build_tests(jobs_dict):
+    add_build_entries(jobs_dict, "test", False, predicate_exclude_nonlinux_and_libtorch)
+
+
+def add_binary_build_uploads(jobs_dict):
+    add_build_entries(jobs_dict, "upload", False)
+
+
+def add_smoke_test_specs(jobs_dict):
+    add_build_entries(jobs_dict, "test", True)
+
 
 def get_nightly_tests():
 
     configs = gen_build_env_list(False)
-    filtered_configs = filter(predicate_exclude_macos, configs)
+    filtered_configs = filter(predicate_exclude_nonlinux_and_libtorch, configs)
 
     tests = []
     for conf_options in filtered_configs:
-        yaml_item = conf_options.gen_workflow_job("test", nightly=True)
-        tests.append(yaml_item)
+        params = {"requires": ["setup", conf_options.gen_build_name("build")]}
+        tests.append({conf_options.gen_build_name("test"): params})
 
     return tests
 
+def get_nightly_uploads():
+
+    configs = gen_build_env_list(False)
+
+    def gen_config(conf, phase_dependency):
+        return {
+            conf.gen_build_name("upload"): OrderedDict([
+                ("context", "org-member"),
+                ("requires", ["setup", conf.gen_build_name(phase_dependency)]),
+            ]),
+        }
+
+    mylist = []
+    for conf in configs:
+        phase_dependency = "test" if predicate_exclude_nonlinux_and_libtorch(conf) else "build"
+        mylist.append(gen_config(conf, phase_dependency))
+
+    return mylist
+
+
+def gen_schedule_tree(cron_timing):
+    return [{
+        "schedule": {
+            "cron": miniutils.quote(cron_timing),
+            "filters": {
+                "branches": {
+                    "only": ["master"],
+                },
+            },
+        },
+    }]
+
+
+def add_jobs_and_render(jobs_dict, toplevel_key, smoke, cron_schedule):
+
+    jobs_list = ["setup"]
 
-def get_jobs(toplevel_key, smoke):
-    jobs_list = []
     configs = gen_build_env_list(smoke)
-    phase = "build" if toplevel_key == "binarybuilds" else "test"
     for build_config in configs:
-        jobs_list.append(build_config.gen_workflow_job(phase, nightly=True))
+        build_name = build_config.gen_build_name("build")
+        jobs_list.append({build_name: {"requires": ["setup"]}})
 
-    return jobs_list
+    jobs_dict[toplevel_key] = OrderedDict(
+        triggers=gen_schedule_tree(cron_schedule),
+        jobs=jobs_list,
+    )
+
+    graph = visualization.generate_graph(get_root(smoke, toplevel_key))
+    graph.draw(toplevel_key + "-config-dimensions.png", prog="twopi")
 
 
-def get_binary_build_jobs():
-    return get_jobs("binarybuilds", False)
+def add_binary_build_jobs(jobs_dict):
+    add_jobs_and_render(jobs_dict, "binarybuilds", False, "5 5 * * *")
 
 
-def get_binary_smoke_test_jobs():
-    return get_jobs("binarysmoketests", True)
+def add_binary_smoke_test_jobs(jobs_dict):
+    add_jobs_and_render(jobs_dict, "binarysmoketests", True, "15 16 * * *")

.circleci/cimodel/data/caffe2_build_data.py

@@ -1,12 +1,38 @@
-from cimodel.lib.conf_tree import ConfigNode, XImportant
+#!/usr/bin/env python3
+
+from cimodel.lib.conf_tree import ConfigNode, X, XImportant
 from cimodel.lib.conf_tree import Ver
 
 
 CONFIG_TREE_DATA = [
+    (Ver("ubuntu", "14.04"), [
+        (Ver("gcc", "4.8"), [X("py2")]),
+        (Ver("gcc", "4.9"), [X("py2")]),
+    ]),
     (Ver("ubuntu", "16.04"), [
-        ([Ver("clang", "7")], [XImportant("onnx_main_py3.6"),
-                               XImportant("onnx_ort1_py3.6"),
-                               XImportant("onnx_ort2_py3.6")]),
+        (Ver("cuda", "9.0"), [
+            # TODO make explicit that this is a "secret TensorRT build"
+            #  (see https://github.com/pytorch/pytorch/pull/17323#discussion_r259446749)
+            # TODO Uh oh, were we supposed to make this one important?!
+            X("py2"),
+            XImportant("cmake"),
+        ]),
+        (Ver("cuda", "9.1"), [XImportant("py2")]),
+        (Ver("mkl"), [XImportant("py2")]),
+        (Ver("gcc", "5"), [XImportant("onnx_py2")]),
+        (Ver("clang", "3.8"), [X("py2")]),
+        (Ver("clang", "3.9"), [X("py2")]),
+        (Ver("clang", "7"), [XImportant("py2"), XImportant("onnx_py3.6")]),
+        (Ver("android"), [XImportant("py2")]),
+    ]),
+    (Ver("centos", "7"), [
+        (Ver("cuda", "9.0"), [X("py2")]),
+    ]),
+    (Ver("macos", "10.13"), [
+        # TODO ios and system aren't related. system qualifies where the python comes
+        #  from (use the system python instead of homebrew or anaconda)
+        (Ver("ios"), [X("py2")]),
+        (Ver("system"), [XImportant("py2")]),
     ]),
 ]
 
@@ -28,22 +54,15 @@ class TreeConfigNode(ConfigNode):
         return [self.child_constructor()(self, k, v) for (k, v) in self.subtree]
 
     def is_build_only(self):
-        if str(self.find_prop("language_version")) == "onnx_main_py3.6" or \
-                str(self.find_prop("language_version")) == "onnx_ort1_py3.6" or \
-                str(self.find_prop("language_version")) == "onnx_ort2_py3.6":
+        if str(self.find_prop("language_version")) == "onnx_py3.6":
             return False
-        return set(str(c) for c in self.find_prop("compiler_version")).intersection({
+        return str(self.find_prop("compiler_version")) in [
+            "gcc4.9",
             "clang3.8",
             "clang3.9",
             "clang7",
             "android",
-        }) or self.find_prop("distro_version").name == "macos"
-
-    def is_test_only(self):
-        if str(self.find_prop("language_version")) == "onnx_ort1_py3.6" or \
-                str(self.find_prop("language_version")) == "onnx_ort2_py3.6":
-            return True
-        return False
+        ] or self.find_prop("distro_version").name == "macos"
 
 
 class TopLevelNode(TreeConfigNode):
@@ -77,7 +96,6 @@ class LanguageConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["language_version"] = node_name
         self.props["build_only"] = self.is_build_only()
-        self.props["test_only"] = self.is_test_only()
 
     def child_constructor(self):
         return ImportantConfigNode

.circleci/cimodel/data/caffe2_build_definitions.py

@@ -1,42 +1,37 @@
+#!/usr/bin/env python3
+
 from collections import OrderedDict
 
 import cimodel.data.dimensions as dimensions
 import cimodel.lib.conf_tree as conf_tree
 from cimodel.lib.conf_tree import Ver
 import cimodel.lib.miniutils as miniutils
+import cimodel.lib.visualization as visualization
 from cimodel.data.caffe2_build_data import CONFIG_TREE_DATA, TopLevelNode
-from cimodel.data.simple.util.branch_filters import gen_filter_dict
+
 
 from dataclasses import dataclass
 
 
 DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/caffe2/"
 
-DOCKER_IMAGE_VERSION = "376"
+DOCKER_IMAGE_VERSION = 287
 
 
 @dataclass
 class Conf:
     language: str
     distro: Ver
-    # There could be multiple compiler versions configured (e.g. nvcc
-    # for gpu files and host compiler (gcc/clang) for cpu files)
-    compilers: [Ver]
+    compiler: Ver
     build_only: bool
-    test_only: bool
     is_important: bool
 
-    @property
-    def compiler_names(self):
-        return [c.name for c in self.compilers]
-
     # TODO: Eventually we can probably just remove the cudnn7 everywhere.
     def get_cudnn_insertion(self):
 
-        omit = self.language == "onnx_main_py3.6" \
-            or self.language == "onnx_ort1_py3.6" \
-            or self.language == "onnx_ort2_py3.6" \
-            or set(self.compiler_names).intersection({"android", "mkl", "clang"}) \
+        omit = self.language == "onnx_py2" \
+            or self.language == "onnx_py3.6" \
+            or self.compiler.name in ["android", "mkl", "clang"] \
             or str(self.distro) in ["ubuntu14.04", "macos10.13"]
 
         return [] if omit else ["cudnn7"]
@@ -48,17 +43,10 @@ class Conf:
         ] + self.get_build_name_middle_parts()
 
     def get_build_name_middle_parts(self):
-        return [str(c) for c in self.compilers] + self.get_cudnn_insertion() + [str(self.distro)]
+        return [str(self.compiler)] + self.get_cudnn_insertion() + [str(self.distro)]
 
     def construct_phase_name(self, phase):
         root_parts = self.get_build_name_root_parts()
-
-        build_name_substitutions = {
-            "onnx_ort1_py3.6": "onnx_main_py3.6",
-            "onnx_ort2_py3.6": "onnx_main_py3.6",
-        }
-        if phase == "build":
-            root_parts = [miniutils.override(r, build_name_substitutions) for r in root_parts]
         return "_".join(root_parts + [phase]).replace(".", "_")
 
     def get_platform(self):
@@ -70,23 +58,22 @@ class Conf:
     def gen_docker_image(self):
 
         lang_substitutions = {
-            "onnx_main_py3.6": "py3.6",
-            "onnx_ort1_py3.6": "py3.6",
-            "onnx_ort2_py3.6": "py3.6",
-            "cmake": "py3",
+            "onnx_py2": "py2",
+            "onnx_py3.6": "py3.6",
+            "cmake": "py2",
         }
 
         lang = miniutils.override(self.language, lang_substitutions)
         parts = [lang] + self.get_build_name_middle_parts()
         return miniutils.quote(DOCKER_IMAGE_PATH_BASE + "-".join(parts) + ":" + str(DOCKER_IMAGE_VERSION))
 
-    def gen_workflow_params(self, phase):
-        parameters = OrderedDict()
+    def gen_yaml_tree(self, phase):
+
+        tuples = []
+
         lang_substitutions = {
-            "onnx_py3": "onnx-py3",
-            "onnx_main_py3.6": "onnx-main-py3.6",
-            "onnx_ort1_py3.6": "onnx-ort1-py3.6",
-            "onnx_ort2_py3.6": "onnx-ort2-py3.6",
+            "onnx_py2": "onnx-py2",
+            "onnx_py3.6": "onnx-py3.6",
         }
 
         lang = miniutils.override(self.language, lang_substitutions)
@@ -96,39 +83,37 @@ class Conf:
             lang,
         ] + self.get_build_name_middle_parts() + [phase]
 
-        build_env_name = "-".join(parts)
-        parameters["build_environment"] = miniutils.quote(build_env_name)
-        if "ios" in self.compiler_names:
-            parameters["build_ios"] = miniutils.quote("1")
+        build_env = "-".join(parts)
+        if not self.distro.name == "macos":
+            build_env = miniutils.quote(build_env)
+
+        tuples.append(("BUILD_ENVIRONMENT", build_env))
+
+        if self.compiler.name == "ios":
+            tuples.append(("BUILD_IOS", miniutils.quote("1")))
+
         if phase == "test":
             # TODO cuda should not be considered a compiler
-            if "cuda" in self.compiler_names:
-                parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
+            if self.compiler.name == "cuda":
+                tuples.append(("USE_CUDA_DOCKER_RUNTIME", miniutils.quote("1")))
 
-        if self.distro.name != "macos":
-            parameters["docker_image"] = self.gen_docker_image()
-            if self.build_only:
-                parameters["build_only"] = miniutils.quote("1")
-        if phase == "test":
-            resource_class = "large" if "cuda" not in self.compiler_names else "gpu.medium"
-            parameters["resource_class"] = resource_class
+        if self.distro.name == "macos":
+            tuples.append(("PYTHON_VERSION", miniutils.quote("2")))
 
-        return parameters
-
-    def gen_workflow_job(self, phase):
-        job_def = OrderedDict()
-        job_def["name"] = self.construct_phase_name(phase)
-
-        if phase == "test":
-            job_def["requires"] = [self.construct_phase_name("build")]
-            job_name = "caffe2_" + self.get_platform() + "_test"
         else:
-            job_name = "caffe2_" + self.get_platform() + "_build"
+            tuples.append(("DOCKER_IMAGE", self.gen_docker_image()))
+            if self.build_only:
+                tuples.append(("BUILD_ONLY", miniutils.quote("1")))
 
-        if not self.is_important:
-            job_def["filters"] = gen_filter_dict()
-        job_def.update(self.gen_workflow_params(phase))
-        return {job_name : job_def}
+        d = OrderedDict({"environment": OrderedDict(tuples)})
+
+        if phase == "test":
+            resource_class = "large" if self.compiler.name != "cuda" else "gpu.medium"
+            d["resource_class"] = resource_class
+
+        d["<<"] = "*" + "_".join(["caffe2", self.get_platform(), phase, "defaults"])
+
+        return d
 
 
 def get_root():
@@ -142,12 +127,12 @@ def instantiate_configs():
     root = get_root()
     found_configs = conf_tree.dfs(root)
     for fc in found_configs:
+
         c = Conf(
             language=fc.find_prop("language_version"),
             distro=fc.find_prop("distro_version"),
-            compilers=fc.find_prop("compiler_version"),
+            compiler=fc.find_prop("compiler_version"),
             build_only=fc.find_prop("build_only"),
-            test_only=fc.find_prop("test_only"),
             is_important=fc.find_prop("important"),
         )
 
@@ -156,19 +141,47 @@ def instantiate_configs():
     return config_list
 
 
-def get_workflow_jobs():
-
+def add_caffe2_builds(jobs_dict):
     configs = instantiate_configs()
-
-    x = []
     for conf_options in configs:
         phases = ["build"]
         if not conf_options.build_only:
             phases = dimensions.PHASES
-        if conf_options.test_only:
-            phases = ["test"]
+        for phase in phases:
+            jobs_dict[conf_options.construct_phase_name(phase)] = conf_options.gen_yaml_tree(phase)
+
+    graph = visualization.generate_graph(get_root())
+    graph.draw("caffe2-config-dimensions.png", prog="twopi")
+
+
+def get_caffe2_workflows():
+
+    configs = instantiate_configs()
+
+    # TODO Why don't we build this config?
+    # See https://github.com/pytorch/pytorch/pull/17323#discussion_r259450540
+    filtered_configs = filter(lambda x: not (str(x.distro) == "ubuntu14.04" and str(x.compiler) == "gcc4.9"), configs)
+
+    x = []
+    for conf_options in filtered_configs:
+
+        phases = ["build"]
+        if not conf_options.build_only:
+            phases = dimensions.PHASES
 
         for phase in phases:
-            x.append(conf_options.gen_workflow_job(phase))
+
+            requires = ["setup"]
+            sub_d = {"requires": requires}
+
+            if phase == "test":
+                requires.append(conf_options.construct_phase_name("build"))
+
+            if not conf_options.is_important:
+                # If you update this, update
+                # pytorch_build_definitions.py too
+                sub_d["filters"] = {"branches": {"only": ["master", r"/ci-all\/.*/"]}}
+
+            x.append({conf_options.construct_phase_name(phase): sub_d})
 
     return x

.circleci/cimodel/data/dimensions.py

@@ -1,14 +1,23 @@
+#!/usr/bin/env python3
+
+
 PHASES = ["build", "test"]
 
 CUDA_VERSIONS = [
     None,  # cpu build
     "92",
-    "101",
-    "102",
+    "100",
 ]
 
 STANDARD_PYTHON_VERSIONS = [
+    "2.7",
+    "3.5",
+    "3.6",
+    "3.7",
+]
+
+CONDA_PYTHON_VERSIONS = [
+    "2.7",
     "3.6",
     "3.7",
-    "3.8"
 ]

.circleci/cimodel/data/pytorch_build_data.py

@@ -1,56 +1,57 @@
+#!/usr/bin/env python3
+
 from cimodel.lib.conf_tree import ConfigNode, X, XImportant
 
 
 CONFIG_TREE_DATA = [
-    ("xenial", [
+    ("trusty", [
         (None, [
+            XImportant("2.7.9"),
+            X("2.7"),
+            X("3.5"),
             X("nightly"),
         ]),
         ("gcc", [
-            ("5.4", [  # All this subtree rebases to master and then build
+            ("4.8", [X("3.6")]),
+            ("5.4", [
                 XImportant("3.6"),
                 ("3.6", [
-                    ("parallel_tbb", [X(True)]),
-                    ("parallel_native", [X(True)]),
+                    ("xla", [XImportant(True)]),
+                    ("namedtensor", [XImportant(True)]),
                 ]),
             ]),
-            # TODO: bring back libtorch test
             ("7", [X("3.6")]),
         ]),
+    ]),
+    ("xenial", [
         ("clang", [
             ("5", [
                 XImportant("3.6"),  # This is actually the ASAN build
+                ("3.6", [
+                    ("namedtensor", [XImportant(True)]),  # ASAN
+                ]),
             ]),
         ]),
         ("cuda", [
-            ("9.2", [
-                X("3.6"),
-                ("3.6", [
-                    ("cuda_gcc_override", [X("gcc5.4")])
-                ])
-            ]),
-            ("10.1", [X("3.6")]),
-            ("10.2", [
+            ("9", [
+                # Note there are magic strings here
+                # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L21
+                # and
+                # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L143
+                # and
+                # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L153
+                # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453144)
+                X("2.7"),
                 XImportant("3.6"),
-                ("3.6", [
-                    ("libtorch", [XImportant(True)])
+                ("2.7", [
+                    ("namedtensor", [XImportant(True)]),
                 ]),
             ]),
+            ("9.2", [X("3.6")]),
+            ("10", [X("3.6")]),
         ]),
-    ]),
-    ("bionic", [
-        ("clang", [
-            ("9", [
-                XImportant("3.6"),
-            ]),
-            ("9", [
-                ("3.6", [
-                    ("xla", [XImportant(True)]),
-                ]),
-            ]),
-        ]),
-        ("gcc", [
-            ("9", [XImportant("3.8")]),
+        ("android", [
+            ("r19c", [XImportant("3.6")]),
         ]),
     ]),
 ]
@@ -94,12 +95,34 @@ class DistroConfigNode(TreeConfigNode):
         distro = self.find_prop("distro_name")
 
         next_nodes = {
+            "trusty": TrustyCompilerConfigNode,
             "xenial": XenialCompilerConfigNode,
-            "bionic": BionicCompilerConfigNode,
         }
         return next_nodes[distro]
 
 
+class TrustyCompilerConfigNode(TreeConfigNode):
+
+    def modify_label(self, label):
+        return label or "<unspecified>"
+
+    def init2(self, node_name):
+        self.props["compiler_name"] = node_name
+
+    def child_constructor(self):
+        return TrustyCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
+
+
+class TrustyCompilerVersionConfigNode(TreeConfigNode):
+
+    def init2(self, node_name):
+        self.props["compiler_version"] = node_name
+
+    # noinspection PyMethodMayBeStatic
+    def child_constructor(self):
+        return PyVerConfigNode
+
+
 class PyVerConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["pyver"] = node_name
@@ -119,12 +142,8 @@ class ExperimentalFeatureConfigNode(TreeConfigNode):
 
         next_nodes = {
             "xla": XlaConfigNode,
-            "parallel_tbb": ParallelTBBConfigNode,
-            "parallel_native": ParallelNativeConfigNode,
-            "libtorch": LibTorchConfigNode,
+            "namedtensor": NamedTensorConfigNode,
             "important": ImportantConfigNode,
-            "build_only": BuildOnlyConfigNode,
-            "cuda_gcc_override": CudaGccOverrideConfigNode
         }
         return next_nodes[experimental_feature]
 
@@ -140,50 +159,12 @@ class XlaConfigNode(TreeConfigNode):
         return ImportantConfigNode
 
 
-class ParallelTBBConfigNode(TreeConfigNode):
+class NamedTensorConfigNode(TreeConfigNode):
     def modify_label(self, label):
-        return "PARALLELTBB=" + str(label)
+        return "NAMEDTENSOR=" + str(label)
 
     def init2(self, node_name):
-        self.props["parallel_backend"] = "paralleltbb"
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class ParallelNativeConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "PARALLELNATIVE=" + str(label)
-
-    def init2(self, node_name):
-        self.props["parallel_backend"] = "parallelnative"
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class LibTorchConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "BUILD_TEST_LIBTORCH=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_libtorch"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class CudaGccOverrideConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["cuda_gcc_override"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-class BuildOnlyConfigNode(TreeConfigNode):
-
-    def init2(self, node_name):
-        self.props["build_only"] = node_name
+        self.props["is_namedtensor"] = node_name
 
     def child_constructor(self):
         return ImportantConfigNode
@@ -202,30 +183,12 @@ class ImportantConfigNode(TreeConfigNode):
 
 class XenialCompilerConfigNode(TreeConfigNode):
 
-    def modify_label(self, label):
-        return label or "<unspecified>"
-
     def init2(self, node_name):
         self.props["compiler_name"] = node_name
 
     # noinspection PyMethodMayBeStatic
     def child_constructor(self):
-
-        return XenialCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
-
-
-class BionicCompilerConfigNode(TreeConfigNode):
-
-    def modify_label(self, label):
-        return label or "<unspecified>"
-
-    def init2(self, node_name):
-        self.props["compiler_name"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-
-        return BionicCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
+        return XenialCompilerVersionConfigNode
 
 
 class XenialCompilerVersionConfigNode(TreeConfigNode):
@@ -235,12 +198,3 @@ class XenialCompilerVersionConfigNode(TreeConfigNode):
     # noinspection PyMethodMayBeStatic
     def child_constructor(self):
         return PyVerConfigNode
-
-
-class BionicCompilerVersionConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["compiler_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return PyVerConfigNode

.circleci/cimodel/data/pytorch_build_definitions.py

@@ -1,35 +1,38 @@
+#!/usr/bin/env python3
+
 from collections import OrderedDict
 
 from cimodel.data.pytorch_build_data import TopLevelNode, CONFIG_TREE_DATA
 import cimodel.data.dimensions as dimensions
 import cimodel.lib.conf_tree as conf_tree
 import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.branch_filters import gen_filter_dict
-from cimodel.data.simple.util.docker_constants import gen_docker_image_path
+import cimodel.lib.visualization as visualization
 
 from dataclasses import dataclass, field
 from typing import List, Optional
 
 
+DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/"
+
+DOCKER_IMAGE_VERSION = 323
+
+
 @dataclass
 class Conf:
     distro: str
     parms: List[str]
-    parms_list_ignored_for_docker_image: Optional[List[str]] = None
     pyver: Optional[str] = None
     cuda_version: Optional[str] = None
     # TODO expand this to cover all the USE_* that we want to test for
     #  tesnrorrt, leveldb, lmdb, redis, opencv, mkldnn, ideep, etc.
     # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453608)
     is_xla: bool = False
-    vulkan: bool = False
     restrict_phases: Optional[List[str]] = None
     gpu_resource: Optional[str] = None
     dependent_tests: List = field(default_factory=list)
     parent_build: Optional['Conf'] = None
-    is_libtorch: bool = False
+    is_namedtensor: bool = False
     is_important: bool = False
-    parallel_backend: Optional[str] = None
 
     # TODO: Eliminate the special casing for docker paths
     # In the short term, we *will* need to support special casing as docker images are merged for caffe2 and pytorch
@@ -42,25 +45,20 @@ class Conf:
         leading.append("pytorch")
         if self.is_xla and not for_docker:
             leading.append("xla")
-        if self.is_libtorch and not for_docker:
-            leading.append("libtorch")
-        if self.parallel_backend is not None and not for_docker:
-            leading.append(self.parallel_backend)
+        if self.is_namedtensor and not for_docker:
+            leading.append("namedtensor")
 
         cuda_parms = []
         if self.cuda_version:
             cuda_parms.extend(["cuda" + self.cuda_version, "cudnn7"])
-        result = leading + ["linux", self.distro] + cuda_parms + self.parms
-        if not for_docker and self.parms_list_ignored_for_docker_image is not None:
-            result = result + self.parms_list_ignored_for_docker_image
-        return result
+        return leading + ["linux", self.distro] + cuda_parms + self.parms
 
     def gen_docker_image_path(self):
 
         parms_source = self.parent_build or self
         base_build_env_name = "-".join(parms_source.get_parms(True))
 
-        return miniutils.quote(gen_docker_image_path(base_build_env_name))
+        return miniutils.quote(DOCKER_IMAGE_PATH_BASE + base_build_env_name + ":" + str(DOCKER_IMAGE_VERSION))
 
     def get_build_job_name_pieces(self, build_or_test):
         return self.get_parms(False) + [build_or_test]
@@ -71,25 +69,44 @@ class Conf:
     def get_dependents(self):
         return self.dependent_tests or []
 
-    def gen_workflow_params(self, phase):
-        parameters = OrderedDict()
-        build_job_name_pieces = self.get_build_job_name_pieces(phase)
+    def gen_yaml_tree(self, build_or_test):
+
+        build_job_name_pieces = self.get_build_job_name_pieces(build_or_test)
 
         build_env_name = "-".join(map(str, build_job_name_pieces))
-        parameters["build_environment"] = miniutils.quote(build_env_name)
-        parameters["docker_image"] = self.gen_docker_image_path()
-        if phase == "test" and self.gpu_resource:
-            parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
-        if phase == "test":
+
+        env_dict = OrderedDict([
+            ("BUILD_ENVIRONMENT", build_env_name),
+            ("DOCKER_IMAGE", self.gen_docker_image_path()),
+        ])
+
+        if self.pyver:
+            env_dict["PYTHON_VERSION"] = miniutils.quote(self.pyver)
+
+        if build_or_test == "test" and self.gpu_resource:
+            env_dict["USE_CUDA_DOCKER_RUNTIME"] = miniutils.quote("1")
+
+        d = {
+            "environment": env_dict,
+            "<<": "*" + "_".join(["pytorch", "linux", build_or_test, "defaults"]),
+        }
+
+        if build_or_test == "test":
             resource_class = "large"
             if self.gpu_resource:
                 resource_class = "gpu." + self.gpu_resource
-            parameters["resource_class"] = resource_class
-        return parameters
 
-    def gen_workflow_job(self, phase):
-        job_def = OrderedDict()
-        job_def["name"] = self.gen_build_name(phase)
+                if self.gpu_resource == "large":
+                    env_dict["MULTI_GPU"] = miniutils.quote("1")
+
+            d["resource_class"] = resource_class
+
+        return d
+
+    def gen_workflow_yaml_item(self, phase):
+
+        # All jobs require the setup job
+        parameters = OrderedDict({"requires": ["setup"]})
 
         if phase == "test":
 
@@ -99,16 +116,14 @@ class Conf:
             #  pytorch build job (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259452641)
 
             dependency_build = self.parent_build or self
-            job_def["requires"] = [dependency_build.gen_build_name("build")]
-            job_name = "pytorch_linux_test"
-        else:
-            job_name = "pytorch_linux_build"
+            parameters["requires"].append(dependency_build.gen_build_name("build"))
 
         if not self.is_important:
-            job_def["filters"] = gen_filter_dict()
-        job_def.update(self.gen_workflow_params(phase))
+            # If you update this, update
+            # caffe2_build_definitions.py too
+            parameters["filters"] = {"branches": {"only": ["master", r"/ci-all\/.*/"]}}
 
-        return {job_name : job_def}
+        return {self.gen_build_name(phase): parameters}
 
 
 # TODO This is a hack to special case some configs just for the workflow list
@@ -117,7 +132,8 @@ class HiddenConf(object):
         self.name = name
         self.parent_build = parent_build
 
-    def gen_workflow_job(self, phase):
+    def gen_workflow_yaml_item(self, phase):
+
         return {self.gen_build_name(phase): {"requires": [self.parent_build.gen_build_name("build")]}}
 
     def gen_build_name(self, _):
@@ -151,13 +167,7 @@ def gen_dependent_configs(xenial_parent_config):
 
         configs.append(c)
 
-    return configs
-
-
-def gen_docs_configs(xenial_parent_config):
-    configs = []
-
-    for x in ["pytorch_python_doc_push", "pytorch_cpp_doc_push", "pytorch_doc_test"]:
+    for x in ["pytorch_short_perf_test_gpu", "pytorch_python_doc_push", "pytorch_cpp_doc_push"]:
         configs.append(HiddenConf(x, parent_build=xenial_parent_config))
 
     return configs
@@ -179,26 +189,20 @@ def instantiate_configs():
 
     root = get_root()
     found_configs = conf_tree.dfs(root)
+    restrict_phases = None
     for fc in found_configs:
 
-        restrict_phases = None
         distro_name = fc.find_prop("distro_name")
-        compiler_name = fc.find_prop("compiler_name")
-        compiler_version = fc.find_prop("compiler_version")
-        is_xla = fc.find_prop("is_xla") or False
-        parms_list_ignored_for_docker_image = []
-
-        vulkan = fc.find_prop("vulkan") or False
-        if vulkan:
-            parms_list_ignored_for_docker_image.append("vulkan")
 
         python_version = None
-        if compiler_name == "cuda" or compiler_name == "android":
+        if distro_name == "xenial":
             python_version = fc.find_prop("pyver")
             parms_list = [fc.find_prop("abbreviated_pyver")]
         else:
             parms_list = ["py" + fc.find_prop("pyver")]
 
+        compiler_name = fc.find_prop("compiler_name")
+
         cuda_version = None
         if compiler_name == "cuda":
             cuda_version = fc.find_prop("compiler_version")
@@ -208,8 +212,6 @@ def instantiate_configs():
             # TODO: do we need clang to compile host binaries like protoc?
             parms_list.append("clang5")
             parms_list.append("android-ndk-" + android_ndk_version)
-            android_abi = fc.find_prop("android_abi")
-            parms_list_ignored_for_docker_image.append(android_abi)
             restrict_phases = ["build"]
 
         elif compiler_name:
@@ -217,22 +219,16 @@ def instantiate_configs():
             parms_list.append(gcc_version)
 
             # TODO: This is a nasty special case
-            if gcc_version == 'clang5' and not is_xla:
+            if compiler_name == "clang":
                 parms_list.append("asan")
-                python_version = fc.find_prop("pyver")
-                parms_list[0] = fc.find_prop("abbreviated_pyver")
 
-        if cuda_version in ["9.2", "10", "10.1", "10.2"]:
+        if cuda_version in ["9.2", "10"]:
             # TODO The gcc version is orthogonal to CUDA version?
-            cuda_gcc_version = fc.find_prop("cuda_gcc_override") or "gcc7"
-            parms_list.append(cuda_gcc_version)
+            parms_list.append("gcc7")
 
-        is_libtorch = fc.find_prop("is_libtorch") or False
+        is_xla = fc.find_prop("is_xla") or False
+        is_namedtensor = fc.find_prop("is_namedtensor") or False
         is_important = fc.find_prop("is_important") or False
-        parallel_backend = fc.find_prop("parallel_backend") or None
-        build_only = fc.find_prop("build_only") or False
-        if build_only and restrict_phases is None:
-            restrict_phases = ["build"]
 
         gpu_resource = None
         if cuda_version and cuda_version != "10":
@@ -241,55 +237,60 @@ def instantiate_configs():
         c = Conf(
             distro_name,
             parms_list,
-            parms_list_ignored_for_docker_image,
             python_version,
             cuda_version,
             is_xla,
-            vulkan,
             restrict_phases,
             gpu_resource,
-            is_libtorch=is_libtorch,
+            is_namedtensor=is_namedtensor,
             is_important=is_important,
-            parallel_backend=parallel_backend,
         )
 
-        # run docs builds on "pytorch-linux-xenial-py3.6-gcc5.4". Docs builds
-        # should run on a CPU-only build that runs on all PRs.
-        if distro_name == 'xenial' and fc.find_prop("pyver") == '3.6' \
-                and cuda_version is None \
-                and parallel_backend is None \
-                and compiler_name == 'gcc' \
-                and fc.find_prop('compiler_version') == '5.4':
-            c.dependent_tests = gen_docs_configs(c)
-
-        if cuda_version == "10.1" and python_version == "3.6" and not is_libtorch:
+        if cuda_version == "9" and python_version == "3.6":
             c.dependent_tests = gen_dependent_configs(c)
 
-        if (compiler_name == "gcc"
-                and compiler_version == "5.4"
-                and not is_libtorch
-                and parallel_backend is None):
-            bc_breaking_check = Conf(
-                "backward-compatibility-check",
-                [],
-                is_xla=False,
-                restrict_phases=["test"],
-                is_libtorch=False,
-                is_important=True,
-                parent_build=c,
-            )
-            c.dependent_tests.append(bc_breaking_check)
-
         config_list.append(c)
 
     return config_list
 
 
-def get_workflow_jobs():
+def add_build_env_defs(jobs_dict):
+
+    mydict = OrderedDict()
+
+    config_list = instantiate_configs()
+    for c in config_list:
+
+        phases = c.restrict_phases or dimensions.PHASES
+
+        for phase in phases:
+
+            # TODO why does this not have a test?
+            if phase == "test" and c.cuda_version == "10":
+                continue
+
+            d = c.gen_yaml_tree(phase)
+            mydict[c.gen_build_name(phase)] = d
+
+            if phase == "test":
+                for x in filter(lambda x: type(x) is not HiddenConf, c.get_dependents()):
+
+                    d = x.gen_yaml_tree(phase)
+                    mydict[x.gen_build_name(phase)] = d
+
+    # this is the circleci api version and probably never changes
+    jobs_dict["version"] = 2
+    jobs_dict["jobs"] = mydict
+
+    graph = visualization.generate_graph(get_root())
+    graph.draw("pytorch-config-dimensions.png", prog="twopi")
+
+
+def get_workflow_list():
 
     config_list = instantiate_configs()
 
-    x = []
+    x = ["setup"]
     for conf_options in config_list:
 
         phases = conf_options.restrict_phases or dimensions.PHASES
@@ -300,10 +301,10 @@ def get_workflow_jobs():
             if phase == "test" and conf_options.cuda_version == "10":
                 continue
 
-            x.append(conf_options.gen_workflow_job(phase))
+            x.append(conf_options.gen_workflow_yaml_item(phase))
 
         # TODO convert to recursion
         for conf in conf_options.get_dependents():
-            x.append(conf.gen_workflow_job("test"))
+            x.append(conf.gen_workflow_yaml_item("test"))
 
     return x

.circleci/cimodel/data/simple/android_definitions.py

@@ -1,92 +0,0 @@
-import cimodel.data.simple.util.branch_filters
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_NDK
-
-
-class AndroidJob:
-    def __init__(self,
-                 variant,
-                 template_name,
-                 is_master_only=True):
-
-        self.variant = variant
-        self.template_name = template_name
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-
-        base_name_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "android",
-            "ndk",
-            "r19c",
-        ] + self.variant + [
-            "build",
-        ]
-
-        full_job_name = "_".join(base_name_parts)
-        build_env_name = "-".join(base_name_parts)
-
-        props_dict = {
-            "name": full_job_name,
-            "build_environment": "\"{}\"".format(build_env_name),
-            "docker_image": "\"{}\"".format(DOCKER_IMAGE_NDK),
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{self.template_name: props_dict}]
-
-
-class AndroidGradleJob:
-    def __init__(self,
-                 job_name,
-                 template_name,
-                 dependencies,
-                 is_master_only=True):
-
-        self.job_name = job_name
-        self.template_name = template_name
-        self.dependencies = dependencies
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-
-        props_dict = {
-            "name": self.job_name,
-            "requires": self.dependencies,
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    AndroidJob(["x86_32"], "pytorch_linux_build", is_master_only=False),
-    AndroidJob(["x86_64"], "pytorch_linux_build"),
-    AndroidJob(["arm", "v7a"], "pytorch_linux_build"),
-    AndroidJob(["arm", "v8a"], "pytorch_linux_build"),
-    AndroidJob(["vulkan", "x86_32"], "pytorch_linux_build", is_master_only=False),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32",
-        "pytorch_android_gradle_build-x86_32",
-        ["pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build"],
-        is_master_only=False),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build",
-        "pytorch_android_gradle_build",
-        ["pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build"]),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/bazel_definitions.py

@@ -1,63 +0,0 @@
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_GCC7
-
-
-def gen_job_name(phase):
-    job_name_parts = [
-        "pytorch",
-        "bazel",
-        phase,
-    ]
-
-    return "_".join(job_name_parts)
-
-
-class BazelJob:
-    def __init__(self, phase, extra_props=None):
-        self.phase = phase
-        self.extra_props = extra_props or {}
-
-    def gen_tree(self):
-
-        template_parts = [
-            "pytorch",
-            "linux",
-            "bazel",
-            self.phase,
-        ]
-
-        build_env_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3.6",
-            "gcc7",
-            "bazel",
-            self.phase,
-        ]
-
-        full_job_name = gen_job_name(self.phase)
-        build_env_name = "-".join(build_env_parts)
-
-        extra_requires = [gen_job_name("build")] if self.phase == "test" else []
-
-        props_dict = {
-            "build_environment": build_env_name,
-            "docker_image": DOCKER_IMAGE_GCC7,
-            "name": full_job_name,
-            "requires": extra_requires,
-        }
-
-        props_dict.update(self.extra_props)
-
-        template_name = "_".join(template_parts)
-        return [{template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    BazelJob("build", {"resource_class": "large"}),
-    BazelJob("test"),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/binary_smoketest.py

@@ -1,193 +0,0 @@
-"""
-TODO: Refactor circleci/cimodel/data/binary_build_data.py to generate this file
-       instead of doing one offs here
- Binary builds (subset, to smoke test that they'll work)
-
- NB: If you modify this file, you need to also modify
- the binary_and_smoke_tests_on_pr variable in
- pytorch-ci-hud to adjust the list of whitelisted builds
- at https://github.com/ezyang/pytorch-ci-hud/blob/master/src/BuildHistoryDisplay.js
-
- Note:
- This binary build is currently broken, see https://github_com/pytorch/pytorch/issues/16710
- - binary_linux_conda_3_6_cu90_devtoolset7_build
- - binary_linux_conda_3_6_cu90_devtoolset7_test
-
- TODO
- we should test a libtorch cuda build, but they take too long
- - binary_linux_libtorch_3_6m_cu90_devtoolset7_static-without-deps_build
-"""
-
-import cimodel.lib.miniutils as miniutils
-import cimodel.data.simple.util.branch_filters
-
-
-class SmoketestJob:
-    def __init__(self,
-                 template_name,
-                 build_env_parts,
-                 docker_image,
-                 job_name,
-                 is_master_only=False,
-                 requires=None,
-                 has_libtorch_variant=False,
-                 extra_props=None):
-
-        self.template_name = template_name
-        self.build_env_parts = build_env_parts
-        self.docker_image = docker_image
-        self.job_name = job_name
-        self.is_master_only = is_master_only
-        self.requires = requires or []
-        self.has_libtorch_variant = has_libtorch_variant
-        self.extra_props = extra_props or {}
-
-    def gen_tree(self):
-
-        props_dict = {
-            "build_environment": " ".join(self.build_env_parts),
-            "name": self.job_name,
-            "requires": self.requires,
-        }
-
-        if self.docker_image:
-            props_dict["docker_image"] = self.docker_image
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        if self.has_libtorch_variant:
-            props_dict["libtorch_variant"] = "shared-with-deps"
-
-        props_dict.update(self.extra_props)
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    SmoketestJob(
-        "binary_linux_build",
-        ["manywheel", "3.7m", "cu102", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_manywheel_3_7m_cu102_devtoolset7_build",
-        is_master_only=True,
-    ),
-    SmoketestJob(
-        "binary_linux_build",
-        ["libtorch", "3.7m", "cpu", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build",
-        is_master_only=False,
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_linux_build",
-        ["libtorch", "3.7m", "cpu", "gcc5.4_cxx11-abi"],
-        "pytorch/pytorch-binary-docker-image-ubuntu16.04:latest",
-        "binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build",
-        is_master_only=False,
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_mac_build",
-        ["wheel", "3.7", "cpu"],
-        None,
-        "binary_macos_wheel_3_7_cpu_build",
-        is_master_only=True,
-    ),
-    # This job has an average run time of 3 hours o.O
-    # Now only running this on master to reduce overhead
-    SmoketestJob(
-        "binary_mac_build",
-        ["libtorch", "3.7", "cpu"],
-        None,
-        "binary_macos_libtorch_3_7_cpu_build",
-        is_master_only=True,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["libtorch", "3.7", "cpu", "debug"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_debug_build",
-        is_master_only=False,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["libtorch", "3.7", "cpu", "release"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_release_build",
-        is_master_only=False,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["wheel", "3.7", "cu102"],
-        None,
-        "binary_windows_wheel_3_7_cu102_build",
-        is_master_only=True,
-    ),
-
-    SmoketestJob(
-        "binary_windows_test",
-        ["libtorch", "3.7", "cpu", "debug"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_debug_test",
-        is_master_only=False,
-        requires=["binary_windows_libtorch_3_7_cpu_debug_build"],
-    ),
-    SmoketestJob(
-        "binary_windows_test",
-        ["libtorch", "3.7", "cpu", "release"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_release_test",
-        is_master_only=False,
-        requires=["binary_windows_libtorch_3_7_cpu_release_build"],
-    ),
-    SmoketestJob(
-        "binary_windows_test",
-        ["wheel", "3.7", "cu102"],
-        None,
-        "binary_windows_wheel_3_7_cu102_test",
-        is_master_only=True,
-        requires=["binary_windows_wheel_3_7_cu102_build"],
-        extra_props={
-            "executor": "windows-with-nvidia-gpu",
-        },
-    ),
-
-
-
-    SmoketestJob(
-        "binary_linux_test",
-        ["manywheel", "3.7m", "cu102", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_manywheel_3_7m_cu102_devtoolset7_test",
-        is_master_only=True,
-        requires=["binary_linux_manywheel_3_7m_cu102_devtoolset7_build"],
-        extra_props={
-            "resource_class": "gpu.medium",
-            "use_cuda_docker_runtime": miniutils.quote((str(1))),
-        },
-    ),
-    SmoketestJob(
-        "binary_linux_test",
-        ["libtorch", "3.7m", "cpu", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_test",
-        is_master_only=False,
-        requires=["binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build"],
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_linux_test",
-        ["libtorch", "3.7m", "cpu", "gcc5.4_cxx11-abi"],
-        "pytorch/pytorch-binary-docker-image-ubuntu16.04:latest",
-        "binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_test",
-        is_master_only=False,
-        requires=["binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build"],
-        has_libtorch_variant=True,
-    ),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/docker_definitions.py

@@ -1,44 +0,0 @@
-from collections import OrderedDict
-
-from cimodel.lib.miniutils import quote
-
-
-# TODO: make this generated from a matrix rather than just a static list
-IMAGE_NAMES = [
-    "pytorch-linux-bionic-cuda11.0-cudnn8-py3.6-gcc9",
-    "pytorch-linux-bionic-cuda11.0-cudnn8-py3.8-gcc9",
-    "pytorch-linux-bionic-cuda10.2-cudnn7-py3.8-gcc9",
-    "pytorch-linux-bionic-py3.6-clang9",
-    "pytorch-linux-bionic-cuda10.2-cudnn7-py3.6-clang9",
-    "pytorch-linux-bionic-py3.8-gcc9",
-    "pytorch-linux-xenial-cuda10-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda10.1-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda11.0-cudnn8-py3-gcc7",
-    "pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc5.4",
-    "pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-py3-clang5-android-ndk-r19c",
-    "pytorch-linux-xenial-py3-clang5-asan",
-    "pytorch-linux-xenial-py3.8",
-    "pytorch-linux-xenial-py3.6-clang7",
-    "pytorch-linux-xenial-py3.6-gcc4.8",
-    "pytorch-linux-xenial-py3.6-gcc5.4",
-    "pytorch-linux-xenial-py3.6-gcc7.2",
-    "pytorch-linux-xenial-py3.6-gcc7",
-    "pytorch-linux-xenial-pynightly",
-    "pytorch-linux-xenial-rocm3.3-py3.6",
-]
-
-
-def get_workflow_jobs():
-    """Generates a list of docker image build definitions"""
-    return [
-        OrderedDict(
-            {
-                "docker_build_job": OrderedDict(
-                    {"name": quote(image_name), "image_name": quote(image_name)}
-                )
-            }
-        )
-        for image_name in IMAGE_NAMES
-    ]

.circleci/cimodel/data/simple/ge_config_tests.py

@@ -1,103 +0,0 @@
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.versions import MultiPartVersion, CudaVersion
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_BASIC, DOCKER_IMAGE_CUDA_10_2
-
-
-class GeConfigTestJob:
-    def __init__(self,
-                 py_version,
-                 gcc_version,
-                 cuda_version,
-                 variant_parts,
-                 extra_requires,
-                 use_cuda_docker=False,
-                 build_env_override=None):
-
-        self.py_version = py_version
-        self.gcc_version = gcc_version
-        self.cuda_version = cuda_version
-        self.variant_parts = variant_parts
-        self.extra_requires = extra_requires
-        self.use_cuda_docker = use_cuda_docker
-        self.build_env_override = build_env_override
-
-    def get_all_parts(self, with_dots):
-
-        maybe_py_version = self.py_version.render_dots_or_parts(with_dots) if self.py_version else []
-        maybe_gcc_version = self.gcc_version.render_dots_or_parts(with_dots) if self.gcc_version else []
-        maybe_cuda_version = self.cuda_version.render_dots_or_parts(with_dots) if self.cuda_version else []
-
-        common_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-        ] + maybe_cuda_version + maybe_py_version + maybe_gcc_version
-
-        return common_parts + self.variant_parts
-
-    def gen_tree(self):
-
-        resource_class = "gpu.medium" if self.use_cuda_docker else "large"
-        docker_image = DOCKER_IMAGE_CUDA_10_2 if self.use_cuda_docker else DOCKER_IMAGE_BASIC
-        full_name = "_".join(self.get_all_parts(False))
-        build_env = self.build_env_override or "-".join(self.get_all_parts(True))
-
-        props_dict = {
-            "name": full_name,
-            "build_environment": build_env,
-            "requires": self.extra_requires,
-            "resource_class": resource_class,
-            "docker_image": docker_image,
-        }
-
-        if self.use_cuda_docker:
-            props_dict["use_cuda_docker_runtime"] = miniutils.quote(str(1))
-
-        return [{"pytorch_linux_test": props_dict}]
-
-
-WORKFLOW_DATA = [
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["ge_config_legacy", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"]),
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["ge_config_profiling", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"]),
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["ge_config_simple", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"],
-    ),
-    GeConfigTestJob(
-        None,
-        None,
-        CudaVersion(10, 2),
-        ["cudnn7", "py3", "ge_config_legacy", "test"],
-        ["pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_build"],
-        use_cuda_docker=True,
-        # TODO Why does the build environment specify cuda10.1, while the
-        # job name is cuda10_2?
-        build_env_override="pytorch-linux-xenial-cuda10.1-cudnn7-ge_config_legacy-test"),
-    GeConfigTestJob(
-        None,
-        None,
-        CudaVersion(10, 2),
-        ["cudnn7", "py3", "ge_config_profiling", "test"],
-        ["pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_build"],
-        use_cuda_docker=True,
-        # TODO Why does the build environment specify cuda10.1, while the
-        # job name is cuda10_2?
-        build_env_override="pytorch-linux-xenial-cuda10.1-cudnn7-ge_config_profiling-test"),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/ios_definitions.py

@@ -1,71 +0,0 @@
-from cimodel.data.simple.util.versions import MultiPartVersion
-
-
-IOS_VERSION = MultiPartVersion([11, 2, 1])
-
-
-class ArchVariant:
-    def __init__(self, name, is_custom=False):
-        self.name = name
-        self.is_custom = is_custom
-
-    def render(self):
-        extra_parts = ["custom"] if self.is_custom else []
-        return "_".join([self.name] + extra_parts)
-
-
-def get_platform(arch_variant_name):
-    return "SIMULATOR" if arch_variant_name == "x86_64" else "OS"
-
-
-class IOSJob:
-    def __init__(self, ios_version, arch_variant, is_org_member_context=True, extra_props=None):
-        self.ios_version = ios_version
-        self.arch_variant = arch_variant
-        self.is_org_member_context = is_org_member_context
-        self.extra_props = extra_props
-
-    def gen_name_parts(self, with_version_dots):
-
-        version_parts = self.ios_version.render_dots_or_parts(with_version_dots)
-        build_variant_suffix = "_".join([self.arch_variant.render(), "build"])
-
-        return [
-            "pytorch",
-            "ios",
-        ] + version_parts + [
-            build_variant_suffix,
-        ]
-
-    def gen_job_name(self):
-        return "_".join(self.gen_name_parts(False))
-
-    def gen_tree(self):
-
-        platform_name = get_platform(self.arch_variant.name)
-
-        props_dict = {
-            "build_environment": "-".join(self.gen_name_parts(True)),
-            "ios_arch": self.arch_variant.name,
-            "ios_platform": platform_name,
-            "name": self.gen_job_name(),
-        }
-
-        if self.is_org_member_context:
-            props_dict["context"] = "org-member"
-
-        if self.extra_props:
-            props_dict.update(self.extra_props)
-
-        return [{"pytorch_ios_build": props_dict}]
-
-
-WORKFLOW_DATA = [
-    IOSJob(IOS_VERSION, ArchVariant("x86_64"), is_org_member_context=False),
-    IOSJob(IOS_VERSION, ArchVariant("arm64")),
-    IOSJob(IOS_VERSION, ArchVariant("arm64", True), extra_props={"op_list": "mobilenetv2.yaml"}),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/macos_definitions.py

@@ -1,28 +0,0 @@
-class MacOsJob:
-    def __init__(self, os_version, is_test=False):
-        self.os_version = os_version
-        self.is_test = is_test
-
-    def gen_tree(self):
-        non_phase_parts = ["pytorch", "macos", self.os_version, "py3"]
-
-        phase_name = "test" if self.is_test else "build"
-
-        full_job_name = "_".join(non_phase_parts + [phase_name])
-
-        test_build_dependency = "_".join(non_phase_parts + ["build"])
-        extra_dependencies = [test_build_dependency] if self.is_test else []
-        job_dependencies = extra_dependencies
-
-        # Yes we name the job after itself, it needs a non-empty value in here
-        # for the YAML output to work.
-        props_dict = {"requires": job_dependencies, "name": full_job_name}
-
-        return [{full_job_name: props_dict}]
-
-
-WORKFLOW_DATA = [MacOsJob("10_13"), MacOsJob("10_13", True)]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/mobile_definitions.py

@@ -1,56 +0,0 @@
-"""
-PyTorch Mobile PR builds (use linux host toolchain + mobile build options)
-"""
-
-import cimodel.lib.miniutils as miniutils
-import cimodel.data.simple.util.branch_filters
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_ASAN, DOCKER_IMAGE_NDK
-
-
-class MobileJob:
-    def __init__(self, docker_image, variant_parts, is_master_only=False):
-        self.docker_image = docker_image
-        self.variant_parts = variant_parts
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-        non_phase_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "mobile",
-        ] + self.variant_parts
-
-        full_job_name = "_".join(non_phase_parts)
-        build_env_name = "-".join(non_phase_parts)
-
-        props_dict = {
-            "build_environment": build_env_name,
-            "build_only": miniutils.quote(str(int(True))),
-            "docker_image": self.docker_image,
-            "name": full_job_name,
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{"pytorch_linux_build": props_dict}]
-
-
-WORKFLOW_DATA = [
-    MobileJob(DOCKER_IMAGE_ASAN, ["build"]),
-    MobileJob(DOCKER_IMAGE_ASAN, ["custom", "build", "static"]),
-
-    # Use LLVM-DEV toolchain in android-ndk-r19c docker image
-    MobileJob(DOCKER_IMAGE_NDK, ["custom", "build", "dynamic"]),
-
-    # Use LLVM-DEV toolchain in android-ndk-r19c docker image
-    # Most of this CI is already covered by "mobile-custom-build-dynamic" job
-    MobileJob(DOCKER_IMAGE_NDK, ["code", "analysis"], True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/nightly_android.py

@@ -1,73 +0,0 @@
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_NDK
-
-
-class AndroidNightlyJob:
-    def __init__(self,
-                 variant,
-                 template_name,
-                 extra_props=None,
-                 with_docker=True,
-                 requires=None,
-                 no_build_suffix=False):
-
-        self.variant = variant
-        self.template_name = template_name
-        self.extra_props = extra_props or {}
-        self.with_docker = with_docker
-        self.requires = requires
-        self.no_build_suffix = no_build_suffix
-
-    def gen_tree(self):
-
-        base_name_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "android",
-            "ndk",
-            "r19c",
-        ] + self.variant
-
-        build_suffix = [] if self.no_build_suffix else ["build"]
-        full_job_name = "_".join(["nightly"] + base_name_parts + build_suffix)
-        build_env_name = "-".join(base_name_parts)
-
-        props_dict = {
-            "name": full_job_name,
-            "requires": self.requires,
-            "filters": {"branches": {"only": "nightly"}},
-        }
-
-        props_dict.update(self.extra_props)
-
-        if self.with_docker:
-            props_dict["docker_image"] = DOCKER_IMAGE_NDK
-            props_dict["build_environment"] = build_env_name
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    AndroidNightlyJob(["x86_32"], "pytorch_linux_build"),
-    AndroidNightlyJob(["x86_64"], "pytorch_linux_build"),
-    AndroidNightlyJob(["arm", "v7a"], "pytorch_linux_build"),
-    AndroidNightlyJob(["arm", "v8a"], "pytorch_linux_build"),
-    AndroidNightlyJob(["android_gradle"], "pytorch_android_gradle_build",
-                      with_docker=False,
-                      requires=[
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build"]),
-    AndroidNightlyJob(["x86_32_android_publish_snapshot"], "pytorch_android_publish_snapshot",
-                      extra_props={"context": "org-member"},
-                      with_docker=False,
-                      requires=["nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_android_gradle_build"],
-                      no_build_suffix=True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/nightly_ios.py

@@ -1,68 +0,0 @@
-import cimodel.data.simple.ios_definitions as ios_definitions
-
-
-class IOSNightlyJob:
-    def __init__(self,
-                 variant,
-                 is_upload=False):
-
-        self.variant = variant
-        self.is_upload = is_upload
-
-    def get_phase_name(self):
-        return "upload" if self.is_upload else "build"
-
-    def get_common_name_pieces(self, with_version_dots):
-
-        extra_name_suffix = [self.get_phase_name()] if self.is_upload else []
-
-        common_name_pieces = [
-            "ios",
-        ] + ios_definitions.IOS_VERSION.render_dots_or_parts(with_version_dots) + [
-            "nightly",
-            self.variant,
-            "build",
-        ] + extra_name_suffix
-
-        return common_name_pieces
-
-    def gen_job_name(self):
-        return "_".join(["pytorch"] + self.get_common_name_pieces(False))
-
-    def gen_tree(self):
-        extra_requires = [x.gen_job_name() for x in BUILD_CONFIGS] if self.is_upload else []
-
-        props_dict = {
-            "build_environment": "-".join(["libtorch"] + self.get_common_name_pieces(True)),
-            "requires": extra_requires,
-            "context": "org-member",
-            "filters": {"branches": {"only": "nightly"}},
-        }
-
-        if not self.is_upload:
-            props_dict["ios_arch"] = self.variant
-            props_dict["ios_platform"] = ios_definitions.get_platform(self.variant)
-            props_dict["name"] = self.gen_job_name()
-
-        template_name = "_".join([
-            "binary",
-            "ios",
-            self.get_phase_name(),
-        ])
-
-        return [{template_name: props_dict}]
-
-
-BUILD_CONFIGS = [
-    IOSNightlyJob("x86_64"),
-    IOSNightlyJob("arm64"),
-]
-
-
-WORKFLOW_DATA = BUILD_CONFIGS + [
-    IOSNightlyJob("binary", is_upload=True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/util/branch_filters.py

@@ -1,22 +0,0 @@
-NON_PR_BRANCH_LIST = [
-    "master",
-    r"/ci-all\/.*/",
-    r"/release\/.*/",
-]
-
-RC_PATTERN = r"/v[0-9]+(\.[0-9]+)*-rc[0-9]+/"
-
-def gen_filter_dict(
-        branches_list=NON_PR_BRANCH_LIST,
-        tags_list=None
-):
-    """Generates a filter dictionary for use with CircleCI's job filter"""
-    filter_dict = {
-        "branches": {
-            "only": branches_list,
-        },
-    }
-
-    if tags_list is not None:
-        filter_dict["tags"] = {"only": tags_list}
-    return filter_dict

.circleci/cimodel/data/simple/util/docker_constants.py

@@ -1,30 +0,0 @@
-AWS_DOCKER_HOST = "308535385114.dkr.ecr.us-east-1.amazonaws.com"
-
-# ARE YOU EDITING THIS NUMBER?  MAKE SURE YOU READ THE GUIDANCE AT THE
-# TOP OF .circleci/config.yml
-DOCKER_IMAGE_TAG = "209062ef-ab58-422a-b295-36c4eed6e906"
-
-
-def gen_docker_image_path(container_type):
-    return "/".join([
-        AWS_DOCKER_HOST,
-        "pytorch",
-        container_type + ":" + DOCKER_IMAGE_TAG,
-    ])
-
-
-DOCKER_IMAGE_BASIC = gen_docker_image_path("pytorch-linux-xenial-py3.6-gcc5.4")
-
-DOCKER_IMAGE_CUDA_10_2 = gen_docker_image_path("pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7")
-
-DOCKER_IMAGE_GCC7 = gen_docker_image_path("pytorch-linux-xenial-py3.6-gcc7")
-
-
-def gen_mobile_docker_name(specifier):
-    container_type = "pytorch-linux-xenial-py3-clang5-" + specifier
-    return gen_docker_image_path(container_type)
-
-
-DOCKER_IMAGE_ASAN = gen_mobile_docker_name("asan")
-
-DOCKER_IMAGE_NDK = gen_mobile_docker_name("android-ndk-r19c")

.circleci/cimodel/data/simple/util/versions.py

@@ -1,31 +0,0 @@
-class MultiPartVersion:
-    def __init__(self, parts, prefix=""):
-        self.parts = parts
-        self.prefix = prefix
-
-    def prefixed_parts(self):
-        """
-        Prepends the first element of the version list
-        with the prefix string.
-        """
-        if self.parts:
-            return [self.prefix + str(self.parts[0])] + list(map(str, self.parts[1:]))
-        else:
-            return [self.prefix]
-
-    def render_dots(self):
-        return ".".join(self.prefixed_parts())
-
-    def render_dots_or_parts(self, with_dots):
-        if with_dots:
-            return [self.render_dots()]
-        else:
-            return self.prefixed_parts()
-
-
-class CudaVersion(MultiPartVersion):
-    def __init__(self, major, minor):
-        self.major = major
-        self.minor = minor
-
-        super().__init__([self.major, self.minor], "cuda")

.circleci/cimodel/data/windows_build_definitions.py

@@ -1,142 +0,0 @@
-import cimodel.data.simple.util.branch_filters
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.versions import CudaVersion
-
-
-class WindowsJob:
-    def __init__(
-        self,
-        test_index,
-        vscode_spec,
-        cuda_version,
-        force_on_cpu=False,
-        master_only_pred=lambda job: job.vscode_spec.year != 2019,
-    ):
-        self.test_index = test_index
-        self.vscode_spec = vscode_spec
-        self.cuda_version = cuda_version
-        self.force_on_cpu = force_on_cpu
-        self.master_only_pred = master_only_pred
-
-    def gen_tree(self):
-
-        base_phase = "build" if self.test_index is None else "test"
-        numbered_phase = (
-            base_phase if self.test_index is None else base_phase + str(self.test_index)
-        )
-
-        key_name = "_".join(["pytorch", "windows", base_phase])
-
-        cpu_forcing_name_parts = ["on", "cpu"] if self.force_on_cpu else []
-
-        target_arch = self.cuda_version.render_dots() if self.cuda_version else "cpu"
-
-        base_name_parts = [
-            "pytorch",
-            "windows",
-            self.vscode_spec.render(),
-            "py36",
-            target_arch,
-        ]
-
-        prerequisite_jobs = []
-        if base_phase == "test":
-            prerequisite_jobs.append("_".join(base_name_parts + ["build"]))
-
-        arch_env_elements = (
-            ["cuda" + str(self.cuda_version.major), "cudnn7"]
-            if self.cuda_version
-            else ["cpu"]
-        )
-
-        build_environment_string = "-".join(
-            ["pytorch", "win"]
-            + self.vscode_spec.get_elements()
-            + arch_env_elements
-            + ["py3"]
-        )
-
-        is_running_on_cuda = bool(self.cuda_version) and not self.force_on_cpu
-
-        props_dict = {
-            "build_environment": build_environment_string,
-            "python_version": miniutils.quote("3.6"),
-            "vc_version": miniutils.quote(self.vscode_spec.dotted_version()),
-            "vc_year": miniutils.quote(str(self.vscode_spec.year)),
-            "vc_product": self.vscode_spec.get_product(),
-            "use_cuda": miniutils.quote(str(int(is_running_on_cuda))),
-            "requires": prerequisite_jobs,
-        }
-
-        if self.master_only_pred(self):
-            props_dict[
-                "filters"
-            ] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        name_parts = base_name_parts + cpu_forcing_name_parts + [numbered_phase]
-
-        if base_phase == "test":
-            test_name = "-".join(["pytorch", "windows", numbered_phase])
-            props_dict["test_name"] = test_name
-
-            if is_running_on_cuda:
-                props_dict["executor"] = "windows-with-nvidia-gpu"
-
-        props_dict["cuda_version"] = (
-            miniutils.quote(str(self.cuda_version.major))
-            if self.cuda_version
-            else "cpu"
-        )
-        props_dict["name"] = "_".join(name_parts)
-
-        return [{key_name: props_dict}]
-
-
-class VcSpec:
-    def __init__(self, year, version_elements=None):
-        self.year = year
-        self.version_elements = version_elements or []
-
-    def get_elements(self):
-        return [self.prefixed_year()] + self.version_elements
-
-    def get_product(self):
-        return "Community" if self.year == 2019 else "BuildTools"
-
-    def dotted_version(self):
-        return ".".join(self.version_elements)
-
-    def prefixed_year(self):
-        return "vs" + str(self.year)
-
-    def render(self):
-        return "_".join(filter(None, [self.prefixed_year(), self.dotted_version()]))
-
-def FalsePred(_):
-    return False
-
-def TruePred(_):
-    return True
-
-WORKFLOW_DATA = [
-    # VS2017 CUDA-10.1
-    WindowsJob(None, VcSpec(2017, ["14", "11"]), CudaVersion(10, 1), master_only_pred=FalsePred),
-    WindowsJob(1, VcSpec(2017, ["14", "11"]), CudaVersion(10, 1)),
-    # VS2017 no-CUDA (builds only)
-    WindowsJob(None, VcSpec(2017, ["14", "16"]), CudaVersion(10, 1)),
-    WindowsJob(None, VcSpec(2017, ["14", "16"]), None),
-    # VS2019 CUDA-10.1
-    WindowsJob(None, VcSpec(2019), CudaVersion(10, 1)),
-    WindowsJob(1, VcSpec(2019), CudaVersion(10, 1)),
-    WindowsJob(2, VcSpec(2019), CudaVersion(10, 1)),
-    # VS2019 CPU-only
-    WindowsJob(None, VcSpec(2019), None),
-    WindowsJob(1, VcSpec(2019), None),
-    WindowsJob(2, VcSpec(2019), None, master_only_pred=TruePred),
-    WindowsJob(1, VcSpec(2019), CudaVersion(10, 1), force_on_cpu=True),
-    WindowsJob(2, VcSpec(2019), CudaVersion(10, 1), force_on_cpu=True, master_only_pred=TruePred),
-]
-
-
-def get_windows_workflows():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/lib/conf_tree.py

@@ -1,3 +1,6 @@
+#!/usr/bin/env python3
+
+
 from dataclasses import dataclass, field
 from typing import Optional, Dict
 

.circleci/cimodel/lib/miniutils.py

@@ -1,3 +1,6 @@
+#!/usr/bin/env python3
+
+
 def quote(s):
     return sandwich('"', s)
 

.circleci/cimodel/lib/miniyaml.py

@@ -1,6 +1,7 @@
-from collections import OrderedDict
+#!/usr/bin/env python3
 
-import cimodel.lib.miniutils as miniutils
+
+from collections import OrderedDict
 
 
 LIST_MARKER = "- "
@@ -8,13 +9,23 @@ INDENTATION_WIDTH = 2
 
 
 def is_dict(data):
-    return type(data) in [dict, OrderedDict]
+    return type(data) is dict or type(data) is OrderedDict
 
 
 def is_collection(data):
     return is_dict(data) or type(data) is list
 
 
+# TODO can eventually drop this custom sorting
+def sortkey(x):
+    k = x[0]
+    return (
+        k == "<<",
+        k != "environment",
+        k,
+    )
+
+
 def render(fh, data, depth, is_list_member=False):
     """
     PyYaml does not allow precise control over the quoting
@@ -28,11 +39,10 @@ def render(fh, data, depth, is_list_member=False):
 
         tuples = list(data.items())
         if type(data) is not OrderedDict:
-            tuples.sort()
+            tuples.sort(key=sortkey)
 
         for i, (k, v) in enumerate(tuples):
-            if not v:
-                continue
+
             # If this dict is itself a list member, the first key gets prefixed with a list marker
             list_marker_prefix = LIST_MARKER if is_list_member and not i else ""
 
@@ -41,12 +51,14 @@ def render(fh, data, depth, is_list_member=False):
 
             render(fh, v, depth + 1 + int(is_list_member))
 
+        # TODO Could eventually drop this cosmetic convention
+        if depth == 2:
+            fh.write("\n")
+
     elif type(data) is list:
         for v in data:
             render(fh, v, depth, True)
 
     else:
-        # use empty quotes to denote an empty string value instead of blank space
-        modified_data = miniutils.quote(data) if data == "" else data
         list_member_prefix = indentation + LIST_MARKER if is_list_member else ""
-        fh.write(list_member_prefix + str(modified_data) + "\n")
+        fh.write(list_member_prefix + str(data) + "\n")

.circleci/cimodel/lib/visualization.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+
+"""
+This module encapsulates dependencies on pygraphviz
+"""
+
+import colorsys
+
+import cimodel.lib.conf_tree as conf_tree
+
+
+def rgb2hex(rgb_tuple):
+    def to_hex(f):
+        return "%02x" % int(f * 255)
+
+    return "#" + "".join(map(to_hex, list(rgb_tuple)))
+
+
+def handle_missing_graphviz(f):
+    """
+    If the user has not installed pygraphviz, this causes
+    calls to the draw() method of the returned object to do nothing.
+    """
+    try:
+        import pygraphviz  # noqa: F401
+        return f
+
+    except ModuleNotFoundError:
+
+        class FakeGraph:
+            def draw(self, *args, **kwargs):
+                pass
+
+        return lambda _: FakeGraph()
+
+
+@handle_missing_graphviz
+def generate_graph(toplevel_config_node):
+    """
+    Traverses the graph once first just to find the max depth
+    """
+
+    config_list = conf_tree.dfs(toplevel_config_node)
+
+    max_depth = 0
+    for config in config_list:
+        max_depth = max(max_depth, config.get_depth())
+
+    # color the nodes using the max depth
+
+    from pygraphviz import AGraph
+    dot = AGraph()
+
+    def node_discovery_callback(node, sibling_index, sibling_count):
+        depth = node.get_depth()
+
+        sat_min, sat_max = 0.1, 0.6
+        sat_range = sat_max - sat_min
+
+        saturation_fraction = sibling_index / float(sibling_count - 1) if sibling_count > 1 else 1
+        saturation = sat_min + sat_range * saturation_fraction
+
+        # TODO Use a hash of the node label to determine the color
+        hue = depth / float(max_depth + 1)
+
+        rgb_tuple = colorsys.hsv_to_rgb(hue, saturation, 1)
+
+        this_node_key = node.get_node_key()
+
+        dot.add_node(
+            this_node_key,
+            label=node.get_label(),
+            style="filled",
+            # fillcolor=hex_color + ":orange",
+            fillcolor=rgb2hex(rgb_tuple),
+            penwidth=3,
+            color=rgb2hex(colorsys.hsv_to_rgb(hue, saturation, 0.9))
+        )
+
+    def child_callback(node, child):
+        this_node_key = node.get_node_key()
+        child_node_key = child.get_node_key()
+        dot.add_edge((this_node_key, child_node_key))
+
+    conf_tree.dfs_recurse(toplevel_config_node, lambda x: None, node_discovery_callback, child_callback)
+    return dot

.circleci/codegen_validation/compare_normalized_yaml.sh

@@ -1,17 +0,0 @@
-#!/bin/bash -xe
-
-
-YAML_FILENAME=verbatim-sources/workflows-pytorch-ge-config-tests.yml
-DIFF_TOOL=meld
-
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-pushd ..
-
-
-$DIFF_TOOL $YAML_FILENAME <(./codegen_validation/normalize_yaml_fragment.py < $YAML_FILENAME)
-
-
-popd

.circleci/codegen_validation/normalize_yaml_fragment.py

@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-
-import os
-import sys
-import yaml
-
-# Need to import modules that lie on an upward-relative path
-sys.path.append(os.path.join(sys.path[0], '..'))
-
-import cimodel.lib.miniyaml as miniyaml
-
-
-def regurgitate(depth, use_pyyaml_formatter=False):
-    data = yaml.safe_load(sys.stdin)
-
-    if use_pyyaml_formatter:
-        output = yaml.dump(data, sort_keys=True)
-        sys.stdout.write(output)
-    else:
-        miniyaml.render(sys.stdout, data, depth)
-
-
-if __name__ == "__main__":
-    regurgitate(3)

.circleci/codegen_validation/overwrite_with_normalized.sh

@@ -1,15 +0,0 @@
-#!/bin/bash -xe
-
-YAML_FILENAME=$1
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-pushd ..
-
-TEMP_FILENAME=$(mktemp)
-
-cat $YAML_FILENAME | ./codegen_validation/normalize_yaml_fragment.py > $TEMP_FILENAME
-mv $TEMP_FILENAME $YAML_FILENAME
-
-popd

.circleci/docker/README.md

@@ -1,19 +0,0 @@
-# Docker images for Jenkins
-
-This directory contains everything needed to build the Docker images
-that are used in our CI
-
-The Dockerfiles located in subdirectories are parameterized to
-conditionally run build stages depending on build arguments passed to
-`docker build`. This lets us use only a few Dockerfiles for many
-images. The different configurations are identified by a freeform
-string that we call a _build environment_. This string is persisted in
-each image as the `BUILD_ENVIRONMENT` environment variable.
-
-See `build.sh` for valid build environments (it's the giant switch).
-
-## Contents
-
-* `build.sh` -- dispatch script to launch all builds
-* `common` -- scripts used to execute individual Docker build stages
-* `ubuntu-cuda` -- Dockerfile for Ubuntu image with CUDA support for nvidia-docker

.circleci/docker/android/AndroidManifest.xml

@@ -1 +0,0 @@
-<manifest package="org.pytorch.deps" />

.circleci/docker/android/build.gradle

@@ -1,68 +0,0 @@
-buildscript {
-    ext {
-        minSdkVersion = 21
-        targetSdkVersion = 28
-        compileSdkVersion = 28
-        buildToolsVersion = '28.0.3'
-
-        coreVersion = "1.2.0"
-        extJUnitVersion = "1.1.1"
-        runnerVersion = "1.2.0"
-        rulesVersion = "1.2.0"
-        junitVersion = "4.12"
-    }
-
-    repositories {
-        google()
-        mavenLocal()
-        mavenCentral()
-        jcenter()
-    }
-
-    dependencies {
-        classpath 'com.android.tools.build:gradle:3.3.2'
-        classpath "com.jfrog.bintray.gradle:gradle-bintray-plugin:1.8.0"
-        classpath "com.github.dcendents:android-maven-gradle-plugin:2.1"
-        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.9.8"
-    }
-}
-
-repositories {
-    google()
-    jcenter()
-}
-
-apply plugin: 'com.android.library'
-
-android {
-    compileSdkVersion rootProject.compileSdkVersion
-    buildToolsVersion rootProject.buildToolsVersion
-
-    defaultConfig {
-        minSdkVersion minSdkVersion
-        targetSdkVersion targetSdkVersion
-    }
-
-    sourceSets {
-        main {
-            manifest.srcFile 'AndroidManifest.xml'
-        }
-    }
-}
-
-dependencies {
-    implementation 'com.android.support:appcompat-v7:28.0.0'
-    implementation 'androidx.appcompat:appcompat:1.0.0'
-    implementation 'com.facebook.fbjni:fbjni-java-only:0.0.3'
-    implementation 'com.google.code.findbugs:jsr305:3.0.1'
-    implementation 'com.facebook.soloader:nativeloader:0.8.0'
-
-    implementation 'junit:junit:' + rootProject.junitVersion
-    implementation 'androidx.test:core:' + rootProject.coreVersion
-
-    implementation 'junit:junit:' + rootProject.junitVersion
-    implementation 'androidx.test:core:' + rootProject.coreVersion
-    implementation 'androidx.test.ext:junit:' + rootProject.extJUnitVersion
-    implementation 'androidx.test:rules:' + rootProject.rulesVersion
-    implementation 'androidx.test:runner:' + rootProject.runnerVersion
-}

.circleci/docker/build.sh

@@ -1,340 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-image="$1"
-shift
-
-if [ -z "${image}" ]; then
-  echo "Usage: $0 IMAGE"
-  exit 1
-fi
-
-# TODO: Generalize
-OS="ubuntu"
-DOCKERFILE="${OS}/Dockerfile"
-if [[ "$image" == *-cuda* ]]; then
-  DOCKERFILE="${OS}-cuda/Dockerfile"
-elif [[ "$image" == *-rocm* ]]; then
-  DOCKERFILE="${OS}-rocm/Dockerfile"
-fi
-
-if [[ "$image" == *-trusty* ]]; then
-  UBUNTU_VERSION=14.04
-elif [[ "$image" == *-xenial* ]]; then
-  UBUNTU_VERSION=16.04
-elif [[ "$image" == *-artful* ]]; then
-  UBUNTU_VERSION=17.10
-elif [[ "$image" == *-bionic* ]]; then
-  UBUNTU_VERSION=18.04
-elif [[ "$image" == *-focal* ]]; then
-  UBUNTU_VERSION=20.04
-fi
-
-TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/14.04/x86_64"
-
-# It's annoying to rename jobs every time you want to rewrite a
-# configuration, so we hardcode everything here rather than do it
-# from scratch
-case "$image" in
-  pytorch-linux-xenial-py3.8)
-    # TODO: This is a hack, get rid of this as soon as you get rid of the travis downloads
-    TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/16.04/x86_64"
-    TRAVIS_PYTHON_VERSION=3.8
-    GCC_VERSION=7
-    # Do not install PROTOBUF, DB, and VISION as a test
-    ;;
-  pytorch-linux-xenial-py3.6-gcc4.8)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=4.8
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-py3.6-gcc5.4)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=5
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-py3.6-gcc7.2)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    # Do not install PROTOBUF, DB, and VISION as a test
-    ;;
-  pytorch-linux-xenial-py3.6-gcc7)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-pynightly)
-    TRAVIS_PYTHON_VERSION=nightly
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc5.4)
-    CUDA_VERSION=9.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=5
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7)
-    CUDA_VERSION=9.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda10-cudnn7-py3-gcc7)
-    CUDA_VERSION=10.0
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda10.1-cudnn7-py3-gcc7)
-    CUDA_VERSION=10.1
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-cuda11.0-cudnn8-py3-gcc7)
-    UBUNTU_VERSION=16.04-rc
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-py3-clang5-asan)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=5.0
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-py3-clang5-android-ndk-r19c)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=5.0
-    LLVMDEV=yes
-    PROTOBUF=yes
-    ANDROID=yes
-    ANDROID_NDK_VERSION=r19c
-    GRADLE_VERSION=4.10.3
-    CMAKE_VERSION=3.7.0
-    NINJA_VERSION=1.9.0
-    ;;
-  pytorch-linux-xenial-py3.6-clang7)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-py3.6-clang9)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-py3.8-gcc9)
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda10.2-cudnn7-py3.6-clang9)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda10.2-cudnn7-py3.8-gcc9)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda11.0-cudnn8-py3.6-gcc9)
-    UBUNTU_VERSION=18.04-rc
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-bionic-cuda11.0-cudnn8-py3.8-gcc9)
-    UBUNTU_VERSION=18.04-rc
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-rocm3.3-py3.6)
-    ANACONDA_PYTHON_VERSION=3.6
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ROCM_VERSION=3.3
-    # newer cmake version required
-    CMAKE_VERSION=3.6.3
-    ;;
-  pytorch-linux-bionic-rocm3.3-py3.6)
-    ANACONDA_PYTHON_VERSION=3.6
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ROCM_VERSION=3.3
-    ;;
-esac
-
-# Set Jenkins UID and GID if running Jenkins
-if [ -n "${JENKINS:-}" ]; then
-  JENKINS_UID=$(id -u jenkins)
-  JENKINS_GID=$(id -g jenkins)
-fi
-
-tmp_tag="tmp-$(cat /dev/urandom | tr -dc 'a-z' | fold -w 32 | head -n 1)"
-
-# Build image
-# TODO: build-arg THRIFT is not turned on for any image, remove it once we confirm
-# it's no longer needed.
-docker build \
-       --no-cache \
-       --progress=plain \
-       --build-arg "TRAVIS_DL_URL_PREFIX=${TRAVIS_DL_URL_PREFIX}" \
-       --build-arg "BUILD_ENVIRONMENT=${image}" \
-       --build-arg "PROTOBUF=${PROTOBUF:-}" \
-       --build-arg "THRIFT=${THRIFT:-}" \
-       --build-arg "LLVMDEV=${LLVMDEV:-}" \
-       --build-arg "DB=${DB:-}" \
-       --build-arg "VISION=${VISION:-}" \
-       --build-arg "EC2=${EC2:-}" \
-       --build-arg "JENKINS=${JENKINS:-}" \
-       --build-arg "JENKINS_UID=${JENKINS_UID:-}" \
-       --build-arg "JENKINS_GID=${JENKINS_GID:-}" \
-       --build-arg "UBUNTU_VERSION=${UBUNTU_VERSION}" \
-       --build-arg "CLANG_VERSION=${CLANG_VERSION}" \
-       --build-arg "ANACONDA_PYTHON_VERSION=${ANACONDA_PYTHON_VERSION}" \
-       --build-arg "TRAVIS_PYTHON_VERSION=${TRAVIS_PYTHON_VERSION}" \
-       --build-arg "GCC_VERSION=${GCC_VERSION}" \
-       --build-arg "CUDA_VERSION=${CUDA_VERSION}" \
-       --build-arg "CUDNN_VERSION=${CUDNN_VERSION}" \
-       --build-arg "ANDROID=${ANDROID}" \
-       --build-arg "ANDROID_NDK=${ANDROID_NDK_VERSION}" \
-       --build-arg "GRADLE_VERSION=${GRADLE_VERSION}" \
-       --build-arg "CMAKE_VERSION=${CMAKE_VERSION:-}" \
-       --build-arg "NINJA_VERSION=${NINJA_VERSION:-}" \
-       --build-arg "KATEX=${KATEX:-}" \
-       --build-arg "ROCM_VERSION=${ROCM_VERSION:-}" \
-       -f $(dirname ${DOCKERFILE})/Dockerfile \
-       -t "$tmp_tag" \
-       "$@" \
-       .
-
-function drun() {
-  docker run --rm "$tmp_tag" $*
-}
-
-if [[ "$OS" == "ubuntu" ]]; then
-  if !(drun lsb_release -a 2>&1 | grep -qF Ubuntu); then
-    echo "OS=ubuntu, but:"
-    drun lsb_release -a
-    exit 1
-  fi
-  if !(drun lsb_release -a 2>&1 | grep -qF "$UBUNTU_VERSION"); then
-    echo "UBUNTU_VERSION=$UBUNTU_VERSION, but:"
-    drun lsb_release -a
-    exit 1
-  fi
-fi
-
-if [ -n "$TRAVIS_PYTHON_VERSION" ]; then
-  if [[ "$TRAVIS_PYTHON_VERSION" != nightly ]]; then
-    if !(drun python --version 2>&1 | grep -qF "Python $TRAVIS_PYTHON_VERSION"); then
-      echo "TRAVIS_PYTHON_VERSION=$TRAVIS_PYTHON_VERSION, but:"
-      drun python --version
-      exit 1
-    fi
-  else
-    echo "Please manually check nightly is OK:"
-    drun python --version
-  fi
-fi
-
-if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
-  if !(drun python --version 2>&1 | grep -qF "Python $ANACONDA_PYTHON_VERSION"); then
-    echo "ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION, but:"
-    drun python --version
-    exit 1
-  fi
-fi
-
-if [ -n "$GCC_VERSION" ]; then
-  if !(drun gcc --version 2>&1 | grep -q " $GCC_VERSION\\W"); then
-    echo "GCC_VERSION=$GCC_VERSION, but:"
-    drun gcc --version
-    exit 1
-  fi
-fi
-
-if [ -n "$CLANG_VERSION" ]; then
-  if !(drun clang --version 2>&1 | grep -qF "clang version $CLANG_VERSION"); then
-    echo "CLANG_VERSION=$CLANG_VERSION, but:"
-    drun clang --version
-    exit 1
-  fi
-fi
-
-if [ -n "$KATEX" ]; then
-  if !(drun katex --version); then
-    echo "KATEX=$KATEX, but:"
-    drun katex --version
-    exit 1
-  fi
-fi

.circleci/docker/build_docker.sh

@@ -1,53 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-retry () {
-    $*  || (sleep 1 && $*) || (sleep 2 && $*)
-}
-
-# If UPSTREAM_BUILD_ID is set (see trigger job), then we can
-# use it to tag this build with the same ID used to tag all other
-# base image builds. Also, we can try and pull the previous
-# image first, to avoid rebuilding layers that haven't changed.
-
-#until we find a way to reliably reuse previous build, this last_tag is not in use
-# last_tag="$(( CIRCLE_BUILD_NUM - 1 ))"
-tag="${CIRCLE_WORKFLOW_ID}"
-
-
-registry="308535385114.dkr.ecr.us-east-1.amazonaws.com"
-image="${registry}/pytorch/${IMAGE_NAME}"
-
-login() {
-  aws ecr get-authorization-token --region us-east-1 --output text --query 'authorizationData[].authorizationToken' |
-    base64 -d |
-    cut -d: -f2 |
-    docker login -u AWS --password-stdin "$1"
-}
-
-# Retry on timeouts (can happen on job stampede).
-retry login "${registry}"
-
-# Logout on exit
-trap "docker logout ${registry}" EXIT
-
-# export EC2=1
-# export JENKINS=1
-
-# Try to pull the previous image (perhaps we can reuse some layers)
-# if [ -n "${last_tag}" ]; then
-#   docker pull "${image}:${last_tag}" || true
-# fi
-
-# Build new image
-./build.sh ${IMAGE_NAME} -t "${image}:${tag}"
-
-docker push "${image}:${tag}"
-
-# TODO: Get rid of duplicate tagging once ${DOCKER_TAG} becomes the default
-docker tag "${image}:${tag}" "${image}:${DOCKER_TAG}"
-docker push "${image}:${DOCKER_TAG}"
-
-docker save -o "${IMAGE_NAME}:${tag}.tar" "${image}:${tag}"
-aws s3 cp "${IMAGE_NAME}:${tag}.tar" "s3://ossci-linux-build/pytorch/base/${IMAGE_NAME}:${tag}.tar" --acl public-read

.circleci/docker/common/install_android.sh

@@ -1,129 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "${ANDROID_NDK}" ]
-
-apt-get update
-apt-get install -y --no-install-recommends autotools-dev autoconf unzip
-apt-get autoclean && apt-get clean
-rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-pushd /tmp
-curl -Os --retry 3 https://dl.google.com/android/repository/android-ndk-${ANDROID_NDK}-linux-x86_64.zip
-popd
-_ndk_dir=/opt/ndk
-mkdir -p "$_ndk_dir"
-unzip -qo /tmp/android*.zip -d "$_ndk_dir"
-_versioned_dir=$(find "$_ndk_dir/" -mindepth 1 -maxdepth 1 -type d)
-mv "$_versioned_dir"/* "$_ndk_dir"/
-rmdir "$_versioned_dir"
-rm -rf /tmp/*
-
-# Install OpenJDK
-# https://hub.docker.com/r/picoded/ubuntu-openjdk-8-jdk/dockerfile/
-
-sudo apt-get update && \
-    apt-get install -y openjdk-8-jdk && \
-    apt-get install -y ant && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/* && \
-    rm -rf /var/cache/oracle-jdk8-installer;
-
-# Fix certificate issues, found as of
-# https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/983302
-
-sudo apt-get update && \
-    apt-get install -y ca-certificates-java && \
-    apt-get clean && \
-    update-ca-certificates -f && \
-    rm -rf /var/lib/apt/lists/* && \
-    rm -rf /var/cache/oracle-jdk8-installer;
-
-export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
-
-# Installing android sdk
-# https://github.com/circleci/circleci-images/blob/staging/android/Dockerfile.m4
-
-_sdk_version=sdk-tools-linux-3859397.zip
-_android_home=/opt/android/sdk
-
-rm -rf $_android_home
-sudo mkdir -p $_android_home
-curl --silent --show-error --location --fail --retry 3 --output /tmp/$_sdk_version https://dl.google.com/android/repository/$_sdk_version
-sudo unzip -q /tmp/$_sdk_version -d $_android_home
-rm /tmp/$_sdk_version
-
-sudo chmod -R 777 $_android_home
-
-export ANDROID_HOME=$_android_home
-export ADB_INSTALL_TIMEOUT=120
-
-export PATH="${ANDROID_HOME}/emulator:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools:${PATH}"
-echo "PATH:${PATH}"
-alias sdkmanager="$ANDROID_HOME/tools/bin/sdkmanager"
-
-sudo mkdir ~/.android && sudo echo '### User Sources for Android SDK Manager' > ~/.android/repositories.cfg
-sudo chmod -R 777 ~/.android
-
-yes | sdkmanager --licenses
-yes | sdkmanager --update
-
-sdkmanager \
-  "tools" \
-  "platform-tools" \
-  "emulator"
-
-sdkmanager \
-  "build-tools;28.0.3" \
-  "build-tools;29.0.2"
-
-sdkmanager \
-  "platforms;android-28" \
-  "platforms;android-29"
-sdkmanager --list
-
-# Installing Gradle
-echo "GRADLE_VERSION:${GRADLE_VERSION}"
-_gradle_home=/opt/gradle
-sudo rm -rf $gradle_home
-sudo mkdir -p $_gradle_home
-
-wget --no-verbose --output-document=/tmp/gradle.zip \
-"https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip"
-
-sudo unzip -q /tmp/gradle.zip -d $_gradle_home
-rm /tmp/gradle.zip
-
-sudo chmod -R 777 $_gradle_home
-
-export GRADLE_HOME=$_gradle_home/gradle-$GRADLE_VERSION
-alias gradle="${GRADLE_HOME}/bin/gradle"
-
-export PATH="${GRADLE_HOME}/bin/:${PATH}"
-echo "PATH:${PATH}"
-
-gradle --version
-
-mkdir /var/lib/jenkins/gradledeps
-cp build.gradle /var/lib/jenkins/gradledeps
-cp AndroidManifest.xml /var/lib/jenkins/gradledeps
-
-pushd /var/lib/jenkins
-
-export GRADLE_LOCAL_PROPERTIES=gradledeps/local.properties
-rm -f $GRADLE_LOCAL_PROPERTIES
-echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-
-chown -R jenkins /var/lib/jenkins/gradledeps
-chgrp -R jenkins /var/lib/jenkins/gradledeps
-
-sudo -H -u jenkins $GRADLE_HOME/bin/gradle -p /var/lib/jenkins/gradledeps -g /var/lib/jenkins/.gradle --refresh-dependencies --debug --stacktrace assemble
-
-chown -R jenkins /var/lib/jenkins/.gradle
-chgrp -R jenkins /var/lib/jenkins/.gradle
-
-popd
-
-rm -rf /var/lib/jenkins/.gradle/daemon

.circleci/docker/common/install_base.sh

@@ -1,75 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# NVIDIA dockers for RC releases use tag names like `11.0-cudnn8-devel-ubuntu18.04-rc`,
-# for this case we will set UBUNTU_VERSION to `18.04-rc` so that the Dockerfile could
-# find the correct image. As a result, here we have to check for
-#   "$UBUNTU_VERSION" == "18.04"*
-# instead of
-#   "$UBUNTU_VERSION" == "18.04"
-if [[ "$UBUNTU_VERSION" == "18.04"* ]]; then
-  cmake3="cmake=3.10*"
-else
-  cmake3="cmake=3.5*"
-fi
-
-# Install common dependencies
-apt-get update
-# TODO: Some of these may not be necessary
-# TODO: libiomp also gets installed by conda, aka there's a conflict
-ccache_deps="asciidoc docbook-xml docbook-xsl xsltproc"
-numpy_deps="gfortran"
-apt-get install -y --no-install-recommends \
-  $ccache_deps \
-  $numpy_deps \
-  ${cmake3} \
-  apt-transport-https \
-  autoconf \
-  automake \
-  build-essential \
-  ca-certificates \
-  curl \
-  git \
-  libatlas-base-dev \
-  libc6-dbg \
-  libiomp-dev \
-  libyaml-dev \
-  libz-dev \
-  libjpeg-dev \
-  libasound2-dev \
-  libsndfile-dev \
-  python \
-  python-dev \
-  python-setuptools \
-  python-wheel \
-  software-properties-common \
-  sudo \
-  wget \
-  vim
-
-# Install Valgrind separately since the apt-get version is too old.
-mkdir valgrind_build && cd valgrind_build
-VALGRIND_VERSION=3.15.0
-if ! wget http://valgrind.org/downloads/valgrind-${VALGRIND_VERSION}.tar.bz2
-then
-  wget https://sourceware.org/ftp/valgrind/valgrind-${VALGRIND_VERSION}.tar.bz2
-fi
-tar -xjf valgrind-${VALGRIND_VERSION}.tar.bz2
-cd valgrind-${VALGRIND_VERSION}
-./configure --prefix=/usr/local
-make -j 4
-sudo make install
-cd ../../
-rm -rf valgrind_build
-alias valgrind="/usr/local/bin/valgrind"
-
-# TODO: THIS IS A HACK!!!
-# distributed nccl(2) tests are a bit busted, see https://github.com/pytorch/pytorch/issues/5877
-if dpkg -s libnccl-dev; then
-  apt-get remove -y libnccl-dev libnccl2 --allow-change-held-packages
-fi
-
-# Cleanup package manager
-apt-get autoclean && apt-get clean
-rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

.circleci/docker/common/install_cache.sh

@@ -1,35 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-mkdir -p /opt/cache/bin
-mkdir -p /opt/cache/lib
-sed -e 's|PATH="\(.*\)"|PATH="/opt/cache/bin:\1"|g' -i /etc/environment
-export PATH="/opt/cache/bin:$PATH"
-
-# Setup compiler cache
-curl --retry 3 https://s3.amazonaws.com/ossci-linux/sccache -o /opt/cache/bin/sccache
-chmod a+x /opt/cache/bin/sccache
-
-function write_sccache_stub() {
-  printf "#!/bin/sh\nexec sccache $(which $1) \$*" > "/opt/cache/bin/$1"
-  chmod a+x "/opt/cache/bin/$1"
-}
-
-write_sccache_stub cc
-write_sccache_stub c++
-write_sccache_stub gcc
-write_sccache_stub g++
-write_sccache_stub clang
-write_sccache_stub clang++
-
-if [ -n "$CUDA_VERSION" ]; then
-  # TODO: This is a workaround for the fact that PyTorch's FindCUDA
-  # implementation cannot find nvcc if it is setup this way, because it
-  # appears to search for the nvcc in PATH, and use its path to infer
-  # where CUDA is installed.  Instead, we install an nvcc symlink outside
-  # of the PATH, and set CUDA_NVCC_EXECUTABLE so that we make use of it.
-
-  printf "#!/bin/sh\nexec sccache $(which nvcc) \"\$@\"" > /opt/cache/lib/nvcc
-  chmod a+x /opt/cache/lib/nvcc
-fi

.circleci/docker/common/install_clang.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-if [ -n "$CLANG_VERSION" ]; then
-
-  if [[ $CLANG_VERSION == 7 && $UBUNTU_VERSION == 16.04 ]]; then
-    wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
-    sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main"
-  elif [[ $CLANG_VERSION == 9 && $UBUNTU_VERSION == 18.04 ]]; then
-    sudo apt-get update
-    # gpg-agent is not available by default on 18.04
-    sudo apt-get install  -y --no-install-recommends gpg-agent
-    wget --no-check-certificate -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add  -
-    apt-add-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-${CLANG_VERSION} main"
-  fi
-
-  sudo apt-get update
-  apt-get install -y --no-install-recommends clang-"$CLANG_VERSION"
-  apt-get install -y --no-install-recommends llvm-"$CLANG_VERSION"
-
-  # Install dev version of LLVM.
-  if [ -n "$LLVMDEV" ]; then
-    sudo apt-get install -y --no-install-recommends llvm-"$CLANG_VERSION"-dev
-  fi
-
-  # Use update-alternatives to make this version the default
-  # TODO: Decide if overriding gcc as well is a good idea
-  # update-alternatives --install /usr/bin/gcc gcc /usr/bin/clang-"$CLANG_VERSION" 50
-  # update-alternatives --install /usr/bin/g++ g++ /usr/bin/clang++-"$CLANG_VERSION" 50
-  update-alternatives --install /usr/bin/clang clang /usr/bin/clang-"$CLANG_VERSION" 50
-  update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-"$CLANG_VERSION" 50
-
-  # clang's packaging is a little messed up (the runtime libs aren't
-  # added into the linker path), so give it a little help
-  clang_lib=("/usr/lib/llvm-$CLANG_VERSION/lib/clang/"*"/lib/linux")
-  echo "$clang_lib" > /etc/ld.so.conf.d/clang.conf
-  ldconfig
-
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-fi

.circleci/docker/common/install_cmake.sh

@@ -1,16 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "$CMAKE_VERSION" ]
-
-# Turn 3.6.3 into v3.6
-path=$(echo "${CMAKE_VERSION}" | sed -e 's/\([0-9].[0-9]\+\).*/v\1/')
-file="cmake-${CMAKE_VERSION}-Linux-x86_64.tar.gz"
-
-# Download and install specific CMake version in /usr/local
-pushd /tmp
-curl -Os --retry 3 "https://cmake.org/files/${path}/${file}"
-tar -C /usr/local --strip-components 1 --no-same-owner -zxf cmake-*.tar.gz
-rm -f cmake-*.tar.gz
-popd

.circleci/docker/common/install_conda.sh

@@ -1,96 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# Optionally install conda
-if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
-  BASE_URL="https://repo.anaconda.com/miniconda"
-
-  MAJOR_PYTHON_VERSION=$(echo "$ANACONDA_PYTHON_VERSION" | cut -d . -f 1)
-
-  case "$MAJOR_PYTHON_VERSION" in
-    2)
-      CONDA_FILE="Miniconda2-latest-Linux-x86_64.sh"
-    ;;
-    3)
-      CONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
-    ;;
-    *)
-      echo "Unsupported ANACONDA_PYTHON_VERSION: $ANACONDA_PYTHON_VERSION"
-      exit 1
-      ;;
-  esac
-
-  mkdir /opt/conda
-  chown jenkins:jenkins /opt/conda
-
-  as_jenkins() {
-    # NB: unsetting the environment variables works around a conda bug
-    # https://github.com/conda/conda/issues/6576
-    # NB: Pass on PATH and LD_LIBRARY_PATH to sudo invocation
-    # NB: This must be run from a directory that jenkins has access to,
-    # works around https://github.com/conda/conda-package-handling/pull/34
-    sudo -H -u jenkins env -u SUDO_UID -u SUDO_GID -u SUDO_COMMAND -u SUDO_USER env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
-  }
-
-  pushd /tmp
-  wget -q "${BASE_URL}/${CONDA_FILE}"
-  chmod +x "${CONDA_FILE}"
-  as_jenkins ./"${CONDA_FILE}" -b -f -p "/opt/conda"
-  popd
-
-  # NB: Don't do this, rely on the rpath to get it right
-  #echo "/opt/conda/lib" > /etc/ld.so.conf.d/conda-python.conf
-  #ldconfig
-  sed -e 's|PATH="\(.*\)"|PATH="/opt/conda/bin:\1"|g' -i /etc/environment
-  export PATH="/opt/conda/bin:$PATH"
-
-  # Ensure we run conda in a directory that jenkins has write access to
-  pushd /opt/conda
-
-  # Track latest conda update
-  as_jenkins conda update -n base conda
-
-  # Install correct Python version
-  as_jenkins conda install python="$ANACONDA_PYTHON_VERSION"
-
-  conda_install() {
-    # Ensure that the install command don't upgrade/downgrade Python
-    # This should be called as
-    #   conda_install pkg1 pkg2 ... [-c channel]
-    as_jenkins conda install -q -y python="$ANACONDA_PYTHON_VERSION" $*
-  }
-
-  # Install PyTorch conda deps, as per https://github.com/pytorch/pytorch README
-  # DO NOT install cmake here as it would install a version newer than 3.5, but
-  # we want to pin to version 3.5.
-  if [ "$ANACONDA_PYTHON_VERSION" = "3.8" ]; then
-    # DO NOT install typing if installing python-3.8, since its part of python-3.8 core packages
-    # Install llvm-8 as it is required to compile llvmlite-0.30.0 from source
-    conda_install numpy pyyaml mkl mkl-include setuptools cffi future six llvmdev=8.0.0
-  else
-    conda_install numpy pyyaml mkl mkl-include setuptools cffi typing future six
-  fi
-  if [[ "$CUDA_VERSION" == 9.2* ]]; then
-    conda_install magma-cuda92 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.0* ]]; then
-    conda_install magma-cuda100 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.1* ]]; then
-    conda_install magma-cuda101 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.2* ]]; then
-    conda_install magma-cuda102 -c pytorch
-  fi
-
-  # TODO: This isn't working atm
-  conda_install nnpack -c killeent
-
-  # Install some other packages
-  # TODO: Why is scipy pinned
-  # numba & llvmlite is pinned because of https://github.com/numba/numba/issues/4368
-  # scikit-learn is pinned because of
-  # https://github.com/scikit-learn/scikit-learn/issues/14485 (affects gcc 5.5
-  # only)
-  as_jenkins pip install --progress-bar off pytest scipy==1.1.0 scikit-learn==0.20.3 scikit-image librosa>=0.6.2 psutil numba==0.46.0 llvmlite==0.30.0
-
-  popd
-fi

.circleci/docker/common/install_db.sh

@@ -1,61 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# This function installs protobuf 2.6
-install_protobuf_26() {
-  pb_dir="/usr/temp_pb_install_dir"
-  mkdir -p $pb_dir
-
-  # On the nvidia/cuda:9-cudnn7-devel-centos7 image we need this symlink or
-  # else it will fail with
-  #   g++: error: ./../lib64/crti.o: No such file or directory
-  ln -s /usr/lib64 "$pb_dir/lib64"
-
-  curl -LO "https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz"
-  tar -xvz -C "$pb_dir" --strip-components 1 -f protobuf-2.6.1.tar.gz
-  pushd "$pb_dir" && ./configure && make && make check && sudo make install && sudo ldconfig
-  popd
-  rm -rf $pb_dir
-}
-
-install_ubuntu() {
-  apt-get update
-  apt-get install -y --no-install-recommends \
-          libhiredis-dev \
-          libleveldb-dev \
-          liblmdb-dev \
-          libsnappy-dev
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Need EPEL for many packages we depend on.
-  # See http://fedoraproject.org/wiki/EPEL
-  yum --enablerepo=extras install -y epel-release
-
-  yum install -y \
-      hiredis-devel \
-      leveldb-devel \
-      lmdb-devel \
-      snappy-devel
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install base packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/common/install_gcc.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-if [ -n "$GCC_VERSION" ]; then
-
-  # Need the official toolchain repo to get alternate packages
-  add-apt-repository ppa:ubuntu-toolchain-r/test
-  apt-get update
-  if [ "$UBUNTU_VERSION" = "16.04" -a "$GCC_VERSION" = "5" ]; then
-    apt-get install -y g++-5=5.4.0-6ubuntu1~16.04.12
-  else
-    apt-get install -y g++-$GCC_VERSION
-  fi
-
-  update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-"$GCC_VERSION" 50
-  update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-"$GCC_VERSION" 50
-
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-fi

.circleci/docker/common/install_jni.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-mkdir -p /usr/local/include
-cp jni.h /usr/local/include

.circleci/docker/common/install_katex.sh

@@ -1,20 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-if [ -n "$KATEX" ]; then
-
-  curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
-  sudo apt-get install -y nodejs
-
-  curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
-  echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
-
-  apt-get update
-  apt-get install -y --no-install-recommends yarn
-  yarn global add katex --prefix /usr/local
-
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-fi

.circleci/docker/common/install_llvm.sh

@@ -1,30 +0,0 @@
-#!/bin/bash
-
-set -ex
-llvm_url="https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/llvm-9.0.1.src.tar.xz"
-
-mkdir /opt/llvm
-
-pushd /tmp
-wget --no-verbose --output-document=llvm.tar.xz "$llvm_url"
-mkdir llvm
-tar -xf llvm.tar.xz -C llvm --strip-components 1
-rm -f llvm.tar.xz
-
-cd llvm
-mkdir build
-cd build
-cmake -G "Unix Makefiles" \
-  -DCMAKE_BUILD_TYPE=MinSizeRel \
-  -DLLVM_ENABLE_ASSERTIONS=ON \
-  -DCMAKE_INSTALL_PREFIX=/opt/llvm \
-  -DLLVM_TARGETS_TO_BUILD="host" \
-  -DLLVM_BUILD_TOOLS=OFF \
-  -DLLVM_BUILD_UTILS=OFF \
-  -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON \
-  ../
-
-make -j4
-sudo make install
-
-popd

.circleci/docker/common/install_ninja.sh

@@ -1,13 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "$NINJA_VERSION" ]
-
-url="https://github.com/ninja-build/ninja/releases/download/v${NINJA_VERSION}/ninja-linux.zip"
-
-pushd /tmp
-wget --no-verbose --output-document=ninja-linux.zip "$url"
-unzip ninja-linux.zip -d /usr/local/bin
-rm -f ninja-linux.zip
-popd

.circleci/docker/common/install_protobuf.sh

@@ -1,56 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# This function installs protobuf 2.6
-install_protobuf_26() {
-  pb_dir="/usr/temp_pb_install_dir"
-  mkdir -p $pb_dir
-
-  # On the nvidia/cuda:9-cudnn7-devel-centos7 image we need this symlink or
-  # else it will fail with
-  #   g++: error: ./../lib64/crti.o: No such file or directory
-  ln -s /usr/lib64 "$pb_dir/lib64"
-
-  curl -LO "https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz"
-  tar -xvz -C "$pb_dir" --strip-components 1 -f protobuf-2.6.1.tar.gz
-  pushd "$pb_dir" && ./configure && make && make check && sudo make install && sudo ldconfig
-  popd
-  rm -rf $pb_dir
-}
-
-install_ubuntu() {
-  # Ubuntu 14.04 ships with protobuf 2.5, but ONNX needs protobuf >= 2.6
-  # so we install that here if on 14.04
-  # Ubuntu 14.04 also has cmake 2.8.12 as the default option, so we will
-  # install cmake3 here and use cmake3.
-  apt-get update
-  if [[ "$UBUNTU_VERSION" == 14.04 ]]; then
-    apt-get install -y --no-install-recommends cmake3
-    install_protobuf_26
-  else
-    apt-get install -y --no-install-recommends \
-            libprotobuf-dev \
-            protobuf-compiler
-  fi
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Centos7 ships with protobuf 2.5, but ONNX needs protobuf >= 2.6
-  # so we always install install that here
-  install_protobuf_26
-}
-
-# Install base packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/common/install_rocm.sh

@@ -1,89 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-install_ubuntu() {
-    apt-get update
-    if [[ $UBUNTU_VERSION == 18.04 ]]; then
-      # gpg-agent is not available by default on 18.04
-      apt-get install -y --no-install-recommends gpg-agent
-    fi
-    apt-get install -y wget
-    apt-get install -y libopenblas-dev
-
-    # Need the libc++1 and libc++abi1 libraries to allow torch._C to load at runtime
-    apt-get install -y libc++1
-    apt-get install -y libc++abi1
-
-    DEB_ROCM_REPO=http://repo.radeon.com/rocm/apt/${ROCM_VERSION}
-    # Add rocm repository
-    wget -qO - $DEB_ROCM_REPO/rocm.gpg.key | apt-key add -
-    echo "deb [arch=amd64] $DEB_ROCM_REPO xenial main" > /etc/apt/sources.list.d/rocm.list
-    apt-get update --allow-insecure-repositories
-
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
-                   rocm-dev \
-                   rocm-utils \
-                   rocfft \
-                   miopen-hip \
-                   rocblas \
-                   hipsparse \
-                   rocrand \
-                   hipcub \
-                   rocthrust \
-                   rccl \
-                   rocprofiler-dev \
-                   roctracer-dev
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-
-  yum update -y
-  yum install -y wget
-  yum install -y openblas-devel
-
-  yum install -y epel-release
-  yum install -y dkms kernel-headers-`uname -r` kernel-devel-`uname -r`
-
-  echo "[ROCm]" > /etc/yum.repos.d/rocm.repo
-  echo "name=ROCm" >> /etc/yum.repos.d/rocm.repo
-  echo "baseurl=http://repo.radeon.com/rocm/yum/rpm/" >> /etc/yum.repos.d/rocm.repo
-  echo "enabled=1" >> /etc/yum.repos.d/rocm.repo
-  echo "gpgcheck=0" >> /etc/yum.repos.d/rocm.repo
-
-  yum update -y
-
-  yum install -y \
-                   rocm-dev \
-                   rocm-utils \
-                   rocfft \
-                   miopen-hip \
-                   rocblas \
-                   hipsparse \
-                   rocrand \
-                   rccl \
-                   hipcub \
-                   rocthrust \
-                   rocprofiler-dev \
-                   roctracer-dev
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install Python packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/common/install_thrift.sh

@@ -1,14 +0,0 @@
-apt-get update
-apt-get install -y sudo wget libboost-dev libboost-test-dev libboost-program-options-dev libboost-filesystem-dev libboost-thread-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev
-wget https://www-us.apache.org/dist/thrift/0.12.0/thrift-0.12.0.tar.gz
-tar -xvf thrift-0.12.0.tar.gz
-cd thrift-0.12.0
-for file in ./compiler/cpp/Makefile*; do
-  sed -i 's/\-Werror//' $file
-done
-./bootstrap.sh
-./configure --without-php --without-java --without-python --without-nodejs --without-go --without-ruby
-sudo make
-sudo make install
-cd ..
-rm thrift-0.12.0.tar.gz

.circleci/docker/common/install_travis_python.sh

@@ -1,97 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-as_jenkins() {
-  # NB: Preserve PATH and LD_LIBRARY_PATH changes
-  sudo -H -u jenkins env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
-}
-
-if [ -n "$TRAVIS_PYTHON_VERSION" ]; then
-
-  mkdir -p /opt/python
-  chown jenkins:jenkins /opt/python
-
-  # Download Python binary from Travis
-  pushd tmp
-  as_jenkins wget --quiet ${TRAVIS_DL_URL_PREFIX}/python-$TRAVIS_PYTHON_VERSION.tar.bz2
-  # NB: The tarball also comes with /home/travis virtualenv that we
-  # don't care about.  (Maybe we should, but we've worked around the
-  # "how do I install to python" issue by making this entire directory
-  # user-writable "lol")
-  # NB: Relative ordering of opt/python and flags matters
-  as_jenkins tar xjf python-$TRAVIS_PYTHON_VERSION.tar.bz2 --strip-components=2 --directory /opt/python opt/python
-  popd
-
-  echo "/opt/python/$TRAVIS_PYTHON_VERSION/lib" > /etc/ld.so.conf.d/travis-python.conf
-  ldconfig
-  sed -e 's|PATH="\(.*\)"|PATH="/opt/python/'"$TRAVIS_PYTHON_VERSION"'/bin:\1"|g' -i /etc/environment
-  export PATH="/opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH"
-
-  python --version
-  pip --version
-
-  # Install pip from source.
-  # The python-pip package on Ubuntu Trusty is old
-  # and upon install numpy doesn't use the binary
-  # distribution, and fails to compile it from source.
-  pushd tmp
-  as_jenkins curl -L -O https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz
-  as_jenkins tar zxf pip-9.0.1.tar.gz
-  pushd pip-9.0.1
-  as_jenkins python setup.py install
-  popd
-  rm -rf pip-9.0.1*
-  popd
-
-  # Install pip packages
-  as_jenkins pip install --upgrade pip
-
-  pip --version
-
-  if [[ "$TRAVIS_PYTHON_VERSION" == nightly ]]; then
-      # These two packages have broken Cythonizations uploaded
-      # to PyPi, see:
-      #
-      #  - https://github.com/numpy/numpy/issues/10500
-      #  - https://github.com/yaml/pyyaml/issues/117
-      #
-      # Furthermore, the released version of Cython does not
-      # have these issues fixed.
-      #
-      # While we are waiting on fixes for these, we build
-      # from Git for now.  Feel free to delete this conditional
-      # branch if things start working again (you may need
-      # to do this if these packages regress on Git HEAD.)
-      as_jenkins pip install git+https://github.com/cython/cython.git
-      as_jenkins pip install git+https://github.com/numpy/numpy.git
-      as_jenkins pip install git+https://github.com/yaml/pyyaml.git
-  else
-      as_jenkins pip install numpy pyyaml
-  fi
-
-  as_jenkins pip install \
-      future \
-      hypothesis \
-      protobuf \
-      pytest \
-      pillow \
-      typing
-
-  as_jenkins pip install mkl mkl-devel
-
-  # SciPy does not support Python 3.7 or Python 2.7.9
-  if [[ "$TRAVIS_PYTHON_VERSION" != nightly ]] && [[ "$TRAVIS_PYTHON_VERSION" != "2.7.9" ]]; then
-      as_jenkins pip install scipy==1.1.0 scikit-image librosa>=0.6.2
-  fi
-
-  # Install psutil for dataloader tests
-  as_jenkins pip install psutil
-
-  # Install dill for serialization tests
-  as_jenkins pip install "dill>=0.3.1"
-
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-fi

.circleci/docker/common/install_user.sh

@@ -1,20 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# Mirror jenkins user in container
-echo "jenkins:x:1014:1014::/var/lib/jenkins:" >> /etc/passwd
-echo "jenkins:x:1014:" >> /etc/group
-
-# Create $HOME
-mkdir -p /var/lib/jenkins
-chown jenkins:jenkins /var/lib/jenkins
-mkdir -p /var/lib/jenkins/.ccache
-chown jenkins:jenkins /var/lib/jenkins/.ccache
-
-# Allow writing to /usr/local (for make install)
-chown jenkins:jenkins /usr/local
-
-# Allow sudo
-# TODO: Maybe we shouldn't
-echo 'jenkins ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/jenkins

.circleci/docker/common/install_vision.sh

@@ -1,57 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# This function installs protobuf 2.6
-install_protobuf_26() {
-  pb_dir="/usr/temp_pb_install_dir"
-  mkdir -p $pb_dir
-
-  # On the nvidia/cuda:9-cudnn7-devel-centos7 image we need this symlink or
-  # else it will fail with
-  #   g++: error: ./../lib64/crti.o: No such file or directory
-  ln -s /usr/lib64 "$pb_dir/lib64"
-
-  curl -LO "https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz"
-  tar -xvz -C "$pb_dir" --strip-components 1 -f protobuf-2.6.1.tar.gz
-  pushd "$pb_dir" && ./configure && make && make check && sudo make install && sudo ldconfig
-  popd
-  rm -rf $pb_dir
-}
-
-install_ubuntu() {
-  apt-get update
-  apt-get install -y --no-install-recommends \
-          libopencv-dev \
-          libavcodec-dev
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Need EPEL for many packages we depend on.
-  # See http://fedoraproject.org/wiki/EPEL
-  yum --enablerepo=extras install -y epel-release
-
-  yum install -y \
-      opencv-devel \
-      ffmpeg-devel
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install base packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/ubuntu-cuda/Dockerfile

@@ -1,94 +0,0 @@
-ARG UBUNTU_VERSION
-ARG CUDA_VERSION
-ARG CUDNN_VERSION
-
-FROM nvidia/cuda:${CUDA_VERSION}-cudnn${CUDNN_VERSION}-devel-ubuntu${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-ARG CUDA_VERSION
-ARG CUDNN_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install katex
-ARG KATEX
-ADD ./common/install_katex.sh install_katex.sh
-RUN bash ./install_katex.sh && rm install_katex.sh
-
-# Install conda
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# Install gcc
-ARG GCC_VERSION
-ADD ./common/install_gcc.sh install_gcc.sh
-RUN bash ./install_gcc.sh && rm install_gcc.sh
-
-# Install clang
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# Install non-standard Python versions (via Travis binaries)
-ARG TRAVIS_PYTHON_VERSION
-ENV PATH /opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH
-ADD ./common/install_travis_python.sh install_travis_python.sh
-RUN bash ./install_travis_python.sh && rm install_travis_python.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-ENV CUDA_NVCC_EXECUTABLE=/opt/cache/lib/nvcc
-
-# Add jni.h for java host build
-ADD ./common/install_jni.sh install_jni.sh
-ADD ./java/jni.h jni.h
-RUN bash ./install_jni.sh && rm install_jni.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-# AWS specific CUDA build guidance
-ENV TORCH_CUDA_ARCH_LIST Maxwell
-ENV TORCH_NVCC_FLAGS "-Xfatbin -compress-all"
-
-# Install LLVM dev version
-ADD ./common/install_llvm.sh install_llvm.sh
-RUN bash ./install_llvm.sh
-
-USER jenkins
-CMD ["bash"]

.circleci/docker/ubuntu-rocm/.gitignore

@@ -1 +0,0 @@
-*.sh

.circleci/docker/ubuntu-rocm/Dockerfile

@@ -1,86 +0,0 @@
-ARG UBUNTU_VERSION
-
-FROM ubuntu:${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install clang
-ARG LLVMDEV
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install conda
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# Install rocm
-ARG ROCM_VERSION
-ADD ./common/install_rocm.sh install_rocm.sh
-RUN bash ./install_rocm.sh
-RUN rm install_rocm.sh
-ENV PATH /opt/rocm/bin:$PATH
-ENV PATH /opt/rocm/hcc/bin:$PATH
-ENV PATH /opt/rocm/hip/bin:$PATH
-ENV PATH /opt/rocm/opencl/bin:$PATH
-ENV HIP_PLATFORM hcc
-ENV LANG C.UTF-8
-ENV LC_ALL C.UTF-8
-
-# (optional) Install non-default CMake version
-ARG CMAKE_VERSION
-ADD ./common/install_cmake.sh install_cmake.sh
-RUN if [ -n "${CMAKE_VERSION}" ]; then bash ./install_cmake.sh; fi
-RUN rm install_cmake.sh
-
-# (optional) Install non-default Ninja version
-ARG NINJA_VERSION
-ADD ./common/install_ninja.sh install_ninja.sh
-RUN if [ -n "${NINJA_VERSION}" ]; then bash ./install_ninja.sh; fi
-RUN rm install_ninja.sh
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-USER jenkins
-CMD ["bash"]

.circleci/docker/ubuntu/Dockerfile

@@ -1,119 +0,0 @@
-ARG UBUNTU_VERSION
-
-FROM ubuntu:${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install clang
-ARG LLVMDEV
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# (optional) Install thrift.
-ARG THRIFT
-ADD ./common/install_thrift.sh install_thrift.sh
-RUN if [ -n "${THRIFT}" ]; then bash ./install_thrift.sh; fi
-RUN rm install_thrift.sh
-ENV INSTALLED_THRIFT ${THRIFT}
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install katex
-ARG KATEX
-ADD ./common/install_katex.sh install_katex.sh
-RUN bash ./install_katex.sh && rm install_katex.sh
-
-# Install conda
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# Install gcc
-ARG GCC_VERSION
-ADD ./common/install_gcc.sh install_gcc.sh
-RUN bash ./install_gcc.sh && rm install_gcc.sh
-
-# Install non-standard Python versions (via Travis binaries)
-ARG TRAVIS_PYTHON_VERSION
-ARG TRAVIS_DL_URL_PREFIX
-ENV PATH /opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH
-ADD ./common/install_travis_python.sh install_travis_python.sh
-RUN bash ./install_travis_python.sh && rm install_travis_python.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# (optional) Install Android NDK
-ARG ANDROID
-ARG ANDROID_NDK
-ARG GRADLE_VERSION
-ADD ./common/install_android.sh install_android.sh
-ADD ./android/AndroidManifest.xml AndroidManifest.xml
-ADD ./android/build.gradle build.gradle
-RUN if [ -n "${ANDROID}" ]; then bash ./install_android.sh; fi
-RUN rm install_android.sh
-RUN rm AndroidManifest.xml
-RUN rm build.gradle
-ENV INSTALLED_ANDROID ${ANDROID}
-
-# (optional) Install non-default CMake version
-ARG CMAKE_VERSION
-ADD ./common/install_cmake.sh install_cmake.sh
-RUN if [ -n "${CMAKE_VERSION}" ]; then bash ./install_cmake.sh; fi
-RUN rm install_cmake.sh
-
-# (optional) Install non-default Ninja version
-ARG NINJA_VERSION
-ADD ./common/install_ninja.sh install_ninja.sh
-RUN if [ -n "${NINJA_VERSION}" ]; then bash ./install_ninja.sh; fi
-RUN rm install_ninja.sh
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-
-# Add jni.h for java host build
-ADD ./common/install_jni.sh install_jni.sh
-ADD ./java/jni.h jni.h
-RUN bash ./install_jni.sh && rm install_jni.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-# Install LLVM dev version
-ADD ./common/install_llvm.sh install_llvm.sh
-RUN bash ./install_llvm.sh
-
-USER jenkins
-CMD ["bash"]

.circleci/ecr_gc_docker/Dockerfile

@@ -1,13 +0,0 @@
-FROM ubuntu:16.04
-
-RUN apt-get update && apt-get install -y python-pip git && rm -rf /var/lib/apt/lists/* /var/log/dpkg.log
-
-ADD requirements.txt /requirements.txt
-
-RUN pip install -r /requirements.txt
-
-ADD gc.py /usr/bin/gc.py
-
-ADD docker_hub.py /usr/bin/docker_hub.py
-
-ENTRYPOINT ["/usr/bin/gc.py"]

.circleci/ecr_gc_docker/docker_hub.py

@@ -1,125 +0,0 @@
-#!/usr/bin/env python
-
-from collections import namedtuple
-
-import boto3
-import requests
-import os
-
-
-IMAGE_INFO = namedtuple(
-    "IMAGE_INFO", ("repo", "tag", "size", "last_updated_at", "last_updated_by")
-)
-
-
-def build_access_token(username, passwordtr):
-    r = requests.post(
-        "https://hub.docker.com/v2/users/login/",
-        data={"username": username, "password": password},
-    )
-    r.raise_for_status()
-    token = r.json().get("token")
-    return {"Authorization": "JWT " + token}
-
-
-def list_repos(user, token):
-    r = requests.get("https://hub.docker.com/v2/repositories/" + user, headers=token)
-    r.raise_for_status()
-    ret = sorted(
-        repo["user"] + "/" + repo["name"] for repo in r.json().get("results", [])
-    )
-    if ret:
-        print("repos found:")
-        print("".join("\n\t" + r for r in ret))
-    return ret
-
-
-def list_tags(repo, token):
-    r = requests.get(
-        "https://hub.docker.com/v2/repositories/" + repo + "/tags", headers=token
-    )
-    r.raise_for_status()
-    return [
-        IMAGE_INFO(
-            repo=repo,
-            tag=t["name"],
-            size=t["full_size"],
-            last_updated_at=t["last_updated"],
-            last_updated_by=t["last_updater_username"],
-        )
-        for t in r.json().get("results", [])
-    ]
-
-
-def save_to_s3(tags):
-    table_content = ""
-    client = boto3.client("s3")
-    for t in tags:
-        table_content += (
-            "<tr><td>{repo}</td><td>{tag}</td><td>{size}</td>"
-            "<td>{last_updated_at}</td><td>{last_updated_by}</td></tr>"
-        ).format(
-            repo=t.repo,
-            tag=t.tag,
-            size=t.size,
-            last_updated_at=t.last_updated_at,
-            last_updated_by=t.last_updated_by,
-        )
-    html_body = """
-    <html>
-        <head>
-            <link rel="stylesheet"
-                href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
-                integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh"
-                crossorigin="anonymous">
-            <link rel="stylesheet" type="text/css"
-                href="https://cdn.datatables.net/1.10.20/css/jquery.dataTables.css">
-            <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js">
-            </script>
-            <script type="text/javascript" charset="utf8"
-                src="https://cdn.datatables.net/1.10.20/js/jquery.dataTables.js"></script>
-            <title> docker image info</title>
-        </head>
-        <body>
-            <table class="table table-striped table-hover" id="docker">
-            <caption>Docker images on docker hub</caption>
-            <thead class="thead-dark">
-                <tr>
-                <th scope="col">repo</th>
-                <th scope="col">tag</th>
-                <th scope="col">size</th>
-                <th scope="col">last_updated_at</th>
-                <th scope="col">last_updated_by</th>
-                </tr>
-            </thead>
-            <tbody>
-                {table_content}
-            </tbody>
-            </table>
-        </body>
-        <script>
-            $(document).ready( function () {{
-                $('#docker').DataTable({{paging: false}});
-            }} );py
-        </script>
-    </html>
-    """.format(
-        table_content=table_content
-    )
-    client.put_object(
-        Bucket="docker.pytorch.org",
-        ACL="public-read",
-        Key="docker_hub.html",
-        Body=html_body,
-        ContentType="text/html",
-    )
-
-
-if __name__ == "__main__":
-    username = os.environ.get("DOCKER_HUB_USERNAME")
-    password = os.environ.get("DOCKER_HUB_PASSWORD")
-    token = build_access_token(username, password)
-    tags = []
-    for repo in list_repos("pytorch", token):
-        tags.extend(list_tags(repo, token))
-    save_to_s3(tags)

.circleci/ecr_gc_docker/gc.py

@@ -1,214 +0,0 @@
-#!/usr/bin/env python
-
-import argparse
-import datetime
-import boto3
-import pytz
-import sys
-import re
-
-
-def save_to_s3(project, data):
-    table_content = ""
-    client = boto3.client("s3")
-    for repo, tag, window, age, pushed in data:
-        table_content += "<tr><td>{repo}</td><td>{tag}</td><td>{window}</td><td>{age}</td><td>{pushed}</td></tr>".format(
-            repo=repo, tag=tag, window=window, age=age, pushed=pushed
-        )
-    html_body = """
-    <html>
-        <head>
-            <link rel="stylesheet"
-                href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
-                integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh"
-                crossorigin="anonymous">
-            <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.20/css/jquery.dataTables.css">
-            <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
-            <script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.20/js/jquery.dataTables.js"></script>
-            <title>{project} nightly and permanent docker image info</title>
-        </head>
-        <body>
-            <table class="table table-striped table-hover" id="docker">
-            <thead class="thead-dark">
-                <tr>
-                <th scope="col">repo</th>
-                <th scope="col">tag</th>
-                <th scope="col">keep window</th>
-                <th scope="col">age</th>
-                <th scope="col">pushed at</th>
-                </tr>
-            </thead>
-            <tbody>
-                {table_content}
-            </tbody>
-            </table>
-        </body>
-        <script>
-            $(document).ready( function () {{
-                $('#docker').DataTable({{paging: false}});
-            }} );
-        </script>
-    </html>
-    """.format(
-        project=project, table_content=table_content
-    )
-
-    # for pytorch, file can be found at
-    # http://ossci-docker.s3-website.us-east-1.amazonaws.com/pytorch.html
-    # and later one we can config docker.pytorch.org to point to the location
-
-    client.put_object(
-        Bucket="docker.pytorch.org",
-        ACL="public-read",
-        Key="{project}.html".format(project=project),
-        Body=html_body,
-        ContentType="text/html",
-    )
-
-
-def repos(client):
-    paginator = client.get_paginator("describe_repositories")
-    pages = paginator.paginate(registryId="308535385114")
-    for page in pages:
-        for repo in page["repositories"]:
-            yield repo
-
-
-def images(client, repository):
-    paginator = client.get_paginator("describe_images")
-    pages = paginator.paginate(
-        registryId="308535385114", repositoryName=repository["repositoryName"]
-    )
-    for page in pages:
-        for image in page["imageDetails"]:
-            yield image
-
-
-parser = argparse.ArgumentParser(description="Delete old Docker tags from registry")
-parser.add_argument(
-    "--dry-run", action="store_true", help="Dry run; print tags that would be deleted"
-)
-parser.add_argument(
-    "--keep-stable-days",
-    type=int,
-    default=14,
-    help="Days of stable Docker tags to keep (non per-build images)",
-)
-parser.add_argument(
-    "--keep-unstable-days",
-    type=int,
-    default=1,
-    help="Days of unstable Docker tags to keep (per-build images)",
-)
-parser.add_argument(
-    "--filter-prefix",
-    type=str,
-    default="",
-    help="Only run cleanup for repositories with this prefix",
-)
-parser.add_argument(
-    "--ignore-tags",
-    type=str,
-    default="",
-    help="Never cleanup these tags (comma separated)",
-)
-args = parser.parse_args()
-
-if not args.ignore_tags or not args.filter_prefix:
-    print(
-        """
-Missing required arguments --ignore-tags and --filter-prefix
-
-You must specify --ignore-tags and --filter-prefix to avoid accidentally
-pruning a stable Docker tag which is being actively used.  This will
-make you VERY SAD.  So pay attention.
-
-First, which filter-prefix do you want?  The list of valid prefixes
-is in jobs/private.groovy under the 'docker-registry-cleanup' job.
-You probably want either pytorch or caffe2.
-
-Second, which ignore-tags do you want?  It should be whatever the most
-up-to-date DockerVersion for the repository in question is.  Follow
-the imports of jobs/pytorch.groovy to find them.
-"""
-    )
-    sys.exit(1)
-
-client = boto3.client("ecr", region_name="us-east-1")
-stable_window = datetime.timedelta(days=args.keep_stable_days)
-unstable_window = datetime.timedelta(days=args.keep_unstable_days)
-now = datetime.datetime.now(pytz.UTC)
-ignore_tags = args.ignore_tags.split(",")
-
-
-def chunks(chunkable, n):
-    """ Yield successive n-sized chunks from l.
-    """
-    for i in range(0, len(chunkable), n):
-        yield chunkable[i : i + n]
-
-SHA_PATTERN = re.compile(r'^[0-9a-f]{40}$')
-def looks_like_git_sha(tag):
-    """Returns a boolean to check if a tag looks like a git sha
-
-    For reference a sha1 is 40 characters with only 0-9a-f and contains no
-    "-" characters
-    """
-    return re.match(SHA_PATTERN, tag) is not None
-
-stable_window_tags = []
-for repo in repos(client):
-    repositoryName = repo["repositoryName"]
-    if not repositoryName.startswith(args.filter_prefix):
-        continue
-
-    # Keep list of image digests to delete for this repository
-    digest_to_delete = []
-    print(repositoryName)
-
-    for image in images(client, repo):
-        tags = image.get("imageTags")
-        if not isinstance(tags, (list,)) or len(tags) == 0:
-            continue
-
-        tag = tags[0]
-        created = image["imagePushedAt"]
-        age = now - created
-        if any([
-                looks_like_git_sha(tag),
-                tag.isdigit(),
-                tag.count("-") == 4,  # TODO: Remove, this no longer applies as tags are now built using a SHA1
-                tag in ignore_tags]):
-            window = stable_window
-            if tag in ignore_tags:
-                stable_window_tags.append((repositoryName, tag, "", age, created))
-            elif age < window:
-                stable_window_tags.append((repositoryName, tag, window, age, created))
-        else:
-            window = unstable_window
-
-        if tag in ignore_tags:
-            print("Ignoring tag {}:{} (age: {})".format(repositoryName, tag, age))
-            continue
-        if age < window:
-            print("Not deleting manifest for tag {}:{} (age: {})".format(repositoryName, tag, age))
-            continue
-
-        if args.dry_run:
-            print("(dry run) Deleting manifest for tag {}:{} (age: {})".format(repositoryName, tag, age))
-        else:
-            print("Deleting manifest for tag{}:{} (age: {})".format(repositoryName, tag, age))
-            digest_to_delete.append(image["imageDigest"])
-
-    # Issue batch delete for all images to delete for this repository
-    # Note that as of 2018-07-25, the maximum number of images you can
-    # delete in a single batch is 100, so chunk our list into batches of
-    # 100
-    for c in chunks(digest_to_delete, 100):
-        client.batch_delete_image(
-            registryId="308535385114",
-            repositoryName=repositoryName,
-            imageIds=[{"imageDigest": digest} for digest in c],
-        )
-
-    save_to_s3(args.filter_prefix, stable_window_tags)

.circleci/ecr_gc_docker/requirements.txt

@@ -1,3 +0,0 @@
-boto3
-pytz
-requests

.circleci/generate_config_yml.py

@@ -6,24 +6,13 @@ Please see README.md in this directory for details.
 """
 
 import os
-import shutil
 import sys
-from collections import OrderedDict, namedtuple
+import shutil
+from collections import namedtuple, OrderedDict
 
+import cimodel.data.pytorch_build_definitions as pytorch_build_definitions
 import cimodel.data.binary_build_definitions as binary_build_definitions
 import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
-import cimodel.data.pytorch_build_definitions as pytorch_build_definitions
-import cimodel.data.simple.android_definitions
-import cimodel.data.simple.bazel_definitions
-import cimodel.data.simple.binary_smoketest
-import cimodel.data.simple.docker_definitions
-import cimodel.data.simple.ge_config_tests
-import cimodel.data.simple.ios_definitions
-import cimodel.data.simple.macos_definitions
-import cimodel.data.simple.mobile_definitions
-import cimodel.data.simple.nightly_android
-import cimodel.data.simple.nightly_ios
-import cimodel.data.windows_build_definitions as windows_build_definitions
 import cimodel.lib.miniutils as miniutils
 import cimodel.lib.miniyaml as miniyaml
 
@@ -32,7 +21,6 @@ class File(object):
     """
     Verbatim copy the contents of a file into config.yml
     """
-
     def __init__(self, filename):
         self.filename = filename
 
@@ -41,7 +29,7 @@ class File(object):
             shutil.copyfileobj(fh, output_filehandle)
 
 
-class FunctionGen(namedtuple("FunctionGen", "function depth")):
+class FunctionGen(namedtuple('FunctionGen', 'function depth')):
     __slots__ = ()
 
 
@@ -51,14 +39,15 @@ class Treegen(FunctionGen):
     """
 
     def write(self, output_filehandle):
-        miniyaml.render(output_filehandle, self.function(), self.depth)
+        build_dict = OrderedDict()
+        self.function(build_dict)
+        miniyaml.render(output_filehandle, build_dict, self.depth)
 
 
 class Listgen(FunctionGen):
     """
     Insert the content of a YAML list into config.yml
     """
-
     def write(self, output_filehandle):
         miniyaml.render(output_filehandle, self.function(), self.depth)
 
@@ -68,6 +57,7 @@ def horizontal_rule():
 
 
 class Header(object):
+
     def __init__(self, title, summary=None):
         self.title = title
         self.summary_lines = summary or []
@@ -81,81 +71,49 @@ class Header(object):
             output_filehandle.write(line + "\n")
 
 
-def gen_build_workflows_tree():
-    build_workflows_functions = [
-        pytorch_build_definitions.get_workflow_jobs,
-        cimodel.data.simple.macos_definitions.get_workflow_jobs,
-        cimodel.data.simple.android_definitions.get_workflow_jobs,
-        cimodel.data.simple.ios_definitions.get_workflow_jobs,
-        cimodel.data.simple.mobile_definitions.get_workflow_jobs,
-        cimodel.data.simple.ge_config_tests.get_workflow_jobs,
-        cimodel.data.simple.bazel_definitions.get_workflow_jobs,
-        caffe2_build_definitions.get_workflow_jobs,
-        cimodel.data.simple.binary_smoketest.get_workflow_jobs,
-        cimodel.data.simple.nightly_ios.get_workflow_jobs,
-        cimodel.data.simple.nightly_android.get_workflow_jobs,
-        windows_build_definitions.get_windows_workflows,
-    ]
-
-    binary_build_functions = [
-        binary_build_definitions.get_binary_build_jobs,
-        binary_build_definitions.get_nightly_tests,
-        binary_build_definitions.get_nightly_uploads,
-        binary_build_definitions.get_post_upload_jobs,
-        binary_build_definitions.get_binary_smoke_test_jobs,
-    ]
-
-    docker_builder_functions = [
-        cimodel.data.simple.docker_definitions.get_workflow_jobs
-    ]
-
-    return {
-        "workflows": {
-            "binary_builds": {
-                "when": r"<< pipeline.parameters.run_binary_tests >>",
-                "jobs": [f() for f in binary_build_functions],
-            },
-            "docker_build": OrderedDict(
-                {
-                    "triggers": [
-                        {
-                            "schedule": {
-                                "cron": miniutils.quote("0 15 * * 0"),
-                                "filters": {"branches": {"only": ["master"]}},
-                            }
-                        }
-                    ],
-                    "jobs": [f() for f in docker_builder_functions],
-                }
-            ),
-            "build": {"jobs": [f() for f in build_workflows_functions]},
-        }
-    }
-
-
 # Order of this list matters to the generated config.yml.
 YAML_SOURCES = [
     File("header-section.yml"),
-    File("commands.yml"),
+    File("linux-build-defaults.yml"),
+    File("macos-build-defaults.yml"),
     File("nightly-binary-build-defaults.yml"),
-    Header("Build parameters"),
-    File("build-parameters/pytorch-build-params.yml"),
-    File("build-parameters/caffe2-build-params.yml"),
-    File("build-parameters/binary-build-params.yml"),
-    File("build-parameters/promote-build-params.yml"),
-    Header("Job specs"),
-    File("job-specs/pytorch-job-specs.yml"),
-    File("job-specs/caffe2-job-specs.yml"),
-    File("job-specs/binary-job-specs.yml"),
-    File("job-specs/job-specs-custom.yml"),
-    File("job-specs/job-specs-promote.yml"),
-    File("job-specs/binary_update_htmls.yml"),
-    File("job-specs/binary-build-tests.yml"),
-    File("job-specs/docker_jobs.yml"),
-    Header("Workflows"),
-    Treegen(gen_build_workflows_tree, 0),
-    File("workflows/workflows-ecr-gc.yml"),
-    File("workflows/workflows-promote.yml"),
+    File("linux-binary-build-defaults.yml"),
+    File("macos-binary-build-defaults.yml"),
+    File("nightly-build-smoke-tests-defaults.yml"),
+    Header("Job specifications job specs"),
+    Treegen(pytorch_build_definitions.add_build_env_defs, 0),
+    File("job-specs-setup.yml"),
+    File("job-specs-custom.yml"),
+    Treegen(caffe2_build_definitions.add_caffe2_builds, 1),
+    File("binary_update_htmls.yml"),
+    Header("Binary build specs individual job specifications"),
+    Treegen(binary_build_definitions.add_binary_build_specs, 1),
+    Header(
+        "Binary build tests", [
+            "These are the smoke tests run right after the build, before the upload.",
+            "If these fail, the upload doesn't happen."
+        ]
+    ),
+    Treegen(binary_build_definitions.add_binary_build_tests, 1),
+    File("binary-build-tests.yml"),
+    Header("Binary build uploads"),
+    Treegen(binary_build_definitions.add_binary_build_uploads, 1),
+    Header("Smoke test specs individual job specifications"),
+    Treegen(binary_build_definitions.add_smoke_test_specs, 1),
+    File("workflows.yml"),
+    Listgen(pytorch_build_definitions.get_workflow_list, 3),
+    File("workflows-pytorch-macos-builds.yml"),
+    Listgen(caffe2_build_definitions.get_caffe2_workflows, 3),
+    File("workflows-binary-builds-smoke-subset.yml"),
+    Header("Daily smoke test trigger"),
+    Treegen(binary_build_definitions.add_binary_smoke_test_jobs, 1),
+    Header("Daily binary build trigger"),
+    Treegen(binary_build_definitions.add_binary_build_jobs, 1),
+    Header("Nightly tests"),
+    Listgen(binary_build_definitions.get_nightly_tests, 3),
+    File("workflows-nightly-uploads-header.yml"),
+    Listgen(binary_build_definitions.get_nightly_uploads, 3),
+    File("workflows-s3-html.yml"),
 ]
 
 

.circleci/scripts/binary_checkout.sh

@@ -1,20 +1,9 @@
 #!/bin/bash
 set -eux -o pipefail
-
-retry () {
-    $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-}
-
-
 # This step runs on multiple executors with different envfile locations
 if [[ "$(uname)" == Darwin ]]; then
   # macos executor (builds and tests)
   workdir="/Users/distiller/project"
-elif [[ "$OSTYPE" == "msys" ]]; then
-  # windows executor (builds and tests)
-  rm -rf /c/w
-  ln -s "/c/Users/circleci/project" /c/w
-  workdir="/c/w"
 elif [[ -d "/home/circleci/project" ]]; then
   # machine executor (binary tests)
   workdir="/home/circleci/project"
@@ -24,17 +13,11 @@ else
 fi
 
 # It is very important that this stays in sync with binary_populate_env.sh
-if [[ "$OSTYPE" == "msys" ]]; then
-  # We need to make the paths as short as possible on Windows
-  export PYTORCH_ROOT="$workdir/p"
-  export BUILDER_ROOT="$workdir/b"
-else
-  export PYTORCH_ROOT="$workdir/pytorch"
-  export BUILDER_ROOT="$workdir/builder"
-fi
+export PYTORCH_ROOT="$workdir/pytorch"
+export BUILDER_ROOT="$workdir/builder"
 
 # Clone the Pytorch branch
-retry git clone https://github.com/pytorch/pytorch.git "$PYTORCH_ROOT"
+git clone https://github.com/pytorch/pytorch.git "$PYTORCH_ROOT"
 pushd "$PYTORCH_ROOT"
 if [[ -n "${CIRCLE_PR_NUMBER:-}" ]]; then
   # "smoke" binary build on PRs
@@ -50,14 +33,16 @@ else
   echo "Can't tell what to checkout"
   exit 1
 fi
-retry git submodule update --init --recursive
+git submodule update --init --recursive --quiet
 echo "Using Pytorch from "
 git --no-pager log --max-count 1
 popd
 
 # Clone the Builder master repo
-retry git clone -q https://github.com/pytorch/builder.git "$BUILDER_ROOT"
+git clone -q https://github.com/pytorch/builder.git "$BUILDER_ROOT"
 pushd "$BUILDER_ROOT"
+git fetch origin
+git reset origin/master --hard
 echo "Using builder from "
 git --no-pager log --max-count 1
 popd

.circleci/scripts/binary_install_miniconda.sh

@@ -31,9 +31,9 @@ fi
 
 conda_sh="$workdir/install_miniconda.sh"
 if [[ "$(uname)" == Darwin ]]; then
-  curl --retry 3 -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+  retry curl -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
 else
-  curl --retry 3 -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+  retry curl -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
 fi
 chmod +x "$conda_sh"
 "$conda_sh" -b -p "$MINICONDA_ROOT"

.circleci/scripts/binary_ios_build.sh

@@ -1,42 +0,0 @@
-#!/bin/bash
-set -ex -o pipefail
-
-echo ""
-echo "DIR: $(pwd)"
-WORKSPACE=/Users/distiller/workspace
-PROJ_ROOT=/Users/distiller/project
-export TCLLIBPATH="/usr/local/lib"
-
-# Install conda
-curl --retry 3 -o ~/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
-chmod +x ~/conda.sh
-/bin/bash ~/conda.sh -b -p ~/anaconda
-export PATH="~/anaconda/bin:${PATH}"
-source ~/anaconda/bin/activate
-
-# Install dependencies
-conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing requests --yes
-export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
-
-# sync submodules
-cd ${PROJ_ROOT}
-git submodule sync
-git submodule update --init --recursive
-
-# run build script
-chmod a+x ${PROJ_ROOT}/scripts/build_ios.sh
-echo "########################################################"
-cat ${PROJ_ROOT}/scripts/build_ios.sh
-echo "########################################################"
-echo "IOS_ARCH: ${IOS_ARCH}"
-echo "IOS_PLATFORM: ${IOS_PLATFORM}"
-export IOS_ARCH=${IOS_ARCH}
-export IOS_PLATFORM=${IOS_PLATFORM}
-unbuffer ${PROJ_ROOT}/scripts/build_ios.sh 2>&1 | ts
-
-#store the binary
-cd ${WORKSPACE}
-DEST_DIR=${WORKSPACE}/ios
-mkdir -p ${DEST_DIR}
-cp -R ${PROJ_ROOT}/build_ios/install ${DEST_DIR}
-mv ${DEST_DIR}/install ${DEST_DIR}/${IOS_ARCH}

.circleci/scripts/binary_ios_test.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-set -ex -o pipefail
-
-echo ""
-echo "DIR: $(pwd)"
-PROJ_ROOT=/Users/distiller/project
-cd ${PROJ_ROOT}/ios/TestApp
-# install fastlane
-sudo gem install bundler && bundle install
-# install certificates
-echo "${IOS_CERT_KEY}" >> cert.txt
-base64 --decode cert.txt -o Certificates.p12
-rm cert.txt
-bundle exec fastlane install_cert
-# install the provisioning profile
-PROFILE=TestApp_CI.mobileprovision
-PROVISIONING_PROFILES=~/Library/MobileDevice/Provisioning\ Profiles
-mkdir -pv "${PROVISIONING_PROFILES}"
-cd "${PROVISIONING_PROFILES}"
-echo "${IOS_SIGN_KEY}" >> cert.txt
-base64 --decode cert.txt -o ${PROFILE}
-rm cert.txt
-# run the ruby build script
-if ! [ -x "$(command -v xcodebuild)" ]; then
-    echo 'Error: xcodebuild is not installed.'
-    exit 1
-fi 
-PROFILE=TestApp_CI
-ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM} -c ${PROFILE} -t ${IOS_DEV_TEAM_ID}

.circleci/scripts/binary_ios_upload.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-set -ex -o pipefail
-
-echo ""
-echo "DIR: $(pwd)"
-WORKSPACE=/Users/distiller/workspace
-PROJ_ROOT=/Users/distiller/project
-ARTIFACTS_DIR=${WORKSPACE}/ios
-ls ${ARTIFACTS_DIR}
-ZIP_DIR=${WORKSPACE}/zip
-mkdir -p ${ZIP_DIR}/install/lib
-mkdir -p ${ZIP_DIR}/src
-# copy header files
-cp -R ${ARTIFACTS_DIR}/arm64/include ${ZIP_DIR}/install/
-# build a FAT bianry
-cd ${ZIP_DIR}/install/lib
-target_libs=(libc10.a libclog.a libcpuinfo.a libeigen_blas.a libpthreadpool.a libpytorch_qnnpack.a libtorch_cpu.a libtorch.a libXNNPACK.a)
-for lib in ${target_libs[*]}
-do
-    if [ -f "${ARTIFACTS_DIR}/x86_64/lib/${lib}" ] && [ -f "${ARTIFACTS_DIR}/arm64/lib/${lib}" ]; then
-        libs=("${ARTIFACTS_DIR}/x86_64/lib/${lib}" "${ARTIFACTS_DIR}/arm64/lib/${lib}")
-        lipo -create "${libs[@]}" -o ${ZIP_DIR}/install/lib/${lib}
-    fi
-done
-lipo -i ${ZIP_DIR}/install/lib/*.a
-# copy the umbrella header and license
-cp ${PROJ_ROOT}/ios/LibTorch.h ${ZIP_DIR}/src/
-cp ${PROJ_ROOT}/LICENSE ${ZIP_DIR}/
-# zip the library
-ZIPFILE=libtorch_ios_nightly_build.zip
-cd ${ZIP_DIR}
-#for testing
-touch version.txt
-echo $(date +%s) > version.txt
-zip -r ${ZIPFILE} install src version.txt LICENSE
-# upload to aws
-brew install awscli
-set +x
-export AWS_ACCESS_KEY_ID=${AWS_S3_ACCESS_KEY_FOR_PYTORCH_BINARY_UPLOAD}
-export AWS_SECRET_ACCESS_KEY=${AWS_S3_ACCESS_SECRET_FOR_PYTORCH_BINARY_UPLOAD}
-set +x
-# echo "AWS KEY: ${AWS_ACCESS_KEY_ID}"
-# echo "AWS SECRET: ${AWS_SECRET_ACCESS_KEY}"
-aws s3 cp ${ZIPFILE} s3://ossci-ios-build/ --acl public-read

.circleci/scripts/binary_linux_test.sh

@@ -9,15 +9,11 @@ set -eux -o pipefail
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   retry conda create -qyn testenv python="$DESIRED_PYTHON"
   source activate testenv >/dev/null
-elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
+elif [[ "$DESIRED_PYTHON" == 2.7mu ]]; then
+  export PATH="/opt/python/cp27-cp27mu/bin:\$PATH"
+else
   python_nodot="\$(echo $DESIRED_PYTHON | tr -d m.u)"
-  python_path="/opt/python/cp\$python_nodot-cp\${python_nodot}"
-  # Prior to Python 3.8 paths were suffixed with an 'm'
-  if [[ -d  "\${python_path}/bin" ]]; then
-    export PATH="\${python_path}/bin:\$PATH"
-  elif [[ -d "\${python_path}m/bin" ]]; then
-    export PATH="\${python_path}m/bin:\$PATH"
-  fi
+  export PATH="/opt/python/cp\$python_nodot-cp\${python_nodot}m/bin:\$PATH"
 fi
 
 # Install the package
@@ -29,12 +25,9 @@ fi
 pkg="/final_pkgs/\$(ls /final_pkgs)"
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   conda install -y "\$pkg" --offline
-  if [[ "$DESIRED_CUDA" == 'cpu' ]]; then
-    retry conda install -y cpuonly -c pytorch
-  fi
   retry conda install -yq future numpy protobuf six
   if [[ "$DESIRED_CUDA" != 'cpu' ]]; then
-    # DESIRED_CUDA is in format cu90 or cu102
+    # DESIRED_CUDA is in format cu90 or cu100
     if [[ "${#DESIRED_CUDA}" == 4 ]]; then
       cu_ver="${DESIRED_CUDA:2:1}.${DESIRED_CUDA:3}"
     else
@@ -42,19 +35,13 @@ if [[ "$PACKAGE_TYPE" == conda ]]; then
     fi
     retry conda install -yq -c pytorch "cudatoolkit=\${cu_ver}"
   fi
-elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
+else
   pip install "\$pkg"
   retry pip install -q future numpy protobuf six
 fi
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  pkg="\$(ls /final_pkgs/*-latest.zip)"
-  unzip "\$pkg" -d /tmp
-  cd /tmp/libtorch
-fi
 
 # Test the package
 /builder/check_binary.sh
-
 # =================== The above code will be executed inside Docker container ===================
 EOL
 echo

.circleci/scripts/binary_linux_upload.sh

@@ -5,6 +5,15 @@ set -eu -o pipefail
 set +x
 declare -x "AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
 declare -x "AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
+cat >/home/circleci/project/login_to_anaconda.sh <<EOL
+set +x
+echo "Trying to login to Anaconda"
+yes | anaconda login \
+    --username "$PYTORCH_BINARY_PJH5_CONDA_USERNAME" \
+    --password "$PYTORCH_BINARY_PJH5_CONDA_PASSWORD"
+set -x
+EOL
+chmod +x /home/circleci/project/login_to_anaconda.sh
 
 #!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
 # DO NOT TURN -x ON BEFORE THIS LINE
@@ -12,37 +21,20 @@ declare -x "AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
 set -eux -o pipefail
 export PATH="$MINICONDA_ROOT/bin:$PATH"
 
-# This gets set in binary_populate_env.sh, but lets have a sane default just in case
-PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly}
-# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
-#       The only difference is the trailing slash
-# Strip trailing slashes if there
-CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
-BACKUP_BUCKET="s3://pytorch-backup"
-
-retry pip install -q awscli
 # Upload the package to the final location
 pushd /home/circleci/project/final_pkgs
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   retry conda install -yq anaconda-client
-  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload  "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
-  # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-  # Because there's no actual conda command to read this
-  subdir=$(tar -xOf ./*.bz2 info/index.json | grep subdir  | cut -d ':' -f2 | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//')
-  BACKUP_DIR="conda/${subdir}"
+  retry timeout 30 /home/circleci/project/login_to_anaconda.sh
+  anaconda upload "$(ls)" -u pytorch --label main --no-progress --force
 elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
+  retry pip install -q awscli
   s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
   for pkg in $(ls); do
     retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
   done
-  BACKUP_DIR="libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
 else
+  retry pip install -q awscli
   s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
   retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
-  BACKUP_DIR="whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-fi
-
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  s3_dir="${BACKUP_BUCKET}/${CIRCLE_TAG}/${BACKUP_DIR}"
-  retry aws s3 cp --recursive . "$s3_dir"
 fi

.circleci/scripts/binary_macos_test.sh

@@ -5,30 +5,26 @@ source "/Users/distiller/project/env"
 export "PATH=$workdir/miniconda/bin:$PATH"
 pkg="$workdir/final_pkgs/$(ls $workdir/final_pkgs)"
 
+# Don't test libtorch
+# TODO we should test libtorch
+if [[ "$PACKAGE_TYPE" == libtorch ]]; then
+  exit 0
+fi
+
 # Create a new test env
 # TODO cut all this out into a separate test job and have an entirely different
 # miniconda
-if [[ "$PACKAGE_TYPE" != libtorch ]]; then
-  source deactivate || true
-  conda create -qyn test python="$DESIRED_PYTHON"
-  source activate test >/dev/null
-fi
+source deactivate || true
+conda create -qyn test python="$DESIRED_PYTHON"
+source activate test >/dev/null
 
 # Install the package
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  pkg="$(ls $workdir/final_pkgs/*-latest.zip)"
-  unzip "$pkg" -d /tmp
-  cd /tmp/libtorch
-elif [[ "$PACKAGE_TYPE" == conda ]]; then
+if [[ "$PACKAGE_TYPE" == conda ]]; then
   conda install -y "$pkg" --offline
 else
   pip install "$pkg" --no-index --no-dependencies -v
 fi
 
 # Test
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  $workdir/builder/check_binary.sh
-else
-  pushd "$workdir/pytorch"
-  $workdir/builder/run_tests.sh "$PACKAGE_TYPE" "$DESIRED_PYTHON" "$DESIRED_CUDA"
-fi
+pushd "$workdir/pytorch"
+$workdir/builder/run_tests.sh "$PACKAGE_TYPE" "$DESIRED_PYTHON" "$DESIRED_CUDA"

.circleci/scripts/binary_macos_upload.sh

@@ -4,6 +4,15 @@ set -eu -o pipefail
 set +x
 export AWS_ACCESS_KEY_ID="${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
 export AWS_SECRET_ACCESS_KEY="${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
+cat >/Users/distiller/project/login_to_anaconda.sh <<EOL
+set +x
+echo "Trying to login to Anaconda"
+yes | anaconda login \
+    --username "$PYTORCH_BINARY_PJH5_CONDA_USERNAME" \
+    --password "$PYTORCH_BINARY_PJH5_CONDA_PASSWORD"
+set -x
+EOL
+chmod +x /Users/distiller/project/login_to_anaconda.sh
 
 #!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
 # DO NOT TURN -x ON BEFORE THIS LINE
@@ -13,36 +22,19 @@ set -eux -o pipefail
 source "/Users/distiller/project/env"
 export "PATH=$workdir/miniconda/bin:$PATH"
 
-# This gets set in binary_populate_env.sh, but lets have a sane default just in case
-PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly}
-# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
-#       The only difference is the trailing slash
-# Strip trailing slashes if there
-CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
-BACKUP_BUCKET="s3://pytorch-backup"
-
-retry pip install -q awscli
 pushd "$workdir/final_pkgs"
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   retry conda install -yq anaconda-client
-  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
-  # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-  # Because there's no actual conda command to read this
-  subdir=$(tar -xOf ./*.bz2 info/index.json | grep subdir  | cut -d ':' -f2 | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//')
-  BACKUP_DIR="conda/${subdir}"
+  retry /Users/distiller/project/login_to_anaconda.sh
+  retry anaconda upload "$(ls)" -u pytorch-nightly --label main --no-progress --force
 elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
+  retry pip install -q awscli
   s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
   for pkg in $(ls); do
     retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
   done
-  BACKUP_DIR="libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
 else
+  retry pip install -q awscli
   s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
   retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
-  BACKUP_DIR="whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-fi
-
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  s3_dir="${BACKUP_BUCKET}/${CIRCLE_TAG}/${BACKUP_DIR}"
-  retry aws s3 cp --recursive . "$s3_dir"
 fi

.circleci/scripts/binary_populate_env.sh

@@ -2,31 +2,11 @@
 set -eux -o pipefail
 export TZ=UTC
 
-tagged_version() {
-  # Grabs version from either the env variable CIRCLE_TAG
-  # or the pytorch git described version
-  if [[ "$OSTYPE" == "msys" ]]; then
-    GIT_DESCRIBE="git --git-dir ${workdir}/p/.git describe"
-  else
-    GIT_DESCRIBE="git --git-dir ${workdir}/pytorch/.git describe"
-  fi
-  if [[ -n "${CIRCLE_TAG:-}" ]]; then
-    echo "${CIRCLE_TAG}"
-  elif ${GIT_DESCRIBE} --exact --tags >/dev/null; then
-    ${GIT_DESCRIBE} --tags
-  else
-    return 1
-  fi
-}
-
 # We need to write an envfile to persist these variables to following
 # steps, but the location of the envfile depends on the circleci executor
 if [[ "$(uname)" == Darwin ]]; then
   # macos executor (builds and tests)
   workdir="/Users/distiller/project"
-elif [[ "$OSTYPE" == "msys" ]]; then
-  # windows executor (builds and tests)
-  workdir="/c/w"
 elif [[ -d "/home/circleci/project" ]]; then
   # machine executor (binary tests)
   workdir="/home/circleci/project"
@@ -43,76 +23,42 @@ configs=($BUILD_ENVIRONMENT)
 export PACKAGE_TYPE="${configs[0]}"
 export DESIRED_PYTHON="${configs[1]}"
 export DESIRED_CUDA="${configs[2]}"
-if [[ "${BUILD_FOR_SYSTEM:-}" == "windows" ]]; then
-  export DESIRED_DEVTOOLSET=""
-  export LIBTORCH_CONFIG="${configs[3]:-}"
-  if [[ "$LIBTORCH_CONFIG" == 'debug' ]]; then
-    export DEBUG=1
-  fi
-else
-  export DESIRED_DEVTOOLSET="${configs[3]:-}"
-fi
+export DESIRED_DEVTOOLSET="${configs[3]:-}"
 if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then
   export BUILD_PYTHONLESS=1
 fi
 
 # Pick docker image
-export DOCKER_IMAGE=${DOCKER_IMAGE:-}
-if [[ -z "$DOCKER_IMAGE" ]]; then
-  if [[ "$PACKAGE_TYPE" == conda ]]; then
-    export DOCKER_IMAGE="pytorch/conda-cuda"
-  elif [[ "$DESIRED_CUDA" == cpu ]]; then
-    export DOCKER_IMAGE="pytorch/manylinux-cuda100"
-  else
-    export DOCKER_IMAGE="pytorch/manylinux-cuda${DESIRED_CUDA:2}"
-  fi
+if [[ "$PACKAGE_TYPE" == conda ]]; then
+  export DOCKER_IMAGE="soumith/conda-cuda"
+elif [[ "$DESIRED_CUDA" == cpu ]]; then
+  export DOCKER_IMAGE="soumith/manylinux-cuda100"
+else
+  export DOCKER_IMAGE="soumith/manylinux-cuda${DESIRED_CUDA:2}"
+fi
+
+# Upload to parallel folder for gcc abis
+# All nightlies used to be devtoolset3, then devtoolset7 was added as a build
+# option, so the upload was redirected to nightly/devtoolset7 to avoid
+# conflicts with other binaries (there shouldn't be any conflicts). Now we are
+# making devtoolset7 the default.
+if [[ "$DESIRED_DEVTOOLSET" == 'devtoolset7' || "$(uname)" == 'Darwin' ]]; then
+  export PIP_UPLOAD_FOLDER='nightly/'
+else
+  # On linux machines, this shouldn't actually be called anymore. This is just
+  # here for extra safety.
+  export PIP_UPLOAD_FOLDER='nightly/devtoolset3/'
 fi
 
-# Default to nightly, since that's where this normally uploads to
-PIP_UPLOAD_FOLDER='nightly/'
 # We put this here so that OVERRIDE_PACKAGE_VERSION below can read from it
 export DATE="$(date -u +%Y%m%d)"
-#TODO: We should be pulling semver version from the base version.txt
-BASE_BUILD_VERSION="1.6.0.dev$DATE"
-# Change BASE_BUILD_VERSION to git tag when on a git tag
-# Use 'git -C' to make doubly sure we're in the correct directory for checking
-# the git tag
-if tagged_version >/dev/null; then
-  # Switch upload folder to 'test/' if we are on a tag
-  PIP_UPLOAD_FOLDER='test/'
-  # Grab git tag, remove prefixed v and remove everything after -
-  # Used to clean up tags that are for release candidates like v1.6.0-rc1
-  # Turns tag v1.6.0-rc1 -> v1.6.0
-  BASE_BUILD_VERSION="$(tagged_version | sed -e 's/^v//' -e 's/-.*$//')"
-fi
-if [[ "$(uname)" == 'Darwin' ]] || [[ "$DESIRED_CUDA" == "cu102" ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
-  export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}"
+if [[ "$(uname)" == 'Darwin' ]]; then
+  export PYTORCH_BUILD_VERSION="1.2.0.dev$DATE"
 else
-  export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}+$DESIRED_CUDA"
+  export PYTORCH_BUILD_VERSION="1.2.0.dev$DATE+$DESIRED_CUDA"
 fi
 export PYTORCH_BUILD_NUMBER=1
 
-
-JAVA_HOME=
-BUILD_JNI=OFF
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  POSSIBLE_JAVA_HOMES=()
-  POSSIBLE_JAVA_HOMES+=(/usr/local)
-  POSSIBLE_JAVA_HOMES+=(/usr/lib/jvm/java-8-openjdk-amd64)
-  POSSIBLE_JAVA_HOMES+=(/Library/Java/JavaVirtualMachines/*.jdk/Contents/Home)
-  for JH in "${POSSIBLE_JAVA_HOMES[@]}" ; do
-    if [[ -e "$JH/include/jni.h" ]] ; then
-      echo "Found jni.h under $JH"
-      JAVA_HOME="$JH"
-      BUILD_JNI=ON
-      break
-    fi
-  done
-  if [ -z "$JAVA_HOME" ]; then
-    echo "Did not find jni.h"
-  fi
-fi
-
 cat >>"$envfile" <<EOL
 # =================== The following code will be executed inside Docker container ===================
 export TZ=UTC
@@ -124,36 +70,24 @@ export DESIRED_CUDA="$DESIRED_CUDA"
 export LIBTORCH_VARIANT="${LIBTORCH_VARIANT:-}"
 export BUILD_PYTHONLESS="${BUILD_PYTHONLESS:-}"
 export DESIRED_DEVTOOLSET="$DESIRED_DEVTOOLSET"
-if [[ "${BUILD_FOR_SYSTEM:-}" == "windows" ]]; then
-  export LIBTORCH_CONFIG="${LIBTORCH_CONFIG:-}"
-  export DEBUG="${DEBUG:-}"
-fi
 
 export DATE="$DATE"
-export NIGHTLIES_DATE_PREAMBLE=1.6.0.dev
+export NIGHTLIES_DATE_PREAMBLE=1.2.0.dev
 export PYTORCH_BUILD_VERSION="$PYTORCH_BUILD_VERSION"
 export PYTORCH_BUILD_NUMBER="$PYTORCH_BUILD_NUMBER"
 export OVERRIDE_PACKAGE_VERSION="$PYTORCH_BUILD_VERSION"
 
-# TODO: We don't need this anymore IIUC
-export TORCH_PACKAGE_NAME='torch'
+export TORCH_PACKAGE_NAME='torch-nightly'
 export TORCH_CONDA_BUILD_FOLDER='pytorch-nightly'
 
 export USE_FBGEMM=1
-export JAVA_HOME=$JAVA_HOME
-export BUILD_JNI=$BUILD_JNI
 export PIP_UPLOAD_FOLDER="$PIP_UPLOAD_FOLDER"
 export DOCKER_IMAGE="$DOCKER_IMAGE"
 
 export workdir="$workdir"
 export MAC_PACKAGE_WORK_DIR="$workdir"
-if [[ "$OSTYPE" == "msys" ]]; then
-  export PYTORCH_ROOT="$workdir/p"
-  export BUILDER_ROOT="$workdir/b"
-else
-  export PYTORCH_ROOT="$workdir/pytorch"
-  export BUILDER_ROOT="$workdir/builder"
-fi
+export PYTORCH_ROOT="$workdir/pytorch"
+export BUILDER_ROOT="$workdir/builder"
 export MINICONDA_ROOT="$workdir/miniconda"
 export PYTORCH_FINAL_PACKAGE_DIR="$workdir/final_pkgs"
 

.circleci/scripts/binary_run_in_docker.sh

@@ -16,12 +16,31 @@ set -eux -o pipefail
 # Expect actual code to be written to this file
 chmod +x /home/circleci/project/ci_test_script.sh
 
-VOLUME_MOUNTS="-v /home/circleci/project/:/circleci_stuff -v /home/circleci/project/final_pkgs:/final_pkgs -v ${PYTORCH_ROOT}:/pytorch -v ${BUILDER_ROOT}:/builder"
 # Run the docker
 if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --runtime=nvidia ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
+  export id=$(docker run --runtime=nvidia -t -d "${DOCKER_IMAGE}")
 else
-  export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
+  export id=$(docker run -t -d "${DOCKER_IMAGE}")
+fi
+
+# Copy the envfile and script with all the code to run into the docker.
+docker cp /home/circleci/project/. "$id:/circleci_stuff"
+
+# Copy built packages into the docker to test. This should only exist on the
+# binary test jobs. The package should've been created from a binary build job,
+# whhich persisted the package to a CircleCI workspace, which this job then
+# copies into a GPU enabled docker for testing
+if [[ -d "/home/circleci/project/final_pkgs" ]]; then
+  docker cp /home/circleci/project/final_pkgs "$id:/final_pkgs"
+fi
+
+# Copy the needed repos into the docker. These do not exist in the smoke test
+# jobs, since the smoke test jobs do not need the Pytorch source code.
+if [[ -d "$PYTORCH_ROOT" ]]; then
+  docker cp "$PYTORCH_ROOT" "$id:/pytorch"
+fi
+if [[ -d "$BUILDER_ROOT" ]]; then
+  docker cp "$BUILDER_ROOT" "$id:/builder"
 fi
 
 # Execute the test script that was populated by an earlier section

.circleci/scripts/binary_windows_build.sh

@@ -1,41 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/c/w/env"
-mkdir -p "$PYTORCH_FINAL_PACKAGE_DIR"
-
-export CUDA_VERSION="${DESIRED_CUDA/cu/}"
-export USE_SCCACHE=1
-export SCCACHE_BUCKET=ossci-compiler-cache-windows
-export NIGHTLIES_PYTORCH_ROOT="$PYTORCH_ROOT"
-
-if [[ "$CUDA_VERSION" == "92" || "$CUDA_VERSION" == "100" ]]; then
-  export VC_YEAR=2017
-else
-  export VC_YEAR=2019
-fi
-
-set +x
-export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}
-export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}
-set -x
-
-if [[ "$CIRCLECI" == 'true' && -d "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages\\_Instances" ]]; then
-  mv "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages\\_Instances" .
-  rm -rf "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-  mkdir -p "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-  mv _Instances "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-fi
-
-echo "Free space on filesystem before build:"
-df -h
-
-pushd "$BUILDER_ROOT"
-if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
-  ./windows/internal/build_conda.bat
-elif [[ "$PACKAGE_TYPE" == 'wheel' || "$PACKAGE_TYPE" == 'libtorch' ]]; then
-  ./windows/internal/build_wheels.bat
-fi
-
-echo "Free space on filesystem after build:"
-df -h

.circleci/scripts/binary_windows_test.sh

@@ -1,19 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/c/w/env"
-
-export CUDA_VERSION="${DESIRED_CUDA/cu/}"
-export VC_YEAR=2017
-
-if [[ "$CUDA_VERSION" == "92" || "$CUDA_VERSION" == "100" ]]; then
-  export VC_YEAR=2017
-else
-  export VC_YEAR=2019
-fi
-
-pushd "$BUILDER_ROOT"
-
-./windows/internal/smoke_test.bat
-
-popd

.circleci/scripts/binary_windows_upload.sh

@@ -1,47 +0,0 @@
-#!/bin/bash
-set -eu -o pipefail
-set +x
-declare -x "AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
-declare -x "AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
-
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-# DO NOT TURN -x ON BEFORE THIS LINE
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-set -eux -o pipefail
-
-source "/env"
-
-# This gets set in binary_populate_env.sh, but lets have a sane default just in case
-PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly/}
-# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
-#       The only difference is the trailing slash
-# Strip trailing slashes if there
-CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
-BACKUP_BUCKET="s3://pytorch-backup"
-
-retry pip install -q awscli
-pushd /root/workspace/final_pkgs
-# Upload the package to the final location
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  retry conda install -yq anaconda-client
-  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload  "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
-  # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-  # Because there's no actual conda command to read this
-  subdir=$(tar -xOf ./*.bz2 info/index.json | grep subdir  | cut -d ':' -f2 | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//')
-  BACKUP_DIR="conda/${subdir}"
-elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  for pkg in $(ls); do
-    retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
-  done
-  BACKUP_DIR="libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-else
-  s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
-  BACKUP_DIR="whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-fi
-
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  s3_dir="${BACKUP_BUCKET}/${CIRCLE_TAG}/${BACKUP_DIR}"
-  retry aws s3 cp --recursive . "$s3_dir"
-fi

.circleci/scripts/build_android_gradle.sh

@@ -1,85 +0,0 @@
-#!/usr/bin/env bash
-set -eux -o pipefail
-
-export ANDROID_NDK_HOME=/opt/ndk
-export ANDROID_HOME=/opt/android/sdk
-
-# Must be in sync with GRADLE_VERSION in docker image for android
-# https://github.com/pietern/pytorch-dockerfiles/blob/master/build.sh#L155
-export GRADLE_VERSION=4.10.3
-export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
-export GRADLE_PATH=$GRADLE_HOME/bin/gradle
-
-BUILD_ANDROID_INCLUDE_DIR_x86=~/workspace/build_android/install/include
-BUILD_ANDROID_LIB_DIR_x86=~/workspace/build_android/install/lib
-
-BUILD_ANDROID_INCLUDE_DIR_x86_64=~/workspace/build_android_install_x86_64/install/include
-BUILD_ANDROID_LIB_DIR_x86_64=~/workspace/build_android_install_x86_64/install/lib
-
-BUILD_ANDROID_INCLUDE_DIR_arm_v7a=~/workspace/build_android_install_arm_v7a/install/include
-BUILD_ANDROID_LIB_DIR_arm_v7a=~/workspace/build_android_install_arm_v7a/install/lib
-
-BUILD_ANDROID_INCLUDE_DIR_arm_v8a=~/workspace/build_android_install_arm_v8a/install/include
-BUILD_ANDROID_LIB_DIR_arm_v8a=~/workspace/build_android_install_arm_v8a/install/lib
-
-PYTORCH_ANDROID_SRC_MAIN_DIR=~/workspace/android/pytorch_android/src/main
-
-JNI_INCLUDE_DIR=${PYTORCH_ANDROID_SRC_MAIN_DIR}/cpp/libtorch_include
-mkdir -p $JNI_INCLUDE_DIR
-
-JNI_LIBS_DIR=${PYTORCH_ANDROID_SRC_MAIN_DIR}/jniLibs
-mkdir -p $JNI_LIBS_DIR
-
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_x86} ${JNI_INCLUDE_DIR}/x86
-ln -s ${BUILD_ANDROID_LIB_DIR_x86} ${JNI_LIBS_DIR}/x86
-
-if [[ "${BUILD_ENVIRONMENT}" != *-gradle-build-only-x86_32* ]]; then
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_x86_64} ${JNI_INCLUDE_DIR}/x86_64
-ln -s ${BUILD_ANDROID_LIB_DIR_x86_64} ${JNI_LIBS_DIR}/x86_64
-
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v7a} ${JNI_INCLUDE_DIR}/armeabi-v7a
-ln -s ${BUILD_ANDROID_LIB_DIR_arm_v7a} ${JNI_LIBS_DIR}/armeabi-v7a
-
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v8a} ${JNI_INCLUDE_DIR}/arm64-v8a
-ln -s ${BUILD_ANDROID_LIB_DIR_arm_v8a} ${JNI_LIBS_DIR}/arm64-v8a
-fi
-
-env
-echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
-
-GRADLE_PARAMS="-p android assembleRelease --debug --stacktrace"
-if [[ "${BUILD_ENVIRONMENT}" == *-gradle-build-only-x86_32* ]]; then
-    GRADLE_PARAMS+=" -PABI_FILTERS=x86"
-fi
-
-if [ -n "{GRADLE_OFFLINE:-}" ]; then
-    GRADLE_PARAMS+=" --offline"
-fi
-
-# touch gradle cache files to prevent expiration
-while IFS= read -r -d '' file
-do
-  touch "$file" || true
-done < <(find /var/lib/jenkins/.gradle -type f -print0)
-
-env
-
-export GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
-rm -f $GRADLE_LOCAL_PROPERTIES
-echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-echo "cmake.dir=/usr/local" >> $GRADLE_LOCAL_PROPERTIES
-
-$GRADLE_PATH $GRADLE_PARAMS
-
-find . -type f -name "*.a" -exec ls -lh {} \;
-
-while IFS= read -r -d '' file
-do
-  echo
-  echo "$file"
-  ls -lah "$file"
-  zipinfo -l "$file"
-done < <(find . -type f -name '*.aar' -print0)
-
-find . -type f -name *aar -print | xargs tar cfvz ~/workspace/android/artifacts.tgz

.circleci/scripts/cpp_doc_push_script.sh

@@ -53,10 +53,11 @@ sudo apt-get -y install doxygen
 # Generate ATen files
 pushd "${pt_checkout}"
 pip install -r requirements.txt
-time python aten/src/ATen/gen.py \
+time GEN_TO_SOURCE=1 python aten/src/ATen/gen.py \
   -s aten/src/ATen \
   -d build/aten/src/ATen \
   aten/src/ATen/Declarations.cwrap \
+  aten/src/THNN/generic/THNN.h \
   aten/src/THCUNN/generic/THCUNN.h \
   aten/src/ATen/nn.yaml \
   aten/src/ATen/native/native_functions.yaml
@@ -72,10 +73,10 @@ time python tools/setup_helpers/generate_code.py \
 
 # Build the docs
 pushd docs/cpp
-pip install breathe==4.13.0 bs4 lxml six
+pip install breathe==4.11.1 bs4 lxml six
 pip install --no-cache-dir -e "git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme"
 pip install exhale>=0.2.1
-pip install sphinx==2.4.4
+pip install sphinx==1.8.5
 # Uncomment once it is fixed
 # pip install -r requirements.txt
 time make VERBOSE=1 html -j

.circleci/scripts/publish_android_snapshot.sh

@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-# DO NOT ADD 'set -x' not to reveal CircleCI secret context environment variables
-set -eu -o pipefail
-
-export ANDROID_NDK_HOME=/opt/ndk
-export ANDROID_HOME=/opt/android/sdk
-
-export GRADLE_VERSION=4.10.3
-export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
-export GRADLE_PATH=$GRADLE_HOME/bin/gradle
-
-echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
-ls -la ~/workspace
-
-GRADLE_PROPERTIES=~/workspace/android/gradle.properties
-
-IS_SNAPSHOT="$(grep 'VERSION_NAME=[0-9\.]\+-SNAPSHOT' "$GRADLE_PROPERTIES")"
-echo "IS_SNAPSHOT:$IS_SNAPSHOT"
-
-if [ -z "$IS_SNAPSHOT" ]; then
-  echo "Error: version is not snapshot."
-elif [ -z "$SONATYPE_NEXUS_USERNAME" ]; then
-  echo "Error: missing env variable SONATYPE_NEXUS_USERNAME."
-elif [ -z "$SONATYPE_NEXUS_PASSWORD" ]; then
-  echo "Error: missing env variable SONATYPE_NEXUS_PASSWORD."
-elif [ -z "$ANDROID_SIGN_KEY" ]; then
-  echo "Error: missing env variable ANDROID_SIGN_KEY."
-elif [ -z "$ANDROID_SIGN_PASS" ]; then
-  echo "Error: missing env variable ANDROID_SIGN_PASS."
-else
-  GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
-  rm -f $GRADLE_LOCAL_PROPERTIES
-
-  echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-  echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-
-  echo "SONATYPE_NEXUS_USERNAME=${SONATYPE_NEXUS_USERNAME}" >> $GRADLE_PROPERTIES
-  echo "SONATYPE_NEXUS_PASSWORD=${SONATYPE_NEXUS_PASSWORD}" >> $GRADLE_PROPERTIES
-
-  echo "signing.keyId=${ANDROID_SIGN_KEY}" >> $GRADLE_PROPERTIES
-  echo "signing.password=${ANDROID_SIGN_PASS}" >> $GRADLE_PROPERTIES
-
-  $GRADLE_PATH -p ~/workspace/android/ uploadArchives
-fi

.circleci/scripts/python_doc_push_script.sh

@@ -71,30 +71,8 @@ cp -a ../vision/docs/source source/torchvision
 # Build the docs
 pip -q install -r requirements.txt || true
 if [ "$is_master_doc" = true ]; then
-  # TODO: fix gh-38011 then enable this which changes warnings into errors
-  # export SPHINXOPTS="-WT --keep-going"
   make html
-  make coverage
-  # Now we have the coverage report, we need to make sure it is empty.
-  # Count the number of lines in the file and turn that number into a variable
-  # $lines. The `cut -f1 ...` is to only parse the number, not the filename
-  # Skip the report header by subtracting 2: the header will be output even if
-  # there are no undocumented items.
-  #
-  # Also: see docs/source/conf.py for "coverage_ignore*" items, which should
-  # be documented then removed from there.
-  lines=$(wc -l build/coverage/python.txt 2>/dev/null |cut -f1 -d' ')
-  undocumented=$(($lines - 2))
-  if [ $undocumented -lt 0 ]; then
-    echo coverage output not found
-    exit 1
-  elif [ $undocumented -gt 0 ]; then
-    echo undocumented objects found:
-    cat build/coverage/python.txt
-    exit 1
-  fi
 else
-  # Don't fail the build on coverage problems
   make html-stable
 fi
 
@@ -112,12 +90,6 @@ else
   find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>$version \&#x25BC</a>@g"
 fi
 
-# Prevent Google from indexing $install_path/_modules. This folder contains
-# generated source files.
-# NB: the following only works on gnu sed. The sed shipped with mac os is different.
-# One can `brew install gnu-sed` on a mac and then use "gsed" instead of "sed".
-find "$install_path/_modules" -name "*.html" -print0 | xargs -0 sed -i '/<head>/a \ \ <meta name="robots" content="noindex">'
-
 git add "$install_path" || true
 git status
 git config user.email "soumith+bot@pytorch.org"

.circleci/scripts/setup_ci_environment.sh

@@ -2,7 +2,7 @@
 set -ex -o pipefail
 
 # Set up NVIDIA docker repo
-curl -s -L --retry 3 https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
+curl -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
 echo "deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
 echo "deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
 echo "deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
@@ -13,15 +13,6 @@ sudo rm -f /etc/apt/heroku.list
 sudo rm -f /etc/apt/openjdk-r-ubuntu-ppa-xenial.list
 sudo rm -f /etc/apt/partner.list
 
-retry () {
-    $*  || $* || $* || $* || $*
-}
-
-# Method adapted from here: https://askubuntu.com/questions/875213/apt-get-to-retry-downloading
-# (with use of tee to avoid permissions problems)
-# This is better than retrying the whole apt-get command
-echo "APT::Acquire::Retries \"3\";" | sudo tee /etc/apt/apt.conf.d/80-retries
-
 sudo apt-get -y update
 sudo apt-get -y remove linux-image-generic linux-headers-generic linux-generic docker-ce
 # WARNING: Docker version is hardcoded here; you must update the
@@ -36,11 +27,7 @@ sudo apt-get -y remove linux-image-generic linux-headers-generic linux-generic d
 # Ubuntu version (e.g., docker run -it ubuntu:16.04) and then ask
 # apt what the packages you need are.  Note that the CircleCI image
 # comes with Docker.
-#
-# Using 'retry' here as belt-and-suspenders even though we are
-# presumably retrying at the single-package level via the
-# apt.conf.d/80-retries technique.
-retry sudo apt-get -y install \
+sudo apt-get -y install \
   linux-headers-$(uname -r) \
   linux-image-generic \
   moreutils \
@@ -51,11 +38,14 @@ retry sudo apt-get -y install \
 
 sudo pkill -SIGHUP dockerd
 
+retry () {
+    $*  || $* || $* || $* || $*
+}
 
 retry sudo pip -q install awscli==1.16.35
 
 if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  DRIVER_FN="NVIDIA-Linux-x86_64-440.59.run"
+  DRIVER_FN="NVIDIA-Linux-x86_64-410.104.run"
   wget "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"
   sudo /bin/bash "$DRIVER_FN" -s --no-drm || (sudo cat /var/log/nvidia-installer.log && false)
   nvidia-smi
@@ -64,6 +54,7 @@ fi
 if [[ "${BUILD_ENVIRONMENT}" == *-build ]]; then
   echo "declare -x IN_CIRCLECI=1" > /home/circleci/project/env
   echo "declare -x COMMIT_SOURCE=${CIRCLE_BRANCH:-}" >> /home/circleci/project/env
+  echo "declare -x PYTHON_VERSION=${PYTHON_VERSION:-}" >> /home/circleci/project/env
   echo "declare -x SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2" >> /home/circleci/project/env
   if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
     echo "declare -x TORCH_CUDA_ARCH_LIST=5.2" >> /home/circleci/project/env
@@ -76,14 +67,12 @@ if [[ "${BUILD_ENVIRONMENT}" == *-build ]]; then
   if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
     # This IAM user allows write access to S3 bucket for sccache & bazels3cache
     set +x
-    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
     echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
     echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
     set -x
   else
     # This IAM user allows write access to S3 bucket for sccache
     set +x
-    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
     echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
     echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
     set -x

.circleci/scripts/setup_linux_system_environment.sh

@@ -2,7 +2,7 @@
 set -eux -o pipefail
 
 # Set up CircleCI GPG keys for apt, if needed
-curl --retry 3 -s -L https://packagecloud.io/circleci/trusty/gpgkey | sudo apt-key add -
+curl -L https://packagecloud.io/circleci/trusty/gpgkey | sudo apt-key add -
 
 # Stop background apt updates.  Hypothetically, the kill should not
 # be necessary, because stop is supposed to send a kill signal to

.circleci/scripts/should_run_job.py

@@ -0,0 +1,96 @@
+import argparse
+import re
+import sys
+
+# Modify this variable if you want to change the set of default jobs
+# which are run on all pull requests.
+#
+# WARNING: Actually, this is a lie; we're currently also controlling
+# the set of jobs to run via the Workflows filters in CircleCI config.
+
+default_set = [
+    # PyTorch CPU
+    # Selected oldest Python 2 version to ensure Python 2 coverage
+    'pytorch-linux-trusty-py2.7.9',
+    # PyTorch CUDA
+    'pytorch-linux-xenial-cuda9-cudnn7-py3',
+    # PyTorch ASAN
+    'pytorch-linux-xenial-py3-clang5-asan',
+    # PyTorch DEBUG
+    'pytorch-linux-trusty-py3.6-gcc5.4',
+
+    # Caffe2 CPU
+    'caffe2-py2-mkl-ubuntu16.04',
+    # Caffe2 CUDA
+    'caffe2-py2-cuda9.1-cudnn7-ubuntu16.04',
+    # Caffe2 ONNX
+    'caffe2-onnx-py2-gcc5-ubuntu16.04',
+    'caffe2-onnx-py3.6-clang7-ubuntu16.04',
+    # Caffe2 Clang
+    'caffe2-py2-clang7-ubuntu16.04',
+    # Caffe2 CMake
+    'caffe2-cmake-cuda9.0-cudnn7-ubuntu16.04',
+
+    # Binaries
+    'manywheel 2.7mu cpu devtoolset7',
+    'libtorch 2.7m cpu devtoolset7',
+
+    # Caffe2 Android
+    'caffe2-py2-android-ubuntu16.04',
+    # Caffe2 OSX
+    'caffe2-py2-system-macos10.13',
+    # PyTorch OSX
+    'pytorch-macos-10.13-cuda9.2-cudnn7-py3',
+    # PyTorch Android
+    'pytorch-linux-xenial-py3-clang5-android-ndk-r19c',
+
+    # XLA
+    'pytorch-xla-linux-trusty-py3.6-gcc5.4',
+
+    # Other checks
+    'pytorch-short-perf-test-gpu',
+    'pytorch-python-doc-push',
+    'pytorch-cpp-doc-push',
+]
+
+# Takes in commit message to analyze via stdin
+#
+# This script will query Git and attempt to determine if we should
+# run the current CI job under question
+#
+# NB: Try to avoid hard-coding names here, so there's less place to update when jobs
+# are updated/renamed
+#
+# Semantics in the presence of multiple tags:
+#   - Let D be the set of default builds
+#   - Let S be the set of explicitly specified builds
+#   - Run S \/ D
+
+parser = argparse.ArgumentParser()
+parser.add_argument('build_environment')
+args = parser.parse_args()
+
+commit_msg = sys.stdin.read()
+
+# Matches anything that looks like [foo ci] or [ci foo] or [foo test]
+# or [test foo]
+RE_MARKER = re.compile(r'\[(?:([^ \[\]]+) )?(?:ci|test)(?: ([^ \[\]]+))?\]')
+
+markers = RE_MARKER.finditer(commit_msg)
+
+for m in markers:
+    if m.group(1) and m.group(2):
+        print("Unrecognized marker: {}".format(m.group(0)))
+        continue
+    spec = m.group(1) or m.group(2)
+    if spec in args.build_environment or spec == 'all':
+        print("Accepting {} due to commit marker {}".format(args.build_environment, m.group(0)))
+        sys.exit(0)
+
+for spec in default_set:
+    if spec in args.build_environment:
+        print("Accepting {} as part of default set".format(args.build_environment))
+        sys.exit(0)
+
+print("Rejecting {}".format(args.build_environment))
+sys.exit(1)

.circleci/scripts/should_run_job.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+set -exu -o pipefail
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+# Check if we should actually run
+echo "BUILD_ENVIRONMENT: ${BUILD_ENVIRONMENT:-}"
+echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
+if [ -z "${BUILD_ENVIRONMENT:-}" ]; then
+  echo "Cannot run should_run_job.sh if BUILD_ENVIRONMENT is not defined!"
+  echo "CircleCI scripts are probably misconfigured."
+  exit 1
+fi
+if ! [ -e "$SCRIPT_DIR/COMMIT_MSG" ]; then
+  echo "Cannot run should_run_job.sh if you don't have COMMIT_MSG"
+  echo "written out.  Are you perhaps running the wrong copy of this script?"
+  echo "You should be running the copy in ~/workspace; SCRIPT_DIR=$SCRIPT_DIR"
+  exit 1
+fi
+if [ -n "${CIRCLE_PULL_REQUEST:-}" ]; then
+  if [[ $CIRCLE_BRANCH != "ci-all/"* ]]; then
+    # Don't swallow "script doesn't exist
+    [ -e "$SCRIPT_DIR/should_run_job.py"  ]
+    if ! python "$SCRIPT_DIR/should_run_job.py" "${BUILD_ENVIRONMENT:-}" < "$SCRIPT_DIR/COMMIT_MSG" ; then
+      circleci step halt
+      exit
+    fi
+  fi
+fi

.circleci/scripts/upload_binary_size_to_scuba.py

@@ -1,145 +0,0 @@
-import glob
-import json
-import logging
-import os
-import os.path
-import pathlib
-import re
-import sys
-import time
-import zipfile
-
-import requests
-
-
-def get_size(file_dir):
-    try:
-        # we should only expect one file, if no, something is wrong
-        file_name = glob.glob(os.path.join(file_dir, "*"))[0]
-        return os.stat(file_name).st_size
-    except:
-        logging.exception(f"error getting file from: {file_dir}")
-        return 0
-
-
-def build_message(size):
-    pkg_type, py_ver, cu_ver, *_ = os.environ.get("BUILD_ENVIRONMENT", "").split() + [
-        None,
-        None,
-        None,
-    ]
-    os_name = os.uname()[0].lower()
-    if os_name == "darwin":
-        os_name = "macos"
-    return {
-        "normal": {
-            "os": os_name,
-            "pkg_type": pkg_type,
-            "py_ver": py_ver,
-            "cu_ver": cu_ver,
-            "pr": os.environ.get("CIRCLE_PR_NUMBER"),
-            "build_num": os.environ.get("CIRCLE_BUILD_NUM"),
-            "sha1": os.environ.get("CIRCLE_SHA1"),
-            "branch": os.environ.get("CIRCLE_BRANCH"),
-        },
-        "int": {
-            "time": int(time.time()),
-            "size": size,
-            "commit_time": int(os.environ.get("COMMIT_TIME", "0")),
-        },
-    }
-
-
-def send_message(messages):
-    access_token = os.environ.get("SCRIBE_GRAPHQL_ACCESS_TOKEN")
-    if not access_token:
-        raise ValueError("Can't find access token from environment variable")
-    url = "https://graph.facebook.com/scribe_logs"
-    r = requests.post(
-        url,
-        data={
-            "access_token": access_token,
-            "logs": json.dumps(
-                [
-                    {
-                        "category": "perfpipe_pytorch_binary_size",
-                        "message": json.dumps(message),
-                        "line_escape": False,
-                    }
-                    for message in messages
-                ]
-            ),
-        },
-    )
-    print(r.text)
-    r.raise_for_status()
-
-
-def report_android_sizes(file_dir):
-    def gen_sizes():
-        # we should only expect one file, if no, something is wrong
-        aar_files = list(pathlib.Path(file_dir).rglob("pytorch_android-*.aar"))
-        if len(aar_files) != 1:
-            logging.exception(f"error getting aar files from: {file_dir} / {aar_files}")
-            return
-
-        aar_file = aar_files[0]
-        zf = zipfile.ZipFile(aar_file)
-        for info in zf.infolist():
-            # Scan ".so" libs in `jni` folder. Examples:
-            # jni/arm64-v8a/libfbjni.so
-            # jni/arm64-v8a/libpytorch_jni.so
-            m = re.match(r"^jni/([^/]+)/(.*\.so)$", info.filename)
-            if not m:
-                continue
-            arch, lib = m.groups()
-            # report per architecture library size
-            yield [arch, lib, info.compress_size, info.file_size]
-
-        # report whole package size
-        yield ["aar", aar_file.name, os.stat(aar_file).st_size, 0]
-
-    def gen_messages():
-        android_build_type = os.environ.get("ANDROID_BUILD_TYPE")
-        for arch, lib, comp_size, uncomp_size in gen_sizes():
-            print(android_build_type, arch, lib, comp_size, uncomp_size)
-            yield {
-                "normal": {
-                    "os": "android",
-                    # TODO: create dedicated columns
-                    "pkg_type": "{}/{}/{}".format(android_build_type, arch, lib),
-                    "cu_ver": "",  # dummy value for derived field `build_name`
-                    "py_ver": "",  # dummy value for derived field `build_name`
-                    "pr": os.environ.get("CIRCLE_PR_NUMBER"),
-                    "build_num": os.environ.get("CIRCLE_BUILD_NUM"),
-                    "sha1": os.environ.get("CIRCLE_SHA1"),
-                    "branch": os.environ.get("CIRCLE_BRANCH"),
-                },
-                "int": {
-                    "time": int(time.time()),
-                    "commit_time": int(os.environ.get("COMMIT_TIME", "0")),
-                    "size": comp_size,
-                    "raw_size": uncomp_size,
-                },
-            }
-
-    send_message(list(gen_messages()))
-
-
-if __name__ == "__main__":
-    file_dir = os.environ.get(
-        "PYTORCH_FINAL_PACKAGE_DIR", "/home/circleci/project/final_pkgs"
-    )
-    if len(sys.argv) == 2:
-        file_dir = sys.argv[1]
-    print("checking dir: " + file_dir)
-
-    if "-android" in os.environ.get("BUILD_ENVIRONMENT", ""):
-        report_android_sizes(file_dir)
-    else:
-        size = get_size(file_dir)
-        if size != 0:
-            try:
-                send_message([build_message(size)])
-            except:
-                logging.exception("can't send message")

.circleci/scripts/vs_install.ps1

@@ -1,34 +0,0 @@
-$VS_DOWNLOAD_LINK = "https://aka.ms/vs/15/release/vs_buildtools.exe"
-$COLLECT_DOWNLOAD_LINK = "https://aka.ms/vscollect.exe"
-$VS_INSTALL_ARGS = @("--nocache","--quiet","--wait", "--add Microsoft.VisualStudio.Workload.VCTools",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Tools.14.11",
-                                                     "--add Microsoft.Component.MSBuild",
-                                                     "--add Microsoft.VisualStudio.Component.Roslyn.Compiler",
-                                                     "--add Microsoft.VisualStudio.Component.TextTemplating",
-                                                     "--add Microsoft.VisualStudio.Component.VC.CoreIde",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
-                                                     "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
-                                                     "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Win81")
-
-curl.exe --retry 3 -kL $VS_DOWNLOAD_LINK --output vs_installer.exe
-if ($LASTEXITCODE -ne 0) {
-    echo "Download of the VS 2017 installer failed"
-    exit 1
-}
-
-$process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_INSTALL_ARGS -NoNewWindow -Wait -PassThru
-Remove-Item -Path vs_installer.exe -Force
-$exitCode = $process.ExitCode
-if (($exitCode -ne 0) -and ($exitCode -ne 3010)) {
-    echo "VS 2017 installer exited with code $exitCode, which should be one of [0, 3010]."
-    curl.exe --retry 3 -kL $COLLECT_DOWNLOAD_LINK --output Collect.exe
-    if ($LASTEXITCODE -ne 0) {
-        echo "Download of the VS Collect tool failed."
-        exit 1
-    }
-    Start-Process "${PWD}\Collect.exe" -NoNewWindow -Wait -PassThru
-    New-Item -Path "C:\w\build-results" -ItemType "directory" -Force
-    Copy-Item -Path "C:\Users\circleci\AppData\Local\Temp\vslogs.zip" -Destination "C:\w\build-results\"
-    exit 1
-}

.circleci/scripts/windows_cuda_install.sh

@@ -1,37 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-curl --retry 3 -kLO https://ossci-windows.s3.amazonaws.com/cuda_10.1.243_426.00_win10.exe
-7z x cuda_10.1.243_426.00_win10.exe -ocuda_10.1.243_426.00_win10
-cd cuda_10.1.243_426.00_win10
-mkdir cuda_install_logs
-
-set +e
-
-./setup.exe -s nvcc_10.1 cuobjdump_10.1 nvprune_10.1 cupti_10.1 cublas_10.1 cublas_dev_10.1 cudart_10.1 cufft_10.1 cufft_dev_10.1 curand_10.1 curand_dev_10.1 cusolver_10.1 cusolver_dev_10.1 cusparse_10.1 cusparse_dev_10.1 nvgraph_10.1 nvgraph_dev_10.1 npp_10.1 npp_dev_10.1 nvrtc_10.1 nvrtc_dev_10.1 nvml_dev_10.1 -loglevel:6 -log:"$(pwd -W)/cuda_install_logs"
-
-set -e
-
-if [[ "${VC_YEAR}" == "2017" ]]; then
-    cp -r CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions/* "C:/Program Files (x86)/Microsoft Visual Studio/2017/${VC_PRODUCT}/Common7/IDE/VC/VCTargets/BuildCustomizations/"
-else
-    cp -r CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions/* "C:/Program Files (x86)/Microsoft Visual Studio/2019/${VC_PRODUCT}/MSBuild/Microsoft/VC/v160/BuildCustomizations/"
-fi
-
-curl --retry 3 -kLO https://ossci-windows.s3.amazonaws.com/NvToolsExt.7z
-7z x NvToolsExt.7z -oNvToolsExt
-mkdir -p "C:/Program Files/NVIDIA Corporation/NvToolsExt"
-cp -r NvToolsExt/* "C:/Program Files/NVIDIA Corporation/NvToolsExt/"
-export NVTOOLSEXT_PATH="C:\\Program Files\\NVIDIA Corporation\\NvToolsExt\\"
-
-if ! ls "/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin/nvcc.exe"
-then
-    echo "CUDA installation failed"
-    mkdir -p /c/w/build-results
-    7z a "c:\\w\\build-results\\cuda_install_logs.7z" cuda_install_logs
-    exit 1
-fi
-
-cd ..
-rm -rf ./cuda_10.1.243_426.00_win10
-rm -f ./cuda_10.1.243_426.00_win10.exe

.circleci/validate-docker-version.py

@@ -1,45 +0,0 @@
-#!/usr/bin/env python3
-import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
-import cimodel.data.simple.util.docker_constants as pytorch_docker_constants
-
-from yaml import load
-
-try:
-    from yaml import CLoader as Loader
-except ImportError:
-    from yaml import Loader
-
-
-def load_config(filename=".circleci/config.yml"):
-    with open(filename, "r") as fh:
-        return load("".join(fh.readlines()), Loader)
-
-
-def load_tags_for_projects(workflow_config):
-    return {
-        v["ecr_gc_job"]["project"]: v["ecr_gc_job"]["tags_to_keep"]
-        for v in workflow_config["workflows"]["ecr_gc"]["jobs"]
-        if isinstance(v, dict) and "ecr_gc_job" in v
-    }
-
-
-def check_version(job, tags, expected_version):
-    valid_versions = tags[job].split(",")
-    if expected_version not in valid_versions:
-        raise RuntimeError(
-            "We configured {} to use Docker version {}; but this "
-            "version is not configured in job ecr_gc_job_for_{}.  Non-deployed versions will be "
-            "garbage collected two weeks after they are created.  DO NOT LAND "
-            "THIS TO MASTER without also updating ossci-job-dsl with this version."
-            "\n\nDeployed versions: {}".format(job, expected_version, job, tags[job])
-        )
-
-
-def validate_docker_version():
-    tags = load_tags_for_projects(load_config())
-    check_version("pytorch", tags, pytorch_docker_constants.DOCKER_IMAGE_TAG)
-    check_version("caffe2", tags, caffe2_build_definitions.DOCKER_IMAGE_VERSION)
-
-
-if __name__ == "__main__":
-    validate_docker_version()

.circleci/verbatim-sources/binary-build-tests.yml

@@ -0,0 +1,20 @@
+
+# There is currently no testing for libtorch TODO
+#  binary_linux_libtorch_2.7m_cpu_test:
+#    environment:
+#      BUILD_ENVIRONMENT: "libtorch 2.7m cpu"
+#    resource_class: gpu.medium
+#    <<: *binary_linux_test
+#
+#  binary_linux_libtorch_2.7m_cu90_test:
+#    environment:
+#      BUILD_ENVIRONMENT: "libtorch 2.7m cu90"
+#    resource_class: gpu.medium
+#    <<: *binary_linux_test
+#
+#  binary_linux_libtorch_2.7m_cu100_test:
+#    environment:
+#      BUILD_ENVIRONMENT: "libtorch 2.7m cu100"
+#    resource_class: gpu.medium
+#    <<: *binary_linux_test
+

.circleci/verbatim-sources/binary_update_htmls.yml

@@ -0,0 +1,98 @@
+  
+  # update_s3_htmls job
+  # These jobs create html files for every cpu/cu## folder in s3. The html
+  # files just store the names of all the files in that folder (which are
+  # binary files (.whl files)). This is to allow pip installs of the latest
+  # version in a folder without having to know the latest date. Pip has a flag
+  # -f that you can pass an html file listing a bunch of packages, and pip will
+  # then install the one with the most recent version.
+  update_s3_htmls: &update_s3_htmls
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_workspace:
+        at: ~/workspace
+    - run:
+        <<: *setup_linux_system_environment
+    - run:
+        <<: *binary_checkout
+    # N.B. we do not run binary_populate_env. The only variable we need is
+    # PIP_UPLOAD_FOLDER (which is 'nightly/' for the nightlies and '' for
+    # releases, and sometimes other things for special cases). Instead we
+    # expect PIP_UPLOAD_FOLDER to be passed directly in the env. This is
+    # because, unlike all the other binary jobs, these jobs only get run once,
+    # in a separate workflow. They are not a step in other binary jobs like
+    # build, test, upload.
+    #
+    # You could attach this to every job, or include it in the upload step if
+    # you wanted. You would need to add binary_populate_env in this case to
+    # make sure it has the same upload folder as the job it's attached to. This
+    # function is idempotent, so it won't hurt anything; it's just a little
+    # unnescessary"
+    - run:
+        name: Update s3 htmls
+        no_output_timeout: "1h"
+        command: |
+          set +x
+          echo "declare -x \"AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}\"" >> /home/circleci/project/env
+          echo "declare -x \"AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}\"" >> /home/circleci/project/env
+          source /home/circleci/project/env
+          set -eux -o pipefail
+          retry () {
+              $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
+          }
+          retry pip install awscli==1.6
+          "/home/circleci/project/builder/cron/update_s3_htmls.sh"
+
+  # Update s3 htmls for the nightlies
+  update_s3_htmls_for_nightlies:
+    environment:
+      PIP_UPLOAD_FOLDER: "nightly/"
+    <<: *update_s3_htmls
+
+  # Update s3 htmls for the nightlies for devtoolset7
+  update_s3_htmls_for_nightlies_devtoolset7:
+    environment:
+      PIP_UPLOAD_FOLDER: "nightly/devtoolset7/"
+    <<: *update_s3_htmls
+
+
+  # upload_binary_logs job
+  # The builder hud at pytorch.org/builder shows the sizes of all the binaries
+  # over time. It gets this info from html files stored in S3, which this job
+  # populates every day.
+  upload_binary_sizes: &upload_binary_sizes
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_workspace:
+        at: ~/workspace
+    - run:
+        <<: *setup_linux_system_environment
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_install_miniconda
+    - run:
+        name: Upload binary sizes
+        no_output_timeout: "1h"
+        command: |
+          set +x
+          echo "declare -x \"AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}\"" > /home/circleci/project/env
+          echo "declare -x \"AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}\"" >> /home/circleci/project/env
+          export DATE="$(date -u +%Y_%m_%d)"
+          retry () {
+              $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
+          }
+          source /home/circleci/project/env
+          set -eux -o pipefail
+
+          # This is hardcoded to match binary_install_miniconda.sh
+          export PATH="/home/circleci/project/miniconda/bin:$PATH"
+          # Not any awscli will work. Most won't. This one will work
+          retry conda create -qyn aws36 python=3.6
+          source activate aws36
+          pip install awscli==1.16.46
+
+          "/home/circleci/project/builder/cron/upload_binary_sizes.sh"
+

.circleci/verbatim-sources/build-parameters/binary-build-params.yml

@@ -1,66 +0,0 @@
-binary_linux_build_params: &binary_linux_build_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    libtorch_variant:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "2xlarge+"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    LIBTORCH_VARIANT: << parameters.libtorch_variant >>
-    ANACONDA_USER: pytorch
-  resource_class: << parameters.resource_class >>
-  docker:
-    - image: << parameters.docker_image >>
-
-binary_linux_test_upload_params: &binary_linux_test_upload_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    libtorch_variant:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "medium"
-    use_cuda_docker_runtime:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    DOCKER_IMAGE: << parameters.docker_image >>
-    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
-    LIBTORCH_VARIANT: << parameters.libtorch_variant >>
-  resource_class: << parameters.resource_class >>
-
-binary_mac_params: &binary_mac_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-
-binary_windows_params: &binary_windows_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    executor:
-      type: string
-      default: "windows-cpu-with-nvidia-cuda"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    BUILD_FOR_SYSTEM: windows
-    JOB_EXECUTOR: <<parameters.executor>>

.circleci/verbatim-sources/build-parameters/caffe2-build-params.yml

@@ -1,27 +0,0 @@
-caffe2_params: &caffe2_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    build_ios:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    use_cuda_docker_runtime:
-      type: string
-      default: ""
-    build_only:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "large"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    BUILD_IOS: << parameters.build_ios >>
-    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
-    DOCKER_IMAGE: << parameters.docker_image >>
-    BUILD_ONLY: << parameters.build_only >>
-  resource_class: << parameters.resource_class >>

.circleci/verbatim-sources/build-parameters/promote-build-params.yml

@@ -1,14 +0,0 @@
-
-promote_common: &promote_common
-  docker:
-    - image: pytorch/release
-  parameters:
-    package_name:
-      description: "package name to promote"
-      type: string
-      default: ""
-  environment:
-    PACKAGE_NAME: << parameters.package_name >>
-    ANACONDA_API_TOKEN: ${CONDA_PYTORCHBOT_TOKEN}
-    AWS_ACCESS_KEY_ID: ${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}
-    AWS_SECRET_ACCESS_KEY: ${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}

.circleci/verbatim-sources/build-parameters/pytorch-build-params.yml

@@ -1,85 +0,0 @@
-pytorch_params: &pytorch_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "large"
-    use_cuda_docker_runtime:
-      type: string
-      default: ""
-    build_only:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    DOCKER_IMAGE: << parameters.docker_image >>
-    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
-    BUILD_ONLY: << parameters.build_only >>
-  resource_class: << parameters.resource_class >>
-
-pytorch_ios_params: &pytorch_ios_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    ios_arch:
-      type: string
-      default: ""
-    ios_platform:
-      type: string
-      default: ""
-    op_list:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    IOS_ARCH: << parameters.ios_arch >>
-    IOS_PLATFORM: << parameters.ios_platform >>
-    SELECTED_OP_LIST: << parameters.op_list >>
-
-pytorch_windows_params: &pytorch_windows_params
-  parameters:
-    executor:
-      type: string
-      default: "windows-cpu-with-nvidia-cuda"
-    build_environment:
-      type: string
-      default: ""
-    test_name:
-      type: string
-      default: ""
-    cuda_version:
-      type: string
-      default: "10"
-    python_version:
-      type: string
-      default: "3.6"
-    vc_version:
-      type: string
-      default: "14.11"
-    vc_year:
-      type: string
-      default: "2017"
-    vc_product:
-      type: string
-      default: "BuildTools"
-    use_cuda:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: <<parameters.build_environment>>
-    SCCACHE_BUCKET: "ossci-compiler-cache"
-    CUDA_VERSION: <<parameters.cuda_version>>
-    PYTHON_VERSION: <<parameters.python_version>>
-    VC_VERSION: <<parameters.vc_version>>
-    VC_YEAR: <<parameters.vc_year>>
-    VC_PRODUCT: <<parameters.vc_product>>
-    USE_CUDA: <<parameters.use_cuda>>
-    TORCH_CUDA_ARCH_LIST: "7.5"
-    JOB_BASE_NAME: <<parameters.test_name>>
-    JOB_EXECUTOR: <<parameters.executor>>

.circleci/verbatim-sources/commands.yml

@@ -1,133 +0,0 @@
-commands:
-
-  # Must be run after attaching workspace from previous steps
-  load_shared_env:
-    description: "Loads .circleci/shared/env_file into ${BASH_ENV}"
-    parameters:
-      # For some weird reason we decide to reattach our workspace to ~/workspace so
-      # in the vein of making it simple let's assume our share env_file is here
-      root:
-        type: string
-        default: "~/workspace"
-    steps:
-      - run:
-          name: "Load .circleci/shared/env_file into ${BASH_ENV}"
-          command: |
-            if [[ -f  "<< parameters.root >>/.circleci/shared/env_file" ]]; then
-              cat << parameters.root >>/.circleci/shared/env_file >> ${BASH_ENV}
-            else
-              echo "We didn't have a shared env file, that's weird"
-            fi
-
-  # This system setup script is meant to run before the CI-related scripts, e.g.,
-  # installing Git client, checking out code, setting up CI env, and
-  # building/testing.
-  setup_linux_system_environment:
-    steps:
-      - run:
-          name: Set Up System Environment
-          no_output_timeout: "1h"
-          command: .circleci/scripts/setup_linux_system_environment.sh
-
-  setup_ci_environment:
-    steps:
-      - run:
-          name: Set Up CI Environment After attach_workspace
-          no_output_timeout: "1h"
-          command: .circleci/scripts/setup_ci_environment.sh
-
-  brew_update:
-    description: "Update Homebrew and install base formulae"
-    steps:
-      - run:
-          name: Update Homebrew
-          no_output_timeout: "10m"
-          command: |
-            set -ex
-
-            # Update repositories manually.
-            # Running `brew update` produces a comparison between the
-            # current checkout and the updated checkout, which takes a
-            # very long time because the existing checkout is 2y old.
-            for path in $(find /usr/local/Homebrew -type d -name .git)
-            do
-            cd $path/..
-            git fetch --depth=1 origin
-            git reset --hard origin/master
-            done
-
-            export HOMEBREW_NO_AUTO_UPDATE=1
-
-            # Install expect and moreutils so that we can call `unbuffer` and `ts`.
-            # moreutils installs a `parallel` executable by default, which conflicts
-            # with the executable from the GNU `parallel`, so we must unlink GNU
-            # `parallel` first, and relink it afterwards.
-            brew unlink parallel
-            brew install moreutils
-            brew link parallel --overwrite
-            brew install expect
-
-  brew_install:
-    description: "Install Homebrew formulae"
-    parameters:
-      formulae:
-        type: string
-        default: ""
-    steps:
-      - run:
-          name: Install << parameters.formulae >>
-          no_output_timeout: "10m"
-          command: |
-            set -ex
-            export HOMEBREW_NO_AUTO_UPDATE=1
-            brew install << parameters.formulae >>
-
-  run_brew_for_macos_build:
-    steps:
-      - brew_update
-      - brew_install:
-          formulae: libomp
-
-  run_brew_for_ios_build:
-    steps:
-      - brew_update
-      - brew_install:
-          formulae: libtool
-
-  optional_merge_target_branch:
-    steps:
-      - run:
-          name: (Optional) Merge target branch
-          no_output_timeout: "10m"
-          command: |
-            if [ -n "$CIRCLE_PULL_REQUEST" ]; then
-              PR_NUM=$(basename $CIRCLE_PULL_REQUEST)
-              CIRCLE_PR_BASE_BRANCH=$(curl -s https://api.github.com/repos/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/pulls/$PR_NUM | jq -r '.base.ref')
-              if [[ "${BUILD_ENVIRONMENT}" == *"xla"* || "${BUILD_ENVIRONMENT}" == *"gcc5"* ]] ; then
-                set -x
-                git config --global user.email "circleci.ossci@gmail.com"
-                git config --global user.name "CircleCI"
-                git config remote.origin.url https://github.com/pytorch/pytorch.git
-                git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master
-                git fetch --tags --progress https://github.com/pytorch/pytorch.git +refs/heads/master:refs/remotes/origin/master --depth=100 --quiet
-                # PRs generated from ghstack has format CIRCLE_PR_BASE_BRANCH=gh/xxx/1234/base
-                if [[ "${CIRCLE_PR_BASE_BRANCH}" == "gh/"* ]]; then
-                  CIRCLE_PR_BASE_BRANCH=master
-                fi
-                export GIT_MERGE_TARGET=`git log -n 1 --pretty=format:"%H" origin/$CIRCLE_PR_BASE_BRANCH`
-                echo "GIT_MERGE_TARGET: " ${GIT_MERGE_TARGET}
-                export GIT_COMMIT=${CIRCLE_SHA1}
-                echo "GIT_COMMIT: " ${GIT_COMMIT}
-                git checkout -f ${GIT_COMMIT}
-                git reset --hard ${GIT_COMMIT}
-                git merge --allow-unrelated-histories --no-edit --no-ff ${GIT_MERGE_TARGET}
-                echo "Merged $CIRCLE_PR_BASE_BRANCH branch before building in environment $BUILD_ENVIRONMENT"
-                set +x
-              else
-                echo "No need to merge with $CIRCLE_PR_BASE_BRANCH, skipping..."
-              fi
-            else
-              echo "This is not a pull request, skipping..."
-            fi
-
-

.circleci/verbatim-sources/header-section.yml

@@ -1,16 +1,15 @@
 # WARNING: DO NOT EDIT THIS FILE DIRECTLY!!!
 # See the README.md in this directory.
 
-# IMPORTANT: To update Docker image version, please follow
-# the instructions at
-# https://github.com/pytorch/pytorch/wiki/Docker-image-build-on-CircleCI
-
-version: 2.1
-
-parameters:
-  run_binary_tests:
-    type: boolean
-    default: false
+# IMPORTANT: To update Docker image version, please first update
+# https://github.com/pytorch/ossci-job-dsl/blob/master/src/main/groovy/ossci/pytorch/DockerVersion.groovy and
+# https://github.com/pytorch/ossci-job-dsl/blob/master/src/main/groovy/ossci/caffe2/DockerVersion.groovy,
+# and then update DOCKER_IMAGE_VERSION at the top of the following files:
+# * cimodel/data/pytorch_build_definitions.py
+# * cimodel/data/caffe2_build_definitions.py
+# And the inline copies of the variable in
+# * verbatim-sources/job-specs-custom.yml
+#   (grep for DOCKER_IMAGE)
 
 docker_config_defaults: &docker_config_defaults
   user: jenkins
@@ -19,16 +18,46 @@ docker_config_defaults: &docker_config_defaults
     aws_access_key_id: ${CIRCLECI_AWS_ACCESS_KEY_FOR_ECR_READ_WRITE_V4}
     aws_secret_access_key: ${CIRCLECI_AWS_SECRET_KEY_FOR_ECR_READ_WRITE_V4}
 
-executors:
-  windows-with-nvidia-gpu:
-    machine:
-      resource_class: windows.gpu.nvidia.medium
-      image: windows-server-2019-nvidia:stable
-      shell: bash.exe
+# This system setup script is meant to run before the CI-related scripts, e.g.,
+# installing Git client, checking out code, setting up CI env, and
+# building/testing.
+setup_linux_system_environment: &setup_linux_system_environment
+  name: Set Up System Environment
+  no_output_timeout: "1h"
+  command: ~/workspace/.circleci/scripts/setup_linux_system_environment.sh
+
+# NB: This (and the command below) must be run after attaching
+# ~/workspace.  This is NOT the default working directory (that's
+# ~/project); this workspace is generated by the setup job.
+should_run_job: &should_run_job
+  name: Should Run Job After attach_workspace
+  no_output_timeout: "2m"
+  command: ~/workspace/.circleci/scripts/should_run_job.sh
+
+setup_ci_environment: &setup_ci_environment
+  name: Set Up CI Environment After attach_workspace
+  no_output_timeout: "1h"
+  command: ~/workspace/.circleci/scripts/setup_ci_environment.sh
+
+# Installs expect and moreutils so that we can call `unbuffer` and `ts`.
+# Also installs OpenMP
+# !!!!NOTE!!!! this is copied into a binary_macos_brew_update job which is the
+# same but does not install libomp. If you are changing this, consider if you
+# need to change that step as well.
+macos_brew_update: &macos_brew_update
+  name: Brew update and install moreutils, expect and libomp
+  no_output_timeout: "1h"
+  command: |
+    set -ex
+    # See https://discourse.brew.sh/t/fetching-homebrew-repos-is-slow/5374/3
+    brew untap caskroom/homebrew-cask
+    # moreutils installs a `parallel` executable by default, which conflicts
+    # with the executable from the GNU `parallel`, so we must unlink GNU
+    # `parallel` first, and relink it afterwards
+    brew update
+    brew unlink parallel
+    brew install moreutils
+    brew link parallel --overwrite
+    brew install expect
+    brew install libomp
 
-  windows-cpu-with-nvidia-cuda:
-    machine:
-      # we will change to CPU host when it's ready
-      resource_class: windows.xlarge
-      image: windows-server-2019-vs2019:stable
-      shell: bash.exe

.circleci/verbatim-sources/job-specs-custom.yml

@@ -0,0 +1,267 @@
+  pytorch_short_perf_test_gpu:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-short-perf-test-gpu
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:323"
+      PYTHON_VERSION: "3.6"
+      USE_CUDA_DOCKER_RUNTIME: "1"
+    resource_class: gpu.medium
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_workspace:
+        at: ~/workspace
+    - run:
+        <<: *should_run_job
+    - run:
+        <<: *setup_linux_system_environment
+    - run:
+        <<: *setup_ci_environment
+    - run:
+        name: Perf Test
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          export id=$(docker run --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+
+          docker cp $id:/var/lib/jenkins/workspace/env /home/circleci/project/env
+          # This IAM user allows write access to S3 bucket for perf test numbers
+          set +x
+          echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_PERF_TEST_S3_BUCKET_V4}" >> /home/circleci/project/env
+          echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_PERF_TEST_S3_BUCKET_V4}" >> /home/circleci/project/env
+          set -x
+          docker cp /home/circleci/project/env $id:/var/lib/jenkins/workspace/env
+
+          export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/short-perf-test-gpu.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+  pytorch_python_doc_push:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-python-doc-push
+      # TODO: stop hardcoding this
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:323"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_workspace:
+        at: ~/workspace
+    - run:
+        <<: *should_run_job
+    - run:
+        <<: *setup_linux_system_environment
+    - run:
+        <<: *setup_ci_environment
+    - run:
+        name: Doc Build and Push
+        no_output_timeout: "1h"
+        command: |
+          set -ex
+          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+
+          # master branch docs push
+          if [[ "${CIRCLE_BRANCH}" == "master" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/master master site") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # stable release docs push. Due to some circleci limitations, we keep
+          # an eternal PR open for merging v1.2.0 -> master for this job.
+          # XXX: The following code is only run on the v1.2.0 branch, which might
+          # not be exactly the same as what you see here.
+          elif [[ "${CIRCLE_BRANCH}" == "v1.2.0" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/stable 1.2.0 site-v1.2.0") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # For open PRs: Do a dry_run of the docs build, don't push build
+          else
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/master master site dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          fi
+
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Save the docs build so we can debug any problems
+          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
+          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
+          docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
+
+  pytorch_cpp_doc_push:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-cpp-doc-push
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:323"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - attach_workspace:
+        at: ~/workspace
+    - run:
+        <<: *should_run_job
+    - run:
+        <<: *setup_linux_system_environment
+    - run:
+        <<: *setup_ci_environment
+    - run:
+        name: Doc Build and Push
+        no_output_timeout: "1h"
+        command: |
+          set -ex
+          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+
+          # master branch docs push
+          if [[ "${CIRCLE_BRANCH}" == "master" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/master master") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # stable release docs push. Due to some circleci limitations, we keep
+          # an eternal PR open (#16502) for merging v1.0.1 -> master for this job.
+          # XXX: The following code is only run on the v1.0.1 branch, which might
+          # not be exactly the same as what you see here.
+          elif [[ "${CIRCLE_BRANCH}" == "v1.0.1" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/stable 1.0.1") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # For open PRs: Do a dry_run of the docs build, don't push build
+          else
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/master master dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          fi
+
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Save the docs build so we can debug any problems
+          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
+          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
+          docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
+
+  pytorch_macos_10_13_py3_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
+    macos:
+      xcode: "9.0"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - attach_workspace:
+          at: ~/workspace
+      - run:
+          <<: *should_run_job
+      - checkout
+      - run:
+          <<: *macos_brew_update
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+
+            export IN_CIRCLECI=1
+
+            # Install sccache
+            sudo curl https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
+            sudo chmod +x /usr/local/bin/sccache
+
+            export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
+            # This IAM user allows write access to S3 bucket for sccache
+            set +x
+            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            set -x
+
+            chmod a+x .jenkins/pytorch/macos-build.sh
+            unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
+
+            mkdir -p /Users/distiller/pytorch-ci-env/workspace
+
+            # copy with -a to preserve relative structure (e.g., symlinks), and be recursive
+            cp -a /Users/distiller/project/. /Users/distiller/pytorch-ci-env/workspace
+      - persist_to_workspace:
+          root: /Users/distiller/pytorch-ci-env
+          paths:
+            - "*"
+
+  pytorch_macos_10_13_py3_test:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
+    macos:
+      xcode: "9.0"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      # This workspace also carries binaries from the build job
+      - attach_workspace:
+          at: ~/workspace
+      - run:
+          <<: *should_run_job
+      - run:
+          <<: *macos_brew_update
+      - run:
+          name: Test
+          no_output_timeout: "1h"
+          command: |
+            set -e
+            export IN_CIRCLECI=1
+
+            # copy with -a to preserve relative structure (e.g., symlinks), and be recursive
+            # TODO: I'm not sure why we can't just run our job in
+            # ~/workspace and call it a day
+            # NB: Yes, you need workspace twice
+            cp -a ~/workspace/workspace/. /Users/distiller/project
+
+            chmod a+x .jenkins/pytorch/macos-test.sh
+            unbuffer .jenkins/pytorch/macos-test.sh 2>&1 | ts
+
+  pytorch_macos_10_13_cuda9_2_cudnn7_py3_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
+    macos:
+      xcode: "9.0"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - attach_workspace:
+          at: ~/workspace
+      - run:
+          <<: *should_run_job
+      - checkout
+      - run:
+          <<: *macos_brew_update
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+
+            export IN_CIRCLECI=1
+
+            # Install CUDA 9.2
+            sudo rm -rf ~/cuda_9.2.64_mac_installer.app || true
+            curl https://s3.amazonaws.com/ossci-macos/cuda_9.2.64_mac_installer.zip -o ~/cuda_9.2.64_mac_installer.zip
+            unzip ~/cuda_9.2.64_mac_installer.zip -d ~/
+            sudo ~/cuda_9.2.64_mac_installer.app/Contents/MacOS/CUDAMacOSXInstaller --accept-eula --no-window
+            sudo cp /usr/local/cuda/lib/libcuda.dylib /Developer/NVIDIA/CUDA-9.2/lib/libcuda.dylib
+            sudo rm -rf /usr/local/cuda || true
+
+            # Install cuDNN 7.1 for CUDA 9.2
+            curl https://s3.amazonaws.com/ossci-macos/cudnn-9.2-osx-x64-v7.1.tgz -o ~/cudnn-9.2-osx-x64-v7.1.tgz
+            rm -rf ~/cudnn-9.2-osx-x64-v7.1 && mkdir ~/cudnn-9.2-osx-x64-v7.1
+            tar -xzvf ~/cudnn-9.2-osx-x64-v7.1.tgz -C ~/cudnn-9.2-osx-x64-v7.1
+            sudo cp ~/cudnn-9.2-osx-x64-v7.1/cuda/include/cudnn.h /Developer/NVIDIA/CUDA-9.2/include/
+            sudo cp ~/cudnn-9.2-osx-x64-v7.1/cuda/lib/libcudnn* /Developer/NVIDIA/CUDA-9.2/lib/
+            sudo chmod a+r /Developer/NVIDIA/CUDA-9.2/include/cudnn.h /Developer/NVIDIA/CUDA-9.2/lib/libcudnn*
+
+            # Install sccache
+            sudo curl https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
+            sudo chmod +x /usr/local/bin/sccache
+            export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
+            # This IAM user allows write access to S3 bucket for sccache
+            set +x
+            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            set -x
+
+            git submodule sync && git submodule update -q --init --recursive
+            chmod a+x .jenkins/pytorch/macos-build.sh
+            unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts