move to eigenteam github for eigen submodule

* [docs] Update broadcasting and cuda semantics notes

* Update multiprocessing.rst

* address comments

* Address comments

.bazelrc

@@ -1,3 +0,0 @@
-build --copt=--std=c++14
-build --copt=-I.
-build --copt=-isystem --copt bazel-out/k8-fastbuild/bin

.bazelversion

@@ -1 +0,0 @@
-3.1.0

.circleci/.gitignore

@@ -1,2 +0,0 @@
-*.svg
-*.png

.circleci/README.md

@@ -1,504 +0,0 @@
-Structure of CI
-===============
-
-setup job:
-1. Does a git checkout
-2. Persists CircleCI scripts (everything in `.circleci`) into a workspace.  Why?
-   We don't always do a Git checkout on all subjobs, but we usually
-   still want to be able to call scripts one way or another in a subjob.
-   Persisting files this way lets us have access to them without doing a
-   checkout.  This workspace is conventionally mounted on `~/workspace`
-   (this is distinguished from `~/project`, which is the conventional
-   working directory that CircleCI will default to starting your jobs
-   in.)
-3. Write out the commit message to `.circleci/COMMIT_MSG`.  This is so
-   we can determine in subjobs if we should actually run the jobs or
-   not, even if there isn't a Git checkout.
-
-
-
-
-CircleCI configuration generator
-================================
-
-One may no longer make changes to the `.circleci/config.yml` file directly.
-Instead, one must edit these Python scripts or files in the `verbatim-sources/` directory.
-
-
-Usage
-----------
-
-1. Make changes to these scripts.
-2. Run the `regenerate.sh` script in this directory and commit the script changes and the resulting change to `config.yml`.
-
-You'll see a build failure on TravisCI if the scripts don't agree with the checked-in version.
-
-
-Motivation
-----------
-
-These scripts establish a single, authoritative source of documentation for the CircleCI configuration matrix.
-The documentation, in the form of diagrams, is automatically generated and cannot drift out of sync with the YAML content.
-
-Furthermore, consistency is enforced within the YAML config itself, by using a single source of data to generate
-multiple parts of the file.
-
-* Facilitates one-off culling/enabling of CI configs for testing PRs on special targets
-
-Also see https://github.com/pytorch/pytorch/issues/17038
-
-
-Future direction
-----------------
-
-### Declaring sparse config subsets
-See comment [here](https://github.com/pytorch/pytorch/pull/17323#pullrequestreview-206945747):
-
-In contrast with a full recursive tree traversal of configuration dimensions,
-> in the future future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
-
-----------------
-----------------
-
-# How do the binaries / nightlies / releases work?
-
-### What is a binary?
-
-A binary or package (used interchangeably) is a pre-built collection of c++ libraries, header files, python bits, and other files. We build these and distribute them so that users do not need to install from source.
-
-A **binary configuration** is a collection of
-
-* release or nightly
-    * releases are stable, nightlies are beta and built every night
-* python version
-    * linux: 3.5m, 3.6m 3.7m (mu is wide unicode or something like that. It usually doesn't matter but you should know that it exists)
-    * macos: 3.6, 3.7, 3.8
-    * windows: 3.6, 3.7, 3.8
-* cpu version
-    * cpu, cuda 9.0, cuda 10.0
-    * The supported cuda versions occasionally change
-* operating system
-    * Linux - these are all built on CentOS. There haven't been any problems in the past building on CentOS and using on Ubuntu
-    * MacOS
-    * Windows - these are built on Azure pipelines
-* devtoolset version (gcc compiler version)
-    * This only matters on Linux cause only Linux uses gcc. tldr is gcc made a backwards incompatible change from gcc 4.8 to gcc 5, because it had to change how it implemented std::vector and std::string
-
-### Where are the binaries?
-
-The binaries are built in CircleCI. There are nightly binaries built every night at 9pm PST (midnight EST) and release binaries corresponding to Pytorch releases, usually every few months.
-
-We have 3 types of binary packages
-
-* pip packages - nightlies are stored on s3 (pip install -f <a s3 url>). releases are stored in a pip repo (pip install torch) (ask Soumith about this)
-* conda packages - nightlies and releases are both stored in a conda repo. Nighty packages have a '_nightly' suffix
-* libtorch packages - these are zips of all the c++ libraries, header files, and sometimes dependencies. These are c++ only
-    * shared with dependencies (the only supported option for Windows)
-    * static with dependencies
-    * shared without dependencies
-    * static without dependencies
-
-All binaries are built in CircleCI workflows except Windows. There are checked-in workflows (committed into the .circleci/config.yml) to build the nightlies every night. Releases are built by manually pushing a PR that builds the suite of release binaries (overwrite the config.yml to build the release)
-
-# CircleCI structure of the binaries
-
-Some quick vocab:
-
-* A\**workflow** is a CircleCI concept; it is a DAG of '**jobs**'. ctrl-f 'workflows' on\https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml to see the workflows.
-* **jobs** are a sequence of '**steps**'
-* **steps** are usually just a bash script or a builtin CircleCI command.* All steps run in new environments, environment variables declared in one script DO NOT persist to following steps*
-* CircleCI has a **workspace**, which is essentially a cache between steps of the *same job* in which you can store artifacts between steps.
-
-## How are the workflows structured?
-
-The nightly binaries have 3 workflows. We have one job (actually 3 jobs:  build, test, and upload) per binary configuration
-
-1. binarybuilds
-    1. every day midnight EST
-    2. linux: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/linux-binary-build-defaults.yml
-    3. macos: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/macos-binary-build-defaults.yml
-    4. For each binary configuration, e.g. linux_conda_3.7_cpu there is a
-        1. binary_linux_conda_3.7_cpu_build
-            1. Builds the build. On linux jobs this uses the 'docker executor'.
-            2. Persists the package to the workspace
-        2. binary_linux_conda_3.7_cpu_test
-            1. Loads the package to the workspace
-            2. Spins up a docker image (on Linux), mapping the package and code repos into the docker
-            3. Runs some smoke tests in the docker
-            4. (Actually, for macos this is a step rather than a separate job)
-        3. binary_linux_conda_3.7_cpu_upload
-            1. Logs in to aws/conda
-            2. Uploads the package
-2. update_s3_htmls
-    1. every day 5am EST
-    2. https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/binary_update_htmls.yml
-    3. See below for what these are for and why they're needed
-    4. Three jobs that each examine the current contents of aws and the conda repo and update some html files in s3
-3. binarysmoketests
-    1. every day
-    2. https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/nightly-build-smoke-tests-defaults.yml
-    3. For each binary configuration, e.g. linux_conda_3.7_cpu there is a
-        1. smoke_linux_conda_3.7_cpu
-            1. Downloads the package from the cloud, e.g. using the official pip or conda instructions
-            2. Runs the smoke tests
-
-## How are the jobs structured?
-
-The jobs are in https://github.com/pytorch/pytorch/tree/master/.circleci/verbatim-sources . Jobs are made of multiple steps. There are some shared steps used by all the binaries/smokes. Steps of these jobs are all delegated to scripts in https://github.com/pytorch/pytorch/tree/master/.circleci/scripts .
-
-* Linux jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/linux-binary-build-defaults.yml
-    * binary_linux_build.sh
-    * binary_linux_test.sh
-    * binary_linux_upload.sh
-* MacOS jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/macos-binary-build-defaults.yml
-    * binary_macos_build.sh
-    * binary_macos_test.sh
-    * binary_macos_upload.sh
-* Update html jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/binary_update_htmls.yml
-    * These delegate from the pytorch/builder repo
-    * https://github.com/pytorch/builder/blob/master/cron/update_s3_htmls.sh
-    * https://github.com/pytorch/builder/blob/master/cron/upload_binary_sizes.sh
-* Smoke jobs (both linux and macos): https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/nightly-build-smoke-tests-defaults.yml
-    * These delegate from the pytorch/builder repo
-    * https://github.com/pytorch/builder/blob/master/run_tests.sh
-    * https://github.com/pytorch/builder/blob/master/smoke_test.sh
-    * https://github.com/pytorch/builder/blob/master/check_binary.sh
-* Common shared code (shared across linux and macos): https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/nightly-binary-build-defaults.yml
-    * binary_checkout.sh - checks out pytorch/builder repo. Right now this also checks out pytorch/pytorch, but it shouldn't. pytorch/pytorch should just be shared through the workspace. This can handle being run before binary_populate_env.sh
-    * binary_populate_env.sh - parses BUILD_ENVIRONMENT into the separate env variables that make up a binary configuration. Also sets lots of default values, the date, the version strings, the location of folders in s3, all sorts of things. This generally has to be run before other steps.
-    * binary_install_miniconda.sh - Installs miniconda, cross platform. Also hacks this for the update_binary_sizes job that doesn't have the right env variables
-    * binary_run_in_docker.sh - Takes a bash script file (the actual test code) from a hardcoded location, spins up a docker image, and runs the script inside the docker image
-
-### **Why do the steps all refer to scripts?**
-
-CircleCI creates a  final yaml file by inlining every <<* segment, so if we were to keep all the code in the config.yml itself then the config size would go over 4 MB and cause infra problems.
-
-### **What is binary_run_in_docker for?**
-
-So, CircleCI has several executor types: macos, machine, and docker are the ones we use. The 'machine' executor gives you two cores on some linux vm. The 'docker' executor gives you considerably more cores (nproc was 32 instead of 2 back when I tried in February). Since the dockers are faster, we try to run everything that we can in dockers. Thus
-
-* linux build jobs use the docker executor. Running them on the docker executor was at least 2x faster than running them on the machine executor
-* linux test jobs use the machine executor and spin up their own docker. Why this nonsense? It's cause we run nvidia-docker for our GPU tests; any code that calls into the CUDA runtime needs to be run on nvidia-docker. To run a nvidia-docker you need to install some nvidia packages on the host machine and then call docker with the '—runtime nvidia' argument. CircleCI doesn't support this, so we have to do it ourself.
-    * This is not just a mere inconvenience. **This blocks all of our linux tests from using more than 2 cores.** But there is nothing that we can do about it, but wait for a fix on circleci's side. Right now, we only run some smoke tests (some simple imports) on the binaries, but this also affects non-binary test jobs.
-* linux upload jobs use the machine executor. The upload jobs are so short that it doesn't really matter what they use
-* linux smoke test jobs use the machine executor for the same reason as the linux test jobs
-
-binary_run_in_docker.sh is a way to share the docker start-up code between the binary test jobs and the binary smoke test jobs
-
-### **Why does binary_checkout also checkout pytorch? Why shouldn't it?**
-
-We want all the nightly binary jobs to run on the exact same git commit, so we wrote our own checkout logic to ensure that the same commit was always picked. Later circleci changed that to use a single pytorch checkout and persist it through the workspace (they did this because our config file was too big, so they wanted to take a lot of the setup code into scripts, but the scripts needed the code repo to exist to be called, so they added a prereq step called 'setup' to checkout the code and persist the needed scripts to the workspace). The changes to the binary jobs were not properly tested, so they all broke from missing pytorch code no longer existing. We hotfixed the problem by adding the pytorch checkout back to binary_checkout, so now there's two checkouts of pytorch on the binary jobs. This problem still needs to be fixed, but it takes careful tracing of which code is being called where.
-
-# Azure Pipelines structure of the binaries
-
-TODO: fill in stuff
-
-## How are the workflows structured?
-
-TODO: fill in stuff
-
-## How are the jobs structured?
-
-TODO: fill in stuff
-
-# Code structure of the binaries (circleci agnostic)
-
-## Overview
-
-The code that runs the binaries lives in two places, in the normal [github.com/pytorch/pytorch](http://github.com/pytorch/pytorch), but also in [github.com/pytorch/builder](http://github.com/pytorch/builder) , which is a repo that defines how all the binaries are built. The relevant code is
-
-
-```
-# All code needed to set-up environments for build code to run in,
-# but only code that is specific to the current CI system
-pytorch/pytorch
-- .circleci/                # Folder that holds all circleci related stuff
-  - config.yml              # GENERATED file that actually controls all circleci behavior
-  - verbatim-sources        # Used to generate job/workflow sections in ^
-  - scripts/                # Code needed to prepare circleci environments for binary build scripts
-
-- setup.py                  # Builds pytorch. This is wrapped in pytorch/builder
-- cmake files               # used in normal building of pytorch
-
-# All code needed to prepare a binary build, given an environment
-# with all the right variables/packages/paths.
-pytorch/builder
-
-# Given an installed binary and a proper python env, runs some checks
-# to make sure the binary was built the proper way. Checks things like
-# the library dependencies, symbols present, etc.
-- check_binary.sh
-
-# Given an installed binary, runs python tests to make sure everything
-# is in order. These should be de-duped. Right now they both run smoke
-# tests, but are called from different places. Usually just call some
-# import statements, but also has overlap with check_binary.sh above
-- run_tests.sh
-- smoke_test.sh
-
-# Folders that govern how packages are built. See paragraphs below
-
-- conda/
-  - build_pytorch.sh          # Entrypoint. Delegates to proper conda build folder
-  - switch_cuda_version.sh    # Switches activate CUDA installation in Docker
-  - pytorch-nightly/          # Build-folder
-- manywheel/
-  - build_cpu.sh              # Entrypoint for cpu builds
-  - build.sh                  # Entrypoint for CUDA builds
-  - build_common.sh           # Actual build script that ^^ call into
-- wheel/
-  - build_wheel.sh            # Entrypoint for wheel builds
-- windows/
-  - build_pytorch.bat         # Entrypoint for wheel builds on Windows
-```
-
-Every type of package has an entrypoint build script that handles the all the important logic.
-
-## Conda
-
-Linux, MacOS and Windows use the same code flow for the conda builds.
-
-Conda packages are built with conda-build, see https://conda.io/projects/conda-build/en/latest/resources/commands/conda-build.html
-
-Basically, you pass `conda build` a build folder (pytorch-nightly/ above) that contains a build script and a meta.yaml. The meta.yaml specifies in what python environment to build the package in, and what dependencies the resulting package should have, and the build script gets called in the env to build the thing.
-tldr; on conda-build is
-
-1. Creates a brand new conda environment, based off of deps in the meta.yaml
-    1. Note that environment variables do not get passed into this build env unless they are specified in the meta.yaml
-    2. If the build fails this environment will stick around. You can activate it for much easier debugging. The “General Python” section below explains what exactly a python “environment” is.
-2. Calls build.sh in the environment
-3. Copies the finished package to a new conda env, also specified by the meta.yaml
-4. Runs some simple import tests (if specified in the meta.yaml)
-5. Saves the finished package as a tarball
-
-The build.sh we use is essentially a wrapper around ```python setup.py build``` , but it also manually copies in some of our dependent libraries into the resulting tarball and messes with some rpaths.
-
-The entrypoint file `builder/conda/build_conda.sh` is complicated because
-
-* It works for Linux, MacOS and Windows
-    * The mac builds used to create their own environments, since they all used to be on the same machine. There’s now a lot of extra logic to handle conda envs. This extra machinery could be removed
-* It used to handle testing too, which adds more logic messing with python environments too. This extra machinery could be removed.
-
-## Manywheels (linux pip and libtorch packages)
-
-Manywheels are pip packages for linux distros. Note that these manywheels are not actually manylinux compliant.
-
-`builder/manywheel/build_cpu.sh` and `builder/manywheel/build.sh` (for CUDA builds) just set different env vars and then call into `builder/manywheel/build_common.sh`
-
-The entrypoint file `builder/manywheel/build_common.sh` is really really complicated because
-
-* This used to handle building for several different python versions at the same time. The loops have been removed, but there's still unnecessary folders and movements here and there.
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This used to handle testing the pip packages too. This is why there’s testing code at the end that messes with python installations and stuff
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This also builds libtorch packages
-    * This should really be separate. libtorch packages are c++ only and have no python. They should not share infra with all the python specific stuff in this file.
-* There is a lot of messing with rpaths. This is necessary, but could be made much much simpler if the above issues were fixed.
-
-## Wheels (MacOS pip and libtorch packages)
-
-The entrypoint file `builder/wheel/build_wheel.sh` is complicated because
-
-* The mac builds used to all run on one machine (we didn’t have autoscaling mac machines till circleci). So this script handled siloing itself by setting-up and tearing-down its build env and siloing itself into its own build directory.
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This also builds libtorch packages
-    * Ditto the comment above. This should definitely be separated out.
-
-Note that the MacOS Python wheels are still built in conda environments. Some of the dependencies present during build also come from conda.
-
-## Windows Wheels (Windows pip and libtorch packages)
-
-The entrypoint file `builder/windows/build_pytorch.bat` is complicated because
-
-* This used to handle building for several different python versions at the same time. This is why there are loops everywhere
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This used to handle testing the pip packages too. This is why there’s testing code at the end that messes with python installations and stuff
-    * The script is never used this way anymore. This extra machinery could be removed.
-* This also builds libtorch packages
-    * This should really be separate. libtorch packages are c++ only and have no python. They should not share infra with all the python specific stuff in this file.
-
-Note that the Windows Python wheels are still built in conda environments. Some of the dependencies present during build also come from conda.
-
-## General notes
-
-### Note on run_tests.sh, smoke_test.sh, and check_binary.sh
-
-* These should all be consolidated
-* These must run on all OS types: MacOS, Linux, and Windows
-* These all run smoke tests at the moment. They inspect the packages some, maybe run a few import statements. They DO NOT run the python tests nor the cpp tests. The idea is that python tests on master and PR merges will catch all breakages. All these tests have to do is make sure the special binary machinery didn’t mess anything up.
-* There are separate run_tests.sh and smoke_test.sh because one used to be called by the smoke jobs and one used to be called by the binary test jobs (see circleci structure section above). This is still true actually, but these could be united into a single script that runs these checks, given an installed pytorch package.
-
-### Note on libtorch
-
-Libtorch packages are built in the wheel build scripts: manywheel/build_*.sh for linux and build_wheel.sh for mac. There are several things wrong with this
-
-* It’s confusing. Most of those scripts deal with python specifics.
-* The extra conditionals everywhere severely complicate the wheel build scripts
-* The process for building libtorch is different from the official instructions (a plain call to cmake, or a call to a script)
-
-### Note on docker images / Dockerfiles
-
-All linux builds occur in docker images. The docker images are
-
-* pytorch/conda-cuda
-    * Has ALL CUDA versions installed. The script pytorch/builder/conda/switch_cuda_version.sh sets /usr/local/cuda to a symlink to e.g. /usr/local/cuda-10.0 to enable different CUDA builds
-    * Also used for cpu builds
-* pytorch/manylinux-cuda90
-* pytorch/manylinux-cuda92
-* pytorch/manylinux-cuda100
-    * Also used for cpu builds
-
-The Dockerfiles are available in pytorch/builder, but there is no circleci job or script to build these docker images, and they cannot be run locally (unless you have the correct local packages/paths). Only Soumith can build them right now.
-
-### General Python
-
-* This is still a good explanation of python installations https://caffe2.ai/docs/faq.html#why-do-i-get-import-errors-in-python-when-i-try-to-use-caffe2
-
-# How to manually rebuild the binaries
-
-tldr; make a PR that looks like https://github.com/pytorch/pytorch/pull/21159
-
-Sometimes we want to push a change to master and then rebuild all of today's binaries after that change. As of May 30, 2019 there isn't a way to manually run a workflow in the UI. You can manually re-run a workflow, but it will use the exact same git commits as the first run and will not include any changes. So we have to make a PR and then force circleci to run the binary workflow instead of the normal tests. The above PR is an example of how to do this; essentially you copy-paste the binarybuilds workflow steps into the default workflow steps. If you need to point the builder repo to a different commit then you'd need to change https://github.com/pytorch/pytorch/blob/master/.circleci/scripts/binary_checkout.sh#L42-L45 to checkout what you want.
-
-## How to test changes to the binaries via .circleci
-
-Writing PRs that test the binaries is annoying, since the default circleci jobs that run on PRs are not the jobs that you want to run. Likely, changes to the binaries will touch something under .circleci/ and require that .circleci/config.yml be regenerated (.circleci/config.yml controls all .circleci behavior, and is generated using ```.circleci/regenerate.sh``` in python 3.7). But you also need to manually hardcode the binary jobs that you want to test into the .circleci/config.yml workflow, so you should actually make at least two commits, one for your changes and one to temporarily hardcode jobs. See https://github.com/pytorch/pytorch/pull/22928 as an example of how to do this.
-
-```
-# Make your changes
-touch .circleci/verbatim-sources/nightly-binary-build-defaults.yml
-
-# Regenerate the yaml, has to be in python 3.7
-.circleci/regenerate.sh
-
-# Make a commit
-git add .circleci *
-git commit -m "My real changes"
-git push origin my_branch
-
-# Now hardcode the jobs that you want in the .circleci/config.yml workflows section
-# Also eliminate ensure-consistency and should_run_job checks
-# e.g. https://github.com/pytorch/pytorch/commit/2b3344bfed8772fe86e5210cc4ee915dee42b32d
-
-# Make a commit you won't keep
-git add .circleci
-git commit -m "[DO NOT LAND] testing binaries for above changes"
-git push origin my_branch
-
-# Now you need to make some changes to the first commit.
-git rebase -i HEAD~2 # mark the first commit as 'edit'
-
-# Make the changes
-touch .circleci/verbatim-sources/nightly-binary-build-defaults.yml
-.circleci/regenerate.sh
-
-# Ammend the commit and recontinue
-git add .circleci
-git commit --amend
-git rebase --continue
-
-# Update the PR, need to force since the commits are different now
-git push origin my_branch --force
-```
-
-The advantage of this flow is that you can make new changes to the base commit and regenerate the .circleci without having to re-write which binary jobs you want to test on. The downside is that all updates will be force pushes.
-
-## How to build a binary locally
-
-### Linux
-
-You can build Linux binaries locally easily using docker.
-
-```
-# Run the docker
-# Use the correct docker image, pytorch/conda-cuda used here as an example
-#
-# -v path/to/foo:path/to/bar makes path/to/foo on your local machine (the
-#    machine that you're running the command on) accessible to the docker
-#    container at path/to/bar. So if you then run `touch path/to/bar/baz`
-#    in the docker container then you will see path/to/foo/baz on your local
-#    machine. You could also clone the pytorch and builder repos in the docker.
-#
-# If you're building a CUDA binary then use `nvidia-docker run` instead, see below.
-#
-# If you know how, add ccache as a volume too and speed up everything
-docker run \
-    -v your/pytorch/repo:/pytorch \
-    -v your/builder/repo:/builder \
-    -v where/you/want/packages/to/appear:/final_pkgs \
-    -it pytorch/conda-cuda /bin/bash
-
-# Export whatever variables are important to you. All variables that you'd
-# possibly need are in .circleci/scripts/binary_populate_env.sh
-# You should probably always export at least these 3 variables
-export PACKAGE_TYPE=conda
-export DESIRED_PYTHON=3.6
-export DESIRED_CUDA=cpu
-
-# Call the entrypoint
-# `|& tee foo.log` just copies all stdout and stderr output to foo.log
-# The builds generate lots of output so you probably need this when
-# building locally.
-/builder/conda/build_pytorch.sh |& tee build_output.log
-```
-
-**Building CUDA binaries on docker**
-
-To build a CUDA binary you need to use `nvidia-docker run` instead of just `docker run` (or you can manually pass `--runtime=nvidia`). This adds some needed libraries and things to build CUDA stuff.
-
-You can build CUDA binaries on CPU only machines, but you can only run CUDA binaries on CUDA machines. This means that you can build a CUDA binary on a docker on your laptop if you so choose (though it’s gonna take a loong time).
-
-For Facebook employees, ask about beefy machines that have docker support and use those instead of your laptop; it will be 5x as fast.
-
-### MacOS
-
-There’s no easy way to generate reproducible hermetic MacOS environments. If you have a Mac laptop then you can try emulating the .circleci environments as much as possible, but you probably have packages in /usr/local/, possibly installed by brew, that will probably interfere with the build. If you’re trying to repro an error on a Mac build in .circleci and you can’t seem to repro locally, then my best advice is actually to iterate on .circleci    :/
-
-But if you want to try, then I’d recommend
-
-```
-# Create a new terminal
-# Clear your LD_LIBRARY_PATH and trim as much out of your PATH as you
-# know how to do
-
-# Install a new miniconda
-# First remove any other python or conda installation from your PATH
-# Always install miniconda 3, even if building for Python <3
-new_conda="~/my_new_conda"
-conda_sh="$new_conda/install_miniconda.sh"
-curl -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
-chmod +x "$conda_sh"
-"$conda_sh" -b -p "$MINICONDA_ROOT"
-rm -f "$conda_sh"
-export PATH="~/my_new_conda/bin:$PATH"
-
-# Create a clean python env
-# All MacOS builds use conda to manage the python env and dependencies
-# that are built with, even the pip packages
-conda create -yn binary python=2.7
-conda activate binary
-
-# Export whatever variables are important to you. All variables that you'd
-# possibly need are in .circleci/scripts/binary_populate_env.sh
-# You should probably always export at least these 3 variables
-export PACKAGE_TYPE=conda
-export DESIRED_PYTHON=3.6
-export DESIRED_CUDA=cpu
-
-# Call the entrypoint you want
-path/to/builder/wheel/build_wheel.sh
-```
-
-N.B. installing a brand new miniconda is important. This has to do with how conda installations work. See the “General Python” section above, but tldr; is that
-
-1. You make the ‘conda’ command accessible by prepending `path/to/conda_root/bin` to your PATH.
-2. You make a new env and activate it, which then also gets prepended to your PATH. Now you have `path/to/conda_root/envs/new_env/bin:path/to/conda_root/bin:$PATH`
-3. Now say you (or some code that you ran) call python executable `foo`
-    1. if you installed `foo` in `new_env`, then `path/to/conda_root/envs/new_env/bin/foo` will get called, as expected.
-    2. But if you forgot to installed `foo` in `new_env` but happened to previously install it in your root conda env (called ‘base’), then unix/linux will still find `path/to/conda_root/bin/foo` . This is dangerous, since `foo` can be a different version than you want; `foo` can even be for an incompatible python version!
-
-Newer conda versions and proper python hygiene can prevent this, but just install a new miniconda to be safe.
-
-### Windows
-
-TODO: fill in

.circleci/cimodel/data/binary_build_data.py

@@ -1,188 +0,0 @@
-"""
-This module models the tree of configuration variants
-for "smoketest" builds.
-
-Each subclass of ConfigNode represents a layer of the configuration hierarchy.
-These tree nodes encapsulate the logic for whether a branch of the hierarchy
-should be "pruned".
-"""
-
-from collections import OrderedDict
-
-from cimodel.lib.conf_tree import ConfigNode
-import cimodel.data.dimensions as dimensions
-
-
-LINKING_DIMENSIONS = [
-    "shared",
-    "static",
-]
-
-
-DEPS_INCLUSION_DIMENSIONS = [
-    "with-deps",
-    "without-deps",
-]
-
-
-def get_processor_arch_name(cuda_version):
-    return "cpu" if not cuda_version else "cu" + cuda_version
-
-
-LINUX_PACKAGE_VARIANTS = OrderedDict(
-    manywheel=[
-        "3.6m",
-        "3.7m",
-        "3.8m",
-    ],
-    conda=dimensions.STANDARD_PYTHON_VERSIONS,
-    libtorch=[
-        "3.7m",
-    ],
-)
-
-CONFIG_TREE_DATA = OrderedDict(
-    linux=(dimensions.CUDA_VERSIONS, LINUX_PACKAGE_VARIANTS),
-    macos=([None], OrderedDict(
-        wheel=dimensions.STANDARD_PYTHON_VERSIONS,
-        conda=dimensions.STANDARD_PYTHON_VERSIONS,
-        libtorch=[
-            "3.7",
-        ],
-    )),
-    windows=(dimensions.CUDA_VERSIONS, OrderedDict(
-        wheel=dimensions.STANDARD_PYTHON_VERSIONS,
-        conda=dimensions.STANDARD_PYTHON_VERSIONS,
-        libtorch=[
-            "3.7",
-        ],
-    )),
-)
-
-# GCC config variants:
-#
-# All the nightlies (except libtorch with new gcc ABI) are built with devtoolset7,
-# which can only build with old gcc ABI. It is better than devtoolset3
-# because it understands avx512, which is needed for good fbgemm performance.
-#
-# Libtorch with new gcc ABI is built with gcc 5.4 on Ubuntu 16.04.
-LINUX_GCC_CONFIG_VARIANTS = OrderedDict(
-    manywheel=['devtoolset7'],
-    conda=['devtoolset7'],
-    libtorch=[
-        "devtoolset7",
-        "gcc5.4_cxx11-abi",
-    ],
-)
-
-WINDOWS_LIBTORCH_CONFIG_VARIANTS = [
-    "debug",
-    "release",
-]
-
-
-class TopLevelNode(ConfigNode):
-    def __init__(self, node_name, config_tree_data, smoke):
-        super(TopLevelNode, self).__init__(None, node_name)
-
-        self.config_tree_data = config_tree_data
-        self.props["smoke"] = smoke
-
-    def get_children(self):
-        return [OSConfigNode(self, x, c, p) for (x, (c, p)) in self.config_tree_data.items()]
-
-
-class OSConfigNode(ConfigNode):
-    def __init__(self, parent, os_name, cuda_versions, py_tree):
-        super(OSConfigNode, self).__init__(parent, os_name)
-
-        self.py_tree = py_tree
-        self.props["os_name"] = os_name
-        self.props["cuda_versions"] = cuda_versions
-
-    def get_children(self):
-        return [PackageFormatConfigNode(self, k, v) for k, v in self.py_tree.items()]
-
-
-class PackageFormatConfigNode(ConfigNode):
-    def __init__(self, parent, package_format, python_versions):
-        super(PackageFormatConfigNode, self).__init__(parent, package_format)
-
-        self.props["python_versions"] = python_versions
-        self.props["package_format"] = package_format
-
-    def get_children(self):
-        if self.find_prop("os_name") == "linux":
-            return [LinuxGccConfigNode(self, v) for v in LINUX_GCC_CONFIG_VARIANTS[self.find_prop("package_format")]]
-        elif self.find_prop("os_name") == "windows" and self.find_prop("package_format") == "libtorch":
-            return [WindowsLibtorchConfigNode(self, v) for v in WINDOWS_LIBTORCH_CONFIG_VARIANTS]
-        else:
-            return [ArchConfigNode(self, v) for v in self.find_prop("cuda_versions")]
-
-
-class LinuxGccConfigNode(ConfigNode):
-    def __init__(self, parent, gcc_config_variant):
-        super(LinuxGccConfigNode, self).__init__(parent, "GCC_CONFIG_VARIANT=" + str(gcc_config_variant))
-
-        self.props["gcc_config_variant"] = gcc_config_variant
-
-    def get_children(self):
-        cuda_versions = self.find_prop("cuda_versions")
-
-        # XXX devtoolset7 on CUDA 9.0 is temporarily disabled
-        # see https://github.com/pytorch/pytorch/issues/20066
-        if self.find_prop("gcc_config_variant") == 'devtoolset7':
-            cuda_versions = filter(lambda x: x != "90", cuda_versions)
-
-        return [ArchConfigNode(self, v) for v in cuda_versions]
-
-
-class WindowsLibtorchConfigNode(ConfigNode):
-    def __init__(self, parent, libtorch_config_variant):
-        super(WindowsLibtorchConfigNode, self).__init__(parent, "LIBTORCH_CONFIG_VARIANT=" + str(libtorch_config_variant))
-
-        self.props["libtorch_config_variant"] = libtorch_config_variant
-
-    def get_children(self):
-        return [ArchConfigNode(self, v) for v in self.find_prop("cuda_versions")]
-
-
-class ArchConfigNode(ConfigNode):
-    def __init__(self, parent, cu):
-        super(ArchConfigNode, self).__init__(parent, get_processor_arch_name(cu))
-
-        self.props["cu"] = cu
-
-    def get_children(self):
-        return [PyVersionConfigNode(self, v) for v in self.find_prop("python_versions")]
-
-
-class PyVersionConfigNode(ConfigNode):
-    def __init__(self, parent, pyver):
-        super(PyVersionConfigNode, self).__init__(parent, pyver)
-
-        self.props["pyver"] = pyver
-
-    def get_children(self):
-        package_format = self.find_prop("package_format")
-        os_name = self.find_prop("os_name")
-
-        has_libtorch_variants = package_format == "libtorch" and os_name == "linux"
-        linking_variants = LINKING_DIMENSIONS if has_libtorch_variants else []
-
-        return [LinkingVariantConfigNode(self, v) for v in linking_variants]
-
-
-class LinkingVariantConfigNode(ConfigNode):
-    def __init__(self, parent, linking_variant):
-        super(LinkingVariantConfigNode, self).__init__(parent, linking_variant)
-
-    def get_children(self):
-        return [DependencyInclusionConfigNode(self, v) for v in DEPS_INCLUSION_DIMENSIONS]
-
-
-class DependencyInclusionConfigNode(ConfigNode):
-    def __init__(self, parent, deps_variant):
-        super(DependencyInclusionConfigNode, self).__init__(parent, deps_variant)
-
-        self.props["libtorch_variant"] = "-".join([self.parent.get_label(), self.get_label()])

.circleci/cimodel/data/binary_build_definitions.py

@@ -1,210 +0,0 @@
-from collections import OrderedDict
-
-import cimodel.data.simple.util.branch_filters as branch_filters
-import cimodel.data.binary_build_data as binary_build_data
-import cimodel.lib.conf_tree as conf_tree
-import cimodel.lib.miniutils as miniutils
-
-class Conf(object):
-    def __init__(self, os, cuda_version, pydistro, parms, smoke, libtorch_variant, gcc_config_variant, libtorch_config_variant):
-
-        self.os = os
-        self.cuda_version = cuda_version
-        self.pydistro = pydistro
-        self.parms = parms
-        self.smoke = smoke
-        self.libtorch_variant = libtorch_variant
-        self.gcc_config_variant = gcc_config_variant
-        self.libtorch_config_variant = libtorch_config_variant
-
-    def gen_build_env_parms(self):
-        elems = [self.pydistro] + self.parms + [binary_build_data.get_processor_arch_name(self.cuda_version)]
-        if self.gcc_config_variant is not None:
-            elems.append(str(self.gcc_config_variant))
-        if self.libtorch_config_variant is not None:
-            elems.append(str(self.libtorch_config_variant))
-        return elems
-
-    def gen_docker_image(self):
-        if self.gcc_config_variant == 'gcc5.4_cxx11-abi':
-            return miniutils.quote("pytorch/pytorch-binary-docker-image-ubuntu16.04:latest")
-
-        docker_word_substitution = {
-            "manywheel": "manylinux",
-            "libtorch": "manylinux",
-        }
-
-        docker_distro_prefix = miniutils.override(self.pydistro, docker_word_substitution)
-
-        # The cpu nightlies are built on the pytorch/manylinux-cuda102 docker image
-        alt_docker_suffix = self.cuda_version or "102"
-        docker_distro_suffix = "" if self.pydistro == "conda" else alt_docker_suffix
-        return miniutils.quote("pytorch/" + docker_distro_prefix + "-cuda" + docker_distro_suffix)
-
-    def get_name_prefix(self):
-        return "smoke" if self.smoke else "binary"
-
-    def gen_build_name(self, build_or_test, nightly):
-
-        parts = [self.get_name_prefix(), self.os] + self.gen_build_env_parms()
-
-        if nightly:
-            parts.append("nightly")
-
-        if self.libtorch_variant:
-            parts.append(self.libtorch_variant)
-
-        if not self.smoke:
-            parts.append(build_or_test)
-
-        joined = "_".join(parts)
-        return joined.replace(".", "_")
-
-    def gen_workflow_job(self, phase, upload_phase_dependency=None, nightly=False):
-        job_def = OrderedDict()
-        job_def["name"] = self.gen_build_name(phase, nightly)
-        job_def["build_environment"] = miniutils.quote(" ".join(self.gen_build_env_parms()))
-        if self.smoke:
-            job_def["requires"] = [
-                "update_s3_htmls",
-            ]
-            job_def["filters"] = branch_filters.gen_filter_dict(
-                branches_list=["nightly"],
-                tags_list=[branch_filters.RC_PATTERN],
-            )
-        else:
-            if phase in ["upload"]:
-                filter_branch = "nightly"
-            else:
-                filter_branch = r"/.*/"
-            job_def["filters"] = branch_filters.gen_filter_dict(
-                branches_list=[filter_branch],
-                tags_list=[branch_filters.RC_PATTERN],
-            )
-        if self.libtorch_variant:
-            job_def["libtorch_variant"] = miniutils.quote(self.libtorch_variant)
-        if phase == "test":
-            if not self.smoke:
-                job_def["requires"] = [self.gen_build_name("build", nightly)]
-            if not (self.smoke and self.os == "macos") and self.os != "windows":
-                job_def["docker_image"] = self.gen_docker_image()
-
-            if self.os != "windows" and self.cuda_version:
-                job_def["use_cuda_docker_runtime"] = miniutils.quote("1")
-        else:
-            if self.os == "linux" and phase != "upload":
-                job_def["docker_image"] = self.gen_docker_image()
-
-        if phase == "test":
-            if self.cuda_version:
-                if self.os == "windows":
-                    job_def["executor"] = "windows-with-nvidia-gpu"
-                else:
-                    job_def["resource_class"] = "gpu.medium"
-        if phase == "upload":
-            job_def["context"] = "org-member"
-            job_def["requires"] = [
-                self.gen_build_name(upload_phase_dependency, nightly)
-            ]
-
-        os_name = miniutils.override(self.os, {"macos": "mac"})
-        job_name = "_".join([self.get_name_prefix(), os_name, phase])
-        return {job_name : job_def}
-
-def get_root(smoke, name):
-
-    return binary_build_data.TopLevelNode(
-        name,
-        binary_build_data.CONFIG_TREE_DATA,
-        smoke,
-    )
-
-
-def gen_build_env_list(smoke):
-
-    root = get_root(smoke, "N/A")
-    config_list = conf_tree.dfs(root)
-
-    newlist = []
-    for c in config_list:
-        conf = Conf(
-            c.find_prop("os_name"),
-            c.find_prop("cu"),
-            c.find_prop("package_format"),
-            [c.find_prop("pyver")],
-            c.find_prop("smoke"),
-            c.find_prop("libtorch_variant"),
-            c.find_prop("gcc_config_variant"),
-            c.find_prop("libtorch_config_variant"),
-        )
-        newlist.append(conf)
-
-    return newlist
-
-def predicate_exclude_macos(config):
-    return config.os == "linux" or config.os == "windows"
-
-def get_nightly_uploads():
-    configs = gen_build_env_list(False)
-    mylist = []
-    for conf in configs:
-        phase_dependency = "test" if predicate_exclude_macos(conf) else "build"
-        mylist.append(conf.gen_workflow_job("upload", phase_dependency, nightly=True))
-
-    return mylist
-
-def get_post_upload_jobs():
-    """Generate jobs to update HTML indices and report binary sizes"""
-    configs = gen_build_env_list(False)
-    common_job_def = {
-        "context": "org-member",
-        "filters": branch_filters.gen_filter_dict(
-            branches_list=["nightly"],
-            tags_list=[branch_filters.RC_PATTERN],
-        ),
-        "requires": [],
-    }
-    for conf in configs:
-        upload_job_name = conf.gen_build_name(
-            build_or_test="upload",
-            nightly=True
-        )
-        common_job_def["requires"].append(upload_job_name)
-    return [
-        {
-            "update_s3_htmls": {
-                "name": "update_s3_htmls",
-                **common_job_def,
-            },
-        },
-    ]
-
-def get_nightly_tests():
-
-    configs = gen_build_env_list(False)
-    filtered_configs = filter(predicate_exclude_macos, configs)
-
-    tests = []
-    for conf_options in filtered_configs:
-        yaml_item = conf_options.gen_workflow_job("test", nightly=True)
-        tests.append(yaml_item)
-
-    return tests
-
-
-def get_jobs(toplevel_key, smoke):
-    jobs_list = []
-    configs = gen_build_env_list(smoke)
-    phase = "build" if toplevel_key == "binarybuilds" else "test"
-    for build_config in configs:
-        jobs_list.append(build_config.gen_workflow_job(phase, nightly=True))
-
-    return jobs_list
-
-
-def get_binary_build_jobs():
-    return get_jobs("binarybuilds", False)
-
-
-def get_binary_smoke_test_jobs():
-    return get_jobs("binarysmoketests", True)

.circleci/cimodel/data/caffe2_build_data.py

@@ -1,91 +0,0 @@
-from cimodel.lib.conf_tree import ConfigNode, XImportant
-from cimodel.lib.conf_tree import Ver
-
-
-CONFIG_TREE_DATA = [
-    (Ver("ubuntu", "16.04"), [
-        ([Ver("clang", "7")], [XImportant("onnx_main_py3.6"),
-                               XImportant("onnx_ort1_py3.6"),
-                               XImportant("onnx_ort2_py3.6")]),
-    ]),
-]
-
-
-class TreeConfigNode(ConfigNode):
-    def __init__(self, parent, node_name, subtree):
-        super(TreeConfigNode, self).__init__(parent, self.modify_label(node_name))
-        self.subtree = subtree
-        self.init2(node_name)
-
-    # noinspection PyMethodMayBeStatic
-    def modify_label(self, label):
-        return str(label)
-
-    def init2(self, node_name):
-        pass
-
-    def get_children(self):
-        return [self.child_constructor()(self, k, v) for (k, v) in self.subtree]
-
-    def is_build_only(self):
-        if str(self.find_prop("language_version")) == "onnx_main_py3.6" or \
-                str(self.find_prop("language_version")) == "onnx_ort1_py3.6" or \
-                str(self.find_prop("language_version")) == "onnx_ort2_py3.6":
-            return False
-        return set(str(c) for c in self.find_prop("compiler_version")).intersection({
-            "clang3.8",
-            "clang3.9",
-            "clang7",
-            "android",
-        }) or self.find_prop("distro_version").name == "macos"
-
-    def is_test_only(self):
-        if str(self.find_prop("language_version")) == "onnx_ort1_py3.6" or \
-                str(self.find_prop("language_version")) == "onnx_ort2_py3.6":
-            return True
-        return False
-
-
-class TopLevelNode(TreeConfigNode):
-    def __init__(self, node_name, subtree):
-        super(TopLevelNode, self).__init__(None, node_name, subtree)
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return DistroConfigNode
-
-
-class DistroConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["distro_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return CompilerConfigNode
-
-
-class CompilerConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["compiler_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return LanguageConfigNode
-
-
-class LanguageConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["language_version"] = node_name
-        self.props["build_only"] = self.is_build_only()
-        self.props["test_only"] = self.is_test_only()
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class ImportantConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["important"] = True
-
-    def get_children(self):
-        return []

.circleci/cimodel/data/caffe2_build_definitions.py

@@ -1,174 +0,0 @@
-from collections import OrderedDict
-
-import cimodel.data.dimensions as dimensions
-import cimodel.lib.conf_tree as conf_tree
-from cimodel.lib.conf_tree import Ver
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.caffe2_build_data import CONFIG_TREE_DATA, TopLevelNode
-from cimodel.data.simple.util.branch_filters import gen_filter_dict
-
-from dataclasses import dataclass
-
-
-DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/caffe2/"
-
-DOCKER_IMAGE_VERSION = "376"
-
-
-@dataclass
-class Conf:
-    language: str
-    distro: Ver
-    # There could be multiple compiler versions configured (e.g. nvcc
-    # for gpu files and host compiler (gcc/clang) for cpu files)
-    compilers: [Ver]
-    build_only: bool
-    test_only: bool
-    is_important: bool
-
-    @property
-    def compiler_names(self):
-        return [c.name for c in self.compilers]
-
-    # TODO: Eventually we can probably just remove the cudnn7 everywhere.
-    def get_cudnn_insertion(self):
-
-        omit = self.language == "onnx_main_py3.6" \
-            or self.language == "onnx_ort1_py3.6" \
-            or self.language == "onnx_ort2_py3.6" \
-            or set(self.compiler_names).intersection({"android", "mkl", "clang"}) \
-            or str(self.distro) in ["ubuntu14.04", "macos10.13"]
-
-        return [] if omit else ["cudnn7"]
-
-    def get_build_name_root_parts(self):
-        return [
-            "caffe2",
-            self.language,
-        ] + self.get_build_name_middle_parts()
-
-    def get_build_name_middle_parts(self):
-        return [str(c) for c in self.compilers] + self.get_cudnn_insertion() + [str(self.distro)]
-
-    def construct_phase_name(self, phase):
-        root_parts = self.get_build_name_root_parts()
-
-        build_name_substitutions = {
-            "onnx_ort1_py3.6": "onnx_main_py3.6",
-            "onnx_ort2_py3.6": "onnx_main_py3.6",
-        }
-        if phase == "build":
-            root_parts = [miniutils.override(r, build_name_substitutions) for r in root_parts]
-        return "_".join(root_parts + [phase]).replace(".", "_")
-
-    def get_platform(self):
-        platform = self.distro.name
-        if self.distro.name != "macos":
-            platform = "linux"
-        return platform
-
-    def gen_docker_image(self):
-
-        lang_substitutions = {
-            "onnx_main_py3.6": "py3.6",
-            "onnx_ort1_py3.6": "py3.6",
-            "onnx_ort2_py3.6": "py3.6",
-            "cmake": "py3",
-        }
-
-        lang = miniutils.override(self.language, lang_substitutions)
-        parts = [lang] + self.get_build_name_middle_parts()
-        return miniutils.quote(DOCKER_IMAGE_PATH_BASE + "-".join(parts) + ":" + str(DOCKER_IMAGE_VERSION))
-
-    def gen_workflow_params(self, phase):
-        parameters = OrderedDict()
-        lang_substitutions = {
-            "onnx_py3": "onnx-py3",
-            "onnx_main_py3.6": "onnx-main-py3.6",
-            "onnx_ort1_py3.6": "onnx-ort1-py3.6",
-            "onnx_ort2_py3.6": "onnx-ort2-py3.6",
-        }
-
-        lang = miniutils.override(self.language, lang_substitutions)
-
-        parts = [
-            "caffe2",
-            lang,
-        ] + self.get_build_name_middle_parts() + [phase]
-
-        build_env_name = "-".join(parts)
-        parameters["build_environment"] = miniutils.quote(build_env_name)
-        if "ios" in self.compiler_names:
-            parameters["build_ios"] = miniutils.quote("1")
-        if phase == "test":
-            # TODO cuda should not be considered a compiler
-            if "cuda" in self.compiler_names:
-                parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
-
-        if self.distro.name != "macos":
-            parameters["docker_image"] = self.gen_docker_image()
-            if self.build_only:
-                parameters["build_only"] = miniutils.quote("1")
-        if phase == "test":
-            resource_class = "large" if "cuda" not in self.compiler_names else "gpu.medium"
-            parameters["resource_class"] = resource_class
-
-        return parameters
-
-    def gen_workflow_job(self, phase):
-        job_def = OrderedDict()
-        job_def["name"] = self.construct_phase_name(phase)
-
-        if phase == "test":
-            job_def["requires"] = [self.construct_phase_name("build")]
-            job_name = "caffe2_" + self.get_platform() + "_test"
-        else:
-            job_name = "caffe2_" + self.get_platform() + "_build"
-
-        if not self.is_important:
-            job_def["filters"] = gen_filter_dict()
-        job_def.update(self.gen_workflow_params(phase))
-        return {job_name : job_def}
-
-
-def get_root():
-    return TopLevelNode("Caffe2 Builds", CONFIG_TREE_DATA)
-
-
-def instantiate_configs():
-
-    config_list = []
-
-    root = get_root()
-    found_configs = conf_tree.dfs(root)
-    for fc in found_configs:
-        c = Conf(
-            language=fc.find_prop("language_version"),
-            distro=fc.find_prop("distro_version"),
-            compilers=fc.find_prop("compiler_version"),
-            build_only=fc.find_prop("build_only"),
-            test_only=fc.find_prop("test_only"),
-            is_important=fc.find_prop("important"),
-        )
-
-        config_list.append(c)
-
-    return config_list
-
-
-def get_workflow_jobs():
-
-    configs = instantiate_configs()
-
-    x = []
-    for conf_options in configs:
-        phases = ["build"]
-        if not conf_options.build_only:
-            phases = dimensions.PHASES
-        if conf_options.test_only:
-            phases = ["test"]
-
-        for phase in phases:
-            x.append(conf_options.gen_workflow_job(phase))
-
-    return x

.circleci/cimodel/data/dimensions.py

@@ -1,14 +0,0 @@
-PHASES = ["build", "test"]
-
-CUDA_VERSIONS = [
-    None,  # cpu build
-    "92",
-    "101",
-    "102",
-]
-
-STANDARD_PYTHON_VERSIONS = [
-    "3.6",
-    "3.7",
-    "3.8"
-]

.circleci/cimodel/data/pytorch_build_data.py

@@ -1,246 +0,0 @@
-from cimodel.lib.conf_tree import ConfigNode, X, XImportant
-
-
-CONFIG_TREE_DATA = [
-    ("xenial", [
-        (None, [
-            X("nightly"),
-        ]),
-        ("gcc", [
-            ("5.4", [  # All this subtree rebases to master and then build
-                XImportant("3.6"),
-                ("3.6", [
-                    ("parallel_tbb", [X(True)]),
-                    ("parallel_native", [X(True)]),
-                ]),
-            ]),
-            # TODO: bring back libtorch test
-            ("7", [X("3.6")]),
-        ]),
-        ("clang", [
-            ("5", [
-                XImportant("3.6"),  # This is actually the ASAN build
-            ]),
-        ]),
-        ("cuda", [
-            ("9.2", [
-                X("3.6"),
-                ("3.6", [
-                    ("cuda_gcc_override", [X("gcc5.4")])
-                ])
-            ]),
-            ("10.1", [X("3.6")]),
-            ("10.2", [
-                XImportant("3.6"),
-                ("3.6", [
-                    ("libtorch", [XImportant(True)])
-                ]),
-            ]),
-        ]),
-    ]),
-    ("bionic", [
-        ("clang", [
-            ("9", [
-                XImportant("3.6"),
-            ]),
-            ("9", [
-                ("3.6", [
-                    ("xla", [XImportant(True)]),
-                ]),
-            ]),
-        ]),
-        ("gcc", [
-            ("9", [XImportant("3.8")]),
-        ]),
-    ]),
-]
-
-
-def get_major_pyver(dotted_version):
-    parts = dotted_version.split(".")
-    return "py" + parts[0]
-
-
-class TreeConfigNode(ConfigNode):
-    def __init__(self, parent, node_name, subtree):
-        super(TreeConfigNode, self).__init__(parent, self.modify_label(node_name))
-        self.subtree = subtree
-        self.init2(node_name)
-
-    def modify_label(self, label):
-        return label
-
-    def init2(self, node_name):
-        pass
-
-    def get_children(self):
-        return [self.child_constructor()(self, k, v) for (k, v) in self.subtree]
-
-
-class TopLevelNode(TreeConfigNode):
-    def __init__(self, node_name, subtree):
-        super(TopLevelNode, self).__init__(None, node_name, subtree)
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return DistroConfigNode
-
-
-class DistroConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["distro_name"] = node_name
-
-    def child_constructor(self):
-        distro = self.find_prop("distro_name")
-
-        next_nodes = {
-            "xenial": XenialCompilerConfigNode,
-            "bionic": BionicCompilerConfigNode,
-        }
-        return next_nodes[distro]
-
-
-class PyVerConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["pyver"] = node_name
-        self.props["abbreviated_pyver"] = get_major_pyver(node_name)
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return ExperimentalFeatureConfigNode
-
-
-class ExperimentalFeatureConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["experimental_feature"] = node_name
-
-    def child_constructor(self):
-        experimental_feature = self.find_prop("experimental_feature")
-
-        next_nodes = {
-            "xla": XlaConfigNode,
-            "parallel_tbb": ParallelTBBConfigNode,
-            "parallel_native": ParallelNativeConfigNode,
-            "libtorch": LibTorchConfigNode,
-            "important": ImportantConfigNode,
-            "build_only": BuildOnlyConfigNode,
-            "cuda_gcc_override": CudaGccOverrideConfigNode
-        }
-        return next_nodes[experimental_feature]
-
-
-class XlaConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "XLA=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_xla"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class ParallelTBBConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "PARALLELTBB=" + str(label)
-
-    def init2(self, node_name):
-        self.props["parallel_backend"] = "paralleltbb"
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class ParallelNativeConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "PARALLELNATIVE=" + str(label)
-
-    def init2(self, node_name):
-        self.props["parallel_backend"] = "parallelnative"
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class LibTorchConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "BUILD_TEST_LIBTORCH=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_libtorch"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class CudaGccOverrideConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["cuda_gcc_override"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-class BuildOnlyConfigNode(TreeConfigNode):
-
-    def init2(self, node_name):
-        self.props["build_only"] = node_name
-
-    def child_constructor(self):
-        return ImportantConfigNode
-
-
-class ImportantConfigNode(TreeConfigNode):
-    def modify_label(self, label):
-        return "IMPORTANT=" + str(label)
-
-    def init2(self, node_name):
-        self.props["is_important"] = node_name
-
-    def get_children(self):
-        return []
-
-
-class XenialCompilerConfigNode(TreeConfigNode):
-
-    def modify_label(self, label):
-        return label or "<unspecified>"
-
-    def init2(self, node_name):
-        self.props["compiler_name"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-
-        return XenialCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
-
-
-class BionicCompilerConfigNode(TreeConfigNode):
-
-    def modify_label(self, label):
-        return label or "<unspecified>"
-
-    def init2(self, node_name):
-        self.props["compiler_name"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-
-        return BionicCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
-
-
-class XenialCompilerVersionConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["compiler_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return PyVerConfigNode
-
-
-class BionicCompilerVersionConfigNode(TreeConfigNode):
-    def init2(self, node_name):
-        self.props["compiler_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return PyVerConfigNode

.circleci/cimodel/data/pytorch_build_definitions.py

@@ -1,309 +0,0 @@
-from collections import OrderedDict
-
-from cimodel.data.pytorch_build_data import TopLevelNode, CONFIG_TREE_DATA
-import cimodel.data.dimensions as dimensions
-import cimodel.lib.conf_tree as conf_tree
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.branch_filters import gen_filter_dict
-from cimodel.data.simple.util.docker_constants import gen_docker_image_path
-
-from dataclasses import dataclass, field
-from typing import List, Optional
-
-
-@dataclass
-class Conf:
-    distro: str
-    parms: List[str]
-    parms_list_ignored_for_docker_image: Optional[List[str]] = None
-    pyver: Optional[str] = None
-    cuda_version: Optional[str] = None
-    # TODO expand this to cover all the USE_* that we want to test for
-    #  tesnrorrt, leveldb, lmdb, redis, opencv, mkldnn, ideep, etc.
-    # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453608)
-    is_xla: bool = False
-    vulkan: bool = False
-    restrict_phases: Optional[List[str]] = None
-    gpu_resource: Optional[str] = None
-    dependent_tests: List = field(default_factory=list)
-    parent_build: Optional['Conf'] = None
-    is_libtorch: bool = False
-    is_important: bool = False
-    parallel_backend: Optional[str] = None
-
-    # TODO: Eliminate the special casing for docker paths
-    # In the short term, we *will* need to support special casing as docker images are merged for caffe2 and pytorch
-    def get_parms(self, for_docker):
-        leading = []
-        # We just don't run non-important jobs on pull requests;
-        # previously we also named them in a way to make it obvious
-        # if self.is_important and not for_docker:
-        #    leading.append("AAA")
-        leading.append("pytorch")
-        if self.is_xla and not for_docker:
-            leading.append("xla")
-        if self.is_libtorch and not for_docker:
-            leading.append("libtorch")
-        if self.parallel_backend is not None and not for_docker:
-            leading.append(self.parallel_backend)
-
-        cuda_parms = []
-        if self.cuda_version:
-            cuda_parms.extend(["cuda" + self.cuda_version, "cudnn7"])
-        result = leading + ["linux", self.distro] + cuda_parms + self.parms
-        if not for_docker and self.parms_list_ignored_for_docker_image is not None:
-            result = result + self.parms_list_ignored_for_docker_image
-        return result
-
-    def gen_docker_image_path(self):
-
-        parms_source = self.parent_build or self
-        base_build_env_name = "-".join(parms_source.get_parms(True))
-
-        return miniutils.quote(gen_docker_image_path(base_build_env_name))
-
-    def get_build_job_name_pieces(self, build_or_test):
-        return self.get_parms(False) + [build_or_test]
-
-    def gen_build_name(self, build_or_test):
-        return ("_".join(map(str, self.get_build_job_name_pieces(build_or_test)))).replace(".", "_").replace("-", "_")
-
-    def get_dependents(self):
-        return self.dependent_tests or []
-
-    def gen_workflow_params(self, phase):
-        parameters = OrderedDict()
-        build_job_name_pieces = self.get_build_job_name_pieces(phase)
-
-        build_env_name = "-".join(map(str, build_job_name_pieces))
-        parameters["build_environment"] = miniutils.quote(build_env_name)
-        parameters["docker_image"] = self.gen_docker_image_path()
-        if phase == "test" and self.gpu_resource:
-            parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
-        if phase == "test":
-            resource_class = "large"
-            if self.gpu_resource:
-                resource_class = "gpu." + self.gpu_resource
-            parameters["resource_class"] = resource_class
-        return parameters
-
-    def gen_workflow_job(self, phase):
-        job_def = OrderedDict()
-        job_def["name"] = self.gen_build_name(phase)
-
-        if phase == "test":
-
-            # TODO When merging the caffe2 and pytorch jobs, it might be convenient for a while to make a
-            #  caffe2 test job dependent on a pytorch build job. This way we could quickly dedup the repeated
-            #  build of pytorch in the caffe2 build job, and just run the caffe2 tests off of a completed
-            #  pytorch build job (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259452641)
-
-            dependency_build = self.parent_build or self
-            job_def["requires"] = [dependency_build.gen_build_name("build")]
-            job_name = "pytorch_linux_test"
-        else:
-            job_name = "pytorch_linux_build"
-
-        if not self.is_important:
-            job_def["filters"] = gen_filter_dict()
-        job_def.update(self.gen_workflow_params(phase))
-
-        return {job_name : job_def}
-
-
-# TODO This is a hack to special case some configs just for the workflow list
-class HiddenConf(object):
-    def __init__(self, name, parent_build=None):
-        self.name = name
-        self.parent_build = parent_build
-
-    def gen_workflow_job(self, phase):
-        return {self.gen_build_name(phase): {"requires": [self.parent_build.gen_build_name("build")]}}
-
-    def gen_build_name(self, _):
-        return self.name
-
-
-# TODO Convert these to graph nodes
-def gen_dependent_configs(xenial_parent_config):
-
-    extra_parms = [
-        (["multigpu"], "large"),
-        (["NO_AVX2"], "medium"),
-        (["NO_AVX", "NO_AVX2"], "medium"),
-        (["slow"], "medium"),
-        (["nogpu"], None),
-    ]
-
-    configs = []
-    for parms, gpu in extra_parms:
-
-        c = Conf(
-            xenial_parent_config.distro,
-            ["py3"] + parms,
-            pyver="3.6",
-            cuda_version=xenial_parent_config.cuda_version,
-            restrict_phases=["test"],
-            gpu_resource=gpu,
-            parent_build=xenial_parent_config,
-            is_important=xenial_parent_config.is_important,
-        )
-
-        configs.append(c)
-
-    return configs
-
-
-def gen_docs_configs(xenial_parent_config):
-    configs = []
-
-    for x in ["pytorch_python_doc_push", "pytorch_cpp_doc_push", "pytorch_doc_test"]:
-        configs.append(HiddenConf(x, parent_build=xenial_parent_config))
-
-    return configs
-
-
-def get_root():
-    return TopLevelNode("PyTorch Builds", CONFIG_TREE_DATA)
-
-
-def gen_tree():
-    root = get_root()
-    configs_list = conf_tree.dfs(root)
-    return configs_list
-
-
-def instantiate_configs():
-
-    config_list = []
-
-    root = get_root()
-    found_configs = conf_tree.dfs(root)
-    for fc in found_configs:
-
-        restrict_phases = None
-        distro_name = fc.find_prop("distro_name")
-        compiler_name = fc.find_prop("compiler_name")
-        compiler_version = fc.find_prop("compiler_version")
-        is_xla = fc.find_prop("is_xla") or False
-        parms_list_ignored_for_docker_image = []
-
-        vulkan = fc.find_prop("vulkan") or False
-        if vulkan:
-            parms_list_ignored_for_docker_image.append("vulkan")
-
-        python_version = None
-        if compiler_name == "cuda" or compiler_name == "android":
-            python_version = fc.find_prop("pyver")
-            parms_list = [fc.find_prop("abbreviated_pyver")]
-        else:
-            parms_list = ["py" + fc.find_prop("pyver")]
-
-        cuda_version = None
-        if compiler_name == "cuda":
-            cuda_version = fc.find_prop("compiler_version")
-
-        elif compiler_name == "android":
-            android_ndk_version = fc.find_prop("compiler_version")
-            # TODO: do we need clang to compile host binaries like protoc?
-            parms_list.append("clang5")
-            parms_list.append("android-ndk-" + android_ndk_version)
-            android_abi = fc.find_prop("android_abi")
-            parms_list_ignored_for_docker_image.append(android_abi)
-            restrict_phases = ["build"]
-
-        elif compiler_name:
-            gcc_version = compiler_name + (fc.find_prop("compiler_version") or "")
-            parms_list.append(gcc_version)
-
-            # TODO: This is a nasty special case
-            if gcc_version == 'clang5' and not is_xla:
-                parms_list.append("asan")
-                python_version = fc.find_prop("pyver")
-                parms_list[0] = fc.find_prop("abbreviated_pyver")
-
-        if cuda_version in ["9.2", "10", "10.1", "10.2"]:
-            # TODO The gcc version is orthogonal to CUDA version?
-            cuda_gcc_version = fc.find_prop("cuda_gcc_override") or "gcc7"
-            parms_list.append(cuda_gcc_version)
-
-        is_libtorch = fc.find_prop("is_libtorch") or False
-        is_important = fc.find_prop("is_important") or False
-        parallel_backend = fc.find_prop("parallel_backend") or None
-        build_only = fc.find_prop("build_only") or False
-        if build_only and restrict_phases is None:
-            restrict_phases = ["build"]
-
-        gpu_resource = None
-        if cuda_version and cuda_version != "10":
-            gpu_resource = "medium"
-
-        c = Conf(
-            distro_name,
-            parms_list,
-            parms_list_ignored_for_docker_image,
-            python_version,
-            cuda_version,
-            is_xla,
-            vulkan,
-            restrict_phases,
-            gpu_resource,
-            is_libtorch=is_libtorch,
-            is_important=is_important,
-            parallel_backend=parallel_backend,
-        )
-
-        # run docs builds on "pytorch-linux-xenial-py3.6-gcc5.4". Docs builds
-        # should run on a CPU-only build that runs on all PRs.
-        if distro_name == 'xenial' and fc.find_prop("pyver") == '3.6' \
-                and cuda_version is None \
-                and parallel_backend is None \
-                and compiler_name == 'gcc' \
-                and fc.find_prop('compiler_version') == '5.4':
-            c.dependent_tests = gen_docs_configs(c)
-
-        if cuda_version == "10.1" and python_version == "3.6" and not is_libtorch:
-            c.dependent_tests = gen_dependent_configs(c)
-
-        if (compiler_name == "gcc"
-                and compiler_version == "5.4"
-                and not is_libtorch
-                and parallel_backend is None):
-            bc_breaking_check = Conf(
-                "backward-compatibility-check",
-                [],
-                is_xla=False,
-                restrict_phases=["test"],
-                is_libtorch=False,
-                is_important=True,
-                parent_build=c,
-            )
-            c.dependent_tests.append(bc_breaking_check)
-
-        config_list.append(c)
-
-    return config_list
-
-
-def get_workflow_jobs():
-
-    config_list = instantiate_configs()
-
-    x = []
-    for conf_options in config_list:
-
-        phases = conf_options.restrict_phases or dimensions.PHASES
-
-        for phase in phases:
-
-            # TODO why does this not have a test?
-            if phase == "test" and conf_options.cuda_version == "10":
-                continue
-
-            x.append(conf_options.gen_workflow_job(phase))
-
-        # TODO convert to recursion
-        for conf in conf_options.get_dependents():
-            x.append(conf.gen_workflow_job("test"))
-
-    return x

.circleci/cimodel/data/simple/android_definitions.py

@@ -1,92 +0,0 @@
-import cimodel.data.simple.util.branch_filters
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_NDK
-
-
-class AndroidJob:
-    def __init__(self,
-                 variant,
-                 template_name,
-                 is_master_only=True):
-
-        self.variant = variant
-        self.template_name = template_name
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-
-        base_name_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "android",
-            "ndk",
-            "r19c",
-        ] + self.variant + [
-            "build",
-        ]
-
-        full_job_name = "_".join(base_name_parts)
-        build_env_name = "-".join(base_name_parts)
-
-        props_dict = {
-            "name": full_job_name,
-            "build_environment": "\"{}\"".format(build_env_name),
-            "docker_image": "\"{}\"".format(DOCKER_IMAGE_NDK),
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{self.template_name: props_dict}]
-
-
-class AndroidGradleJob:
-    def __init__(self,
-                 job_name,
-                 template_name,
-                 dependencies,
-                 is_master_only=True):
-
-        self.job_name = job_name
-        self.template_name = template_name
-        self.dependencies = dependencies
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-
-        props_dict = {
-            "name": self.job_name,
-            "requires": self.dependencies,
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    AndroidJob(["x86_32"], "pytorch_linux_build", is_master_only=False),
-    AndroidJob(["x86_64"], "pytorch_linux_build"),
-    AndroidJob(["arm", "v7a"], "pytorch_linux_build"),
-    AndroidJob(["arm", "v8a"], "pytorch_linux_build"),
-    AndroidJob(["vulkan", "x86_32"], "pytorch_linux_build", is_master_only=False),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32",
-        "pytorch_android_gradle_build-x86_32",
-        ["pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build"],
-        is_master_only=False),
-    AndroidGradleJob(
-        "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build",
-        "pytorch_android_gradle_build",
-        ["pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build",
-         "pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build"]),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/bazel_definitions.py

@@ -1,63 +0,0 @@
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_GCC7
-
-
-def gen_job_name(phase):
-    job_name_parts = [
-        "pytorch",
-        "bazel",
-        phase,
-    ]
-
-    return "_".join(job_name_parts)
-
-
-class BazelJob:
-    def __init__(self, phase, extra_props=None):
-        self.phase = phase
-        self.extra_props = extra_props or {}
-
-    def gen_tree(self):
-
-        template_parts = [
-            "pytorch",
-            "linux",
-            "bazel",
-            self.phase,
-        ]
-
-        build_env_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3.6",
-            "gcc7",
-            "bazel",
-            self.phase,
-        ]
-
-        full_job_name = gen_job_name(self.phase)
-        build_env_name = "-".join(build_env_parts)
-
-        extra_requires = [gen_job_name("build")] if self.phase == "test" else []
-
-        props_dict = {
-            "build_environment": build_env_name,
-            "docker_image": DOCKER_IMAGE_GCC7,
-            "name": full_job_name,
-            "requires": extra_requires,
-        }
-
-        props_dict.update(self.extra_props)
-
-        template_name = "_".join(template_parts)
-        return [{template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    BazelJob("build", {"resource_class": "large"}),
-    BazelJob("test"),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/binary_smoketest.py

@@ -1,193 +0,0 @@
-"""
-TODO: Refactor circleci/cimodel/data/binary_build_data.py to generate this file
-       instead of doing one offs here
- Binary builds (subset, to smoke test that they'll work)
-
- NB: If you modify this file, you need to also modify
- the binary_and_smoke_tests_on_pr variable in
- pytorch-ci-hud to adjust the list of whitelisted builds
- at https://github.com/ezyang/pytorch-ci-hud/blob/master/src/BuildHistoryDisplay.js
-
- Note:
- This binary build is currently broken, see https://github_com/pytorch/pytorch/issues/16710
- - binary_linux_conda_3_6_cu90_devtoolset7_build
- - binary_linux_conda_3_6_cu90_devtoolset7_test
-
- TODO
- we should test a libtorch cuda build, but they take too long
- - binary_linux_libtorch_3_6m_cu90_devtoolset7_static-without-deps_build
-"""
-
-import cimodel.lib.miniutils as miniutils
-import cimodel.data.simple.util.branch_filters
-
-
-class SmoketestJob:
-    def __init__(self,
-                 template_name,
-                 build_env_parts,
-                 docker_image,
-                 job_name,
-                 is_master_only=False,
-                 requires=None,
-                 has_libtorch_variant=False,
-                 extra_props=None):
-
-        self.template_name = template_name
-        self.build_env_parts = build_env_parts
-        self.docker_image = docker_image
-        self.job_name = job_name
-        self.is_master_only = is_master_only
-        self.requires = requires or []
-        self.has_libtorch_variant = has_libtorch_variant
-        self.extra_props = extra_props or {}
-
-    def gen_tree(self):
-
-        props_dict = {
-            "build_environment": " ".join(self.build_env_parts),
-            "name": self.job_name,
-            "requires": self.requires,
-        }
-
-        if self.docker_image:
-            props_dict["docker_image"] = self.docker_image
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        if self.has_libtorch_variant:
-            props_dict["libtorch_variant"] = "shared-with-deps"
-
-        props_dict.update(self.extra_props)
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    SmoketestJob(
-        "binary_linux_build",
-        ["manywheel", "3.7m", "cu102", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_manywheel_3_7m_cu102_devtoolset7_build",
-        is_master_only=True,
-    ),
-    SmoketestJob(
-        "binary_linux_build",
-        ["libtorch", "3.7m", "cpu", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build",
-        is_master_only=False,
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_linux_build",
-        ["libtorch", "3.7m", "cpu", "gcc5.4_cxx11-abi"],
-        "pytorch/pytorch-binary-docker-image-ubuntu16.04:latest",
-        "binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build",
-        is_master_only=False,
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_mac_build",
-        ["wheel", "3.7", "cpu"],
-        None,
-        "binary_macos_wheel_3_7_cpu_build",
-        is_master_only=True,
-    ),
-    # This job has an average run time of 3 hours o.O
-    # Now only running this on master to reduce overhead
-    SmoketestJob(
-        "binary_mac_build",
-        ["libtorch", "3.7", "cpu"],
-        None,
-        "binary_macos_libtorch_3_7_cpu_build",
-        is_master_only=True,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["libtorch", "3.7", "cpu", "debug"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_debug_build",
-        is_master_only=False,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["libtorch", "3.7", "cpu", "release"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_release_build",
-        is_master_only=False,
-    ),
-    SmoketestJob(
-        "binary_windows_build",
-        ["wheel", "3.7", "cu102"],
-        None,
-        "binary_windows_wheel_3_7_cu102_build",
-        is_master_only=True,
-    ),
-
-    SmoketestJob(
-        "binary_windows_test",
-        ["libtorch", "3.7", "cpu", "debug"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_debug_test",
-        is_master_only=False,
-        requires=["binary_windows_libtorch_3_7_cpu_debug_build"],
-    ),
-    SmoketestJob(
-        "binary_windows_test",
-        ["libtorch", "3.7", "cpu", "release"],
-        None,
-        "binary_windows_libtorch_3_7_cpu_release_test",
-        is_master_only=False,
-        requires=["binary_windows_libtorch_3_7_cpu_release_build"],
-    ),
-    SmoketestJob(
-        "binary_windows_test",
-        ["wheel", "3.7", "cu102"],
-        None,
-        "binary_windows_wheel_3_7_cu102_test",
-        is_master_only=True,
-        requires=["binary_windows_wheel_3_7_cu102_build"],
-        extra_props={
-            "executor": "windows-with-nvidia-gpu",
-        },
-    ),
-
-
-
-    SmoketestJob(
-        "binary_linux_test",
-        ["manywheel", "3.7m", "cu102", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_manywheel_3_7m_cu102_devtoolset7_test",
-        is_master_only=True,
-        requires=["binary_linux_manywheel_3_7m_cu102_devtoolset7_build"],
-        extra_props={
-            "resource_class": "gpu.medium",
-            "use_cuda_docker_runtime": miniutils.quote((str(1))),
-        },
-    ),
-    SmoketestJob(
-        "binary_linux_test",
-        ["libtorch", "3.7m", "cpu", "devtoolset7"],
-        "pytorch/manylinux-cuda102",
-        "binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_test",
-        is_master_only=False,
-        requires=["binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build"],
-        has_libtorch_variant=True,
-    ),
-    SmoketestJob(
-        "binary_linux_test",
-        ["libtorch", "3.7m", "cpu", "gcc5.4_cxx11-abi"],
-        "pytorch/pytorch-binary-docker-image-ubuntu16.04:latest",
-        "binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_test",
-        is_master_only=False,
-        requires=["binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build"],
-        has_libtorch_variant=True,
-    ),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/docker_definitions.py

@@ -1,44 +0,0 @@
-from collections import OrderedDict
-
-from cimodel.lib.miniutils import quote
-
-
-# TODO: make this generated from a matrix rather than just a static list
-IMAGE_NAMES = [
-    "pytorch-linux-bionic-cuda11.0-cudnn8-py3.6-gcc9",
-    "pytorch-linux-bionic-cuda11.0-cudnn8-py3.8-gcc9",
-    "pytorch-linux-bionic-cuda10.2-cudnn7-py3.8-gcc9",
-    "pytorch-linux-bionic-py3.6-clang9",
-    "pytorch-linux-bionic-cuda10.2-cudnn7-py3.6-clang9",
-    "pytorch-linux-bionic-py3.8-gcc9",
-    "pytorch-linux-xenial-cuda10-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda10.1-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-cuda11.0-cudnn8-py3-gcc7",
-    "pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc5.4",
-    "pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7",
-    "pytorch-linux-xenial-py3-clang5-android-ndk-r19c",
-    "pytorch-linux-xenial-py3-clang5-asan",
-    "pytorch-linux-xenial-py3.8",
-    "pytorch-linux-xenial-py3.6-clang7",
-    "pytorch-linux-xenial-py3.6-gcc4.8",
-    "pytorch-linux-xenial-py3.6-gcc5.4",
-    "pytorch-linux-xenial-py3.6-gcc7.2",
-    "pytorch-linux-xenial-py3.6-gcc7",
-    "pytorch-linux-xenial-pynightly",
-    "pytorch-linux-xenial-rocm3.3-py3.6",
-]
-
-
-def get_workflow_jobs():
-    """Generates a list of docker image build definitions"""
-    return [
-        OrderedDict(
-            {
-                "docker_build_job": OrderedDict(
-                    {"name": quote(image_name), "image_name": quote(image_name)}
-                )
-            }
-        )
-        for image_name in IMAGE_NAMES
-    ]

.circleci/cimodel/data/simple/ge_config_tests.py

@@ -1,103 +0,0 @@
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.versions import MultiPartVersion, CudaVersion
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_BASIC, DOCKER_IMAGE_CUDA_10_2
-
-
-class GeConfigTestJob:
-    def __init__(self,
-                 py_version,
-                 gcc_version,
-                 cuda_version,
-                 variant_parts,
-                 extra_requires,
-                 use_cuda_docker=False,
-                 build_env_override=None):
-
-        self.py_version = py_version
-        self.gcc_version = gcc_version
-        self.cuda_version = cuda_version
-        self.variant_parts = variant_parts
-        self.extra_requires = extra_requires
-        self.use_cuda_docker = use_cuda_docker
-        self.build_env_override = build_env_override
-
-    def get_all_parts(self, with_dots):
-
-        maybe_py_version = self.py_version.render_dots_or_parts(with_dots) if self.py_version else []
-        maybe_gcc_version = self.gcc_version.render_dots_or_parts(with_dots) if self.gcc_version else []
-        maybe_cuda_version = self.cuda_version.render_dots_or_parts(with_dots) if self.cuda_version else []
-
-        common_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-        ] + maybe_cuda_version + maybe_py_version + maybe_gcc_version
-
-        return common_parts + self.variant_parts
-
-    def gen_tree(self):
-
-        resource_class = "gpu.medium" if self.use_cuda_docker else "large"
-        docker_image = DOCKER_IMAGE_CUDA_10_2 if self.use_cuda_docker else DOCKER_IMAGE_BASIC
-        full_name = "_".join(self.get_all_parts(False))
-        build_env = self.build_env_override or "-".join(self.get_all_parts(True))
-
-        props_dict = {
-            "name": full_name,
-            "build_environment": build_env,
-            "requires": self.extra_requires,
-            "resource_class": resource_class,
-            "docker_image": docker_image,
-        }
-
-        if self.use_cuda_docker:
-            props_dict["use_cuda_docker_runtime"] = miniutils.quote(str(1))
-
-        return [{"pytorch_linux_test": props_dict}]
-
-
-WORKFLOW_DATA = [
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["ge_config_legacy", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"]),
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["ge_config_profiling", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"]),
-    GeConfigTestJob(
-        MultiPartVersion([3, 6], "py"),
-        MultiPartVersion([5, 4], "gcc"),
-        None,
-        ["ge_config_simple", "test"],
-        ["pytorch_linux_xenial_py3_6_gcc5_4_build"],
-    ),
-    GeConfigTestJob(
-        None,
-        None,
-        CudaVersion(10, 2),
-        ["cudnn7", "py3", "ge_config_legacy", "test"],
-        ["pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_build"],
-        use_cuda_docker=True,
-        # TODO Why does the build environment specify cuda10.1, while the
-        # job name is cuda10_2?
-        build_env_override="pytorch-linux-xenial-cuda10.1-cudnn7-ge_config_legacy-test"),
-    GeConfigTestJob(
-        None,
-        None,
-        CudaVersion(10, 2),
-        ["cudnn7", "py3", "ge_config_profiling", "test"],
-        ["pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_build"],
-        use_cuda_docker=True,
-        # TODO Why does the build environment specify cuda10.1, while the
-        # job name is cuda10_2?
-        build_env_override="pytorch-linux-xenial-cuda10.1-cudnn7-ge_config_profiling-test"),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/ios_definitions.py

@@ -1,71 +0,0 @@
-from cimodel.data.simple.util.versions import MultiPartVersion
-
-
-IOS_VERSION = MultiPartVersion([11, 2, 1])
-
-
-class ArchVariant:
-    def __init__(self, name, is_custom=False):
-        self.name = name
-        self.is_custom = is_custom
-
-    def render(self):
-        extra_parts = ["custom"] if self.is_custom else []
-        return "_".join([self.name] + extra_parts)
-
-
-def get_platform(arch_variant_name):
-    return "SIMULATOR" if arch_variant_name == "x86_64" else "OS"
-
-
-class IOSJob:
-    def __init__(self, ios_version, arch_variant, is_org_member_context=True, extra_props=None):
-        self.ios_version = ios_version
-        self.arch_variant = arch_variant
-        self.is_org_member_context = is_org_member_context
-        self.extra_props = extra_props
-
-    def gen_name_parts(self, with_version_dots):
-
-        version_parts = self.ios_version.render_dots_or_parts(with_version_dots)
-        build_variant_suffix = "_".join([self.arch_variant.render(), "build"])
-
-        return [
-            "pytorch",
-            "ios",
-        ] + version_parts + [
-            build_variant_suffix,
-        ]
-
-    def gen_job_name(self):
-        return "_".join(self.gen_name_parts(False))
-
-    def gen_tree(self):
-
-        platform_name = get_platform(self.arch_variant.name)
-
-        props_dict = {
-            "build_environment": "-".join(self.gen_name_parts(True)),
-            "ios_arch": self.arch_variant.name,
-            "ios_platform": platform_name,
-            "name": self.gen_job_name(),
-        }
-
-        if self.is_org_member_context:
-            props_dict["context"] = "org-member"
-
-        if self.extra_props:
-            props_dict.update(self.extra_props)
-
-        return [{"pytorch_ios_build": props_dict}]
-
-
-WORKFLOW_DATA = [
-    IOSJob(IOS_VERSION, ArchVariant("x86_64"), is_org_member_context=False),
-    IOSJob(IOS_VERSION, ArchVariant("arm64")),
-    IOSJob(IOS_VERSION, ArchVariant("arm64", True), extra_props={"op_list": "mobilenetv2.yaml"}),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/macos_definitions.py

@@ -1,28 +0,0 @@
-class MacOsJob:
-    def __init__(self, os_version, is_test=False):
-        self.os_version = os_version
-        self.is_test = is_test
-
-    def gen_tree(self):
-        non_phase_parts = ["pytorch", "macos", self.os_version, "py3"]
-
-        phase_name = "test" if self.is_test else "build"
-
-        full_job_name = "_".join(non_phase_parts + [phase_name])
-
-        test_build_dependency = "_".join(non_phase_parts + ["build"])
-        extra_dependencies = [test_build_dependency] if self.is_test else []
-        job_dependencies = extra_dependencies
-
-        # Yes we name the job after itself, it needs a non-empty value in here
-        # for the YAML output to work.
-        props_dict = {"requires": job_dependencies, "name": full_job_name}
-
-        return [{full_job_name: props_dict}]
-
-
-WORKFLOW_DATA = [MacOsJob("10_13"), MacOsJob("10_13", True)]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/mobile_definitions.py

@@ -1,56 +0,0 @@
-"""
-PyTorch Mobile PR builds (use linux host toolchain + mobile build options)
-"""
-
-import cimodel.lib.miniutils as miniutils
-import cimodel.data.simple.util.branch_filters
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_ASAN, DOCKER_IMAGE_NDK
-
-
-class MobileJob:
-    def __init__(self, docker_image, variant_parts, is_master_only=False):
-        self.docker_image = docker_image
-        self.variant_parts = variant_parts
-        self.is_master_only = is_master_only
-
-    def gen_tree(self):
-        non_phase_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "mobile",
-        ] + self.variant_parts
-
-        full_job_name = "_".join(non_phase_parts)
-        build_env_name = "-".join(non_phase_parts)
-
-        props_dict = {
-            "build_environment": build_env_name,
-            "build_only": miniutils.quote(str(int(True))),
-            "docker_image": self.docker_image,
-            "name": full_job_name,
-        }
-
-        if self.is_master_only:
-            props_dict["filters"] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        return [{"pytorch_linux_build": props_dict}]
-
-
-WORKFLOW_DATA = [
-    MobileJob(DOCKER_IMAGE_ASAN, ["build"]),
-    MobileJob(DOCKER_IMAGE_ASAN, ["custom", "build", "static"]),
-
-    # Use LLVM-DEV toolchain in android-ndk-r19c docker image
-    MobileJob(DOCKER_IMAGE_NDK, ["custom", "build", "dynamic"]),
-
-    # Use LLVM-DEV toolchain in android-ndk-r19c docker image
-    # Most of this CI is already covered by "mobile-custom-build-dynamic" job
-    MobileJob(DOCKER_IMAGE_NDK, ["code", "analysis"], True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/nightly_android.py

@@ -1,73 +0,0 @@
-from cimodel.data.simple.util.docker_constants import DOCKER_IMAGE_NDK
-
-
-class AndroidNightlyJob:
-    def __init__(self,
-                 variant,
-                 template_name,
-                 extra_props=None,
-                 with_docker=True,
-                 requires=None,
-                 no_build_suffix=False):
-
-        self.variant = variant
-        self.template_name = template_name
-        self.extra_props = extra_props or {}
-        self.with_docker = with_docker
-        self.requires = requires
-        self.no_build_suffix = no_build_suffix
-
-    def gen_tree(self):
-
-        base_name_parts = [
-            "pytorch",
-            "linux",
-            "xenial",
-            "py3",
-            "clang5",
-            "android",
-            "ndk",
-            "r19c",
-        ] + self.variant
-
-        build_suffix = [] if self.no_build_suffix else ["build"]
-        full_job_name = "_".join(["nightly"] + base_name_parts + build_suffix)
-        build_env_name = "-".join(base_name_parts)
-
-        props_dict = {
-            "name": full_job_name,
-            "requires": self.requires,
-            "filters": {"branches": {"only": "nightly"}},
-        }
-
-        props_dict.update(self.extra_props)
-
-        if self.with_docker:
-            props_dict["docker_image"] = DOCKER_IMAGE_NDK
-            props_dict["build_environment"] = build_env_name
-
-        return [{self.template_name: props_dict}]
-
-
-WORKFLOW_DATA = [
-    AndroidNightlyJob(["x86_32"], "pytorch_linux_build"),
-    AndroidNightlyJob(["x86_64"], "pytorch_linux_build"),
-    AndroidNightlyJob(["arm", "v7a"], "pytorch_linux_build"),
-    AndroidNightlyJob(["arm", "v8a"], "pytorch_linux_build"),
-    AndroidNightlyJob(["android_gradle"], "pytorch_android_gradle_build",
-                      with_docker=False,
-                      requires=[
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build",
-                          "nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build"]),
-    AndroidNightlyJob(["x86_32_android_publish_snapshot"], "pytorch_android_publish_snapshot",
-                      extra_props={"context": "org-member"},
-                      with_docker=False,
-                      requires=["nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_android_gradle_build"],
-                      no_build_suffix=True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/nightly_ios.py

@@ -1,68 +0,0 @@
-import cimodel.data.simple.ios_definitions as ios_definitions
-
-
-class IOSNightlyJob:
-    def __init__(self,
-                 variant,
-                 is_upload=False):
-
-        self.variant = variant
-        self.is_upload = is_upload
-
-    def get_phase_name(self):
-        return "upload" if self.is_upload else "build"
-
-    def get_common_name_pieces(self, with_version_dots):
-
-        extra_name_suffix = [self.get_phase_name()] if self.is_upload else []
-
-        common_name_pieces = [
-            "ios",
-        ] + ios_definitions.IOS_VERSION.render_dots_or_parts(with_version_dots) + [
-            "nightly",
-            self.variant,
-            "build",
-        ] + extra_name_suffix
-
-        return common_name_pieces
-
-    def gen_job_name(self):
-        return "_".join(["pytorch"] + self.get_common_name_pieces(False))
-
-    def gen_tree(self):
-        extra_requires = [x.gen_job_name() for x in BUILD_CONFIGS] if self.is_upload else []
-
-        props_dict = {
-            "build_environment": "-".join(["libtorch"] + self.get_common_name_pieces(True)),
-            "requires": extra_requires,
-            "context": "org-member",
-            "filters": {"branches": {"only": "nightly"}},
-        }
-
-        if not self.is_upload:
-            props_dict["ios_arch"] = self.variant
-            props_dict["ios_platform"] = ios_definitions.get_platform(self.variant)
-            props_dict["name"] = self.gen_job_name()
-
-        template_name = "_".join([
-            "binary",
-            "ios",
-            self.get_phase_name(),
-        ])
-
-        return [{template_name: props_dict}]
-
-
-BUILD_CONFIGS = [
-    IOSNightlyJob("x86_64"),
-    IOSNightlyJob("arm64"),
-]
-
-
-WORKFLOW_DATA = BUILD_CONFIGS + [
-    IOSNightlyJob("binary", is_upload=True),
-]
-
-
-def get_workflow_jobs():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/data/simple/util/branch_filters.py

@@ -1,22 +0,0 @@
-NON_PR_BRANCH_LIST = [
-    "master",
-    r"/ci-all\/.*/",
-    r"/release\/.*/",
-]
-
-RC_PATTERN = r"/v[0-9]+(\.[0-9]+)*-rc[0-9]+/"
-
-def gen_filter_dict(
-        branches_list=NON_PR_BRANCH_LIST,
-        tags_list=None
-):
-    """Generates a filter dictionary for use with CircleCI's job filter"""
-    filter_dict = {
-        "branches": {
-            "only": branches_list,
-        },
-    }
-
-    if tags_list is not None:
-        filter_dict["tags"] = {"only": tags_list}
-    return filter_dict

.circleci/cimodel/data/simple/util/docker_constants.py

@@ -1,30 +0,0 @@
-AWS_DOCKER_HOST = "308535385114.dkr.ecr.us-east-1.amazonaws.com"
-
-# ARE YOU EDITING THIS NUMBER?  MAKE SURE YOU READ THE GUIDANCE AT THE
-# TOP OF .circleci/config.yml
-DOCKER_IMAGE_TAG = "209062ef-ab58-422a-b295-36c4eed6e906"
-
-
-def gen_docker_image_path(container_type):
-    return "/".join([
-        AWS_DOCKER_HOST,
-        "pytorch",
-        container_type + ":" + DOCKER_IMAGE_TAG,
-    ])
-
-
-DOCKER_IMAGE_BASIC = gen_docker_image_path("pytorch-linux-xenial-py3.6-gcc5.4")
-
-DOCKER_IMAGE_CUDA_10_2 = gen_docker_image_path("pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7")
-
-DOCKER_IMAGE_GCC7 = gen_docker_image_path("pytorch-linux-xenial-py3.6-gcc7")
-
-
-def gen_mobile_docker_name(specifier):
-    container_type = "pytorch-linux-xenial-py3-clang5-" + specifier
-    return gen_docker_image_path(container_type)
-
-
-DOCKER_IMAGE_ASAN = gen_mobile_docker_name("asan")
-
-DOCKER_IMAGE_NDK = gen_mobile_docker_name("android-ndk-r19c")

.circleci/cimodel/data/simple/util/versions.py

@@ -1,31 +0,0 @@
-class MultiPartVersion:
-    def __init__(self, parts, prefix=""):
-        self.parts = parts
-        self.prefix = prefix
-
-    def prefixed_parts(self):
-        """
-        Prepends the first element of the version list
-        with the prefix string.
-        """
-        if self.parts:
-            return [self.prefix + str(self.parts[0])] + list(map(str, self.parts[1:]))
-        else:
-            return [self.prefix]
-
-    def render_dots(self):
-        return ".".join(self.prefixed_parts())
-
-    def render_dots_or_parts(self, with_dots):
-        if with_dots:
-            return [self.render_dots()]
-        else:
-            return self.prefixed_parts()
-
-
-class CudaVersion(MultiPartVersion):
-    def __init__(self, major, minor):
-        self.major = major
-        self.minor = minor
-
-        super().__init__([self.major, self.minor], "cuda")

.circleci/cimodel/data/windows_build_definitions.py

@@ -1,142 +0,0 @@
-import cimodel.data.simple.util.branch_filters
-import cimodel.lib.miniutils as miniutils
-from cimodel.data.simple.util.versions import CudaVersion
-
-
-class WindowsJob:
-    def __init__(
-        self,
-        test_index,
-        vscode_spec,
-        cuda_version,
-        force_on_cpu=False,
-        master_only_pred=lambda job: job.vscode_spec.year != 2019,
-    ):
-        self.test_index = test_index
-        self.vscode_spec = vscode_spec
-        self.cuda_version = cuda_version
-        self.force_on_cpu = force_on_cpu
-        self.master_only_pred = master_only_pred
-
-    def gen_tree(self):
-
-        base_phase = "build" if self.test_index is None else "test"
-        numbered_phase = (
-            base_phase if self.test_index is None else base_phase + str(self.test_index)
-        )
-
-        key_name = "_".join(["pytorch", "windows", base_phase])
-
-        cpu_forcing_name_parts = ["on", "cpu"] if self.force_on_cpu else []
-
-        target_arch = self.cuda_version.render_dots() if self.cuda_version else "cpu"
-
-        base_name_parts = [
-            "pytorch",
-            "windows",
-            self.vscode_spec.render(),
-            "py36",
-            target_arch,
-        ]
-
-        prerequisite_jobs = []
-        if base_phase == "test":
-            prerequisite_jobs.append("_".join(base_name_parts + ["build"]))
-
-        arch_env_elements = (
-            ["cuda" + str(self.cuda_version.major), "cudnn7"]
-            if self.cuda_version
-            else ["cpu"]
-        )
-
-        build_environment_string = "-".join(
-            ["pytorch", "win"]
-            + self.vscode_spec.get_elements()
-            + arch_env_elements
-            + ["py3"]
-        )
-
-        is_running_on_cuda = bool(self.cuda_version) and not self.force_on_cpu
-
-        props_dict = {
-            "build_environment": build_environment_string,
-            "python_version": miniutils.quote("3.6"),
-            "vc_version": miniutils.quote(self.vscode_spec.dotted_version()),
-            "vc_year": miniutils.quote(str(self.vscode_spec.year)),
-            "vc_product": self.vscode_spec.get_product(),
-            "use_cuda": miniutils.quote(str(int(is_running_on_cuda))),
-            "requires": prerequisite_jobs,
-        }
-
-        if self.master_only_pred(self):
-            props_dict[
-                "filters"
-            ] = cimodel.data.simple.util.branch_filters.gen_filter_dict()
-
-        name_parts = base_name_parts + cpu_forcing_name_parts + [numbered_phase]
-
-        if base_phase == "test":
-            test_name = "-".join(["pytorch", "windows", numbered_phase])
-            props_dict["test_name"] = test_name
-
-            if is_running_on_cuda:
-                props_dict["executor"] = "windows-with-nvidia-gpu"
-
-        props_dict["cuda_version"] = (
-            miniutils.quote(str(self.cuda_version.major))
-            if self.cuda_version
-            else "cpu"
-        )
-        props_dict["name"] = "_".join(name_parts)
-
-        return [{key_name: props_dict}]
-
-
-class VcSpec:
-    def __init__(self, year, version_elements=None):
-        self.year = year
-        self.version_elements = version_elements or []
-
-    def get_elements(self):
-        return [self.prefixed_year()] + self.version_elements
-
-    def get_product(self):
-        return "Community" if self.year == 2019 else "BuildTools"
-
-    def dotted_version(self):
-        return ".".join(self.version_elements)
-
-    def prefixed_year(self):
-        return "vs" + str(self.year)
-
-    def render(self):
-        return "_".join(filter(None, [self.prefixed_year(), self.dotted_version()]))
-
-def FalsePred(_):
-    return False
-
-def TruePred(_):
-    return True
-
-WORKFLOW_DATA = [
-    # VS2017 CUDA-10.1
-    WindowsJob(None, VcSpec(2017, ["14", "11"]), CudaVersion(10, 1), master_only_pred=FalsePred),
-    WindowsJob(1, VcSpec(2017, ["14", "11"]), CudaVersion(10, 1)),
-    # VS2017 no-CUDA (builds only)
-    WindowsJob(None, VcSpec(2017, ["14", "16"]), CudaVersion(10, 1)),
-    WindowsJob(None, VcSpec(2017, ["14", "16"]), None),
-    # VS2019 CUDA-10.1
-    WindowsJob(None, VcSpec(2019), CudaVersion(10, 1)),
-    WindowsJob(1, VcSpec(2019), CudaVersion(10, 1)),
-    WindowsJob(2, VcSpec(2019), CudaVersion(10, 1)),
-    # VS2019 CPU-only
-    WindowsJob(None, VcSpec(2019), None),
-    WindowsJob(1, VcSpec(2019), None),
-    WindowsJob(2, VcSpec(2019), None, master_only_pred=TruePred),
-    WindowsJob(1, VcSpec(2019), CudaVersion(10, 1), force_on_cpu=True),
-    WindowsJob(2, VcSpec(2019), CudaVersion(10, 1), force_on_cpu=True, master_only_pred=TruePred),
-]
-
-
-def get_windows_workflows():
-    return [item.gen_tree() for item in WORKFLOW_DATA]

.circleci/cimodel/lib/conf_tree.py

@@ -1,107 +0,0 @@
-from dataclasses import dataclass, field
-from typing import Optional, Dict
-
-
-def X(val):
-    """
-    Compact way to write a leaf node
-    """
-    return val, []
-
-
-def XImportant(name):
-    """Compact way to write an important (run on PRs) leaf node"""
-    return (name, [("important", [X(True)])])
-
-
-@dataclass
-class Ver:
-    """
-    Represents a product with a version number
-    """
-    name: str
-    version: str = ""
-
-    def __str__(self):
-        return self.name + self.version
-
-
-@dataclass
-class ConfigNode:
-    parent: Optional['ConfigNode']
-    node_name: str
-    props: Dict[str, str] = field(default_factory=dict)
-
-    def get_label(self):
-        return self.node_name
-
-    # noinspection PyMethodMayBeStatic
-    def get_children(self):
-        return []
-
-    def get_parents(self):
-        return (self.parent.get_parents() + [self.parent.get_label()]) if self.parent else []
-
-    def get_depth(self):
-        return len(self.get_parents())
-
-    def get_node_key(self):
-        return "%".join(self.get_parents() + [self.get_label()])
-
-    def find_prop(self, propname, searched=None):
-        """
-        Checks if its own dictionary has
-        the property, otherwise asks parent node.
-        """
-
-        if searched is None:
-            searched = []
-
-        searched.append(self.node_name)
-
-        if propname in self.props:
-            return self.props[propname]
-        elif self.parent:
-            return self.parent.find_prop(propname, searched)
-        else:
-            # raise Exception('Property "%s" does not exist anywhere in the tree! Searched: %s' % (propname, searched))
-            return None
-
-
-def dfs_recurse(
-        node,
-        leaf_callback=lambda x: None,
-        discovery_callback=lambda x, y, z: None,
-        child_callback=lambda x, y: None,
-        sibling_index=0,
-        sibling_count=1):
-
-    discovery_callback(node, sibling_index, sibling_count)
-
-    node_children = node.get_children()
-    if node_children:
-        for i, child in enumerate(node_children):
-            child_callback(node, child)
-
-            dfs_recurse(
-                child,
-                leaf_callback,
-                discovery_callback,
-                child_callback,
-                i,
-                len(node_children),
-            )
-    else:
-        leaf_callback(node)
-
-
-def dfs(toplevel_config_node):
-
-    config_list = []
-
-    def leaf_callback(node):
-        config_list.append(node)
-
-    dfs_recurse(toplevel_config_node, leaf_callback)
-
-    return config_list

.circleci/cimodel/lib/miniutils.py

@@ -1,10 +0,0 @@
-def quote(s):
-    return sandwich('"', s)
-
-
-def sandwich(bread, jam):
-    return bread + jam + bread
-
-
-def override(word, substitutions):
-    return substitutions.get(word, word)

.circleci/cimodel/lib/miniyaml.py

@@ -1,52 +0,0 @@
-from collections import OrderedDict
-
-import cimodel.lib.miniutils as miniutils
-
-
-LIST_MARKER = "- "
-INDENTATION_WIDTH = 2
-
-
-def is_dict(data):
-    return type(data) in [dict, OrderedDict]
-
-
-def is_collection(data):
-    return is_dict(data) or type(data) is list
-
-
-def render(fh, data, depth, is_list_member=False):
-    """
-    PyYaml does not allow precise control over the quoting
-    behavior, especially for merge references.
-    Therefore, we use this custom YAML renderer.
-    """
-
-    indentation = " " * INDENTATION_WIDTH * depth
-
-    if is_dict(data):
-
-        tuples = list(data.items())
-        if type(data) is not OrderedDict:
-            tuples.sort()
-
-        for i, (k, v) in enumerate(tuples):
-            if not v:
-                continue
-            # If this dict is itself a list member, the first key gets prefixed with a list marker
-            list_marker_prefix = LIST_MARKER if is_list_member and not i else ""
-
-            trailing_whitespace = "\n" if is_collection(v) else " "
-            fh.write(indentation + list_marker_prefix + k + ":" + trailing_whitespace)
-
-            render(fh, v, depth + 1 + int(is_list_member))
-
-    elif type(data) is list:
-        for v in data:
-            render(fh, v, depth, True)
-
-    else:
-        # use empty quotes to denote an empty string value instead of blank space
-        modified_data = miniutils.quote(data) if data == "" else data
-        list_member_prefix = indentation + LIST_MARKER if is_list_member else ""
-        fh.write(list_member_prefix + str(modified_data) + "\n")

.circleci/codegen_validation/compare_normalized_yaml.sh

@@ -1,17 +0,0 @@
-#!/bin/bash -xe
-
-
-YAML_FILENAME=verbatim-sources/workflows-pytorch-ge-config-tests.yml
-DIFF_TOOL=meld
-
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-pushd ..
-
-
-$DIFF_TOOL $YAML_FILENAME <(./codegen_validation/normalize_yaml_fragment.py < $YAML_FILENAME)
-
-
-popd

.circleci/codegen_validation/normalize_yaml_fragment.py

@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-
-import os
-import sys
-import yaml
-
-# Need to import modules that lie on an upward-relative path
-sys.path.append(os.path.join(sys.path[0], '..'))
-
-import cimodel.lib.miniyaml as miniyaml
-
-
-def regurgitate(depth, use_pyyaml_formatter=False):
-    data = yaml.safe_load(sys.stdin)
-
-    if use_pyyaml_formatter:
-        output = yaml.dump(data, sort_keys=True)
-        sys.stdout.write(output)
-    else:
-        miniyaml.render(sys.stdout, data, depth)
-
-
-if __name__ == "__main__":
-    regurgitate(3)

.circleci/codegen_validation/overwrite_with_normalized.sh

@@ -1,15 +0,0 @@
-#!/bin/bash -xe
-
-YAML_FILENAME=$1
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-pushd ..
-
-TEMP_FILENAME=$(mktemp)
-
-cat $YAML_FILENAME | ./codegen_validation/normalize_yaml_fragment.py > $TEMP_FILENAME
-mv $TEMP_FILENAME $YAML_FILENAME
-
-popd

.circleci/docker/README.md

@@ -1,19 +0,0 @@
-# Docker images for Jenkins
-
-This directory contains everything needed to build the Docker images
-that are used in our CI
-
-The Dockerfiles located in subdirectories are parameterized to
-conditionally run build stages depending on build arguments passed to
-`docker build`. This lets us use only a few Dockerfiles for many
-images. The different configurations are identified by a freeform
-string that we call a _build environment_. This string is persisted in
-each image as the `BUILD_ENVIRONMENT` environment variable.
-
-See `build.sh` for valid build environments (it's the giant switch).
-
-## Contents
-
-* `build.sh` -- dispatch script to launch all builds
-* `common` -- scripts used to execute individual Docker build stages
-* `ubuntu-cuda` -- Dockerfile for Ubuntu image with CUDA support for nvidia-docker

.circleci/docker/android/AndroidManifest.xml

@@ -1 +0,0 @@
-<manifest package="org.pytorch.deps" />

.circleci/docker/android/build.gradle

@@ -1,68 +0,0 @@
-buildscript {
-    ext {
-        minSdkVersion = 21
-        targetSdkVersion = 28
-        compileSdkVersion = 28
-        buildToolsVersion = '28.0.3'
-
-        coreVersion = "1.2.0"
-        extJUnitVersion = "1.1.1"
-        runnerVersion = "1.2.0"
-        rulesVersion = "1.2.0"
-        junitVersion = "4.12"
-    }
-
-    repositories {
-        google()
-        mavenLocal()
-        mavenCentral()
-        jcenter()
-    }
-
-    dependencies {
-        classpath 'com.android.tools.build:gradle:3.3.2'
-        classpath "com.jfrog.bintray.gradle:gradle-bintray-plugin:1.8.0"
-        classpath "com.github.dcendents:android-maven-gradle-plugin:2.1"
-        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.9.8"
-    }
-}
-
-repositories {
-    google()
-    jcenter()
-}
-
-apply plugin: 'com.android.library'
-
-android {
-    compileSdkVersion rootProject.compileSdkVersion
-    buildToolsVersion rootProject.buildToolsVersion
-
-    defaultConfig {
-        minSdkVersion minSdkVersion
-        targetSdkVersion targetSdkVersion
-    }
-
-    sourceSets {
-        main {
-            manifest.srcFile 'AndroidManifest.xml'
-        }
-    }
-}
-
-dependencies {
-    implementation 'com.android.support:appcompat-v7:28.0.0'
-    implementation 'androidx.appcompat:appcompat:1.0.0'
-    implementation 'com.facebook.fbjni:fbjni-java-only:0.0.3'
-    implementation 'com.google.code.findbugs:jsr305:3.0.1'
-    implementation 'com.facebook.soloader:nativeloader:0.8.0'
-
-    implementation 'junit:junit:' + rootProject.junitVersion
-    implementation 'androidx.test:core:' + rootProject.coreVersion
-
-    implementation 'junit:junit:' + rootProject.junitVersion
-    implementation 'androidx.test:core:' + rootProject.coreVersion
-    implementation 'androidx.test.ext:junit:' + rootProject.extJUnitVersion
-    implementation 'androidx.test:rules:' + rootProject.rulesVersion
-    implementation 'androidx.test:runner:' + rootProject.runnerVersion
-}

.circleci/docker/build.sh

@@ -1,340 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-image="$1"
-shift
-
-if [ -z "${image}" ]; then
-  echo "Usage: $0 IMAGE"
-  exit 1
-fi
-
-# TODO: Generalize
-OS="ubuntu"
-DOCKERFILE="${OS}/Dockerfile"
-if [[ "$image" == *-cuda* ]]; then
-  DOCKERFILE="${OS}-cuda/Dockerfile"
-elif [[ "$image" == *-rocm* ]]; then
-  DOCKERFILE="${OS}-rocm/Dockerfile"
-fi
-
-if [[ "$image" == *-trusty* ]]; then
-  UBUNTU_VERSION=14.04
-elif [[ "$image" == *-xenial* ]]; then
-  UBUNTU_VERSION=16.04
-elif [[ "$image" == *-artful* ]]; then
-  UBUNTU_VERSION=17.10
-elif [[ "$image" == *-bionic* ]]; then
-  UBUNTU_VERSION=18.04
-elif [[ "$image" == *-focal* ]]; then
-  UBUNTU_VERSION=20.04
-fi
-
-TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/14.04/x86_64"
-
-# It's annoying to rename jobs every time you want to rewrite a
-# configuration, so we hardcode everything here rather than do it
-# from scratch
-case "$image" in
-  pytorch-linux-xenial-py3.8)
-    # TODO: This is a hack, get rid of this as soon as you get rid of the travis downloads
-    TRAVIS_DL_URL_PREFIX="https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/16.04/x86_64"
-    TRAVIS_PYTHON_VERSION=3.8
-    GCC_VERSION=7
-    # Do not install PROTOBUF, DB, and VISION as a test
-    ;;
-  pytorch-linux-xenial-py3.6-gcc4.8)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=4.8
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-py3.6-gcc5.4)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=5
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-py3.6-gcc7.2)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    # Do not install PROTOBUF, DB, and VISION as a test
-    ;;
-  pytorch-linux-xenial-py3.6-gcc7)
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-pynightly)
-    TRAVIS_PYTHON_VERSION=nightly
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc5.4)
-    CUDA_VERSION=9.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=5
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7)
-    CUDA_VERSION=9.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda10-cudnn7-py3-gcc7)
-    CUDA_VERSION=10.0
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-cuda10.1-cudnn7-py3-gcc7)
-    CUDA_VERSION=10.1
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-cuda11.0-cudnn8-py3-gcc7)
-    UBUNTU_VERSION=16.04-rc
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-py3-clang5-asan)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=5.0
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-xenial-py3-clang5-android-ndk-r19c)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=5.0
-    LLVMDEV=yes
-    PROTOBUF=yes
-    ANDROID=yes
-    ANDROID_NDK_VERSION=r19c
-    GRADLE_VERSION=4.10.3
-    CMAKE_VERSION=3.7.0
-    NINJA_VERSION=1.9.0
-    ;;
-  pytorch-linux-xenial-py3.6-clang7)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=7
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-py3.6-clang9)
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-py3.8-gcc9)
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda10.2-cudnn7-py3.6-clang9)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.6
-    CLANG_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda10.2-cudnn7-py3.8-gcc9)
-    CUDA_VERSION=10.2
-    CUDNN_VERSION=7
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ;;
-  pytorch-linux-bionic-cuda11.0-cudnn8-py3.6-gcc9)
-    UBUNTU_VERSION=18.04-rc
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.6
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-bionic-cuda11.0-cudnn8-py3.8-gcc9)
-    UBUNTU_VERSION=18.04-rc
-    CUDA_VERSION=11.0
-    CUDNN_VERSION=8
-    ANACONDA_PYTHON_VERSION=3.8
-    GCC_VERSION=9
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    KATEX=yes
-    ;;
-  pytorch-linux-xenial-rocm3.3-py3.6)
-    ANACONDA_PYTHON_VERSION=3.6
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ROCM_VERSION=3.3
-    # newer cmake version required
-    CMAKE_VERSION=3.6.3
-    ;;
-  pytorch-linux-bionic-rocm3.3-py3.6)
-    ANACONDA_PYTHON_VERSION=3.6
-    PROTOBUF=yes
-    DB=yes
-    VISION=yes
-    ROCM_VERSION=3.3
-    ;;
-esac
-
-# Set Jenkins UID and GID if running Jenkins
-if [ -n "${JENKINS:-}" ]; then
-  JENKINS_UID=$(id -u jenkins)
-  JENKINS_GID=$(id -g jenkins)
-fi
-
-tmp_tag="tmp-$(cat /dev/urandom | tr -dc 'a-z' | fold -w 32 | head -n 1)"
-
-# Build image
-# TODO: build-arg THRIFT is not turned on for any image, remove it once we confirm
-# it's no longer needed.
-docker build \
-       --no-cache \
-       --progress=plain \
-       --build-arg "TRAVIS_DL_URL_PREFIX=${TRAVIS_DL_URL_PREFIX}" \
-       --build-arg "BUILD_ENVIRONMENT=${image}" \
-       --build-arg "PROTOBUF=${PROTOBUF:-}" \
-       --build-arg "THRIFT=${THRIFT:-}" \
-       --build-arg "LLVMDEV=${LLVMDEV:-}" \
-       --build-arg "DB=${DB:-}" \
-       --build-arg "VISION=${VISION:-}" \
-       --build-arg "EC2=${EC2:-}" \
-       --build-arg "JENKINS=${JENKINS:-}" \
-       --build-arg "JENKINS_UID=${JENKINS_UID:-}" \
-       --build-arg "JENKINS_GID=${JENKINS_GID:-}" \
-       --build-arg "UBUNTU_VERSION=${UBUNTU_VERSION}" \
-       --build-arg "CLANG_VERSION=${CLANG_VERSION}" \
-       --build-arg "ANACONDA_PYTHON_VERSION=${ANACONDA_PYTHON_VERSION}" \
-       --build-arg "TRAVIS_PYTHON_VERSION=${TRAVIS_PYTHON_VERSION}" \
-       --build-arg "GCC_VERSION=${GCC_VERSION}" \
-       --build-arg "CUDA_VERSION=${CUDA_VERSION}" \
-       --build-arg "CUDNN_VERSION=${CUDNN_VERSION}" \
-       --build-arg "ANDROID=${ANDROID}" \
-       --build-arg "ANDROID_NDK=${ANDROID_NDK_VERSION}" \
-       --build-arg "GRADLE_VERSION=${GRADLE_VERSION}" \
-       --build-arg "CMAKE_VERSION=${CMAKE_VERSION:-}" \
-       --build-arg "NINJA_VERSION=${NINJA_VERSION:-}" \
-       --build-arg "KATEX=${KATEX:-}" \
-       --build-arg "ROCM_VERSION=${ROCM_VERSION:-}" \
-       -f $(dirname ${DOCKERFILE})/Dockerfile \
-       -t "$tmp_tag" \
-       "$@" \
-       .
-
-function drun() {
-  docker run --rm "$tmp_tag" $*
-}
-
-if [[ "$OS" == "ubuntu" ]]; then
-  if !(drun lsb_release -a 2>&1 | grep -qF Ubuntu); then
-    echo "OS=ubuntu, but:"
-    drun lsb_release -a
-    exit 1
-  fi
-  if !(drun lsb_release -a 2>&1 | grep -qF "$UBUNTU_VERSION"); then
-    echo "UBUNTU_VERSION=$UBUNTU_VERSION, but:"
-    drun lsb_release -a
-    exit 1
-  fi
-fi
-
-if [ -n "$TRAVIS_PYTHON_VERSION" ]; then
-  if [[ "$TRAVIS_PYTHON_VERSION" != nightly ]]; then
-    if !(drun python --version 2>&1 | grep -qF "Python $TRAVIS_PYTHON_VERSION"); then
-      echo "TRAVIS_PYTHON_VERSION=$TRAVIS_PYTHON_VERSION, but:"
-      drun python --version
-      exit 1
-    fi
-  else
-    echo "Please manually check nightly is OK:"
-    drun python --version
-  fi
-fi
-
-if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
-  if !(drun python --version 2>&1 | grep -qF "Python $ANACONDA_PYTHON_VERSION"); then
-    echo "ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION, but:"
-    drun python --version
-    exit 1
-  fi
-fi
-
-if [ -n "$GCC_VERSION" ]; then
-  if !(drun gcc --version 2>&1 | grep -q " $GCC_VERSION\\W"); then
-    echo "GCC_VERSION=$GCC_VERSION, but:"
-    drun gcc --version
-    exit 1
-  fi
-fi
-
-if [ -n "$CLANG_VERSION" ]; then
-  if !(drun clang --version 2>&1 | grep -qF "clang version $CLANG_VERSION"); then
-    echo "CLANG_VERSION=$CLANG_VERSION, but:"
-    drun clang --version
-    exit 1
-  fi
-fi
-
-if [ -n "$KATEX" ]; then
-  if !(drun katex --version); then
-    echo "KATEX=$KATEX, but:"
-    drun katex --version
-    exit 1
-  fi
-fi

.circleci/docker/build_docker.sh

@@ -1,53 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-retry () {
-    $*  || (sleep 1 && $*) || (sleep 2 && $*)
-}
-
-# If UPSTREAM_BUILD_ID is set (see trigger job), then we can
-# use it to tag this build with the same ID used to tag all other
-# base image builds. Also, we can try and pull the previous
-# image first, to avoid rebuilding layers that haven't changed.
-
-#until we find a way to reliably reuse previous build, this last_tag is not in use
-# last_tag="$(( CIRCLE_BUILD_NUM - 1 ))"
-tag="${CIRCLE_WORKFLOW_ID}"
-
-
-registry="308535385114.dkr.ecr.us-east-1.amazonaws.com"
-image="${registry}/pytorch/${IMAGE_NAME}"
-
-login() {
-  aws ecr get-authorization-token --region us-east-1 --output text --query 'authorizationData[].authorizationToken' |
-    base64 -d |
-    cut -d: -f2 |
-    docker login -u AWS --password-stdin "$1"
-}
-
-# Retry on timeouts (can happen on job stampede).
-retry login "${registry}"
-
-# Logout on exit
-trap "docker logout ${registry}" EXIT
-
-# export EC2=1
-# export JENKINS=1
-
-# Try to pull the previous image (perhaps we can reuse some layers)
-# if [ -n "${last_tag}" ]; then
-#   docker pull "${image}:${last_tag}" || true
-# fi
-
-# Build new image
-./build.sh ${IMAGE_NAME} -t "${image}:${tag}"
-
-docker push "${image}:${tag}"
-
-# TODO: Get rid of duplicate tagging once ${DOCKER_TAG} becomes the default
-docker tag "${image}:${tag}" "${image}:${DOCKER_TAG}"
-docker push "${image}:${DOCKER_TAG}"
-
-docker save -o "${IMAGE_NAME}:${tag}.tar" "${image}:${tag}"
-aws s3 cp "${IMAGE_NAME}:${tag}.tar" "s3://ossci-linux-build/pytorch/base/${IMAGE_NAME}:${tag}.tar" --acl public-read

.circleci/docker/common/install_android.sh

@@ -1,129 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "${ANDROID_NDK}" ]
-
-apt-get update
-apt-get install -y --no-install-recommends autotools-dev autoconf unzip
-apt-get autoclean && apt-get clean
-rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-pushd /tmp
-curl -Os --retry 3 https://dl.google.com/android/repository/android-ndk-${ANDROID_NDK}-linux-x86_64.zip
-popd
-_ndk_dir=/opt/ndk
-mkdir -p "$_ndk_dir"
-unzip -qo /tmp/android*.zip -d "$_ndk_dir"
-_versioned_dir=$(find "$_ndk_dir/" -mindepth 1 -maxdepth 1 -type d)
-mv "$_versioned_dir"/* "$_ndk_dir"/
-rmdir "$_versioned_dir"
-rm -rf /tmp/*
-
-# Install OpenJDK
-# https://hub.docker.com/r/picoded/ubuntu-openjdk-8-jdk/dockerfile/
-
-sudo apt-get update && \
-    apt-get install -y openjdk-8-jdk && \
-    apt-get install -y ant && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/* && \
-    rm -rf /var/cache/oracle-jdk8-installer;
-
-# Fix certificate issues, found as of
-# https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/983302
-
-sudo apt-get update && \
-    apt-get install -y ca-certificates-java && \
-    apt-get clean && \
-    update-ca-certificates -f && \
-    rm -rf /var/lib/apt/lists/* && \
-    rm -rf /var/cache/oracle-jdk8-installer;
-
-export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
-
-# Installing android sdk
-# https://github.com/circleci/circleci-images/blob/staging/android/Dockerfile.m4
-
-_sdk_version=sdk-tools-linux-3859397.zip
-_android_home=/opt/android/sdk
-
-rm -rf $_android_home
-sudo mkdir -p $_android_home
-curl --silent --show-error --location --fail --retry 3 --output /tmp/$_sdk_version https://dl.google.com/android/repository/$_sdk_version
-sudo unzip -q /tmp/$_sdk_version -d $_android_home
-rm /tmp/$_sdk_version
-
-sudo chmod -R 777 $_android_home
-
-export ANDROID_HOME=$_android_home
-export ADB_INSTALL_TIMEOUT=120
-
-export PATH="${ANDROID_HOME}/emulator:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools:${PATH}"
-echo "PATH:${PATH}"
-alias sdkmanager="$ANDROID_HOME/tools/bin/sdkmanager"
-
-sudo mkdir ~/.android && sudo echo '### User Sources for Android SDK Manager' > ~/.android/repositories.cfg
-sudo chmod -R 777 ~/.android
-
-yes | sdkmanager --licenses
-yes | sdkmanager --update
-
-sdkmanager \
-  "tools" \
-  "platform-tools" \
-  "emulator"
-
-sdkmanager \
-  "build-tools;28.0.3" \
-  "build-tools;29.0.2"
-
-sdkmanager \
-  "platforms;android-28" \
-  "platforms;android-29"
-sdkmanager --list
-
-# Installing Gradle
-echo "GRADLE_VERSION:${GRADLE_VERSION}"
-_gradle_home=/opt/gradle
-sudo rm -rf $gradle_home
-sudo mkdir -p $_gradle_home
-
-wget --no-verbose --output-document=/tmp/gradle.zip \
-"https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip"
-
-sudo unzip -q /tmp/gradle.zip -d $_gradle_home
-rm /tmp/gradle.zip
-
-sudo chmod -R 777 $_gradle_home
-
-export GRADLE_HOME=$_gradle_home/gradle-$GRADLE_VERSION
-alias gradle="${GRADLE_HOME}/bin/gradle"
-
-export PATH="${GRADLE_HOME}/bin/:${PATH}"
-echo "PATH:${PATH}"
-
-gradle --version
-
-mkdir /var/lib/jenkins/gradledeps
-cp build.gradle /var/lib/jenkins/gradledeps
-cp AndroidManifest.xml /var/lib/jenkins/gradledeps
-
-pushd /var/lib/jenkins
-
-export GRADLE_LOCAL_PROPERTIES=gradledeps/local.properties
-rm -f $GRADLE_LOCAL_PROPERTIES
-echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-
-chown -R jenkins /var/lib/jenkins/gradledeps
-chgrp -R jenkins /var/lib/jenkins/gradledeps
-
-sudo -H -u jenkins $GRADLE_HOME/bin/gradle -p /var/lib/jenkins/gradledeps -g /var/lib/jenkins/.gradle --refresh-dependencies --debug --stacktrace assemble
-
-chown -R jenkins /var/lib/jenkins/.gradle
-chgrp -R jenkins /var/lib/jenkins/.gradle
-
-popd
-
-rm -rf /var/lib/jenkins/.gradle/daemon

.circleci/docker/common/install_base.sh

@@ -1,75 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# NVIDIA dockers for RC releases use tag names like `11.0-cudnn8-devel-ubuntu18.04-rc`,
-# for this case we will set UBUNTU_VERSION to `18.04-rc` so that the Dockerfile could
-# find the correct image. As a result, here we have to check for
-#   "$UBUNTU_VERSION" == "18.04"*
-# instead of
-#   "$UBUNTU_VERSION" == "18.04"
-if [[ "$UBUNTU_VERSION" == "18.04"* ]]; then
-  cmake3="cmake=3.10*"
-else
-  cmake3="cmake=3.5*"
-fi
-
-# Install common dependencies
-apt-get update
-# TODO: Some of these may not be necessary
-# TODO: libiomp also gets installed by conda, aka there's a conflict
-ccache_deps="asciidoc docbook-xml docbook-xsl xsltproc"
-numpy_deps="gfortran"
-apt-get install -y --no-install-recommends \
-  $ccache_deps \
-  $numpy_deps \
-  ${cmake3} \
-  apt-transport-https \
-  autoconf \
-  automake \
-  build-essential \
-  ca-certificates \
-  curl \
-  git \
-  libatlas-base-dev \
-  libc6-dbg \
-  libiomp-dev \
-  libyaml-dev \
-  libz-dev \
-  libjpeg-dev \
-  libasound2-dev \
-  libsndfile-dev \
-  python \
-  python-dev \
-  python-setuptools \
-  python-wheel \
-  software-properties-common \
-  sudo \
-  wget \
-  vim
-
-# Install Valgrind separately since the apt-get version is too old.
-mkdir valgrind_build && cd valgrind_build
-VALGRIND_VERSION=3.15.0
-if ! wget http://valgrind.org/downloads/valgrind-${VALGRIND_VERSION}.tar.bz2
-then
-  wget https://sourceware.org/ftp/valgrind/valgrind-${VALGRIND_VERSION}.tar.bz2
-fi
-tar -xjf valgrind-${VALGRIND_VERSION}.tar.bz2
-cd valgrind-${VALGRIND_VERSION}
-./configure --prefix=/usr/local
-make -j 4
-sudo make install
-cd ../../
-rm -rf valgrind_build
-alias valgrind="/usr/local/bin/valgrind"
-
-# TODO: THIS IS A HACK!!!
-# distributed nccl(2) tests are a bit busted, see https://github.com/pytorch/pytorch/issues/5877
-if dpkg -s libnccl-dev; then
-  apt-get remove -y libnccl-dev libnccl2 --allow-change-held-packages
-fi
-
-# Cleanup package manager
-apt-get autoclean && apt-get clean
-rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

.circleci/docker/common/install_cache.sh

@@ -1,35 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-mkdir -p /opt/cache/bin
-mkdir -p /opt/cache/lib
-sed -e 's|PATH="\(.*\)"|PATH="/opt/cache/bin:\1"|g' -i /etc/environment
-export PATH="/opt/cache/bin:$PATH"
-
-# Setup compiler cache
-curl --retry 3 https://s3.amazonaws.com/ossci-linux/sccache -o /opt/cache/bin/sccache
-chmod a+x /opt/cache/bin/sccache
-
-function write_sccache_stub() {
-  printf "#!/bin/sh\nexec sccache $(which $1) \$*" > "/opt/cache/bin/$1"
-  chmod a+x "/opt/cache/bin/$1"
-}
-
-write_sccache_stub cc
-write_sccache_stub c++
-write_sccache_stub gcc
-write_sccache_stub g++
-write_sccache_stub clang
-write_sccache_stub clang++
-
-if [ -n "$CUDA_VERSION" ]; then
-  # TODO: This is a workaround for the fact that PyTorch's FindCUDA
-  # implementation cannot find nvcc if it is setup this way, because it
-  # appears to search for the nvcc in PATH, and use its path to infer
-  # where CUDA is installed.  Instead, we install an nvcc symlink outside
-  # of the PATH, and set CUDA_NVCC_EXECUTABLE so that we make use of it.
-
-  printf "#!/bin/sh\nexec sccache $(which nvcc) \"\$@\"" > /opt/cache/lib/nvcc
-  chmod a+x /opt/cache/lib/nvcc
-fi

.circleci/docker/common/install_clang.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-if [ -n "$CLANG_VERSION" ]; then
-
-  if [[ $CLANG_VERSION == 7 && $UBUNTU_VERSION == 16.04 ]]; then
-    wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
-    sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main"
-  elif [[ $CLANG_VERSION == 9 && $UBUNTU_VERSION == 18.04 ]]; then
-    sudo apt-get update
-    # gpg-agent is not available by default on 18.04
-    sudo apt-get install  -y --no-install-recommends gpg-agent
-    wget --no-check-certificate -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add  -
-    apt-add-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-${CLANG_VERSION} main"
-  fi
-
-  sudo apt-get update
-  apt-get install -y --no-install-recommends clang-"$CLANG_VERSION"
-  apt-get install -y --no-install-recommends llvm-"$CLANG_VERSION"
-
-  # Install dev version of LLVM.
-  if [ -n "$LLVMDEV" ]; then
-    sudo apt-get install -y --no-install-recommends llvm-"$CLANG_VERSION"-dev
-  fi
-
-  # Use update-alternatives to make this version the default
-  # TODO: Decide if overriding gcc as well is a good idea
-  # update-alternatives --install /usr/bin/gcc gcc /usr/bin/clang-"$CLANG_VERSION" 50
-  # update-alternatives --install /usr/bin/g++ g++ /usr/bin/clang++-"$CLANG_VERSION" 50
-  update-alternatives --install /usr/bin/clang clang /usr/bin/clang-"$CLANG_VERSION" 50
-  update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-"$CLANG_VERSION" 50
-
-  # clang's packaging is a little messed up (the runtime libs aren't
-  # added into the linker path), so give it a little help
-  clang_lib=("/usr/lib/llvm-$CLANG_VERSION/lib/clang/"*"/lib/linux")
-  echo "$clang_lib" > /etc/ld.so.conf.d/clang.conf
-  ldconfig
-
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-fi

.circleci/docker/common/install_cmake.sh

@@ -1,16 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "$CMAKE_VERSION" ]
-
-# Turn 3.6.3 into v3.6
-path=$(echo "${CMAKE_VERSION}" | sed -e 's/\([0-9].[0-9]\+\).*/v\1/')
-file="cmake-${CMAKE_VERSION}-Linux-x86_64.tar.gz"
-
-# Download and install specific CMake version in /usr/local
-pushd /tmp
-curl -Os --retry 3 "https://cmake.org/files/${path}/${file}"
-tar -C /usr/local --strip-components 1 --no-same-owner -zxf cmake-*.tar.gz
-rm -f cmake-*.tar.gz
-popd

.circleci/docker/common/install_conda.sh

@@ -1,96 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# Optionally install conda
-if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
-  BASE_URL="https://repo.anaconda.com/miniconda"
-
-  MAJOR_PYTHON_VERSION=$(echo "$ANACONDA_PYTHON_VERSION" | cut -d . -f 1)
-
-  case "$MAJOR_PYTHON_VERSION" in
-    2)
-      CONDA_FILE="Miniconda2-latest-Linux-x86_64.sh"
-    ;;
-    3)
-      CONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
-    ;;
-    *)
-      echo "Unsupported ANACONDA_PYTHON_VERSION: $ANACONDA_PYTHON_VERSION"
-      exit 1
-      ;;
-  esac
-
-  mkdir /opt/conda
-  chown jenkins:jenkins /opt/conda
-
-  as_jenkins() {
-    # NB: unsetting the environment variables works around a conda bug
-    # https://github.com/conda/conda/issues/6576
-    # NB: Pass on PATH and LD_LIBRARY_PATH to sudo invocation
-    # NB: This must be run from a directory that jenkins has access to,
-    # works around https://github.com/conda/conda-package-handling/pull/34
-    sudo -H -u jenkins env -u SUDO_UID -u SUDO_GID -u SUDO_COMMAND -u SUDO_USER env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
-  }
-
-  pushd /tmp
-  wget -q "${BASE_URL}/${CONDA_FILE}"
-  chmod +x "${CONDA_FILE}"
-  as_jenkins ./"${CONDA_FILE}" -b -f -p "/opt/conda"
-  popd
-
-  # NB: Don't do this, rely on the rpath to get it right
-  #echo "/opt/conda/lib" > /etc/ld.so.conf.d/conda-python.conf
-  #ldconfig
-  sed -e 's|PATH="\(.*\)"|PATH="/opt/conda/bin:\1"|g' -i /etc/environment
-  export PATH="/opt/conda/bin:$PATH"
-
-  # Ensure we run conda in a directory that jenkins has write access to
-  pushd /opt/conda
-
-  # Track latest conda update
-  as_jenkins conda update -n base conda
-
-  # Install correct Python version
-  as_jenkins conda install python="$ANACONDA_PYTHON_VERSION"
-
-  conda_install() {
-    # Ensure that the install command don't upgrade/downgrade Python
-    # This should be called as
-    #   conda_install pkg1 pkg2 ... [-c channel]
-    as_jenkins conda install -q -y python="$ANACONDA_PYTHON_VERSION" $*
-  }
-
-  # Install PyTorch conda deps, as per https://github.com/pytorch/pytorch README
-  # DO NOT install cmake here as it would install a version newer than 3.5, but
-  # we want to pin to version 3.5.
-  if [ "$ANACONDA_PYTHON_VERSION" = "3.8" ]; then
-    # DO NOT install typing if installing python-3.8, since its part of python-3.8 core packages
-    # Install llvm-8 as it is required to compile llvmlite-0.30.0 from source
-    conda_install numpy pyyaml mkl mkl-include setuptools cffi future six llvmdev=8.0.0
-  else
-    conda_install numpy pyyaml mkl mkl-include setuptools cffi typing future six
-  fi
-  if [[ "$CUDA_VERSION" == 9.2* ]]; then
-    conda_install magma-cuda92 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.0* ]]; then
-    conda_install magma-cuda100 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.1* ]]; then
-    conda_install magma-cuda101 -c pytorch
-  elif [[ "$CUDA_VERSION" == 10.2* ]]; then
-    conda_install magma-cuda102 -c pytorch
-  fi
-
-  # TODO: This isn't working atm
-  conda_install nnpack -c killeent
-
-  # Install some other packages
-  # TODO: Why is scipy pinned
-  # numba & llvmlite is pinned because of https://github.com/numba/numba/issues/4368
-  # scikit-learn is pinned because of
-  # https://github.com/scikit-learn/scikit-learn/issues/14485 (affects gcc 5.5
-  # only)
-  as_jenkins pip install --progress-bar off pytest scipy==1.1.0 scikit-learn==0.20.3 scikit-image librosa>=0.6.2 psutil numba==0.46.0 llvmlite==0.30.0
-
-  popd
-fi

.circleci/docker/common/install_db.sh

@@ -1,61 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# This function installs protobuf 2.6
-install_protobuf_26() {
-  pb_dir="/usr/temp_pb_install_dir"
-  mkdir -p $pb_dir
-
-  # On the nvidia/cuda:9-cudnn7-devel-centos7 image we need this symlink or
-  # else it will fail with
-  #   g++: error: ./../lib64/crti.o: No such file or directory
-  ln -s /usr/lib64 "$pb_dir/lib64"
-
-  curl -LO "https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz"
-  tar -xvz -C "$pb_dir" --strip-components 1 -f protobuf-2.6.1.tar.gz
-  pushd "$pb_dir" && ./configure && make && make check && sudo make install && sudo ldconfig
-  popd
-  rm -rf $pb_dir
-}
-
-install_ubuntu() {
-  apt-get update
-  apt-get install -y --no-install-recommends \
-          libhiredis-dev \
-          libleveldb-dev \
-          liblmdb-dev \
-          libsnappy-dev
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Need EPEL for many packages we depend on.
-  # See http://fedoraproject.org/wiki/EPEL
-  yum --enablerepo=extras install -y epel-release
-
-  yum install -y \
-      hiredis-devel \
-      leveldb-devel \
-      lmdb-devel \
-      snappy-devel
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install base packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/common/install_gcc.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-if [ -n "$GCC_VERSION" ]; then
-
-  # Need the official toolchain repo to get alternate packages
-  add-apt-repository ppa:ubuntu-toolchain-r/test
-  apt-get update
-  if [ "$UBUNTU_VERSION" = "16.04" -a "$GCC_VERSION" = "5" ]; then
-    apt-get install -y g++-5=5.4.0-6ubuntu1~16.04.12
-  else
-    apt-get install -y g++-$GCC_VERSION
-  fi
-
-  update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-"$GCC_VERSION" 50
-  update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-"$GCC_VERSION" 50
-
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-fi

.circleci/docker/common/install_jni.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-mkdir -p /usr/local/include
-cp jni.h /usr/local/include

.circleci/docker/common/install_katex.sh

@@ -1,20 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-if [ -n "$KATEX" ]; then
-
-  curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
-  sudo apt-get install -y nodejs
-
-  curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
-  echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
-
-  apt-get update
-  apt-get install -y --no-install-recommends yarn
-  yarn global add katex --prefix /usr/local
-
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-fi

.circleci/docker/common/install_llvm.sh

@@ -1,30 +0,0 @@
-#!/bin/bash
-
-set -ex
-llvm_url="https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/llvm-9.0.1.src.tar.xz"
-
-mkdir /opt/llvm
-
-pushd /tmp
-wget --no-verbose --output-document=llvm.tar.xz "$llvm_url"
-mkdir llvm
-tar -xf llvm.tar.xz -C llvm --strip-components 1
-rm -f llvm.tar.xz
-
-cd llvm
-mkdir build
-cd build
-cmake -G "Unix Makefiles" \
-  -DCMAKE_BUILD_TYPE=MinSizeRel \
-  -DLLVM_ENABLE_ASSERTIONS=ON \
-  -DCMAKE_INSTALL_PREFIX=/opt/llvm \
-  -DLLVM_TARGETS_TO_BUILD="host" \
-  -DLLVM_BUILD_TOOLS=OFF \
-  -DLLVM_BUILD_UTILS=OFF \
-  -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON \
-  ../
-
-make -j4
-sudo make install
-
-popd

.circleci/docker/common/install_ninja.sh

@@ -1,13 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-[ -n "$NINJA_VERSION" ]
-
-url="https://github.com/ninja-build/ninja/releases/download/v${NINJA_VERSION}/ninja-linux.zip"
-
-pushd /tmp
-wget --no-verbose --output-document=ninja-linux.zip "$url"
-unzip ninja-linux.zip -d /usr/local/bin
-rm -f ninja-linux.zip
-popd

.circleci/docker/common/install_protobuf.sh

@@ -1,56 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# This function installs protobuf 2.6
-install_protobuf_26() {
-  pb_dir="/usr/temp_pb_install_dir"
-  mkdir -p $pb_dir
-
-  # On the nvidia/cuda:9-cudnn7-devel-centos7 image we need this symlink or
-  # else it will fail with
-  #   g++: error: ./../lib64/crti.o: No such file or directory
-  ln -s /usr/lib64 "$pb_dir/lib64"
-
-  curl -LO "https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz"
-  tar -xvz -C "$pb_dir" --strip-components 1 -f protobuf-2.6.1.tar.gz
-  pushd "$pb_dir" && ./configure && make && make check && sudo make install && sudo ldconfig
-  popd
-  rm -rf $pb_dir
-}
-
-install_ubuntu() {
-  # Ubuntu 14.04 ships with protobuf 2.5, but ONNX needs protobuf >= 2.6
-  # so we install that here if on 14.04
-  # Ubuntu 14.04 also has cmake 2.8.12 as the default option, so we will
-  # install cmake3 here and use cmake3.
-  apt-get update
-  if [[ "$UBUNTU_VERSION" == 14.04 ]]; then
-    apt-get install -y --no-install-recommends cmake3
-    install_protobuf_26
-  else
-    apt-get install -y --no-install-recommends \
-            libprotobuf-dev \
-            protobuf-compiler
-  fi
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Centos7 ships with protobuf 2.5, but ONNX needs protobuf >= 2.6
-  # so we always install install that here
-  install_protobuf_26
-}
-
-# Install base packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/common/install_rocm.sh

@@ -1,89 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-install_ubuntu() {
-    apt-get update
-    if [[ $UBUNTU_VERSION == 18.04 ]]; then
-      # gpg-agent is not available by default on 18.04
-      apt-get install -y --no-install-recommends gpg-agent
-    fi
-    apt-get install -y wget
-    apt-get install -y libopenblas-dev
-
-    # Need the libc++1 and libc++abi1 libraries to allow torch._C to load at runtime
-    apt-get install -y libc++1
-    apt-get install -y libc++abi1
-
-    DEB_ROCM_REPO=http://repo.radeon.com/rocm/apt/${ROCM_VERSION}
-    # Add rocm repository
-    wget -qO - $DEB_ROCM_REPO/rocm.gpg.key | apt-key add -
-    echo "deb [arch=amd64] $DEB_ROCM_REPO xenial main" > /etc/apt/sources.list.d/rocm.list
-    apt-get update --allow-insecure-repositories
-
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
-                   rocm-dev \
-                   rocm-utils \
-                   rocfft \
-                   miopen-hip \
-                   rocblas \
-                   hipsparse \
-                   rocrand \
-                   hipcub \
-                   rocthrust \
-                   rccl \
-                   rocprofiler-dev \
-                   roctracer-dev
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-
-  yum update -y
-  yum install -y wget
-  yum install -y openblas-devel
-
-  yum install -y epel-release
-  yum install -y dkms kernel-headers-`uname -r` kernel-devel-`uname -r`
-
-  echo "[ROCm]" > /etc/yum.repos.d/rocm.repo
-  echo "name=ROCm" >> /etc/yum.repos.d/rocm.repo
-  echo "baseurl=http://repo.radeon.com/rocm/yum/rpm/" >> /etc/yum.repos.d/rocm.repo
-  echo "enabled=1" >> /etc/yum.repos.d/rocm.repo
-  echo "gpgcheck=0" >> /etc/yum.repos.d/rocm.repo
-
-  yum update -y
-
-  yum install -y \
-                   rocm-dev \
-                   rocm-utils \
-                   rocfft \
-                   miopen-hip \
-                   rocblas \
-                   hipsparse \
-                   rocrand \
-                   rccl \
-                   hipcub \
-                   rocthrust \
-                   rocprofiler-dev \
-                   roctracer-dev
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install Python packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/common/install_thrift.sh

@@ -1,14 +0,0 @@
-apt-get update
-apt-get install -y sudo wget libboost-dev libboost-test-dev libboost-program-options-dev libboost-filesystem-dev libboost-thread-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev
-wget https://www-us.apache.org/dist/thrift/0.12.0/thrift-0.12.0.tar.gz
-tar -xvf thrift-0.12.0.tar.gz
-cd thrift-0.12.0
-for file in ./compiler/cpp/Makefile*; do
-  sed -i 's/\-Werror//' $file
-done
-./bootstrap.sh
-./configure --without-php --without-java --without-python --without-nodejs --without-go --without-ruby
-sudo make
-sudo make install
-cd ..
-rm thrift-0.12.0.tar.gz

.circleci/docker/common/install_travis_python.sh

@@ -1,97 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-as_jenkins() {
-  # NB: Preserve PATH and LD_LIBRARY_PATH changes
-  sudo -H -u jenkins env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
-}
-
-if [ -n "$TRAVIS_PYTHON_VERSION" ]; then
-
-  mkdir -p /opt/python
-  chown jenkins:jenkins /opt/python
-
-  # Download Python binary from Travis
-  pushd tmp
-  as_jenkins wget --quiet ${TRAVIS_DL_URL_PREFIX}/python-$TRAVIS_PYTHON_VERSION.tar.bz2
-  # NB: The tarball also comes with /home/travis virtualenv that we
-  # don't care about.  (Maybe we should, but we've worked around the
-  # "how do I install to python" issue by making this entire directory
-  # user-writable "lol")
-  # NB: Relative ordering of opt/python and flags matters
-  as_jenkins tar xjf python-$TRAVIS_PYTHON_VERSION.tar.bz2 --strip-components=2 --directory /opt/python opt/python
-  popd
-
-  echo "/opt/python/$TRAVIS_PYTHON_VERSION/lib" > /etc/ld.so.conf.d/travis-python.conf
-  ldconfig
-  sed -e 's|PATH="\(.*\)"|PATH="/opt/python/'"$TRAVIS_PYTHON_VERSION"'/bin:\1"|g' -i /etc/environment
-  export PATH="/opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH"
-
-  python --version
-  pip --version
-
-  # Install pip from source.
-  # The python-pip package on Ubuntu Trusty is old
-  # and upon install numpy doesn't use the binary
-  # distribution, and fails to compile it from source.
-  pushd tmp
-  as_jenkins curl -L -O https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz
-  as_jenkins tar zxf pip-9.0.1.tar.gz
-  pushd pip-9.0.1
-  as_jenkins python setup.py install
-  popd
-  rm -rf pip-9.0.1*
-  popd
-
-  # Install pip packages
-  as_jenkins pip install --upgrade pip
-
-  pip --version
-
-  if [[ "$TRAVIS_PYTHON_VERSION" == nightly ]]; then
-      # These two packages have broken Cythonizations uploaded
-      # to PyPi, see:
-      #
-      #  - https://github.com/numpy/numpy/issues/10500
-      #  - https://github.com/yaml/pyyaml/issues/117
-      #
-      # Furthermore, the released version of Cython does not
-      # have these issues fixed.
-      #
-      # While we are waiting on fixes for these, we build
-      # from Git for now.  Feel free to delete this conditional
-      # branch if things start working again (you may need
-      # to do this if these packages regress on Git HEAD.)
-      as_jenkins pip install git+https://github.com/cython/cython.git
-      as_jenkins pip install git+https://github.com/numpy/numpy.git
-      as_jenkins pip install git+https://github.com/yaml/pyyaml.git
-  else
-      as_jenkins pip install numpy pyyaml
-  fi
-
-  as_jenkins pip install \
-      future \
-      hypothesis \
-      protobuf \
-      pytest \
-      pillow \
-      typing
-
-  as_jenkins pip install mkl mkl-devel
-
-  # SciPy does not support Python 3.7 or Python 2.7.9
-  if [[ "$TRAVIS_PYTHON_VERSION" != nightly ]] && [[ "$TRAVIS_PYTHON_VERSION" != "2.7.9" ]]; then
-      as_jenkins pip install scipy==1.1.0 scikit-image librosa>=0.6.2
-  fi
-
-  # Install psutil for dataloader tests
-  as_jenkins pip install psutil
-
-  # Install dill for serialization tests
-  as_jenkins pip install "dill>=0.3.1"
-
-  # Cleanup package manager
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-fi

.circleci/docker/common/install_user.sh

@@ -1,20 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# Mirror jenkins user in container
-echo "jenkins:x:1014:1014::/var/lib/jenkins:" >> /etc/passwd
-echo "jenkins:x:1014:" >> /etc/group
-
-# Create $HOME
-mkdir -p /var/lib/jenkins
-chown jenkins:jenkins /var/lib/jenkins
-mkdir -p /var/lib/jenkins/.ccache
-chown jenkins:jenkins /var/lib/jenkins/.ccache
-
-# Allow writing to /usr/local (for make install)
-chown jenkins:jenkins /usr/local
-
-# Allow sudo
-# TODO: Maybe we shouldn't
-echo 'jenkins ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/jenkins

.circleci/docker/common/install_vision.sh

@@ -1,57 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-# This function installs protobuf 2.6
-install_protobuf_26() {
-  pb_dir="/usr/temp_pb_install_dir"
-  mkdir -p $pb_dir
-
-  # On the nvidia/cuda:9-cudnn7-devel-centos7 image we need this symlink or
-  # else it will fail with
-  #   g++: error: ./../lib64/crti.o: No such file or directory
-  ln -s /usr/lib64 "$pb_dir/lib64"
-
-  curl -LO "https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz"
-  tar -xvz -C "$pb_dir" --strip-components 1 -f protobuf-2.6.1.tar.gz
-  pushd "$pb_dir" && ./configure && make && make check && sudo make install && sudo ldconfig
-  popd
-  rm -rf $pb_dir
-}
-
-install_ubuntu() {
-  apt-get update
-  apt-get install -y --no-install-recommends \
-          libopencv-dev \
-          libavcodec-dev
-
-  # Cleanup
-  apt-get autoclean && apt-get clean
-  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-}
-
-install_centos() {
-  # Need EPEL for many packages we depend on.
-  # See http://fedoraproject.org/wiki/EPEL
-  yum --enablerepo=extras install -y epel-release
-
-  yum install -y \
-      opencv-devel \
-      ffmpeg-devel
-
-  # Cleanup
-  yum clean all
-  rm -rf /var/cache/yum
-  rm -rf /var/lib/yum/yumdb
-  rm -rf /var/lib/yum/history
-}
-
-# Install base packages depending on the base OS
-if [ -f /etc/lsb-release ]; then
-  install_ubuntu
-elif [ -f /etc/os-release ]; then
-  install_centos
-else
-  echo "Unable to determine OS..."
-  exit 1
-fi

.circleci/docker/ubuntu-cuda/Dockerfile

@@ -1,94 +0,0 @@
-ARG UBUNTU_VERSION
-ARG CUDA_VERSION
-ARG CUDNN_VERSION
-
-FROM nvidia/cuda:${CUDA_VERSION}-cudnn${CUDNN_VERSION}-devel-ubuntu${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-ARG CUDA_VERSION
-ARG CUDNN_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install katex
-ARG KATEX
-ADD ./common/install_katex.sh install_katex.sh
-RUN bash ./install_katex.sh && rm install_katex.sh
-
-# Install conda
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# Install gcc
-ARG GCC_VERSION
-ADD ./common/install_gcc.sh install_gcc.sh
-RUN bash ./install_gcc.sh && rm install_gcc.sh
-
-# Install clang
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# Install non-standard Python versions (via Travis binaries)
-ARG TRAVIS_PYTHON_VERSION
-ENV PATH /opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH
-ADD ./common/install_travis_python.sh install_travis_python.sh
-RUN bash ./install_travis_python.sh && rm install_travis_python.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-ENV CUDA_NVCC_EXECUTABLE=/opt/cache/lib/nvcc
-
-# Add jni.h for java host build
-ADD ./common/install_jni.sh install_jni.sh
-ADD ./java/jni.h jni.h
-RUN bash ./install_jni.sh && rm install_jni.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-# AWS specific CUDA build guidance
-ENV TORCH_CUDA_ARCH_LIST Maxwell
-ENV TORCH_NVCC_FLAGS "-Xfatbin -compress-all"
-
-# Install LLVM dev version
-ADD ./common/install_llvm.sh install_llvm.sh
-RUN bash ./install_llvm.sh
-
-USER jenkins
-CMD ["bash"]

.circleci/docker/ubuntu-rocm/.gitignore

@@ -1 +0,0 @@
-*.sh

.circleci/docker/ubuntu-rocm/Dockerfile

@@ -1,86 +0,0 @@
-ARG UBUNTU_VERSION
-
-FROM ubuntu:${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install clang
-ARG LLVMDEV
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install conda
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# Install rocm
-ARG ROCM_VERSION
-ADD ./common/install_rocm.sh install_rocm.sh
-RUN bash ./install_rocm.sh
-RUN rm install_rocm.sh
-ENV PATH /opt/rocm/bin:$PATH
-ENV PATH /opt/rocm/hcc/bin:$PATH
-ENV PATH /opt/rocm/hip/bin:$PATH
-ENV PATH /opt/rocm/opencl/bin:$PATH
-ENV HIP_PLATFORM hcc
-ENV LANG C.UTF-8
-ENV LC_ALL C.UTF-8
-
-# (optional) Install non-default CMake version
-ARG CMAKE_VERSION
-ADD ./common/install_cmake.sh install_cmake.sh
-RUN if [ -n "${CMAKE_VERSION}" ]; then bash ./install_cmake.sh; fi
-RUN rm install_cmake.sh
-
-# (optional) Install non-default Ninja version
-ARG NINJA_VERSION
-ADD ./common/install_ninja.sh install_ninja.sh
-RUN if [ -n "${NINJA_VERSION}" ]; then bash ./install_ninja.sh; fi
-RUN rm install_ninja.sh
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-USER jenkins
-CMD ["bash"]

.circleci/docker/ubuntu/Dockerfile

@@ -1,119 +0,0 @@
-ARG UBUNTU_VERSION
-
-FROM ubuntu:${UBUNTU_VERSION}
-
-ARG UBUNTU_VERSION
-
-ENV DEBIAN_FRONTEND noninteractive
-
-# Install common dependencies (so that this step can be cached separately)
-ARG EC2
-ADD ./common/install_base.sh install_base.sh
-RUN bash ./install_base.sh && rm install_base.sh
-
-# Install clang
-ARG LLVMDEV
-ARG CLANG_VERSION
-ADD ./common/install_clang.sh install_clang.sh
-RUN bash ./install_clang.sh && rm install_clang.sh
-
-# (optional) Install thrift.
-ARG THRIFT
-ADD ./common/install_thrift.sh install_thrift.sh
-RUN if [ -n "${THRIFT}" ]; then bash ./install_thrift.sh; fi
-RUN rm install_thrift.sh
-ENV INSTALLED_THRIFT ${THRIFT}
-
-# Install user
-ADD ./common/install_user.sh install_user.sh
-RUN bash ./install_user.sh && rm install_user.sh
-
-# Install katex
-ARG KATEX
-ADD ./common/install_katex.sh install_katex.sh
-RUN bash ./install_katex.sh && rm install_katex.sh
-
-# Install conda
-ENV PATH /opt/conda/bin:$PATH
-ARG ANACONDA_PYTHON_VERSION
-ADD ./common/install_conda.sh install_conda.sh
-RUN bash ./install_conda.sh && rm install_conda.sh
-
-# Install gcc
-ARG GCC_VERSION
-ADD ./common/install_gcc.sh install_gcc.sh
-RUN bash ./install_gcc.sh && rm install_gcc.sh
-
-# Install non-standard Python versions (via Travis binaries)
-ARG TRAVIS_PYTHON_VERSION
-ARG TRAVIS_DL_URL_PREFIX
-ENV PATH /opt/python/$TRAVIS_PYTHON_VERSION/bin:$PATH
-ADD ./common/install_travis_python.sh install_travis_python.sh
-RUN bash ./install_travis_python.sh && rm install_travis_python.sh
-
-# (optional) Install protobuf for ONNX
-ARG PROTOBUF
-ADD ./common/install_protobuf.sh install_protobuf.sh
-RUN if [ -n "${PROTOBUF}" ]; then bash ./install_protobuf.sh; fi
-RUN rm install_protobuf.sh
-ENV INSTALLED_PROTOBUF ${PROTOBUF}
-
-# (optional) Install database packages like LMDB and LevelDB
-ARG DB
-ADD ./common/install_db.sh install_db.sh
-RUN if [ -n "${DB}" ]; then bash ./install_db.sh; fi
-RUN rm install_db.sh
-ENV INSTALLED_DB ${DB}
-
-# (optional) Install vision packages like OpenCV and ffmpeg
-ARG VISION
-ADD ./common/install_vision.sh install_vision.sh
-RUN if [ -n "${VISION}" ]; then bash ./install_vision.sh; fi
-RUN rm install_vision.sh
-ENV INSTALLED_VISION ${VISION}
-
-# (optional) Install Android NDK
-ARG ANDROID
-ARG ANDROID_NDK
-ARG GRADLE_VERSION
-ADD ./common/install_android.sh install_android.sh
-ADD ./android/AndroidManifest.xml AndroidManifest.xml
-ADD ./android/build.gradle build.gradle
-RUN if [ -n "${ANDROID}" ]; then bash ./install_android.sh; fi
-RUN rm install_android.sh
-RUN rm AndroidManifest.xml
-RUN rm build.gradle
-ENV INSTALLED_ANDROID ${ANDROID}
-
-# (optional) Install non-default CMake version
-ARG CMAKE_VERSION
-ADD ./common/install_cmake.sh install_cmake.sh
-RUN if [ -n "${CMAKE_VERSION}" ]; then bash ./install_cmake.sh; fi
-RUN rm install_cmake.sh
-
-# (optional) Install non-default Ninja version
-ARG NINJA_VERSION
-ADD ./common/install_ninja.sh install_ninja.sh
-RUN if [ -n "${NINJA_VERSION}" ]; then bash ./install_ninja.sh; fi
-RUN rm install_ninja.sh
-
-# Install ccache/sccache (do this last, so we get priority in PATH)
-ADD ./common/install_cache.sh install_cache.sh
-ENV PATH /opt/cache/bin:$PATH
-RUN bash ./install_cache.sh && rm install_cache.sh
-
-# Add jni.h for java host build
-ADD ./common/install_jni.sh install_jni.sh
-ADD ./java/jni.h jni.h
-RUN bash ./install_jni.sh && rm install_jni.sh
-
-# Include BUILD_ENVIRONMENT environment variable in image
-ARG BUILD_ENVIRONMENT
-ENV BUILD_ENVIRONMENT ${BUILD_ENVIRONMENT}
-
-# Install LLVM dev version
-ADD ./common/install_llvm.sh install_llvm.sh
-RUN bash ./install_llvm.sh
-
-USER jenkins
-CMD ["bash"]

.circleci/ecr_gc_docker/Dockerfile

@@ -1,13 +0,0 @@
-FROM ubuntu:16.04
-
-RUN apt-get update && apt-get install -y python-pip git && rm -rf /var/lib/apt/lists/* /var/log/dpkg.log
-
-ADD requirements.txt /requirements.txt
-
-RUN pip install -r /requirements.txt
-
-ADD gc.py /usr/bin/gc.py
-
-ADD docker_hub.py /usr/bin/docker_hub.py
-
-ENTRYPOINT ["/usr/bin/gc.py"]

.circleci/ecr_gc_docker/docker_hub.py

@@ -1,125 +0,0 @@
-#!/usr/bin/env python
-
-from collections import namedtuple
-
-import boto3
-import requests
-import os
-
-
-IMAGE_INFO = namedtuple(
-    "IMAGE_INFO", ("repo", "tag", "size", "last_updated_at", "last_updated_by")
-)
-
-
-def build_access_token(username, passwordtr):
-    r = requests.post(
-        "https://hub.docker.com/v2/users/login/",
-        data={"username": username, "password": password},
-    )
-    r.raise_for_status()
-    token = r.json().get("token")
-    return {"Authorization": "JWT " + token}
-
-
-def list_repos(user, token):
-    r = requests.get("https://hub.docker.com/v2/repositories/" + user, headers=token)
-    r.raise_for_status()
-    ret = sorted(
-        repo["user"] + "/" + repo["name"] for repo in r.json().get("results", [])
-    )
-    if ret:
-        print("repos found:")
-        print("".join("\n\t" + r for r in ret))
-    return ret
-
-
-def list_tags(repo, token):
-    r = requests.get(
-        "https://hub.docker.com/v2/repositories/" + repo + "/tags", headers=token
-    )
-    r.raise_for_status()
-    return [
-        IMAGE_INFO(
-            repo=repo,
-            tag=t["name"],
-            size=t["full_size"],
-            last_updated_at=t["last_updated"],
-            last_updated_by=t["last_updater_username"],
-        )
-        for t in r.json().get("results", [])
-    ]
-
-
-def save_to_s3(tags):
-    table_content = ""
-    client = boto3.client("s3")
-    for t in tags:
-        table_content += (
-            "<tr><td>{repo}</td><td>{tag}</td><td>{size}</td>"
-            "<td>{last_updated_at}</td><td>{last_updated_by}</td></tr>"
-        ).format(
-            repo=t.repo,
-            tag=t.tag,
-            size=t.size,
-            last_updated_at=t.last_updated_at,
-            last_updated_by=t.last_updated_by,
-        )
-    html_body = """
-    <html>
-        <head>
-            <link rel="stylesheet"
-                href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
-                integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh"
-                crossorigin="anonymous">
-            <link rel="stylesheet" type="text/css"
-                href="https://cdn.datatables.net/1.10.20/css/jquery.dataTables.css">
-            <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js">
-            </script>
-            <script type="text/javascript" charset="utf8"
-                src="https://cdn.datatables.net/1.10.20/js/jquery.dataTables.js"></script>
-            <title> docker image info</title>
-        </head>
-        <body>
-            <table class="table table-striped table-hover" id="docker">
-            <caption>Docker images on docker hub</caption>
-            <thead class="thead-dark">
-                <tr>
-                <th scope="col">repo</th>
-                <th scope="col">tag</th>
-                <th scope="col">size</th>
-                <th scope="col">last_updated_at</th>
-                <th scope="col">last_updated_by</th>
-                </tr>
-            </thead>
-            <tbody>
-                {table_content}
-            </tbody>
-            </table>
-        </body>
-        <script>
-            $(document).ready( function () {{
-                $('#docker').DataTable({{paging: false}});
-            }} );py
-        </script>
-    </html>
-    """.format(
-        table_content=table_content
-    )
-    client.put_object(
-        Bucket="docker.pytorch.org",
-        ACL="public-read",
-        Key="docker_hub.html",
-        Body=html_body,
-        ContentType="text/html",
-    )
-
-
-if __name__ == "__main__":
-    username = os.environ.get("DOCKER_HUB_USERNAME")
-    password = os.environ.get("DOCKER_HUB_PASSWORD")
-    token = build_access_token(username, password)
-    tags = []
-    for repo in list_repos("pytorch", token):
-        tags.extend(list_tags(repo, token))
-    save_to_s3(tags)

.circleci/ecr_gc_docker/gc.py

@@ -1,214 +0,0 @@
-#!/usr/bin/env python
-
-import argparse
-import datetime
-import boto3
-import pytz
-import sys
-import re
-
-
-def save_to_s3(project, data):
-    table_content = ""
-    client = boto3.client("s3")
-    for repo, tag, window, age, pushed in data:
-        table_content += "<tr><td>{repo}</td><td>{tag}</td><td>{window}</td><td>{age}</td><td>{pushed}</td></tr>".format(
-            repo=repo, tag=tag, window=window, age=age, pushed=pushed
-        )
-    html_body = """
-    <html>
-        <head>
-            <link rel="stylesheet"
-                href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
-                integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh"
-                crossorigin="anonymous">
-            <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.20/css/jquery.dataTables.css">
-            <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
-            <script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.20/js/jquery.dataTables.js"></script>
-            <title>{project} nightly and permanent docker image info</title>
-        </head>
-        <body>
-            <table class="table table-striped table-hover" id="docker">
-            <thead class="thead-dark">
-                <tr>
-                <th scope="col">repo</th>
-                <th scope="col">tag</th>
-                <th scope="col">keep window</th>
-                <th scope="col">age</th>
-                <th scope="col">pushed at</th>
-                </tr>
-            </thead>
-            <tbody>
-                {table_content}
-            </tbody>
-            </table>
-        </body>
-        <script>
-            $(document).ready( function () {{
-                $('#docker').DataTable({{paging: false}});
-            }} );
-        </script>
-    </html>
-    """.format(
-        project=project, table_content=table_content
-    )
-
-    # for pytorch, file can be found at
-    # http://ossci-docker.s3-website.us-east-1.amazonaws.com/pytorch.html
-    # and later one we can config docker.pytorch.org to point to the location
-
-    client.put_object(
-        Bucket="docker.pytorch.org",
-        ACL="public-read",
-        Key="{project}.html".format(project=project),
-        Body=html_body,
-        ContentType="text/html",
-    )
-
-
-def repos(client):
-    paginator = client.get_paginator("describe_repositories")
-    pages = paginator.paginate(registryId="308535385114")
-    for page in pages:
-        for repo in page["repositories"]:
-            yield repo
-
-
-def images(client, repository):
-    paginator = client.get_paginator("describe_images")
-    pages = paginator.paginate(
-        registryId="308535385114", repositoryName=repository["repositoryName"]
-    )
-    for page in pages:
-        for image in page["imageDetails"]:
-            yield image
-
-
-parser = argparse.ArgumentParser(description="Delete old Docker tags from registry")
-parser.add_argument(
-    "--dry-run", action="store_true", help="Dry run; print tags that would be deleted"
-)
-parser.add_argument(
-    "--keep-stable-days",
-    type=int,
-    default=14,
-    help="Days of stable Docker tags to keep (non per-build images)",
-)
-parser.add_argument(
-    "--keep-unstable-days",
-    type=int,
-    default=1,
-    help="Days of unstable Docker tags to keep (per-build images)",
-)
-parser.add_argument(
-    "--filter-prefix",
-    type=str,
-    default="",
-    help="Only run cleanup for repositories with this prefix",
-)
-parser.add_argument(
-    "--ignore-tags",
-    type=str,
-    default="",
-    help="Never cleanup these tags (comma separated)",
-)
-args = parser.parse_args()
-
-if not args.ignore_tags or not args.filter_prefix:
-    print(
-        """
-Missing required arguments --ignore-tags and --filter-prefix
-
-You must specify --ignore-tags and --filter-prefix to avoid accidentally
-pruning a stable Docker tag which is being actively used.  This will
-make you VERY SAD.  So pay attention.
-
-First, which filter-prefix do you want?  The list of valid prefixes
-is in jobs/private.groovy under the 'docker-registry-cleanup' job.
-You probably want either pytorch or caffe2.
-
-Second, which ignore-tags do you want?  It should be whatever the most
-up-to-date DockerVersion for the repository in question is.  Follow
-the imports of jobs/pytorch.groovy to find them.
-"""
-    )
-    sys.exit(1)
-
-client = boto3.client("ecr", region_name="us-east-1")
-stable_window = datetime.timedelta(days=args.keep_stable_days)
-unstable_window = datetime.timedelta(days=args.keep_unstable_days)
-now = datetime.datetime.now(pytz.UTC)
-ignore_tags = args.ignore_tags.split(",")
-
-
-def chunks(chunkable, n):
-    """ Yield successive n-sized chunks from l.
-    """
-    for i in range(0, len(chunkable), n):
-        yield chunkable[i : i + n]
-
-SHA_PATTERN = re.compile(r'^[0-9a-f]{40}$')
-def looks_like_git_sha(tag):
-    """Returns a boolean to check if a tag looks like a git sha
-
-    For reference a sha1 is 40 characters with only 0-9a-f and contains no
-    "-" characters
-    """
-    return re.match(SHA_PATTERN, tag) is not None
-
-stable_window_tags = []
-for repo in repos(client):
-    repositoryName = repo["repositoryName"]
-    if not repositoryName.startswith(args.filter_prefix):
-        continue
-
-    # Keep list of image digests to delete for this repository
-    digest_to_delete = []
-    print(repositoryName)
-
-    for image in images(client, repo):
-        tags = image.get("imageTags")
-        if not isinstance(tags, (list,)) or len(tags) == 0:
-            continue
-
-        tag = tags[0]
-        created = image["imagePushedAt"]
-        age = now - created
-        if any([
-                looks_like_git_sha(tag),
-                tag.isdigit(),
-                tag.count("-") == 4,  # TODO: Remove, this no longer applies as tags are now built using a SHA1
-                tag in ignore_tags]):
-            window = stable_window
-            if tag in ignore_tags:
-                stable_window_tags.append((repositoryName, tag, "", age, created))
-            elif age < window:
-                stable_window_tags.append((repositoryName, tag, window, age, created))
-        else:
-            window = unstable_window
-
-        if tag in ignore_tags:
-            print("Ignoring tag {}:{} (age: {})".format(repositoryName, tag, age))
-            continue
-        if age < window:
-            print("Not deleting manifest for tag {}:{} (age: {})".format(repositoryName, tag, age))
-            continue
-
-        if args.dry_run:
-            print("(dry run) Deleting manifest for tag {}:{} (age: {})".format(repositoryName, tag, age))
-        else:
-            print("Deleting manifest for tag{}:{} (age: {})".format(repositoryName, tag, age))
-            digest_to_delete.append(image["imageDigest"])
-
-    # Issue batch delete for all images to delete for this repository
-    # Note that as of 2018-07-25, the maximum number of images you can
-    # delete in a single batch is 100, so chunk our list into batches of
-    # 100
-    for c in chunks(digest_to_delete, 100):
-        client.batch_delete_image(
-            registryId="308535385114",
-            repositoryName=repositoryName,
-            imageIds=[{"imageDigest": digest} for digest in c],
-        )
-
-    save_to_s3(args.filter_prefix, stable_window_tags)

.circleci/ecr_gc_docker/requirements.txt

@@ -1,3 +0,0 @@
-boto3
-pytz
-requests

.circleci/ensure-consistency.py

@@ -1,39 +0,0 @@
-#!/usr/bin/env python3
-
-import os
-import subprocess
-import sys
-import tempfile
-
-import generate_config_yml
-
-
-CHECKED_IN_FILE = "config.yml"
-REGENERATION_SCRIPT = "regenerate.sh"
-
-PARENT_DIR = os.path.basename(os.path.dirname(os.path.abspath(__file__)))
-README_PATH = os.path.join(PARENT_DIR, "README.md")
-
-ERROR_MESSAGE_TEMPLATE = """
-The checked-in CircleCI "%s" file does not match what was generated by the scripts.
-Please re-run the "%s" script in the "%s" directory and commit the result. See "%s" for more information.
-"""
-
-
-def check_consistency():
-
-    _, temp_filename = tempfile.mkstemp("-generated-config.yml")
-
-    with open(temp_filename, "w") as fh:
-        generate_config_yml.stitch_sources(fh)
-
-    try:
-        subprocess.check_call(["cmp", temp_filename, CHECKED_IN_FILE])
-    except subprocess.CalledProcessError:
-        sys.exit(ERROR_MESSAGE_TEMPLATE % (CHECKED_IN_FILE, REGENERATION_SCRIPT, PARENT_DIR, README_PATH))
-    finally:
-        os.remove(temp_filename)
-
-
-if __name__ == "__main__":
-    check_consistency()

.circleci/generate_config_yml.py

@@ -1,169 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-This script is the source of truth for config.yml.
-Please see README.md in this directory for details.
-"""
-
-import os
-import shutil
-import sys
-from collections import OrderedDict, namedtuple
-
-import cimodel.data.binary_build_definitions as binary_build_definitions
-import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
-import cimodel.data.pytorch_build_definitions as pytorch_build_definitions
-import cimodel.data.simple.android_definitions
-import cimodel.data.simple.bazel_definitions
-import cimodel.data.simple.binary_smoketest
-import cimodel.data.simple.docker_definitions
-import cimodel.data.simple.ge_config_tests
-import cimodel.data.simple.ios_definitions
-import cimodel.data.simple.macos_definitions
-import cimodel.data.simple.mobile_definitions
-import cimodel.data.simple.nightly_android
-import cimodel.data.simple.nightly_ios
-import cimodel.data.windows_build_definitions as windows_build_definitions
-import cimodel.lib.miniutils as miniutils
-import cimodel.lib.miniyaml as miniyaml
-
-
-class File(object):
-    """
-    Verbatim copy the contents of a file into config.yml
-    """
-
-    def __init__(self, filename):
-        self.filename = filename
-
-    def write(self, output_filehandle):
-        with open(os.path.join("verbatim-sources", self.filename)) as fh:
-            shutil.copyfileobj(fh, output_filehandle)
-
-
-class FunctionGen(namedtuple("FunctionGen", "function depth")):
-    __slots__ = ()
-
-
-class Treegen(FunctionGen):
-    """
-    Insert the content of a YAML tree into config.yml
-    """
-
-    def write(self, output_filehandle):
-        miniyaml.render(output_filehandle, self.function(), self.depth)
-
-
-class Listgen(FunctionGen):
-    """
-    Insert the content of a YAML list into config.yml
-    """
-
-    def write(self, output_filehandle):
-        miniyaml.render(output_filehandle, self.function(), self.depth)
-
-
-def horizontal_rule():
-    return "".join("#" * 78)
-
-
-class Header(object):
-    def __init__(self, title, summary=None):
-        self.title = title
-        self.summary_lines = summary or []
-
-    def write(self, output_filehandle):
-        text_lines = [self.title] + self.summary_lines
-        comment_lines = ["# " + x for x in text_lines]
-        lines = miniutils.sandwich([horizontal_rule()], comment_lines)
-
-        for line in filter(None, lines):
-            output_filehandle.write(line + "\n")
-
-
-def gen_build_workflows_tree():
-    build_workflows_functions = [
-        pytorch_build_definitions.get_workflow_jobs,
-        cimodel.data.simple.macos_definitions.get_workflow_jobs,
-        cimodel.data.simple.android_definitions.get_workflow_jobs,
-        cimodel.data.simple.ios_definitions.get_workflow_jobs,
-        cimodel.data.simple.mobile_definitions.get_workflow_jobs,
-        cimodel.data.simple.ge_config_tests.get_workflow_jobs,
-        cimodel.data.simple.bazel_definitions.get_workflow_jobs,
-        caffe2_build_definitions.get_workflow_jobs,
-        cimodel.data.simple.binary_smoketest.get_workflow_jobs,
-        cimodel.data.simple.nightly_ios.get_workflow_jobs,
-        cimodel.data.simple.nightly_android.get_workflow_jobs,
-        windows_build_definitions.get_windows_workflows,
-    ]
-
-    binary_build_functions = [
-        binary_build_definitions.get_binary_build_jobs,
-        binary_build_definitions.get_nightly_tests,
-        binary_build_definitions.get_nightly_uploads,
-        binary_build_definitions.get_post_upload_jobs,
-        binary_build_definitions.get_binary_smoke_test_jobs,
-    ]
-
-    docker_builder_functions = [
-        cimodel.data.simple.docker_definitions.get_workflow_jobs
-    ]
-
-    return {
-        "workflows": {
-            "binary_builds": {
-                "when": r"<< pipeline.parameters.run_binary_tests >>",
-                "jobs": [f() for f in binary_build_functions],
-            },
-            "docker_build": OrderedDict(
-                {
-                    "triggers": [
-                        {
-                            "schedule": {
-                                "cron": miniutils.quote("0 15 * * 0"),
-                                "filters": {"branches": {"only": ["master"]}},
-                            }
-                        }
-                    ],
-                    "jobs": [f() for f in docker_builder_functions],
-                }
-            ),
-            "build": {"jobs": [f() for f in build_workflows_functions]},
-        }
-    }
-
-
-# Order of this list matters to the generated config.yml.
-YAML_SOURCES = [
-    File("header-section.yml"),
-    File("commands.yml"),
-    File("nightly-binary-build-defaults.yml"),
-    Header("Build parameters"),
-    File("build-parameters/pytorch-build-params.yml"),
-    File("build-parameters/caffe2-build-params.yml"),
-    File("build-parameters/binary-build-params.yml"),
-    File("build-parameters/promote-build-params.yml"),
-    Header("Job specs"),
-    File("job-specs/pytorch-job-specs.yml"),
-    File("job-specs/caffe2-job-specs.yml"),
-    File("job-specs/binary-job-specs.yml"),
-    File("job-specs/job-specs-custom.yml"),
-    File("job-specs/job-specs-promote.yml"),
-    File("job-specs/binary_update_htmls.yml"),
-    File("job-specs/binary-build-tests.yml"),
-    File("job-specs/docker_jobs.yml"),
-    Header("Workflows"),
-    Treegen(gen_build_workflows_tree, 0),
-    File("workflows/workflows-ecr-gc.yml"),
-    File("workflows/workflows-promote.yml"),
-]
-
-
-def stitch_sources(output_filehandle):
-    for f in YAML_SOURCES:
-        f.write(output_filehandle)
-
-
-if __name__ == "__main__":
-
-    stitch_sources(sys.stdout)

.circleci/regenerate.sh

@@ -1,8 +0,0 @@
-#!/bin/bash -xe
-
-# Allows this script to be invoked from any directory:
-cd $(dirname "$0")
-
-NEW_FILE=$(mktemp)
-./generate_config_yml.py > $NEW_FILE
-cp $NEW_FILE config.yml

.circleci/scripts/README.md

@@ -1,4 +0,0 @@
-All the scripts in this directory are callable from `~/workspace/.circleci/scripts/foo.sh`.
-Don't try to call them as `.circleci/scripts/foo.sh`, that won't
-(necessarily) work.  See Note [Workspace for CircleCI scripts] in
-job-specs-setup.yml for more details.

.circleci/scripts/binary_checkout.sh

@@ -1,63 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-retry () {
-    $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-}
-
-
-# This step runs on multiple executors with different envfile locations
-if [[ "$(uname)" == Darwin ]]; then
-  # macos executor (builds and tests)
-  workdir="/Users/distiller/project"
-elif [[ "$OSTYPE" == "msys" ]]; then
-  # windows executor (builds and tests)
-  rm -rf /c/w
-  ln -s "/c/Users/circleci/project" /c/w
-  workdir="/c/w"
-elif [[ -d "/home/circleci/project" ]]; then
-  # machine executor (binary tests)
-  workdir="/home/circleci/project"
-else
-  # docker executor (binary builds)
-  workdir="/"
-fi
-
-# It is very important that this stays in sync with binary_populate_env.sh
-if [[ "$OSTYPE" == "msys" ]]; then
-  # We need to make the paths as short as possible on Windows
-  export PYTORCH_ROOT="$workdir/p"
-  export BUILDER_ROOT="$workdir/b"
-else
-  export PYTORCH_ROOT="$workdir/pytorch"
-  export BUILDER_ROOT="$workdir/builder"
-fi
-
-# Clone the Pytorch branch
-retry git clone https://github.com/pytorch/pytorch.git "$PYTORCH_ROOT"
-pushd "$PYTORCH_ROOT"
-if [[ -n "${CIRCLE_PR_NUMBER:-}" ]]; then
-  # "smoke" binary build on PRs
-  git fetch --force origin "pull/${CIRCLE_PR_NUMBER}/head:remotes/origin/pull/${CIRCLE_PR_NUMBER}"
-  git reset --hard "$CIRCLE_SHA1"
-  git checkout -q -B "$CIRCLE_BRANCH"
-  git reset --hard "$CIRCLE_SHA1"
-elif [[ -n "${CIRCLE_SHA1:-}" ]]; then
-  # Scheduled workflows & "smoke" binary build on master on PR merges
-  git reset --hard "$CIRCLE_SHA1"
-  git checkout -q -B master
-else
-  echo "Can't tell what to checkout"
-  exit 1
-fi
-retry git submodule update --init --recursive
-echo "Using Pytorch from "
-git --no-pager log --max-count 1
-popd
-
-# Clone the Builder master repo
-retry git clone -q https://github.com/pytorch/builder.git "$BUILDER_ROOT"
-pushd "$BUILDER_ROOT"
-echo "Using builder from "
-git --no-pager log --max-count 1
-popd

.circleci/scripts/binary_install_miniconda.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-
-set -eux -o pipefail
-
-# This step runs on multiple executors with different envfile locations
-if [[ "$(uname)" == Darwin ]]; then
-  envfile="/Users/distiller/project/env"
-elif [[ -d "/home/circleci/project" ]]; then
-  # machine executor (binary tests)
-  envfile="/home/circleci/project/env"
-else
-  # docker executor (binary builds)
-  envfile="/env"
-fi
-
-# TODO this is super hacky and ugly. Basically, the binary_update_html job does
-# not have an env file, since it does not call binary_populate_env.sh, since it
-# does not have a BUILD_ENVIRONMENT. So for this one case, which we detect by a
-# lack of an env file, we manually export the environment variables that we
-# need to install miniconda
-if [[ ! -f "$envfile" ]]; then
-  MINICONDA_ROOT="/home/circleci/project/miniconda"
-  workdir="/home/circleci/project"
-  retry () {
-      $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
-  }
-  export -f retry
-else
-  source "$envfile"
-fi
-
-conda_sh="$workdir/install_miniconda.sh"
-if [[ "$(uname)" == Darwin ]]; then
-  curl --retry 3 -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
-else
-  curl --retry 3 -o "$conda_sh" https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-fi
-chmod +x "$conda_sh"
-"$conda_sh" -b -p "$MINICONDA_ROOT"
-rm -f "$conda_sh"
-
-# We can't actually add miniconda to the PATH in the envfile, because that
-# breaks 'unbuffer' in Mac jobs. This is probably because conda comes with
-# a tclsh, which then gets inserted before the tclsh needed in /usr/bin

.circleci/scripts/binary_ios_build.sh

@@ -1,42 +0,0 @@
-#!/bin/bash
-set -ex -o pipefail
-
-echo ""
-echo "DIR: $(pwd)"
-WORKSPACE=/Users/distiller/workspace
-PROJ_ROOT=/Users/distiller/project
-export TCLLIBPATH="/usr/local/lib"
-
-# Install conda
-curl --retry 3 -o ~/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
-chmod +x ~/conda.sh
-/bin/bash ~/conda.sh -b -p ~/anaconda
-export PATH="~/anaconda/bin:${PATH}"
-source ~/anaconda/bin/activate
-
-# Install dependencies
-conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing requests --yes
-export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
-
-# sync submodules
-cd ${PROJ_ROOT}
-git submodule sync
-git submodule update --init --recursive
-
-# run build script
-chmod a+x ${PROJ_ROOT}/scripts/build_ios.sh
-echo "########################################################"
-cat ${PROJ_ROOT}/scripts/build_ios.sh
-echo "########################################################"
-echo "IOS_ARCH: ${IOS_ARCH}"
-echo "IOS_PLATFORM: ${IOS_PLATFORM}"
-export IOS_ARCH=${IOS_ARCH}
-export IOS_PLATFORM=${IOS_PLATFORM}
-unbuffer ${PROJ_ROOT}/scripts/build_ios.sh 2>&1 | ts
-
-#store the binary
-cd ${WORKSPACE}
-DEST_DIR=${WORKSPACE}/ios
-mkdir -p ${DEST_DIR}
-cp -R ${PROJ_ROOT}/build_ios/install ${DEST_DIR}
-mv ${DEST_DIR}/install ${DEST_DIR}/${IOS_ARCH}

.circleci/scripts/binary_ios_test.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-set -ex -o pipefail
-
-echo ""
-echo "DIR: $(pwd)"
-PROJ_ROOT=/Users/distiller/project
-cd ${PROJ_ROOT}/ios/TestApp
-# install fastlane
-sudo gem install bundler && bundle install
-# install certificates
-echo "${IOS_CERT_KEY}" >> cert.txt
-base64 --decode cert.txt -o Certificates.p12
-rm cert.txt
-bundle exec fastlane install_cert
-# install the provisioning profile
-PROFILE=TestApp_CI.mobileprovision
-PROVISIONING_PROFILES=~/Library/MobileDevice/Provisioning\ Profiles
-mkdir -pv "${PROVISIONING_PROFILES}"
-cd "${PROVISIONING_PROFILES}"
-echo "${IOS_SIGN_KEY}" >> cert.txt
-base64 --decode cert.txt -o ${PROFILE}
-rm cert.txt
-# run the ruby build script
-if ! [ -x "$(command -v xcodebuild)" ]; then
-    echo 'Error: xcodebuild is not installed.'
-    exit 1
-fi 
-PROFILE=TestApp_CI
-ruby ${PROJ_ROOT}/scripts/xcode_build.rb -i ${PROJ_ROOT}/build_ios/install -x ${PROJ_ROOT}/ios/TestApp/TestApp.xcodeproj -p ${IOS_PLATFORM} -c ${PROFILE} -t ${IOS_DEV_TEAM_ID}

.circleci/scripts/binary_ios_upload.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-set -ex -o pipefail
-
-echo ""
-echo "DIR: $(pwd)"
-WORKSPACE=/Users/distiller/workspace
-PROJ_ROOT=/Users/distiller/project
-ARTIFACTS_DIR=${WORKSPACE}/ios
-ls ${ARTIFACTS_DIR}
-ZIP_DIR=${WORKSPACE}/zip
-mkdir -p ${ZIP_DIR}/install/lib
-mkdir -p ${ZIP_DIR}/src
-# copy header files
-cp -R ${ARTIFACTS_DIR}/arm64/include ${ZIP_DIR}/install/
-# build a FAT bianry
-cd ${ZIP_DIR}/install/lib
-target_libs=(libc10.a libclog.a libcpuinfo.a libeigen_blas.a libpthreadpool.a libpytorch_qnnpack.a libtorch_cpu.a libtorch.a libXNNPACK.a)
-for lib in ${target_libs[*]}
-do
-    if [ -f "${ARTIFACTS_DIR}/x86_64/lib/${lib}" ] && [ -f "${ARTIFACTS_DIR}/arm64/lib/${lib}" ]; then
-        libs=("${ARTIFACTS_DIR}/x86_64/lib/${lib}" "${ARTIFACTS_DIR}/arm64/lib/${lib}")
-        lipo -create "${libs[@]}" -o ${ZIP_DIR}/install/lib/${lib}
-    fi
-done
-lipo -i ${ZIP_DIR}/install/lib/*.a
-# copy the umbrella header and license
-cp ${PROJ_ROOT}/ios/LibTorch.h ${ZIP_DIR}/src/
-cp ${PROJ_ROOT}/LICENSE ${ZIP_DIR}/
-# zip the library
-ZIPFILE=libtorch_ios_nightly_build.zip
-cd ${ZIP_DIR}
-#for testing
-touch version.txt
-echo $(date +%s) > version.txt
-zip -r ${ZIPFILE} install src version.txt LICENSE
-# upload to aws
-brew install awscli
-set +x
-export AWS_ACCESS_KEY_ID=${AWS_S3_ACCESS_KEY_FOR_PYTORCH_BINARY_UPLOAD}
-export AWS_SECRET_ACCESS_KEY=${AWS_S3_ACCESS_SECRET_FOR_PYTORCH_BINARY_UPLOAD}
-set +x
-# echo "AWS KEY: ${AWS_ACCESS_KEY_ID}"
-# echo "AWS SECRET: ${AWS_SECRET_ACCESS_KEY}"
-aws s3 cp ${ZIPFILE} s3://ossci-ios-build/ --acl public-read

.circleci/scripts/binary_linux_build.sh

@@ -1,30 +0,0 @@
-#!/bin/bash
-
-echo "RUNNING ON $(uname -a) WITH $(nproc) CPUS AND $(free -m)"
-set -eux -o pipefail
-source /env
-
-# Defaults here so they can be changed in one place
-export MAX_JOBS=12
-
-# Parse the parameters
-if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
-  build_script='conda/build_pytorch.sh'
-elif [[ "$DESIRED_CUDA" == cpu ]]; then
-  build_script='manywheel/build_cpu.sh'
-else
-  build_script='manywheel/build.sh'
-fi
-
-# We want to call unbuffer, which calls tclsh which finds the expect
-# package. The expect was installed by yum into /usr/bin so we want to
-# find /usr/bin/tclsh, but this is shadowed by /opt/conda/bin/tclsh in
-# the conda docker images, so we prepend it to the path here.
-if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
-  mkdir /just_tclsh_bin
-  ln -s /usr/bin/tclsh /just_tclsh_bin/tclsh
-  export PATH=/just_tclsh_bin:$PATH
-fi
-
-# Build the package
-SKIP_ALL_TESTS=1 unbuffer "/builder/$build_script" | ts

.circleci/scripts/binary_linux_test.sh

@@ -1,63 +0,0 @@
-#!/bin/bash
-
-source /home/circleci/project/env
-cat >/home/circleci/project/ci_test_script.sh <<EOL
-# =================== The following code will be executed inside Docker container ===================
-set -eux -o pipefail
-
-# Set up Python
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  retry conda create -qyn testenv python="$DESIRED_PYTHON"
-  source activate testenv >/dev/null
-elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
-  python_nodot="\$(echo $DESIRED_PYTHON | tr -d m.u)"
-  python_path="/opt/python/cp\$python_nodot-cp\${python_nodot}"
-  # Prior to Python 3.8 paths were suffixed with an 'm'
-  if [[ -d  "\${python_path}/bin" ]]; then
-    export PATH="\${python_path}/bin:\$PATH"
-  elif [[ -d "\${python_path}m/bin" ]]; then
-    export PATH="\${python_path}m/bin:\$PATH"
-  fi
-fi
-
-# Install the package
-# These network calls should not have 'retry's because they are installing
-# locally and aren't actually network calls
-# TODO there is duplicated and inconsistent test-python-env setup across this
-#   file, builder/smoke_test.sh, and builder/run_tests.sh, and also in the
-#   conda build scripts themselves. These should really be consolidated
-pkg="/final_pkgs/\$(ls /final_pkgs)"
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  conda install -y "\$pkg" --offline
-  if [[ "$DESIRED_CUDA" == 'cpu' ]]; then
-    retry conda install -y cpuonly -c pytorch
-  fi
-  retry conda install -yq future numpy protobuf six
-  if [[ "$DESIRED_CUDA" != 'cpu' ]]; then
-    # DESIRED_CUDA is in format cu90 or cu102
-    if [[ "${#DESIRED_CUDA}" == 4 ]]; then
-      cu_ver="${DESIRED_CUDA:2:1}.${DESIRED_CUDA:3}"
-    else
-      cu_ver="${DESIRED_CUDA:2:2}.${DESIRED_CUDA:4}"
-    fi
-    retry conda install -yq -c pytorch "cudatoolkit=\${cu_ver}"
-  fi
-elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
-  pip install "\$pkg"
-  retry pip install -q future numpy protobuf six
-fi
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  pkg="\$(ls /final_pkgs/*-latest.zip)"
-  unzip "\$pkg" -d /tmp
-  cd /tmp/libtorch
-fi
-
-# Test the package
-/builder/check_binary.sh
-
-# =================== The above code will be executed inside Docker container ===================
-EOL
-echo
-echo
-echo "The script that will run in the next step is:"
-cat /home/circleci/project/ci_test_script.sh

.circleci/scripts/binary_linux_upload.sh

@@ -1,48 +0,0 @@
-#!/bin/bash
-# Do NOT set -x
-source /home/circleci/project/env
-set -eu -o pipefail
-set +x
-declare -x "AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
-declare -x "AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
-
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-# DO NOT TURN -x ON BEFORE THIS LINE
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-set -eux -o pipefail
-export PATH="$MINICONDA_ROOT/bin:$PATH"
-
-# This gets set in binary_populate_env.sh, but lets have a sane default just in case
-PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly}
-# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
-#       The only difference is the trailing slash
-# Strip trailing slashes if there
-CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
-BACKUP_BUCKET="s3://pytorch-backup"
-
-retry pip install -q awscli
-# Upload the package to the final location
-pushd /home/circleci/project/final_pkgs
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  retry conda install -yq anaconda-client
-  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload  "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
-  # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-  # Because there's no actual conda command to read this
-  subdir=$(tar -xOf ./*.bz2 info/index.json | grep subdir  | cut -d ':' -f2 | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//')
-  BACKUP_DIR="conda/${subdir}"
-elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  for pkg in $(ls); do
-    retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
-  done
-  BACKUP_DIR="libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-else
-  s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
-  BACKUP_DIR="whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-fi
-
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  s3_dir="${BACKUP_BUCKET}/${CIRCLE_TAG}/${BACKUP_DIR}"
-  retry aws s3 cp --recursive . "$s3_dir"
-fi

.circleci/scripts/binary_macos_build.sh

@@ -1,24 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/Users/distiller/project/env"
-mkdir -p "$PYTORCH_FINAL_PACKAGE_DIR"
-
-# For some reason `unbuffer` breaks if we change the PATH here, so we
-# write a script with the PATH change in it and unbuffer the whole
-# thing
-build_script="$workdir/build_script.sh"
-touch "$build_script"
-chmod +x "$build_script"
-
-# Build
-cat >"$build_script" <<EOL
-export PATH="$workdir/miniconda/bin:$PATH"
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  "$workdir/builder/conda/build_pytorch.sh"
-else
-  export TORCH_PACKAGE_NAME="$(echo $TORCH_PACKAGE_NAME | tr '-' '_')"
-  "$workdir/builder/wheel/build_wheel.sh"
-fi
-EOL
-unbuffer "$build_script" | ts

.circleci/scripts/binary_macos_test.sh

@@ -1,34 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/Users/distiller/project/env"
-export "PATH=$workdir/miniconda/bin:$PATH"
-pkg="$workdir/final_pkgs/$(ls $workdir/final_pkgs)"
-
-# Create a new test env
-# TODO cut all this out into a separate test job and have an entirely different
-# miniconda
-if [[ "$PACKAGE_TYPE" != libtorch ]]; then
-  source deactivate || true
-  conda create -qyn test python="$DESIRED_PYTHON"
-  source activate test >/dev/null
-fi
-
-# Install the package
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  pkg="$(ls $workdir/final_pkgs/*-latest.zip)"
-  unzip "$pkg" -d /tmp
-  cd /tmp/libtorch
-elif [[ "$PACKAGE_TYPE" == conda ]]; then
-  conda install -y "$pkg" --offline
-else
-  pip install "$pkg" --no-index --no-dependencies -v
-fi
-
-# Test
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  $workdir/builder/check_binary.sh
-else
-  pushd "$workdir/pytorch"
-  $workdir/builder/run_tests.sh "$PACKAGE_TYPE" "$DESIRED_PYTHON" "$DESIRED_CUDA"
-fi

.circleci/scripts/binary_macos_upload.sh

@@ -1,48 +0,0 @@
-#!/bin/bash
-# Do NOT set -x
-set -eu -o pipefail
-set +x
-export AWS_ACCESS_KEY_ID="${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
-export AWS_SECRET_ACCESS_KEY="${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
-
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-# DO NOT TURN -x ON BEFORE THIS LINE
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-set -eux -o pipefail
-
-source "/Users/distiller/project/env"
-export "PATH=$workdir/miniconda/bin:$PATH"
-
-# This gets set in binary_populate_env.sh, but lets have a sane default just in case
-PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly}
-# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
-#       The only difference is the trailing slash
-# Strip trailing slashes if there
-CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
-BACKUP_BUCKET="s3://pytorch-backup"
-
-retry pip install -q awscli
-pushd "$workdir/final_pkgs"
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  retry conda install -yq anaconda-client
-  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
-  # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-  # Because there's no actual conda command to read this
-  subdir=$(tar -xOf ./*.bz2 info/index.json | grep subdir  | cut -d ':' -f2 | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//')
-  BACKUP_DIR="conda/${subdir}"
-elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  for pkg in $(ls); do
-    retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
-  done
-  BACKUP_DIR="libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-else
-  s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
-  BACKUP_DIR="whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-fi
-
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  s3_dir="${BACKUP_BUCKET}/${CIRCLE_TAG}/${BACKUP_DIR}"
-  retry aws s3 cp --recursive . "$s3_dir"
-fi

.circleci/scripts/binary_populate_env.sh

@@ -1,172 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-export TZ=UTC
-
-tagged_version() {
-  # Grabs version from either the env variable CIRCLE_TAG
-  # or the pytorch git described version
-  if [[ "$OSTYPE" == "msys" ]]; then
-    GIT_DESCRIBE="git --git-dir ${workdir}/p/.git describe"
-  else
-    GIT_DESCRIBE="git --git-dir ${workdir}/pytorch/.git describe"
-  fi
-  if [[ -n "${CIRCLE_TAG:-}" ]]; then
-    echo "${CIRCLE_TAG}"
-  elif ${GIT_DESCRIBE} --exact --tags >/dev/null; then
-    ${GIT_DESCRIBE} --tags
-  else
-    return 1
-  fi
-}
-
-# We need to write an envfile to persist these variables to following
-# steps, but the location of the envfile depends on the circleci executor
-if [[ "$(uname)" == Darwin ]]; then
-  # macos executor (builds and tests)
-  workdir="/Users/distiller/project"
-elif [[ "$OSTYPE" == "msys" ]]; then
-  # windows executor (builds and tests)
-  workdir="/c/w"
-elif [[ -d "/home/circleci/project" ]]; then
-  # machine executor (binary tests)
-  workdir="/home/circleci/project"
-else
-  # docker executor (binary builds)
-  workdir="/"
-fi
-envfile="$workdir/env"
-touch "$envfile"
-chmod +x "$envfile"
-
-# Parse the BUILD_ENVIRONMENT to package type, python, and cuda
-configs=($BUILD_ENVIRONMENT)
-export PACKAGE_TYPE="${configs[0]}"
-export DESIRED_PYTHON="${configs[1]}"
-export DESIRED_CUDA="${configs[2]}"
-if [[ "${BUILD_FOR_SYSTEM:-}" == "windows" ]]; then
-  export DESIRED_DEVTOOLSET=""
-  export LIBTORCH_CONFIG="${configs[3]:-}"
-  if [[ "$LIBTORCH_CONFIG" == 'debug' ]]; then
-    export DEBUG=1
-  fi
-else
-  export DESIRED_DEVTOOLSET="${configs[3]:-}"
-fi
-if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then
-  export BUILD_PYTHONLESS=1
-fi
-
-# Pick docker image
-export DOCKER_IMAGE=${DOCKER_IMAGE:-}
-if [[ -z "$DOCKER_IMAGE" ]]; then
-  if [[ "$PACKAGE_TYPE" == conda ]]; then
-    export DOCKER_IMAGE="pytorch/conda-cuda"
-  elif [[ "$DESIRED_CUDA" == cpu ]]; then
-    export DOCKER_IMAGE="pytorch/manylinux-cuda100"
-  else
-    export DOCKER_IMAGE="pytorch/manylinux-cuda${DESIRED_CUDA:2}"
-  fi
-fi
-
-# Default to nightly, since that's where this normally uploads to
-PIP_UPLOAD_FOLDER='nightly/'
-# We put this here so that OVERRIDE_PACKAGE_VERSION below can read from it
-export DATE="$(date -u +%Y%m%d)"
-#TODO: We should be pulling semver version from the base version.txt
-BASE_BUILD_VERSION="1.6.0.dev$DATE"
-# Change BASE_BUILD_VERSION to git tag when on a git tag
-# Use 'git -C' to make doubly sure we're in the correct directory for checking
-# the git tag
-if tagged_version >/dev/null; then
-  # Switch upload folder to 'test/' if we are on a tag
-  PIP_UPLOAD_FOLDER='test/'
-  # Grab git tag, remove prefixed v and remove everything after -
-  # Used to clean up tags that are for release candidates like v1.6.0-rc1
-  # Turns tag v1.6.0-rc1 -> v1.6.0
-  BASE_BUILD_VERSION="$(tagged_version | sed -e 's/^v//' -e 's/-.*$//')"
-fi
-if [[ "$(uname)" == 'Darwin' ]] || [[ "$DESIRED_CUDA" == "cu102" ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
-  export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}"
-else
-  export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}+$DESIRED_CUDA"
-fi
-export PYTORCH_BUILD_NUMBER=1
-
-
-JAVA_HOME=
-BUILD_JNI=OFF
-if [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  POSSIBLE_JAVA_HOMES=()
-  POSSIBLE_JAVA_HOMES+=(/usr/local)
-  POSSIBLE_JAVA_HOMES+=(/usr/lib/jvm/java-8-openjdk-amd64)
-  POSSIBLE_JAVA_HOMES+=(/Library/Java/JavaVirtualMachines/*.jdk/Contents/Home)
-  for JH in "${POSSIBLE_JAVA_HOMES[@]}" ; do
-    if [[ -e "$JH/include/jni.h" ]] ; then
-      echo "Found jni.h under $JH"
-      JAVA_HOME="$JH"
-      BUILD_JNI=ON
-      break
-    fi
-  done
-  if [ -z "$JAVA_HOME" ]; then
-    echo "Did not find jni.h"
-  fi
-fi
-
-cat >>"$envfile" <<EOL
-# =================== The following code will be executed inside Docker container ===================
-export TZ=UTC
-echo "Running on $(uname -a) at $(date)"
-
-export PACKAGE_TYPE="$PACKAGE_TYPE"
-export DESIRED_PYTHON="$DESIRED_PYTHON"
-export DESIRED_CUDA="$DESIRED_CUDA"
-export LIBTORCH_VARIANT="${LIBTORCH_VARIANT:-}"
-export BUILD_PYTHONLESS="${BUILD_PYTHONLESS:-}"
-export DESIRED_DEVTOOLSET="$DESIRED_DEVTOOLSET"
-if [[ "${BUILD_FOR_SYSTEM:-}" == "windows" ]]; then
-  export LIBTORCH_CONFIG="${LIBTORCH_CONFIG:-}"
-  export DEBUG="${DEBUG:-}"
-fi
-
-export DATE="$DATE"
-export NIGHTLIES_DATE_PREAMBLE=1.6.0.dev
-export PYTORCH_BUILD_VERSION="$PYTORCH_BUILD_VERSION"
-export PYTORCH_BUILD_NUMBER="$PYTORCH_BUILD_NUMBER"
-export OVERRIDE_PACKAGE_VERSION="$PYTORCH_BUILD_VERSION"
-
-# TODO: We don't need this anymore IIUC
-export TORCH_PACKAGE_NAME='torch'
-export TORCH_CONDA_BUILD_FOLDER='pytorch-nightly'
-
-export USE_FBGEMM=1
-export JAVA_HOME=$JAVA_HOME
-export BUILD_JNI=$BUILD_JNI
-export PIP_UPLOAD_FOLDER="$PIP_UPLOAD_FOLDER"
-export DOCKER_IMAGE="$DOCKER_IMAGE"
-
-export workdir="$workdir"
-export MAC_PACKAGE_WORK_DIR="$workdir"
-if [[ "$OSTYPE" == "msys" ]]; then
-  export PYTORCH_ROOT="$workdir/p"
-  export BUILDER_ROOT="$workdir/b"
-else
-  export PYTORCH_ROOT="$workdir/pytorch"
-  export BUILDER_ROOT="$workdir/builder"
-fi
-export MINICONDA_ROOT="$workdir/miniconda"
-export PYTORCH_FINAL_PACKAGE_DIR="$workdir/final_pkgs"
-
-export CIRCLE_TAG="${CIRCLE_TAG:-}"
-export CIRCLE_SHA1="$CIRCLE_SHA1"
-export CIRCLE_PR_NUMBER="${CIRCLE_PR_NUMBER:-}"
-export CIRCLE_BRANCH="$CIRCLE_BRANCH"
-# =================== The above code will be executed inside Docker container ===================
-EOL
-
-echo 'retry () {' >> "$envfile"
-echo '    $*  || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)' >> "$envfile"
-echo '}' >> "$envfile"
-echo 'export -f retry' >> "$envfile"
-
-cat "$envfile"

.circleci/scripts/binary_run_in_docker.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-
-# This section is used in the binary_test and smoke_test jobs. It expects
-# 'binary_populate_env' to have populated /home/circleci/project/env and it
-# expects another section to populate /home/circleci/project/ci_test_script.sh
-# with the code to run in the docker
-
-# Expect all needed environment variables to be written to this file
-source /home/circleci/project/env
-echo "Running the following code in Docker"
-cat /home/circleci/project/ci_test_script.sh
-echo
-echo
-set -eux -o pipefail
-
-# Expect actual code to be written to this file
-chmod +x /home/circleci/project/ci_test_script.sh
-
-VOLUME_MOUNTS="-v /home/circleci/project/:/circleci_stuff -v /home/circleci/project/final_pkgs:/final_pkgs -v ${PYTORCH_ROOT}:/pytorch -v ${BUILDER_ROOT}:/builder"
-# Run the docker
-if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --runtime=nvidia ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
-else
-  export id=$(docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined ${VOLUME_MOUNTS} -t -d "${DOCKER_IMAGE}")
-fi
-
-# Execute the test script that was populated by an earlier section
-export COMMAND='((echo "source /circleci_stuff/env && /circleci_stuff/ci_test_script.sh") | docker exec -i "$id" bash) 2>&1'
-echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts

.circleci/scripts/binary_windows_build.sh

@@ -1,41 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/c/w/env"
-mkdir -p "$PYTORCH_FINAL_PACKAGE_DIR"
-
-export CUDA_VERSION="${DESIRED_CUDA/cu/}"
-export USE_SCCACHE=1
-export SCCACHE_BUCKET=ossci-compiler-cache-windows
-export NIGHTLIES_PYTORCH_ROOT="$PYTORCH_ROOT"
-
-if [[ "$CUDA_VERSION" == "92" || "$CUDA_VERSION" == "100" ]]; then
-  export VC_YEAR=2017
-else
-  export VC_YEAR=2019
-fi
-
-set +x
-export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}
-export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}
-set -x
-
-if [[ "$CIRCLECI" == 'true' && -d "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages\\_Instances" ]]; then
-  mv "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages\\_Instances" .
-  rm -rf "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-  mkdir -p "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-  mv _Instances "C:\\ProgramData\\Microsoft\\VisualStudio\\Packages"
-fi
-
-echo "Free space on filesystem before build:"
-df -h
-
-pushd "$BUILDER_ROOT"
-if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
-  ./windows/internal/build_conda.bat
-elif [[ "$PACKAGE_TYPE" == 'wheel' || "$PACKAGE_TYPE" == 'libtorch' ]]; then
-  ./windows/internal/build_wheels.bat
-fi
-
-echo "Free space on filesystem after build:"
-df -h

.circleci/scripts/binary_windows_test.sh

@@ -1,19 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-source "/c/w/env"
-
-export CUDA_VERSION="${DESIRED_CUDA/cu/}"
-export VC_YEAR=2017
-
-if [[ "$CUDA_VERSION" == "92" || "$CUDA_VERSION" == "100" ]]; then
-  export VC_YEAR=2017
-else
-  export VC_YEAR=2019
-fi
-
-pushd "$BUILDER_ROOT"
-
-./windows/internal/smoke_test.bat
-
-popd

.circleci/scripts/binary_windows_upload.sh

@@ -1,47 +0,0 @@
-#!/bin/bash
-set -eu -o pipefail
-set +x
-declare -x "AWS_ACCESS_KEY_ID=${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}"
-declare -x "AWS_SECRET_ACCESS_KEY=${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}"
-
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-# DO NOT TURN -x ON BEFORE THIS LINE
-#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
-set -eux -o pipefail
-
-source "/env"
-
-# This gets set in binary_populate_env.sh, but lets have a sane default just in case
-PIP_UPLOAD_FOLDER=${PIP_UPLOAD_FOLDER:-nightly/}
-# TODO: Combine CONDA_UPLOAD_CHANNEL and PIP_UPLOAD_FOLDER into one variable
-#       The only difference is the trailing slash
-# Strip trailing slashes if there
-CONDA_UPLOAD_CHANNEL=$(echo "${PIP_UPLOAD_FOLDER}" | sed 's:/*$::')
-BACKUP_BUCKET="s3://pytorch-backup"
-
-retry pip install -q awscli
-pushd /root/workspace/final_pkgs
-# Upload the package to the final location
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  retry conda install -yq anaconda-client
-  retry anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload  "$(ls)" -u "pytorch-${CONDA_UPLOAD_CHANNEL}" --label main --no-progress --force
-  # Fetch  platform (eg. win-64, linux-64, etc.) from index file
-  # Because there's no actual conda command to read this
-  subdir=$(tar -xOf ./*.bz2 info/index.json | grep subdir  | cut -d ':' -f2 | sed -e 's/[[:space:]]//' -e 's/"//g' -e 's/,//')
-  BACKUP_DIR="conda/${subdir}"
-elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
-  s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  for pkg in $(ls); do
-    retry aws s3 cp "$pkg" "$s3_dir" --acl public-read
-  done
-  BACKUP_DIR="libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-else
-  s3_dir="s3://pytorch/whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-  retry aws s3 cp "$(ls)" "$s3_dir" --acl public-read
-  BACKUP_DIR="whl/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"
-fi
-
-if [[ -n "${CIRCLE_TAG:-}" ]]; then
-  s3_dir="${BACKUP_BUCKET}/${CIRCLE_TAG}/${BACKUP_DIR}"
-  retry aws s3 cp --recursive . "$s3_dir"
-fi

.circleci/scripts/build_android_gradle.sh

@@ -1,85 +0,0 @@
-#!/usr/bin/env bash
-set -eux -o pipefail
-
-export ANDROID_NDK_HOME=/opt/ndk
-export ANDROID_HOME=/opt/android/sdk
-
-# Must be in sync with GRADLE_VERSION in docker image for android
-# https://github.com/pietern/pytorch-dockerfiles/blob/master/build.sh#L155
-export GRADLE_VERSION=4.10.3
-export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
-export GRADLE_PATH=$GRADLE_HOME/bin/gradle
-
-BUILD_ANDROID_INCLUDE_DIR_x86=~/workspace/build_android/install/include
-BUILD_ANDROID_LIB_DIR_x86=~/workspace/build_android/install/lib
-
-BUILD_ANDROID_INCLUDE_DIR_x86_64=~/workspace/build_android_install_x86_64/install/include
-BUILD_ANDROID_LIB_DIR_x86_64=~/workspace/build_android_install_x86_64/install/lib
-
-BUILD_ANDROID_INCLUDE_DIR_arm_v7a=~/workspace/build_android_install_arm_v7a/install/include
-BUILD_ANDROID_LIB_DIR_arm_v7a=~/workspace/build_android_install_arm_v7a/install/lib
-
-BUILD_ANDROID_INCLUDE_DIR_arm_v8a=~/workspace/build_android_install_arm_v8a/install/include
-BUILD_ANDROID_LIB_DIR_arm_v8a=~/workspace/build_android_install_arm_v8a/install/lib
-
-PYTORCH_ANDROID_SRC_MAIN_DIR=~/workspace/android/pytorch_android/src/main
-
-JNI_INCLUDE_DIR=${PYTORCH_ANDROID_SRC_MAIN_DIR}/cpp/libtorch_include
-mkdir -p $JNI_INCLUDE_DIR
-
-JNI_LIBS_DIR=${PYTORCH_ANDROID_SRC_MAIN_DIR}/jniLibs
-mkdir -p $JNI_LIBS_DIR
-
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_x86} ${JNI_INCLUDE_DIR}/x86
-ln -s ${BUILD_ANDROID_LIB_DIR_x86} ${JNI_LIBS_DIR}/x86
-
-if [[ "${BUILD_ENVIRONMENT}" != *-gradle-build-only-x86_32* ]]; then
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_x86_64} ${JNI_INCLUDE_DIR}/x86_64
-ln -s ${BUILD_ANDROID_LIB_DIR_x86_64} ${JNI_LIBS_DIR}/x86_64
-
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v7a} ${JNI_INCLUDE_DIR}/armeabi-v7a
-ln -s ${BUILD_ANDROID_LIB_DIR_arm_v7a} ${JNI_LIBS_DIR}/armeabi-v7a
-
-ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v8a} ${JNI_INCLUDE_DIR}/arm64-v8a
-ln -s ${BUILD_ANDROID_LIB_DIR_arm_v8a} ${JNI_LIBS_DIR}/arm64-v8a
-fi
-
-env
-echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
-
-GRADLE_PARAMS="-p android assembleRelease --debug --stacktrace"
-if [[ "${BUILD_ENVIRONMENT}" == *-gradle-build-only-x86_32* ]]; then
-    GRADLE_PARAMS+=" -PABI_FILTERS=x86"
-fi
-
-if [ -n "{GRADLE_OFFLINE:-}" ]; then
-    GRADLE_PARAMS+=" --offline"
-fi
-
-# touch gradle cache files to prevent expiration
-while IFS= read -r -d '' file
-do
-  touch "$file" || true
-done < <(find /var/lib/jenkins/.gradle -type f -print0)
-
-env
-
-export GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
-rm -f $GRADLE_LOCAL_PROPERTIES
-echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-echo "cmake.dir=/usr/local" >> $GRADLE_LOCAL_PROPERTIES
-
-$GRADLE_PATH $GRADLE_PARAMS
-
-find . -type f -name "*.a" -exec ls -lh {} \;
-
-while IFS= read -r -d '' file
-do
-  echo
-  echo "$file"
-  ls -lah "$file"
-  zipinfo -l "$file"
-done < <(find . -type f -name '*.aar' -print0)
-
-find . -type f -name *aar -print | xargs tar cfvz ~/workspace/android/artifacts.tgz

.circleci/scripts/cpp_doc_push_script.sh

@@ -1,126 +0,0 @@
-# =================== The following code **should** be executed inside Docker container ===================
-
-# Install dependencies
-sudo apt-get -y update
-sudo apt-get -y install expect-dev
-
-# This is where the local pytorch install in the docker image is located
-pt_checkout="/var/lib/jenkins/workspace"
-
-# Since we're cat-ing this file, we need to escape all $'s
-echo "cpp_doc_push_script.sh: Invoked with $*"
-
-# Argument 1: Where to copy the built documentation for Python API to
-# (pytorch.github.io/$install_path)
-install_path="$1"
-if [ -z "$install_path" ]; then
-echo "error: cpp_doc_push_script.sh: install_path (arg1) not specified"
-  exit 1
-fi
-
-# Argument 2: What version of the Python API docs we are building.
-version="$2"
-if [ -z "$version" ]; then
-echo "error: cpp_doc_push_script.sh: version (arg2) not specified"
-  exit 1
-fi
-
-is_master_doc=false
-if [ "$version" == "master" ]; then
-  is_master_doc=true
-fi
-
-# Argument 3: (optional) If present, we will NOT do any pushing. Used for testing.
-dry_run=false
-if [ "$3" != "" ]; then
-  dry_run=true
-fi
-
-echo "install_path: $install_path  version: $version  dry_run: $dry_run"
-
-# ======================== Building PyTorch C++ API Docs ========================
-
-echo "Building PyTorch C++ API docs..."
-
-# Clone the cppdocs repo
-rm -rf cppdocs
-git clone https://github.com/pytorch/cppdocs
-
-set -ex
-
-sudo apt-get -y install doxygen
-
-# Generate ATen files
-pushd "${pt_checkout}"
-pip install -r requirements.txt
-time python aten/src/ATen/gen.py \
-  -s aten/src/ATen \
-  -d build/aten/src/ATen \
-  aten/src/ATen/Declarations.cwrap \
-  aten/src/THCUNN/generic/THCUNN.h \
-  aten/src/ATen/nn.yaml \
-  aten/src/ATen/native/native_functions.yaml
-
-# Copy some required files
-cp aten/src/ATen/common_with_cwrap.py tools/shared/cwrap_common.py
-cp torch/_utils_internal.py tools/shared
-
-# Generate PyTorch files
-time python tools/setup_helpers/generate_code.py \
-  --declarations-path build/aten/src/ATen/Declarations.yaml \
-  --nn-path aten/src/
-
-# Build the docs
-pushd docs/cpp
-pip install breathe==4.13.0 bs4 lxml six
-pip install --no-cache-dir -e "git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme"
-pip install exhale>=0.2.1
-pip install sphinx==2.4.4
-# Uncomment once it is fixed
-# pip install -r requirements.txt
-time make VERBOSE=1 html -j
-
-popd
-popd
-
-pushd cppdocs
-
-# Purge everything with some exceptions
-mkdir /tmp/cppdocs-sync
-mv _config.yml README.md /tmp/cppdocs-sync/
-rm -rf *
-
-# Copy over all the newly generated HTML
-cp -r "${pt_checkout}"/docs/cpp/build/html/* .
-
-# Copy back _config.yml
-rm -rf _config.yml
-mv /tmp/cppdocs-sync/* .
-
-# Make a new commit
-git add . || true
-git status
-git config user.email "soumith+bot@pytorch.org"
-git config user.name "pytorchbot"
-# If there aren't changes, don't make a commit; push is no-op
-git commit -m "Automatic sync on $(date)" || true
-git status
-
-if [ "$dry_run" = false ]; then
-  echo "Pushing to https://github.com/pytorch/cppdocs"
-  set +x
-/usr/bin/expect <<DONE
-  spawn git push -u origin master
-  expect "Username*"
-  send "pytorchbot\n"
-  expect "Password*"
-  send "$::env(GITHUB_PYTORCHBOT_TOKEN)\n"
-  expect eof
-DONE
-  set -x
-else
-  echo "Skipping push due to dry_run"
-fi
-
-popd
-# =================== The above code **should** be executed inside Docker container ===================

.circleci/scripts/publish_android_snapshot.sh

@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-# DO NOT ADD 'set -x' not to reveal CircleCI secret context environment variables
-set -eu -o pipefail
-
-export ANDROID_NDK_HOME=/opt/ndk
-export ANDROID_HOME=/opt/android/sdk
-
-export GRADLE_VERSION=4.10.3
-export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
-export GRADLE_PATH=$GRADLE_HOME/bin/gradle
-
-echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
-ls -la ~/workspace
-
-GRADLE_PROPERTIES=~/workspace/android/gradle.properties
-
-IS_SNAPSHOT="$(grep 'VERSION_NAME=[0-9\.]\+-SNAPSHOT' "$GRADLE_PROPERTIES")"
-echo "IS_SNAPSHOT:$IS_SNAPSHOT"
-
-if [ -z "$IS_SNAPSHOT" ]; then
-  echo "Error: version is not snapshot."
-elif [ -z "$SONATYPE_NEXUS_USERNAME" ]; then
-  echo "Error: missing env variable SONATYPE_NEXUS_USERNAME."
-elif [ -z "$SONATYPE_NEXUS_PASSWORD" ]; then
-  echo "Error: missing env variable SONATYPE_NEXUS_PASSWORD."
-elif [ -z "$ANDROID_SIGN_KEY" ]; then
-  echo "Error: missing env variable ANDROID_SIGN_KEY."
-elif [ -z "$ANDROID_SIGN_PASS" ]; then
-  echo "Error: missing env variable ANDROID_SIGN_PASS."
-else
-  GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
-  rm -f $GRADLE_LOCAL_PROPERTIES
-
-  echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
-  echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
-
-  echo "SONATYPE_NEXUS_USERNAME=${SONATYPE_NEXUS_USERNAME}" >> $GRADLE_PROPERTIES
-  echo "SONATYPE_NEXUS_PASSWORD=${SONATYPE_NEXUS_PASSWORD}" >> $GRADLE_PROPERTIES
-
-  echo "signing.keyId=${ANDROID_SIGN_KEY}" >> $GRADLE_PROPERTIES
-  echo "signing.password=${ANDROID_SIGN_PASS}" >> $GRADLE_PROPERTIES
-
-  $GRADLE_PATH -p ~/workspace/android/ uploadArchives
-fi

.circleci/scripts/python_doc_push_script.sh

@@ -1,146 +0,0 @@
-# =================== The following code **should** be executed inside Docker container ===================
-
-# Install dependencies
-sudo apt-get -y update
-sudo apt-get -y install expect-dev
-
-# This is where the local pytorch install in the docker image is located
-pt_checkout="/var/lib/jenkins/workspace"
-
-echo "python_doc_push_script.sh: Invoked with $*"
-
-set -ex
-
-# Argument 1: Where to copy the built documentation to
-# (pytorch.github.io/$install_path)
-install_path="$1"
-if [ -z "$install_path" ]; then
-echo "error: python_doc_push_script.sh: install_path (arg1) not specified"
-  exit 1
-fi
-
-# Argument 2: What version of the docs we are building.
-version="$2"
-if [ -z "$version" ]; then
-echo "error: python_doc_push_script.sh: version (arg2) not specified"
-  exit 1
-fi
-
-is_master_doc=false
-if [ "$version" == "master" ]; then
-  is_master_doc=true
-fi
-
-# Argument 3: The branch to push to. Usually is "site"
-branch="$3"
-if [ -z "$branch" ]; then
-echo "error: python_doc_push_script.sh: branch (arg3) not specified"
-  exit 1
-fi
-
-# Argument 4: (optional) If present, we will NOT do any pushing. Used for testing.
-dry_run=false
-if [ "$4" != "" ]; then
-  dry_run=true
-fi
-
-echo "install_path: $install_path  version: $version  dry_run: $dry_run"
-
-git clone https://github.com/pytorch/pytorch.github.io -b $branch
-pushd pytorch.github.io
-
-export LC_ALL=C
-export PATH=/opt/conda/bin:$PATH
-
-rm -rf pytorch || true
-
-# Install TensorBoard in python 3 so torch.utils.tensorboard classes render
-pip install -q https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py3-none-any.whl
-
-# Get all the documentation sources, put them in one place
-pushd "$pt_checkout"
-git clone https://github.com/pytorch/vision
-pushd vision
-conda install -q pillow
-time python setup.py install
-popd
-pushd docs
-rm -rf source/torchvision
-cp -a ../vision/docs/source source/torchvision
-
-# Build the docs
-pip -q install -r requirements.txt || true
-if [ "$is_master_doc" = true ]; then
-  # TODO: fix gh-38011 then enable this which changes warnings into errors
-  # export SPHINXOPTS="-WT --keep-going"
-  make html
-  make coverage
-  # Now we have the coverage report, we need to make sure it is empty.
-  # Count the number of lines in the file and turn that number into a variable
-  # $lines. The `cut -f1 ...` is to only parse the number, not the filename
-  # Skip the report header by subtracting 2: the header will be output even if
-  # there are no undocumented items.
-  #
-  # Also: see docs/source/conf.py for "coverage_ignore*" items, which should
-  # be documented then removed from there.
-  lines=$(wc -l build/coverage/python.txt 2>/dev/null |cut -f1 -d' ')
-  undocumented=$(($lines - 2))
-  if [ $undocumented -lt 0 ]; then
-    echo coverage output not found
-    exit 1
-  elif [ $undocumented -gt 0 ]; then
-    echo undocumented objects found:
-    cat build/coverage/python.txt
-    exit 1
-  fi
-else
-  # Don't fail the build on coverage problems
-  make html-stable
-fi
-
-# Move them into the docs repo
-popd
-popd
-git rm -rf "$install_path" || true
-mv "$pt_checkout/docs/build/html" "$install_path"
-
-# Add the version handler by search and replace.
-# XXX: Consider moving this to the docs Makefile or site build
-if [ "$is_master_doc" = true ]; then
-  find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>\1 \&#x25BC</a>@g"
-else
-  find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>$version \&#x25BC</a>@g"
-fi
-
-# Prevent Google from indexing $install_path/_modules. This folder contains
-# generated source files.
-# NB: the following only works on gnu sed. The sed shipped with mac os is different.
-# One can `brew install gnu-sed` on a mac and then use "gsed" instead of "sed".
-find "$install_path/_modules" -name "*.html" -print0 | xargs -0 sed -i '/<head>/a \ \ <meta name="robots" content="noindex">'
-
-git add "$install_path" || true
-git status
-git config user.email "soumith+bot@pytorch.org"
-git config user.name "pytorchbot"
-# If there aren't changes, don't make a commit; push is no-op
-git commit -m "auto-generating sphinx docs" || true
-git status
-
-if [ "$dry_run" = false ]; then
-  echo "Pushing to pytorch.github.io:$branch"
-  set +x
-/usr/bin/expect <<DONE
-  spawn git push origin $branch
-  expect "Username*"
-  send "pytorchbot\n"
-  expect "Password*"
-  send "$::env(GITHUB_PYTORCHBOT_TOKEN)\n"
-  expect eof
-DONE
-  set -x
-else
-  echo "Skipping push due to dry_run"
-fi
-
-popd
-# =================== The above code **should** be executed inside Docker container ===================

.circleci/scripts/setup_ci_environment.sh

@@ -1,98 +0,0 @@
-#!/usr/bin/env bash
-set -ex -o pipefail
-
-# Set up NVIDIA docker repo
-curl -s -L --retry 3 https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
-echo "deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
-echo "deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
-echo "deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
-
-# Remove unnecessary sources
-sudo rm -f /etc/apt/sources.list.d/google-chrome.list
-sudo rm -f /etc/apt/heroku.list
-sudo rm -f /etc/apt/openjdk-r-ubuntu-ppa-xenial.list
-sudo rm -f /etc/apt/partner.list
-
-retry () {
-    $*  || $* || $* || $* || $*
-}
-
-# Method adapted from here: https://askubuntu.com/questions/875213/apt-get-to-retry-downloading
-# (with use of tee to avoid permissions problems)
-# This is better than retrying the whole apt-get command
-echo "APT::Acquire::Retries \"3\";" | sudo tee /etc/apt/apt.conf.d/80-retries
-
-sudo apt-get -y update
-sudo apt-get -y remove linux-image-generic linux-headers-generic linux-generic docker-ce
-# WARNING: Docker version is hardcoded here; you must update the
-# version number below for docker-ce and nvidia-docker2 to get newer
-# versions of Docker.  We hardcode these numbers because we kept
-# getting broken CI when Docker would update their docker version,
-# and nvidia-docker2 would be out of date for a day until they
-# released a newer version of their package.
-#
-# How to figure out what the correct versions of these packages are?
-# My preferred method is to start a Docker instance of the correct
-# Ubuntu version (e.g., docker run -it ubuntu:16.04) and then ask
-# apt what the packages you need are.  Note that the CircleCI image
-# comes with Docker.
-#
-# Using 'retry' here as belt-and-suspenders even though we are
-# presumably retrying at the single-package level via the
-# apt.conf.d/80-retries technique.
-retry sudo apt-get -y install \
-  linux-headers-$(uname -r) \
-  linux-image-generic \
-  moreutils \
-  docker-ce=5:18.09.4~3-0~ubuntu-xenial \
-  nvidia-container-runtime=2.0.0+docker18.09.4-1 \
-  nvidia-docker2=2.0.3+docker18.09.4-1 \
-  expect-dev
-
-sudo pkill -SIGHUP dockerd
-
-
-retry sudo pip -q install awscli==1.16.35
-
-if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  DRIVER_FN="NVIDIA-Linux-x86_64-440.59.run"
-  wget "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"
-  sudo /bin/bash "$DRIVER_FN" -s --no-drm || (sudo cat /var/log/nvidia-installer.log && false)
-  nvidia-smi
-fi
-
-if [[ "${BUILD_ENVIRONMENT}" == *-build ]]; then
-  echo "declare -x IN_CIRCLECI=1" > /home/circleci/project/env
-  echo "declare -x COMMIT_SOURCE=${CIRCLE_BRANCH:-}" >> /home/circleci/project/env
-  echo "declare -x SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2" >> /home/circleci/project/env
-  if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-    echo "declare -x TORCH_CUDA_ARCH_LIST=5.2" >> /home/circleci/project/env
-  fi
-  export SCCACHE_MAX_JOBS=`expr $(nproc) - 1`
-  export MEMORY_LIMIT_MAX_JOBS=8  # the "large" resource class on CircleCI has 32 CPU cores, if we use all of them we'll OOM
-  export MAX_JOBS=$(( ${SCCACHE_MAX_JOBS} > ${MEMORY_LIMIT_MAX_JOBS} ? ${MEMORY_LIMIT_MAX_JOBS} : ${SCCACHE_MAX_JOBS} ))
-  echo "declare -x MAX_JOBS=${MAX_JOBS}" >> /home/circleci/project/env
-
-  if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
-    # This IAM user allows write access to S3 bucket for sccache & bazels3cache
-    set +x
-    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
-    echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
-    echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
-    set -x
-  else
-    # This IAM user allows write access to S3 bucket for sccache
-    set +x
-    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
-    echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
-    echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
-    set -x
-  fi
-fi
-
-# This IAM user only allows read-write access to ECR
-set +x
-export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_ECR_READ_WRITE_V4:-}
-export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_ECR_READ_WRITE_V4:-}
-eval $(aws ecr get-login --region us-east-1 --no-include-email)
-set -x

.circleci/scripts/setup_linux_system_environment.sh

@@ -1,50 +0,0 @@
-#!/usr/bin/env bash
-set -eux -o pipefail
-
-# Set up CircleCI GPG keys for apt, if needed
-curl --retry 3 -s -L https://packagecloud.io/circleci/trusty/gpgkey | sudo apt-key add -
-
-# Stop background apt updates.  Hypothetically, the kill should not
-# be necessary, because stop is supposed to send a kill signal to
-# the process, but we've added it for good luck.  Also
-# hypothetically, it's supposed to be unnecessary to wait for
-# the process to block.  We also have that line for good luck.
-# If you like, try deleting them and seeing if it works.
-sudo systemctl stop apt-daily.service || true
-sudo systemctl kill --kill-who=all apt-daily.service || true
-
-sudo systemctl stop unattended-upgrades.service || true
-sudo systemctl kill --kill-who=all unattended-upgrades.service || true
-
-# wait until `apt-get update` has been killed
-while systemctl is-active --quiet apt-daily.service
-do
-    sleep 1;
-done
-while systemctl is-active --quiet unattended-upgrades.service
-do
-    sleep 1;
-done
-
-# See if we actually were successful
-systemctl list-units --all | cat
-
-# For good luck, try even harder to kill apt-get
-sudo pkill apt-get || true
-
-# For even better luck, purge unattended-upgrades
-sudo apt-get purge -y unattended-upgrades
-
-cat /etc/apt/sources.list
-
-# For the bestest luck, kill again now
-sudo pkill apt || true
-sudo pkill dpkg || true
-
-# Try to detect if apt/dpkg is stuck
-if ps auxfww | grep '[a]pt'; then
-  echo "WARNING: There are leftover apt processes; subsequent apt update will likely fail"
-fi
-if ps auxfww | grep '[d]pkg'; then
-  echo "WARNING: There are leftover dpkg processes; subsequent apt update will likely fail"
-fi

.circleci/scripts/upload_binary_size_to_scuba.py

@@ -1,145 +0,0 @@
-import glob
-import json
-import logging
-import os
-import os.path
-import pathlib
-import re
-import sys
-import time
-import zipfile
-
-import requests
-
-
-def get_size(file_dir):
-    try:
-        # we should only expect one file, if no, something is wrong
-        file_name = glob.glob(os.path.join(file_dir, "*"))[0]
-        return os.stat(file_name).st_size
-    except:
-        logging.exception(f"error getting file from: {file_dir}")
-        return 0
-
-
-def build_message(size):
-    pkg_type, py_ver, cu_ver, *_ = os.environ.get("BUILD_ENVIRONMENT", "").split() + [
-        None,
-        None,
-        None,
-    ]
-    os_name = os.uname()[0].lower()
-    if os_name == "darwin":
-        os_name = "macos"
-    return {
-        "normal": {
-            "os": os_name,
-            "pkg_type": pkg_type,
-            "py_ver": py_ver,
-            "cu_ver": cu_ver,
-            "pr": os.environ.get("CIRCLE_PR_NUMBER"),
-            "build_num": os.environ.get("CIRCLE_BUILD_NUM"),
-            "sha1": os.environ.get("CIRCLE_SHA1"),
-            "branch": os.environ.get("CIRCLE_BRANCH"),
-        },
-        "int": {
-            "time": int(time.time()),
-            "size": size,
-            "commit_time": int(os.environ.get("COMMIT_TIME", "0")),
-        },
-    }
-
-
-def send_message(messages):
-    access_token = os.environ.get("SCRIBE_GRAPHQL_ACCESS_TOKEN")
-    if not access_token:
-        raise ValueError("Can't find access token from environment variable")
-    url = "https://graph.facebook.com/scribe_logs"
-    r = requests.post(
-        url,
-        data={
-            "access_token": access_token,
-            "logs": json.dumps(
-                [
-                    {
-                        "category": "perfpipe_pytorch_binary_size",
-                        "message": json.dumps(message),
-                        "line_escape": False,
-                    }
-                    for message in messages
-                ]
-            ),
-        },
-    )
-    print(r.text)
-    r.raise_for_status()
-
-
-def report_android_sizes(file_dir):
-    def gen_sizes():
-        # we should only expect one file, if no, something is wrong
-        aar_files = list(pathlib.Path(file_dir).rglob("pytorch_android-*.aar"))
-        if len(aar_files) != 1:
-            logging.exception(f"error getting aar files from: {file_dir} / {aar_files}")
-            return
-
-        aar_file = aar_files[0]
-        zf = zipfile.ZipFile(aar_file)
-        for info in zf.infolist():
-            # Scan ".so" libs in `jni` folder. Examples:
-            # jni/arm64-v8a/libfbjni.so
-            # jni/arm64-v8a/libpytorch_jni.so
-            m = re.match(r"^jni/([^/]+)/(.*\.so)$", info.filename)
-            if not m:
-                continue
-            arch, lib = m.groups()
-            # report per architecture library size
-            yield [arch, lib, info.compress_size, info.file_size]
-
-        # report whole package size
-        yield ["aar", aar_file.name, os.stat(aar_file).st_size, 0]
-
-    def gen_messages():
-        android_build_type = os.environ.get("ANDROID_BUILD_TYPE")
-        for arch, lib, comp_size, uncomp_size in gen_sizes():
-            print(android_build_type, arch, lib, comp_size, uncomp_size)
-            yield {
-                "normal": {
-                    "os": "android",
-                    # TODO: create dedicated columns
-                    "pkg_type": "{}/{}/{}".format(android_build_type, arch, lib),
-                    "cu_ver": "",  # dummy value for derived field `build_name`
-                    "py_ver": "",  # dummy value for derived field `build_name`
-                    "pr": os.environ.get("CIRCLE_PR_NUMBER"),
-                    "build_num": os.environ.get("CIRCLE_BUILD_NUM"),
-                    "sha1": os.environ.get("CIRCLE_SHA1"),
-                    "branch": os.environ.get("CIRCLE_BRANCH"),
-                },
-                "int": {
-                    "time": int(time.time()),
-                    "commit_time": int(os.environ.get("COMMIT_TIME", "0")),
-                    "size": comp_size,
-                    "raw_size": uncomp_size,
-                },
-            }
-
-    send_message(list(gen_messages()))
-
-
-if __name__ == "__main__":
-    file_dir = os.environ.get(
-        "PYTORCH_FINAL_PACKAGE_DIR", "/home/circleci/project/final_pkgs"
-    )
-    if len(sys.argv) == 2:
-        file_dir = sys.argv[1]
-    print("checking dir: " + file_dir)
-
-    if "-android" in os.environ.get("BUILD_ENVIRONMENT", ""):
-        report_android_sizes(file_dir)
-    else:
-        size = get_size(file_dir)
-        if size != 0:
-            try:
-                send_message([build_message(size)])
-            except:
-                logging.exception("can't send message")

.circleci/scripts/vs_install.ps1

@@ -1,34 +0,0 @@
-$VS_DOWNLOAD_LINK = "https://aka.ms/vs/15/release/vs_buildtools.exe"
-$COLLECT_DOWNLOAD_LINK = "https://aka.ms/vscollect.exe"
-$VS_INSTALL_ARGS = @("--nocache","--quiet","--wait", "--add Microsoft.VisualStudio.Workload.VCTools",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Tools.14.11",
-                                                     "--add Microsoft.Component.MSBuild",
-                                                     "--add Microsoft.VisualStudio.Component.Roslyn.Compiler",
-                                                     "--add Microsoft.VisualStudio.Component.TextTemplating",
-                                                     "--add Microsoft.VisualStudio.Component.VC.CoreIde",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
-                                                     "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
-                                                     "--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
-                                                     "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Win81")
-
-curl.exe --retry 3 -kL $VS_DOWNLOAD_LINK --output vs_installer.exe
-if ($LASTEXITCODE -ne 0) {
-    echo "Download of the VS 2017 installer failed"
-    exit 1
-}
-
-$process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_INSTALL_ARGS -NoNewWindow -Wait -PassThru
-Remove-Item -Path vs_installer.exe -Force
-$exitCode = $process.ExitCode
-if (($exitCode -ne 0) -and ($exitCode -ne 3010)) {
-    echo "VS 2017 installer exited with code $exitCode, which should be one of [0, 3010]."
-    curl.exe --retry 3 -kL $COLLECT_DOWNLOAD_LINK --output Collect.exe
-    if ($LASTEXITCODE -ne 0) {
-        echo "Download of the VS Collect tool failed."
-        exit 1
-    }
-    Start-Process "${PWD}\Collect.exe" -NoNewWindow -Wait -PassThru
-    New-Item -Path "C:\w\build-results" -ItemType "directory" -Force
-    Copy-Item -Path "C:\Users\circleci\AppData\Local\Temp\vslogs.zip" -Destination "C:\w\build-results\"
-    exit 1
-}

.circleci/scripts/windows_cuda_install.sh

@@ -1,37 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-
-curl --retry 3 -kLO https://ossci-windows.s3.amazonaws.com/cuda_10.1.243_426.00_win10.exe
-7z x cuda_10.1.243_426.00_win10.exe -ocuda_10.1.243_426.00_win10
-cd cuda_10.1.243_426.00_win10
-mkdir cuda_install_logs
-
-set +e
-
-./setup.exe -s nvcc_10.1 cuobjdump_10.1 nvprune_10.1 cupti_10.1 cublas_10.1 cublas_dev_10.1 cudart_10.1 cufft_10.1 cufft_dev_10.1 curand_10.1 curand_dev_10.1 cusolver_10.1 cusolver_dev_10.1 cusparse_10.1 cusparse_dev_10.1 nvgraph_10.1 nvgraph_dev_10.1 npp_10.1 npp_dev_10.1 nvrtc_10.1 nvrtc_dev_10.1 nvml_dev_10.1 -loglevel:6 -log:"$(pwd -W)/cuda_install_logs"
-
-set -e
-
-if [[ "${VC_YEAR}" == "2017" ]]; then
-    cp -r CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions/* "C:/Program Files (x86)/Microsoft Visual Studio/2017/${VC_PRODUCT}/Common7/IDE/VC/VCTargets/BuildCustomizations/"
-else
-    cp -r CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions/* "C:/Program Files (x86)/Microsoft Visual Studio/2019/${VC_PRODUCT}/MSBuild/Microsoft/VC/v160/BuildCustomizations/"
-fi
-
-curl --retry 3 -kLO https://ossci-windows.s3.amazonaws.com/NvToolsExt.7z
-7z x NvToolsExt.7z -oNvToolsExt
-mkdir -p "C:/Program Files/NVIDIA Corporation/NvToolsExt"
-cp -r NvToolsExt/* "C:/Program Files/NVIDIA Corporation/NvToolsExt/"
-export NVTOOLSEXT_PATH="C:\\Program Files\\NVIDIA Corporation\\NvToolsExt\\"
-
-if ! ls "/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin/nvcc.exe"
-then
-    echo "CUDA installation failed"
-    mkdir -p /c/w/build-results
-    7z a "c:\\w\\build-results\\cuda_install_logs.7z" cuda_install_logs
-    exit 1
-fi
-
-cd ..
-rm -rf ./cuda_10.1.243_426.00_win10
-rm -f ./cuda_10.1.243_426.00_win10.exe

.circleci/validate-docker-version.py

@@ -1,45 +0,0 @@
-#!/usr/bin/env python3
-import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
-import cimodel.data.simple.util.docker_constants as pytorch_docker_constants
-
-from yaml import load
-
-try:
-    from yaml import CLoader as Loader
-except ImportError:
-    from yaml import Loader
-
-
-def load_config(filename=".circleci/config.yml"):
-    with open(filename, "r") as fh:
-        return load("".join(fh.readlines()), Loader)
-
-
-def load_tags_for_projects(workflow_config):
-    return {
-        v["ecr_gc_job"]["project"]: v["ecr_gc_job"]["tags_to_keep"]
-        for v in workflow_config["workflows"]["ecr_gc"]["jobs"]
-        if isinstance(v, dict) and "ecr_gc_job" in v
-    }
-
-
-def check_version(job, tags, expected_version):
-    valid_versions = tags[job].split(",")
-    if expected_version not in valid_versions:
-        raise RuntimeError(
-            "We configured {} to use Docker version {}; but this "
-            "version is not configured in job ecr_gc_job_for_{}.  Non-deployed versions will be "
-            "garbage collected two weeks after they are created.  DO NOT LAND "
-            "THIS TO MASTER without also updating ossci-job-dsl with this version."
-            "\n\nDeployed versions: {}".format(job, expected_version, job, tags[job])
-        )
-
-
-def validate_docker_version():
-    tags = load_tags_for_projects(load_config())
-    check_version("pytorch", tags, pytorch_docker_constants.DOCKER_IMAGE_TAG)
-    check_version("caffe2", tags, caffe2_build_definitions.DOCKER_IMAGE_VERSION)
-
-
-if __name__ == "__main__":
-    validate_docker_version()

.circleci/verbatim-sources/build-parameters/binary-build-params.yml

@@ -1,66 +0,0 @@
-binary_linux_build_params: &binary_linux_build_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    libtorch_variant:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "2xlarge+"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    LIBTORCH_VARIANT: << parameters.libtorch_variant >>
-    ANACONDA_USER: pytorch
-  resource_class: << parameters.resource_class >>
-  docker:
-    - image: << parameters.docker_image >>
-
-binary_linux_test_upload_params: &binary_linux_test_upload_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    libtorch_variant:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "medium"
-    use_cuda_docker_runtime:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    DOCKER_IMAGE: << parameters.docker_image >>
-    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
-    LIBTORCH_VARIANT: << parameters.libtorch_variant >>
-  resource_class: << parameters.resource_class >>
-
-binary_mac_params: &binary_mac_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-
-binary_windows_params: &binary_windows_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    executor:
-      type: string
-      default: "windows-cpu-with-nvidia-cuda"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    BUILD_FOR_SYSTEM: windows
-    JOB_EXECUTOR: <<parameters.executor>>

.circleci/verbatim-sources/build-parameters/caffe2-build-params.yml

@@ -1,27 +0,0 @@
-caffe2_params: &caffe2_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    build_ios:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    use_cuda_docker_runtime:
-      type: string
-      default: ""
-    build_only:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "large"
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    BUILD_IOS: << parameters.build_ios >>
-    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
-    DOCKER_IMAGE: << parameters.docker_image >>
-    BUILD_ONLY: << parameters.build_only >>
-  resource_class: << parameters.resource_class >>

.circleci/verbatim-sources/build-parameters/promote-build-params.yml

@@ -1,14 +0,0 @@
-
-promote_common: &promote_common
-  docker:
-    - image: pytorch/release
-  parameters:
-    package_name:
-      description: "package name to promote"
-      type: string
-      default: ""
-  environment:
-    PACKAGE_NAME: << parameters.package_name >>
-    ANACONDA_API_TOKEN: ${CONDA_PYTORCHBOT_TOKEN}
-    AWS_ACCESS_KEY_ID: ${PYTORCH_BINARY_AWS_ACCESS_KEY_ID}
-    AWS_SECRET_ACCESS_KEY: ${PYTORCH_BINARY_AWS_SECRET_ACCESS_KEY}

.circleci/verbatim-sources/build-parameters/pytorch-build-params.yml

@@ -1,85 +0,0 @@
-pytorch_params: &pytorch_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    docker_image:
-      type: string
-      default: ""
-    resource_class:
-      type: string
-      default: "large"
-    use_cuda_docker_runtime:
-      type: string
-      default: ""
-    build_only:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    DOCKER_IMAGE: << parameters.docker_image >>
-    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
-    BUILD_ONLY: << parameters.build_only >>
-  resource_class: << parameters.resource_class >>
-
-pytorch_ios_params: &pytorch_ios_params
-  parameters:
-    build_environment:
-      type: string
-      default: ""
-    ios_arch:
-      type: string
-      default: ""
-    ios_platform:
-      type: string
-      default: ""
-    op_list:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: << parameters.build_environment >>
-    IOS_ARCH: << parameters.ios_arch >>
-    IOS_PLATFORM: << parameters.ios_platform >>
-    SELECTED_OP_LIST: << parameters.op_list >>
-
-pytorch_windows_params: &pytorch_windows_params
-  parameters:
-    executor:
-      type: string
-      default: "windows-cpu-with-nvidia-cuda"
-    build_environment:
-      type: string
-      default: ""
-    test_name:
-      type: string
-      default: ""
-    cuda_version:
-      type: string
-      default: "10"
-    python_version:
-      type: string
-      default: "3.6"
-    vc_version:
-      type: string
-      default: "14.11"
-    vc_year:
-      type: string
-      default: "2017"
-    vc_product:
-      type: string
-      default: "BuildTools"
-    use_cuda:
-      type: string
-      default: ""
-  environment:
-    BUILD_ENVIRONMENT: <<parameters.build_environment>>
-    SCCACHE_BUCKET: "ossci-compiler-cache"
-    CUDA_VERSION: <<parameters.cuda_version>>
-    PYTHON_VERSION: <<parameters.python_version>>
-    VC_VERSION: <<parameters.vc_version>>
-    VC_YEAR: <<parameters.vc_year>>
-    VC_PRODUCT: <<parameters.vc_product>>
-    USE_CUDA: <<parameters.use_cuda>>
-    TORCH_CUDA_ARCH_LIST: "7.5"
-    JOB_BASE_NAME: <<parameters.test_name>>
-    JOB_EXECUTOR: <<parameters.executor>>

.circleci/verbatim-sources/commands.yml

@@ -1,133 +0,0 @@
-commands:
-
-  # Must be run after attaching workspace from previous steps
-  load_shared_env:
-    description: "Loads .circleci/shared/env_file into ${BASH_ENV}"
-    parameters:
-      # For some weird reason we decide to reattach our workspace to ~/workspace so
-      # in the vein of making it simple let's assume our share env_file is here
-      root:
-        type: string
-        default: "~/workspace"
-    steps:
-      - run:
-          name: "Load .circleci/shared/env_file into ${BASH_ENV}"
-          command: |
-            if [[ -f  "<< parameters.root >>/.circleci/shared/env_file" ]]; then
-              cat << parameters.root >>/.circleci/shared/env_file >> ${BASH_ENV}
-            else
-              echo "We didn't have a shared env file, that's weird"
-            fi
-
-  # This system setup script is meant to run before the CI-related scripts, e.g.,
-  # installing Git client, checking out code, setting up CI env, and
-  # building/testing.
-  setup_linux_system_environment:
-    steps:
-      - run:
-          name: Set Up System Environment
-          no_output_timeout: "1h"
-          command: .circleci/scripts/setup_linux_system_environment.sh
-
-  setup_ci_environment:
-    steps:
-      - run:
-          name: Set Up CI Environment After attach_workspace
-          no_output_timeout: "1h"
-          command: .circleci/scripts/setup_ci_environment.sh
-
-  brew_update:
-    description: "Update Homebrew and install base formulae"
-    steps:
-      - run:
-          name: Update Homebrew
-          no_output_timeout: "10m"
-          command: |
-            set -ex
-
-            # Update repositories manually.
-            # Running `brew update` produces a comparison between the
-            # current checkout and the updated checkout, which takes a
-            # very long time because the existing checkout is 2y old.
-            for path in $(find /usr/local/Homebrew -type d -name .git)
-            do
-            cd $path/..
-            git fetch --depth=1 origin
-            git reset --hard origin/master
-            done
-
-            export HOMEBREW_NO_AUTO_UPDATE=1
-
-            # Install expect and moreutils so that we can call `unbuffer` and `ts`.
-            # moreutils installs a `parallel` executable by default, which conflicts
-            # with the executable from the GNU `parallel`, so we must unlink GNU
-            # `parallel` first, and relink it afterwards.
-            brew unlink parallel
-            brew install moreutils
-            brew link parallel --overwrite
-            brew install expect
-
-  brew_install:
-    description: "Install Homebrew formulae"
-    parameters:
-      formulae:
-        type: string
-        default: ""
-    steps:
-      - run:
-          name: Install << parameters.formulae >>
-          no_output_timeout: "10m"
-          command: |
-            set -ex
-            export HOMEBREW_NO_AUTO_UPDATE=1
-            brew install << parameters.formulae >>
-
-  run_brew_for_macos_build:
-    steps:
-      - brew_update
-      - brew_install:
-          formulae: libomp
-
-  run_brew_for_ios_build:
-    steps:
-      - brew_update
-      - brew_install:
-          formulae: libtool
-
-  optional_merge_target_branch:
-    steps:
-      - run:
-          name: (Optional) Merge target branch
-          no_output_timeout: "10m"
-          command: |
-            if [ -n "$CIRCLE_PULL_REQUEST" ]; then
-              PR_NUM=$(basename $CIRCLE_PULL_REQUEST)
-              CIRCLE_PR_BASE_BRANCH=$(curl -s https://api.github.com/repos/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/pulls/$PR_NUM | jq -r '.base.ref')
-              if [[ "${BUILD_ENVIRONMENT}" == *"xla"* || "${BUILD_ENVIRONMENT}" == *"gcc5"* ]] ; then
-                set -x
-                git config --global user.email "circleci.ossci@gmail.com"
-                git config --global user.name "CircleCI"
-                git config remote.origin.url https://github.com/pytorch/pytorch.git
-                git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master
-                git fetch --tags --progress https://github.com/pytorch/pytorch.git +refs/heads/master:refs/remotes/origin/master --depth=100 --quiet
-                # PRs generated from ghstack has format CIRCLE_PR_BASE_BRANCH=gh/xxx/1234/base
-                if [[ "${CIRCLE_PR_BASE_BRANCH}" == "gh/"* ]]; then
-                  CIRCLE_PR_BASE_BRANCH=master
-                fi
-                export GIT_MERGE_TARGET=`git log -n 1 --pretty=format:"%H" origin/$CIRCLE_PR_BASE_BRANCH`
-                echo "GIT_MERGE_TARGET: " ${GIT_MERGE_TARGET}
-                export GIT_COMMIT=${CIRCLE_SHA1}
-                echo "GIT_COMMIT: " ${GIT_COMMIT}
-                git checkout -f ${GIT_COMMIT}
-                git reset --hard ${GIT_COMMIT}
-                git merge --allow-unrelated-histories --no-edit --no-ff ${GIT_MERGE_TARGET}
-                echo "Merged $CIRCLE_PR_BASE_BRANCH branch before building in environment $BUILD_ENVIRONMENT"
-                set +x
-              else
-                echo "No need to merge with $CIRCLE_PR_BASE_BRANCH, skipping..."
-              fi
-            else
-              echo "This is not a pull request, skipping..."
-            fi
-
-

.circleci/verbatim-sources/header-section.yml

@@ -1,34 +0,0 @@
-# WARNING: DO NOT EDIT THIS FILE DIRECTLY!!!
-# See the README.md in this directory.
-
-# IMPORTANT: To update Docker image version, please follow
-# the instructions at
-# https://github.com/pytorch/pytorch/wiki/Docker-image-build-on-CircleCI
-
-version: 2.1
-
-parameters:
-  run_binary_tests:
-    type: boolean
-    default: false
-
-docker_config_defaults: &docker_config_defaults
-  user: jenkins
-  aws_auth:
-    # This IAM user only allows read-write access to ECR
-    aws_access_key_id: ${CIRCLECI_AWS_ACCESS_KEY_FOR_ECR_READ_WRITE_V4}
-    aws_secret_access_key: ${CIRCLECI_AWS_SECRET_KEY_FOR_ECR_READ_WRITE_V4}
-
-executors:
-  windows-with-nvidia-gpu:
-    machine:
-      resource_class: windows.gpu.nvidia.medium
-      image: windows-server-2019-nvidia:stable
-      shell: bash.exe
-
-  windows-cpu-with-nvidia-cuda:
-    machine:
-      # we will change to CPU host when it's ready
-      resource_class: windows.xlarge
-      image: windows-server-2019-vs2019:stable
-      shell: bash.exe