Enabling inplace relu

Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/28710

Test Plan: Imported from OSS

Differential Revision: D18146120

Pulled By: z-a-f

fbshipit-source-id: d8f0982f5a2ae35f7deb34e67cdb64be700a9d6c

.circleci/README.md

@@ -1,3 +1,23 @@
+Structure of CI
+===============
+
+setup job:
+1. Does a git checkout
+2. Persists CircleCI scripts (everything in `.circleci`) into a workspace.  Why?
+   We don't always do a Git checkout on all subjobs, but we usually
+   still want to be able to call scripts one way or another in a subjob.
+   Persisting files this way lets us have access to them without doing a
+   checkout.  This workspace is conventionally mounted on `~/workspace`
+   (this is distinguished from `~/project`, which is the conventional
+   working directory that CircleCI will default to starting your jobs
+   in.)
+3. Write out the commit message to `.circleci/COMMIT_MSG`.  This is so
+   we can determine in subjobs if we should actually run the jobs or
+   not, even if there isn't a Git checkout.
+
+
+
+
 CircleCI configuration generator
 ================================
 
@@ -35,4 +55,450 @@ Future direction
 See comment [here](https://github.com/pytorch/pytorch/pull/17323#pullrequestreview-206945747):
 
 In contrast with a full recursive tree traversal of configuration dimensions,
-> in the future future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
+> in the future future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
+
+----------------
+----------------
+
+# How do the binaries / nightlies / releases work?
+
+### What is a binary?
+
+A binary or package (used interchangeably) is a pre-built collection of c++ libraries, header files, python bits, and other files. We build these and distribute them so that users do not need to install from source.
+
+A **binary configuration** is a collection of
+
+* release or nightly
+    * releases are stable, nightlies are beta and built every night
+* python version
+    * linux: 2.7m, 2.7mu, 3.5m, 3.6m 3.7m (mu is wide unicode or something like that. It usually doesn't matter but you should know that it exists)
+    * macos: 2.7, 3.5, 3.6, 3.7
+    * windows: 3.5, 3.6, 3.7
+* cpu version
+    * cpu, cuda 9.0, cuda 10.0
+    * The supported cuda versions occasionally change
+* operating system
+    * Linux - these are all built on CentOS. There haven't been any problems in the past building on CentOS and using on Ubuntu
+    * MacOS
+    * Windows - these are built on Azure pipelines
+* devtoolset version (gcc compiler version)
+    * This only matters on Linux cause only Linux uses gcc. tldr is gcc made a backwards incompatible change from gcc 4.8 to gcc 5, because it had to change how it implemented std::vector and std::string
+
+### Where are the binaries?
+
+The binaries are built in CircleCI. There are nightly binaries built every night at 9pm PST (midnight EST) and release binaries corresponding to Pytorch releases, usually every few months.
+
+We have 3 types of binary packages
+
+* pip packages - nightlies are stored on s3 (pip install -f <a s3 url>). releases are stored in a pip repo (pip install torch) (ask Soumith about this)
+* conda packages - nightlies and releases are both stored in a conda repo. Nighty packages have a '_nightly' suffix
+* libtorch packages - these are zips of all the c++ libraries, header files, and sometimes dependencies. These are c++ only
+    * shared with dependencies (the only supported option for Windows)
+    * static with dependencies
+    * shared without dependencies
+    * static without dependencies
+
+All binaries are built in CircleCI workflows except Windows. There are checked-in workflows (committed into the .circleci/config.yml) to build the nightlies every night. Releases are built by manually pushing a PR that builds the suite of release binaries (overwrite the config.yml to build the release)
+
+# CircleCI structure of the binaries
+
+Some quick vocab:
+
+* A\**workflow** is a CircleCI concept; it is a DAG of '**jobs**'. ctrl-f 'workflows' on\https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml to see the workflows.
+* **jobs** are a sequence of '**steps**'
+* **steps** are usually just a bash script or a builtin CircleCI command.* All steps run in new environments, environment variables declared in one script DO NOT persist to following steps*
+* CircleCI has a **workspace**, which is essentially a cache between steps of the *same job* in which you can store artifacts between steps.
+
+## How are the workflows structured?
+
+The nightly binaries have 3 workflows. We have one job (actually 3 jobs:  build, test, and upload) per binary configuration
+
+1. binarybuilds
+    1. every day midnight EST
+    2. linux: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/linux-binary-build-defaults.yml
+    3. macos: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/macos-binary-build-defaults.yml
+    4. For each binary configuration, e.g. linux_conda_3.7_cpu there is a
+        1. binary_linux_conda_3.7_cpu_build
+            1. Builds the build. On linux jobs this uses the 'docker executor'.
+            2. Persists the package to the workspace
+        2. binary_linux_conda_3.7_cpu_test
+            1. Loads the package to the workspace
+            2. Spins up a docker image (on Linux), mapping the package and code repos into the docker
+            3. Runs some smoke tests in the docker
+            4. (Actually, for macos this is a step rather than a separate job)
+        3. binary_linux_conda_3.7_cpu_upload
+            1. Logs in to aws/conda
+            2. Uploads the package
+2. update_s3_htmls
+    1. every day 5am EST
+    2. https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/binary_update_htmls.yml
+    3. See below for what these are for and why they're needed
+    4. Three jobs that each examine the current contents of aws and the conda repo and update some html files in s3
+3. binarysmoketests
+    1. every day
+    2. https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/nightly-build-smoke-tests-defaults.yml
+    3. For each binary configuration, e.g. linux_conda_3.7_cpu there is a
+        1. smoke_linux_conda_3.7_cpu
+            1. Downloads the package from the cloud, e.g. using the official pip or conda instructions
+            2. Runs the smoke tests
+
+## How are the jobs structured?
+
+The jobs are in https://github.com/pytorch/pytorch/tree/master/.circleci/verbatim-sources . Jobs are made of multiple steps. There are some shared steps used by all the binaries/smokes. Steps of these jobs are all delegated to scripts in https://github.com/pytorch/pytorch/tree/master/.circleci/scripts .
+
+* Linux jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/linux-binary-build-defaults.yml
+    * binary_linux_build.sh
+    * binary_linux_test.sh
+    * binary_linux_upload.sh
+* MacOS jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/macos-binary-build-defaults.yml
+    * binary_macos_build.sh
+    * binary_macos_test.sh
+    * binary_macos_upload.sh
+* Update html jobs: https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/binary_update_htmls.yml
+    * These delegate from the pytorch/builder repo
+    * https://github.com/pytorch/builder/blob/master/cron/update_s3_htmls.sh
+    * https://github.com/pytorch/builder/blob/master/cron/upload_binary_sizes.sh
+* Smoke jobs (both linux and macos): https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/nightly-build-smoke-tests-defaults.yml
+    * These delegate from the pytorch/builder repo
+    * https://github.com/pytorch/builder/blob/master/run_tests.sh
+    * https://github.com/pytorch/builder/blob/master/smoke_test.sh
+    * https://github.com/pytorch/builder/blob/master/check_binary.sh
+* Common shared code (shared across linux and macos): https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/nightly-binary-build-defaults.yml
+    * binary_checkout.sh - checks out pytorch/builder repo. Right now this also checks out pytorch/pytorch, but it shouldn't. pytorch/pytorch should just be shared through the workspace. This can handle being run before binary_populate_env.sh
+    * binary_populate_env.sh - parses BUILD_ENVIRONMENT into the separate env variables that make up a binary configuration. Also sets lots of default values, the date, the version strings, the location of folders in s3, all sorts of things. This generally has to be run before other steps.
+    * binary_install_miniconda.sh - Installs miniconda, cross platform. Also hacks this for the update_binary_sizes job that doesn't have the right env variables
+    * binary_run_in_docker.sh - Takes a bash script file (the actual test code) from a hardcoded location, spins up a docker image, and runs the script inside the docker image
+
+### **Why do the steps all refer to scripts?**
+
+CircleCI creates a  final yaml file by inlining every <<* segment, so if we were to keep all the code in the config.yml itself then the config size would go over 4 MB and cause infra problems.
+
+### **What is binary_run_in_docker for?**
+
+So, CircleCI has several executor types: macos, machine, and docker are the ones we use. The 'machine' executor gives you two cores on some linux vm. The 'docker' executor gives you considerably more cores (nproc was 32 instead of 2 back when I tried in February). Since the dockers are faster, we try to run everything that we can in dockers. Thus
+
+* linux build jobs use the docker executor. Running them on the docker executor was at least 2x faster than running them on the machine executor
+* linux test jobs use the machine executor and spin up their own docker. Why this nonsense? It's cause we run nvidia-docker for our GPU tests; any code that calls into the CUDA runtime needs to be run on nvidia-docker. To run a nvidia-docker you need to install some nvidia packages on the host machine and then call docker with the '—runtime nvidia' argument. CircleCI doesn't support this, so we have to do it ourself.
+    * This is not just a mere inconvenience. **This blocks all of our linux tests from using more than 2 cores.** But there is nothing that we can do about it, but wait for a fix on circleci's side. Right now, we only run some smoke tests (some simple imports) on the binaries, but this also affects non-binary test jobs.
+* linux upload jobs use the machine executor. The upload jobs are so short that it doesn't really matter what they use
+* linux smoke test jobs use the machine executor for the same reason as the linux test jobs
+
+binary_run_in_docker.sh is a way to share the docker start-up code between the binary test jobs and the binary smoke test jobs
+
+### **Why does binary_checkout also checkout pytorch? Why shouldn't it?**
+
+We want all the nightly binary jobs to run on the exact same git commit, so we wrote our own checkout logic to ensure that the same commit was always picked. Later circleci changed that to use a single pytorch checkout and persist it through the workspace (they did this because our config file was too big, so they wanted to take a lot of the setup code into scripts, but the scripts needed the code repo to exist to be called, so they added a prereq step called 'setup' to checkout the code and persist the needed scripts to the workspace). The changes to the binary jobs were not properly tested, so they all broke from missing pytorch code no longer existing. We hotfixed the problem by adding the pytorch checkout back to binary_checkout, so now there's two checkouts of pytorch on the binary jobs. This problem still needs to be fixed, but it takes careful tracing of which code is being called where.
+
+# Azure Pipelines structure of the binaries
+
+TODO: fill in stuff
+
+## How are the workflows structured?
+
+TODO: fill in stuff
+
+## How are the jobs structured?
+
+TODO: fill in stuff
+
+# Code structure of the binaries (circleci agnostic)
+
+## Overview
+
+The code that runs the binaries lives in two places, in the normal [github.com/pytorch/pytorch](http://github.com/pytorch/pytorch), but also in [github.com/pytorch/builder](http://github.com/pytorch/builder) , which is a repo that defines how all the binaries are built. The relevant code is
+
+
+```
+# All code needed to set-up environments for build code to run in,
+# but only code that is specific to the current CI system
+pytorch/pytorch
+- .circleci/                # Folder that holds all circleci related stuff
+  - config.yml              # GENERATED file that actually controls all circleci behavior
+  - verbatim-sources        # Used to generate job/workflow sections in ^
+  - scripts/                # Code needed to prepare circleci environments for binary build scripts
+
+- setup.py                  # Builds pytorch. This is wrapped in pytorch/builder
+- cmake files               # used in normal building of pytorch
+
+# All code needed to prepare a binary build, given an environment
+# with all the right variables/packages/paths.
+pytorch/builder
+
+# Given an installed binary and a proper python env, runs some checks
+# to make sure the binary was built the proper way. Checks things like
+# the library dependencies, symbols present, etc.
+- check_binary.sh
+
+# Given an installed binary, runs python tests to make sure everything
+# is in order. These should be de-duped. Right now they both run smoke
+# tests, but are called from different places. Usually just call some
+# import statements, but also has overlap with check_binary.sh above
+- run_tests.sh
+- smoke_test.sh
+
+# Folders that govern how packages are built. See paragraphs below
+
+- conda/
+  - build_pytorch.sh          # Entrypoint. Delegates to proper conda build folder
+  - switch_cuda_version.sh    # Switches activate CUDA installation in Docker
+  - pytorch-nightly/          # Build-folder
+- manywheel/
+  - build_cpu.sh              # Entrypoint for cpu builds
+  - build.sh                  # Entrypoint for CUDA builds
+  - build_common.sh           # Actual build script that ^^ call into
+- wheel/
+  - build_wheel.sh            # Entrypoint for wheel builds
+- windows/
+  - build_pytorch.bat         # Entrypoint for wheel builds on Windows
+```
+
+Every type of package has an entrypoint build script that handles the all the important logic.
+
+## Conda
+
+Linux, MacOS and Windows use the same code flow for the conda builds.
+
+Conda packages are built with conda-build, see https://conda.io/projects/conda-build/en/latest/resources/commands/conda-build.html
+
+Basically, you pass `conda build` a build folder (pytorch-nightly/ above) that contains a build script and a meta.yaml. The meta.yaml specifies in what python environment to build the package in, and what dependencies the resulting package should have, and the build script gets called in the env to build the thing.
+tldr; on conda-build is
+
+1. Creates a brand new conda environment, based off of deps in the meta.yaml
+    1. Note that environment variables do not get passed into this build env unless they are specified in the meta.yaml
+    2. If the build fails this environment will stick around. You can activate it for much easier debugging. The “General Python” section below explains what exactly a python “environment” is.
+2. Calls build.sh in the environment
+3. Copies the finished package to a new conda env, also specified by the meta.yaml
+4. Runs some simple import tests (if specified in the meta.yaml)
+5. Saves the finished package as a tarball
+
+The build.sh we use is essentially a wrapper around ```python setup.py build``` , but it also manually copies in some of our dependent libraries into the resulting tarball and messes with some rpaths.
+
+The entrypoint file `builder/conda/build_conda.sh` is complicated because
+
+* It works for Linux, MacOS and Windows
+    * The mac builds used to create their own environments, since they all used to be on the same machine. There’s now a lot of extra logic to handle conda envs. This extra machinery could be removed
+* It used to handle testing too, which adds more logic messing with python environments too. This extra machinery could be removed.
+
+## Manywheels (linux pip and libtorch packages)
+
+Manywheels are pip packages for linux distros. Note that these manywheels are not actually manylinux compliant.
+
+`builder/manywheel/build_cpu.sh` and `builder/manywheel/build.sh` (for CUDA builds) just set different env vars and then call into `builder/manywheel/build_common.sh`
+
+The entrypoint file `builder/manywheel/build_common.sh` is really really complicated because
+
+* This used to handle building for several different python versions at the same time. The loops have been removed, but there's still unnecessary folders and movements here and there.
+    * The script is never used this way anymore. This extra machinery could be removed.
+* This used to handle testing the pip packages too. This is why there’s testing code at the end that messes with python installations and stuff
+    * The script is never used this way anymore. This extra machinery could be removed.
+* This also builds libtorch packages
+    * This should really be separate. libtorch packages are c++ only and have no python. They should not share infra with all the python specific stuff in this file.
+* There is a lot of messing with rpaths. This is necessary, but could be made much much simpler if the above issues were fixed.
+
+## Wheels (MacOS pip and libtorch packages)
+
+The entrypoint file `builder/wheel/build_wheel.sh` is complicated because
+
+* The mac builds used to all run on one machine (we didn’t have autoscaling mac machines till circleci). So this script handled siloing itself by setting-up and tearing-down its build env and siloing itself into its own build directory.
+    * The script is never used this way anymore. This extra machinery could be removed.
+* This also builds libtorch packages
+    * Ditto the comment above. This should definitely be separated out.
+
+Note that the MacOS Python wheels are still built in conda environments. Some of the dependencies present during build also come from conda.
+
+## Windows Wheels (Windows pip and libtorch packages)
+
+The entrypoint file `builder/windows/build_pytorch.bat` is complicated because
+
+* This used to handle building for several different python versions at the same time. This is why there are loops everywhere
+    * The script is never used this way anymore. This extra machinery could be removed.
+* This used to handle testing the pip packages too. This is why there’s testing code at the end that messes with python installations and stuff
+    * The script is never used this way anymore. This extra machinery could be removed.
+* This also builds libtorch packages
+    * This should really be separate. libtorch packages are c++ only and have no python. They should not share infra with all the python specific stuff in this file.
+
+Note that the Windows Python wheels are still built in conda environments. Some of the dependencies present during build also come from conda.
+
+## General notes
+
+### Note on run_tests.sh, smoke_test.sh, and check_binary.sh
+
+* These should all be consolidated
+* These must run on all OS types: MacOS, Linux, and Windows
+* These all run smoke tests at the moment. They inspect the packages some, maybe run a few import statements. They DO NOT run the python tests nor the cpp tests. The idea is that python tests on master and PR merges will catch all breakages. All these tests have to do is make sure the special binary machinery didn’t mess anything up.
+* There are separate run_tests.sh and smoke_test.sh because one used to be called by the smoke jobs and one used to be called by the binary test jobs (see circleci structure section above). This is still true actually, but these could be united into a single script that runs these checks, given an installed pytorch package.
+
+### Note on libtorch
+
+Libtorch packages are built in the wheel build scripts: manywheel/build_*.sh for linux and build_wheel.sh for mac. There are several things wrong with this
+
+* It’s confusing. Most of those scripts deal with python specifics.
+* The extra conditionals everywhere severely complicate the wheel build scripts
+* The process for building libtorch is different from the official instructions (a plain call to cmake, or a call to a script)
+
+### Note on docker images / Dockerfiles
+
+All linux builds occur in docker images. The docker images are
+
+* soumith/conda-cuda
+    * Has ALL CUDA versions installed. The script pytorch/builder/conda/switch_cuda_version.sh sets /usr/local/cuda to a symlink to e.g. /usr/local/cuda-10.0 to enable different CUDA builds
+    * Also used for cpu builds
+* soumith/manylinux-cuda90
+* soumith/manylinux-cuda92
+* soumith/manylinux-cuda100
+    * Also used for cpu builds
+
+The Dockerfiles are available in pytorch/builder, but there is no circleci job or script to build these docker images, and they cannot be run locally (unless you have the correct local packages/paths). Only Soumith can build them right now.
+
+### General Python
+
+* This is still a good explanation of python installations https://caffe2.ai/docs/faq.html#why-do-i-get-import-errors-in-python-when-i-try-to-use-caffe2
+
+# How to manually rebuild the binaries
+
+tldr; make a PR that looks like https://github.com/pytorch/pytorch/pull/21159
+
+Sometimes we want to push a change to master and then rebuild all of today's binaries after that change. As of May 30, 2019 there isn't a way to manually run a workflow in the UI. You can manually re-run a workflow, but it will use the exact same git commits as the first run and will not include any changes. So we have to make a PR and then force circleci to run the binary workflow instead of the normal tests. The above PR is an example of how to do this; essentially you copy-paste the binarybuilds workflow steps into the default workflow steps. If you need to point the builder repo to a different commit then you'd need to change https://github.com/pytorch/pytorch/blob/master/.circleci/scripts/binary_checkout.sh#L42-L45 to checkout what you want.
+
+## How to test changes to the binaries via .circleci
+
+Writing PRs that test the binaries is annoying, since the default circleci jobs that run on PRs are not the jobs that you want to run. Likely, changes to the binaries will touch something under .circleci/ and require that .circleci/config.yml be regenerated (.circleci/config.yml controls all .circleci behavior, and is generated using ```.circleci/regenerate.sh``` in python 3.7). But you also need to manually hardcode the binary jobs that you want to test into the .circleci/config.yml workflow, so you should actually make at least two commits, one for your changes and one to temporarily hardcode jobs. See https://github.com/pytorch/pytorch/pull/22928 as an example of how to do this.
+
+```
+# Make your changes
+touch .circleci/verbatim-sources/nightly-binary-build-defaults.yml
+
+# Regenerate the yaml, has to be in python 3.7
+.circleci/regenerate.sh
+
+# Make a commit
+git add .circleci *
+git commit -m "My real changes"
+git push origin my_branch
+
+# Now hardcode the jobs that you want in the .circleci/config.yml workflows section
+# Also eliminate ensure-consistency and should_run_job checks
+# e.g. https://github.com/pytorch/pytorch/commit/2b3344bfed8772fe86e5210cc4ee915dee42b32d
+
+# Make a commit you won't keep
+git add .circleci
+git commit -m "[DO NOT LAND] testing binaries for above changes"
+git push origin my_branch
+
+# Now you need to make some changes to the first commit.
+git rebase -i HEAD~2 # mark the first commit as 'edit'
+
+# Make the changes
+touch .circleci/verbatim-sources/nightly-binary-build-defaults.yml
+.circleci/regenerate.sh
+
+# Ammend the commit and recontinue
+git add .circleci
+git commit --amend
+git rebase --continue
+
+# Update the PR, need to force since the commits are different now
+git push origin my_branch --force
+```
+
+The advantage of this flow is that you can make new changes to the base commit and regenerate the .circleci without having to re-write which binary jobs you want to test on. The downside is that all updates will be force pushes.
+
+## How to build a binary locally
+
+### Linux
+
+You can build Linux binaries locally easily using docker.
+
+```
+# Run the docker
+# Use the correct docker image, soumith/conda-cuda used here as an example
+#
+# -v path/to/foo:path/to/bar makes path/to/foo on your local machine (the
+#    machine that you're running the command on) accessible to the docker
+#    container at path/to/bar. So if you then run `touch path/to/bar/baz`
+#    in the docker container then you will see path/to/foo/baz on your local
+#    machine. You could also clone the pytorch and builder repos in the docker.
+#
+# If you're building a CUDA binary then use `nvidia-docker run` instead, see below.
+#
+# If you know how, add ccache as a volume too and speed up everything
+docker run \
+    -v your/pytorch/repo:/pytorch \
+    -v your/builder/repo:/builder \
+    -v where/you/want/packages/to/appear:/final_pkgs \
+    -it soumith/conda-cuda /bin/bash
+
+# Export whatever variables are important to you. All variables that you'd
+# possibly need are in .circleci/scripts/binary_populate_env.sh
+# You should probably always export at least these 3 variables
+export PACKAGE_TYPE=conda
+export DESIRED_PYTHON=3.6
+export DESIRED_CUDA=cpu
+
+# Call the entrypoint
+# `|& tee foo.log` just copies all stdout and stderr output to foo.log
+# The builds generate lots of output so you probably need this when
+# building locally.
+/builder/conda/build_pytorch.sh |& tee build_output.log
+```
+
+**Building CUDA binaries on docker**
+
+To build a CUDA binary you need to use `nvidia-docker run` instead of just `docker run` (or you can manually pass `--runtime=nvidia`). This adds some needed libraries and things to build CUDA stuff.
+
+You can build CUDA binaries on CPU only machines, but you can only run CUDA binaries on CUDA machines. This means that you can build a CUDA binary on a docker on your laptop if you so choose (though it’s gonna take a loong time).
+
+For Facebook employees, ask about beefy machines that have docker support and use those instead of your laptop; it will be 5x as fast.
+
+### MacOS
+
+There’s no easy way to generate reproducible hermetic MacOS environments. If you have a Mac laptop then you can try emulating the .circleci environments as much as possible, but you probably have packages in /usr/local/, possibly installed by brew, that will probably interfere with the build. If you’re trying to repro an error on a Mac build in .circleci and you can’t seem to repro locally, then my best advice is actually to iterate on .circleci    :/
+
+But if you want to try, then I’d recommend
+
+```
+# Create a new terminal
+# Clear your LD_LIBRARY_PATH and trim as much out of your PATH as you
+# know how to do
+
+# Install a new miniconda
+# First remove any other python or conda installation from your PATH
+# Always install miniconda 3, even if building for Python <3
+new_conda="~/my_new_conda"
+conda_sh="$new_conda/install_miniconda.sh"
+curl -o "$conda_sh" https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+chmod +x "$conda_sh"
+"$conda_sh" -b -p "$MINICONDA_ROOT"
+rm -f "$conda_sh"
+export PATH="~/my_new_conda/bin:$PATH"
+
+# Create a clean python env
+# All MacOS builds use conda to manage the python env and dependencies
+# that are built with, even the pip packages
+conda create -yn binary python=2.7
+conda activate binary
+
+# Export whatever variables are important to you. All variables that you'd
+# possibly need are in .circleci/scripts/binary_populate_env.sh
+# You should probably always export at least these 3 variables
+export PACKAGE_TYPE=conda
+export DESIRED_PYTHON=3.6
+export DESIRED_CUDA=cpu
+
+# Call the entrypoint you want
+path/to/builder/wheel/build_wheel.sh
+```
+
+N.B. installing a brand new miniconda is important. This has to do with how conda installations work. See the “General Python” section above, but tldr; is that
+
+1. You make the ‘conda’ command accessible by prepending `path/to/conda_root/bin` to your PATH.
+2. You make a new env and activate it, which then also gets prepended to your PATH. Now you have `path/to/conda_root/envs/new_env/bin:path/to/conda_root/bin:$PATH`
+3. Now say you (or some code that you ran) call python executable `foo`
+    1. if you installed `foo` in `new_env`, then `path/to/conda_root/envs/new_env/bin/foo` will get called, as expected.
+    2. But if you forgot to installed `foo` in `new_env` but happened to previously install it in your root conda env (called ‘base’), then unix/linux will still find `path/to/conda_root/bin/foo` . This is dangerous, since `foo` can be a different version than you want; `foo` can even be for an incompatible python version!
+
+Newer conda versions and proper python hygiene can prevent this, but just install a new miniconda to be safe.
+
+### Windows
+
+TODO: fill in

.circleci/cimodel/data/binary_build_data.py

@@ -59,11 +59,21 @@ CONFIG_TREE_DATA = OrderedDict(
     )),
 )
 
-
-DEVTOOLSET_VERSIONS = [
-    3,
-    7,
-]
+# GCC config variants:
+#
+# All the nightlies (except libtorch with new gcc ABI) are built with devtoolset7,
+# which can only build with old gcc ABI. It is better than devtoolset3
+# because it understands avx512, which is needed for good fbgemm performance.
+#
+# Libtorch with new gcc ABI is built with gcc 5.4 on Ubuntu 16.04.
+LINUX_GCC_CONFIG_VARIANTS = OrderedDict(
+    manywheel=['devtoolset7'],
+    conda=['devtoolset7'],
+    libtorch=[
+        "devtoolset7",
+        "gcc5.4_cxx11-abi",
+    ],
+)
 
 
 class TopLevelNode(ConfigNode):
@@ -97,24 +107,24 @@ class PackageFormatConfigNode(ConfigNode):
         self.props["package_format"] = package_format
 
     def get_children(self):
-        if self.find_prop("os_name") == "linux" and self.find_prop("package_format") != "conda":
-            return [LinuxGccConfigNode(self, v) for v in DEVTOOLSET_VERSIONS]
+        if self.find_prop("os_name") == "linux":
+            return [LinuxGccConfigNode(self, v) for v in LINUX_GCC_CONFIG_VARIANTS[self.find_prop("package_format")]]
         else:
             return [ArchConfigNode(self, v) for v in self.find_prop("cuda_versions")]
 
 
 class LinuxGccConfigNode(ConfigNode):
-    def __init__(self, parent, devtoolset_version):
-        super(LinuxGccConfigNode, self).__init__(parent, "DEVTOOLSET=" + str(devtoolset_version))
+    def __init__(self, parent, gcc_config_variant):
+        super(LinuxGccConfigNode, self).__init__(parent, "GCC_CONFIG_VARIANT=" + str(gcc_config_variant))
 
-        self.props["devtoolset_version"] = devtoolset_version
+        self.props["gcc_config_variant"] = gcc_config_variant
 
     def get_children(self):
         cuda_versions = self.find_prop("cuda_versions")
 
         # XXX devtoolset7 on CUDA 9.0 is temporarily disabled
         # see https://github.com/pytorch/pytorch/issues/20066
-        if self.find_prop("devtoolset_version") == 7:
+        if self.find_prop("gcc_config_variant") == 'devtoolset7':
             cuda_versions = filter(lambda x: x != "90", cuda_versions)
 
         return [ArchConfigNode(self, v) for v in cuda_versions]
@@ -142,7 +152,7 @@ class PyVersionConfigNode(ConfigNode):
         package_format = self.find_prop("package_format")
         os_name = self.find_prop("os_name")
 
-        has_libtorch_variants = smoke and package_format == "libtorch" and os_name == "linux"
+        has_libtorch_variants = package_format == "libtorch" and os_name == "linux"
         linking_variants = LINKING_DIMENSIONS if has_libtorch_variants else []
 
         return [LinkingVariantConfigNode(self, v) for v in linking_variants]

.circleci/cimodel/data/binary_build_definitions.py

@@ -9,7 +9,7 @@ import cimodel.lib.visualization as visualization
 
 
 class Conf(object):
-    def __init__(self, os, cuda_version, pydistro, parms, smoke, libtorch_variant, devtoolset_version):
+    def __init__(self, os, cuda_version, pydistro, parms, smoke, libtorch_variant, gcc_config_variant):
 
         self.os = os
         self.cuda_version = cuda_version
@@ -17,15 +17,17 @@ class Conf(object):
         self.parms = parms
         self.smoke = smoke
         self.libtorch_variant = libtorch_variant
-        self.devtoolset_version = devtoolset_version
+        self.gcc_config_variant = gcc_config_variant
 
     def gen_build_env_parms(self):
         elems = [self.pydistro] + self.parms + [binary_build_data.get_processor_arch_name(self.cuda_version)]
-        if self.devtoolset_version is not None:
-            elems.append("devtoolset" + str(self.devtoolset_version))
+        if self.gcc_config_variant is not None:
+            elems.append(str(self.gcc_config_variant))
         return elems
 
     def gen_docker_image(self):
+        if self.gcc_config_variant == 'gcc5.4_cxx11-abi':
+            return miniutils.quote("soumith/conda-cuda-cxx11-ubuntu1604:latest")
 
         docker_word_substitution = {
             "manywheel": "manylinux",
@@ -34,8 +36,8 @@ class Conf(object):
 
         docker_distro_prefix = miniutils.override(self.pydistro, docker_word_substitution)
 
-        # The cpu nightlies are built on the soumith/manylinux-cuda80 docker image
-        alt_docker_suffix = self.cuda_version or "80"
+        # The cpu nightlies are built on the soumith/manylinux-cuda100 docker image
+        alt_docker_suffix = self.cuda_version or "100"
         docker_distro_suffix = "" if self.pydistro == "conda" else alt_docker_suffix
         return miniutils.quote("soumith/" + docker_distro_prefix + "-cuda" + docker_distro_suffix)
 
@@ -46,44 +48,45 @@ class Conf(object):
 
         parts = [self.get_name_prefix(), self.os] + self.gen_build_env_parms()
 
-        if self.smoke:
-            if self.libtorch_variant:
-                parts.append(self.libtorch_variant)
-        else:
+        if self.libtorch_variant:
+            parts.append(self.libtorch_variant)
+
+        if not self.smoke:
             parts.append(build_or_test)
 
-        return "_".join(parts)
-
-    def gen_yaml_tree(self, build_or_test):
-
-        env_tuples = [("BUILD_ENVIRONMENT", miniutils.quote(" ".join(self.gen_build_env_parms())))]
+        joined = "_".join(parts)
+        return joined.replace(".", "_")
 
+    def gen_workflow_job(self, phase, upload_phase_dependency=None):
+        job_def = OrderedDict()
+        job_def["name"] = self.gen_build_name(phase)
+        job_def["build_environment"] = miniutils.quote(" ".join(self.gen_build_env_parms()))
+        job_def["requires"] = ["setup"]
+        job_def["filters"] = {"branches": {"only": "nightly"}}
         if self.libtorch_variant:
-            env_tuples.append(("LIBTORCH_VARIANT", miniutils.quote(self.libtorch_variant)))
+            job_def["libtorch_variant"] = miniutils.quote(self.libtorch_variant)
+        if phase == "test":
+            if not self.smoke:
+                job_def["requires"].append(self.gen_build_name("build"))
+            if not (self.smoke and self.os == "macos"):
+                job_def["docker_image"] = self.gen_docker_image()
+
+            if self.cuda_version:
+                job_def["use_cuda_docker_runtime"] = miniutils.quote("1")
+        else:
+            if self.os == "linux" and phase != "upload":
+                job_def["docker_image"] = self.gen_docker_image()
+
+        if phase == "test":
+            if self.cuda_version:
+                job_def["resource_class"] = "gpu.medium"
+        if phase == "upload":
+            job_def["context"] = "org-member"
+            job_def["requires"] = ["setup", self.gen_build_name(upload_phase_dependency)]
 
         os_name = miniutils.override(self.os, {"macos": "mac"})
-        d = {"<<": "*" + "_".join([self.get_name_prefix(), os_name, build_or_test])}
-
-        if build_or_test == "test":
-
-            if not (self.smoke and self.os == "macos"):
-                env_tuples.append(("DOCKER_IMAGE", self.gen_docker_image()))
-
-            if self.cuda_version:
-                env_tuples.append(("USE_CUDA_DOCKER_RUNTIME", miniutils.quote("1")))
-
-        else:
-            if self.os == "linux" and build_or_test != "upload":
-                d["docker"] = [{"image": self.gen_docker_image()}]
-
-        d["environment"] = OrderedDict(env_tuples)
-
-        if build_or_test == "test":
-            if self.cuda_version:
-                d["resource_class"] = "gpu.medium"
-
-        return d
-
+        job_name = "_".join([self.get_name_prefix(), os_name, phase])
+        return {job_name : job_def}
 
 def get_root(smoke, name):
 
@@ -108,7 +111,7 @@ def gen_build_env_list(smoke):
             [c.find_prop("pyver")],
             c.find_prop("smoke"),
             c.find_prop("libtorch_variant"),
-            c.find_prop("devtoolset_version"),
+            c.find_prop("gcc_config_variant"),
         )
         newlist.append(conf)
 
@@ -116,31 +119,17 @@ def gen_build_env_list(smoke):
 
 
 def predicate_exclude_nonlinux_and_libtorch(config):
-    return config.os == "linux" and (config.smoke or config.pydistro != "libtorch")
+    return config.os == "linux"
 
 
-def add_build_entries(jobs_dict, phase, smoke, filter_predicate=lambda x: True):
-
-    configs = gen_build_env_list(smoke)
-    for conf_options in filter(filter_predicate, configs):
-        jobs_dict[conf_options.gen_build_name(phase)] = conf_options.gen_yaml_tree(phase)
-
-
-def add_binary_build_specs(jobs_dict):
-    add_build_entries(jobs_dict, "build", False)
-
-
-def add_binary_build_tests(jobs_dict):
-    add_build_entries(jobs_dict, "test", False, predicate_exclude_nonlinux_and_libtorch)
-
-
-def add_binary_build_uploads(jobs_dict):
-    add_build_entries(jobs_dict, "upload", False)
-
-
-def add_smoke_test_specs(jobs_dict):
-    add_build_entries(jobs_dict, "test", True)
+def get_nightly_uploads():
+    configs = gen_build_env_list(False)
+    mylist = []
+    for conf in configs:
+        phase_dependency = "test" if predicate_exclude_nonlinux_and_libtorch(conf) else "build"
+        mylist.append(conf.gen_workflow_job("upload", phase_dependency))
 
+    return mylist
 
 def get_nightly_tests():
 
@@ -149,55 +138,22 @@ def get_nightly_tests():
 
     tests = []
     for conf_options in filtered_configs:
-        params = {"requires": ["setup", conf_options.gen_build_name("build")]}
-        tests.append({conf_options.gen_build_name("test"): params})
+        yaml_item = conf_options.gen_workflow_job("test")
+        tests.append(yaml_item)
 
     return tests
 
-def get_nightly_uploads():
-
-    configs = gen_build_env_list(False)
-
-    def gen_config(conf, phase_dependency):
-        return {
-            conf.gen_build_name("upload"): OrderedDict([
-                ("context", "org-member"),
-                ("requires", ["setup", conf.gen_build_name(phase_dependency)]),
-            ]),
-        }
-
-    mylist = []
-    for conf in configs:
-        phase_dependency = "test" if predicate_exclude_nonlinux_and_libtorch(conf) else "build"
-        mylist.append(gen_config(conf, phase_dependency))
-
-    return mylist
-
-
-def gen_schedule_tree(cron_timing):
-    return [{
-        "schedule": {
-            "cron": miniutils.quote(cron_timing),
-            "filters": {
-                "branches": {
-                    "only": ["master"],
-                },
-            },
-        },
-    }]
-
 
 def add_jobs_and_render(jobs_dict, toplevel_key, smoke, cron_schedule):
 
     jobs_list = ["setup"]
 
     configs = gen_build_env_list(smoke)
+    phase = "build" if toplevel_key == "binarybuilds" else "test"
     for build_config in configs:
-        build_name = build_config.gen_build_name("build")
-        jobs_list.append({build_name: {"requires": ["setup"]}})
+        jobs_list.append(build_config.gen_workflow_job(phase))
 
     jobs_dict[toplevel_key] = OrderedDict(
-        triggers=gen_schedule_tree(cron_schedule),
         jobs=jobs_list,
     )
 

.circleci/cimodel/data/caffe2_build_data.py

@@ -1,8 +1,7 @@
 #!/usr/bin/env python3
 
-from cimodel.lib.conf_tree import ConfigNode, X
+from cimodel.lib.conf_tree import ConfigNode, X, XImportant
 from cimodel.lib.conf_tree import Ver
-import cimodel.data.dimensions as dimensions
 
 
 CONFIG_TREE_DATA = [
@@ -14,16 +13,17 @@ CONFIG_TREE_DATA = [
         (Ver("cuda", "9.0"), [
             # TODO make explicit that this is a "secret TensorRT build"
             #  (see https://github.com/pytorch/pytorch/pull/17323#discussion_r259446749)
+            # TODO Uh oh, were we supposed to make this one important?!
             X("py2"),
-            X("cmake"),
+            XImportant("cmake"),
         ]),
-        (Ver("cuda", "9.1"), [X("py2")]),
-        (Ver("mkl"), [X("py2")]),
-        (Ver("gcc", "5"), [X("onnx_py2")]),
+        (Ver("cuda", "10.1"), [XImportant("py3.5")]),  # TensorRT 6 build
+        (Ver("mkl"), [XImportant("py2")]),
+        (Ver("gcc", "5"), [XImportant("onnx_py2")]),
         (Ver("clang", "3.8"), [X("py2")]),
         (Ver("clang", "3.9"), [X("py2")]),
-        (Ver("clang", "7"), [X("py2")]),
-        (Ver("android"), [X("py2")]),
+        (Ver("clang", "7"), [XImportant("py2"), XImportant("onnx_py3.6")]),
+        (Ver("android"), [XImportant("py2")]),
     ]),
     (Ver("centos", "7"), [
         (Ver("cuda", "9.0"), [X("py2")]),
@@ -32,7 +32,7 @@ CONFIG_TREE_DATA = [
         # TODO ios and system aren't related. system qualifies where the python comes
         #  from (use the system python instead of homebrew or anaconda)
         (Ver("ios"), [X("py2")]),
-        (Ver("system"), [X("py2")]),
+        (Ver("system"), [XImportant("py2")]),
     ]),
 ]
 
@@ -54,6 +54,8 @@ class TreeConfigNode(ConfigNode):
         return [self.child_constructor()(self, k, v) for (k, v) in self.subtree]
 
     def is_build_only(self):
+        if str(self.find_prop("language_version")) == "onnx_py3.6":
+            return False
         return str(self.find_prop("compiler_version")) in [
             "gcc4.9",
             "clang3.8",
@@ -95,16 +97,13 @@ class LanguageConfigNode(TreeConfigNode):
         self.props["language_version"] = node_name
         self.props["build_only"] = self.is_build_only()
 
-    def get_children(self):
-
-        children = []
-        for phase in dimensions.PHASES:
-            if phase == "build" or not self.props["build_only"]:
-                children.append(PhaseConfigNode(self, phase, []))
-
-        return children
+    def child_constructor(self):
+        return ImportantConfigNode
 
 
-class PhaseConfigNode(TreeConfigNode):
+class ImportantConfigNode(TreeConfigNode):
     def init2(self, node_name):
-        self.props["phase_name"] = node_name
+        self.props["important"] = True
+
+    def get_children(self):
+        return []

.circleci/cimodel/data/caffe2_build_definitions.py

@@ -2,30 +2,34 @@
 
 from collections import OrderedDict
 
+import cimodel.data.dimensions as dimensions
 import cimodel.lib.conf_tree as conf_tree
+from cimodel.lib.conf_tree import Ver
 import cimodel.lib.miniutils as miniutils
-import cimodel.lib.visualization as visualization
 from cimodel.data.caffe2_build_data import CONFIG_TREE_DATA, TopLevelNode
 
 
+from dataclasses import dataclass
+
+
 DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/caffe2/"
 
-DOCKER_IMAGE_VERSION = 276
+DOCKER_IMAGE_VERSION = 324
 
 
-class Conf(object):
-    def __init__(self, language, distro, compiler, phase, build_only):
-
-        self.language = language
-        self.distro = distro
-        self.compiler = compiler
-        self.phase = phase
-        self.build_only = build_only
+@dataclass
+class Conf:
+    language: str
+    distro: Ver
+    compiler: Ver
+    build_only: bool
+    is_important: bool
 
     # TODO: Eventually we can probably just remove the cudnn7 everywhere.
     def get_cudnn_insertion(self):
 
         omit = self.language == "onnx_py2" \
+            or self.language == "onnx_py3.6" \
             or self.compiler.name in ["android", "mkl", "clang"] \
             or str(self.distro) in ["ubuntu14.04", "macos10.13"]
 
@@ -44,9 +48,6 @@ class Conf(object):
         root_parts = self.get_build_name_root_parts()
         return "_".join(root_parts + [phase]).replace(".", "_")
 
-    def get_name(self):
-        return self.construct_phase_name(self.phase)
-
     def get_platform(self):
         platform = self.distro.name
         if self.distro.name != "macos":
@@ -57,6 +58,7 @@ class Conf(object):
 
         lang_substitutions = {
             "onnx_py2": "py2",
+            "onnx_py3.6": "py3.6",
             "cmake": "py2",
         }
 
@@ -64,12 +66,11 @@ class Conf(object):
         parts = [lang] + self.get_build_name_middle_parts()
         return miniutils.quote(DOCKER_IMAGE_PATH_BASE + "-".join(parts) + ":" + str(DOCKER_IMAGE_VERSION))
 
-    def gen_yaml_tree(self):
-
-        tuples = []
-
+    def gen_workflow_params(self, phase):
+        parameters = OrderedDict()
         lang_substitutions = {
             "onnx_py2": "onnx-py2",
+            "onnx_py3.6": "onnx-py3.6",
         }
 
         lang = miniutils.override(self.language, lang_substitutions)
@@ -77,39 +78,42 @@ class Conf(object):
         parts = [
             "caffe2",
             lang,
-        ] + self.get_build_name_middle_parts() + [self.phase]
-
-        build_env = "-".join(parts)
-        if not self.distro.name == "macos":
-            build_env = miniutils.quote(build_env)
-
-        tuples.append(("BUILD_ENVIRONMENT", build_env))
+        ] + self.get_build_name_middle_parts() + [phase]
 
+        build_env_name = "-".join(parts)
+        parameters["build_environment"] = miniutils.quote(build_env_name)
         if self.compiler.name == "ios":
-            tuples.append(("BUILD_IOS", miniutils.quote("1")))
-
-        if self.phase == "test":
+            parameters["build_ios"] = miniutils.quote("1")
+        if phase == "test":
             # TODO cuda should not be considered a compiler
             if self.compiler.name == "cuda":
-                tuples.append(("USE_CUDA_DOCKER_RUNTIME", miniutils.quote("1")))
+                parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
 
-        if self.distro.name == "macos":
-            tuples.append(("PYTHON_VERSION", miniutils.quote("2")))
-
-        else:
-            tuples.append(("DOCKER_IMAGE", self.gen_docker_image()))
+        if self.distro.name != "macos":
+            parameters["docker_image"] = self.gen_docker_image()
             if self.build_only:
-                tuples.append(("BUILD_ONLY", miniutils.quote("1")))
-
-        d = OrderedDict({"environment": OrderedDict(tuples)})
-
-        if self.phase == "test":
+                parameters["build_only"] = miniutils.quote("1")
+        if phase == "test":
             resource_class = "large" if self.compiler.name != "cuda" else "gpu.medium"
-            d["resource_class"] = resource_class
+            parameters["resource_class"] = resource_class
 
-        d["<<"] = "*" + "_".join(["caffe2", self.get_platform(), self.phase, "defaults"])
+        return parameters
 
-        return d
+    def gen_workflow_job(self, phase):
+        job_def = OrderedDict()
+        job_def["name"] = self.construct_phase_name(phase)
+        job_def["requires"] = ["setup"]
+
+        if phase == "test":
+            job_def["requires"].append(self.construct_phase_name("build"))
+            job_name = "caffe2_" + self.get_platform() + "_test"
+        else:
+            job_name = "caffe2_" + self.get_platform() + "_build"
+
+        if not self.is_important:
+            job_def["filters"] = {"branches": {"only": ["master", r"/ci-all\/.*/"]}}
+        job_def.update(self.gen_workflow_params(phase))
+        return {job_name : job_def}
 
 
 def get_root():
@@ -125,11 +129,11 @@ def instantiate_configs():
     for fc in found_configs:
 
         c = Conf(
-            fc.find_prop("language_version"),
-            fc.find_prop("distro_version"),
-            fc.find_prop("compiler_version"),
-            fc.find_prop("phase_name"),
-            fc.find_prop("build_only"),
+            language=fc.find_prop("language_version"),
+            distro=fc.find_prop("distro_version"),
+            compiler=fc.find_prop("compiler_version"),
+            build_only=fc.find_prop("build_only"),
+            is_important=fc.find_prop("important"),
         )
 
         config_list.append(c)
@@ -137,17 +141,7 @@ def instantiate_configs():
     return config_list
 
 
-def add_caffe2_builds(jobs_dict):
-
-    configs = instantiate_configs()
-    for conf_options in configs:
-        jobs_dict[conf_options.get_name()] = conf_options.gen_yaml_tree()
-
-    graph = visualization.generate_graph(get_root())
-    graph.draw("caffe2-config-dimensions.png", prog="twopi")
-
-
-def get_caffe2_workflows():
+def get_workflow_jobs():
 
     configs = instantiate_configs()
 
@@ -158,11 +152,11 @@ def get_caffe2_workflows():
     x = []
     for conf_options in filtered_configs:
 
-        requires = ["setup"]
+        phases = ["build"]
+        if not conf_options.build_only:
+            phases = dimensions.PHASES
 
-        if conf_options.phase == "test":
-            requires.append(conf_options.construct_phase_name("build"))
-
-        x.append({conf_options.get_name(): {"requires": requires}})
+        for phase in phases:
+            x.append(conf_options.gen_workflow_job(phase))
 
     return x

.circleci/cimodel/data/dimensions.py

@@ -5,8 +5,9 @@ PHASES = ["build", "test"]
 
 CUDA_VERSIONS = [
     None,  # cpu build
-    "90",
+    "92",
     "100",
+    "101",
 ]
 
 STANDARD_PYTHON_VERSIONS = [

.circleci/cimodel/data/pytorch_build_data.py

@@ -1,31 +1,38 @@
 #!/usr/bin/env python3
 
-from cimodel.lib.conf_tree import ConfigNode, X
+from cimodel.lib.conf_tree import ConfigNode, X, XImportant
 
 
 CONFIG_TREE_DATA = [
-    ("trusty", [
+    ("xenial", [
         (None, [
-            X("2.7.9"),
+            XImportant("2.7.9"),
             X("2.7"),
-            ("3.5", [("important", [X(True)])]),
+            XImportant("3.5"),  # Not run on all PRs, but should be included on [test all]
             X("nightly"),
         ]),
         ("gcc", [
             ("4.8", [X("3.6")]),
-            ("5.4", [
-                X("3.6"),
+            ("5.4", [  # All this subtree rebases to master and then build
+                XImportant("3.6"),
                 ("3.6", [
-                    ("xla", [X(True)]),
-                    ("namedtensor", [X(True)]),
+                    ("namedtensor", [XImportant(True)]),
                 ]),
             ]),
             ("7", [X("3.6")]),
         ]),
-    ]),
-    ("xenial", [
         ("clang", [
-            ("5", [X("3.6")]),
+            ("5", [
+                XImportant("3.6"),  # This is actually the ASAN build
+                ("3.6", [
+                    ("namedtensor", [XImportant(True)]),  # ASAN
+                ]),
+            ]),
+            ("7", [
+                ("3.6", [
+                    ("xla", [XImportant(True)]),
+                ]),
+            ]),
         ]),
         ("cuda", [
             ("9", [
@@ -36,14 +43,25 @@ CONFIG_TREE_DATA = [
                 # and
                 # https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/build.sh#L153
                 # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453144)
-                ("2.7", [("important", [X(True)])]),
-                X("3.6"),
+                X("2.7"),
+                XImportant("3.6"),
+                ("2.7", [
+                    ("namedtensor", [XImportant(True)]),
+                ]),
             ]),
             ("9.2", [X("3.6")]),
             ("10", [X("3.6")]),
+            ("10.1", [X("3.6")]),
         ]),
         ("android", [
-            ("r19c", [X("3.6")]),
+            ("r19c", [
+                ("3.6", [
+                    ("android_abi", [XImportant("x86_32")]),
+                    ("android_abi", [X("x86_64")]),
+                    ("android_abi", [X("arm-v7a")]),
+                    ("android_abi", [X("arm-v8a")]),
+                ])
+            ]),
         ]),
     ]),
 ]
@@ -87,34 +105,11 @@ class DistroConfigNode(TreeConfigNode):
         distro = self.find_prop("distro_name")
 
         next_nodes = {
-            "trusty": TrustyCompilerConfigNode,
             "xenial": XenialCompilerConfigNode,
         }
         return next_nodes[distro]
 
 
-class TrustyCompilerConfigNode(TreeConfigNode):
-
-    def modify_label(self, label):
-        return label or "<unspecified>"
-
-    def init2(self, node_name):
-        self.props["compiler_name"] = node_name
-
-    def child_constructor(self):
-        return TrustyCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
-
-
-class TrustyCompilerVersionConfigNode(TreeConfigNode):
-
-    def init2(self, node_name):
-        self.props["compiler_version"] = node_name
-
-    # noinspection PyMethodMayBeStatic
-    def child_constructor(self):
-        return PyVerConfigNode
-
-
 class PyVerConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["pyver"] = node_name
@@ -136,6 +131,7 @@ class ExperimentalFeatureConfigNode(TreeConfigNode):
             "xla": XlaConfigNode,
             "namedtensor": NamedTensorConfigNode,
             "important": ImportantConfigNode,
+            "android_abi": AndroidAbiConfigNode,
         }
         return next_nodes[experimental_feature]
 
@@ -147,6 +143,9 @@ class XlaConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["is_xla"] = node_name
 
+    def child_constructor(self):
+        return ImportantConfigNode
+
 
 class NamedTensorConfigNode(TreeConfigNode):
     def modify_label(self, label):
@@ -155,6 +154,16 @@ class NamedTensorConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["is_namedtensor"] = node_name
 
+    def child_constructor(self):
+        return ImportantConfigNode
+
+class AndroidAbiConfigNode(TreeConfigNode):
+
+    def init2(self, node_name):
+        self.props["android_abi"] = node_name
+
+    def child_constructor(self):
+        return ImportantConfigNode
 
 class ImportantConfigNode(TreeConfigNode):
     def modify_label(self, label):
@@ -163,15 +172,22 @@ class ImportantConfigNode(TreeConfigNode):
     def init2(self, node_name):
         self.props["is_important"] = node_name
 
+    def get_children(self):
+        return []
+
 
 class XenialCompilerConfigNode(TreeConfigNode):
 
+    def modify_label(self, label):
+        return label or "<unspecified>"
+
     def init2(self, node_name):
         self.props["compiler_name"] = node_name
 
     # noinspection PyMethodMayBeStatic
     def child_constructor(self):
-        return XenialCompilerVersionConfigNode
+
+        return XenialCompilerVersionConfigNode if self.props["compiler_name"] else PyVerConfigNode
 
 
 class XenialCompilerVersionConfigNode(TreeConfigNode):

.circleci/cimodel/data/pytorch_build_definitions.py

@@ -6,51 +6,44 @@ from cimodel.data.pytorch_build_data import TopLevelNode, CONFIG_TREE_DATA
 import cimodel.data.dimensions as dimensions
 import cimodel.lib.conf_tree as conf_tree
 import cimodel.lib.miniutils as miniutils
-import cimodel.lib.visualization as visualization
+
+from dataclasses import dataclass, field
+from typing import List, Optional
 
 
 DOCKER_IMAGE_PATH_BASE = "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/"
 
-DOCKER_IMAGE_VERSION = 300
+# ARE YOU EDITING THIS NUMBER?  MAKE SURE YOU READ THE GUIDANCE AT THE
+# TOP OF .circleci/config.yml
+DOCKER_IMAGE_VERSION = 347
 
 
-class Conf(object):
-    def __init__(self,
-                 distro,
-                 parms,
-                 pyver=None,
-                 cuda_version=None,
-                 is_xla=False,
-                 restrict_phases=None,
-                 gpu_resource=None,
-                 dependent_tests=None,
-                 parent_build=None,
-                 is_namedtensor=False,
-                 is_important=False):
-
-        self.distro = distro
-        self.pyver = pyver
-        self.parms = parms
-        self.cuda_version = cuda_version
-
-        # TODO expand this to cover all the USE_* that we want to test for
-        #  tesnrorrt, leveldb, lmdb, redis, opencv, mkldnn, ideep, etc.
-        # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453608)
-        self.is_xla = is_xla
-        self.is_namedtensor = is_namedtensor
-        self.is_important = is_important
-
-        self.restrict_phases = restrict_phases
-        self.gpu_resource = gpu_resource
-        self.dependent_tests = dependent_tests or []
-        self.parent_build = parent_build
+@dataclass
+class Conf:
+    distro: str
+    parms: List[str]
+    parms_list_ignored_for_docker_image: Optional[List[str]] = None
+    pyver: Optional[str] = None
+    cuda_version: Optional[str] = None
+    # TODO expand this to cover all the USE_* that we want to test for
+    #  tesnrorrt, leveldb, lmdb, redis, opencv, mkldnn, ideep, etc.
+    # (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259453608)
+    is_xla: bool = False
+    restrict_phases: Optional[List[str]] = None
+    gpu_resource: Optional[str] = None
+    dependent_tests: List = field(default_factory=list)
+    parent_build: Optional['Conf'] = None
+    is_namedtensor: bool = False
+    is_important: bool = False
 
     # TODO: Eliminate the special casing for docker paths
     # In the short term, we *will* need to support special casing as docker images are merged for caffe2 and pytorch
     def get_parms(self, for_docker):
         leading = []
-        if self.is_important and not for_docker:
-            leading.append("AAA")
+        # We just don't run non-important jobs on pull requests;
+        # previously we also named them in a way to make it obvious
+        # if self.is_important and not for_docker:
+        #    leading.append("AAA")
         leading.append("pytorch")
         if self.is_xla and not for_docker:
             leading.append("xla")
@@ -60,7 +53,10 @@ class Conf(object):
         cuda_parms = []
         if self.cuda_version:
             cuda_parms.extend(["cuda" + self.cuda_version, "cudnn7"])
-        return leading + ["linux", self.distro] + cuda_parms + self.parms
+        result = leading + ["linux", self.distro] + cuda_parms + self.parms
+        if (not for_docker and self.parms_list_ignored_for_docker_image is not None):
+            result = result + self.parms_list_ignored_for_docker_image
+        return result
 
     def gen_docker_image_path(self):
 
@@ -78,44 +74,27 @@ class Conf(object):
     def get_dependents(self):
         return self.dependent_tests or []
 
-    def gen_yaml_tree(self, build_or_test):
-
-        build_job_name_pieces = self.get_build_job_name_pieces(build_or_test)
+    def gen_workflow_params(self, phase):
+        parameters = OrderedDict()
+        build_job_name_pieces = self.get_build_job_name_pieces(phase)
 
         build_env_name = "-".join(map(str, build_job_name_pieces))
-
-        env_dict = OrderedDict([
-            ("BUILD_ENVIRONMENT", build_env_name),
-            ("DOCKER_IMAGE", self.gen_docker_image_path()),
-        ])
-
-        if self.pyver:
-            env_dict["PYTHON_VERSION"] = miniutils.quote(self.pyver)
-
-        if build_or_test == "test" and self.gpu_resource:
-            env_dict["USE_CUDA_DOCKER_RUNTIME"] = miniutils.quote("1")
-
-        d = {
-            "environment": env_dict,
-            "<<": "*" + "_".join(["pytorch", "linux", build_or_test, "defaults"]),
-        }
-
-        if build_or_test == "test":
+        parameters["build_environment"] = miniutils.quote(build_env_name)
+        parameters["docker_image"] = self.gen_docker_image_path()
+        if phase == "test" and self.gpu_resource:
+            parameters["use_cuda_docker_runtime"] = miniutils.quote("1")
+        if phase == "test":
             resource_class = "large"
             if self.gpu_resource:
                 resource_class = "gpu." + self.gpu_resource
+            parameters["resource_class"] = resource_class
+        return parameters
 
-                if self.gpu_resource == "large":
-                    env_dict["MULTI_GPU"] = miniutils.quote("1")
-
-            d["resource_class"] = resource_class
-
-        return d
-
-    def gen_workflow_yaml_item(self, phase):
-
+    def gen_workflow_job(self, phase):
         # All jobs require the setup job
-        parameters = OrderedDict({"requires": ["setup"]})
+        job_def = OrderedDict()
+        job_def["name"] = self.gen_build_name(phase)
+        job_def["requires"] = ["setup"]
 
         if phase == "test":
 
@@ -125,9 +104,19 @@ class Conf(object):
             #  pytorch build job (from https://github.com/pytorch/pytorch/pull/17323#discussion_r259452641)
 
             dependency_build = self.parent_build or self
-            parameters["requires"].append(dependency_build.gen_build_name("build"))
+            job_def["requires"].append(dependency_build.gen_build_name("build"))
+            job_name = "pytorch_linux_test"
+        else:
+            job_name = "pytorch_linux_build"
 
-        return {self.gen_build_name(phase): parameters}
+
+        if not self.is_important:
+            # If you update this, update
+            # caffe2_build_definitions.py too
+            job_def["filters"] = {"branches": {"only": ["master", r"/ci-all\/.*/"]}}
+        job_def.update(self.gen_workflow_params(phase))
+
+        return {job_name : job_def}
 
 
 # TODO This is a hack to special case some configs just for the workflow list
@@ -136,8 +125,7 @@ class HiddenConf(object):
         self.name = name
         self.parent_build = parent_build
 
-    def gen_workflow_yaml_item(self, phase):
-
+    def gen_workflow_job(self, phase):
         return {self.gen_build_name(phase): {"requires": [self.parent_build.gen_build_name("build")]}}
 
     def gen_build_name(self, _):
@@ -166,11 +154,12 @@ def gen_dependent_configs(xenial_parent_config):
             restrict_phases=["test"],
             gpu_resource=gpu,
             parent_build=xenial_parent_config,
+            is_important=xenial_parent_config.is_important,
         )
 
         configs.append(c)
 
-    for x in ["pytorch_short_perf_test_gpu", "pytorch_doc_push"]:
+    for x in ["pytorch_short_perf_test_gpu", "pytorch_python_doc_push", "pytorch_cpp_doc_push"]:
         configs.append(HiddenConf(x, parent_build=xenial_parent_config))
 
     return configs
@@ -196,16 +185,18 @@ def instantiate_configs():
     for fc in found_configs:
 
         distro_name = fc.find_prop("distro_name")
+        compiler_name = fc.find_prop("compiler_name")
+        compiler_version = fc.find_prop("compiler_version")
+        is_xla = fc.find_prop("is_xla") or False
+        parms_list_ignored_for_docker_image = []
 
         python_version = None
-        if distro_name == "xenial":
+        if compiler_name == "cuda" or compiler_name == "android":
             python_version = fc.find_prop("pyver")
             parms_list = [fc.find_prop("abbreviated_pyver")]
         else:
             parms_list = ["py" + fc.find_prop("pyver")]
 
-        compiler_name = fc.find_prop("compiler_name")
-
         cuda_version = None
         if compiler_name == "cuda":
             cuda_version = fc.find_prop("compiler_version")
@@ -215,20 +206,24 @@ def instantiate_configs():
             # TODO: do we need clang to compile host binaries like protoc?
             parms_list.append("clang5")
             parms_list.append("android-ndk-" + android_ndk_version)
+            android_abi = fc.find_prop("android_abi")
+            parms_list_ignored_for_docker_image.append(android_abi)
             restrict_phases = ["build"]
 
         elif compiler_name:
             gcc_version = compiler_name + (fc.find_prop("compiler_version") or "")
             parms_list.append(gcc_version)
 
-            if compiler_name == "clang":
+            # TODO: This is a nasty special case
+            if compiler_name == "clang" and not is_xla:
                 parms_list.append("asan")
+                python_version = fc.find_prop("pyver")
+                parms_list[0] = fc.find_prop("abbreviated_pyver")
 
-        if cuda_version in ["9.2", "10"]:
+        if cuda_version in ["9.2", "10", "10.1"]:
             # TODO The gcc version is orthogonal to CUDA version?
             parms_list.append("gcc7")
 
-        is_xla = fc.find_prop("is_xla") or False
         is_namedtensor = fc.find_prop("is_namedtensor") or False
         is_important = fc.find_prop("is_important") or False
 
@@ -239,6 +234,7 @@ def instantiate_configs():
         c = Conf(
             distro_name,
             parms_list,
+            parms_list_ignored_for_docker_image,
             python_version,
             cuda_version,
             is_xla,
@@ -251,44 +247,26 @@ def instantiate_configs():
         if cuda_version == "9" and python_version == "3.6":
             c.dependent_tests = gen_dependent_configs(c)
 
+        if (compiler_name == "gcc"
+                and compiler_version == "5.4"
+                and not is_namedtensor):
+            bc_breaking_check = Conf(
+                "backward-compatibility-check",
+                [],
+                is_xla=False,
+                restrict_phases=["test"],
+                is_namedtensor=False,
+                is_important=True,
+                parent_build=c,
+            )
+            c.dependent_tests.append(bc_breaking_check)
+
         config_list.append(c)
 
     return config_list
 
 
-def add_build_env_defs(jobs_dict):
-
-    mydict = OrderedDict()
-
-    config_list = instantiate_configs()
-    for c in config_list:
-
-        phases = c.restrict_phases or dimensions.PHASES
-
-        for phase in phases:
-
-            # TODO why does this not have a test?
-            if phase == "test" and c.cuda_version == "10":
-                continue
-
-            d = c.gen_yaml_tree(phase)
-            mydict[c.gen_build_name(phase)] = d
-
-            if phase == "test":
-                for x in filter(lambda x: type(x) is not HiddenConf, c.get_dependents()):
-
-                    d = x.gen_yaml_tree(phase)
-                    mydict[x.gen_build_name(phase)] = d
-
-    # this is the circleci api version and probably never changes
-    jobs_dict["version"] = 2
-    jobs_dict["jobs"] = mydict
-
-    graph = visualization.generate_graph(get_root())
-    graph.draw("pytorch-config-dimensions.png", prog="twopi")
-
-
-def get_workflow_list():
+def get_workflow_jobs():
 
     config_list = instantiate_configs()
 
@@ -303,10 +281,10 @@ def get_workflow_list():
             if phase == "test" and conf_options.cuda_version == "10":
                 continue
 
-            x.append(conf_options.gen_workflow_yaml_item(phase))
+            x.append(conf_options.gen_workflow_job(phase))
 
         # TODO convert to recursion
         for conf in conf_options.get_dependents():
-            x.append(conf.gen_workflow_yaml_item("test"))
+            x.append(conf.gen_workflow_job("test"))
 
     return x

.circleci/cimodel/lib/conf_tree.py

@@ -1,6 +1,10 @@
 #!/usr/bin/env python3
 
 
+from dataclasses import dataclass, field
+from typing import Optional, Dict
+
+
 def X(val):
     """
     Compact way to write a leaf node
@@ -8,23 +12,28 @@ def X(val):
     return val, []
 
 
-class Ver(object):
+def XImportant(name):
+    """Compact way to write an important (run on PRs) leaf node"""
+    return (name, [("important", [X(True)])])
+
+
+@dataclass
+class Ver:
     """
     Represents a product with a version number
     """
-    def __init__(self, name, version=""):
-        self.name = name
-        self.version = version
+    name: str
+    version: str = ""
 
     def __str__(self):
         return self.name + self.version
 
 
-class ConfigNode(object):
-    def __init__(self, parent, node_name):
-        self.parent = parent
-        self.node_name = node_name
-        self.props = {}
+@dataclass
+class ConfigNode:
+    parent: Optional['ConfigNode']
+    node_name: str
+    props: Dict[str, str] = field(default_factory=dict)
 
     def get_label(self):
         return self.node_name

.circleci/cimodel/lib/miniyaml.py

@@ -9,23 +9,13 @@ INDENTATION_WIDTH = 2
 
 
 def is_dict(data):
-    return type(data) is dict or type(data) is OrderedDict
+    return type(data) in [dict, OrderedDict]
 
 
 def is_collection(data):
     return is_dict(data) or type(data) is list
 
 
-# TODO can eventually drop this custom sorting
-def sortkey(x):
-    k = x[0]
-    return (
-        k == "<<",
-        k != "environment",
-        k,
-    )
-
-
 def render(fh, data, depth, is_list_member=False):
     """
     PyYaml does not allow precise control over the quoting
@@ -39,7 +29,7 @@ def render(fh, data, depth, is_list_member=False):
 
         tuples = list(data.items())
         if type(data) is not OrderedDict:
-            tuples.sort(key=sortkey)
+            tuples.sort()
 
         for i, (k, v) in enumerate(tuples):
 
@@ -51,10 +41,6 @@ def render(fh, data, depth, is_list_member=False):
 
             render(fh, v, depth + 1 + int(is_list_member))
 
-        # TODO Could eventually drop this cosmetic convention
-        if depth == 2:
-            fh.write("\n")
-
     elif type(data) is list:
         for v in data:
             render(fh, v, depth, True)

.circleci/generate_config_yml.py

@@ -74,41 +74,33 @@ class Header(object):
 # Order of this list matters to the generated config.yml.
 YAML_SOURCES = [
     File("header-section.yml"),
-    File("linux-build-defaults.yml"),
-    File("macos-build-defaults.yml"),
+    File("commands.yml"),
     File("nightly-binary-build-defaults.yml"),
-    File("linux-binary-build-defaults.yml"),
-    File("macos-binary-build-defaults.yml"),
-    File("nightly-build-smoke-tests-defaults.yml"),
-    Header("Job specifications job specs"),
-    Treegen(pytorch_build_definitions.add_build_env_defs, 0),
+    Header("Build parameters"),
+    File("pytorch-build-params.yml"),
+    File("caffe2-build-params.yml"),
+    File("binary-build-params.yml"),
+    Header("Job specs"),
+    File("pytorch-job-specs.yml"),
+    File("caffe2-job-specs.yml"),
+    File("binary-job-specs.yml"),
     File("job-specs-setup.yml"),
     File("job-specs-custom.yml"),
-    Treegen(caffe2_build_definitions.add_caffe2_builds, 1),
     File("binary_update_htmls.yml"),
-    Header("Binary build specs individual job specifications"),
-    Treegen(binary_build_definitions.add_binary_build_specs, 1),
-    Header(
-        "Binary build tests", [
-            "These are the smoke tests run right after the build, before the upload.",
-            "If these fail, the upload doesn't happen."
-        ]
-    ),
-    Treegen(binary_build_definitions.add_binary_build_tests, 1),
     File("binary-build-tests.yml"),
-    Header("Binary build uploads"),
-    Treegen(binary_build_definitions.add_binary_build_uploads, 1),
-    Header("Smoke test specs individual job specifications"),
-    Treegen(binary_build_definitions.add_smoke_test_specs, 1),
     File("workflows.yml"),
-    Listgen(pytorch_build_definitions.get_workflow_list, 3),
+    Listgen(pytorch_build_definitions.get_workflow_jobs, 3),
     File("workflows-pytorch-macos-builds.yml"),
-    Listgen(caffe2_build_definitions.get_caffe2_workflows, 3),
+    File("workflows-pytorch-android-gradle-build.yml"),
+    File("workflows-pytorch-ios-builds.yml"),
+    Listgen(caffe2_build_definitions.get_workflow_jobs, 3),
     File("workflows-binary-builds-smoke-subset.yml"),
     Header("Daily smoke test trigger"),
     Treegen(binary_build_definitions.add_binary_smoke_test_jobs, 1),
     Header("Daily binary build trigger"),
     Treegen(binary_build_definitions.add_binary_build_jobs, 1),
+    File("workflows-nightly-ios-binary-builds.yml"),
+    File("workflows-nightly-android-binary-builds.yml"),
     Header("Nightly tests"),
     Listgen(binary_build_definitions.get_nightly_tests, 3),
     File("workflows-nightly-uploads-header.yml"),

.circleci/scripts/README.md

@@ -0,0 +1,4 @@
+All the scripts in this directory are callable from `~/workspace/.circleci/scripts/foo.sh`.
+Don't try to call them as `.circleci/scripts/foo.sh`, that won't
+(necessarily) work.  See Note [Workspace for CircleCI scripts] in
+job-specs-setup.yml for more details.

.circleci/scripts/binary_checkout.sh

@@ -41,8 +41,6 @@ popd
 # Clone the Builder master repo
 git clone -q https://github.com/pytorch/builder.git "$BUILDER_ROOT"
 pushd "$BUILDER_ROOT"
-git fetch origin
-git reset origin/master --hard
 echo "Using builder from "
 git --no-pager log --max-count 1
 popd

.circleci/scripts/binary_ios_build.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+set -eux -o pipefail
+
+echo ""
+echo "PWD: ${PWD}"
+WORKSPACE=/Users/distiller/workspace
+PROJ_ROOT=/Users/distiller/project
+export TCLLIBPATH="/usr/local/lib" 
+# Install conda
+curl -o ~/Downloads/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+chmod +x ~/Downloads/conda.sh
+/bin/bash ~/Downloads/conda.sh -b -p ~/anaconda
+export PATH="~/anaconda/bin:${PATH}"
+source ~/anaconda/bin/activate
+# Install dependencies
+conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing requests --yes
+export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
+# sync submodules
+cd ${PROJ_ROOT}
+git submodule sync
+git submodule update --init --recursive
+# run build script
+chmod a+x ${PROJ_ROOT}/scripts/build_ios.sh
+echo "########################################################"
+cat ${PROJ_ROOT}/scripts/build_ios.sh
+echo "########################################################"
+echo "IOS_ARCH: ${IOS_ARCH}"
+echo "IOS_PLATFORM: ${IOS_PLATFORM}"
+export BUILD_PYTORCH_MOBILE=1
+export IOS_ARCH=${IOS_ARCH}
+export IOS_PLATFORM=${IOS_PLATFORM}
+unbuffer ${PROJ_ROOT}/scripts/build_ios.sh 2>&1 | ts
+#store the binary
+cd ${WORKSPACE}
+DEST_DIR=${WORKSPACE}/ios
+mkdir -p ${DEST_DIR}
+cp -R ${PROJ_ROOT}/build_ios/install ${DEST_DIR}
+mv ${DEST_DIR}/install ${DEST_DIR}/${IOS_ARCH}

.circleci/scripts/binary_ios_upload.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+set -eux -o pipefail
+
+echo ""
+echo "PWD: $(pwd)"
+WORKSPACE=/Users/distiller/workspace
+PROJ_ROOT=/Users/distiller/project
+ARTIFACTS_DIR=${WORKSPACE}/ios
+ls ${ARTIFACTS_DIR}
+ZIP_DIR=${WORKSPACE}/zip
+mkdir -p ${ZIP_DIR}/install/lib
+mkdir -p ${ZIP_DIR}/src
+# copy header files
+cp -R ${ARTIFACTS_DIR}/arm64/include ${ZIP_DIR}/install/
+# build a FAT bianry
+cd ${ZIP_DIR}/install/lib
+target_libs=(libc10.a libclog.a libcpuinfo.a libeigen_blas.a libpytorch_qnnpack.a libtorch.a)
+for lib in ${target_libs[*]}
+do
+    libs=(${ARTIFACTS_DIR}/x86_64/lib/${lib} ${ARTIFACTS_DIR}/arm64/lib/${lib})
+    lipo -create "${libs[@]}" -o ${ZIP_DIR}/install/lib/${lib}
+done
+# for nnpack, we only support arm64 build
+cp ${ARTIFACTS_DIR}/arm64/lib/libnnpack.a ./
+lipo -i ${ZIP_DIR}/install/lib/*.a
+# copy the umbrella header and license
+cp ${PROJ_ROOT}/ios/LibTorch.h ${ZIP_DIR}/src/
+cp ${PROJ_ROOT}/LICENSE ${ZIP_DIR}/
+# zip the library
+ZIPFILE=libtorch_ios_nightly_build.zip
+cd ${ZIP_DIR}
+#for testing
+touch version.txt
+echo $(date +%s) > version.txt
+zip -r ${ZIPFILE} install src version.txt LICENSE
+# upload to aws
+brew install awscli
+set +x
+export AWS_ACCESS_KEY_ID=${AWS_S3_ACCESS_KEY_FOR_PYTORCH_BINARY_UPLOAD}
+export AWS_SECRET_ACCESS_KEY=${AWS_S3_ACCESS_SECRET_FOR_PYTORCH_BINARY_UPLOAD}
+set +x
+# echo "AWS KEY: ${AWS_ACCESS_KEY_ID}"
+# echo "AWS SECRET: ${AWS_SECRET_ACCESS_KEY}"
+aws s3 cp ${ZIPFILE} s3://ossci-ios-build/ --acl public-read

.circleci/scripts/binary_linux_build.sh

@@ -19,7 +19,7 @@ fi
 # We want to call unbuffer, which calls tclsh which finds the expect
 # package. The expect was installed by yum into /usr/bin so we want to
 # find /usr/bin/tclsh, but this is shadowed by /opt/conda/bin/tclsh in
-# the conda docker images.
+# the conda docker images, so we prepend it to the path here.
 if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
   mkdir /just_tclsh_bin
   ln -s /usr/bin/tclsh /just_tclsh_bin/tclsh

.circleci/scripts/binary_linux_test.sh

@@ -11,7 +11,7 @@ if [[ "$PACKAGE_TYPE" == conda ]]; then
   source activate testenv >/dev/null
 elif [[ "$DESIRED_PYTHON" == 2.7mu ]]; then
   export PATH="/opt/python/cp27-cp27mu/bin:\$PATH"
-else
+elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
   python_nodot="\$(echo $DESIRED_PYTHON | tr -d m.u)"
   export PATH="/opt/python/cp\$python_nodot-cp\${python_nodot}m/bin:\$PATH"
 fi
@@ -25,6 +25,9 @@ fi
 pkg="/final_pkgs/\$(ls /final_pkgs)"
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   conda install -y "\$pkg" --offline
+  if [[ "$DESIRED_CUDA" == 'cpu' ]]; then
+    conda install -y cpuonly -c pytorch
+  fi
   retry conda install -yq future numpy protobuf six
   if [[ "$DESIRED_CUDA" != 'cpu' ]]; then
     # DESIRED_CUDA is in format cu90 or cu100
@@ -35,10 +38,15 @@ if [[ "$PACKAGE_TYPE" == conda ]]; then
     fi
     retry conda install -yq -c pytorch "cudatoolkit=\${cu_ver}"
   fi
-else
+elif [[ "$PACKAGE_TYPE" != libtorch ]]; then
   pip install "\$pkg"
   retry pip install -q future numpy protobuf six
 fi
+if [[ "$PACKAGE_TYPE" == libtorch ]]; then
+  pkg="\$(ls /final_pkgs/*-latest.zip)"
+  unzip "\$pkg" -d /tmp
+  cd /tmp/libtorch
+fi
 
 # Test the package
 /builder/check_binary.sh

.circleci/scripts/binary_linux_upload.sh

@@ -26,7 +26,7 @@ pushd /home/circleci/project/final_pkgs
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   retry conda install -yq anaconda-client
   retry timeout 30 /home/circleci/project/login_to_anaconda.sh
-  anaconda upload "$(ls)" -u pytorch --label main --no-progress --force
+  anaconda upload "$(ls)" -u pytorch-nightly --label main --no-progress --force
 elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
   retry pip install -q awscli
   s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"

.circleci/scripts/binary_macos_upload.sh

@@ -26,7 +26,7 @@ pushd "$workdir/final_pkgs"
 if [[ "$PACKAGE_TYPE" == conda ]]; then
   retry conda install -yq anaconda-client
   retry /Users/distiller/project/login_to_anaconda.sh
-  retry anaconda upload "$(ls)" -u pytorch --label main --no-progress --force
+  retry anaconda upload "$(ls)" -u pytorch-nightly --label main --no-progress --force
 elif [[ "$PACKAGE_TYPE" == libtorch ]]; then
   retry pip install -q awscli
   s3_dir="s3://pytorch/libtorch/${PIP_UPLOAD_FOLDER}${DESIRED_CUDA}/"

.circleci/scripts/binary_populate_env.sh

@@ -29,29 +29,37 @@ if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then
 fi
 
 # Pick docker image
-if [[ "$PACKAGE_TYPE" == conda ]]; then
-  export DOCKER_IMAGE="soumith/conda-cuda"
-elif [[ "$DESIRED_CUDA" == cpu ]]; then
-  export DOCKER_IMAGE="soumith/manylinux-cuda80"
-else
-  export DOCKER_IMAGE="soumith/manylinux-cuda${DESIRED_CUDA:2}"
+export DOCKER_IMAGE=${DOCKER_IMAGE:-}
+if [[ -z "$DOCKER_IMAGE" ]]; then
+  if [[ "$PACKAGE_TYPE" == conda ]]; then
+    export DOCKER_IMAGE="soumith/conda-cuda"
+  elif [[ "$DESIRED_CUDA" == cpu ]]; then
+    export DOCKER_IMAGE="soumith/manylinux-cuda100"
+  else
+    export DOCKER_IMAGE="soumith/manylinux-cuda${DESIRED_CUDA:2}"
+  fi
 fi
 
-# Upload to parallel folder for gcc abis
-if [[ "$DESIRED_DEVTOOLSET" == 'devtoolset7' ]]; then
-  export PIP_UPLOAD_FOLDER='nightly/devtoolset7/'
-  if [[ "$PACKAGE_TYPE" == 'conda' ]]; then
-    echo "We don't handle conda builds with gcc ABI of 1, since we don't"
-    echo "want to add a new package name to the conda builds"
-    exit 1
-  fi
-else
+# Upload to parallel folder for devtoolsets
+# All nightlies used to be devtoolset3, then devtoolset7 was added as a build
+# option, so the upload was redirected to nightly/devtoolset7 to avoid
+# conflicts with other binaries (there shouldn't be any conflicts). Now we are
+# making devtoolset7 the default.
+if [[ "$DESIRED_DEVTOOLSET" == 'devtoolset7' || "$DESIRED_DEVTOOLSET" == *"cxx11-abi"* || "$(uname)" == 'Darwin' ]]; then
   export PIP_UPLOAD_FOLDER='nightly/'
+else
+  # On linux machines, this shouldn't actually be called anymore. This is just
+  # here for extra safety.
+  export PIP_UPLOAD_FOLDER='nightly/devtoolset3/'
 fi
 
 # We put this here so that OVERRIDE_PACKAGE_VERSION below can read from it
 export DATE="$(date -u +%Y%m%d)"
-export PYTORCH_BUILD_VERSION="1.2.0.dev$DATE"
+if [[ "$(uname)" == 'Darwin' ]] || [[ "$DESIRED_CUDA" == "cu100" ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
+  export PYTORCH_BUILD_VERSION="1.3.0.dev$DATE"
+else
+  export PYTORCH_BUILD_VERSION="1.3.0.dev$DATE+$DESIRED_CUDA"
+fi
 export PYTORCH_BUILD_NUMBER=1
 
 cat >>"$envfile" <<EOL
@@ -67,15 +75,16 @@ export BUILD_PYTHONLESS="${BUILD_PYTHONLESS:-}"
 export DESIRED_DEVTOOLSET="$DESIRED_DEVTOOLSET"
 
 export DATE="$DATE"
-export NIGHTLIES_DATE_PREAMBLE=1.2.0.dev
+export NIGHTLIES_DATE_PREAMBLE=1.3.0.dev
 export PYTORCH_BUILD_VERSION="$PYTORCH_BUILD_VERSION"
 export PYTORCH_BUILD_NUMBER="$PYTORCH_BUILD_NUMBER"
 export OVERRIDE_PACKAGE_VERSION="$PYTORCH_BUILD_VERSION"
 
-export TORCH_PACKAGE_NAME='torch-nightly'
+# TODO: We don't need this anymore IIUC
+export TORCH_PACKAGE_NAME='torch'
 export TORCH_CONDA_BUILD_FOLDER='pytorch-nightly'
 
-export NO_FBGEMM=1
+export USE_FBGEMM=1
 export PIP_UPLOAD_FOLDER="$PIP_UPLOAD_FOLDER"
 export DOCKER_IMAGE="$DOCKER_IMAGE"
 

.circleci/scripts/build_android_gradle.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+set -eux -o pipefail
+
+export ANDROID_NDK_HOME=/opt/ndk
+export ANDROID_HOME=/opt/android/sdk
+
+export GRADLE_VERSION=4.10.3
+export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
+export GRADLE_PATH=$GRADLE_HOME/bin/gradle
+
+BUILD_ANDROID_INCLUDE_DIR_x86=~/workspace/build_android/install/include
+BUILD_ANDROID_LIB_DIR_x86=~/workspace/build_android/install/lib
+
+BUILD_ANDROID_INCLUDE_DIR_x86_64=~/workspace/build_android_install_x86_64/install/include
+BUILD_ANDROID_LIB_DIR_x86_64=~/workspace/build_android_install_x86_64/install/lib
+
+BUILD_ANDROID_INCLUDE_DIR_arm_v7a=~/workspace/build_android_install_arm_v7a/install/include
+BUILD_ANDROID_LIB_DIR_arm_v7a=~/workspace/build_android_install_arm_v7a/install/lib
+
+BUILD_ANDROID_INCLUDE_DIR_arm_v8a=~/workspace/build_android_install_arm_v8a/install/include
+BUILD_ANDROID_LIB_DIR_arm_v8a=~/workspace/build_android_install_arm_v8a/install/lib
+
+PYTORCH_ANDROID_SRC_MAIN_DIR=~/workspace/android/pytorch_android/src/main
+
+JNI_INCLUDE_DIR=${PYTORCH_ANDROID_SRC_MAIN_DIR}/cpp/libtorch_include
+mkdir -p $JNI_INCLUDE_DIR
+
+JNI_LIBS_DIR=${PYTORCH_ANDROID_SRC_MAIN_DIR}/jniLibs
+mkdir -p $JNI_LIBS_DIR
+
+ln -s ${BUILD_ANDROID_INCLUDE_DIR_x86} ${JNI_INCLUDE_DIR}/x86
+ln -s ${BUILD_ANDROID_LIB_DIR_x86} ${JNI_LIBS_DIR}/x86
+
+if [[ "${BUILD_ENVIRONMENT}" != *-gradle-build-only-x86_32* ]]; then
+ln -s ${BUILD_ANDROID_INCLUDE_DIR_x86_64} ${JNI_INCLUDE_DIR}/x86_64
+ln -s ${BUILD_ANDROID_LIB_DIR_x86_64} ${JNI_LIBS_DIR}/x86_64
+
+ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v7a} ${JNI_INCLUDE_DIR}/armeabi-v7a
+ln -s ${BUILD_ANDROID_LIB_DIR_arm_v7a} ${JNI_LIBS_DIR}/armeabi-v7a
+
+ln -s ${BUILD_ANDROID_INCLUDE_DIR_arm_v8a} ${JNI_INCLUDE_DIR}/arm64-v8a
+ln -s ${BUILD_ANDROID_LIB_DIR_arm_v8a} ${JNI_LIBS_DIR}/arm64-v8a
+fi
+
+env
+echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
+
+export GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
+rm -f $GRADLE_LOCAL_PROPERTIES
+echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
+echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
+
+if [[ "${BUILD_ENVIRONMENT}" == *-gradle-build-only-x86_32* ]]; then
+    $GRADLE_PATH -PABI_FILTERS=x86 -p ~/workspace/android/ assembleRelease
+else
+    $GRADLE_PATH -p ~/workspace/android/ assembleRelease
+fi
+
+find . -type f -name *aar -print | xargs tar cfvz ~/workspace/android/artifacts.tgz

.circleci/scripts/cpp_doc_push_script.sh

@@ -0,0 +1,127 @@
+# =================== The following code **should** be executed inside Docker container ===================
+
+# Install dependencies
+sudo apt-get -y update
+sudo apt-get -y install expect-dev
+
+# This is where the local pytorch install in the docker image is located
+pt_checkout="/var/lib/jenkins/workspace"
+
+# Since we're cat-ing this file, we need to escape all $'s
+echo "cpp_doc_push_script.sh: Invoked with $*"
+
+# Argument 1: Where to copy the built documentation for Python API to
+# (pytorch.github.io/$install_path)
+install_path="$1"
+if [ -z "$install_path" ]; then
+echo "error: cpp_doc_push_script.sh: install_path (arg1) not specified"
+  exit 1
+fi
+
+# Argument 2: What version of the Python API docs we are building.
+version="$2"
+if [ -z "$version" ]; then
+echo "error: cpp_doc_push_script.sh: version (arg2) not specified"
+  exit 1
+fi
+
+is_master_doc=false
+if [ "$version" == "master" ]; then
+  is_master_doc=true
+fi
+
+# Argument 3: (optional) If present, we will NOT do any pushing. Used for testing.
+dry_run=false
+if [ "$3" != "" ]; then
+  dry_run=true
+fi
+
+echo "install_path: $install_path  version: $version  dry_run: $dry_run"
+
+# ======================== Building PyTorch C++ API Docs ========================
+
+echo "Building PyTorch C++ API docs..."
+
+# Clone the cppdocs repo
+rm -rf cppdocs
+git clone https://github.com/pytorch/cppdocs
+
+set -ex
+
+sudo apt-get -y install doxygen
+
+# Generate ATen files
+pushd "${pt_checkout}"
+pip install -r requirements.txt
+time GEN_TO_SOURCE=1 python aten/src/ATen/gen.py \
+  -s aten/src/ATen \
+  -d build/aten/src/ATen \
+  aten/src/ATen/Declarations.cwrap \
+  aten/src/THNN/generic/THNN.h \
+  aten/src/THCUNN/generic/THCUNN.h \
+  aten/src/ATen/nn.yaml \
+  aten/src/ATen/native/native_functions.yaml
+
+# Copy some required files
+cp aten/src/ATen/common_with_cwrap.py tools/shared/cwrap_common.py
+cp torch/_utils_internal.py tools/shared
+
+# Generate PyTorch files
+time python tools/setup_helpers/generate_code.py \
+  --declarations-path build/aten/src/ATen/Declarations.yaml \
+  --nn-path aten/src/
+
+# Build the docs
+pushd docs/cpp
+pip install breathe==4.11.1 bs4 lxml six
+pip install --no-cache-dir -e "git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme"
+pip install exhale>=0.2.1
+pip install sphinx==1.8.5
+# Uncomment once it is fixed
+# pip install -r requirements.txt
+time make VERBOSE=1 html -j
+
+popd
+popd
+
+pushd cppdocs
+
+# Purge everything with some exceptions
+mkdir /tmp/cppdocs-sync
+mv _config.yml README.md /tmp/cppdocs-sync/
+rm -rf *
+
+# Copy over all the newly generated HTML
+cp -r "${pt_checkout}"/docs/cpp/build/html/* .
+
+# Copy back _config.yml
+rm -rf _config.yml
+mv /tmp/cppdocs-sync/* .
+
+# Make a new commit
+git add . || true
+git status
+git config user.email "soumith+bot@pytorch.org"
+git config user.name "pytorchbot"
+# If there aren't changes, don't make a commit; push is no-op
+git commit -m "Automatic sync on $(date)" || true
+git status
+
+if [ "$dry_run" = false ]; then
+  echo "Pushing to https://github.com/pytorch/cppdocs"
+  set +x
+/usr/bin/expect <<DONE
+  spawn git push -u origin master
+  expect "Username*"
+  send "pytorchbot\n"
+  expect "Password*"
+  send "$::env(GITHUB_PYTORCHBOT_TOKEN)\n"
+  expect eof
+DONE
+  set -x
+else
+  echo "Skipping push due to dry_run"
+fi
+
+popd
+# =================== The above code **should** be executed inside Docker container ===================

.circleci/scripts/publish_android_snapshot.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# DO NOT ADD 'set -x' not to reveal CircleCI secret context environment variables
+set -eu -o pipefail
+
+export ANDROID_NDK_HOME=/opt/ndk
+export ANDROID_HOME=/opt/android/sdk
+
+export GRADLE_VERSION=4.10.3
+export GRADLE_HOME=/opt/gradle/gradle-$GRADLE_VERSION
+export GRADLE_PATH=$GRADLE_HOME/bin/gradle
+
+echo "BUILD_ENVIRONMENT:$BUILD_ENVIRONMENT"
+ls -la ~/workspace
+
+GRADLE_PROPERTIES=~/workspace/android/gradle.properties
+
+IS_SNAPSHOT="$(grep 'VERSION_NAME=[0-9\.]\+-SNAPSHOT' "$GRADLE_PROPERTIES")"
+echo "IS_SNAPSHOT:$IS_SNAPSHOT"
+
+if [ -z "$IS_SNAPSHOT" ]; then
+  echo "Error: version is not snapshot."
+elif [ -z "$SONATYPE_NEXUS_USERNAME" ]; then
+  echo "Error: missing env variable SONATYPE_NEXUS_USERNAME."
+elif [ -z "$SONATYPE_NEXUS_PASSWORD" ]; then
+  echo "Error: missing env variable SONATYPE_NEXUS_PASSWORD."
+elif [ -z "$ANDROID_SIGN_KEY" ]; then
+  echo "Error: missing env variable ANDROID_SIGN_KEY."
+elif [ -z "$ANDROID_SIGN_PASS" ]; then
+  echo "Error: missing env variable ANDROID_SIGN_PASS."
+else
+  GRADLE_LOCAL_PROPERTIES=~/workspace/android/local.properties
+  rm -f $GRADLE_LOCAL_PROPERTIES
+
+  echo "sdk.dir=/opt/android/sdk" >> $GRADLE_LOCAL_PROPERTIES
+  echo "ndk.dir=/opt/ndk" >> $GRADLE_LOCAL_PROPERTIES
+
+  echo "SONATYPE_NEXUS_USERNAME=${SONATYPE_NEXUS_USERNAME}" >> $GRADLE_PROPERTIES
+  echo "SONATYPE_NEXUS_PASSWORD=${SONATYPE_NEXUS_PASSWORD}" >> $GRADLE_PROPERTIES
+
+  echo "signing.keyId=${ANDROID_SIGN_KEY}" >> $GRADLE_PROPERTIES
+  echo "signing.password=${ANDROID_SIGN_PASS}" >> $GRADLE_PROPERTIES
+
+  $GRADLE_PATH -p ~/workspace/android/ uploadArchives
+fi

.circleci/scripts/python_doc_push_script.sh

@@ -0,0 +1,118 @@
+# =================== The following code **should** be executed inside Docker container ===================
+
+# Install dependencies
+sudo apt-get -y update
+sudo apt-get -y install expect-dev
+
+# This is where the local pytorch install in the docker image is located
+pt_checkout="/var/lib/jenkins/workspace"
+
+echo "python_doc_push_script.sh: Invoked with $*"
+
+set -ex
+
+# Argument 1: Where to copy the built documentation to
+# (pytorch.github.io/$install_path)
+install_path="$1"
+if [ -z "$install_path" ]; then
+echo "error: python_doc_push_script.sh: install_path (arg1) not specified"
+  exit 1
+fi
+
+# Argument 2: What version of the docs we are building.
+version="$2"
+if [ -z "$version" ]; then
+echo "error: python_doc_push_script.sh: version (arg2) not specified"
+  exit 1
+fi
+
+is_master_doc=false
+if [ "$version" == "master" ]; then
+  is_master_doc=true
+fi
+
+# Argument 3: The branch to push to. Usually is "site"
+branch="$3"
+if [ -z "$branch" ]; then
+echo "error: python_doc_push_script.sh: branch (arg3) not specified"
+  exit 1
+fi
+
+# Argument 4: (optional) If present, we will NOT do any pushing. Used for testing.
+dry_run=false
+if [ "$4" != "" ]; then
+  dry_run=true
+fi
+
+echo "install_path: $install_path  version: $version  dry_run: $dry_run"
+
+git clone https://github.com/pytorch/pytorch.github.io -b $branch
+pushd pytorch.github.io
+
+export LC_ALL=C
+export PATH=/opt/conda/bin:$PATH
+
+rm -rf pytorch || true
+
+# Install TensorBoard in python 3 so torch.utils.tensorboard classes render
+pip install -q https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py3-none-any.whl
+
+# Get all the documentation sources, put them in one place
+pushd "$pt_checkout"
+git clone https://github.com/pytorch/vision
+pushd vision
+conda install -q pillow
+time python setup.py install
+popd
+pushd docs
+rm -rf source/torchvision
+cp -a ../vision/docs/source source/torchvision
+
+# Build the docs
+pip -q install -r requirements.txt || true
+if [ "$is_master_doc" = true ]; then
+  make html
+else
+  make html-stable
+fi
+
+# Move them into the docs repo
+popd
+popd
+git rm -rf "$install_path" || true
+mv "$pt_checkout/docs/build/html" "$install_path"
+
+# Add the version handler by search and replace.
+# XXX: Consider moving this to the docs Makefile or site build
+if [ "$is_master_doc" = true ]; then
+  find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>\1 \&#x25BC</a>@g"
+else
+  find "$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>$version \&#x25BC</a>@g"
+fi
+
+git add "$install_path" || true
+git status
+git config user.email "soumith+bot@pytorch.org"
+git config user.name "pytorchbot"
+# If there aren't changes, don't make a commit; push is no-op
+git commit -m "auto-generating sphinx docs" || true
+git status
+
+if [ "$dry_run" = false ]; then
+  echo "Pushing to pytorch.github.io:$branch"
+  set +x
+/usr/bin/expect <<DONE
+  spawn git push origin $branch
+  expect "Username*"
+  send "pytorchbot\n"
+  expect "Password*"
+  send "$::env(GITHUB_PYTORCHBOT_TOKEN)\n"
+  expect eof
+DONE
+  set -x
+else
+  echo "Skipping push due to dry_run"
+fi
+
+popd
+# =================== The above code **should** be executed inside Docker container ===================

.circleci/scripts/setup_ci_environment.sh

@@ -1,52 +1,18 @@
 #!/usr/bin/env bash
 set -ex -o pipefail
 
-# Check if we should actually run
-echo "BUILD_ENVIRONMENT: ${BUILD_ENVIRONMENT}"
-echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
-if [[ "${BUILD_ENVIRONMENT}" == *-slow-* ]]; then
-  if ! [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
-    # It's a PR; test for [slow ci] tag on the TOPMOST commit
-    topmost_commit=$(git log --format='%B' -n 1 HEAD)
-    if !(echo $topmost_commit | grep -q -e '\[slow ci\]' -e '\[ci slow\]' -e '\[test slow\]' -e '\[slow test\]'); then
-      circleci step halt
-      exit
-    fi
-  fi
-fi
-if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
-  if ! [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
-    # It's a PR; test for [xla ci] tag on the TOPMOST commit
-    topmost_commit=$(git log --format='%B' -n 1 HEAD)
-    if !(echo $topmost_commit | grep -q -e '\[xla ci\]' -e '\[ci xla\]' -e '\[test xla\]' -e '\[xla test\]'); then
-      # NB: This doesn't halt everything, just this job.  So
-      # the rest of the workflow will keep going and you need
-      # to make sure you halt there too.  Blegh.
-      circleci step halt
-      exit
-    fi
-  fi
-fi
-if [[ "${BUILD_ENVIRONMENT}" == *namedtensor* ]]; then
-  if ! [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
-    # It's a PR; test for [namedtensor] tag on the TOPMOST commit
-    topmost_commit=$(git log --format='%B' -n 1 HEAD)
-    if !(echo $topmost_commit | grep -q -e '\[namedtensor\]' -e '\[ci namedtensor\]' -e '\[namedtensor ci\]'); then
-      # NB: This doesn't halt everything, just this job.  So
-      # the rest of the workflow will keep going and you need
-      # to make sure you halt there too.  Blegh.
-      circleci step halt
-      exit
-    fi
-  fi
-fi
-
 # Set up NVIDIA docker repo
 curl -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
 echo "deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
 echo "deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
 echo "deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
 
+# Remove unnecessary sources
+sudo rm -f /etc/apt/sources.list.d/google-chrome.list
+sudo rm -f /etc/apt/heroku.list
+sudo rm -f /etc/apt/openjdk-r-ubuntu-ppa-xenial.list
+sudo rm -f /etc/apt/partner.list
+
 sudo apt-get -y update
 sudo apt-get -y remove linux-image-generic linux-headers-generic linux-generic docker-ce
 # WARNING: Docker version is hardcoded here; you must update the
@@ -72,10 +38,14 @@ sudo apt-get -y install \
 
 sudo pkill -SIGHUP dockerd
 
-sudo pip -q install awscli==1.16.35
+retry () {
+    $*  || $* || $* || $* || $*
+}
+
+retry sudo pip -q install awscli==1.16.35
 
 if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
-  DRIVER_FN="NVIDIA-Linux-x86_64-410.104.run"
+  DRIVER_FN="NVIDIA-Linux-x86_64-430.40.run"
   wget "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"
   sudo /bin/bash "$DRIVER_FN" -s --no-drm || (sudo cat /var/log/nvidia-installer.log && false)
   nvidia-smi
@@ -84,7 +54,6 @@ fi
 if [[ "${BUILD_ENVIRONMENT}" == *-build ]]; then
   echo "declare -x IN_CIRCLECI=1" > /home/circleci/project/env
   echo "declare -x COMMIT_SOURCE=${CIRCLE_BRANCH:-}" >> /home/circleci/project/env
-  echo "declare -x PYTHON_VERSION=${PYTHON_VERSION:-}" >> /home/circleci/project/env
   echo "declare -x SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2" >> /home/circleci/project/env
   if [ -n "${USE_CUDA_DOCKER_RUNTIME:-}" ]; then
     echo "declare -x TORCH_CUDA_ARCH_LIST=5.2" >> /home/circleci/project/env
@@ -97,12 +66,14 @@ if [[ "${BUILD_ENVIRONMENT}" == *-build ]]; then
   if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
     # This IAM user allows write access to S3 bucket for sccache & bazels3cache
     set +x
+    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
     echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
     echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_AND_XLA_BAZEL_S3_BUCKET_V2:-}" >> /home/circleci/project/env
     set -x
   else
     # This IAM user allows write access to S3 bucket for sccache
     set +x
+    echo "declare -x XLA_CLANG_CACHE_S3_BUCKET_NAME=${XLA_CLANG_CACHE_S3_BUCKET_NAME:-}" >> /home/circleci/project/env
     echo "declare -x AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
     echo "declare -x AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4:-}" >> /home/circleci/project/env
     set -x

.circleci/scripts/setup_linux_system_environment.sh

@@ -29,10 +29,22 @@ done
 # See if we actually were successful
 systemctl list-units --all | cat
 
+# For good luck, try even harder to kill apt-get
+sudo pkill apt-get || true
+
+# For even better luck, purge unattended-upgrades
 sudo apt-get purge -y unattended-upgrades
 
 cat /etc/apt/sources.list
 
-# Bail out early if we detect apt/dpkg is stuck
-ps auxfww | (! grep '[a]pt')
-ps auxfww | (! grep '[d]pkg')
+# For the bestest luck, kill again now
+sudo pkill apt || true
+sudo pkill dpkg || true
+
+# Try to detect if apt/dpkg is stuck
+if ps auxfww | grep '[a]pt'; then
+  echo "WARNING: There are leftover apt processes; subsequent apt update will likely fail"
+fi
+if ps auxfww | grep '[d]pkg'; then
+  echo "WARNING: There are leftover dpkg processes; subsequent apt update will likely fail"
+fi

.circleci/scripts/should_run_job.py

@@ -0,0 +1,132 @@
+import argparse
+import re
+import sys
+
+# Modify this variable if you want to change the set of default jobs
+# which are run on all pull requests.
+#
+# WARNING: Actually, this is a lie; we're currently also controlling
+# the set of jobs to run via the Workflows filters in CircleCI config.
+
+default_set = set([
+    # PyTorch CPU
+    # Selected oldest Python 2 version to ensure Python 2 coverage
+    'pytorch-linux-xenial-py2.7.9',
+    # PyTorch CUDA
+    'pytorch-linux-xenial-cuda9-cudnn7-py3',
+    # PyTorch ASAN
+    'pytorch-linux-xenial-py3-clang5-asan',
+    # PyTorch DEBUG
+    'pytorch-linux-xenial-py3.6-gcc5.4',
+
+    # Caffe2 CPU
+    'caffe2-py2-mkl-ubuntu16.04',
+    # Caffe2 CUDA
+    'caffe2-py3.5-cuda10.1-cudnn7-ubuntu16.04',
+    # Caffe2 ONNX
+    'caffe2-onnx-py2-gcc5-ubuntu16.04',
+    'caffe2-onnx-py3.6-clang7-ubuntu16.04',
+    # Caffe2 Clang
+    'caffe2-py2-clang7-ubuntu16.04',
+    # Caffe2 CMake
+    'caffe2-cmake-cuda9.0-cudnn7-ubuntu16.04',
+
+    # Binaries
+    'manywheel 2.7mu cpu devtoolset7',
+    'libtorch 2.7m cpu devtoolset7',
+    'libtorch 2.7m cpu gcc5.4_cxx11-abi',
+    'libtorch-ios-10.2.1-nightly-x86_64-build',
+    'libtorch-ios-10.2.1-nightly-arm64-build',
+    'libtorch-ios-10.2.1-nightly-binary-build-upload',
+
+    # Caffe2 Android
+    'caffe2-py2-android-ubuntu16.04',
+    # Caffe2 OSX
+    'caffe2-py2-system-macos10.13',
+    # PyTorch OSX
+    'pytorch-macos-10.13-py3',
+    'pytorch-macos-10.13-cuda9.2-cudnn7-py3',
+    # PyTorch Android
+    'pytorch-linux-xenial-py3-clang5-android-ndk-r19c-x86_32-build',
+    # PyTorch Android gradle
+    'pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-only-x86_32',
+    # Pytorch iOS builds
+    'pytorch-ios-10.2.1-x86_64_build',
+    'pytorch-ios-10.2.1-arm64_build',
+
+    # Pytorch backward compatibility check
+    'pytorch-linux-backward-compatibility-check-test',
+
+    # XLA
+    'pytorch-xla-linux-xenial-py3.6-clang7',
+
+    # Named tensor
+    "pytorch-namedtensor-linux-xenial-py3.6-gcc5.4",
+    "pytorch-namedtensor-linux-xenial-py3-clang5-asan",
+    "pytorch-namedtensor-linux-xenial-cuda9-cudnn7-py2",
+
+    # Other checks
+    'pytorch-short-perf-test-gpu',
+    'pytorch-python-doc-push',
+    'pytorch-cpp-doc-push',
+])
+
+# Collection of jobs that are *temporarily* excluded from running on PRs.
+# Use this if there is a long-running job breakage that we can't fix with a
+# single revert.
+skip_override = {
+    # example entry:
+    # 'pytorch-cpp-doc-push': "https://github.com/pytorch/pytorch/issues/<related issue>"
+}
+
+# Takes in commit message to analyze via stdin
+#
+# This script will query Git and attempt to determine if we should
+# run the current CI job under question
+#
+# NB: Try to avoid hard-coding names here, so there's less place to update when jobs
+# are updated/renamed
+#
+# Semantics in the presence of multiple tags:
+#   - Let D be the set of default builds
+#   - Let S be the set of explicitly specified builds
+#   - Let O be the set of temporarily skipped builds
+#   - Run S \/ (D - O)
+
+parser = argparse.ArgumentParser()
+parser.add_argument('build_environment')
+args = parser.parse_args()
+
+commit_msg = sys.stdin.read()
+
+# Matches anything that looks like [foo ci] or [ci foo] or [foo test]
+# or [test foo]
+RE_MARKER = re.compile(r'\[(?:([^ \[\]]+) )?(?:ci|test)(?: ([^ \[\]]+))?\]')
+
+markers = RE_MARKER.finditer(commit_msg)
+
+for m in markers:
+    if m.group(1) and m.group(2):
+        print("Unrecognized marker: {}".format(m.group(0)))
+        continue
+    spec = m.group(1) or m.group(2)
+    if spec is None:
+        print("Unrecognized marker: {}".format(m.group(0)))
+        continue
+    if spec in args.build_environment or spec == 'all':
+        print("Accepting {} due to commit marker {}".format(args.build_environment, m.group(0)))
+        sys.exit(0)
+
+skip_override_set = set(skip_override.keys())
+should_run_set = default_set - skip_override_set
+for spec in should_run_set:
+    if spec in args.build_environment:
+        print("Accepting {} as part of default set".format(args.build_environment))
+        sys.exit(0)
+
+print("Rejecting {}".format(args.build_environment))
+for spec, issue in skip_override.items():
+    if spec in args.build_environment:
+        print("This job is temporarily excluded from running on PRs. Reason: {}".format(issue))
+        break
+sys.exit(1)

.circleci/scripts/should_run_job.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+set -exu -o pipefail
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+# Check if we should actually run
+echo "BUILD_ENVIRONMENT: ${BUILD_ENVIRONMENT:-}"
+echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
+if [ -z "${BUILD_ENVIRONMENT:-}" ]; then
+  echo "Cannot run should_run_job.sh if BUILD_ENVIRONMENT is not defined!"
+  echo "CircleCI scripts are probably misconfigured."
+  exit 1
+fi
+if ! [ -e "$SCRIPT_DIR/COMMIT_MSG" ]; then
+  echo "Cannot run should_run_job.sh if you don't have COMMIT_MSG"
+  echo "written out.  Are you perhaps running the wrong copy of this script?"
+  echo "You should be running the copy in ~/workspace; SCRIPT_DIR=$SCRIPT_DIR"
+  exit 1
+fi
+if [ -n "${CIRCLE_PULL_REQUEST:-}" ]; then
+  if [[ $CIRCLE_BRANCH != "ci-all/"* ]] && [[ $CIRCLE_BRANCH != "nightly" ]] &&  [[ $CIRCLE_BRANCH != "postnightly" ]] ; then
+    # Don't swallow "script doesn't exist
+    [ -e "$SCRIPT_DIR/should_run_job.py"  ]
+    if ! python "$SCRIPT_DIR/should_run_job.py" "${BUILD_ENVIRONMENT:-}" < "$SCRIPT_DIR/COMMIT_MSG" ; then
+      circleci step halt
+      exit
+    fi
+  fi
+fi

.circleci/validate-docker-version.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+
+import urllib.request
+import re
+
+import cimodel.data.pytorch_build_definitions as pytorch_build_definitions
+import cimodel.data.caffe2_build_definitions as caffe2_build_definitions
+
+RE_VERSION = re.compile(r'allDeployedVersions = "([0-9,]+)"')
+
+URL_TEMPLATE = (
+    "https://raw.githubusercontent.com/pytorch/ossci-job-dsl/"
+    "master/src/main/groovy/ossci/{}/DockerVersion.groovy"
+)
+
+def check_version(job, expected_version):
+    url = URL_TEMPLATE.format(job)
+    with urllib.request.urlopen(url) as f:
+        contents = f.read().decode('utf-8')
+        m = RE_VERSION.search(contents)
+        if not m:
+            raise RuntimeError(
+                "Unbelievable! I could not find the variable allDeployedVersions in "
+                "{}; did the organization of ossci-job-dsl change?\n\nFull contents:\n{}"
+                .format(url, contents)
+            )
+        valid_versions = [int(v) for v in m.group(1).split(',')]
+        if expected_version not in valid_versions:
+            raise RuntimeError(
+                "We configured {} to use Docker version {}; but this "
+                "version is not deployed in {}.  Non-deployed versions will be "
+                "garbage collected two weeks after they are created.  DO NOT LAND "
+                "THIS TO MASTER without also updating ossci-job-dsl with this version."
+                "\n\nDeployed versions: {}"
+                .format(job, expected_version, url, m.group(1))
+            )
+
+def validate_docker_version():
+    check_version('pytorch', pytorch_build_definitions.DOCKER_IMAGE_VERSION)
+    check_version('caffe2', caffe2_build_definitions.DOCKER_IMAGE_VERSION)
+
+
+if __name__ == "__main__":
+    validate_docker_version()

.circleci/verbatim-sources/binary-build-params.yml

@@ -0,0 +1,54 @@
+binary_linux_build_params: &binary_linux_build_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+    docker_image:
+      type: string
+      default: ""
+    libtorch_variant:
+      type: string
+      default: ""
+    resource_class:
+      type: string
+      default: "2xlarge+"
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+    LIBTORCH_VARIANT: << parameters.libtorch_variant >>
+    ANACONDA_USER: pytorch
+  resource_class: << parameters.resource_class >>
+  docker:
+    - image: << parameters.docker_image >>
+
+binary_linux_test_upload_params: &binary_linux_test_upload_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+    docker_image:
+      type: string
+      default: ""
+    libtorch_variant:
+      type: string
+      default: ""
+    resource_class:
+      type: string
+      default: "medium"
+    use_cuda_docker_runtime:
+      type: string
+      default: ""
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+    DOCKER_IMAGE: << parameters.docker_image >>
+    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
+    LIBTORCH_VARIANT: << parameters.libtorch_variant >>
+  resource_class: << parameters.resource_class >>
+
+binary_mac_params: &binary_mac_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+

.circleci/verbatim-sources/binary-job-specs.yml

@@ -0,0 +1,261 @@
+  binary_linux_build:
+    <<: *binary_linux_build_params
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_populate_env
+    - run:
+        name: Install unbuffer and ts
+        command: |
+            set -eux -o pipefail
+            source /env
+            OS_NAME=`awk -F= '/^NAME/{print $2}' /etc/os-release`
+            if [[ "$OS_NAME" == *"CentOS Linux"* ]]; then
+              retry yum -q -y install epel-release
+              retry yum -q -y install expect moreutils
+            elif [[ "$OS_NAME" == *"Ubuntu"* ]]; then
+              retry apt-get update
+              retry apt-get -y install expect moreutils
+              conda install -y -c eumetsat expect
+              conda install -y cmake
+            fi
+    - run:
+        name: Update compiler to devtoolset7
+        command: |
+            set -eux -o pipefail
+            source /env
+            if [[ "$DESIRED_DEVTOOLSET" == 'devtoolset7' ]]; then
+              source "/builder/update_compiler.sh"
+
+              # Env variables are not persisted into the next step
+              echo "export PATH=$PATH" >> /env
+              echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> /env
+            else
+              echo "Not updating compiler"
+            fi
+    - run:
+        name: Build
+        no_output_timeout: "1h"
+        command: |
+            source "/pytorch/.circleci/scripts/binary_linux_build.sh"
+    - persist_to_workspace:
+        root: /
+        paths: final_pkgs
+
+    # This should really just be another step of the binary_linux_build job above.
+    # This isn't possible right now b/c the build job uses the docker executor
+    # (otherwise they'd be really really slow) but this one uses the macine
+    # executor (b/c we have to run the docker with --runtime=nvidia and we can't do
+    # that on the docker executor)
+  binary_linux_test:
+    <<: *binary_linux_test_upload_params
+    machine:
+        image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    # TODO: We shouldn't attach the workspace multiple times
+    - attach_workspace:
+        at: /home/circleci/project
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_populate_env
+    - run:
+        name: Prepare test code
+        no_output_timeout: "1h"
+        command: ~/workspace/.circleci/scripts/binary_linux_test.sh
+    - run:
+        <<: *binary_run_in_docker
+
+  binary_linux_upload:
+    <<: *binary_linux_test_upload_params
+    machine:
+        image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - attach_workspace:
+        at: /home/circleci/project
+    - run:
+        <<: *binary_populate_env
+    - run:
+        <<: *binary_install_miniconda
+    - run:
+        name: Upload
+        no_output_timeout: "1h"
+        command: ~/workspace/.circleci/scripts/binary_linux_upload.sh
+
+  # Nighlty build smoke tests defaults
+  # These are the second-round smoke tests. These make sure that the binaries are
+  # correct from a user perspective, testing that they exist from the cloud are
+  # are runnable. Note that the pytorch repo is never cloned into these jobs
+  ##############################################################################
+  smoke_linux_test:
+    <<: *binary_linux_test_upload_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - attach_workspace:
+        at: ~/workspace
+    - attach_workspace:
+        at: /home/circleci/project
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_populate_env
+    - run:
+        name: Test
+        no_output_timeout: "1h"
+        command: |
+          set -ex
+          cat >/home/circleci/project/ci_test_script.sh \<<EOL
+          # The following code will be executed inside Docker container
+          set -eux -o pipefail
+          /builder/smoke_test.sh
+          # The above code will be executed inside Docker container
+          EOL
+    - run:
+        <<: *binary_run_in_docker
+
+  smoke_mac_test:
+    <<: *binary_linux_test_upload_params
+    macos:
+      xcode: "9.0"
+    steps:
+      - attach_workspace:
+          at: ~/workspace
+      - attach_workspace: # TODO - we can `cp` from ~/workspace
+          at: /Users/distiller/project
+      - run:
+          <<: *binary_checkout
+      - run:
+          <<: *binary_populate_env
+      - brew_update
+      - run:
+          <<: *binary_install_miniconda
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -ex
+            source "/Users/distiller/project/env"
+            export "PATH=$workdir/miniconda/bin:$PATH"
+            # TODO unbuffer and ts this, but it breaks cause miniconda overwrites
+            # tclsh. But unbuffer and ts aren't that important so they're just
+            # disabled for now
+            ./builder/smoke_test.sh
+
+  binary_mac_build:
+    <<: *binary_mac_params
+    macos:
+      xcode: "9.0"
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_populate_env
+    - brew_update
+    - run:
+        <<: *binary_install_miniconda
+
+    - run:
+        name: Build
+        no_output_timeout: "1h"
+        command: |
+          # Do not set -u here; there is some problem with CircleCI
+          # variable expansion with PROMPT_COMMAND
+          set -ex -o pipefail
+          script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_build.sh"
+          cat "$script"
+          source "$script"
+
+    - run:
+        name: Test
+        no_output_timeout: "1h"
+        command: |
+          # Do not set -u here; there is some problem with CircleCI
+          # variable expansion with PROMPT_COMMAND
+          set -ex -o pipefail
+          script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_test.sh"
+          cat "$script"
+          source "$script"
+
+    - persist_to_workspace:
+        root: /Users/distiller/project
+        paths: final_pkgs
+
+  binary_mac_upload: &binary_mac_upload
+    <<: *binary_mac_params
+    macos:
+      xcode: "9.0"
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - run:
+        <<: *binary_checkout
+    - run:
+        <<: *binary_populate_env
+    - brew_update
+    - run:
+        <<: *binary_install_miniconda
+    - attach_workspace: # TODO - we can `cp` from ~/workspace
+        at: /Users/distiller/project
+    - run:
+        name: Upload
+        no_output_timeout: "10m"
+        command: |
+          script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_upload.sh"
+          cat "$script"
+          source "$script"
+
+  binary_ios_build:
+    <<: *pytorch_ios_params
+    macos:
+      xcode: "10.2.1"
+    steps:
+    - attach_workspace:
+        at: ~/workspace
+    - should_run_job
+    - checkout
+    - run_brew_for_ios_build
+    - run:
+        name: Build
+        contxt: org-member
+        no_output_timeout: "1h"
+        command: |
+          script="/Users/distiller/project/.circleci/scripts/binary_ios_build.sh"
+          cat "$script"
+          source "$script"
+    - persist_to_workspace:
+        root: /Users/distiller/workspace/
+        paths: ios
+  
+  binary_ios_upload: 
+    <<: *pytorch_ios_params
+    macos:
+      xcode: "10.2.1"
+    steps:
+    - attach_workspace:
+        at: ~/workspace
+    - should_run_job
+    - checkout
+    - run_brew_for_ios_build
+    - run:
+        name: Upload
+        no_output_timeout: "1h"
+        command: |
+          script="/Users/distiller/project/.circleci/scripts/binary_ios_upload.sh"
+          cat "$script"
+          source "$script"

.circleci/verbatim-sources/binary_update_htmls.yml

@@ -1,4 +1,4 @@
-  
+
   # update_s3_htmls job
   # These jobs create html files for every cpu/cu## folder in s3. The html
   # files just store the names of all the files in that folder (which are
@@ -12,8 +12,7 @@
     steps:
     - attach_workspace:
         at: ~/workspace
-    - run:
-        <<: *setup_linux_system_environment
+    - setup_linux_system_environment
     - run:
         <<: *binary_checkout
     # N.B. we do not run binary_populate_env. The only variable we need is
@@ -67,8 +66,7 @@
     steps:
     - attach_workspace:
         at: ~/workspace
-    - run:
-        <<: *setup_linux_system_environment
+    - setup_linux_system_environment
     - run:
         <<: *binary_checkout
     - run:

.circleci/verbatim-sources/caffe2-build-params.yml

@@ -0,0 +1,28 @@
+caffe2_params: &caffe2_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+    build_ios:
+      type: string
+      default: ""
+    docker_image:
+      type: string
+      default: ""
+    use_cuda_docker_runtime:
+      type: string
+      default: ""
+    build_only:
+      type: string
+      default: ""
+    resource_class:
+      type: string
+      default: "large"
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+    BUILD_IOS: << parameters.build_ios >>
+    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
+    DOCKER_IMAGE: << parameters.docker_image >>
+    BUILD_ONLY: << parameters.build_only >>
+  resource_class: << parameters.resource_class >>
+

.circleci/verbatim-sources/caffe2-job-specs.yml

@@ -0,0 +1,200 @@
+  caffe2_linux_build:
+    <<: *caffe2_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: Build
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          cat >/home/circleci/project/ci_build_script.sh \<<EOL
+          # =================== The following code will be executed inside Docker container ===================
+          set -ex
+          export BUILD_ENVIRONMENT="$BUILD_ENVIRONMENT"
+
+          # Reinitialize submodules
+          git submodule sync && git submodule update -q --init --recursive
+
+          # conda must be added to the path for Anaconda builds (this location must be
+          # the same as that in install_anaconda.sh used to build the docker image)
+          if [[ "${BUILD_ENVIRONMENT}" == conda* ]]; then
+            export PATH=/opt/conda/bin:$PATH
+            sudo chown -R jenkins:jenkins '/opt/conda'
+          fi
+
+          # Build
+          ./.jenkins/caffe2/build.sh
+
+          # Show sccache stats if it is running
+          if pgrep sccache > /dev/null; then
+            sccache --show-stats
+          fi
+          # =================== The above code will be executed inside Docker container ===================
+          EOL
+          chmod +x /home/circleci/project/ci_build_script.sh
+
+          echo "DOCKER_IMAGE: "${DOCKER_IMAGE}
+          time docker pull ${DOCKER_IMAGE} >/dev/null
+          export id=$(docker run -t -d -w /var/lib/jenkins ${DOCKER_IMAGE})
+          docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_build_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Push intermediate Docker image for next phase to use
+          if [ -z "${BUILD_ONLY}" ]; then
+            if [[ "$BUILD_ENVIRONMENT" == *cmake* ]]; then
+              export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-cmake-${CIRCLE_SHA1}
+            else
+              export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+            fi
+            docker commit "$id" ${COMMIT_DOCKER_IMAGE}
+            time docker push ${COMMIT_DOCKER_IMAGE}
+          fi
+
+  caffe2_linux_test:
+    <<: *caffe2_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        name: Test
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          # TODO: merge this into Caffe2 test.sh
+          cat >/home/circleci/project/ci_test_script.sh \<<EOL
+          # =================== The following code will be executed inside Docker container ===================
+          set -ex
+
+          export BUILD_ENVIRONMENT="$BUILD_ENVIRONMENT"
+
+          # libdc1394 (dependency of OpenCV) expects /dev/raw1394 to exist...
+          sudo ln /dev/null /dev/raw1394
+
+          # conda must be added to the path for Anaconda builds (this location must be
+          # the same as that in install_anaconda.sh used to build the docker image)
+          if [[ "${BUILD_ENVIRONMENT}" == conda* ]]; then
+            export PATH=/opt/conda/bin:$PATH
+          fi
+
+          # Upgrade SSL module to avoid old SSL warnings
+          pip -q install --user --upgrade pyOpenSSL ndg-httpsclient pyasn1
+
+          pip -q install --user -b /tmp/pip_install_onnx "file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx"
+
+          # Build
+          ./.jenkins/caffe2/test.sh
+
+          # Remove benign core dumps.
+          # These are tests for signal handling (including SIGABRT).
+          rm -f ./crash/core.fatal_signal_as.*
+          rm -f ./crash/core.logging_test.*
+          # =================== The above code will be executed inside Docker container ===================
+          EOL
+          chmod +x /home/circleci/project/ci_test_script.sh
+
+          if [[ "$BUILD_ENVIRONMENT" == *cmake* ]]; then
+            export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-cmake-${CIRCLE_SHA1}
+          else
+            export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          fi
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
+            export id=$(docker run --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+          else
+            export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+          fi
+          docker cp /home/circleci/project/. "$id:/var/lib/jenkins/workspace"
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_test_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+  caffe2_macos_build:
+    <<: *caffe2_params
+    macos:
+      xcode: "9.0"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - should_run_job
+      - checkout
+      - run_brew_for_macos_build
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+
+            export IN_CIRCLECI=1
+
+            brew install cmake
+
+            # Reinitialize submodules
+            git submodule sync && git submodule update -q --init --recursive
+
+            # Reinitialize path (see man page for path_helper(8))
+            eval `/usr/libexec/path_helper -s`
+
+            export PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:$PATH
+
+            # Install Anaconda if we need to
+            if [ -n "${CAFFE2_USE_ANACONDA}" ]; then
+              rm -rf ${TMPDIR}/anaconda
+              curl -o ${TMPDIR}/conda.sh https://repo.continuum.io/miniconda/Miniconda${ANACONDA_VERSION}-latest-MacOSX-x86_64.sh
+              chmod +x ${TMPDIR}/conda.sh
+              /bin/bash ${TMPDIR}/conda.sh -b -p ${TMPDIR}/anaconda
+              rm -f ${TMPDIR}/conda.sh
+              export PATH="${TMPDIR}/anaconda/bin:${PATH}"
+              source ${TMPDIR}/anaconda/bin/activate
+            fi
+
+            pip -q install numpy
+
+            # Install sccache
+            sudo curl https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
+            sudo chmod +x /usr/local/bin/sccache
+            export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
+
+            # This IAM user allows write access to S3 bucket for sccache
+            set +x
+            export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
+            set -x
+
+            export SCCACHE_BIN=${PWD}/sccache_bin
+            mkdir -p ${SCCACHE_BIN}
+            if which sccache > /dev/null; then
+              printf "#!/bin/sh\nexec sccache $(which clang++) \$*" > "${SCCACHE_BIN}/clang++"
+              chmod a+x "${SCCACHE_BIN}/clang++"
+
+              printf "#!/bin/sh\nexec sccache $(which clang) \$*" > "${SCCACHE_BIN}/clang"
+              chmod a+x "${SCCACHE_BIN}/clang"
+
+              export PATH="${SCCACHE_BIN}:$PATH"
+            fi
+
+            # Build
+            if [ "${BUILD_IOS:-0}" -eq 1 ]; then
+              unbuffer scripts/build_ios.sh 2>&1 | ts
+            elif [ -n "${CAFFE2_USE_ANACONDA}" ]; then
+              # All conda build logic should be in scripts/build_anaconda.sh
+              unbuffer scripts/build_anaconda.sh 2>&1 | ts
+            else
+              unbuffer scripts/build_local.sh 2>&1 | ts
+            fi
+
+            # Show sccache stats if it is running
+            if which sccache > /dev/null; then
+              sccache --show-stats
+            fi

.circleci/verbatim-sources/commands.yml

@@ -0,0 +1,90 @@
+commands:
+  # NB: This command must be run as the first command in a job. It
+  # attaches the workspace at ~/workspace; this workspace is generated
+  # by the setup job. Note that ~/workspace is not the default working
+  # directory (that's ~/project).
+  should_run_job:
+    description: "Test if the job should run or not"
+    steps:
+      - attach_workspace:
+          name: Attaching workspace
+          at: ~/workspace
+      - run:
+          name: Should run job
+          no_output_timeout: "2m"
+          command: ~/workspace/.circleci/scripts/should_run_job.sh
+
+  # This system setup script is meant to run before the CI-related scripts, e.g.,
+  # installing Git client, checking out code, setting up CI env, and
+  # building/testing.
+  setup_linux_system_environment:
+    steps:
+      - run:
+          name: Set Up System Environment
+          no_output_timeout: "1h"
+          command: ~/workspace/.circleci/scripts/setup_linux_system_environment.sh
+
+  setup_ci_environment:
+    steps:
+      - run:
+          name: Set Up CI Environment After attach_workspace
+          no_output_timeout: "1h"
+          command: ~/workspace/.circleci/scripts/setup_ci_environment.sh
+
+  brew_update:
+    description: "Update Homebrew and install base formulae"
+    steps:
+      - run:
+          name: Update Homebrew
+          no_output_timeout: "10m"
+          command: |
+            set -ex
+
+            # Update repositories manually.
+            # Running `brew update` produces a comparison between the
+            # current checkout and the updated checkout, which takes a
+            # very long time because the existing checkout is 2y old.
+            for path in $(find /usr/local/Homebrew -type d -name .git)
+            do
+            cd $path/..
+            git fetch --depth=1 origin
+            git reset --hard origin/master
+            done
+
+            export HOMEBREW_NO_AUTO_UPDATE=1
+
+            # Install expect and moreutils so that we can call `unbuffer` and `ts`.
+            # moreutils installs a `parallel` executable by default, which conflicts
+            # with the executable from the GNU `parallel`, so we must unlink GNU
+            # `parallel` first, and relink it afterwards.
+            brew unlink parallel
+            brew install moreutils
+            brew link parallel --overwrite
+            brew install expect
+
+  brew_install:
+    description: "Install Homebrew formulae"
+    parameters:
+      formulae:
+        type: string
+        default: ""
+    steps:
+      - run:
+          name: Install << parameters.formulae >>
+          no_output_timeout: "10m"
+          command: |
+            set -ex
+            export HOMEBREW_NO_AUTO_UPDATE=1
+            brew install << parameters.formulae >>
+
+  run_brew_for_macos_build:
+    steps:
+      - brew_update
+      - brew_install:
+          formulae: libomp
+
+  run_brew_for_ios_build:
+    steps:
+      - brew_update
+      - brew_install:
+          formulae: libtool

.circleci/verbatim-sources/header-section.yml

@@ -7,6 +7,11 @@
 # and then update DOCKER_IMAGE_VERSION at the top of the following files:
 # * cimodel/data/pytorch_build_definitions.py
 # * cimodel/data/caffe2_build_definitions.py
+# And the inline copies of the variable in
+# * verbatim-sources/job-specs-custom.yml
+#   (grep for DOCKER_IMAGE)
+
+version: 2.1
 
 docker_config_defaults: &docker_config_defaults
   user: jenkins
@@ -14,148 +19,3 @@ docker_config_defaults: &docker_config_defaults
     # This IAM user only allows read-write access to ECR
     aws_access_key_id: ${CIRCLECI_AWS_ACCESS_KEY_FOR_ECR_READ_WRITE_V4}
     aws_secret_access_key: ${CIRCLECI_AWS_SECRET_KEY_FOR_ECR_READ_WRITE_V4}
-
-# This system setup script is meant to run before the CI-related scripts, e.g.,
-# installing Git client, checking out code, setting up CI env, and
-# building/testing.
-setup_linux_system_environment: &setup_linux_system_environment
-  name: Set Up System Environment
-  no_output_timeout: "1h"
-  command: ~/workspace/.circleci/scripts/setup_linux_system_environment.sh
-
-install_doc_push_script: &install_doc_push_script
-  name: Install the doc push script
-  no_output_timeout: "2m"
-  command: |
-    cat >/home/circleci/project/doc_push_script.sh <<EOL
-    # =================== The following code **should** be executed inside Docker container ===================
-
-    # This is where the local pytorch install in the docker image is located
-    pt_checkout="/var/lib/jenkins/workspace"
-
-    # Since we're cat-ing this file, we need to escape all $'s
-    echo "doc_push_script.sh: Invoked with \$*"
-
-    git clone https://yf225:${GITHUB_PYTORCHBOT_TOKEN}@github.com/pytorch/pytorch.github.io -b site
-    pushd pytorch.github.io
-
-    set -ex
-
-    # Argument 1: Where to copy the built documentation to
-    # (pytorch.github.io/$install_path)
-    install_path="\$1"
-    if [ -z "\$install_path" ]; then
-    echo "error: doc_push_script.sh: install_path (arg1) not specified"
-      exit 1
-    fi
-
-    # Argument 2: What version of the docs we are building.
-    version="\$2"
-    if [ -z "\$version" ]; then
-    echo "error: doc_push_script.sh: version (arg2) not specified"
-      exit 1
-    fi
-
-    is_master_doc=false
-    if [ "\$version" == "master" ]; then
-      is_master_doc=true
-    fi
-
-    # Argument 3: (optional) If present, we will NOT do any pushing. Used for testing.
-    dry_run=false
-    if [ "\$3" != "" ]; then
-      dry_run=true
-    fi
-
-    echo "install_path: \$install_path  version: \$version  dry_run: \$dry_run"
-
-    export LC_ALL=C
-    export PATH=/opt/conda/bin:$PATH
-
-    rm -rf pytorch || true
-
-    # Install TensorBoard in python 3 so torch.utils.tensorboard classes render
-    pip install -q https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py3-none-any.whl
-
-    # Get all the documentation sources, put them in one place
-    pushd "\$pt_checkout"
-    git clone https://github.com/pytorch/vision
-    pushd vision
-    conda install -q pillow
-    time python setup.py install
-    popd
-    pushd docs
-    rm -rf source/torchvision
-    cp -a ../vision/docs/source source/torchvision
-
-    # Build the docs
-    pip -q install -r requirements.txt || true
-    if [ "\$is_master_doc" = true ]; then
-      make html
-    else
-      make html-stable
-    fi
-
-    # Move them into the docs repo
-    popd
-    popd
-    git rm -rf "\$install_path" || true
-    mv "\$pt_checkout/docs/build/html" "\$install_path"
-
-    # Add the version handler by search and replace.
-    # XXX: Consider moving this to the docs Makefile or site build
-    if [ "\$is_master_doc" = true ]; then
-      find "\$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>\1 \&#x25BC</a>@g"
-    else
-      find "\$install_path" -name "*.html" -print0 | xargs -0 perl -pi -w -e "s@master\s+\((\d\.\d\.[A-Fa-f0-9]+\+[A-Fa-f0-9]+)\s+\)@<a href='http://pytorch.org/docs/versions.html'>\$version \&#x25BC</a>@g"
-    fi
-
-    git add "\$install_path" || true
-    git status
-    git config user.email "soumith+bot@pytorch.org"
-    git config user.name "pytorchbot"
-    # If there aren't changes, don't make a commit; push is no-op
-    git commit -m "auto-generating sphinx docs" || true
-    git status
-
-    if [ "\$dry_run" = false ]; then
-      echo "Pushing to pytorch.github.io:site"
-      git push origin site
-    else
-      echo "Skipping push due to dry_run"
-    fi
-
-    popd
-    # =================== The above code **should** be executed inside Docker container ===================
-    EOL
-    chmod +x /home/circleci/project/doc_push_script.sh
-
-# `setup_ci_environment` has to be run **after** the ``checkout`` step because
-# it writes into the checkout directory and otherwise git will complain
-# that
-#   Directory (/home/circleci/project) you are trying to checkout to is not empty and not git repository
-setup_ci_environment: &setup_ci_environment
-  name: Set Up CI Environment After Checkout
-  no_output_timeout: "1h"
-  command: ~/workspace/.circleci/scripts/setup_ci_environment.sh
-
-# Installs expect and moreutils so that we can call `unbuffer` and `ts`.
-# Also installs OpenMP
-# !!!!NOTE!!!! this is copied into a binary_macos_brew_update job which is the
-# same but does not install libomp. If you are changing this, consider if you
-# need to change that step as well.
-macos_brew_update: &macos_brew_update
-  name: Brew update and install moreutils, expect and libomp
-  no_output_timeout: "1h"
-  command: |
-    set -ex
-    # moreutils installs a `parallel` executable by default, which conflicts
-    # with the executable from the GNU `parallel`, so we must unlink GNU
-    # `parallel` first, and relink it afterwards
-    brew update
-    brew unlink parallel
-    brew install moreutils
-    brew link parallel --overwrite
-    brew install expect
-    brew install libomp
-

.circleci/verbatim-sources/job-specs-custom.yml

@@ -1,20 +1,17 @@
-  
   pytorch_short_perf_test_gpu:
     environment:
       BUILD_ENVIRONMENT: pytorch-short-perf-test-gpu
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:300"
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:347"
       PYTHON_VERSION: "3.6"
       USE_CUDA_DOCKER_RUNTIME: "1"
     resource_class: gpu.medium
     machine:
       image: ubuntu-1604:201903-01
     steps:
-    - attach_workspace:
-        at: ~/workspace
-    - run:
-        <<: *setup_linux_system_environment
-    - run:
-        <<: *setup_ci_environment
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - setup_ci_environment
     - run:
         name: Perf Test
         no_output_timeout: "1h"
@@ -22,7 +19,7 @@
           set -e
           export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
           echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-          docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
           export id=$(docker run --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
 
           docker cp $id:/var/lib/jenkins/workspace/env /home/circleci/project/env
@@ -36,48 +33,43 @@
           export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/short-perf-test-gpu.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
           echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
 
-  pytorch_doc_push:
+  pytorch_python_doc_push:
     environment:
-      BUILD_ENVIRONMENT: pytorch-doc-push
-      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:300"
+      BUILD_ENVIRONMENT: pytorch-python-doc-push
+      # TODO: stop hardcoding this
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:347"
     resource_class: large
     machine:
       image: ubuntu-1604:201903-01
     steps:
-    - attach_workspace:
-        at: ~/workspace
-    - run:
-        <<: *setup_linux_system_environment
-    - run:
-        <<: *setup_ci_environment
-    - run:
-        <<: *install_doc_push_script
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - setup_ci_environment
     - run:
         name: Doc Build and Push
         no_output_timeout: "1h"
         command: |
-          set -e
+          set -ex
           export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
           echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-          docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
           export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
 
-          docker cp /home/circleci/project/doc_push_script.sh $id:/var/lib/jenkins/workspace/doc_push_script.sh
-
           # master branch docs push
           if [[ "${CIRCLE_BRANCH}" == "master" ]]; then
-            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./doc_push_script.sh docs/master master") | docker exec -u jenkins -i "$id" bash) 2>&1'
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/master master site") | docker exec -u jenkins -i "$id" bash) 2>&1'
 
           # stable release docs push. Due to some circleci limitations, we keep
-          # an eternal PR open (#16502) for merging v1.0.1 -> master for this job.
-          # XXX: The following code is only run on the v1.0.1 branch, which might
+          # an eternal PR open for merging v1.3.0 -> master for this job.
+          # XXX: The following code is only run on the v1.3.0 branch, which might
           # not be exactly the same as what you see here.
-          elif [[ "${CIRCLE_BRANCH}" == "v1.0.1" ]]; then
-            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./doc_push_script.sh docs/stable 1.0.1") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          elif [[ "${CIRCLE_BRANCH}" == "v1.3.0" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/stable 1.3.0 site-v1.3.0 dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
 
           # For open PRs: Do a dry_run of the docs build, don't push build
           else
-            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./doc_push_script.sh docs/master master dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/python_doc_push_script.sh docs/master master site dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
           fi
 
           echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
@@ -85,30 +77,75 @@
           # Save the docs build so we can debug any problems
           export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
           docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
-          docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
+          time docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
+
+  pytorch_cpp_doc_push:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-cpp-doc-push
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py3:347"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        name: Doc Build and Push
+        no_output_timeout: "1h"
+        command: |
+          set -ex
+          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+
+          # master branch docs push
+          if [[ "${CIRCLE_BRANCH}" == "master" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/master master") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # stable release docs push. Due to some circleci limitations, we keep
+          # an eternal PR open (#16502) for merging v1.0.1 -> master for this job.
+          # XXX: The following code is only run on the v1.0.1 branch, which might
+          # not be exactly the same as what you see here.
+          elif [[ "${CIRCLE_BRANCH}" == "v1.0.1" ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/stable 1.0.1") | docker exec -u jenkins -i "$id" bash) 2>&1'
+
+          # For open PRs: Do a dry_run of the docs build, don't push build
+          else
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export GITHUB_PYTORCHBOT_TOKEN=${GITHUB_PYTORCHBOT_TOKEN}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && . ./.circleci/scripts/cpp_doc_push_script.sh docs/master master dry_run") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          fi
+
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Save the docs build so we can debug any problems
+          export DEBUG_COMMIT_DOCKER_IMAGE=${COMMIT_DOCKER_IMAGE}-debug
+          docker commit "$id" ${DEBUG_COMMIT_DOCKER_IMAGE}
+          time docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
 
   pytorch_macos_10_13_py3_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
     macos:
       xcode: "9.0"
     steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - should_run_job
       - checkout
-      - run:
-          <<: *macos_brew_update
+      - run_brew_for_macos_build
       - run:
           name: Build
-          environment:
-            BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
           no_output_timeout: "1h"
           command: |
             set -e
-
             export IN_CIRCLECI=1
 
             # Install sccache
             sudo curl https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
             sudo chmod +x /usr/local/bin/sccache
-
             export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
+
             # This IAM user allows write access to S3 bucket for sccache
             set +x
             export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
@@ -118,54 +155,50 @@
             chmod a+x .jenkins/pytorch/macos-build.sh
             unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
 
-            mkdir -p /Users/distiller/pytorch-ci-env/workspace
-
             # copy with -a to preserve relative structure (e.g., symlinks), and be recursive
-            cp -a /Users/distiller/project/. /Users/distiller/pytorch-ci-env/workspace
+            cp -a ~/project ~/workspace
+
       - persist_to_workspace:
-          root: /Users/distiller/pytorch-ci-env
+          root: ~/workspace
           paths:
-            - "*"
+            - miniconda3
+            - project
 
   pytorch_macos_10_13_py3_test:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
     macos:
       xcode: "9.0"
     steps:
-      - run:
-          name: Prepare workspace
-          command: |
-            sudo mkdir -p /Users/distiller/pytorch-ci-env
-            sudo chmod -R 777 /Users/distiller/pytorch-ci-env
-      - attach_workspace:
-          at: /Users/distiller/pytorch-ci-env
-      - run:
-          <<: *macos_brew_update
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      # This workspace also carries binaries from the build job
+      - should_run_job
+      - run_brew_for_macos_build
       - run:
           name: Test
-          environment:
-            BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
           no_output_timeout: "1h"
           command: |
             set -e
             export IN_CIRCLECI=1
 
             # copy with -a to preserve relative structure (e.g., symlinks), and be recursive
-            cp -a /Users/distiller/pytorch-ci-env/workspace/. /Users/distiller/project
+            cp -a ~/workspace/project/. ~/project
 
             chmod a+x .jenkins/pytorch/macos-test.sh
             unbuffer .jenkins/pytorch/macos-test.sh 2>&1 | ts
 
   pytorch_macos_10_13_cuda9_2_cudnn7_py3_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
     macos:
       xcode: "9.0"
     steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - should_run_job
       - checkout
-      - run:
-          <<: *macos_brew_update
+      - run_brew_for_macos_build
       - run:
           name: Build
-          environment:
-            BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
           no_output_timeout: "1h"
           command: |
             set -e
@@ -202,3 +235,212 @@
             chmod a+x .jenkins/pytorch/macos-build.sh
             unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
 
+  pytorch_android_gradle_build:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+      PYTHON_VERSION: "3.6"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - should_run_job
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: pytorch android gradle build
+        no_output_timeout: "1h"
+        command: |
+          set -eux
+          docker_image_commit=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+
+          docker_image_libtorch_android_x86_32=${docker_image_commit}-android-x86_32
+          docker_image_libtorch_android_x86_64=${docker_image_commit}-android-x86_64
+          docker_image_libtorch_android_arm_v7a=${docker_image_commit}-android-arm-v7a
+          docker_image_libtorch_android_arm_v8a=${docker_image_commit}-android-arm-v8a
+
+          echo "docker_image_commit: "${docker_image_commit}
+          echo "docker_image_libtorch_android_x86_32: "${docker_image_libtorch_android_x86_32}
+          echo "docker_image_libtorch_android_x86_64: "${docker_image_libtorch_android_x86_64}
+          echo "docker_image_libtorch_android_arm_v7a: "${docker_image_libtorch_android_arm_v7a}
+          echo "docker_image_libtorch_android_arm_v8a: "${docker_image_libtorch_android_arm_v8a}
+
+          # x86_32
+          time docker pull ${docker_image_libtorch_android_x86_32} >/dev/null
+          export id_x86_32=$(docker run -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # arm-v7a
+          time docker pull ${docker_image_libtorch_android_arm_v7a} >/dev/null
+          export id_arm_v7a=$(docker run -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_arm_v7a})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_arm_v7a" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir ~/workspace/build_android_install_arm_v7a
+          docker cp $id_arm_v7a:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_arm_v7a
+
+          # x86_64
+          time docker pull ${docker_image_libtorch_android_x86_64} >/dev/null
+          export id_x86_64=$(docker run -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_64})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_x86_64" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir ~/workspace/build_android_install_x86_64
+          docker cp $id_x86_64:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_x86_64
+
+          # arm-v8a
+          time docker pull ${docker_image_libtorch_android_arm_v8a} >/dev/null
+          export id_arm_v8a=$(docker run -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_arm_v8a})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace") | docker exec -u jenkins -i "$id_arm_v8a" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir ~/workspace/build_android_install_arm_v8a
+          docker cp $id_arm_v8a:/var/lib/jenkins/workspace/build_android/install ~/workspace/build_android_install_arm_v8a
+
+          docker cp ~/workspace/build_android_install_arm_v7a $id_x86_32:/var/lib/jenkins/workspace/build_android_install_arm_v7a
+          docker cp ~/workspace/build_android_install_x86_64 $id_x86_32:/var/lib/jenkins/workspace/build_android_install_x86_64
+          docker cp ~/workspace/build_android_install_arm_v8a $id_x86_32:/var/lib/jenkins/workspace/build_android_install_arm_v8a
+
+          # run gradle buildRelease
+          export COMMAND='((echo "source ./workspace/env" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir -p ~/workspace/build_android_artifacts
+          docker cp $id_x86_32:/var/lib/jenkins/workspace/android/artifacts.tgz ~/workspace/build_android_artifacts/
+
+          output_image=$docker_image_libtorch_android_x86_32-gradle
+          docker commit "$id_x86_32" ${output_image}
+          time docker push ${output_image}
+    - store_artifacts:
+        path: ~/workspace/build_android_artifacts/artifacts.tgz
+        destination: artifacts.tgz
+
+  pytorch_android_publish_snapshot:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-publish-snapshot
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+      PYTHON_VERSION: "3.6"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - should_run_job
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: pytorch android gradle build
+        no_output_timeout: "1h"
+        command: |
+          set -eux
+          docker_image_commit=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+
+          docker_image_libtorch_android_x86_32_gradle=${docker_image_commit}-android-x86_32-gradle
+
+          echo "docker_image_commit: "${docker_image_commit}
+          echo "docker_image_libtorch_android_x86_32_gradle: "${docker_image_libtorch_android_x86_32_gradle}
+
+          # x86_32
+          time docker pull ${docker_image_libtorch_android_x86_32_gradle} >/dev/null
+          export id_x86_32=$(docker run -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32_gradle})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "export SONATYPE_NEXUS_USERNAME=${SONATYPE_NEXUS_USERNAME}" && echo "export SONATYPE_NEXUS_PASSWORD=${SONATYPE_NEXUS_PASSWORD}" && echo "export ANDROID_SIGN_KEY=${ANDROID_SIGN_KEY}" && echo "export ANDROID_SIGN_PASS=${ANDROID_SIGN_PASS}" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/publish_android_snapshot.sh") | docker exec -u jenkins -i "$id_x86_32" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          output_image=${docker_image_libtorch_android_x86_32_gradle}-publish-snapshot
+          docker commit "$id_x86_32" ${output_image}
+          time docker push ${output_image}
+
+  pytorch_android_gradle_build-x86_32:
+    environment:
+      BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-only-x86_32
+      DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+      PYTHON_VERSION: "3.6"
+    resource_class: large
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    - should_run_job
+    - run:
+        name: filter out not PR runs
+        no_output_timeout: "5m"
+        command: |
+          echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
+          if [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
+            circleci step halt
+          fi
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: pytorch android gradle build only x86_32 (for PR)
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          docker_image_libtorch_android_x86_32=${DOCKER_IMAGE}-${CIRCLE_SHA1}-android-x86_32
+          echo "docker_image_libtorch_android_x86_32: "${docker_image_libtorch_android_x86_32}
+
+          # x86
+          time docker pull ${docker_image_libtorch_android_x86_32} >/dev/null
+          export id=$(docker run -t -d -w /var/lib/jenkins ${docker_image_libtorch_android_x86_32})
+
+          export COMMAND='((echo "source ./workspace/env" && echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "sudo chown -R jenkins workspace && cd workspace && ./.circleci/scripts/build_android_gradle.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          mkdir -p ~/workspace/build_android_x86_32_artifacts
+          docker cp $id:/var/lib/jenkins/workspace/android/artifacts.tgz ~/workspace/build_android_x86_32_artifacts/
+
+          output_image=${docker_image_libtorch_android_x86_32}-gradle
+          docker commit "$id" ${output_image}
+          time docker push ${output_image}
+    - store_artifacts:
+        path: ~/workspace/build_android_x86_32_artifacts/artifacts.tgz
+        destination: artifacts.tgz
+
+  pytorch_ios_build:
+    <<: *pytorch_ios_params
+    macos:
+      xcode: "10.2.1"
+    steps:
+      # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+      - should_run_job
+      - checkout
+      - run_brew_for_ios_build
+      - run:
+          name: Build
+          no_output_timeout: "1h"
+          command: |
+            set -e
+            export IN_CIRCLECI=1
+            WORKSPACE=/Users/distiller/workspace
+            PROJ_ROOT=/Users/distiller/project
+            export TCLLIBPATH="/usr/local/lib"
+
+            # Install conda
+            curl -o ~/Downloads/conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
+            chmod +x ~/Downloads/conda.sh
+            /bin/bash ~/Downloads/conda.sh -b -p ~/anaconda
+            export PATH="~/anaconda/bin:${PATH}"
+            source ~/anaconda/bin/activate
+            # Install dependencies
+            conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing requests --yes
+            # sync submodules
+            cd ${PROJ_ROOT}
+            git submodule sync
+            git submodule update --init --recursive
+            # export
+            export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
+            # run build script
+            chmod a+x ${PROJ_ROOT}/scripts/build_ios.sh
+            echo "IOS_ARCH: ${IOS_ARCH}"
+            echo "IOS_PLATFORM: ${IOS_PLATFORM}"
+            export BUILD_PYTORCH_MOBILE=1
+            export IOS_ARCH=${IOS_ARCH}
+            export IOS_PLATFORM=${IOS_PLATFORM}
+            unbuffer ${PROJ_ROOT}/scripts/build_ios.sh 2>&1 | ts

.circleci/verbatim-sources/job-specs-setup.yml

@@ -1,4 +1,4 @@
-  
+
   setup:
     docker:
       - image: circleci/python:3.7.3
@@ -8,6 +8,26 @@
           name: Ensure config is up to date
           command: ./ensure-consistency.py
           working_directory: .circleci
+      - run:
+          name: Save commit message
+          command: git log --format='%B' -n 1 HEAD > .circleci/scripts/COMMIT_MSG
+      # Note [Workspace for CircleCI scripts]
+      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+      # In the beginning, you wrote your CI scripts in a
+      # .circleci/config.yml file, and life was good.  Your CI
+      # configurations flourished and multiplied.
+      #
+      # Then one day, CircleCI cometh down high and say, "Your YAML file
+      # is too biggeth, it stresses our servers so."  And thus they
+      # asketh us to smite the scripts in the yml file.
+      #
+      # But you can't just put the scripts in the .circleci folder,
+      # because in some jobs, you don't ever actually checkout the
+      # source repository.  Where you gonna get the scripts from?
+      #
+      # Here's how you do it: you persist .circleci/scripts into a
+      # workspace, attach the workspace in your subjobs, and run all
+      # your scripts from there.
       - persist_to_workspace:
           root: .
           paths: .circleci/scripts

.circleci/verbatim-sources/linux-binary-build-defaults.yml

@@ -1,96 +0,0 @@
-
-# binary linux build defaults
-##############################################################################
-binary_linux_build: &binary_linux_build
-  resource_class: 2xlarge+
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *binary_checkout
-  - run:
-      <<: *binary_populate_env
-  - run:
-      name: Install unbuffer and ts
-      command: |
-        set -eux -o pipefail
-        source /env
-        retry yum -q -y install epel-release
-        retry yum -q -y install expect moreutils
-  - run:
-      name: Upgrade gcc version (based on env var)
-      command: |
-        set -eux -o pipefail
-        source /env
-        if [[ "$DESIRED_DEVTOOLSET" == 'devtoolset7' ]]; then
-          source "/builder/upgrade_gcc_abi.sh"
-
-          # Env variables are not persisted into the next step
-          echo "export PATH=$PATH" >> /env
-          echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> /env
-
-          # We need to set this variable manually because
-          # https://github.com/pytorch/pytorch/blob/master/torch/abi-check.cpp
-          # sets the ABI to 0 by default
-          echo "export _GLIBCXX_USE_CXX11_ABI=1" >> /env
-        else
-          echo "Not upgrading gcc version"
-        fi
-  - run:
-      name: Build
-      no_output_timeout: "1h"
-      command: |
-        source "/pytorch/.circleci/scripts/binary_linux_build.sh"
-  - persist_to_workspace:
-      root: /
-      paths: final_pkgs
-
-# This should really just be another step of the binary_linux_build job above.
-# This isn't possible right now b/c the build job uses the docker executor
-# (otherwise they'd be really really slow) but this one uses the macine
-# executor (b/c we have to run the docker with --runtime=nvidia and we can't do
-# that on the docker executor)
-binary_linux_test: &binary_linux_test
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - attach_workspace:
-      at: /home/circleci/project
-  - run:
-      <<: *setup_linux_system_environment
-  - run:
-      <<: *setup_ci_environment
-  - run:
-      <<: *binary_checkout
-  - run:
-      <<: *binary_populate_env
-  - run:
-      name: Prepare test code
-      no_output_timeout: "1h"
-      command: ~/workspace/.circleci/scripts/binary_linux_test.sh
-  - run:
-      <<: *binary_run_in_docker
-
-binary_linux_upload: &binary_linux_upload
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *setup_linux_system_environment
-  - run:
-      <<: *setup_ci_environment
-  - attach_workspace:
-      at: /home/circleci/project
-  - run:
-      <<: *binary_populate_env
-  - run:
-      <<: *binary_install_miniconda
-  - run:
-      name: Upload
-      no_output_timeout: "1h"
-      command: ~/workspace/.circleci/scripts/binary_linux_upload.sh
-

.circleci/verbatim-sources/linux-build-defaults.yml

@@ -1,218 +0,0 @@
-
-
-##############################################################################
-# Linux build defaults
-##############################################################################
-
-pytorch_linux_build_defaults: &pytorch_linux_build_defaults
-  resource_class: large
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *setup_linux_system_environment
-  - checkout
-  - run:
-      <<: *setup_ci_environment
-  - run:
-      name: Build
-      no_output_timeout: "1h"
-      command: |
-        set -e
-        # Pull Docker image and run build
-        echo "DOCKER_IMAGE: "${DOCKER_IMAGE}
-        docker pull ${DOCKER_IMAGE} >/dev/null
-        export id=$(docker run -t -d -w /var/lib/jenkins ${DOCKER_IMAGE})
-
-        git submodule sync && git submodule update -q --init --recursive
-
-        docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
-
-        if [[ ${BUILD_ENVIRONMENT} == *"namedtensor"* ]]; then
-          NAMED_FLAG="export USE_NAMEDTENSOR=1"
-        fi
-
-        export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo '"$NAMED_FLAG"' && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/build.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-        echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-        # Push intermediate Docker image for next phase to use
-        if [ -z "${BUILD_ONLY}" ]; then
-          # Note [namedtensor build image]
-          # The namedtensor build uses the same docker image as
-          # pytorch-linux-trusty-py3.6-gcc5.4-build. In the push step, we have to
-          # distinguish between these two so the test can pick up the correct image.
-          output_image=${DOCKER_IMAGE}-${CIRCLE_SHA1}
-          if [[ ${BUILD_ENVIRONMENT} == *"namedtensor"* ]]; then
-            export COMMIT_DOCKER_IMAGE=$output_image-namedtensor
-          else
-            export COMMIT_DOCKER_IMAGE=$output_image
-          fi
-          docker commit "$id" ${COMMIT_DOCKER_IMAGE}
-          docker push ${COMMIT_DOCKER_IMAGE}
-        fi
-
-pytorch_linux_test_defaults: &pytorch_linux_test_defaults
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *setup_linux_system_environment
-  - checkout
-  - run:
-      <<: *setup_ci_environment
-  - run:
-      name: Test
-      no_output_timeout: "90m"
-      command: |
-        set -e
-        # See Note [namedtensor build image]
-        output_image=${DOCKER_IMAGE}-${CIRCLE_SHA1}
-        if [[ ${BUILD_ENVIRONMENT} == *"namedtensor"* ]]; then
-          export COMMIT_DOCKER_IMAGE=$output_image-namedtensor
-        else
-          export COMMIT_DOCKER_IMAGE=$output_image
-        fi
-        echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-        docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-        if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
-          export id=$(docker run --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-        else
-          export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-        fi
-        if [ -n "${MULTI_GPU}" ]; then
-          export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/multigpu-test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-        else
-          export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-        fi
-        echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-caffe2_linux_build_defaults: &caffe2_linux_build_defaults
-  resource_class: large
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *setup_linux_system_environment
-  - checkout
-  - run:
-      <<: *setup_ci_environment
-  - run:
-      name: Build
-      no_output_timeout: "1h"
-      command: |
-        set -e
-        cat >/home/circleci/project/ci_build_script.sh <<EOL
-        # =================== The following code will be executed inside Docker container ===================
-        set -ex
-        export BUILD_ENVIRONMENT="$BUILD_ENVIRONMENT"
-
-        # Reinitialize submodules
-        git submodule sync && git submodule update -q --init --recursive
-
-        # conda must be added to the path for Anaconda builds (this location must be
-        # the same as that in install_anaconda.sh used to build the docker image)
-        if [[ "${BUILD_ENVIRONMENT}" == conda* ]]; then
-          export PATH=/opt/conda/bin:$PATH
-          sudo chown -R jenkins:jenkins '/opt/conda'
-        fi
-
-        # Build
-        ./.jenkins/caffe2/build.sh
-
-        # Show sccache stats if it is running
-        if pgrep sccache > /dev/null; then
-          sccache --show-stats
-        fi
-        # =================== The above code will be executed inside Docker container ===================
-        EOL
-        chmod +x /home/circleci/project/ci_build_script.sh
-
-        echo "DOCKER_IMAGE: "${DOCKER_IMAGE}
-        docker pull ${DOCKER_IMAGE} >/dev/null
-        export id=$(docker run -t -d -w /var/lib/jenkins ${DOCKER_IMAGE})
-        docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
-
-        export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_build_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-        echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-
-        # Push intermediate Docker image for next phase to use
-        if [ -z "${BUILD_ONLY}" ]; then
-          if [[ "$BUILD_ENVIRONMENT" == *cmake* ]]; then
-            export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-cmake-${CIRCLE_SHA1}
-          else
-            export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
-          fi
-          docker commit "$id" ${COMMIT_DOCKER_IMAGE}
-          docker push ${COMMIT_DOCKER_IMAGE}
-        fi
-
-caffe2_linux_test_defaults: &caffe2_linux_test_defaults
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *setup_linux_system_environment
-  - run:
-      <<: *setup_ci_environment
-  - run:
-      name: Test
-      no_output_timeout: "1h"
-      command: |
-        set -e
-        # TODO: merge this into Caffe2 test.sh
-        cat >/home/circleci/project/ci_test_script.sh <<EOL
-        # =================== The following code will be executed inside Docker container ===================
-        set -ex
-
-        export BUILD_ENVIRONMENT="$BUILD_ENVIRONMENT"
-
-        # libdc1394 (dependency of OpenCV) expects /dev/raw1394 to exist...
-        sudo ln /dev/null /dev/raw1394
-
-        # conda must be added to the path for Anaconda builds (this location must be
-        # the same as that in install_anaconda.sh used to build the docker image)
-        if [[ "${BUILD_ENVIRONMENT}" == conda* ]]; then
-          export PATH=/opt/conda/bin:$PATH
-        fi
-
-        # Upgrade SSL module to avoid old SSL warnings
-        pip -q install --user --upgrade pyOpenSSL ndg-httpsclient pyasn1
-
-        pip -q install --user -b /tmp/pip_install_onnx "file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx"
-
-        # Build
-        ./.jenkins/caffe2/test.sh
-
-        # Remove benign core dumps.
-        # These are tests for signal handling (including SIGABRT).
-        rm -f ./crash/core.fatal_signal_as.*
-        rm -f ./crash/core.logging_test.*
-        # =================== The above code will be executed inside Docker container ===================
-        EOL
-        chmod +x /home/circleci/project/ci_test_script.sh
-
-        if [[ "$BUILD_ENVIRONMENT" == *cmake* ]]; then
-          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-cmake-${CIRCLE_SHA1}
-        else
-          export COMMIT_DOCKER_IMAGE=${DOCKER_IMAGE}-${CIRCLE_SHA1}
-        fi
-        echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
-        docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
-        if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
-          export id=$(docker run --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-        else
-          export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
-        fi
-        docker cp /home/circleci/project/. "$id:/var/lib/jenkins/workspace"
-
-        export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_test_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
-        echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
-

.circleci/verbatim-sources/macos-binary-build-defaults.yml

@@ -1,66 +0,0 @@
-
-##############################################################################
-# Macos binary build defaults
-# The root of everything is /Users/distiller/pytorch-ci-env/workspace
-##############################################################################
-binary_mac_build: &binary_mac_build
-  macos:
-    xcode: "9.0"
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *binary_checkout
-  - run:
-      <<: *binary_populate_env
-  - run:
-      <<: *binary_macos_brew_update
-  - run:
-      <<: *binary_install_miniconda
-
-  - run:
-      name: Build
-      no_output_timeout: "1h"
-      command: |
-        set -eux -o pipefail
-        script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_build.sh"
-        cat "$script"
-        source "$script"
-
-  - run:
-      name: Test
-      no_output_timeout: "1h"
-      command: |
-        set -eux -o pipefail
-        script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_test.sh"
-        cat "$script"
-        source "$script"
-
-  - persist_to_workspace:
-      root: /Users/distiller/project
-      paths: final_pkgs
-
-binary_mac_upload: &binary_mac_upload
-  macos:
-    xcode: "9.0"
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - run:
-      <<: *binary_checkout
-  - run:
-      <<: *binary_populate_env
-  - run:
-      <<: *binary_macos_brew_update
-  - run:
-      <<: *binary_install_miniconda
-  - attach_workspace: # TODO - we can `cp` from ~/workspace
-      at: /Users/distiller/project
-  - run:
-      name: Upload
-      no_output_timeout: "10m"
-      command: |
-        script="/Users/distiller/project/pytorch/.circleci/scripts/binary_macos_upload.sh"
-        cat "$script"
-        source "$script"
-

.circleci/verbatim-sources/macos-build-defaults.yml

@@ -1,85 +0,0 @@
-
-##############################################################################
-# Macos build defaults
-##############################################################################
-
-caffe2_macos_build_defaults: &caffe2_macos_build_defaults
-  macos:
-    xcode: "9.0"
-  steps:
-    - checkout
-    - run:
-        <<: *macos_brew_update
-    - run:
-        name: Build
-        no_output_timeout: "1h"
-        command: |
-          set -e
-
-          export IN_CIRCLECI=1
-
-          brew install cmake
-
-          # Reinitialize submodules
-          git submodule sync && git submodule update -q --init --recursive
-
-          # Reinitialize path (see man page for path_helper(8))
-          eval `/usr/libexec/path_helper -s`
-
-          # Use Homebrew Python if configured to do so
-          if [ "${PYTHON_INSTALLATION}" == "homebrew" ]; then
-            export PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:$PATH
-          fi
-
-          pip -q install numpy
-
-          # Install Anaconda if we need to
-          if [ -n "${CAFFE2_USE_ANACONDA}" ]; then
-            rm -rf ${TMPDIR}/anaconda
-            curl -o ${TMPDIR}/conda.sh https://repo.continuum.io/miniconda/Miniconda${ANACONDA_VERSION}-latest-MacOSX-x86_64.sh
-            chmod +x ${TMPDIR}/conda.sh
-            /bin/bash ${TMPDIR}/conda.sh -b -p ${TMPDIR}/anaconda
-            rm -f ${TMPDIR}/conda.sh
-            export PATH="${TMPDIR}/anaconda/bin:${PATH}"
-            source ${TMPDIR}/anaconda/bin/activate
-          fi
-
-          # Install sccache
-          sudo curl https://s3.amazonaws.com/ossci-macos/sccache --output /usr/local/bin/sccache
-          sudo chmod +x /usr/local/bin/sccache
-          export SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
-
-          # This IAM user allows write access to S3 bucket for sccache
-          set +x
-          export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_SCCACHE_S3_BUCKET_V4}
-          export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_SCCACHE_S3_BUCKET_V4}
-          set -x
-
-          export SCCACHE_BIN=${PWD}/sccache_bin
-          mkdir -p ${SCCACHE_BIN}
-          if which sccache > /dev/null; then
-            printf "#!/bin/sh\nexec sccache $(which clang++) \$*" > "${SCCACHE_BIN}/clang++"
-            chmod a+x "${SCCACHE_BIN}/clang++"
-
-            printf "#!/bin/sh\nexec sccache $(which clang) \$*" > "${SCCACHE_BIN}/clang"
-            chmod a+x "${SCCACHE_BIN}/clang"
-
-            export PATH="${SCCACHE_BIN}:$PATH"
-          fi
-
-          # Build
-          if [ "${BUILD_IOS:-0}" -eq 1 ]; then
-            unbuffer scripts/build_ios.sh 2>&1 | ts
-          elif [ -n "${CAFFE2_USE_ANACONDA}" ]; then
-            # All conda build logic should be in scripts/build_anaconda.sh
-            unbuffer scripts/build_anaconda.sh 2>&1 | ts
-          else
-            unbuffer scripts/build_local.sh 2>&1 | ts
-          fi
-
-          # Show sccache stats if it is running
-          if which sccache > /dev/null; then
-            sccache --show-stats
-          fi
-
-

.circleci/verbatim-sources/nightly-binary-build-defaults.yml

@@ -48,21 +48,3 @@ binary_run_in_docker: &binary_run_in_docker
   # This step only runs on circleci linux machine executors that themselves
   # need to start docker images
   command: ~/workspace/.circleci/scripts/binary_run_in_docker.sh
-
-# This is copied almost verbatim from the macos_brew_update job
-# In version 2.1 and above we could make this a command and pass a parameter to
-# it, but in this version there is no way to pass a parameter to a step
-binary_macos_brew_update: &binary_macos_brew_update
-  name: Brew update and install moreutils and expect
-  no_output_timeout: "1h"
-  command: |
-    set -eux -o pipefail
-    # moreutils installs a `parallel` executable by default, which conflicts
-    # with the executable from the GNU `parallel`, so we must unlink GNU
-    # `parallel` first, and relink it afterwards
-    brew update
-    brew unlink parallel
-    brew install moreutils
-    brew link parallel --overwrite
-    brew install expect
-

.circleci/verbatim-sources/nightly-build-smoke-tests-defaults.yml

@@ -1,64 +0,0 @@
-
-# Nighlty build smoke tests defaults
-# These are the second-round smoke tests. These make sure that the binaries are
-# correct from a user perspective, testing that they exist from the cloud are
-# are runnable. Note that the pytorch repo is never cloned into these jobs
-##############################################################################
-smoke_linux_test: &smoke_linux_test
-  machine:
-    image: ubuntu-1604:201903-01
-  steps:
-  - attach_workspace:
-      at: ~/workspace
-  - attach_workspace:
-      at: /home/circleci/project
-  - run:
-      <<: *setup_linux_system_environment
-  - run:
-      <<: *setup_ci_environment
-  - run:
-      <<: *binary_checkout
-  - run:
-      <<: *binary_populate_env
-  - run:
-      name: Test
-      no_output_timeout: "1h"
-      command: |
-        set -ex
-        cat >/home/circleci/project/ci_test_script.sh <<EOL
-        # The following code will be executed inside Docker container
-        set -eux -o pipefail
-        /builder/smoke_test.sh
-        # The above code will be executed inside Docker container
-        EOL
-  - run:
-      <<: *binary_run_in_docker
-
-smoke_mac_test: &smoke_mac_test
-  macos:
-    xcode: "9.0"
-  steps:
-    - attach_workspace:
-        at: ~/workspace
-    - attach_workspace: # TODO - we can `cp` from ~/workspace
-        at: /Users/distiller/project
-    - run:
-        <<: *binary_checkout
-    - run:
-        <<: *binary_populate_env
-    - run:
-        <<: *binary_macos_brew_update
-    - run:
-        <<: *binary_install_miniconda
-    - run:
-        name: Build
-        no_output_timeout: "1h"
-        command: |
-          set -ex
-          source "/Users/distiller/project/env"
-          export "PATH=$workdir/miniconda/bin:$PATH"
-          # TODO unbuffer and ts this, but it breaks cause miniconda overwrites
-          # tclsh. But unbuffer and ts aren't that important so they're just
-          # disabled for now
-          ./builder/smoke_test.sh
-

.circleci/verbatim-sources/pytorch-build-params.yml

@@ -0,0 +1,39 @@
+pytorch_params: &pytorch_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+    docker_image:
+      type: string
+      default: ""
+    resource_class:
+      type: string
+      default: "large"
+    use_cuda_docker_runtime:
+      type: string
+      default: ""
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+    DOCKER_IMAGE: << parameters.docker_image >>
+    USE_CUDA_DOCKER_RUNTIME: << parameters.use_cuda_docker_runtime >>
+  resource_class: << parameters.resource_class >>
+
+pytorch_ios_params: &pytorch_ios_params
+  parameters:
+    build_environment:
+      type: string
+      default: ""
+    ios_arch:
+      type: string
+      default: ""
+    ios_platform:
+      type: string
+      default: ""
+  environment:
+    BUILD_ENVIRONMENT: << parameters.build_environment >>
+    IOS_ARCH: << parameters.ios_arch >>
+    IOS_PLATFORM: << parameters.ios_platform >>
+
+
+
+

.circleci/verbatim-sources/pytorch-job-specs.yml

@@ -0,0 +1,117 @@
+jobs:
+  pytorch_linux_build:
+    <<: *pytorch_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - checkout
+    - setup_ci_environment
+    - run:
+        name: Build
+        no_output_timeout: "1h"
+        command: |
+          set -e
+          # Pull Docker image and run build
+          echo "DOCKER_IMAGE: "${DOCKER_IMAGE}
+          time docker pull ${DOCKER_IMAGE} >/dev/null
+          export id=$(docker run -t -d -w /var/lib/jenkins ${DOCKER_IMAGE})
+
+          # TODO We may want to move the rebase logic to a separate step after checkout
+          # Rebase to v1.3.0 only if in xenial_py3_6_gcc5_4 case
+          if [[ "${CIRCLE_BRANCH}" != "v1.3.0" && "${BUILD_ENVIRONMENT}" == *"gcc5"* ]]; then
+            echo "Merge v1.3.0 branch into $CIRCLE_BRANCH before build in environment $BUILD_ENVIRONMENT"
+            set -x
+            git config --global user.email "circleci.ossci@gmail.com"
+            git config --global user.name "CircleCI"
+            git config remote.origin.url https://github.com/pytorch/pytorch.git
+            git config --add remote.origin.fetch +refs/heads/v1.3.0:refs/remotes/origin/v1.3.0
+            git fetch --tags --progress https://github.com/pytorch/pytorch.git +refs/heads/v1.3.0:refs/remotes/origin/v1.3.0 --depth=50 --quiet
+            export GIT_MERGE_TARGET=`git log -n 1 --pretty=format:"%H" origin/v1.3.0`
+            echo "GIT_MERGE_TARGET: " ${GIT_MERGE_TARGET}
+            export GIT_COMMIT=${CIRCLE_SHA1}
+            echo "GIT_COMMIT: " ${GIT_COMMIT}
+            git checkout -f ${GIT_COMMIT}
+            git reset --hard ${GIT_COMMIT}
+            git merge --no-edit --no-ff ${GIT_MERGE_TARGET}
+            set +x
+          else
+            echo "Do NOT merge v1.3.0 branch into $CIRCLE_BRANCH in environment $BUILD_ENVIRONMENT"
+          fi
+
+          git submodule sync && git submodule update -q --init --recursive
+
+          docker cp /home/circleci/project/. $id:/var/lib/jenkins/workspace
+
+          if [[ ${BUILD_ENVIRONMENT} == *"namedtensor"* ]]; then
+            NAMED_FLAG="export BUILD_NAMEDTENSOR=1"
+          fi
+
+          export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo '"$NAMED_FLAG"' && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/build.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts
+
+          # Push intermediate Docker image for next phase to use
+          if [ -z "${BUILD_ONLY}" ]; then
+            # Note [Special build images]
+            # The namedtensor and xla builds use the same docker image as
+            # pytorch-linux-trusty-py3.6-gcc5.4-build. In the push step, we have to
+            # distinguish between them so the test can pick up the correct image.
+            output_image=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+            if [[ ${BUILD_ENVIRONMENT} == *"namedtensor"* ]]; then
+              export COMMIT_DOCKER_IMAGE=$output_image-namedtensor
+            elif [[ ${BUILD_ENVIRONMENT} == *"xla"* ]]; then
+              export COMMIT_DOCKER_IMAGE=$output_image-xla
+            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-x86_64"* ]]; then
+              export COMMIT_DOCKER_IMAGE=$output_image-android-x86_64
+            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-arm-v7a"* ]]; then
+              export COMMIT_DOCKER_IMAGE=$output_image-android-arm-v7a
+            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-arm-v8a"* ]]; then
+              export COMMIT_DOCKER_IMAGE=$output_image-android-arm-v8a
+            elif [[ ${BUILD_ENVIRONMENT} == *"android-ndk-r19c-x86_32"* ]]; then
+              export COMMIT_DOCKER_IMAGE=$output_image-android-x86_32
+            else
+              export COMMIT_DOCKER_IMAGE=$output_image
+            fi
+            docker commit "$id" ${COMMIT_DOCKER_IMAGE}
+            time docker push ${COMMIT_DOCKER_IMAGE}
+          fi
+
+  pytorch_linux_test:
+    <<: *pytorch_params
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+    # See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
+    - should_run_job
+    - setup_linux_system_environment
+    - setup_ci_environment
+    - run:
+        name: Test
+        no_output_timeout: "90m"
+        command: |
+          set -e
+          # See Note [Special build images]
+          output_image=${DOCKER_IMAGE}-${CIRCLE_SHA1}
+          if [[ ${BUILD_ENVIRONMENT} == *"namedtensor"* ]]; then
+            export COMMIT_DOCKER_IMAGE=$output_image-namedtensor
+            export NAMED_FLAG="export BUILD_NAMEDTENSOR=1 && export TEST_NAMEDTENSOR=1"
+          elif [[ ${BUILD_ENVIRONMENT} == *"xla"* ]]; then
+            export COMMIT_DOCKER_IMAGE=$output_image-xla
+          else
+            export COMMIT_DOCKER_IMAGE=$output_image
+          fi
+          echo "DOCKER_IMAGE: "${COMMIT_DOCKER_IMAGE}
+          time docker pull ${COMMIT_DOCKER_IMAGE} >/dev/null
+          if [ -n "${USE_CUDA_DOCKER_RUNTIME}" ]; then
+            export id=$(docker run --runtime=nvidia -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+          else
+            export id=$(docker run -t -d -w /var/lib/jenkins ${COMMIT_DOCKER_IMAGE})
+          fi
+          if [[ ${BUILD_ENVIRONMENT} == *"multigpu"* ]]; then
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "${NAMED_FLAG}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/multigpu-test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          else
+            export COMMAND='((echo "export BUILD_ENVIRONMENT=${BUILD_ENVIRONMENT}" && echo "${NAMED_FLAG}" && echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && .jenkins/pytorch/test.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
+          fi
+          echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts

.circleci/verbatim-sources/workflows-binary-builds-smoke-subset.yml

@@ -5,44 +5,97 @@
       # pytorch-ci-hud to adjust the list of whitelisted builds
       # at https://github.com/ezyang/pytorch-ci-hud/blob/master/src/BuildHistoryDisplay.js
 
-      - binary_linux_manywheel_2.7mu_cpu_devtoolset3_build:
+      - binary_linux_build:
+          name: binary_linux_manywheel_2_7mu_cpu_devtoolset7_build
+          build_environment: "manywheel 2.7mu cpu devtoolset7"
           requires:
             - setup
-      - binary_linux_manywheel_3.7m_cu100_devtoolset3_build:
+          docker_image: "soumith/manylinux-cuda100"
+      - binary_linux_build:
+          name: binary_linux_manywheel_3_7m_cu100_devtoolset7_build
+          build_environment: "manywheel 3.7m cu100 devtoolset7"
           requires:
             - setup
-      - binary_linux_conda_2.7_cpu_build:
+          docker_image: "soumith/manylinux-cuda100"
+      - binary_linux_build:
+          name: binary_linux_conda_2_7_cpu_devtoolset7_build
+          build_environment: "conda 2.7 cpu devtoolset7"
           requires:
             - setup
-      # This binary build is currently broken, see https://github.com/pytorch/pytorch/issues/16710
-      # - binary_linux_conda_3.6_cu90_build
+          docker_image: "soumith/conda-cuda"
+      # This binary build is currently broken, see https://github_com/pytorch/pytorch/issues/16710
+      # - binary_linux_conda_3_6_cu90_devtoolset7_build
+      - binary_linux_build:
+          name: binary_linux_libtorch_2_7m_cpu_devtoolset7_shared-with-deps_build
+          build_environment: "libtorch 2.7m cpu devtoolset7"
+          requires:
+            - setup
+          libtorch_variant: "shared-with-deps"
+          docker_image: "soumith/manylinux-cuda100"
+      - binary_linux_build:
+          name: binary_linux_libtorch_2_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build
+          build_environment: "libtorch 2.7m cpu gcc5.4_cxx11-abi"
+          requires:
+            - setup
+          libtorch_variant: "shared-with-deps"
+          docker_image: "yf225/pytorch-binary-docker-image-ubuntu16.04:latest"
       # TODO we should test a libtorch cuda build, but they take too long
-      # - binary_linux_libtorch_2.7m_cu90_devtoolset3_build
-      - binary_macos_wheel_3.6_cpu_build:
+      # - binary_linux_libtorch_2_7m_cu90_devtoolset7_static-without-deps_build
+      - binary_mac_build:
+          name: binary_macos_wheel_3_6_cpu_build
+          build_environment: "wheel 3.6 cpu"
           requires:
             - setup
-      - binary_macos_conda_2.7_cpu_build:
+      - binary_mac_build:
+          name: binary_macos_conda_2_7_cpu_build
+          build_environment: "conda 2.7 cpu"
           requires:
             - setup
-      - binary_macos_libtorch_2.7_cpu_build:
+      - binary_mac_build:
+          name: binary_macos_libtorch_2_7_cpu_build
+          build_environment: "libtorch 2.7 cpu"
           requires:
             - setup
 
-      - binary_linux_manywheel_2.7mu_cpu_devtoolset3_test:
+      - binary_linux_test:
+          name: binary_linux_manywheel_2_7mu_cpu_devtoolset7_test
+          build_environment: "manywheel 2.7mu cpu devtoolset7"
           requires:
             - setup
-            - binary_linux_manywheel_2.7mu_cpu_devtoolset3_build
-      - binary_linux_manywheel_3.7m_cu100_devtoolset3_test:
+            - binary_linux_manywheel_2_7mu_cpu_devtoolset7_build
+          docker_image: "soumith/manylinux-cuda100"
+      - binary_linux_test:
+          name: binary_linux_manywheel_3_7m_cu100_devtoolset7_test
+          build_environment: "manywheel 3.7m cu100 devtoolset7"
           requires:
             - setup
-            - binary_linux_manywheel_3.7m_cu100_devtoolset3_build
-      - binary_linux_conda_2.7_cpu_test:
+            - binary_linux_manywheel_3_7m_cu100_devtoolset7_build
+          docker_image: "soumith/manylinux-cuda100"
+          use_cuda_docker_runtime: "1"
+          resource_class: gpu.medium
+      - binary_linux_test:
+          name: binary_linux_conda_2_7_cpu_devtoolset7_test
+          build_environment: "conda 2.7 cpu devtoolset7"
           requires:
             - setup
-            - binary_linux_conda_2.7_cpu_build
-      # This binary build is currently broken, see https://github.com/pytorch/pytorch/issues/16710
-      # - binary_linux_conda_3.6_cu90_test:
-      #     requires:
-      #       - setup
-      #       - binary_linux_conda_3.6_cu90_build
+            - binary_linux_conda_2_7_cpu_devtoolset7_build
+          docker_image: "soumith/conda-cuda"
+      # This binary build is currently broken, see https://github_com/pytorch/pytorch/issues/16710
+      # - binary_linux_conda_3_6_cu90_devtoolset7_test:
+      - binary_linux_test:
+          name: binary_linux_libtorch_2_7m_cpu_devtoolset7_shared-with-deps_test
+          build_environment: "libtorch 2.7m cpu devtoolset7"
+          requires:
+            - setup
+            - binary_linux_libtorch_2_7m_cpu_devtoolset7_shared-with-deps_build
+          libtorch_variant: "shared-with-deps"
+          docker_image: "soumith/manylinux-cuda100"
+      - binary_linux_test:
+          name: binary_linux_libtorch_2_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_test
+          build_environment: "libtorch 2.7m cpu gcc5.4_cxx11-abi"
+          requires:
+            - setup
+            - binary_linux_libtorch_2_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build
+          libtorch_variant: "shared-with-deps"
+          docker_image: "yf225/pytorch-binary-docker-image-ubuntu16.04:latest"
 

.circleci/verbatim-sources/workflows-nightly-android-binary-builds.yml

@@ -0,0 +1,56 @@
+      - pytorch_linux_build:
+          name: nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build
+          build_environment: "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-x86_32"
+          requires:
+            - setup
+          docker_image: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+          filters:
+            branches:
+              only: nightly
+      - pytorch_linux_build:
+          name: nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build
+          build_environment: "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-x86_64"
+          requires:
+            - setup
+          docker_image: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+          filters:
+            branches:
+              only: nightly
+      - pytorch_linux_build:
+          name: nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build
+          build_environment: "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-arm-v7a"
+          requires:
+            - setup
+          docker_image: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+          filters:
+            branches:
+              only: nightly
+      - pytorch_linux_build:
+          name: nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build
+          build_environment: "pytorch-linux-xenial-py3-clang5-android-ndk-r19c-arm-v8a"
+          requires:
+            - setup
+          docker_image: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:347"
+          filters:
+            branches:
+              only: nightly
+
+      - pytorch_android_gradle_build:
+          name: nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_android_gradle_build
+          requires:
+            - nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build
+            - nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build
+            - nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build
+            - nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build
+          filters:
+            branches:
+              only: nightly
+
+      - pytorch_android_publish_snapshot:
+          name: nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_android_publish_snapshot
+          requires:
+            - nightly_pytorch_linux_xenial_py3_clang5_android_ndk_r19c_android_gradle_build
+          context: org-member
+          filters:
+            branches:
+              only: nightly

.circleci/verbatim-sources/workflows-nightly-ios-binary-builds.yml

@@ -0,0 +1,31 @@
+      # Pytorch iOS binary builds
+      - binary_ios_build:
+          name: pytorch_ios_10_2_1_nightly_x86_64_build
+          build_environment: "libtorch-ios-10.2.1-nightly-x86_64-build"
+          ios_platform: "SIMULATOR"
+          ios_arch: "x86_64"
+          requires: 
+            - setup
+          filters:
+            branches:
+              only: nightly
+      - binary_ios_build:
+          name: pytorch_ios_10_2_1_nightly_arm64_build
+          build_environment: "libtorch-ios-10.2.1-nightly-arm64-build"
+          ios_arch: "arm64"
+          ios_platform: "OS"
+          requires: 
+            - setup
+          filters:
+            branches:
+              only: nightly
+      - binary_ios_upload:
+          build_environment: "libtorch-ios-10.2.1-nightly-binary-build-upload"
+          context: org-member
+          requires:
+            - setup
+            - pytorch_ios_10_2_1_nightly_x86_64_build
+            - pytorch_ios_10_2_1_nightly_arm64_build
+          filters:
+            branches:
+              only: nightly

.circleci/verbatim-sources/workflows-pytorch-android-gradle-build.yml

@@ -0,0 +1,12 @@
+      - pytorch_android_gradle_build-x86_32:
+          name: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32
+          requires:
+            - pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build
+
+      - pytorch_android_gradle_build:
+          name: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build
+          requires:
+            - pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build
+            - pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_64_build
+            - pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v7a_build
+            - pytorch_linux_xenial_py3_clang5_android_ndk_r19c_arm_v8a_build

.circleci/verbatim-sources/workflows-pytorch-ios-builds.yml

@@ -0,0 +1,13 @@
+      # Pytorch iOS PR builds
+      - pytorch_ios_build:
+          name: pytorch_ios_10_2_1_x86_64_build
+          build_environment: "pytorch-ios-10.2.1-x86_64_build"
+          ios_platform: "SIMULATOR"
+          requires:
+            - setup
+      - pytorch_ios_build:
+          name: pytorch_ios_10_2_1_arm64_build
+          build_environment: "pytorch-ios-10.2.1-arm64_build"
+          ios_arch: "arm64"
+          requires:
+            - setup

.circleci/verbatim-sources/workflows-pytorch-macos-builds.yml

@@ -1,3 +1,4 @@
+      # Warning: indentation here matters!
 
       # Pytorch MacOS builds
       - pytorch_macos_10_13_py3_build:
@@ -10,4 +11,3 @@
       - pytorch_macos_10_13_cuda9_2_cudnn7_py3_build:
           requires:
             - setup
-            

.circleci/verbatim-sources/workflows-s3-html.yml

@@ -6,25 +6,30 @@
   # These jobs are all idempotent and very lightweight; they just upload html
   # files that track what binaries are available and what their sizes are.
   update_s3_htmls:
-    triggers:
-      - schedule:
-          cron: "0 9 * * *"
+    jobs:
+      - setup:
           filters:
             branches:
-              only:
-                - master
-    jobs:
-      - setup
+              only: postnightly
       - update_s3_htmls_for_nightlies:
           context: org-member
           requires:
             - setup
+          filters:
+            branches:
+              only: postnightly
       - update_s3_htmls_for_nightlies_devtoolset7:
           context: org-member
           requires:
             - setup
+          filters:
+            branches:
+              only: postnightly
       - upload_binary_sizes:
           context: org-member
           requires:
             - setup
+          filters:
+            branches:
+              only: postnightly
 

.circleci/verbatim-sources/workflows.yml

@@ -7,6 +7,5 @@
 
 # PR jobs pr builds
 workflows:
-  version: 2
   build:
     jobs:

.clang-tidy

@@ -5,6 +5,7 @@ Checks: '
   ,bugprone-*
   ,-bugprone-forward-declaration-namespace
   ,-bugprone-macro-parentheses
+  ,-bugprone-lambda-function-name
   ,cppcoreguidelines-*
   ,-cppcoreguidelines-interfaces-global-init
   ,-cppcoreguidelines-owning-memory

.flake8

@@ -5,6 +5,8 @@ max-line-length = 120
 # E501 is not flexible enough, we're using B950 instead
 ignore =
     E203,E305,E402,E501,E721,E741,F403,F405,F821,F841,F999,W503,W504,C408,E302,W291,E303,
+    # EXE001 is skipped for now because some files use shebang to determine Python version.
+    EXE001,
     # these ignores are from flake8-bugbear; please fix!
     B007,B008,
     # these ignores are from flake8-comprehensions; please fix!

.github/pytorch-probot.yml

@@ -0,0 +1 @@
+tracking_issue: 24422

.github/workflows/lint.yml

@@ -0,0 +1,48 @@
+name: Lint
+
+on:
+  push:
+    branches:
+    - master
+  pull_request:
+
+jobs:
+  flake8-py3:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.7.4
+          architecture: x64
+      - name: Fetch PyTorch
+        uses: actions/checkout@master
+      - name: Checkout PR tip
+        run: |
+          set -eux
+          if [ -z "${GITHUB_HEAD_REF}" ]; then
+            # We are on master, just set the SHA from our current location
+            echo ::set-output name=commit_sha::${GITHUB_SHA}
+          else
+            # We are on a PR, so actions/checkout leaves us on merge commit.
+            # Check out the actual tip of the branch.
+            PR_TIP=$(git rev-parse HEAD^2)
+            git checkout ${PR_TIP}
+            echo ::set-output name=commit_sha::${PR_TIP}
+          fi
+        id: get_pr_tip
+      - name: Run flake8
+        run: |
+          set -eux
+          pip install flake8
+          flake8 --exit-zero > ${GITHUB_WORKSPACE}/flake8-output.txt
+          cat ${GITHUB_WORKSPACE}/flake8-output.txt
+      - name: Add annotations
+        uses: pytorch/add-annotations-github-action@master
+        with:
+          check_name: 'flake8-py3'
+          linter_output_path: 'flake8-output.txt'
+          commit_sha: ${{ steps.get_pr_tip.outputs.commit_sha }}
+          regex: '^(?<filename>.*?):(?<lineNumber>\d+):(?<columnNumber>\d+): (?<errorCode>\w\d+) (?<errorDesc>.*)'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -8,6 +8,9 @@
 
 ## PyTorch
 
+.coverage
+.gradle
+.hypothesis
 .mypy_cache
 */*.pyc
 */*.so*
@@ -27,14 +30,14 @@ dist/
 docs/src/**/*
 docs/cpp/build
 docs/cpp/source/api
+log
 test/.coverage
 test/.hypothesis/
 test/cpp/api/mnist
 test/custom_operator/model.pt
-test/data/gpu_tensors.pt
 test/data/legacy_modules.t7
-test/data/legacy_serialized.pt
-test/data/linear.pt
+test/data/*.pt
+test/backward_compatibility/new_schemas.txt
 dropout_model.pt
 test/generated_type_hints_smoketest.py
 test/htmlcov
@@ -43,6 +46,8 @@ third_party/build/
 tools/shared/_utils_internal.py
 torch.egg-info/
 torch/__init__.pyi
+torch/nn/functional.pyi
+torch/nn/modules/*.pyi
 torch/csrc/autograd/generated/*
 torch/csrc/cudnn/cuDNN.cpp
 torch/csrc/generated
@@ -85,6 +90,7 @@ torch/version.py
 # Root level file used in CI to specify certain env configs.
 # E.g., see .circleci/config.yaml
 env
+.circleci/scripts/COMMIT_MSG
 
 # IPython notebook checkpoints
 .ipynb_checkpoints
@@ -220,11 +226,6 @@ caffe2.egg-info
 # Files generated by CLion
 cmake-build-debug
 
-# Files generated by ctags
-CTAGS
-tags
-TAGS
-
 # BEGIN NOT-CLEAN-FILES (setup.py handles this marker. Do not change.)
 #
 # Below files are not deleted by "setup.py clean".
@@ -239,3 +240,12 @@ TAGS
 # Files generated when a patch is rejected
 *.orig
 *.rej
+
+# Files generated by ctags
+CTAGS
+GTAGS
+GRTAGS
+GSYMS
+GPATH
+tags
+TAGS

.gitmodules

@@ -21,7 +21,7 @@
 [submodule "third_party/protobuf"]
     ignore = dirty
     path = third_party/protobuf
-    url = https://github.com/google/protobuf.git
+    url = https://github.com/protocolbuffers/protobuf.git
 [submodule "third_party/ios-cmake"]
     ignore = dirty
     path = third_party/ios-cmake
@@ -57,7 +57,7 @@
 [submodule "third-party/cpuinfo"]
     ignore = dirty
     path = third_party/cpuinfo
-    url = https://github.com/Maratyszcza/cpuinfo.git
+    url = https://github.com/pytorch/cpuinfo.git
 [submodule "third_party/python-enum"]
     ignore = dirty
     path = third_party/python-enum
@@ -81,7 +81,7 @@
 [submodule "third_party/sleef"]
     ignore = dirty
     path = third_party/sleef
-    url = https://github.com/zdevito/sleef
+    url = https://github.com/shibatch/sleef
 [submodule "third_party/ideep"]
     ignore = dirty
     path = third_party/ideep
@@ -110,3 +110,11 @@
     ignore = dirty
     path = third_party/foxi
     url = https://github.com/houseroad/foxi.git
+[submodule "third_party/tbb"]
+	path = third_party/tbb
+	url = https://github.com/01org/tbb
+	branch = tbb_2018
+[submodule "android/libs/fbjni"]
+    ignore = dirty
+    path = android/libs/fbjni
+    url = https://github.com/IvanKobzarev/fbjni.git

.jenkins/caffe2/bench.sh

@@ -17,30 +17,44 @@ fi
 caffe2_pypath="$(cd /usr && $PYTHON -c 'import os; import caffe2; print(os.path.dirname(os.path.realpath(caffe2.__file__)))')"
 # Resnet50
 if (( $num_gpus == 0 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --use_cpu
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --use_cpu
 fi
 if (( $num_gpus >= 1 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --num_gpus 1
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --train_data null --batch_size 256 --epoch_size 25600 --num_epochs 2 --num_gpus 1 --float16_compute --dtype float16
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --num_gpus 1
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 256 --epoch_size 25600 --num_epochs 2 --num_gpus 1 --float16_compute --dtype float16
 fi
 if (( $num_gpus >= 2 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --train_data null --batch_size 256 --epoch_size 25600 --num_epochs 2 --num_gpus 2
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 256 --epoch_size 25600 --num_epochs 2 --num_gpus 2
 fi
 if (( $num_gpus >= 4 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --train_data null --batch_size 512 --epoch_size 51200 --num_epochs 2 --num_gpus 4
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 512 --epoch_size 51200 --num_epochs 2 --num_gpus 4
 fi
 
 # ResNext
 if (( $num_gpus == 0 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 32 --epoch_size 3200 --num_epochs 2 --use_cpu
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 32 --epoch_size 3200 --num_epochs 2 --use_cpu
 fi
 if (( $num_gpus >= 1 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 32 --epoch_size 3200 --num_epochs 2 --num_gpus 1
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 64 --epoch_size 3200 --num_epochs 2 --num_gpus 1 --float16_compute --dtype float16
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 32 --epoch_size 3200 --num_epochs 2 --num_gpus 1
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 64 --epoch_size 3200 --num_epochs 2 --num_gpus 1 --float16_compute --dtype float16
 fi
 if (( $num_gpus >= 2 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 64 --epoch_size 6400 --num_epochs 2 --num_gpus 2
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 64 --epoch_size 6400 --num_epochs 2 --num_gpus 2
 fi
 if (( $num_gpus >= 4 )); then
-    "$PYTHON" "$caffe2_pypath/python/examples/resnet50_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --num_gpus 4
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --resnext_num_groups 32 --resnext_width_per_group 4 --num_layers 101 --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --num_gpus 4
+fi
+
+# Shufflenet
+if (( $num_gpus == 0 )); then
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 32 --epoch_size 3200 --num_epochs 2 --use_cpu --model shufflenet
+fi
+if (( $num_gpus >= 1 )); then
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 32 --epoch_size 3200 --num_epochs 2 --num_gpus 1 --model shufflenet
+fi
+if (( $num_gpus >= 2 )); then
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 64 --epoch_size 6400 --num_epochs 2 --num_gpus 2 --model shufflenet
+fi
+if (( $num_gpus >= 4 )); then
+    "$PYTHON" "$caffe2_pypath/python/examples/imagenet_trainer.py" --train_data null --batch_size 128 --epoch_size 12800 --num_epochs 2 --num_gpus 4 --model shufflenet
 fi

.jenkins/caffe2/build.sh

@@ -156,7 +156,9 @@ if [[ $BUILD_ENVIRONMENT == *cuda* ]]; then
   build_args+=("TORCH_CUDA_ARCH_LIST=Maxwell")
 
   # Explicitly set path to NVCC such that the symlink to ccache or sccache is used
-  build_args+=("CUDA_NVCC_EXECUTABLE=${CACHE_WRAPPER_DIR}/nvcc")
+  if [ -n "${CACHE_WRAPPER_DIR}" ]; then
+    build_args+=("CUDA_NVCC_EXECUTABLE=${CACHE_WRAPPER_DIR}/nvcc")
+  fi
 
   # Ensure FindCUDA.cmake can infer the right path to the CUDA toolkit.
   # Setting PATH to resolve to the right nvcc alone isn't enough.
@@ -255,7 +257,7 @@ else
 
   # sccache will be stuck if  all cores are used for compiling
   # see https://github.com/pytorch/pytorch/pull/7361
-  if [[ -n "${SCCACHE}" ]]; then
+  if [[ -n "${SCCACHE}" && $BUILD_ENVIRONMENT != *rocm* ]]; then
     export MAX_JOBS=`expr $(nproc) - 1`
   fi
 
@@ -271,4 +273,9 @@ fi
 # Install ONNX into a local directory
 pip install --user -b /tmp/pip_install_onnx "file://${ROOT_DIR}/third_party/onnx#egg=onnx"
 
+if [[ $BUILD_ENVIRONMENT == *rocm* ]]; then
+  # runtime compilation of MIOpen kernels manages to crash sccache - hence undo the wrapping
+  bash tools/amd_build/unwrap_clang.sh
+fi
+
 report_compile_cache_stats

.jenkins/caffe2/test.sh

@@ -90,9 +90,6 @@ rocm_ignore_test=()
 if [[ $BUILD_ENVIRONMENT == *-rocm* ]]; then
   # Currently these tests are failing on ROCM platform:
 
-  # Unknown reasons, need to debug
-  rocm_ignore_test+=("--ignore $caffe2_pypath/python/operator_test/piecewise_linear_transform_test.py")
-
   # On ROCm, RCCL (distributed) development isn't complete.
   # https://github.com/ROCmSoftwarePlatform/rccl
   rocm_ignore_test+=("--ignore $caffe2_pypath/python/data_parallel_model_test.py")
@@ -101,6 +98,12 @@ fi
 # NB: Warnings are disabled because they make it harder to see what
 # the actual erroring test is
 echo "Running Python tests.."
+if [[ "$BUILD_ENVIRONMENT" == *py3* ]]; then
+  # locale setting is required by click package with py3
+  export LC_ALL=C.UTF-8
+  export LANG=C.UTF-8
+fi
+
 pip install --user pytest-sugar
 "$PYTHON" \
   -m pytest \
@@ -121,5 +124,15 @@ pip install --user pytest-sugar
 #####################
 if [[ "$BUILD_ENVIRONMENT" == *onnx* ]]; then
   pip install -q --user git+https://github.com/pytorch/vision.git
+  pip install -q --user ninja
+  # JIT C++ extensions require ninja, so put it into PATH.
+  export PATH="/var/lib/jenkins/.local/bin:$PATH"
+  if [[ "$BUILD_ENVIRONMENT" == *py3* ]]; then
+    # default pip version is too old(9.0.2), unable to support tag `manylinux2010`.
+    # Fix the pip error: Couldn't find a version that satisfies the requirement
+    sudo pip install --upgrade pip
+    pip install -q --user -i https://test.pypi.org/simple/ ort-nightly==0.5.0.dev905
+  fi
   "$ROOT_DIR/scripts/onnx/test.sh"
 fi
+

.jenkins/pytorch/build-asan.sh

@@ -33,7 +33,7 @@ export ASAN_OPTIONS=detect_leaks=0:symbolize=1
 CC="clang" CXX="clang++" LDSHARED="clang --shared" \
   CFLAGS="-fsanitize=address -fsanitize=undefined -fno-sanitize-recover=all -shared-libasan -pthread" \
   CXX_FLAGS="-pthread" \
-  USE_ASAN=1 NO_CUDA=1 USE_MKLDNN=0 \
+  USE_ASAN=1 USE_CUDA=0 USE_MKLDNN=0 \
   python setup.py install
 
 assert_git_not_dirty

.jenkins/pytorch/build.sh

@@ -32,7 +32,7 @@ if [[ "$BUILD_ENVIRONMENT" == *-xenial-cuda9*gcc7* ]] || [[ "$BUILD_ENVIRONMENT"
   sudo mkdir -p /var/run/sshd
 fi
 
-if [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-xenial-py3-clang5-asan* ]]; then
+if [[ "$BUILD_ENVIRONMENT" == *-linux-xenial-py3-clang5-asan* ]]; then
   exec "$(dirname "${BASH_SOURCE[0]}")/build-asan.sh" "$@"
 fi
 
@@ -46,7 +46,7 @@ echo "CMake version:"
 cmake --version
 
 # TODO: Don't run this...
-pip install -q -r requirements.txt || true
+pip_install -r requirements.txt || true
 
 # TODO: Don't install this here
 if ! which conda; then
@@ -54,7 +54,7 @@ if ! which conda; then
   # intel cpu and later run tests on machines with amd cpu.
   # Also leave out two builds to make sure non-mkldnn builds still work.
   if [[ "$BUILD_ENVIRONMENT" != *rocm* && "$BUILD_ENVIRONMENT" != *-trusty-py3.5-* && "$BUILD_ENVIRONMENT" != *-xenial-cuda9-cudnn7-py3-* ]]; then
-    pip install -q mkl mkl-devel
+    pip_install mkl mkl-devel
     export USE_MKLDNN=1
   else
     export USE_MKLDNN=0
@@ -65,9 +65,16 @@ fi
 if [[ "${BUILD_ENVIRONMENT}" == *-android* ]]; then
   export ANDROID_NDK=/opt/ndk
   build_args=()
-  build_args+=("-DBUILD_CAFFE2_MOBILE=OFF")
-  build_args+=("-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')")
-  build_args+=("-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)')")
+  if [[ "${BUILD_ENVIRONMENT}" == *-arm-v7a* ]]; then
+    build_args+=("-DANDROID_ABI=armeabi-v7a")
+  elif [[ "${BUILD_ENVIRONMENT}" == *-arm-v8a* ]]; then
+    build_args+=("-DANDROID_ABI=arm64-v8a")
+  elif [[ "${BUILD_ENVIRONMENT}" == *-x86_32* ]]; then
+    build_args+=("-DANDROID_ABI=x86")
+  elif [[ "${BUILD_ENVIRONMENT}" == *-x86_64* ]]; then
+    build_args+=("-DANDROID_ABI=x86_64")
+  fi
+  export BUILD_PYTORCH_MOBILE=1
   exec ./scripts/build_android.sh "${build_args[@]}" "$@"
 fi
 
@@ -90,7 +97,7 @@ if [[ "$BUILD_ENVIRONMENT" == *rocm* ]]; then
     fi
 
     # Setup wrapper scripts
-    for compiler in cc c++ gcc g++; do
+    for compiler in cc c++ gcc g++ clang clang++; do
       (
         echo "#!/bin/sh"
         echo "exec $SCCACHE $(which $compiler) \"\$@\""
@@ -108,6 +115,10 @@ if [[ "$BUILD_ENVIRONMENT" == *rocm* ]]; then
   # OPENCV is needed to enable ImageInput operator in caffe2 resnet5_trainer
   # LMDB is needed to read datasets from https://download.caffe2.ai/databases/resnet_trainer.zip
   USE_ROCM=1 USE_LMDB=1 USE_OPENCV=1 python setup.py install --user
+
+  # runtime compilation of MIOpen kernels manages to crash sccache - hence undo the wrapping
+  bash tools/amd_build/unwrap_clang.sh
+
   exit 0
 fi
 
@@ -126,7 +137,7 @@ if [[ "$BUILD_ENVIRONMENT" == *ppc64le* ]]; then
   export TORCH_CUDA_ARCH_LIST="6.0"
 fi
 
-if [[ "$BUILD_ENVIRONMENT" == *trusty-py3.6-gcc5.4* ]]; then
+if [[ "$BUILD_ENVIRONMENT" == *xenial-py3.6-gcc5.4* ]]; then
   export DEBUG=1
 fi
 
@@ -136,6 +147,11 @@ if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
   ./xla/scripts/apply_patches.sh
 fi
 
+if [[ "${BUILD_ENVIRONMENT}" == *clang* ]]; then
+  export CC=clang
+  export CXX=clang++
+fi
+
 
 # check that setup.py would fail with bad arguments
 echo "The next three invocations are expected to fail with invalid command error messages."
@@ -146,19 +162,24 @@ echo "The next three invocations are expected to fail with invalid command error
 # ppc64le build fails when WERROR=1
 # set only when building other architectures
 # only use for "python setup.py install" line
-if [[ "$BUILD_ENVIRONMENT" != *ppc64le* ]]; then
+if [[ "$BUILD_ENVIRONMENT" != *ppc64le*  && "$BUILD_ENVIRONMENT" != *clang* ]]; then
   WERROR=1 python setup.py install
 else
   python setup.py install
 fi
 
+if which sccache > /dev/null; then
+  echo 'PyTorch Build Statistics'
+  sccache --show-stats
+fi
+
 assert_git_not_dirty
 
 # Test documentation build
 if [[ "$BUILD_ENVIRONMENT" == *xenial-cuda9-cudnn7-py3* ]]; then
   pushd docs
   # TODO: Don't run this here
-  pip install -q -r requirements.txt || true
+  pip_install -r requirements.txt || true
   LC_ALL=C make html
   popd
   assert_git_not_dirty
@@ -201,10 +222,10 @@ fi
 if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
   # TODO: Move this to Dockerfile.
 
-  pip install -q lark-parser
+  pip_install lark-parser
 
   # Bazel doesn't work with sccache gcc. https://github.com/bazelbuild/bazel/issues/3642
-  sudo add-apt-repository "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-7 main"
+  sudo add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main"
   wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
   sudo apt-get -qq update
 
@@ -235,7 +256,7 @@ if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
     exit 1
   fi
 
-  bazels3cache --bucket=ossci-compiler-cache-circleci-xla --maxEntrySizeBytes=0
+  bazels3cache --bucket=${XLA_CLANG_CACHE_S3_BUCKET_NAME} --maxEntrySizeBytes=0
   pushd xla
   export CC=clang-7 CXX=clang++-7
   # Use cloud cache to build when available.

.jenkins/pytorch/common.sh

@@ -17,9 +17,21 @@ function cleanup {
 
 set -ex
 
+# Save the SCRIPT_DIR absolute path in case later we chdir (as occurs in the gpu perf test)
+SCRIPT_DIR="$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )"
+
 # Required environment variables:
 #   $BUILD_ENVIRONMENT (should be set by your Docker image)
 
+# Figure out which Python to use for ROCm
+if [[ "${BUILD_ENVIRONMENT}" == *rocm* ]] && [[ "${BUILD_ENVIRONMENT}" =~ py((2|3)\.?[0-9]?\.?[0-9]?) ]]; then
+  PYTHON=$(which "python${BASH_REMATCH[1]}")
+  # non-interactive bashs do not expand aliases by default
+  shopt -s expand_aliases
+  export PYTORCH_TEST_WITH_ROCM=1
+  alias python="$PYTHON"
+fi
+
 # This token is used by a parser on Jenkins logs for determining
 # if a failure is a legitimate problem, or a problem with the build
 # system; to find out more, grep for this string in ossci-job-dsl.
@@ -89,7 +101,7 @@ if which sccache > /dev/null; then
   sccache --zero-stats
   function sccache_epilogue() {
     echo '=================== sccache compilation log ==================='
-    python "$(dirname "${BASH_SOURCE[0]}")/print_sccache_log.py" ~/sccache_error.log
+    python "$SCRIPT_DIR/print_sccache_log.py" ~/sccache_error.log 2>/dev/null
     echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
     sccache --show-stats
     sccache --stop-server || true
@@ -144,6 +156,16 @@ if [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-xenial-cuda* ]]; then
   fi
 fi
 
+function pip_install() {
+  # retry 3 times
+  pip install --progress-bar off "$@" || pip install --progress-bar off "$@" || pip install --progress-bar off "$@"
+}
+
+function pip_uninstall() {
+  # uninstall 2 times
+  pip uninstall -y "$@" || pip uninstall -y "$@"
+}
+
 function get_exit_code() {
   set +e
   "$@"

.jenkins/pytorch/macos-build.sh

@@ -1,27 +1,11 @@
 #!/bin/bash
 
 # shellcheck disable=SC2034
-COMPACT_JOB_NAME="${BUILD_ENVIRONMENT}"
-
-export PATH="/usr/local/bin:$PATH"
-source "$(dirname "${BASH_SOURCE[0]}")/common.sh"
-
-# Set up conda environment
-export PYTORCH_ENV_DIR="${HOME}/pytorch-ci-env"
-# If a local installation of conda doesn't exist, we download and install conda
-if [ ! -d "${PYTORCH_ENV_DIR}/miniconda3" ]; then
-  mkdir -p ${PYTORCH_ENV_DIR}
-  curl https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o ${PYTORCH_ENV_DIR}/miniconda3.sh
-  bash ${PYTORCH_ENV_DIR}/miniconda3.sh -b -p ${PYTORCH_ENV_DIR}/miniconda3
-fi
-export PATH="${PYTORCH_ENV_DIR}/miniconda3/bin:$PATH"
-source ${PYTORCH_ENV_DIR}/miniconda3/bin/activate
-conda install -y mkl mkl-include numpy pyyaml setuptools cmake cffi ninja
-rm -rf ${PYTORCH_ENV_DIR}/miniconda3/lib/python3.6/site-packages/torch*
+source "$(dirname "${BASH_SOURCE[0]}")/macos-common.sh"
 
 git submodule sync --recursive
 git submodule update --init --recursive
-export CMAKE_PREFIX_PATH=${PYTORCH_ENV_DIR}/miniconda3/
+export CMAKE_PREFIX_PATH=${WORKSPACE_DIR}/miniconda3/
 
 # Build PyTorch
 if [[ "${BUILD_ENVIRONMENT}" == *cuda9.2* ]]; then
@@ -30,7 +14,7 @@ if [[ "${BUILD_ENVIRONMENT}" == *cuda9.2* ]]; then
   export PATH=/Developer/NVIDIA/CUDA-${CUDA_VERSION}/bin${PATH:+:${PATH}}
   export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-${CUDA_VERSION}/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
   export CUDA_HOME=/Developer/NVIDIA/CUDA-${CUDA_VERSION}
-  export NO_CUDA=0
+  export USE_CUDA=1
 
   if [ -z "${IN_CIRCLECI}" ]; then
     # Eigen gives "explicit specialization of class must precede its first use" error
@@ -43,35 +27,29 @@ else
   fi
 fi
 
-export MACOSX_DEPLOYMENT_TARGET=10.9
-export CXX=clang++
-export CC=clang
 if which sccache > /dev/null; then
-  printf "#!/bin/sh\nexec sccache $(which clang++) \$*" > "${PYTORCH_ENV_DIR}/clang++"
-  chmod a+x "${PYTORCH_ENV_DIR}/clang++"
+  printf "#!/bin/sh\nexec sccache $(which clang++) \$*" > "${WORKSPACE_DIR}/clang++"
+  chmod a+x "${WORKSPACE_DIR}/clang++"
 
-  printf "#!/bin/sh\nexec sccache $(which clang) \$*" > "${PYTORCH_ENV_DIR}/clang"
-  chmod a+x "${PYTORCH_ENV_DIR}/clang"
+  printf "#!/bin/sh\nexec sccache $(which clang) \$*" > "${WORKSPACE_DIR}/clang"
+  chmod a+x "${WORKSPACE_DIR}/clang"
 
   if [[ "${BUILD_ENVIRONMENT}" == *cuda* ]]; then
-    printf "#!/bin/sh\nexec sccache $(which nvcc) \$*" > "${PYTORCH_ENV_DIR}/nvcc"
-    chmod a+x "${PYTORCH_ENV_DIR}/nvcc"
-    export CUDA_NVCC_EXECUTABLE="${PYTORCH_ENV_DIR}/nvcc"
+    printf "#!/bin/sh\nexec sccache $(which nvcc) \$*" > "${WORKSPACE_DIR}/nvcc"
+    chmod a+x "${WORKSPACE_DIR}/nvcc"
+    export CUDA_NVCC_EXECUTABLE="${WORKSPACE_DIR}/nvcc"
   fi
 
-  export PATH="${PYTORCH_ENV_DIR}:$PATH"
+  export PATH="${WORKSPACE_DIR}:$PATH"
 fi
+
 # If we run too many parallel jobs, we will OOM
-export MAX_JOBS=2
-
-export IMAGE_COMMIT_TAG=${BUILD_ENVIRONMENT}-${IMAGE_COMMIT_ID}
-
-python setup.py install
+MAX_JOBS=2 USE_DISTRIBUTED=1 python setup.py install
 
 assert_git_not_dirty
 
 # Upload torch binaries when the build job is finished
 if [ -z "${IN_CIRCLECI}" ]; then
-  7z a ${IMAGE_COMMIT_TAG}.7z ${PYTORCH_ENV_DIR}/miniconda3/lib/python3.6/site-packages/torch*
+  7z a ${IMAGE_COMMIT_TAG}.7z ${WORKSPACE_DIR}/miniconda3/lib/python3.6/site-packages/torch*
   aws s3 cp ${IMAGE_COMMIT_TAG}.7z s3://ossci-macos-build/pytorch/${IMAGE_COMMIT_TAG}.7z --acl public-read
 fi

.jenkins/pytorch/macos-common.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Common prelude for macos-build.sh and macos-test.sh
+
+# shellcheck disable=SC2034
+COMPACT_JOB_NAME="${BUILD_ENVIRONMENT}"
+
+source "$(dirname "${BASH_SOURCE[0]}")/common.sh"
+export PATH="/usr/local/bin:$PATH"
+export WORKSPACE_DIR="${HOME}/workspace"
+mkdir -p ${WORKSPACE_DIR}
+
+# If a local installation of conda doesn't exist, we download and install conda
+if [ ! -d "${WORKSPACE_DIR}/miniconda3" ]; then
+  mkdir -p ${WORKSPACE_DIR}
+  curl https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o ${WORKSPACE_DIR}/miniconda3.sh
+  bash ${WORKSPACE_DIR}/miniconda3.sh -b -p ${WORKSPACE_DIR}/miniconda3
+fi
+export PATH="${WORKSPACE_DIR}/miniconda3/bin:$PATH"
+source ${WORKSPACE_DIR}/miniconda3/bin/activate
+conda install -y mkl mkl-include numpy pyyaml setuptools cmake cffi ninja
+
+# The torch.hub tests make requests to GitHub.
+#
+# The certifi package from conda-forge is new enough to make the
+# following error disappear (included for future reference):
+#
+# > ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
+# > certificate verify failed: unable to get local issuer certificate
+# > (_ssl.c:1056)
+#
+conda install -y -c conda-forge certifi
+
+# Needed by torchvision, which is imported from TestHub in test_utils.py.
+conda install -y pillow
+
+# Building with USE_DISTRIBUTED=1 requires libuv (for Gloo).
+conda install -y libuv pkg-config
+
+# Image commit tag is used to persist the build from the build job
+# and to retrieve the build from the test job.
+export IMAGE_COMMIT_TAG=${BUILD_ENVIRONMENT}-${IMAGE_COMMIT_ID}
+
+# These are required for both the build job and the test job.
+# In the latter to test cpp extensions.
+export MACOSX_DEPLOYMENT_TARGET=10.9
+export CXX=clang++
+export CC=clang

.jenkins/pytorch/macos-test.sh

@@ -1,23 +1,9 @@
 #!/bin/bash
 
 # shellcheck disable=SC2034
-COMPACT_JOB_NAME="${BUILD_ENVIRONMENT}"
+source "$(dirname "${BASH_SOURCE[0]}")/macos-common.sh"
 
-source "$(dirname "${BASH_SOURCE[0]}")/common.sh"
-
-export PATH="/usr/local/bin:$PATH"
-
-# Set up conda environment
-export PYTORCH_ENV_DIR="${HOME}/pytorch-ci-env"
-# If a local installation of conda doesn't exist, we download and install conda
-if [ ! -d "${PYTORCH_ENV_DIR}/miniconda3" ]; then
-  mkdir -p ${PYTORCH_ENV_DIR}
-  curl https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o ${PYTORCH_ENV_DIR}/miniconda3.sh
-  bash ${PYTORCH_ENV_DIR}/miniconda3.sh -b -p ${PYTORCH_ENV_DIR}/miniconda3
-fi
-export PATH="${PYTORCH_ENV_DIR}/miniconda3/bin:$PATH"
-source ${PYTORCH_ENV_DIR}/miniconda3/bin/activate
-conda install -y mkl mkl-include numpy pyyaml setuptools cmake cffi ninja six
+conda install -y six
 pip install -q hypothesis "librosa>=0.6.2" psutil
 
 # faulthandler become built-in since 3.3
@@ -26,12 +12,12 @@ if [[ ! $(python -c "import sys; print(int(sys.version_info >= (3, 3)))") == "1"
 fi
 
 if [ -z "${IN_CIRCLECI}" ]; then
-  rm -rf ${PYTORCH_ENV_DIR}/miniconda3/lib/python3.6/site-packages/torch*
+  rm -rf ${WORKSPACE_DIR}/miniconda3/lib/python3.6/site-packages/torch*
 fi
 
 git submodule sync --recursive
 git submodule update --init --recursive
-export CMAKE_PREFIX_PATH=${PYTORCH_ENV_DIR}/miniconda3/
+export CMAKE_PREFIX_PATH=${WORKSPACE_DIR}/miniconda3/
 
 # Test PyTorch
 if [ -z "${IN_CIRCLECI}" ]; then
@@ -43,19 +29,12 @@ if [ -z "${IN_CIRCLECI}" ]; then
     export DEVELOPER_DIR=/Applications/Xcode9.app/Contents/Developer
   fi
 fi
-export MACOSX_DEPLOYMENT_TARGET=10.9
-export CXX=clang++
-export CC=clang
-# If we run too many parallel jobs, we will OOM
-export MAX_JOBS=2
-
-export IMAGE_COMMIT_TAG=${BUILD_ENVIRONMENT}-${IMAGE_COMMIT_ID}
 
 # Download torch binaries in the test jobs
 if [ -z "${IN_CIRCLECI}" ]; then
-  rm -rf ${PYTORCH_ENV_DIR}/miniconda3/lib/python3.6/site-packages/torch*
+  rm -rf ${WORKSPACE_DIR}/miniconda3/lib/python3.6/site-packages/torch*
   aws s3 cp s3://ossci-macos-build/pytorch/${IMAGE_COMMIT_TAG}.7z ${IMAGE_COMMIT_TAG}.7z
-  7z x ${IMAGE_COMMIT_TAG}.7z -o"${PYTORCH_ENV_DIR}/miniconda3/lib/python3.6/site-packages"
+  7z x ${IMAGE_COMMIT_TAG}.7z -o"${WORKSPACE_DIR}/miniconda3/lib/python3.6/site-packages"
 fi
 
 # Test that OpenMP is enabled
@@ -67,6 +46,10 @@ fi
 popd
 
 test_python_all() {
+  # The CircleCI worker hostname doesn't resolve to an address.
+  # This environment variable makes ProcessGroupGloo default to
+  # using the address associated with the loopback interface.
+  export GLOO_SOCKET_IFNAME=lo0
   echo "Ninja version: $(ninja --version)"
   python test/run_test.py --verbose
   assert_git_not_dirty
@@ -116,6 +99,7 @@ test_custom_script_ops() {
 
   # Run tests Python-side and export a script module.
   python test_custom_ops.py -v
+  python test_custom_classes.py -v
   python model.py --export-script-module=model.pt
   # Run tests C++-side and load the exported script module.
   build/test_custom_ops ./model.pt

.jenkins/pytorch/multigpu-test.sh

@@ -27,5 +27,9 @@ if [ -n "${IN_CIRCLECI}" ]; then
   fi
 fi
 
+python tools/download_mnist.py --quiet -d test/cpp/api/mnist
+OMP_NUM_THREADS=2 TORCH_CPP_TEST_MNIST_PATH="test/cpp/api/mnist" "$PWD/../cpp-build"/caffe2/build/bin/test_api
 time python test/run_test.py --verbose -i distributed
+time python test/run_test.py --verbose -i c10d
+time python test/run_test.py --verbose -i c10d_spawn
 assert_git_not_dirty

.jenkins/pytorch/test.sh

@@ -35,30 +35,30 @@ fi
 # --user breaks ppc64le builds and these packages are already in ppc64le docker
 if [[ "$BUILD_ENVIRONMENT" != *ppc64le* ]]; then
   # JIT C++ extensions require ninja.
-  pip install -q ninja --user
+  pip_install --user ninja
   # ninja is installed in /var/lib/jenkins/.local/bin
   export PATH="/var/lib/jenkins/.local/bin:$PATH"
 
   # TODO: move this to Docker
-  pip install -q hypothesis --user
+  pip_install --user hypothesis
 
   # TODO: move this to Docker
   PYTHON_VERSION=$(python -c 'import platform; print(platform.python_version())'|cut -c1)
   echo $PYTHON_VERSION
-  if [[ $PYTHON_VERSION == "2" ]]; then
-    pip install -q https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py2-none-any.whl --user
-  else
-    pip install -q https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py3-none-any.whl --user
-  fi
-
+  # if [[ $PYTHON_VERSION == "2" ]]; then
+  #   pip_install --user https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py2-none-any.whl
+  # else
+  #   pip_install --user https://s3.amazonaws.com/ossci-linux/wheels/tensorboard-1.14.0a0-py3-none-any.whl
+  # fi
+  pip_install --user tb-nightly
   # mypy will fail to install on Python <3.4.  In that case,
   # we just won't run these tests.
-  pip install mypy --user || true
+  pip_install --user mypy || true
 fi
 
 # faulthandler become built-in since 3.3
 if [[ ! $(python -c "import sys; print(int(sys.version_info >= (3, 3)))") == "1" ]]; then
-  pip install -q faulthandler --user
+  pip_install --user faulthandler
 fi
 
 # DANGER WILL ROBINSON.  The LD_PRELOAD here could cause you problems
@@ -92,10 +92,6 @@ if [[ "$BUILD_ENVIRONMENT" == *asan* ]]; then
     (cd test && ! get_exit_code python -c "import torch; torch._C._crash_if_aten_asan(3)")
 fi
 
-if [[ "$BUILD_ENVIRONMENT" == *rocm* ]]; then
-  export PYTORCH_TEST_WITH_ROCM=1
-fi
-
 if [[ "${BUILD_ENVIRONMENT}" == *-NO_AVX-* ]]; then
   export ATEN_CPU_CAPABILITY=default
 elif [[ "${BUILD_ENVIRONMENT}" == *-NO_AVX2-* ]]; then
@@ -108,7 +104,7 @@ test_python_nn() {
 }
 
 test_python_all_except_nn() {
-  time python test/run_test.py --exclude nn --verbose
+  time python test/run_test.py --exclude nn --verbose --bring-to-front quantization quantized quantized_tensor quantized_nn_mods quantizer
   assert_git_not_dirty
 }
 
@@ -138,22 +134,7 @@ test_aten() {
 }
 
 test_torchvision() {
-  rm -rf ninja
-
-  echo "Installing torchvision at branch master"
-  rm -rf vision
-  # TODO: This git clone is bad, it means pushes to torchvision can break
-  # PyTorch CI
-  git clone https://github.com/pytorch/vision --quiet
-  pushd vision
-  # python setup.py install with a tqdm dependency is broken in the
-  # Travis Python nightly (but not in latest Python nightlies, so
-  # this should be a transient requirement...)
-  # See https://github.com/pytorch/pytorch/issues/7525
-  #time python setup.py install
-  pip install -q --user .
-  popd
-  rm -rf vision
+  pip_install --user git+https://github.com/pytorch/vision.git@2b73a4846773a670632b29fb2fc2ac57df7bce5d
 }
 
 test_libtorch() {
@@ -162,13 +143,13 @@ test_libtorch() {
     python test/cpp/jit/tests_setup.py setup
     CPP_BUILD="$PWD/../cpp-build"
     if [[ "$BUILD_ENVIRONMENT" == *cuda* ]]; then
-      "$CPP_BUILD"/caffe2/bin/test_jit
+      "$CPP_BUILD"/caffe2/build/bin/test_jit
     else
-      "$CPP_BUILD"/caffe2/bin/test_jit "[cpu]"
+      "$CPP_BUILD"/caffe2/build/bin/test_jit "[cpu]"
     fi
     python test/cpp/jit/tests_setup.py shutdown
     python tools/download_mnist.py --quiet -d test/cpp/api/mnist
-    OMP_NUM_THREADS=2 TORCH_CPP_TEST_MNIST_PATH="test/cpp/api/mnist" "$CPP_BUILD"/caffe2/bin/test_api
+    OMP_NUM_THREADS=2 TORCH_CPP_TEST_MNIST_PATH="test/cpp/api/mnist" "$CPP_BUILD"/caffe2/build/bin/test_api
     assert_git_not_dirty
   fi
 }
@@ -181,6 +162,7 @@ test_custom_script_ops() {
     cp -a "$CUSTOM_OP_BUILD" build
     # Run tests Python-side and export a script module.
     python test_custom_ops.py -v
+    python test_custom_classes.py -v
     python model.py --export-script-module=model.pt
     # Run tests C++-side and load the exported script module.
     build/test_custom_ops ./model.pt
@@ -193,22 +175,47 @@ test_xla() {
   export XLA_USE_XRT=1 XRT_DEVICE_MAP="CPU:0;/job:localservice/replica:0/task:0/device:XLA_CPU:0"
   export XRT_WORKERS="localservice:0;grpc://localhost:40934"
   pushd xla
-  python test/test_operations.py
+  echo "Running Python Tests"
+  ./test/run_tests.sh
+
+  echo "Running MNIST Test"
   python test/test_train_mnist.py --tidy
+
+  echo "Running C++ Tests"
+  pushd test/cpp
+  CC=clang-7 CXX=clang++-7 ./run_tests.sh
   popd
   assert_git_not_dirty
 }
 
+# Do NOT run this test before any other tests, like test_python_nn, etc.
+# Because this function uninstalls the torch built from branch, and install
+# nightly version.
+test_backward_compatibility() {
+  set -x
+  pushd test/backward_compatibility
+  python dump_all_function_schemas.py --filename new_schemas.txt
+  pip_uninstall torch
+  pip_install --pre torch -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
+  python check_backward_compatibility.py --new-schemas new_schemas.txt
+  popd
+  set +x
+  assert_git_not_dirty
+}
+
 (cd test && python -c "import torch; print(torch.__config__.show())")
 (cd test && python -c "import torch; print(torch.__config__.parallel_info())")
 
-if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
+if [[ "${BUILD_ENVIRONMENT}" == *backward* ]]; then
+  test_backward_compatibility
+  # Do NOT add tests after bc check tests, see its comment.
+elif [[ "${BUILD_ENVIRONMENT}" == *xla* || "${JOB_BASE_NAME}" == *xla* ]]; then
   test_torchvision
   test_xla
-elif [[ "${BUILD_ENVIRONMENT}" == *-test1 ]]; then
+elif [[ "${BUILD_ENVIRONMENT}" == *-test1 || "${JOB_BASE_NAME}" == *-test1 ]]; then
   test_torchvision
   test_python_nn
-elif [[ "${BUILD_ENVIRONMENT}" == *-test2 ]]; then
+elif [[ "${BUILD_ENVIRONMENT}" == *-test2 || "${JOB_BASE_NAME}" == *-test2 ]]; then
   test_python_all_except_nn
   test_aten
   test_libtorch

.jenkins/pytorch/win-test-helpers/build_pytorch.bat

@@ -6,6 +6,12 @@ if "%DEBUG%" == "1" (
 
 set PATH=C:\Program Files\CMake\bin;C:\Program Files\7-Zip;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Amazon\AWSCLI;%PATH%
 
+:: This inflates our log size slightly, but it is REALLY useful to be
+:: able to see what our cl.exe commands are (since you can actually
+:: just copy-paste them into a local Windows setup to just rebuild a
+:: single file.)
+set CMAKE_VERBOSE_MAKEFILE=1
+
 
 set INSTALLER_DIR=%SCRIPT_HELPERS_DIR%\installation-helpers
 
@@ -69,16 +75,26 @@ set CXX=sccache cl
 
 set CMAKE_GENERATOR=Ninja
 
+:: The following code will try to build PyTorch twice if USE_CUDA is neither 0
+:: nor 1. It is intended so that both builds can be folded into 1 CI run.
+
 if not "%USE_CUDA%"=="1" (
   if "%REBUILD%"=="" (
-    set NO_CUDA=1
+    :: Must save and restore the original value of USE_CUDA, otherwise the
+    :: `if not "%USE_CUDA%"=="0"` line can be messed up.
+    set OLD_USE_CUDA=%USE_CUDA%
+    set USE_CUDA=0
     python setup.py install
+    set USE_CUDA=%OLD_USE_CUDA%
   )
   if errorlevel 1 exit /b 1
   if not errorlevel 0 exit /b 1
 )
 
 if not "%USE_CUDA%"=="0" (
+  :: sccache will fail for CUDA builds if all cores are used for compiling
+  if not defined MAX_JOBS set /A MAX_JOBS=%NUMBER_OF_PROCESSORS%-1
+
   if "%REBUILD%"=="" (
     sccache --show-stats
     sccache --zero-stats
@@ -93,13 +109,12 @@ if not "%USE_CUDA%"=="0" (
 
   set CUDA_NVCC_EXECUTABLE=%TMP_DIR_WIN%\bin\nvcc
 
-  if "%REBUILD%"=="" set NO_CUDA=0
+  if "%REBUILD%"=="" set USE_CUDA=1
 
   python setup.py install --cmake && sccache --show-stats && (
     if "%BUILD_ENVIRONMENT%"=="" (
       echo NOTE: To run `import torch`, please make sure to activate the conda environment by running `call %CONDA_PARENT_DIR%\Miniconda3\Scripts\activate.bat %CONDA_PARENT_DIR%\Miniconda3` in Command Prompt before running Git Bash.
     ) else (
-      mv %CD%\build\bin\test_api.exe %CONDA_PARENT_DIR%\Miniconda3\Lib\site-packages\torch\lib
       7z a %TMP_DIR_WIN%\%IMAGE_COMMIT_TAG%.7z %CONDA_PARENT_DIR%\Miniconda3\Lib\site-packages\torch %CONDA_PARENT_DIR%\Miniconda3\Lib\site-packages\caffe2 && python %SCRIPT_HELPERS_DIR%\upload_image.py %TMP_DIR_WIN%\%IMAGE_COMMIT_TAG%.7z
     )
   )

.jenkins/pytorch/win-test-helpers/download_image.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python
+
 import os
 import sys
 import boto3

.jenkins/pytorch/win-test-helpers/installation-helpers/install_mkl.bat

@@ -1,8 +1,8 @@
 if "%REBUILD%"=="" (
   if "%BUILD_ENVIRONMENT%"=="" (
-    curl -k https://s3.amazonaws.com/ossci-windows/mkl_2018.2.185.7z --output %TMP_DIR_WIN%\mkl.7z
+    curl -k https://s3.amazonaws.com/ossci-windows/mkl_2019.4.245.7z --output %TMP_DIR_WIN%\mkl.7z
   ) else (
-    aws s3 cp s3://ossci-windows/mkl_2018.2.185.7z %TMP_DIR_WIN%\mkl.7z --quiet
+    aws s3 cp s3://ossci-windows/mkl_2019.4.245.7z %TMP_DIR_WIN%\mkl.7z --quiet
   )
   7z x -aoa %TMP_DIR_WIN%\mkl.7z -o%TMP_DIR_WIN%\mkl
 )

.jenkins/pytorch/win-test-helpers/setup_pytorch_env.bat

@@ -19,9 +19,10 @@ if NOT "%BUILD_ENVIRONMENT%"=="" (
 call %CONDA_PARENT_DIR%\Miniconda3\Scripts\activate.bat %CONDA_PARENT_DIR%\Miniconda3
 if NOT "%BUILD_ENVIRONMENT%"=="" (
     :: We have to pin Python version to 3.6.7, until mkl supports Python 3.7
-    call conda install -y -q python=3.6.7 numpy mkl cffi pyyaml boto3 protobuf numba
+    :: Numba is pinned to 0.44.0 to avoid https://github.com/numba/numba/issues/4352
+    call conda install -y -q python=3.6.7 numpy mkl cffi pyyaml boto3 protobuf numba==0.44.0
 )
-pip install -q ninja future hypothesis "librosa>=0.6.2" psutil
+pip install -q ninja future hypothesis "librosa>=0.6.2" psutil pillow
 :: No need to install faulthandler since we only test Python >= 3.6 on Windows
 :: faulthandler is builtin since Python 3.3
 

.jenkins/pytorch/win-test-helpers/test_custom_script_ops.bat

@@ -1,5 +1,6 @@
 call %SCRIPT_HELPERS_DIR%\setup_pytorch_env.bat
 
+git submodule update --init --recursive third_party/pybind11
 cd test\custom_operator
 
 :: Build the custom operator library.
@@ -23,8 +24,18 @@ popd
 
 :: Run tests Python-side and export a script module.
 python test_custom_ops.py -v
+if ERRORLEVEL 1 exit /b 1
+
+:: TODO: fix and re-enable this test
+:: See https://github.com/pytorch/pytorch/issues/25155
+:: python test_custom_classes.py -v
+:: if ERRORLEVEL 1 exit /b 1
+
 python model.py --export-script-module="build/model.pt"
+if ERRORLEVEL 1 exit /b 1
+
 :: Run tests C++-side and load the exported script module.
 cd build
 set PATH=C:\Program Files\NVIDIA Corporation\NvToolsExt\bin\x64;%TMP_DIR_WIN%\build\torch\lib;%PATH%
 test_custom_ops.exe model.pt
+if ERRORLEVEL 1 exit /b 1

.jenkins/pytorch/win-test-helpers/test_libtorch.bat

@@ -1,9 +1,27 @@
 call %SCRIPT_HELPERS_DIR%\setup_pytorch_env.bat
 
-dir
-dir %TMP_DIR_WIN%\build
-dir %TMP_DIR_WIN%\build\torch
-dir %TMP_DIR_WIN%\build\torch\lib
-cd %TMP_DIR_WIN%\build\torch\lib
+cd %TMP_DIR_WIN%\build\torch\bin
 set PATH=C:\Program Files\NVIDIA Corporation\NvToolsExt\bin\x64;%TMP_DIR_WIN%\build\torch\lib;%PATH%
 test_api.exe --gtest_filter="-IntegrationTest.MNIST*"
+
+if errorlevel 1 exit /b 1
+
+cd %TMP_DIR_WIN%\build\torch\test
+for /r "." %%a in (*.exe) do (
+    call :libtorch_check "%%~na" "%%~fa"
+)
+
+goto :eof
+
+:libtorch_check
+:: See https://github.com/pytorch/pytorch/issues/25161
+if "%~1" == "c10_metaprogramming_test" goto :eof
+if "%~1" == "module_test" goto :eof
+:: See https://github.com/pytorch/pytorch/issues/25312
+if "%~1" == "converter_nomigraph_test" goto :eof
+
+echo Running "%~2"
+call "%~2"
+if errorlevel 1 exit /b 1
+
+goto :eof

.jenkins/pytorch/win-test-helpers/test_python_all_except_nn.bat

@@ -1,2 +1,3 @@
 call %SCRIPT_HELPERS_DIR%\setup_pytorch_env.bat
 cd test && python run_test.py --exclude nn --verbose && cd ..
+if ERRORLEVEL 1 exit /b 1

.jenkins/pytorch/win-test-helpers/upload_image.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python
+
 import os
 import sys
 import boto3

.travis.yml

@@ -1,58 +0,0 @@
-# https://travis-ci.org/pytorch/pytorch
-language: python
-dist: trusty
-git:
-  submodules: false
-
-# This reportedly works around an issue downloading packages from pypi on
-# travis.  Consider removing this after the underlying issue is fixed.
-# https://github.com/travis-ci/travis-ci/issues/2389
-sudo: false
-
-matrix:
-    fast_finish: true
-    include:
-      - name: "Ensure consistent CircleCI YAML"
-        python: "3.6"
-        dist: xenial
-        script: cd .circleci && ./ensure-consistency.py
-      - name: "Shellcheck Jenkins scripts"
-        dist: xenial
-        install: sudo apt-get install -y shellcheck
-        script: .jenkins/run-shellcheck.sh
-      - name: "Ensure no tabs"
-        python: "2.7"
-        script:
-          - (! git grep -I -l $'\t' -- . ':(exclude)*.svg' ':(exclude)**Makefile' ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude).gitattributes' ':(exclude).gitmodules' || (echo "The above files have tabs; please convert them to spaces"; false))
-      - name: "Python 2.7 Lint"
-        python: "2.7"
-        install: pip install flake8
-        script: flake8
-      - name: "Python 3.7 Lint"
-        python: "3.7"
-        dist: xenial    # required for Python 3.7 (travis-ci/travis-ci#9069)
-        sudo: required  # required for Python 3.7 (travis-ci/travis-ci#9069)
-        install:
-          - pip install flake8 flake8-mypy flake8-comprehensions flake8-pyi mccabe pycodestyle pyflakes
-          # Apparently Facebook runs master of this one
-          # https://github.com/PyCQA/flake8-bugbear/issues/53
-          - pip install git+https://github.com/PyCQA/flake8-bugbear.git@d9444713a51a9fb6ee8cd2d88fca85e9ff0c2d58
-        script: flake8
-      - name: "MyPy typecheck"
-        python: "3.6"
-        install: pip install mypy mypy-extensions
-        script: mypy @mypy-files.txt
-      - name: "CPP doc check"
-        python: "3.6"
-        install:
-          - sudo apt-get install -y doxygen
-          - pip install -r requirements.txt
-        script: cd docs/cpp/source && ./check-doxygen.sh
-      - name: "clang tidy"
-        python: "3.6"
-        script: tools/run-clang-tidy-in-ci.sh
-
-branches:
-  only:
-  - master
-  - /gh\/.*\/base/

CMakeLists.txt

@@ -5,11 +5,25 @@ cmake_minimum_required(VERSION 3.5 FATAL_ERROR)
 # Use compiler ID "AppleClang" instead of "Clang" for XCode.
 # Not setting this sometimes makes XCode C compiler gets detected as "Clang",
 # even when the C++ one is detected as "AppleClang".
+cmake_policy(SET CMP0010 NEW)
 cmake_policy(SET CMP0025 NEW)
 
+# Suppress warning flags in default MSVC configuration.  It's not
+# mandatory that we do this (and we don't if cmake is old), but it's
+# nice when it's possible, and it's possible on our Windows configs.
+if(NOT CMAKE_VERSION VERSION_LESS 3.15.0)
+  cmake_policy(SET CMP0092 NEW)
+endif()
+
 # ---[ Project and semantic versioning.
 project(Caffe2 CXX C)
 
+if (${CMAKE_SYSTEM_NAME} STREQUAL "Linux")
+  set(LINUX TRUE)
+else()
+  set(LINUX FALSE)
+endif()
+
 set(CMAKE_INSTALL_MESSAGE NEVER)
 
 set(CMAKE_CXX_STANDARD 11)
@@ -19,6 +33,7 @@ endif()
 if (DEFINED GLIBCXX_USE_CXX11_ABI)
   if (${GLIBCXX_USE_CXX11_ABI} EQUAL 1)
     set(CXX_STANDARD_REQUIRED ON)
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_CXX11_ABI=1")
   endif()
 endif()
 
@@ -62,12 +77,28 @@ if(APPLE)
   set(CMAKE_MACOSX_RPATH ON)
 endif()
 
+if (${CMAKE_HOST_SYSTEM_PROCESSOR} MATCHES "(x86_64|i[3-6]+86)")
+  set(CPU_INTEL ON)
+else ()
+  set(CPU_INTEL OFF)
+endif ()
+
+# For non-supported platforms, turn USE_DISTRIBUTED off by default.
+# It is not tested and likely won't work without additional changes.
+if(NOT LINUX)
+  set(USE_DISTRIBUTED OFF CACHE STRING "Use distributed")
+  # On macOS, if USE_DISTRIBUTED is enabled (specified by the user),
+  # then make Gloo build with the libuv transport.
+  if(APPLE AND USE_DISTRIBUTED)
+    set(USE_LIBUV ON CACHE STRING "")
+  endif()
+endif()
+
 # ---[ Options.
 # Note to developers: if you add an option below, make sure you also add it to
 # cmake/Summary.cmake so that the summary prints out the option values.
 include(CMakeDependentOption)
 option(ATEN_NO_TEST "Do not build ATen test binaries" OFF)
-option(BUILD_ATEN_ONLY "Build only a subset focused on ATen only" OFF)
 option(BUILD_BINARY "Build C++ binaries" OFF)
 option(BUILD_DOCS "Build Caffe2 documentation" OFF)
 option(BUILD_CUSTOM_PROTOBUF "Build and use Caffe2's own protobuf under third_party" ON)
@@ -75,6 +106,8 @@ option(BUILD_PYTHON "Build Python binaries" ON)
 option(BUILD_CAFFE2_OPS "Build Caffe2 operators" ON)
 option(BUILD_SHARED_LIBS "Build libcaffe2.so" ON)
 option(BUILD_CAFFE2_MOBILE "Build libcaffe2 for mobile (deprecating)" ON)
+option(BUILD_NAMEDTENSOR "Experimental: compile with namedtensor support" OFF)
+option(USE_STATIC_DISPATCH "Use static dispatch for ATen operators" OFF)
 cmake_dependent_option(
     CAFFE2_LINK_LOCAL_PROTOBUF "If set, build protobuf inside libcaffe2.so." ON
     "BUILD_SHARED_LIBS AND BUILD_CUSTOM_PROTOBUF" OFF)
@@ -93,8 +126,10 @@ option(CAFFE2_STATIC_LINK_CUDA "Statically link CUDA libraries" OFF)
 cmake_dependent_option(
     USE_CUDNN "Use cuDNN" ON
     "USE_CUDA" OFF)
-option(USE_FBGEMM "Use FBGEMM (quantized 8-bit server operators)" OFF)
-option(NAMEDTENSOR_ENABLED "Experimental: compile with namedtensor support" OFF)
+cmake_dependent_option(
+    USE_STATIC_CUDNN "Use cuDNN static libraries" OFF
+    "USE_CUDNN" OFF)
+option(USE_FBGEMM "Use FBGEMM (quantized 8-bit server operators)" ON)
 option(USE_FFMPEG "Use ffmpeg" OFF)
 option(USE_GFLAGS "Use GFLAGS" OFF)
 option(USE_GLOG "Use GLOG" OFF)
@@ -103,11 +138,20 @@ option(USE_LITE_PROTO "Use lite protobuf instead of full." OFF)
 option(USE_LMDB "Use LMDB" OFF)
 option(USE_METAL "Use Metal for iOS build" ON)
 option(USE_NATIVE_ARCH "Use -march=native" OFF)
-option(USE_NCCL "Use NCCL" ON)
-option(USE_SYSTEM_NCCL "Use system-wide NCCL" OFF)
+cmake_dependent_option(
+    USE_NCCL "Use NCCL" ON
+    "USE_CUDA;UNIX;NOT APPLE" OFF)
+cmake_dependent_option(
+    USE_STATIC_NCCL "Use static NCCL" OFF
+    "USE_NCCL" OFF)
+cmake_dependent_option(
+    USE_SYSTEM_NCCL "Use system-wide NCCL" OFF
+    "USE_NCCL" OFF)
 option(USE_NNAPI "Use NNAPI" OFF)
 option(USE_NNPACK "Use NNPACK" ON)
-option(USE_NUMA "Use NUMA (only available on Linux)" ON)
+cmake_dependent_option(
+    USE_NUMA "Use NUMA. Only available on Linux." ON
+    "LINUX" OFF)
 cmake_dependent_option(
     USE_NVRTC "Use NVRTC. Only available if USE_CUDA is on." OFF
     "USE_CUDA" OFF)
@@ -118,6 +162,7 @@ option(USE_OPENCV "Use OpenCV" OFF)
 option(USE_OPENMP "Use OpenMP for parallel code" ON)
 option(USE_PROF "Use profiling" OFF)
 option(USE_QNNPACK "Use QNNPACK (quantized 8-bit operators)" ON)
+option(USE_PYTORCH_QNNPACK "Use ATen/QNNPACK (quantized 8-bit operators)" ON)
 option(USE_REDIS "Use Redis" OFF)
 option(USE_ROCKSDB "Use RocksDB" OFF)
 option(USE_SNPE "Use Qualcomm's SNPE library" OFF)
@@ -126,7 +171,13 @@ option(USE_SYSTEM_EIGEN_INSTALL
 option(USE_TENSORRT "Using Nvidia TensorRT library" OFF)
 option(USE_ZMQ "Use ZMQ" OFF)
 option(USE_ZSTD "Use ZSTD" OFF)
-option(USE_MKLDNN "Use MKLDNN" OFF)
+cmake_dependent_option(
+  USE_MKLDNN "Use MKLDNN. Only available on x86 and x86_64." ON
+  "CPU_INTEL" OFF)
+set(MKLDNN_ENABLE_CONCURRENT_EXEC ${USE_MKLDNN})
+cmake_dependent_option(
+    USE_MKLDNN_CBLAS "Use CBLAS in MKLDNN" OFF
+    "USE_MKLDNN" OFF)
 option(USE_DISTRIBUTED "Use distributed" ON)
 cmake_dependent_option(
     USE_MPI "Use MPI for Caffe2. Only available if USE_DISTRIBUTED is on." ON
@@ -134,43 +185,94 @@ cmake_dependent_option(
 cmake_dependent_option(
     USE_GLOO "Use Gloo. Only available if USE_DISTRIBUTED is on." ON
     "USE_DISTRIBUTED" OFF)
-cmake_dependent_option(
-    USE_GLOO_IBVERBS "Use Gloo IB verbs for distributed. Only available if USE_GLOO is on." OFF
-    "USE_GLOO" OFF)
+option(USE_TBB "Use TBB" OFF)
 
 # Used when building Caffe2 through setup.py
-option(BUILDING_WITH_TORCH_LIBS "Tell cmake if Caffe2 is being built alongside torch libs" OFF)
+option(BUILDING_WITH_TORCH_LIBS "Tell cmake if Caffe2 is being built alongside torch libs" ON)
 
 # /Z7 override option
 # When generating debug symbols, CMake default to use the flag /Zi.
 # However, it is not compatible with sccache. So we rewrite it off.
 # But some users don't use sccache; this override is for them.
-option(MSVC_Z7_OVERRIDE "Work around sccache bug by replacing /Zi and /ZI with /Z7 when using MSVC (if you are not using sccache, you can turn this OFF)" ON)
+cmake_dependent_option(
+  MSVC_Z7_OVERRIDE "Work around sccache bug by replacing /Zi and /ZI with /Z7 when using MSVC (if you are not using sccache, you can turn this OFF)" ON
+  "MSVC" OFF)
 
-SET(ONNX_NAMESPACE "onnx_c2" CACHE STRING "onnx namespace")
+set(ONNX_NAMESPACE "onnx_torch" CACHE STRING "A namespace for ONNX; needed to build with other frameworks that share ONNX.")
+
+# This is a fix for a rare build issue on Ubuntu:
+# symbol lookup error: miniconda3/envs/pytorch-py3.7/lib/libmkl_intel_lp64.so: undefined symbol: mkl_blas_dsyrk
+# https://software.intel.com/en-us/articles/symbol-lookup-error-when-linking-intel-mkl-with-gcc-on-ubuntu
+if(LINUX)
+  set(CMAKE_SHARED_LINKER_FLAGS "-Wl,--no-as-needed")
+endif()
 
-# For MSVC,
-# 1. Replace /Zi and /ZI with /Z7
-# 2. Switch off incremental linking in debug builds
 if (MSVC)
-  if(MSVC_Z7_OVERRIDE)
-    foreach(flag_var
-        CMAKE_CXX_FLAGS CMAKE_CXX_FLAGS_DEBUG CMAKE_CXX_FLAGS_RELEASE
-        CMAKE_CXX_FLAGS_MINSIZEREL CMAKE_CXX_FLAGS_RELWITHDEBINFO)
+  foreach(flag_var
+      CMAKE_C_FLAGS CMAKE_C_FLAGS_DEBUG CMAKE_C_FLAGS_RELEASE
+      CMAKE_C_FLAGS_MINSIZEREL CMAKE_C_FLAGS_RELWITHDEBINFO
+      CMAKE_CXX_FLAGS CMAKE_CXX_FLAGS_DEBUG CMAKE_CXX_FLAGS_RELEASE
+      CMAKE_CXX_FLAGS_MINSIZEREL CMAKE_CXX_FLAGS_RELWITHDEBINFO)
+    # Replace /Zi and /ZI with /Z7
+    if(MSVC_Z7_OVERRIDE)
       if(${flag_var} MATCHES "/Z[iI]")
         string(REGEX REPLACE "/Z[iI]" "/Z7" ${flag_var} "${${flag_var}}")
       endif(${flag_var} MATCHES "/Z[iI]")
-    endforeach(flag_var)
-  endif(MSVC_Z7_OVERRIDE)
+    endif(MSVC_Z7_OVERRIDE)
+    # Turn off warnings on Windows.  In an ideal world we'd be warning
+    # clean on Windows too, but this is too much work for our
+    # non-Windows developers.
+    #
+    # NB: Technically, this is not necessary if CMP0092 was applied
+    # properly, but only cmake >= 3.15 has this policy, so we nail
+    # it one more time just be safe.
+    #
+    # NB2: This is NOT enough to prevent warnings from nvcc on MSVC.  At the
+    # moment only CMP0092 is enough to prevent those warnings too.
+    string(REPLACE "/W3" "" ${flag_var} "${${flag_var}}")
+    # Suppress EHs is overridden by EHa warning
+    string(REPLACE "/EHsc" "" ${flag_var} "${${flag_var}}")
+
+    # Turn off warnings (Windows build is currently is extremely warning
+    # unclean and the warnings aren't telling us anything useful.)
+    #
+    # Turn on EHa; I'm not altogether clear why we use the asynchronous
+    # exception handling model, but someone added it at some point, so
+    # keep using it.
+    string(APPEND ${flag_var} " /w /EHa")
+
+    if (${CAFFE2_USE_MSVC_STATIC_RUNTIME})
+      if(${flag_var} MATCHES "/MD")
+        string(REGEX REPLACE "/MD" "/MT" ${flag_var} "${${flag_var}}")
+      endif(${flag_var} MATCHES "/MD")
+    else()
+      if(${flag_var} MATCHES "/MT")
+        string(REGEX REPLACE "/MT" "/MD" ${flag_var} "${${flag_var}}")
+      endif()
+    endif()
+
+    # /bigobj increases number of sections in .obj file, which is needed to link
+    # against libaries in Python 2.7 under Windows
+    set(${flag_var} "${${flag_var}} /MP /bigobj")
+  endforeach(flag_var)
+
   foreach(flag_var
       CMAKE_SHARED_LINKER_FLAGS_DEBUG CMAKE_STATIC_LINKER_FLAGS_DEBUG
       CMAKE_EXE_LINKER_FLAGS_DEBUG CMAKE_MODULE_LINKER_FLAGS_DEBUG)
+    # Switch off incremental linking in debug builds
     if(${flag_var} MATCHES "/INCREMENTAL" AND NOT ${flag_var} MATCHES "/INCREMENTAL:NO")
       string(REGEX REPLACE "/INCREMENTAL" "/INCREMENTAL:NO" ${flag_var} "${${flag_var}}")
     endif()
   endforeach(flag_var)
-  # Turning off USE_DISTRIBUTED on default
-  set(USE_DISTRIBUTED OFF)
+
+  foreach(flag_var
+      CMAKE_SHARED_LINKER_FLAGS CMAKE_STATIC_LINKER_FLAGS
+      CMAKE_EXE_LINKER_FLAGS CMAKE_MODULE_LINKER_FLAGS)
+    string(APPEND ${flag_var} " /ignore:4049 /ignore:4217")
+  endforeach(flag_var)
+
+  # Try harder
+  list(APPEND CUDA_NVCC_FLAGS "-Xcompiler /w -w")
 endif(MSVC)
 
 # Set INTERN_BUILD_MOBILE for all mobile builds. Components that are not
@@ -179,6 +281,13 @@ if (ANDROID OR IOS)
   set(INTERN_BUILD_MOBILE ON)
 endif()
 
+# Setting `PYTORCH_BUILD_MOBILE` environment variable can force it to do mobile
+# build with host toolchain.
+if (DEFINED ENV{PYTORCH_BUILD_MOBILE})
+  set(INTERN_BUILD_MOBILE ON)
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DC10_MOBILE")
+endif()
+
 # INTERN_BUILD_ATEN_OPS is used to control whether to build ATen/TH operators.
 # It's disabled for caffe2 mobile library.
 if (INTERN_BUILD_MOBILE AND BUILD_CAFFE2_MOBILE)
@@ -192,27 +301,21 @@ endif()
 # When it's disabled it builds libtorch mobile library, which contains ATen/TH ops and native support for
 # TorchScript model, but doesn't contain not-yet-unified caffe2 ops;
 if (INTERN_BUILD_MOBILE AND NOT BUILD_CAFFE2_MOBILE)
+  if (NOT BUILD_SHARED_LIBS)
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DNO_EXPORT")
+  endif()
   set(BUILD_PYTHON OFF)
-  set(BUILD_TORCH ON)
   set(BUILD_CAFFE2_OPS OFF)
   set(USE_DISTRIBUTED OFF)
   set(FEATURE_TORCH_MOBILE ON)
-endif()
-
-if (BUILD_ATEN_ONLY)
-  set(BUILD_CAFFE2_OPS OFF)
-  set(BUILD_PYTHON OFF)
-  set(USE_NUMA OFF)
-  set(USE_LEVELDB OFF)
-  set(USE_GFLAGS OFF)
-  set(USE_GLOG OFF)
-  set(USE_NCCL OFF)
-  set(USE_NNPACK OFF)
-  set(USE_NUMPY OFF)
-  set(USE_OPENCV OFF)
-  set(USE_MKLDNN OFF)
-  set(USE_DISTRIBUTED OFF)
-  set(USE_LMDB OFF)
+  set(NO_API ON)
+  set(USE_FBGEMM OFF)
+  set(USE_PYTORCH_QNNPACK ON)
+  set(USE_QNNPACK OFF)
+  set(USE_STATIC_DISPATCH ON)
+  set(INTERN_DISABLE_ONNX ON)
+  set(INTERN_DISABLE_AUTOGRAD ON)
+  set(INTERN_USE_EIGEN_BLAS ON)
 endif()
 
 # ---[ Utils
@@ -221,8 +324,12 @@ include(cmake/Utils.cmake)
 include(cmake/public/utils.cmake)
 
 # ---[ Version numbers for generated libraries
-set(TORCH_DEFAULT_VERSION "1.0.0")
+set(TORCH_DEFAULT_VERSION "1.1.0")
 set(TORCH_BUILD_VERSION "${TORCH_DEFAULT_VERSION}" CACHE STRING "Torch build version")
+if (DEFINED ENV{PYTORCH_BUILD_VERSION})
+  set(TORCH_BUILD_VERSION "$ENV{PYTORCH_BUILD_VERSION}"
+    CACHE STRING "Torch build version" FORCE)
+endif()
 if (NOT TORCH_BUILD_VERSION)
   # An empty string was specified so force version to the default
   set(TORCH_BUILD_VERSION "${TORCH_DEFAULT_VERSION}"
@@ -269,8 +376,16 @@ if(USE_FBGEMM)
   set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DUSE_FBGEMM")
 endif()
 
-if(NAMEDTENSOR_ENABLED)
-  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DNAMEDTENSOR_ENABLED")
+if(BUILD_NAMEDTENSOR)
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DBUILD_NAMEDTENSOR")
+endif()
+
+if(USE_QNNPACK)
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DUSE_QNNPACK")
+endif()
+
+if(USE_PYTORCH_QNNPACK)
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DUSE_PYTORCH_QNNPACK")
 endif()
 
 # ---[ Whitelist file if whitelist is specified
@@ -353,25 +468,6 @@ if(NOT MSVC)
   set (CMAKE_LINKER_FLAGS_DEBUG "${CMAKE_STATIC_LINKER_FLAGS_DEBUG} -fno-omit-frame-pointer -O0")
   set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-math-errno")
   set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-trapping-math")
-else()
-  foreach(flag_var
-      CMAKE_CXX_FLAGS CMAKE_CXX_FLAGS_DEBUG CMAKE_CXX_FLAGS_RELEASE
-      CMAKE_CXX_FLAGS_MINSIZEREL CMAKE_CXX_FLAGS_RELWITHDEBINFO
-      CMAKE_C_FLAGS CMAKE_C_FLAGS_DEBUG CMAKE_C_FLAGS_RELEASE
-      CMAKE_C_FLAGS_MINSIZEREL CMAKE_C_FLAGS_RELWITHDEBINFO)
-    if (${CAFFE2_USE_MSVC_STATIC_RUNTIME})
-      if(${flag_var} MATCHES "/MD")
-        string(REGEX REPLACE "/MD" "/MT" ${flag_var} "${${flag_var}}")
-      endif(${flag_var} MATCHES "/MD")
-    else()
-      if(${flag_var} MATCHES "/MT")
-        string(REGEX REPLACE "/MT" "/MD" ${flag_var} "${${flag_var}}")
-      endif()
-    endif()
-    # /bigobj increases number of sections in .obj file, which is needed to link
-    # against libaries in Python 2.7 under Windows
-    set(${flag_var} "${${flag_var}} /MP /bigobj")
-  endforeach(flag_var)
 endif()
 
 if (USE_ASAN)
@@ -418,6 +514,7 @@ include_directories(BEFORE ${PROJECT_SOURCE_DIR})
 include_directories(BEFORE ${PROJECT_BINARY_DIR})
 
 include_directories(BEFORE ${PROJECT_SOURCE_DIR}/aten/src/)
+include_directories(BEFORE ${PROJECT_BINARY_DIR}/aten/src/)
 
 # ---[ Main build
 add_subdirectory(c10)

CODEOWNERS

@@ -4,8 +4,18 @@
 /docs/cpp @goldsborough @ebetica @yf225
 /torch/csrc/api/ @ebetica @goldsborough @yf225
 /test/cpp/api/ @ebetica @goldsborough @yf225
-/torch/lib/c10d/ @apaszke @pietern @mrshenli
-/torch/csrc/distributed/ @apaszke @pietern @mrshenli
+/torch/lib/c10d/ @pietern @mrshenli
+/torch/csrc/distributed/ @pietern @mrshenli
 /torch/distributed/ @apaszke @pietern @mrshenli
-/test/test_c10d.py @apaszke @pietern @mrshenli
-/torch/utils/cpp_extension.py @goldsborough @fmassa @apaszke @soumith @ezyang
+/test/test_c10d.py @pietern @mrshenli
+/torch/utils/cpp_extension.py @goldsborough @fmassa @soumith @ezyang
+
+# Not there to stricly require the approval, but to be tagged as a reviewer
+# on the PRs to push them into a high priority inbox.
+/torch/csrc/api/data/ @apaszke
+/torch/csrc/autograd/ @apaszke
+/torch/csrc/jit/ @apaszke
+/torch/nn/ @apaszke
+/torch/autograd/ @apaszke
+/torch/jit/ @apaszke
+/torch/utils/data/ @apaszke

CONTRIBUTING.md

@@ -185,7 +185,7 @@ pytest test/test_nn.py -k Loss -v
 The above is an example of testing a change to Loss functions: this command runs tests such as
 `TestNN.test_BCELoss` and `TestNN.test_MSELoss` and can be useful to save keystrokes.
 
-## Writing documentation
+## Writing Documentation
 
 PyTorch uses [Google style](http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)
 for formatting docstrings. Length of line inside docstrings block must be limited to 80 characters to
@@ -204,7 +204,87 @@ We run Doxygen in CI (Travis) to verify that you do not use invalid Doxygen
 commands. To run this check locally, run `./check-doxygen.sh` from inside
 `docs/cpp`.
 
-## Managing multiple build trees
+### Building Documentation
+
+To build the documentation:
+
+1. Build and install PyTorch
+
+2. Install the prequesities
+
+```bash
+cd docs
+pip install -r requirements.txt
+# `katex` must also be available in your PATH.
+# If you are using Ubuntu or Debian, you can install it with:
+# sudo apt install katex
+```
+
+3. Generate the documentation HTML files. The generated files will be in `docs/build/html`.
+
+```bash
+cd docs
+make html
+```
+
+4. To view HTML files, you must start an HTTP server. For example
+
+```bash
+# Start a server from the current directory (Python 3 only)
+cd docs/build/html
+python -m http.server
+```
+
+If you are developing on a remote machine, you can set up an SSH tunnel so that
+you can access the HTTP server on the remote machine on your local machine. To map
+remote port 8086 to local port 8086, use either of the following commands.
+
+```bash
+# For SSH
+ssh my_machine -L 8086:my_machine:8086
+
+# For Eternal Terminal
+et my_machine -t="8086:8086"
+```
+
+Then navigate to `localhost:8086` in your web browser.
+
+#### Tips
+
+The `.rst` source files live in [docs/source](docs/source). Some of the `.rst`
+files pull in docstrings from PyTorch Python code (for example, via
+the `autofunction` or `autoclass` directives). To vastly shorten doc build times,
+it is helpful to remove the files you are not working on, only keeping the base
+`index.rst` file and the files you are editing. The Sphinx build will produce
+missing file warnings but will still complete. For example, to work on `jit.rst`:
+
+```bash
+cd docs/source
+ls | grep rst | grep -v index | grep -v jit | xargs rm
+
+# Make your changes, build the docs, etc.
+
+# Don't commit the deletions!
+git add index.rst jit.rst 
+...
+```
+
+
+### Adding Documentation Tests
+
+It is easy for code snippets in docstrings and `.rst` files to get out of date. The docs
+build includes the [Sphinx Doctest Extension](https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html),
+which can run code in documentation as a unit test. To use the extension, use
+the `.. testcode::` directive in your `.rst` and docstrings.
+
+To manually run these tests, follow steps 1 and 2 above, then run:
+
+```bash
+cd docs
+make doctest
+```
+
+## Managing Multiple Build Trees
 
 One downside to using `python setup.py develop` is that your development
 version of PyTorch will be installed globally on your account (e.g., if
@@ -243,18 +323,27 @@ only interested in a specific component.
   Caffe2 operators.
 
 On the initial build, you can also speed things up with the environment
-variables `DEBUG` and `NO_CUDA`.
+variables `DEBUG`, `USE_DISTRIBUTED`, `USE_MKLDNN`, `USE_CUDA`, `BUILD_TEST`, `USE_FBGEMM`, `USE_NNPACK` and `USE_QNNPACK`.
 
 - `DEBUG=1` will enable debug builds (-g -O0)
 - `REL_WITH_DEB_INFO=1` will enable debug symbols with optimizations (-g -O3)
-- `NO_CUDA=1` will disable compiling CUDA (in case you are developing on something not CUDA related), to save compile time.
+- `USE_DISTRIBUTED=0` will disable distributed (c10d, gloo, mpi, etc.) build.
+- `USE_MKLDNN=0` will disable using MKL-DNN.
+- `USE_CUDA=0` will disable compiling CUDA (in case you are developing on something not CUDA related), to save compile time.
+- `BUILD_TEST=0` will disable building C++ test binaries.
+- `USE_FBGEMM=0` will disable using FBGEMM (quantized 8-bit server operators).
+- `USE_NNPACK=0` will disable compiling with NNPACK.
+- `USE_QNNPACK=0` will disable QNNPACK build (quantized 8-bit operators).
 
 For example:
 ```bash
-NO_CUDA=1 DEBUG=1 python setup.py develop
+DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 python setup.py develop
 ```
 
-Make sure you continue to pass these flags on subsequent builds.
+For subsequent builds (i.e., when `build/CMakeCache.txt` exists), the build
+options passed for the first time will persist; please run `ccmake build/`, run
+`cmake-gui build/`, or directly edit `build/CMakeCache.txt` to adapt build
+options.
 
 ### Code completion and IDE support
 
@@ -349,6 +438,16 @@ ccache -F 0
 # deploy (and add to ~/.bashrc for later)
 export PATH="/usr/lib/ccache:$PATH"
 ```
+#### Use a faster linker
+If you are editing a single file and rebuilding in a tight loop, the time spent
+linking will dominate. The system linker available in most Linux distributions
+(GNU `ld`) is quite slow. Use a faster linker, like [lld](https://lld.llvm.org/).
+
+The easiest way to use `lld` this is download the
+[latest LLVM binaries](http://releases.llvm.org/download.html#8.0.0) and run:
+```
+ln -s /path/to/downloaded/ld.lld /usr/local/bin/ld
+```
 
 ## CUDA Development tips
 
@@ -359,6 +458,39 @@ If you are working on the CUDA code, here are some useful CUDA debugging tips:
     slow down the build process for about 50% (compared to only `DEBUG=1`), so use wisely.
 2. `cuda-gdb` and `cuda-memcheck` are your best CUDA debugging friends. Unlike`gdb`,
    `cuda-gdb` can display actual values in a CUDA tensor (rather than all zeros).
+3. CUDA supports a lot of C++11 features such as, `std::numeric_limits`, `std::nextafter`,
+   `std::tuple` etc. in device code. Many of such features are possible because of the
+   [--expt-relaxed-constexpr](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#constexpr-functions)
+   nvcc flag. There is a known [issue](https://github.com/ROCm-Developer-Tools/HIP/issues/374)
+   that ROCm errors out on device code, which uses such stl functions.
+4. A good performance metric for a CUDA kernel is the
+   [Effective Memory Bandwidth](https://devblogs.nvidia.com/how-implement-performance-metrics-cuda-cc/).
+   It is useful for you to measure this metric whenever you are writing/optimizing a CUDA
+   kernel. Following script shows how we can measure the effective bandwidth of CUDA `uniform_`
+   kernel.
+   ```python
+   import torch
+   import time
+   size = 128*512
+   nrep = 100
+   nbytes_read_write = 4 # this is number of bytes read + written by a kernel. Change this to fit your kernel.
+
+   for i in range(10):
+       a=torch.Tensor(size).cuda().uniform_()
+       torch.cuda.synchronize()
+       start = time.time()
+       # dry run to alloc
+       out = a.uniform_()
+       torch.cuda.synchronize()
+       start = time.time()
+       for i in range(nrep):
+         out = a.uniform_()
+       torch.cuda.synchronize()
+       end = time.time()
+       timec = (end-start)/nrep
+       print("uniform, size, elements", size, "forward", timec, "bandwidth (GB/s)", size*(nbytes_read_write)*1e-9/timec)
+       size *=2
+   ```
 
 
 Hope this helps, and thanks for considering to contribute.
@@ -472,6 +604,11 @@ static_assert(std::is_same(A*, decltype(A::singleton()))::value, "hmm");
   This causes preprocessor tokens inside the literal like an`#endif`  to be incorrectly
   treated as preprocessor directives. See https://godbolt.org/z/eVTIJq as an example.
 
+* Either MSVC or the Windows headers have a PURE macro defined and will replace
+  any occurrences of the PURE token in code with an empty string. This is why
+  we have AliasAnalysisKind::PURE_FUNCTION and not AliasAnalysisKind::PURE.
+  The same is likely true for other identifiers that we just didn't try to use yet.
+
 ### Running Clang-Tidy
 
 [Clang-Tidy](https://clang.llvm.org/extra/clang-tidy/index.html) is a C++
@@ -505,7 +642,8 @@ which is in PyTorch's `requirements.txt`.
 
 ### Pre-commit Tidy/Linting Hook
 
-We use clang-tidy and flake8 (installed with flake-mypy) to perform additional
+We use clang-tidy and flake8 (installed with flake8-bugbear,
+flake8-comprehensions, flake8-mypy, and flake8-pyi) to perform additional
 formatting and semantic checking of code. We provide a pre-commit git hook for
 performing these checks, before a commit is created:
 
@@ -517,6 +655,100 @@ You'll need to install an appropriately configured flake8; see
 [Lint as you type](https://github.com/pytorch/pytorch/wiki/Lint-as-you-type)
 for documentation on how to do this.
 
+### Building PyTorch with ASAN
+
+[ASAN](https://github.com/google/sanitizers/wiki/AddressSanitizer) is very
+useful for debugging memory errors in C++. We run it in CI, but here's how to
+get the same thing to run on your local machine.
+
+First, install LLVM 8. The easiest way is to get [prebuilt
+binaries](http://releases.llvm.org/download.html#8.0.0) and extract them to
+folder (later called `$LLVM_ROOT`).
+
+Then set up the appropriate scripts. You can put this in your `.bashrc`:
+
+```
+LLVM_ROOT=<wherever your llvm install is>
+PYTORCH_ROOT=<wherever your pytorch checkout is>
+
+LIBASAN_RT="$LLVM_ROOT/lib/clang/8.0.0/lib/linux/libclang_rt.asan-x86_64.so"
+build_with_asan()
+{
+  LD_PRELOAD=${LIBASAN_RT} \
+  CC="$LLVM_ROOT/bin/clang" \
+  CXX="$LLVM_ROOT/bin/clang++" \
+  LDSHARED="clang --shared" \
+  LDFLAGS="-stdlib=libstdc++" \
+  CFLAGS="-fsanitize=address -fno-sanitize-recover=all -shared-libasan -pthread" \
+  CXX_FLAGS="-pthread" \
+  NO_CUDA=1 USE_OPENMP=0 BUILD_CAFFE2_OPS=0 NO_DISTRIBUTED=1 DEBUG=1 \
+  python setup.py develop
+}
+
+run_with_asan()
+{
+  LD_PRELOAD=${LIBASAN_RT} $@
+}
+
+# you can look at build-asan.sh to find the latest options the CI uses
+export ASAN_OPTIONS=detect_leaks=0:symbolize=1:strict_init_order=true
+export UBSAN_OPTIONS=print_stacktrace=1:suppressions=$PYTORCH_ROOT/ubsan.supp
+export ASAN_SYMBOLIZER_PATH=$LLVM_ROOT/bin/llvm-symbolizer
+```
+
+Then you can use the scripts like:
+
+```
+suo-devfair ~/pytorch ❯ build_with_asan
+suo-devfair ~/pytorch ❯ run_with_asan python test/test_jit.py
+```
+
+#### Getting `ccache` to work
+
+The scripts above specify the `clang` and `clang++` binaries directly, which
+bypasses `ccache`. Here's how to get `ccache` to work:
+
+1. Make sure the ccache symlinks for `clang` and `clang++` are set up (see
+   CONTRIBUTING.md)
+2. Make sure `$LLVM_ROOT/bin` is available on your `$PATH`.
+3. Change the `CC` and `CXX` variables in `build_with_asan()` to point
+   directly to `clang` and `clang++`.
+
+#### Why this stuff with `LD_PRELOAD` and `LIBASAN_RT`?
+
+The “standard” workflow for ASAN assumes you have a standalone binary:
+
+1. Recompile your binary with `-fsanitize=address`.
+2. Run the binary, and ASAN will report whatever errors it find.
+
+Unfortunately, PyTorch is a distributed as a shared library that is loaded by
+a third-party executable (Python). It’s too much of a hassle to recompile all
+of Python every time we want to use ASAN. Luckily, the ASAN folks have a
+workaround for cases like this:
+
+1. Recompile your library with `-fsanitize=address -shared-libasan`. The
+   extra `-shared-libasan` tells the compiler to ask for the shared ASAN
+   runtime library.
+2. Use `LD_PRELOAD` to tell the dynamic linker to load the ASAN runtime
+   library before anything else.
+
+More information can be found
+[here](https://github.com/google/sanitizers/wiki/AddressSanitizerAsDso).
+
+#### Why LD_PRELOAD in the build function?
+
+We need `LD_PRELOAD` because there is a cmake check that ensures that a
+simple program builds and runs. If we are building with ASAN as a shared
+library, we need to `LD_PRELOAD` the runtime library, otherwise there will
+dynamic linker errors and the check will fail.
+
+We don’t actually need either of these if we fix the cmake checks.
+
+#### Why no Leak detection?
+
+Python leaks a lot of memory. Possibly we could configure a suppression file,
+but we haven’t gotten around to it.
+
 ## Caffe2 notes
 
 In 2018, we merged Caffe2 into the PyTorch source repository. While the

README.md

@@ -159,7 +159,7 @@ If you want to compile with CUDA support, install
 - [NVIDIA CUDA](https://developer.nvidia.com/cuda-downloads) 9 or above
 - [NVIDIA cuDNN](https://developer.nvidia.com/cudnn) v7 or above
 
-If you want to disable CUDA support, export environment variable `NO_CUDA=1`.
+If you want to disable CUDA support, export environment variable `USE_CUDA=0`.
 Other potentially useful environment variables may be found in `setup.py`.
 
 If you are building for NVIDIA's Jetson platforms (Jetson Nano, TX1, TX2, AGX Xavier), Instructions to [are available here](https://devtalk.nvidia.com/default/topic/1049071/jetson-nano/pytorch-for-jetson-nano/)
@@ -212,27 +212,69 @@ If the version of Visual Studio 2017 is higher than 15.4.5, installing of "VC++
 NVTX is a part of CUDA distributive, where it is called "Nsight Compute". For installing it onto already installed CUDA run CUDA installation once again and check the corresponding checkbox.
 Be sure that CUDA with Nsight Compute is installed after Visual Studio 2017.
 
+Currently VS 2017, VS 2019 and Ninja are supported as the generator of CMake. If `ninja.exe` is detected in `PATH`, then Ninja will be used as the default generator, otherwise it will use VS 2017.
+<br/> If Ninja is selected as the generator, the latest MSVC which is newer than VS 2015 (14.0) will get selected as the underlying toolchain if you have Python > 3.5, otherwise VS 2015 will be selected so you'll have to activate the environment. If you use CMake <= 3.14.2 and has VS 2019 installed, then even if you specify VS 2017 as the generator, VS 2019 will get selected as the generator.
+
+CUDA and MSVC has strong version dependencies, so even if you use VS 2017 / 2019, you will get build errors like `nvcc fatal : Host compiler targets unsupported OS`. For this kind of problem, please install the corresponding VS toolchain in the table below and then you can either specify the toolset during activation (recommended) or set `CUDAHOSTCXX` to override the cuda host compiler (not recommended if there are big version differences).
+
+| CUDA version | Newest supported VS version                             |
+| ------------ | ------------------------------------------------------- |
+| 9.0 / 9.1    | Visual Studio 2017 Update 4 (15.4) (`_MSC_VER` <= 1911) |
+| 9.2          | Visual Studio 2017 Update 5 (15.5) (`_MSC_VER` <= 1912) |
+| 10.0         | Visual Studio 2017 (15.X) (`_MSC_VER` < 1920)           |
+| 10.1         | Visual Studio 2019 (16.X) (`_MSC_VER` < 1930)           |
+
 ```cmd
 cmd
-REM [Optional] The following two lines are needed for Python 2.7, but the support for it is very experimental.
+:: [Optional] Only add the next two lines if you need Python 2.7. If you use Python 3, ignore these two lines.
 set MSSdk=1
 set FORCE_PY27_BUILD=1
 
-set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
-set DISTUTILS_USE_SDK=1
+:: [Optional] If you want to build with VS 2019 generator, please change the value in the next line to `Visual Studio 16 2019`.
+:: Note: This value is useless if Ninja is detected. However, you can force that by using `set USE_NINJA=OFF`.
+set CMAKE_GENERATOR=Visual Studio 15 2017
 
-REM Run "Visual Studio 2017 Developer Command Prompt"
-for /f "usebackq tokens=*" %i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=14.11
+:: Read the content in the previous section carefully before you preceed.
+:: [Optional] If you want to override the underlying toolset used by Ninja and Visual Studio with CUDA, please run the following script block.
+:: "Visual Studio 2017 Developer Command Prompt" will be run automatically.
+:: Make sure you have CMake >= 3.12 before you do this when you use the Visual Studio generator.
+:: It's an essential step if you use Python 3.5.
+set CMAKE_GENERATOR_TOOLSET_VERSION=14.11
+set DISTUTILS_USE_SDK=1
+for /f "usebackq tokens=*" %i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=%CMAKE_GENERATOR_TOOLSET_VERSION%
+
+:: [Optional] If you want to override the cuda host compiler
+set CUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\cl.exe
 
 python setup.py install
 
 ```
 
+##### Adjust Build Options (Optional)
+
+You can adjust the configuration of cmake variables optionally (without building first), by doing
+the following. For example, adjusting the pre-detected directories for CuDNN or BLAS can be done
+with such a step.
+
+On Linux
+```bash
+export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
+python setup.py build --cmake-only
+ccmake build  # or cmake-gui build
+```
+
+On macOS
+```bash
+export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
+MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py build --cmake-only
+ccmake build  # or cmake-gui build
+```
+
 ### Docker Image
 
 Dockerfile is supplied to build images with cuda support and cudnn v7. You can pass `-e PYTHON_VERSION=x.y` flag to specify which Python version is to be used by Miniconda, or leave it unset to use the default. Build from pytorch repo directory as docker needs to copy git repo into docker filesystem while building the image.
 ```
-docker build -t pytorch -f docker/pytorch/Dockerfile .
+docker build -t pytorch -f docker/pytorch/Dockerfile .  # [optional] --build-arg WITH_TORCHVISION=0
 ```
 
 You can also pull a pre-built docker image from Docker Hub and run with nvidia-docker,

android/.gitignore

@@ -0,0 +1,16 @@
+local.properties
+**/*.iml
+.gradle
+gradlew*
+gradle/wrapper
+.idea/*
+.externalNativeBuild
+build
+pytorch_android/src/main/cpp/libtorch_include/x86/**
+pytorch_android/src/main/cpp/libtorch_include/x86_64/**
+pytorch_android/src/main/cpp/libtorch_include/armeabi-v7a/**
+pytorch_android/src/main/cpp/libtorch_include/arm64-v8a/**
+pytorch_android/src/main/jniLibs/x86/**
+pytorch_android/src/main/jniLibs/x86_64/**
+pytorch_android/src/main/jniLibs/armeabi-v7a/**
+pytorch_android/src/main/jniLibs/arm64-v8a/**

android/build.gradle

@@ -0,0 +1,41 @@
+buildscript {
+    ext {
+        minSdkVersion = 21
+        targetSdkVersion = 28
+        compileSdkVersion = 28
+        buildToolsVersion = '28.0.3'
+
+        coreVersion = "1.2.0"
+        extJUnitVersion = "1.1.1"
+        runnerVersion = "1.2.0"
+        rulesVersion = "1.2.0"
+        junitVersion = "4.12"
+    }
+
+    repositories {
+        google()
+        mavenLocal()
+        mavenCentral()
+        jcenter()
+    }
+
+    dependencies {
+        classpath 'com.android.tools.build:gradle:3.3.2'
+        classpath "com.jfrog.bintray.gradle:gradle-bintray-plugin:${GRADLE_BINTRAY_PLUGIN_VERSION}"
+        classpath "com.github.dcendents:android-maven-gradle-plugin:${ANDROID_MAVEN_GRADLE_PLUGIN_VERSION}"
+        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.9.8"
+    }
+}
+
+allprojects {
+    repositories {
+        google()
+        jcenter()
+    }
+}
+
+ext.isPublishing = { ['uploadArchives', 'bintrayUpload'].any { gradle.startParameter.taskNames.contains(it) } }
+
+ext.deps = [
+        jsr305: 'com.google.code.findbugs:jsr305:3.0.1',
+]

android/gradle.properties

@@ -0,0 +1,24 @@
+ABI_FILTERS=armeabi-v7a,arm64-v8a,x86,x86_64
+
+VERSION_NAME=1.3.0
+GROUP=org.pytorch
+MAVEN_GROUP=org.pytorch
+POM_URL=https://github.com/pytorch/pytorch/tree/master/android
+POM_SCM_URL=https://github.com/pytorch/pytorch.git
+POM_SCM_CONNECTION=scm:git:https://github.com/pytorch/pytorch
+POM_SCM_DEV_CONNECTION=scm:git:git@github.com:pytorch/pytorch.git
+POM_LICENSE_NAME=BSD 3-Clause
+POM_LICENSE_URL=https://github.com/pytorch/pytorch/blob/master/LICENSE
+POM_ISSUES_URL=https://github.com/pytorch/pytorch/issues
+POM_LICENSE_DIST=repo
+POM_DEVELOPER_ID=pytorch
+POM_DEVELOPER_NAME=pytorch
+syncWithMavenCentral=true
+
+GRADLE_BINTRAY_PLUGIN_VERSION=1.8.0
+GRADLE_VERSIONS_PLUGIN_VERSION=0.15.0
+ANDROID_MAVEN_GRADLE_PLUGIN_VERSION=2.1
+
+# Gradle internals
+org.gradle.internal.repository.max.retries=1
+org.gradle.jvmargs=-XX:MaxMetaspaceSize=1024m

android/gradle/android_maven_install.gradle

@@ -0,0 +1,32 @@
+apply plugin: 'com.github.dcendents.android-maven'
+
+version = VERSION_NAME
+group = GROUP
+project.archivesBaseName = POM_ARTIFACT_ID
+
+install {
+    repositories.mavenInstaller {
+        pom.project {
+            name POM_NAME
+            artifactId POM_ARTIFACT_ID
+            packaging POM_PACKAGING
+            description POM_DESCRIPTION
+            url projectUrl
+
+            scm {
+                url scmUrl
+                connection scmConnection
+                developerConnection scmDeveloperConnection
+            }
+
+            licenses projectLicenses
+
+            developers {
+                developer {
+                    id developerId
+                    name developerName
+                }
+            }
+        }
+    }
+}

android/gradle/android_tasks.gradle

@@ -0,0 +1,95 @@
+
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.io.FileOutputStream
+import java.util.zip.ZipFile
+
+// Android tasks for Javadoc and sources.jar generation
+
+afterEvaluate { project ->
+    if (POM_PACKAGING == 'aar') {
+        task androidJavadoc(type: Javadoc, dependsOn: assembleDebug) {
+            source += files(android.sourceSets.main.java.srcDirs)
+            failOnError false
+            // This task will try to compile *everything* it finds in the above directory and
+            // will choke on text files it doesn't understand.
+            exclude '**/BUCK'
+            exclude '**/*.md'
+        }
+
+        task androidJavadocJar(type: Jar, dependsOn: androidJavadoc) {
+            classifier = 'javadoc'
+            from androidJavadoc.destinationDir
+        }
+
+        task androidSourcesJar(type: Jar) {
+            classifier = 'sources'
+            from android.sourceSets.main.java.srcDirs
+        }
+
+        android.libraryVariants.all { variant ->
+            def name = variant.name.capitalize()
+            task "jar${name}"(type: Jar, dependsOn: variant.javaCompileProvider) {
+                from variant.javaCompileProvider.get().destinationDir
+            }
+
+            androidJavadoc.doFirst {
+                classpath += files(android.bootClasspath)
+                classpath += files(variant.javaCompileProvider.get().classpath.files)
+                // This is generated by `assembleDebug` and holds the JARs generated by the APT.
+                classpath += fileTree(dir: "$buildDir/intermediates/bundles/debug/", include: '**/*.jar')
+
+                // Process AAR dependencies
+                def aarDependencies = classpath.filter { it.name.endsWith('.aar') }
+                classpath -= aarDependencies
+                aarDependencies.each { aar ->
+                    // Extract classes.jar from the AAR dependency, and add it to the javadoc classpath
+                    def outputPath = "$buildDir/tmp/aarJar/${aar.name.replace('.aar', '.jar')}"
+                    classpath += files(outputPath)
+
+                    // Use a task so the actual extraction only happens before the javadoc task is run
+                    dependsOn task(name: "extract ${aar.name}").doLast {
+                        extractEntry(aar, 'classes.jar', outputPath)
+                    }
+                }
+            }
+        }
+
+        artifacts.add('archives', androidJavadocJar)
+        artifacts.add('archives', androidSourcesJar)
+    }
+
+    if (POM_PACKAGING == 'jar') {
+        task javadocJar(type: Jar, dependsOn: javadoc) {
+            classifier = 'javadoc'
+            from javadoc.destinationDir
+        }
+
+        task sourcesJar(type: Jar, dependsOn: classes) {
+            classifier = 'sources'
+            from sourceSets.main.allSource
+        }
+
+        artifacts.add('archives', javadocJar)
+        artifacts.add('archives', sourcesJar)
+    }
+}
+
+// Utility method to extract only one entry in a zip file
+private def extractEntry(archive, entryPath, outputPath) {
+    if (!archive.exists()) {
+        throw new GradleException("archive $archive not found")
+    }
+
+    def zip = new ZipFile(archive)
+    zip.entries().each {
+        if (it.name == entryPath) {
+            def path = Paths.get(outputPath)
+            if (!Files.exists(path)) {
+                Files.createDirectories(path.getParent())
+                Files.copy(zip.getInputStream(it), path)
+            }
+        }
+    }
+    zip.close()
+}

android/gradle/bintray.gradle

@@ -0,0 +1,64 @@
+apply plugin: 'com.jfrog.bintray'
+
+def getBintrayUsername() {
+    return project.hasProperty('bintrayUsername') ? property('bintrayUsername') : System.getenv('BINTRAY_USERNAME')
+}
+
+def getBintrayApiKey() {
+    return project.hasProperty('bintrayApiKey') ? property('bintrayApiKey') : System.getenv('BINTRAY_API_KEY')
+}
+
+def getBintrayGpgPassword() {
+    return project.hasProperty('bintrayGpgPassword') ? property('bintrayGpgPassword') : System.getenv('BINTRAY_GPG_PASSWORD')
+}
+
+def getMavenCentralUsername() {
+    return project.hasProperty('mavenCentralUsername') ? property('mavenCentralUsername') : System.getenv('MAVEN_CENTRAL_USERNAME')
+}
+
+def getMavenCentralPassword() {
+    return project.hasProperty('mavenCentralPassword') ? property('mavenCentralPassword') : System.getenv('MAVEN_CENTRAL_PASSWORD')
+}
+
+def shouldSyncWithMavenCentral() {
+    return project.hasProperty('syncWithMavenCentral') ? property('syncWithMavenCentral').toBoolean() : false
+}
+
+def dryRunOnly() {
+    return project.hasProperty('dryRun') ? property('dryRun').toBoolean() : false
+}
+
+bintray {
+    user = getBintrayUsername()
+    key = getBintrayApiKey()
+    override = false
+    configurations = ['archives']
+    pkg {
+        repo = bintrayRepo
+        userOrg = bintrayUserOrg
+        name = bintrayName
+        desc = bintrayDescription
+        websiteUrl = projectUrl
+        issueTrackerUrl = issuesUrl
+        vcsUrl = scmUrl
+        licenses = [ POM_LICENSE_NAME ]
+        dryRun = dryRunOnly()
+        override = false
+        publish = true
+        publicDownloadNumbers = true
+        version {
+            name = versionName
+            desc = bintrayDescription
+            gpg {
+                sign = true
+                passphrase = getBintrayGpgPassword()
+            }
+            mavenCentralSync {
+                sync = shouldSyncWithMavenCentral()
+                user = getMavenCentralUsername()
+                password = getMavenCentralPassword()
+                close = '1' // If set to 0, you have to manually click release
+            }
+        }
+    }
+}

android/gradle/gradle_maven_push.gradle

@@ -0,0 +1,81 @@
+apply plugin: 'signing'
+
+version = VERSION_NAME
+group = MAVEN_GROUP
+
+def isReleaseBuild() {
+  return !VERSION_NAME.contains('SNAPSHOT')
+}
+
+def getReleaseRepositoryUrl() {
+  return hasProperty('RELEASE_REPOSITORY_URL') ? RELEASE_REPOSITORY_URL
+      : "https://oss.sonatype.org/service/local/staging/deploy/maven2/"
+}
+
+def getSnapshotRepositoryUrl() {
+  return hasProperty('SNAPSHOT_REPOSITORY_URL') ? SNAPSHOT_REPOSITORY_URL
+      : "https://oss.sonatype.org/content/repositories/snapshots/"
+}
+
+def getRepositoryUsername() {
+  return hasProperty('SONATYPE_NEXUS_USERNAME') ? SONATYPE_NEXUS_USERNAME : ""
+}
+
+def getRepositoryPassword() {
+  return hasProperty('SONATYPE_NEXUS_PASSWORD') ? SONATYPE_NEXUS_PASSWORD : ""
+}
+
+afterEvaluate { project ->
+  uploadArchives {
+    repositories {
+      mavenDeployer {
+        beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) }
+
+        pom.groupId = MAVEN_GROUP
+        pom.artifactId = POM_ARTIFACT_ID
+        pom.version = VERSION_NAME
+
+        repository(url: getReleaseRepositoryUrl()) {
+          authentication(userName: getRepositoryUsername(), password: getRepositoryPassword())
+        }
+        snapshotRepository(url: getSnapshotRepositoryUrl()) {
+          authentication(userName: getRepositoryUsername(), password: getRepositoryPassword())
+        }
+
+        pom.project {
+          name POM_NAME
+          packaging POM_PACKAGING
+          description POM_DESCRIPTION
+          url POM_URL
+
+          scm {
+            url POM_SCM_URL
+            connection POM_SCM_CONNECTION
+            developerConnection POM_SCM_DEV_CONNECTION
+          }
+
+          licenses {
+            license {
+              name POM_LICENSE_NAME
+              url POM_LICENSE_URL
+              distribution POM_LICENSE_DIST
+            }
+          }
+
+          developers {
+            developer {
+              id POM_DEVELOPER_ID
+              name POM_DEVELOPER_NAME
+            }
+          }
+        }
+      }
+    }
+  }
+
+  signing {
+    required { isReleaseBuild() && gradle.taskGraph.hasTask('uploadArchives') }
+    sign configurations.archives
+  }
+
+}

android/gradle/release.gradle

@@ -0,0 +1,5 @@
+apply from: rootProject.file('gradle/android_tasks.gradle')
+
+apply from: rootProject.file('gradle/release_bintray.gradle')
+
+apply from: rootProject.file('gradle/gradle_maven_push.gradle')

android/gradle/release_bintray.gradle

@@ -0,0 +1,32 @@
+ext {
+    bintrayRepo = 'maven'
+    bintrayUserOrg = 'pytorch'
+    bintrayName = "${GROUP}:${POM_ARTIFACT_ID}"
+    bintrayDescription = POM_DESCRIPTION
+    projectUrl = POM_URL
+    issuesUrl = POM_ISSUES_URL
+    scmUrl = POM_SCM_URL
+    scmConnection = POM_SCM_CONNECTION
+    scmDeveloperConnection = POM_SCM_DEV_CONNECTION
+
+    publishedGroupId = GROUP
+    libraryName = 'pytorch_android'
+    artifact = 'pytorch_android'
+
+    developerId = POM_DEVELOPER_ID
+    developerName = POM_DEVELOPER_NAME
+
+    versionName = VERSION_NAME
+
+    projectLicenses = {
+        license = {
+            name = POM_LICENSE_NAME
+            url = POM_LICENSE_URL
+            distribution = POM_LICENSE_DIST
+        }
+    }
+}
+
+apply from: rootProject.file('gradle/android_maven_install.gradle')
+
+apply from: rootProject.file('gradle/bintray.gradle')

android/libs/fbjni_local/build.gradle

@@ -0,0 +1,47 @@
+apply plugin: 'com.android.library'
+apply plugin: 'maven'
+
+android {
+    compileSdkVersion rootProject.compileSdkVersion
+    buildToolsVersion rootProject.buildToolsVersion
+
+    defaultConfig {
+        minSdkVersion rootProject.minSdkVersion
+        targetSdkVersion rootProject.targetSdkVersion
+
+        sourceSets {
+            main {
+                manifest.srcFile '../fbjni/ApplicationManifest.xml'
+                java {
+                    srcDir '../fbjni/java'
+                }
+            }
+        }
+    }
+    buildTypes {
+        debug {
+            minifyEnabled false
+        }
+        release {
+            minifyEnabled false
+        }
+    }
+    externalNativeBuild {
+        cmake {
+            path "../fbjni/CMakeLists.txt"
+        }
+    }
+}
+
+dependencies {
+    compileOnly 'com.google.code.findbugs:jsr305:3.0.1'
+}
+
+apply from: rootProject.file('gradle/release.gradle')
+
+task sourcesJar(type: Jar) {
+    from android.sourceSets.main.java.srcDirs
+    classifier = 'sources'
+}
+
+artifacts.add('archives', sourcesJar)

android/libs/fbjni_local/gradle.properties

@@ -0,0 +1,4 @@
+POM_NAME=pytorch_android_fbjni
+POM_DESCRIPTION=pytorch_android_fbjni
+POM_ARTIFACT_ID=pytorch_android_fbjni
+POM_PACKAGING=aar

android/pytorch_android/CMakeLists.txt

@@ -0,0 +1,63 @@
+cmake_minimum_required(VERSION 3.4.1)
+project(pytorch CXX)
+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_VERBOSE_MAKEFILE ON)
+
+set(pytorch_android_DIR ${CMAKE_CURRENT_LIST_DIR}/src/main/cpp)
+set(libtorch_include_DIR ${pytorch_android_DIR}/libtorch_include/${ANDROID_ABI})
+
+message(STATUS "libtorch dir:${libtorch_DIR}")
+
+file(GLOB pytorch_android_SOURCES
+  ${pytorch_android_DIR}/*.cpp
+)
+
+add_library(pytorch SHARED
+    ${pytorch_android_SOURCES}
+)
+
+target_compile_options(pytorch PRIVATE
+  -fexceptions
+)
+
+target_include_directories(pytorch PUBLIC
+    ${libtorch_include_DIR}
+)
+
+set(BUILD_DIR ${CMAKE_SOURCE_DIR}/build)
+file(MAKE_DIRECTORY ${BUILD_DIR})
+
+set(fbjni_DIR ${CMAKE_CURRENT_LIST_DIR}/../libs/fbjni/)
+set(fbjni_BUILD_DIR ${BUILD_DIR}/fbjni/${ANDROID_ABI})
+
+add_subdirectory(${fbjni_DIR} ${fbjni_BUILD_DIR})
+
+function(import_static_lib name)
+  add_library(${name} STATIC IMPORTED)
+  set_property(
+      TARGET ${name}
+      PROPERTY IMPORTED_LOCATION
+      ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/${name}.a)
+endfunction(import_static_lib)
+
+import_static_lib(libtorch)
+import_static_lib(libc10)
+import_static_lib(libnnpack)
+import_static_lib(libpytorch_qnnpack)
+import_static_lib(libeigen_blas)
+import_static_lib(libcpuinfo)
+import_static_lib(libclog)
+
+target_link_libraries(pytorch
+    fbjni
+    -Wl,--gc-sections
+    -Wl,--whole-archive
+    libtorch
+    -Wl,--no-whole-archive
+    libc10
+    libnnpack
+    libpytorch_qnnpack
+    libeigen_blas
+    libcpuinfo
+    libclog
+)

android/pytorch_android/build.gradle

@@ -0,0 +1,75 @@
+apply plugin: 'com.android.library'
+apply plugin: 'maven'
+
+android {
+    compileSdkVersion rootProject.compileSdkVersion
+    buildToolsVersion rootProject.buildToolsVersion
+
+    defaultConfig {
+        minSdkVersion rootProject.minSdkVersion
+        targetSdkVersion rootProject.targetSdkVersion
+        versionCode 0
+        versionName "0.1"
+
+        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+        ndk {
+            abiFilters ABI_FILTERS.split(",")
+        }
+    }
+    buildTypes {
+        debug {
+            minifyEnabled false
+            debuggable true
+        }
+        release {
+            minifyEnabled false
+        }
+    }
+    sourceSets {
+        main {
+            jniLibs.srcDirs = ['src/main/jniLibs']
+        }
+    }
+    externalNativeBuild {
+        cmake {
+            path "CMakeLists.txt"
+        }
+    }
+
+    packagingOptions {
+        if (rootProject.isPublishing()) {
+            exclude '**/libfbjni.so'
+        } else {
+            pickFirst '**/libfbjni.so'
+        }
+    }
+
+    useLibrary 'android.test.runner'
+    useLibrary 'android.test.base'
+    useLibrary 'android.test.mock'
+}
+
+dependencies {
+    api project(':fbjni')
+
+    implementation 'com.android.support:appcompat-v7:28.0.0'
+
+    testImplementation 'junit:junit:' + rootProject.junitVersion
+    testImplementation 'androidx.test:core:' + rootProject.coreVersion
+
+    androidTestImplementation 'junit:junit:' + rootProject.junitVersion
+    androidTestImplementation 'androidx.test:core:' + rootProject.coreVersion
+    androidTestImplementation 'androidx.test.ext:junit:' + rootProject.extJUnitVersion
+    androidTestImplementation 'androidx.test:rules:' + rootProject.rulesVersion
+    androidTestImplementation 'androidx.test:runner:' + rootProject.runnerVersion
+}
+
+apply from: rootProject.file('gradle/release.gradle')
+
+task sourcesJar(type: Jar) {
+    from android.sourceSets.main.java.srcDirs
+    classifier = 'sources'
+}
+
+artifacts.add('archives', sourcesJar)
+

android/pytorch_android/generate_test_torchscripts.py

@@ -0,0 +1,111 @@
+import torch
+
+OUTPUT_DIR = "src/androidTest/assets/"
+
+def scriptAndSave(module, fileName):
+    print('-' * 80)
+    script_module = torch.jit.script(module)
+    print(script_module.graph)
+    outputFileName = OUTPUT_DIR + fileName
+    script_module.save(outputFileName)
+    print("Saved to " + outputFileName)
+    print('=' * 80)
+
+class Test(torch.jit.ScriptModule):
+    def __init__(self):
+        super(Test, self).__init__()
+
+    @torch.jit.script_method
+    def forward(self, input):
+        return None
+
+    @torch.jit.script_method
+    def eqBool(self, input):
+        # type: (bool) -> bool
+        return input
+
+    @torch.jit.script_method
+    def eqInt(self, input):
+        # type: (int) -> int
+        return input
+
+    @torch.jit.script_method
+    def eqFloat(self, input):
+        # type: (float) -> float
+        return input
+
+    @torch.jit.script_method
+    def eqStr(self, input):
+        # type: (str) -> str
+        return input
+
+    @torch.jit.script_method
+    def eqTensor(self, input):
+        # type: (Tensor) -> Tensor
+        return input
+
+    @torch.jit.script_method
+    def eqDictStrKeyIntValue(self, input):
+        # type: (Dict[str, int]) -> Dict[str, int]
+        return input
+
+    @torch.jit.script_method
+    def eqDictIntKeyIntValue(self, input):
+        # type: (Dict[int, int]) -> Dict[int, int]
+        return input
+
+    @torch.jit.script_method
+    def eqDictFloatKeyIntValue(self, input):
+        # type: (Dict[float, int]) -> Dict[float, int]
+        return input
+
+    @torch.jit.script_method
+    def listIntSumReturnTuple(self, input):
+        # type: (List[int]) -> Tuple[List[int], int]
+        sum = 0
+        for x in input:
+            sum += x
+        return (input, sum)
+
+    @torch.jit.script_method
+    def listBoolConjunction(self, input):
+        # type: (List[bool]) -> bool
+        res = True
+        for x in input:
+            res = res and x
+        return res
+
+    @torch.jit.script_method
+    def listBoolDisjunction(self, input):
+        # type: (List[bool]) -> bool
+        res = False
+        for x in input:
+            res = res or x
+        return res
+
+    @torch.jit.script_method
+    def tupleIntSumReturnTuple(self, input):
+        # type: (Tuple[int, int, int]) -> Tuple[Tuple[int, int, int], int]
+        sum = 0
+        for x in input:
+            sum += x
+        return (input, sum)
+
+    @torch.jit.script_method
+    def optionalIntIsNone(self, input):
+        # type: (Optional[int]) -> bool
+        return input is None
+
+    @torch.jit.script_method
+    def intEq0None(self, input):
+        # type: (int) -> Optional[int]
+        if input == 0:
+            return None
+        return input
+
+    @torch.jit.script_method
+    def str3Concat(self, input):
+        # type: (str) -> str
+        return input + input + input
+
+scriptAndSave(Test(), "test.pt")