mirror of
https://github.com/pytorch/pytorch.git
synced 2025-10-20 12:54:11 +08:00
Reduce set of build/tests which run on PRs. (#20930)
Summary: Resubmit of #20775 Signed-off-by: Edward Z. Yang <ezyang@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/20930 Differential Revision: D15503173 Pulled By: ezyang fbshipit-source-id: a5de8eacf6b29ee26f07ac53c915fff3f4d32569
This commit is contained in:
Edward Yang
committed by
Facebook Github Bot
Facebook Github Bot
parent
c46c6a4fe6
commit
b1df8bfe8a
@ -1,3 +1,23 @@
|
||||
Structure of CI
|
||||
===============
|
||||
|
||||
setup job:
|
||||
1. Does a git checkout
|
||||
2. Persists CircleCI scripts (everything in `.circleci`) into a workspace. Why?
|
||||
We don't always do a Git checkout on all subjobs, but we usually
|
||||
still want to be able to call scripts one way or another in a subjob.
|
||||
Persisting files this way lets us have access to them without doing a
|
||||
checkout. This workspace is conventionally mounted on `~/workspace`
|
||||
(this is distinguished from `~/project`, which is the conventional
|
||||
working directory that CircleCI will default to starting your jobs
|
||||
in.)
|
||||
3. Write out the commit message to `.circleci/COMMIT_MSG`. This is so
|
||||
we can determine in subjobs if we should actually run the jobs or
|
||||
not, even if there isn't a Git checkout.
|
||||
|
||||
|
||||
|
||||
|
||||
CircleCI configuration generator
|
||||
================================
|
||||
|
||||
@ -35,4 +55,4 @@ Future direction
|
||||
See comment [here](https://github.com/pytorch/pytorch/pull/17323#pullrequestreview-206945747):
|
||||
|
||||
In contrast with a full recursive tree traversal of configuration dimensions,
|
||||
> in the future future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
|
||||
> in the future future I think we actually want to decrease our matrix somewhat and have only a few mostly-orthogonal builds that taste as many different features as possible on PRs, plus a more complete suite on every PR and maybe an almost full suite nightly/weekly (we don't have this yet). Specifying PR jobs in the future might be easier to read with an explicit list when we come to this.
|
||||
|
||||
@ -130,12 +130,16 @@ install_doc_push_script: &install_doc_push_script
|
||||
EOL
|
||||
chmod +x /home/circleci/project/doc_push_script.sh
|
||||
|
||||
# `setup_ci_environment` has to be run **after** the ``checkout`` step because
|
||||
# it writes into the checkout directory and otherwise git will complain
|
||||
# that
|
||||
# Directory (/home/circleci/project) you are trying to checkout to is not empty and not git repository
|
||||
# NB: This (and the command below) must be run after attaching
|
||||
# ~/workspace. This is NOT the default working directory (that's
|
||||
# ~/project); this workspace is generated by the setup job.
|
||||
should_run_job: &should_run_job
|
||||
name: Should Run Job After attach_workspace
|
||||
no_output_timeout: "2m"
|
||||
command: ~/workspace/.circleci/scripts/should_run_job.sh
|
||||
|
||||
setup_ci_environment: &setup_ci_environment
|
||||
name: Set Up CI Environment After Checkout
|
||||
name: Set Up CI Environment After attach_workspace
|
||||
no_output_timeout: "1h"
|
||||
command: ~/workspace/.circleci/scripts/setup_ci_environment.sh
|
||||
|
||||
@ -170,8 +174,11 @@ pytorch_linux_build_defaults: &pytorch_linux_build_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- checkout
|
||||
@ -218,11 +225,13 @@ pytorch_linux_test_defaults: &pytorch_linux_test_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- checkout
|
||||
- run:
|
||||
<<: *setup_ci_environment
|
||||
- run:
|
||||
@ -256,8 +265,11 @@ caffe2_linux_build_defaults: &caffe2_linux_build_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- checkout
|
||||
@ -317,10 +329,13 @@ caffe2_linux_test_defaults: &caffe2_linux_test_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_ci_environment
|
||||
- run:
|
||||
@ -386,6 +401,11 @@ caffe2_macos_build_defaults: &caffe2_macos_build_defaults
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- checkout
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
@ -536,8 +556,11 @@ binary_macos_brew_update: &binary_macos_brew_update
|
||||
binary_linux_build: &binary_linux_build
|
||||
resource_class: 2xlarge+
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *binary_checkout
|
||||
- run:
|
||||
@ -586,10 +609,14 @@ binary_linux_test: &binary_linux_test
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
# TODO: We shouldn't attach the workspace multiple times
|
||||
- attach_workspace:
|
||||
at: /home/circleci/project
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -609,8 +636,11 @@ binary_linux_upload: &binary_linux_upload
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -635,8 +665,11 @@ binary_mac_build: &binary_mac_build
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *binary_checkout
|
||||
- run:
|
||||
@ -672,8 +705,11 @@ binary_mac_upload: &binary_mac_upload
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *binary_checkout
|
||||
- run:
|
||||
@ -1010,11 +1046,30 @@ jobs:
|
||||
name: Ensure config is up to date
|
||||
command: ./ensure-consistency.py
|
||||
working_directory: .circleci
|
||||
- run:
|
||||
name: Save commit message
|
||||
command: git log --format='%B' -n 1 HEAD > .circleci/scripts/COMMIT_MSG
|
||||
# Note [Workspace for CircleCI scripts]
|
||||
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
# In the beginning, you wrote your CI scripts in a
|
||||
# .circleci/config.yml file, and life was good. Your CI
|
||||
# configurations flourished and multiplied.
|
||||
#
|
||||
# Then one day, CircleCI cometh down high and say, "Your YAML file
|
||||
# is too biggeth, it stresses our servers so." And thus they
|
||||
# asketh us to smite the scripts in the yml file.
|
||||
#
|
||||
# But you can't just put the scripts in the .circleci folder,
|
||||
# because in some jobs, you don't ever actually checkout the
|
||||
# source repository. Where you gonna get the scripts from?
|
||||
#
|
||||
# Here's how you do it: you persist .circleci/scripts into a
|
||||
# workspace, attach the workspace in your subjobs, and run all
|
||||
# your scripts from there.
|
||||
- persist_to_workspace:
|
||||
root: .
|
||||
paths: .circleci/scripts
|
||||
|
||||
|
||||
pytorch_short_perf_test_gpu:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-short-perf-test-gpu
|
||||
@ -1025,8 +1080,11 @@ jobs:
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -1060,8 +1118,11 @@ jobs:
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -1104,16 +1165,21 @@ jobs:
|
||||
docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
|
||||
|
||||
pytorch_macos_10_13_py3_build:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- checkout
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
- run:
|
||||
name: Build
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
|
||||
no_output_timeout: "1h"
|
||||
command: |
|
||||
set -e
|
||||
@ -1144,44 +1210,50 @@ jobs:
|
||||
- "*"
|
||||
|
||||
pytorch_macos_10_13_py3_test:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
- run:
|
||||
name: Prepare workspace
|
||||
command: |
|
||||
sudo mkdir -p /Users/distiller/pytorch-ci-env
|
||||
sudo chmod -R 777 /Users/distiller/pytorch-ci-env
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
# This workspace also carries binaries from the build job
|
||||
- attach_workspace:
|
||||
at: /Users/distiller/pytorch-ci-env
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
- run:
|
||||
name: Test
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
|
||||
no_output_timeout: "1h"
|
||||
command: |
|
||||
set -e
|
||||
export IN_CIRCLECI=1
|
||||
|
||||
# copy with -a to preserve relative structure (e.g., symlinks), and be recursive
|
||||
cp -a /Users/distiller/pytorch-ci-env/workspace/. /Users/distiller/project
|
||||
# TODO: I'm not sure why we can't just run our job in
|
||||
# ~/workspace and call it a day
|
||||
cp -a ~/workspace/. /Users/distiller/project
|
||||
|
||||
chmod a+x .jenkins/pytorch/macos-test.sh
|
||||
unbuffer .jenkins/pytorch/macos-test.sh 2>&1 | ts
|
||||
|
||||
pytorch_macos_10_13_cuda9_2_cudnn7_py3_build:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- checkout
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
- run:
|
||||
name: Build
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
|
||||
no_output_timeout: "1h"
|
||||
command: |
|
||||
set -e
|
||||
@ -1217,7 +1289,6 @@ jobs:
|
||||
git submodule sync && git submodule update -q --init --recursive
|
||||
chmod a+x .jenkins/pytorch/macos-build.sh
|
||||
unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
|
||||
|
||||
caffe2_py2_gcc4_8_ubuntu14_04_build:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: "caffe2-py2-gcc4.8-ubuntu14.04-build"
|
||||
|
||||
4
.circleci/scripts/README.md
Normal file
4
.circleci/scripts/README.md
Normal file
@ -0,0 +1,4 @@
|
||||
All the scripts in this directory are callable from `~/workspace/.circleci/scripts/foo.sh`.
|
||||
Don't try to call them as `.circleci/scripts/foo.sh`, that won't
|
||||
(necessarily) work. See Note [Workspace for CircleCI scripts] in
|
||||
job-specs-setup.yml for more details.
|
||||
@ -1,46 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -ex -o pipefail
|
||||
|
||||
# Check if we should actually run
|
||||
echo "BUILD_ENVIRONMENT: ${BUILD_ENVIRONMENT}"
|
||||
echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
|
||||
if [[ "${BUILD_ENVIRONMENT}" == *-slow-* ]]; then
|
||||
if ! [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
|
||||
# It's a PR; test for [slow ci] tag on the TOPMOST commit
|
||||
topmost_commit=$(git log --format='%B' -n 1 HEAD)
|
||||
if !(echo $topmost_commit | grep -q -e '\[slow ci\]' -e '\[ci slow\]' -e '\[test slow\]' -e '\[slow test\]'); then
|
||||
circleci step halt
|
||||
exit
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
if [[ "${BUILD_ENVIRONMENT}" == *xla* ]]; then
|
||||
if ! [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
|
||||
# It's a PR; test for [xla ci] tag on the TOPMOST commit
|
||||
topmost_commit=$(git log --format='%B' -n 1 HEAD)
|
||||
if !(echo $topmost_commit | grep -q -e '\[xla ci\]' -e '\[ci xla\]' -e '\[test xla\]' -e '\[xla test\]'); then
|
||||
# NB: This doesn't halt everything, just this job. So
|
||||
# the rest of the workflow will keep going and you need
|
||||
# to make sure you halt there too. Blegh.
|
||||
circleci step halt
|
||||
exit
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
if [[ "${BUILD_ENVIRONMENT}" == *namedtensor* ]]; then
|
||||
if ! [ -z "${CIRCLE_PULL_REQUEST:-}" ]; then
|
||||
# It's a PR; test for [namedtensor] tag on the TOPMOST commit
|
||||
topmost_commit=$(git log --format='%B' -n 1 HEAD)
|
||||
if !(echo $topmost_commit | grep -q -e '\[namedtensor\]' -e '\[ci namedtensor\]' -e '\[namedtensor ci\]'); then
|
||||
# NB: This doesn't halt everything, just this job. So
|
||||
# the rest of the workflow will keep going and you need
|
||||
# to make sure you halt there too. Blegh.
|
||||
circleci step halt
|
||||
exit
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Set up NVIDIA docker repo
|
||||
curl -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
|
||||
echo "deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 /" | sudo tee -a /etc/apt/sources.list.d/nvidia-docker.list
|
||||
|
||||
87
.circleci/scripts/should_run_job.py
Normal file
87
.circleci/scripts/should_run_job.py
Normal file
@ -0,0 +1,87 @@
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
|
||||
# Modify this variable if you want to change the set of default jobs
|
||||
# which are run on all pull requests.
|
||||
|
||||
default_set = [
|
||||
# PyTorch CPU
|
||||
# Selected oldest Python 2 version to ensure Python 2 coverage
|
||||
'pytorch-linux-trusty-py2.7.9',
|
||||
# PyTorch CUDA
|
||||
'pytorch-linux-xenial-cuda9-cudnn7-py3',
|
||||
# PyTorch ASAN
|
||||
'pytorch-linux-xenial-py3-clang5-asan',
|
||||
# PyTorch DEBUG
|
||||
'pytorch-linux-trusty-py3.6-gcc5.4',
|
||||
|
||||
# Caffe2 CPU
|
||||
'caffe2-py2-mkl-ubuntu16.04',
|
||||
# Caffe2 CUDA
|
||||
'caffe2-py2-cuda9.1-cudnn7-ubuntu16.04',
|
||||
# Caffe2 ONNX
|
||||
'caffe2-onnx-py2-gcc5-ubuntu16.04',
|
||||
# Caffe2 Clang
|
||||
'caffe2-py2-clang7-ubuntu16.04',
|
||||
# Caffe2 CMake
|
||||
'caffe2-cmake-cuda9.0-cudnn7-ubuntu16.04',
|
||||
|
||||
# Binaries
|
||||
'manywheel 2.7mu cpu devtoolset3',
|
||||
|
||||
# Caffe2 Android
|
||||
'caffe2-py2-android-ubuntu16.04',
|
||||
# Caffe2 OSX
|
||||
'caffe2-py2-system-macos10.13',
|
||||
# PyTorch OSX
|
||||
'pytorch-macos-10.13-cuda9.2-cudnn7-py3',
|
||||
# PyTorch Android
|
||||
'pytorch-linux-xenial-py3-clang5-android-ndk-r19c',
|
||||
|
||||
# Other checks
|
||||
'pytorch-short-perf-test-gpu',
|
||||
'pytorch-doc-push',
|
||||
]
|
||||
|
||||
# Takes in commit message to analyze via stdin
|
||||
#
|
||||
# This script will query Git and attempt to determine if we should
|
||||
# run the current CI job under question
|
||||
#
|
||||
# NB: Try to avoid hard-coding names here, so there's less place to update when jobs
|
||||
# are updated/renamed
|
||||
#
|
||||
# Semantics in the presence of multiple tags:
|
||||
# - Let D be the set of default builds
|
||||
# - Let S be the set of explicitly specified builds
|
||||
# - Run S \/ D
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('build_environment')
|
||||
args = parser.parse_args()
|
||||
|
||||
commit_msg = sys.stdin.read()
|
||||
|
||||
# Matches anything that looks like [foo ci] or [ci foo] or [foo test]
|
||||
# or [test foo]
|
||||
RE_MARKER = re.compile(r'\[(?:([^ \[\]]+) )?(?:ci|test)(?: ([^ \[\]]+))?\]')
|
||||
|
||||
markers = RE_MARKER.finditer(commit_msg)
|
||||
|
||||
for m in markers:
|
||||
if m.group(1) and m.group(2):
|
||||
print("Unrecognized marker: {}".format(m.group(0)))
|
||||
continue
|
||||
spec = m.group(1) or m.group(2)
|
||||
if spec in args.build_environment or spec == 'all':
|
||||
print("Accepting {} due to commit marker {}".format(args.build_environment, m.group(0)))
|
||||
sys.exit(0)
|
||||
|
||||
for spec in default_set:
|
||||
if spec in args.build_environment:
|
||||
print("Accepting {} as part of default set".format(args.build_environment))
|
||||
sys.exit(0)
|
||||
|
||||
print("Rejecting {}".format(args.build_environment))
|
||||
sys.exit(1)
|
||||
27
.circleci/scripts/should_run_job.sh
Executable file
27
.circleci/scripts/should_run_job.sh
Executable file
@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
set -exu -o pipefail
|
||||
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
|
||||
# Check if we should actually run
|
||||
echo "BUILD_ENVIRONMENT: ${BUILD_ENVIRONMENT:-}"
|
||||
echo "CIRCLE_PULL_REQUEST: ${CIRCLE_PULL_REQUEST:-}"
|
||||
if [ -z "${BUILD_ENVIRONMENT:-}" ]; then
|
||||
echo "Cannot run should_run_job.sh if BUILD_ENVIRONMENT is not defined!"
|
||||
echo "CircleCI scripts are probably misconfigured."
|
||||
exit 1
|
||||
fi
|
||||
if ! [ -e "$SCRIPT_DIR/COMMIT_MSG" ]; then
|
||||
echo "Cannot run should_run_job.sh if you don't have COMMIT_MSG"
|
||||
echo "written out. Are you perhaps running the wrong copy of this script?"
|
||||
echo "You should be running the copy in ~/workspace; SCRIPT_DIR=$SCRIPT_DIR"
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "${CIRCLE_PULL_REQUEST:-}" ]; then
|
||||
# Don't swallow "script doesn't exist
|
||||
[ -e "$SCRIPT_DIR/should_run_job.py" ]
|
||||
if ! python "$SCRIPT_DIR/should_run_job.py" "${BUILD_ENVIRONMENT:-}" < "$SCRIPT_DIR/COMMIT_MSG" ; then
|
||||
circleci step halt
|
||||
exit
|
||||
fi
|
||||
fi
|
||||
@ -130,12 +130,16 @@ install_doc_push_script: &install_doc_push_script
|
||||
EOL
|
||||
chmod +x /home/circleci/project/doc_push_script.sh
|
||||
|
||||
# `setup_ci_environment` has to be run **after** the ``checkout`` step because
|
||||
# it writes into the checkout directory and otherwise git will complain
|
||||
# that
|
||||
# Directory (/home/circleci/project) you are trying to checkout to is not empty and not git repository
|
||||
# NB: This (and the command below) must be run after attaching
|
||||
# ~/workspace. This is NOT the default working directory (that's
|
||||
# ~/project); this workspace is generated by the setup job.
|
||||
should_run_job: &should_run_job
|
||||
name: Should Run Job After attach_workspace
|
||||
no_output_timeout: "2m"
|
||||
command: ~/workspace/.circleci/scripts/should_run_job.sh
|
||||
|
||||
setup_ci_environment: &setup_ci_environment
|
||||
name: Set Up CI Environment After Checkout
|
||||
name: Set Up CI Environment After attach_workspace
|
||||
no_output_timeout: "1h"
|
||||
command: ~/workspace/.circleci/scripts/setup_ci_environment.sh
|
||||
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
|
||||
pytorch_short_perf_test_gpu:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-short-perf-test-gpu
|
||||
@ -9,8 +8,11 @@
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -44,8 +46,11 @@
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -88,16 +93,21 @@
|
||||
docker push ${DEBUG_COMMIT_DOCKER_IMAGE}
|
||||
|
||||
pytorch_macos_10_13_py3_build:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- checkout
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
- run:
|
||||
name: Build
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-build
|
||||
no_output_timeout: "1h"
|
||||
command: |
|
||||
set -e
|
||||
@ -128,44 +138,50 @@
|
||||
- "*"
|
||||
|
||||
pytorch_macos_10_13_py3_test:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
- run:
|
||||
name: Prepare workspace
|
||||
command: |
|
||||
sudo mkdir -p /Users/distiller/pytorch-ci-env
|
||||
sudo chmod -R 777 /Users/distiller/pytorch-ci-env
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
# This workspace also carries binaries from the build job
|
||||
- attach_workspace:
|
||||
at: /Users/distiller/pytorch-ci-env
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
- run:
|
||||
name: Test
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-py3-test
|
||||
no_output_timeout: "1h"
|
||||
command: |
|
||||
set -e
|
||||
export IN_CIRCLECI=1
|
||||
|
||||
# copy with -a to preserve relative structure (e.g., symlinks), and be recursive
|
||||
cp -a /Users/distiller/pytorch-ci-env/workspace/. /Users/distiller/project
|
||||
# TODO: I'm not sure why we can't just run our job in
|
||||
# ~/workspace and call it a day
|
||||
cp -a ~/workspace/. /Users/distiller/project
|
||||
|
||||
chmod a+x .jenkins/pytorch/macos-test.sh
|
||||
unbuffer .jenkins/pytorch/macos-test.sh 2>&1 | ts
|
||||
|
||||
pytorch_macos_10_13_cuda9_2_cudnn7_py3_build:
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- checkout
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
- run:
|
||||
name: Build
|
||||
environment:
|
||||
BUILD_ENVIRONMENT: pytorch-macos-10.13-cuda9.2-cudnn7-py3-build
|
||||
no_output_timeout: "1h"
|
||||
command: |
|
||||
set -e
|
||||
@ -201,4 +217,3 @@
|
||||
git submodule sync && git submodule update -q --init --recursive
|
||||
chmod a+x .jenkins/pytorch/macos-build.sh
|
||||
unbuffer .jenkins/pytorch/macos-build.sh 2>&1 | ts
|
||||
|
||||
|
||||
@ -8,6 +8,26 @@
|
||||
name: Ensure config is up to date
|
||||
command: ./ensure-consistency.py
|
||||
working_directory: .circleci
|
||||
- run:
|
||||
name: Save commit message
|
||||
command: git log --format='%B' -n 1 HEAD > .circleci/scripts/COMMIT_MSG
|
||||
# Note [Workspace for CircleCI scripts]
|
||||
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
# In the beginning, you wrote your CI scripts in a
|
||||
# .circleci/config.yml file, and life was good. Your CI
|
||||
# configurations flourished and multiplied.
|
||||
#
|
||||
# Then one day, CircleCI cometh down high and say, "Your YAML file
|
||||
# is too biggeth, it stresses our servers so." And thus they
|
||||
# asketh us to smite the scripts in the yml file.
|
||||
#
|
||||
# But you can't just put the scripts in the .circleci folder,
|
||||
# because in some jobs, you don't ever actually checkout the
|
||||
# source repository. Where you gonna get the scripts from?
|
||||
#
|
||||
# Here's how you do it: you persist .circleci/scripts into a
|
||||
# workspace, attach the workspace in your subjobs, and run all
|
||||
# your scripts from there.
|
||||
- persist_to_workspace:
|
||||
root: .
|
||||
paths: .circleci/scripts
|
||||
|
||||
@ -4,8 +4,11 @@
|
||||
binary_linux_build: &binary_linux_build
|
||||
resource_class: 2xlarge+
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *binary_checkout
|
||||
- run:
|
||||
@ -54,10 +57,14 @@ binary_linux_test: &binary_linux_test
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
# TODO: We shouldn't attach the workspace multiple times
|
||||
- attach_workspace:
|
||||
at: /home/circleci/project
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
@ -77,8 +84,11 @@ binary_linux_upload: &binary_linux_upload
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
|
||||
@ -9,8 +9,11 @@ pytorch_linux_build_defaults: &pytorch_linux_build_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- checkout
|
||||
@ -57,11 +60,13 @@ pytorch_linux_test_defaults: &pytorch_linux_test_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- checkout
|
||||
- run:
|
||||
<<: *setup_ci_environment
|
||||
- run:
|
||||
@ -95,8 +100,11 @@ caffe2_linux_build_defaults: &caffe2_linux_build_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- checkout
|
||||
@ -156,10 +164,13 @@ caffe2_linux_test_defaults: &caffe2_linux_test_defaults
|
||||
machine:
|
||||
image: ubuntu-1604:201903-01
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *setup_linux_system_environment
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *setup_ci_environment
|
||||
- run:
|
||||
|
||||
@ -7,8 +7,11 @@ binary_mac_build: &binary_mac_build
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *binary_checkout
|
||||
- run:
|
||||
@ -44,8 +47,11 @@ binary_mac_upload: &binary_mac_upload
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- run:
|
||||
<<: *binary_checkout
|
||||
- run:
|
||||
|
||||
@ -7,6 +7,11 @@ caffe2_macos_build_defaults: &caffe2_macos_build_defaults
|
||||
macos:
|
||||
xcode: "9.0"
|
||||
steps:
|
||||
# See Note [Workspace for CircleCI scripts] in job-specs-setup.yml
|
||||
- attach_workspace:
|
||||
at: ~/workspace
|
||||
- run:
|
||||
<<: *should_run_job
|
||||
- checkout
|
||||
- run:
|
||||
<<: *macos_brew_update
|
||||
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@ -85,6 +85,7 @@ torch/version.py
|
||||
# Root level file used in CI to specify certain env configs.
|
||||
# E.g., see .circleci/config.yaml
|
||||
env
|
||||
.circleci/scripts/COMMIT_MSG
|
||||
|
||||
# IPython notebook checkpoints
|
||||
.ipynb_checkpoints
|
||||
|
||||
Reference in New Issue
Block a user