frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 21:14:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

[skip ci] Add simple local actions runner (#56439)

Browse Source 
Summary:
This pulls out shell scripts from an action and runs them locally as a first pass at https://github.com/pytorch/pytorch/issues/55847. A helper script extracts specific steps in some order and runs them:

```bash
$ time -p make lint -j 5  # run lint with 5 CPUs
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'flake8-py3' \
        --step 'Run flake8'
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'mypy' \
        --step 'Run mypy'
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'quick-checks' \
        --step 'Ensure no trailing spaces' \
        --step 'Ensure no tabs' \
        --step 'Ensure no non-breaking spaces' \
        --step 'Ensure canonical include' \
        --step 'Ensure no unqualified noqa' \
        --step 'Ensure no direct cub include' \
        --step 'Ensure correct trailing newlines'
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'cmakelint' \
        --step 'Run cmakelint'
quick-checks: Ensure no direct cub include
quick-checks: Ensure canonical include
quick-checks: Ensure no unqualified noqa
quick-checks: Ensure no non-breaking spaces
quick-checks: Ensure no tabs
quick-checks: Ensure correct trailing newlines
cmakelint: Run cmakelint
quick-checks: Ensure no trailing spaces
mypy: Run mypy
Success: no issues found in 1316 source files
Success: no issues found in 56 source files
flake8-py3: Run flake8
./test.py:1:1: F401 'torch' imported but unused
real 13.89
user 199.63
sys 6.08
```

Mypy/flake8 are by far the slowest, but that's mostly just because they're wasting a bunch of work linting the entire repo.

In followup, we could/should:
* Improve ergonomics (i.e. no output unless there are errors)
* Speed up lint by only linting files changes between origin and HEAD
* Add clang-tidy

Pull Request resolved: https://github.com/pytorch/pytorch/pull/56439

Reviewed By: samestep

Differential Revision: D27888027

Pulled By: driazati

fbshipit-source-id: d6f2a59a45e9d725566688bdac8e909210175996
This commit is contained in:
driazati
2021-04-20 12:14:37 -07:00
committed by Facebook GitHub Bot
parent ab20ba4427
commit 43eb21bff3
4 changed files with 241 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
.github/workflows/lint.yml vendored
View File

@ -45,23 +45,23 @@ jobs:
tools/run_shellcheck.sh .jenkins/pytorch .extracted_scripts
- name: Ensure correct trailing newlines
run: |
(! git grep -Il '' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude)**.expect' ':(exclude)tools/clang_format_hash' | tools/trailing_newlines.py || (echo "The above files do not have correct trailing newlines; please normalize them"; false))
(! git --no-pager grep -Il '' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude)**.expect' ':(exclude)tools/clang_format_hash' | tools/trailing_newlines.py || (echo "The above files do not have correct trailing newlines; please normalize them"; false))
- name: Ensure no trailing spaces
run: |
(! git grep -In '[[:blank:]]$' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' || (echo "The above lines have trailing spaces; please remove them"; false))
(! git --no-pager grep -In '[[:blank:]]$' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' || (echo "The above lines have trailing spaces; please remove them"; false))
- name: Ensure no tabs
run: |
(! git grep -In $'\t' -- . ':(exclude)*.svg' ':(exclude)**Makefile' ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude).gitattributes' ':(exclude).gitmodules' || (echo "The above lines have tabs; please convert them to spaces"; false))
(! git --no-pager grep -In $'\t' -- . ':(exclude)*.svg' ':(exclude)**Makefile' ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude).gitattributes' ':(exclude).gitmodules' || (echo "The above lines have tabs; please convert them to spaces"; false))
- name: Ensure no non-breaking spaces
run: |
(! git grep -In $'\u00a0' -- . || (echo "The above lines have non-breaking spaces (U+00A0); please convert them to spaces (U+0020)"; false))
(! git --no-pager grep -In $'\u00a0' -- . || (echo "The above lines have non-breaking spaces (U+00A0); please convert them to spaces (U+0020)"; false))
- name: Ensure canonical include
run: |
(! git grep -In $'#include "' -- ./c10 ./aten ./torch/csrc ':(exclude)aten/src/ATen/native/quantized/cpu/qnnpack/**' || (echo "The above lines have include with quotes; please convert them to #include <xxxx>"; false))
(! git --no-pager grep -In $'#include "' -- ./c10 ./aten ./torch/csrc ':(exclude)aten/src/ATen/native/quantized/cpu/qnnpack/**' || (echo "The above lines have include with quotes; please convert them to #include <xxxx>"; false))
- name: Ensure no unqualified noqa
run: |
# shellcheck disable=SC2016
(! git grep -InP '# noqa(?!: [A-Z]+\d{3})' -- '**.py' ':(exclude)caffe2' || (echo 'The above lines have unqualified `noqa`; please convert them to `noqa: XXXX`'; false))
(! git --no-pager grep -InP '# noqa(?!: [A-Z]+\d{3})' -- '**.py' ':(exclude)caffe2' || (echo 'The above lines have unqualified `noqa`; please convert them to `noqa: XXXX`'; false))
# note that this next step depends on a clean checkout;
# if you run it locally then it will likely to complain
# about all the generated files in torch/test
@ -79,7 +79,7 @@ jobs:
python torch/testing/check_kernel_launches.py |& tee "${GITHUB_WORKSPACE}"/cuda_kernel_launch_checks.txt
- name: Ensure no direct cub include
run: |
(! git grep -I -no $'#include <cub/' -- ./aten ':(exclude)aten/src/ATen/cuda/cub.cuh' || (echo "The above files have direct cub include; please include ATen/cuda/cub.cuh instead and wrap your cub calls in at::native namespace if necessary"; false))
(! git --no-pager grep -I -no $'#include <cub/' -- ./aten ':(exclude)aten/src/ATen/cuda/cub.cuh' || (echo "The above files have direct cub include; please include ATen/cuda/cub.cuh instead and wrap your cub calls in at::native namespace if necessary"; false))
python2-setup-compat:
runs-on: ubuntu-18.04
@ -153,21 +153,23 @@ jobs:
mkdir flake8-output
cd flake8-output
echo "$HEAD_SHA" > commit-sha.txt
- name: Run flake8
- name: Install dependencies
run: |
set -eux
pip install typing-extensions # for tools/translate_annotations.py
pip install -r requirements-flake8.txt
flake8 --version
- name: Run flake8
run: |
set -eux
flake8 | tee "${GITHUB_WORKSPACE}"/flake8-output.txt
cp flake8-output.txt flake8-output/annotations.json
- name: Translate annotations
if: github.event_name == 'pull_request'
env:
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
tools/translate_annotations.py \
--file=flake8-output.txt \
--file="${GITHUB_WORKSPACE}"/flake8-output.txt \
--regex='^(?P<filename>.*?):(?P<lineNumber>\d+):(?P<columnNumber>\d+): (?P<errorCode>\w+\d+) (?P<errorDesc>.*)' \
--commit="$HEAD_SHA" \
> flake8-output/annotations.json
@ -218,10 +220,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y clang-tidy-11
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-11 1000
- name: Run clang-tidy
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
- name: Generate build files
run: |
set -eux
git remote add upstream https://github.com/pytorch/pytorch
@ -245,6 +244,12 @@ jobs:
--native-functions-path aten/src/ATen/native/native_functions.yaml \
--nn-path aten/src
fi
- name: Run clang-tidy
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
set -eux
# Run Clang-Tidy
# The negative filters below are to exclude files that include onnx_pb.h or
@ -299,13 +304,16 @@ jobs:
architecture: x64
- name: Fetch PyTorch
uses: actions/checkout@v2
- name: Run cmakelint
- name: Install dependencies
run: |
set -eux
pip install cmakelint
cmakelint --version
- name: Run cmakelint
run: |
set -eux
git ls-files -z -- bootstrap '*.cmake' '*.cmake.in' '*CMakeLists.txt' | \
grep -E -z -v '^(cmake/Modules/|cmake/Modules_CUDA_fix/)' | \
grep -E -z -v '^(cmake/Modules/|cmake/Modules_CUDA_fix/|cmake/Caffe2Config.cmake.in|aten/src/ATen/ATenConfig.cmake.in|cmake/Caffe2ConfigVersion.cmake.in|cmake/TorchConfig.cmake.in|cmake/TorchConfigVersion.cmake.in|cmake/cmake_uninstall.cmake.in)' | \
xargs -0 cmakelint --config=.cmakelintrc --spaces=2 --quiet
mypy:

34
CONTRIBUTING.md
View File

@ -10,6 +10,7 @@
- [Unit testing](#unit-testing)
- [Python Unit Testing](#python-unit-testing)
- [Better local unit tests with `pytest`](#better-local-unit-tests-with-pytest)
- [Local linting](#local-linting)
- [Running `mypy`](#running-mypy)
- [C++ Unit Testing](#c-unit-testing)
- [Writing documentation](#writing-documentation)
@ -357,13 +358,44 @@ The above is an example of testing a change to all Loss functions: this
command runs tests such as `TestNN.test_BCELoss` and
`TestNN.test_MSELoss` and can be useful to save keystrokes.
### Local linting
You can run the same linting steps that are used in CI locally via `make`:
```bash
make lint -j 6 # run lint (using 6 parallel jobs)
```
These jobs may require extra dependencies that aren't dependencies of PyTorch
itself, so you can install them via this command, which you should only have to
run once:
```bash
make setup_lint
```
To run a specific linting step, use one of these targets or see the
[`Makefile`](Makefile) for a complete list of options.
```bash
# Check for tabs, trailing newlines, etc.
make quick_checks
make flake8
make mypy
make cmakelint
```
### Running `mypy`
`mypy` is an optional static type checker for Python. We have multiple `mypy`
configs for the PyTorch codebase, so you can run them all using this command:
```bash
for CONFIG in mypy*.ini; do mypy --config="$CONFIG"; done
make mypy
```
See [Guide for adding type annotations to

44
Makefile
View File

@ -30,3 +30,47 @@ shellcheck-gha:
generate-gha-workflows:
./.github/scripts/generate_linux_ci_workflows.py
$(MAKE) shellcheck-gha
setup_lint:
python tools/actions_local_runner.py --file .github/workflows/lint.yml \
--job 'flake8-py3' --step 'Install dependencies'
python tools/actions_local_runner.py --file .github/workflows/lint.yml \
--job 'cmakelint' --step 'Install dependencies'
pip install jinja2
quick_checks:
# TODO: This is broken when 'git config submodule.recurse' is 'true'
@python tools/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'quick-checks' \
--step 'Ensure no trailing spaces' \
--step 'Ensure no tabs' \
--step 'Ensure no non-breaking spaces' \
--step 'Ensure canonical include' \
--step 'Ensure no unqualified noqa' \
--step 'Ensure no direct cub include' \
--step 'Ensure correct trailing newlines'
flake8:
@python tools/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'flake8-py3' \
--step 'Run flake8'
mypy:
@python tools/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'mypy' \
--step 'Run mypy'
cmakelint:
@python tools/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'cmakelint' \
--step 'Run cmakelint'
clang_tidy:
echo "clang-tidy local lint is not yet implemented"
exit 1
lint: flake8 mypy quick_checks cmakelint generate-gha-workflows

140
tools/actions_local_runner.py Executable file
View File

@ -0,0 +1,140 @@
#!/bin/python3
import subprocess
import os
import argparse
import yaml
import asyncio
REPO_ROOT = os.path.dirname(os.path.dirname(__file__))
class col:
HEADER = "\033[95m"
BLUE = "\033[94m"
GREEN = "\033[92m"
YELLOW = "\033[93m"
RED = "\033[91m"
RESET = "\033[0m"
BOLD = "\033[1m"
UNDERLINE = "\033[4m"
def color(the_color, text):
return col.BOLD + the_color + str(text) + col.RESET
def cprint(the_color, text):
print(color(the_color, text))
async def run_step(step, job_name):
env = os.environ.copy()
env["GITHUB_WORKSPACE"] = "/tmp"
script = step["run"]
# We don't need to print the commands for local running
# TODO: Either lint that GHA scripts only use 'set -eux' or make this more
# resilient
script = script.replace("set -eux", "set -eu")
try:
proc = await asyncio.create_subprocess_shell(
script,
shell=True,
cwd=REPO_ROOT,
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
stdout, stderr = await proc.communicate()
cprint(col.BLUE, f'{job_name}: {step["name"]}')
except Exception as e:
cprint(col.BLUE, f'{job_name}: {step["name"]}')
print(e)
stdout = stdout.decode().strip()
stderr = stderr.decode().strip()
if stderr != "":
print(stderr)
if stdout != "":
print(stdout)
async def run_steps(steps, job_name):
coros = [run_step(step, job_name) for step in steps]
await asyncio.gather(*coros)
def grab_specific_steps(steps_to_grab, job):
relevant_steps = []
for step in steps_to_grab:
for actual_step in job["steps"]:
if actual_step["name"].lower().strip() == step.lower().strip():
relevant_steps.append(actual_step)
break
if len(relevant_steps) != len(steps_to_grab):
raise RuntimeError("Missing steps")
return relevant_steps
def grab_all_steps_after(last_step, job):
relevant_steps = []
found = False
for step in job["steps"]:
if found:
relevant_steps.append(step)
if step["name"].lower().strip() == last_step.lower().strip():
found = True
return relevant_steps
def main():
parser = argparse.ArgumentParser(
description="Pull shell scripts out of GitHub actions and run them"
)
parser.add_argument("--file", help="YAML file with actions", required=True)
parser.add_argument("--job", help="job name", required=True)
parser.add_argument("--step", action="append", help="steps to run (in order)")
parser.add_argument(
"--all-steps-after", help="include every step after this one (non inclusive)"
)
args = parser.parse_args()
if args.step is None and args.all_steps_after is None:
raise RuntimeError("1+ --steps or --all-steps-after must be provided")
if args.step is not None and args.all_steps_after is not None:
raise RuntimeError("Only one of --step and --all-steps-after can be used")
action = yaml.safe_load(open(args.file, "r"))
if "jobs" not in action:
raise RuntimeError(f"top level key 'jobs' not found in {args.file}")
jobs = action["jobs"]
if args.job not in jobs:
raise RuntimeError(f"job '{args.job}' not found in {args.file}")
job = jobs[args.job]
if args.step is not None:
relevant_steps = grab_specific_steps(args.step, job)
else:
relevant_steps = grab_all_steps_after(args.all_steps_after, job)
# pprint.pprint(relevant_steps)
asyncio.run(run_steps(relevant_steps, args.job))
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
pass