[skip ci] Add simple local actions runner (#56439)

Summary:
This pulls out shell scripts from an action and runs them locally as a first pass at https://github.com/pytorch/pytorch/issues/55847. A helper script extracts specific steps in some order and runs them:

```bash
$ time -p make lint -j 5  # run lint with 5 CPUs
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'flake8-py3' \
        --step 'Run flake8'
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'mypy' \
        --step 'Run mypy'
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'quick-checks' \
        --step 'Ensure no trailing spaces' \
        --step 'Ensure no tabs' \
        --step 'Ensure no non-breaking spaces' \
        --step 'Ensure canonical include' \
        --step 'Ensure no unqualified noqa' \
        --step 'Ensure no direct cub include' \
        --step 'Ensure correct trailing newlines'
python scripts/actions_local_runner.py \
        --file .github/workflows/lint.yml \
        --job 'cmakelint' \
        --step 'Run cmakelint'
quick-checks: Ensure no direct cub include
quick-checks: Ensure canonical include
quick-checks: Ensure no unqualified noqa
quick-checks: Ensure no non-breaking spaces
quick-checks: Ensure no tabs
quick-checks: Ensure correct trailing newlines
cmakelint: Run cmakelint
quick-checks: Ensure no trailing spaces
mypy: Run mypy
Success: no issues found in 1316 source files
Success: no issues found in 56 source files
flake8-py3: Run flake8
./test.py:1:1: F401 'torch' imported but unused
real 13.89
user 199.63
sys 6.08
```

Mypy/flake8 are by far the slowest, but that's mostly just because they're wasting a bunch of work linting the entire repo.

In followup, we could/should:
* Improve ergonomics (i.e. no output unless there are errors)
* Speed up lint by only linting files changes between origin and HEAD
* Add clang-tidy

Pull Request resolved: https://github.com/pytorch/pytorch/pull/56439

Reviewed By: samestep

Differential Revision: D27888027

Pulled By: driazati

fbshipit-source-id: d6f2a59a45e9d725566688bdac8e909210175996

.github/workflows/lint.yml

@@ -45,23 +45,23 @@ jobs:
           tools/run_shellcheck.sh .jenkins/pytorch .extracted_scripts
       - name: Ensure correct trailing newlines
         run: |
-          (! git grep -Il '' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude)**.expect' ':(exclude)tools/clang_format_hash' | tools/trailing_newlines.py || (echo "The above files do not have correct trailing newlines; please normalize them"; false))
+          (! git --no-pager grep -Il '' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude)**.expect' ':(exclude)tools/clang_format_hash' | tools/trailing_newlines.py || (echo "The above files do not have correct trailing newlines; please normalize them"; false))
       - name: Ensure no trailing spaces
         run: |
-          (! git grep -In '[[:blank:]]$' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' || (echo "The above lines have trailing spaces; please remove them"; false))
+          (! git --no-pager grep -In '[[:blank:]]$' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' || (echo "The above lines have trailing spaces; please remove them"; false))
       - name: Ensure no tabs
         run: |
-          (! git grep -In $'\t' -- . ':(exclude)*.svg' ':(exclude)**Makefile' ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude).gitattributes' ':(exclude).gitmodules' || (echo "The above lines have tabs; please convert them to spaces"; false))
+          (! git --no-pager grep -In $'\t' -- . ':(exclude)*.svg' ':(exclude)**Makefile' ':(exclude)**/contrib/**' ':(exclude)third_party' ':(exclude).gitattributes' ':(exclude).gitmodules' || (echo "The above lines have tabs; please convert them to spaces"; false))
       - name: Ensure no non-breaking spaces
         run: |
-          (! git grep -In $'\u00a0' -- . || (echo "The above lines have non-breaking spaces (U+00A0); please convert them to spaces (U+0020)"; false))
+          (! git --no-pager grep -In $'\u00a0' -- . || (echo "The above lines have non-breaking spaces (U+00A0); please convert them to spaces (U+0020)"; false))
       - name: Ensure canonical include
         run: |
-          (! git grep -In $'#include "' -- ./c10 ./aten ./torch/csrc ':(exclude)aten/src/ATen/native/quantized/cpu/qnnpack/**' || (echo "The above lines have include with quotes; please convert them to #include <xxxx>"; false))
+          (! git --no-pager grep -In $'#include "' -- ./c10 ./aten ./torch/csrc ':(exclude)aten/src/ATen/native/quantized/cpu/qnnpack/**' || (echo "The above lines have include with quotes; please convert them to #include <xxxx>"; false))
       - name: Ensure no unqualified noqa
         run: |
           # shellcheck disable=SC2016
-          (! git grep -InP '# noqa(?!: [A-Z]+\d{3})' -- '**.py' ':(exclude)caffe2' || (echo 'The above lines have unqualified `noqa`; please convert them to `noqa: XXXX`'; false))
+          (! git --no-pager grep -InP '# noqa(?!: [A-Z]+\d{3})' -- '**.py' ':(exclude)caffe2' || (echo 'The above lines have unqualified `noqa`; please convert them to `noqa: XXXX`'; false))
       # note that this next step depends on a clean checkout;
       # if you run it locally then it will likely to complain
       # about all the generated files in torch/test
@@ -79,7 +79,7 @@ jobs:
           python torch/testing/check_kernel_launches.py |& tee "${GITHUB_WORKSPACE}"/cuda_kernel_launch_checks.txt
       - name: Ensure no direct cub include
         run: |
-          (! git grep -I -no $'#include <cub/' --  ./aten  ':(exclude)aten/src/ATen/cuda/cub.cuh' || (echo "The above files have direct cub include; please include ATen/cuda/cub.cuh instead and wrap your cub calls in at::native namespace if necessary"; false))
+          (! git --no-pager grep -I -no $'#include <cub/' --  ./aten  ':(exclude)aten/src/ATen/cuda/cub.cuh' || (echo "The above files have direct cub include; please include ATen/cuda/cub.cuh instead and wrap your cub calls in at::native namespace if necessary"; false))
 
   python2-setup-compat:
     runs-on: ubuntu-18.04
@@ -153,21 +153,23 @@ jobs:
           mkdir flake8-output
           cd flake8-output
           echo "$HEAD_SHA" > commit-sha.txt
-      - name: Run flake8
+      - name: Install dependencies
         run: |
           set -eux
           pip install typing-extensions # for tools/translate_annotations.py
           pip install -r requirements-flake8.txt
           flake8 --version
+      - name: Run flake8
+        run: |
+          set -eux
           flake8 | tee "${GITHUB_WORKSPACE}"/flake8-output.txt
-          cp flake8-output.txt flake8-output/annotations.json
       - name: Translate annotations
         if: github.event_name == 'pull_request'
         env:
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
           tools/translate_annotations.py \
-            --file=flake8-output.txt \
+            --file="${GITHUB_WORKSPACE}"/flake8-output.txt \
             --regex='^(?P<filename>.*?):(?P<lineNumber>\d+):(?P<columnNumber>\d+): (?P<errorCode>\w+\d+) (?P<errorDesc>.*)' \
             --commit="$HEAD_SHA" \
             > flake8-output/annotations.json
@@ -218,10 +220,7 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y clang-tidy-11
           sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-11 1000
-      - name: Run clang-tidy
-        env:
-          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+      - name: Generate build files
         run: |
           set -eux
           git remote add upstream https://github.com/pytorch/pytorch
@@ -245,6 +244,12 @@ jobs:
               --native-functions-path aten/src/ATen/native/native_functions.yaml \
               --nn-path aten/src
           fi
+      - name: Run clang-tidy
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -eux
 
           # Run Clang-Tidy
           # The negative filters below are to exclude files that include onnx_pb.h or
@@ -299,13 +304,16 @@ jobs:
           architecture: x64
       - name: Fetch PyTorch
         uses: actions/checkout@v2
-      - name: Run cmakelint
+      - name: Install dependencies
         run: |
           set -eux
           pip install cmakelint
           cmakelint --version
+      - name: Run cmakelint
+        run: |
+          set -eux
           git ls-files -z -- bootstrap '*.cmake' '*.cmake.in' '*CMakeLists.txt' | \
-          grep -E -z -v '^(cmake/Modules/|cmake/Modules_CUDA_fix/)' | \
+          grep -E -z -v '^(cmake/Modules/|cmake/Modules_CUDA_fix/|cmake/Caffe2Config.cmake.in|aten/src/ATen/ATenConfig.cmake.in|cmake/Caffe2ConfigVersion.cmake.in|cmake/TorchConfig.cmake.in|cmake/TorchConfigVersion.cmake.in|cmake/cmake_uninstall.cmake.in)' | \
           xargs -0 cmakelint --config=.cmakelintrc --spaces=2 --quiet
 
   mypy:

CONTRIBUTING.md

@@ -10,6 +10,7 @@
 - [Unit testing](#unit-testing)
   - [Python Unit Testing](#python-unit-testing)
   - [Better local unit tests with `pytest`](#better-local-unit-tests-with-pytest)
+  - [Local linting](#local-linting)
   - [Running `mypy`](#running-mypy)
   - [C++ Unit Testing](#c-unit-testing)
 - [Writing documentation](#writing-documentation)
@@ -357,13 +358,44 @@ The above is an example of testing a change to all Loss functions: this
 command runs tests such as `TestNN.test_BCELoss` and
 `TestNN.test_MSELoss` and can be useful to save keystrokes.
 
+
+### Local linting
+
+You can run the same linting steps that are used in CI locally via `make`:
+
+```bash
+make lint -j 6  # run lint (using 6 parallel jobs)
+```
+
+These jobs may require extra dependencies that aren't dependencies of PyTorch
+itself, so you can install them via this command, which you should only have to
+run once:
+
+```bash
+make setup_lint
+```
+
+To run a specific linting step, use one of these targets or see the
+[`Makefile`](Makefile) for a complete list of options.
+
+```bash
+# Check for tabs, trailing newlines, etc.
+make quick_checks
+
+make flake8
+
+make mypy
+
+make cmakelint
+```
+
 ### Running `mypy`
 
 `mypy` is an optional static type checker for Python. We have multiple `mypy`
 configs for the PyTorch codebase, so you can run them all using this command:
 
 ```bash
-for CONFIG in mypy*.ini; do mypy --config="$CONFIG"; done
+make mypy
 ```
 
 See [Guide for adding type annotations to

Makefile

@@ -30,3 +30,47 @@ shellcheck-gha:
 generate-gha-workflows:
 	./.github/scripts/generate_linux_ci_workflows.py
 	$(MAKE) shellcheck-gha
+
+setup_lint:
+	python tools/actions_local_runner.py --file .github/workflows/lint.yml \
+	 	--job 'flake8-py3' --step 'Install dependencies'
+	python tools/actions_local_runner.py --file .github/workflows/lint.yml \
+	 	--job 'cmakelint' --step 'Install dependencies'
+	pip install jinja2
+
+quick_checks:
+# TODO: This is broken when 'git config submodule.recurse' is 'true'
+	@python tools/actions_local_runner.py \
+		--file .github/workflows/lint.yml \
+		--job 'quick-checks' \
+		--step 'Ensure no trailing spaces' \
+		--step 'Ensure no tabs' \
+		--step 'Ensure no non-breaking spaces' \
+		--step 'Ensure canonical include' \
+		--step 'Ensure no unqualified noqa' \
+		--step 'Ensure no direct cub include' \
+		--step 'Ensure correct trailing newlines'
+
+flake8:
+	@python tools/actions_local_runner.py \
+		--file .github/workflows/lint.yml \
+		--job 'flake8-py3' \
+		--step 'Run flake8'
+
+mypy:
+	@python tools/actions_local_runner.py \
+		--file .github/workflows/lint.yml \
+		--job 'mypy' \
+		--step 'Run mypy'
+
+cmakelint:
+	@python tools/actions_local_runner.py \
+		--file .github/workflows/lint.yml \
+		--job 'cmakelint' \
+		--step 'Run cmakelint'
+
+clang_tidy:
+	echo "clang-tidy local lint is not yet implemented"
+	exit 1
+
+lint: flake8 mypy quick_checks cmakelint generate-gha-workflows

tools/actions_local_runner.py

@@ -0,0 +1,140 @@
+#!/bin/python3
+
+import subprocess
+import os
+import argparse
+import yaml
+import asyncio
+
+
+REPO_ROOT = os.path.dirname(os.path.dirname(__file__))
+
+
+class col:
+    HEADER = "\033[95m"
+    BLUE = "\033[94m"
+    GREEN = "\033[92m"
+    YELLOW = "\033[93m"
+    RED = "\033[91m"
+    RESET = "\033[0m"
+    BOLD = "\033[1m"
+    UNDERLINE = "\033[4m"
+
+
+def color(the_color, text):
+    return col.BOLD + the_color + str(text) + col.RESET
+
+
+def cprint(the_color, text):
+    print(color(the_color, text))
+
+
+async def run_step(step, job_name):
+    env = os.environ.copy()
+    env["GITHUB_WORKSPACE"] = "/tmp"
+    script = step["run"]
+
+    # We don't need to print the commands for local running
+    # TODO: Either lint that GHA scripts only use 'set -eux' or make this more
+    # resilient
+    script = script.replace("set -eux", "set -eu")
+
+    try:
+        proc = await asyncio.create_subprocess_shell(
+            script,
+            shell=True,
+            cwd=REPO_ROOT,
+            env=env,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        stdout, stderr = await proc.communicate()
+        cprint(col.BLUE, f'{job_name}: {step["name"]}')
+    except Exception as e:
+        cprint(col.BLUE, f'{job_name}: {step["name"]}')
+        print(e)
+
+    stdout = stdout.decode().strip()
+    stderr = stderr.decode().strip()
+
+    if stderr != "":
+        print(stderr)
+    if stdout != "":
+        print(stdout)
+
+
+async def run_steps(steps, job_name):
+    coros = [run_step(step, job_name) for step in steps]
+    await asyncio.gather(*coros)
+
+
+def grab_specific_steps(steps_to_grab, job):
+    relevant_steps = []
+    for step in steps_to_grab:
+        for actual_step in job["steps"]:
+            if actual_step["name"].lower().strip() == step.lower().strip():
+                relevant_steps.append(actual_step)
+                break
+
+    if len(relevant_steps) != len(steps_to_grab):
+        raise RuntimeError("Missing steps")
+
+    return relevant_steps
+
+
+def grab_all_steps_after(last_step, job):
+    relevant_steps = []
+
+    found = False
+    for step in job["steps"]:
+        if found:
+            relevant_steps.append(step)
+        if step["name"].lower().strip() == last_step.lower().strip():
+            found = True
+
+    return relevant_steps
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Pull shell scripts out of GitHub actions and run them"
+    )
+    parser.add_argument("--file", help="YAML file with actions", required=True)
+    parser.add_argument("--job", help="job name", required=True)
+    parser.add_argument("--step", action="append", help="steps to run (in order)")
+    parser.add_argument(
+        "--all-steps-after", help="include every step after this one (non inclusive)"
+    )
+    args = parser.parse_args()
+
+    if args.step is None and args.all_steps_after is None:
+        raise RuntimeError("1+ --steps or --all-steps-after must be provided")
+
+    if args.step is not None and args.all_steps_after is not None:
+        raise RuntimeError("Only one of --step and --all-steps-after can be used")
+
+    action = yaml.safe_load(open(args.file, "r"))
+    if "jobs" not in action:
+        raise RuntimeError(f"top level key 'jobs' not found in {args.file}")
+    jobs = action["jobs"]
+
+    if args.job not in jobs:
+        raise RuntimeError(f"job '{args.job}' not found in {args.file}")
+
+    job = jobs[args.job]
+
+    if args.step is not None:
+        relevant_steps = grab_specific_steps(args.step, job)
+    else:
+        relevant_steps = grab_all_steps_after(args.all_steps_after, job)
+
+    # pprint.pprint(relevant_steps)
+    asyncio.run(run_steps(relevant_steps, args.job))
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        pass