Summary:
Previously `make quicklint` would lint all changed files for both mypy `ini`s, regardless of whether that file was actually supposed to be run under that configuration. This PR fixes that so we are using `tools/mypy_wrapper.py` to check if files should be included.
There's a similar change for `flake8` so that it now only outputs errors once and correctly excludes the paths in `.flake8`.
This also adds a bunch of tests to ensure that `make lint` and `make quicklint` both work and that `make quicklint` is excluding and including what it should.
Fixes#57644
](https://our.intern.facebook.com/intern/diff/28259692/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/57752
Pulled By: driazati
Reviewed By: samestep
Differential Revision: D28259692
fbshipit-source-id: 233d355781230f11f98a6f61e2c07e9f5e737e24
Summary:
Fixes#56738
* `setup_lint` now installs mypy / shellcheck
* the shell used to execute commands is pinned to `bash` (on Ubuntu the default is `dash`, which was causing the false positives in #56738)
* the emoji check marks don't always work, so use more basic ones instead
* adds `Run autogen` step for mypy (for the `lint` step only since it's pretty slow)
](https://our.intern.facebook.com/intern/diff/27972006/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56752
Pulled By: driazati
Reviewed By: samestep
Differential Revision: D27972006
fbshipit-source-id: 624e6c1af2d4f7c8623f420516744922b6b829a5
Summary:
This queries the local git repo for changed files (any changed files, not just committed ones) and sends them to mypy/flake8 instead of the default (which is the whole repo, defined the .flake8 and mypy.ini files). This brings a good speedup (from 15 seconds with no cache to < 1 second from my local testing on this PR).
```bash
make quicklint -j 6
```
It should be noted that the results of this aren’t exactly what’s in the CI, since mypy and flake8 ignore the `include` and `exclude` parts of their config when an explicit list of files is passed in.
](https://our.intern.facebook.com/intern/diff/27901577/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56559
Pulled By: driazati
Reviewed By: malfet
Differential Revision: D27901577
fbshipit-source-id: 99f351cdfe5aba007948aea2b8a78f683c5d8583
Summary:
This pulls out shell scripts from an action and runs them locally as a first pass at https://github.com/pytorch/pytorch/issues/55847. A helper script extracts specific steps in some order and runs them:
```bash
$ time -p make lint -j 5 # run lint with 5 CPUs
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'flake8-py3' \
--step 'Run flake8'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'mypy' \
--step 'Run mypy'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'quick-checks' \
--step 'Ensure no trailing spaces' \
--step 'Ensure no tabs' \
--step 'Ensure no non-breaking spaces' \
--step 'Ensure canonical include' \
--step 'Ensure no unqualified noqa' \
--step 'Ensure no direct cub include' \
--step 'Ensure correct trailing newlines'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'cmakelint' \
--step 'Run cmakelint'
quick-checks: Ensure no direct cub include
quick-checks: Ensure canonical include
quick-checks: Ensure no unqualified noqa
quick-checks: Ensure no non-breaking spaces
quick-checks: Ensure no tabs
quick-checks: Ensure correct trailing newlines
cmakelint: Run cmakelint
quick-checks: Ensure no trailing spaces
mypy: Run mypy
Success: no issues found in 1316 source files
Success: no issues found in 56 source files
flake8-py3: Run flake8
./test.py:1:1: F401 'torch' imported but unused
real 13.89
user 199.63
sys 6.08
```
Mypy/flake8 are by far the slowest, but that's mostly just because they're wasting a bunch of work linting the entire repo.
In followup, we could/should:
* Improve ergonomics (i.e. no output unless there are errors)
* Speed up lint by only linting files changes between origin and HEAD
* Add clang-tidy
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56439
Reviewed By: samestep
Differential Revision: D27888027
Pulled By: driazati
fbshipit-source-id: d6f2a59a45e9d725566688bdac8e909210175996
Summary:
When trying to build caffe2 with python provided by homebrew, I find out there are some errors in the building scripts. The "get_python_cmake_flags.py" script is supposed to find out the correct python library and header file locations. However, due to these errors, this script does not function correctly. After building, caffe2 is linked against the default python library provided by Apple which causes a crash when trying to validate whether or not the installation is successful:
```shell
python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"
```
The fix is as simple as follows:
- Add "shell" so that command substitution could work under Makefile.
- Add blank spaces between -D options so that they are treated as options not makefile targets.
- Print the "flags" variable without the newline character so that they could be utilized by command substitution correctly.
Closes https://github.com/caffe2/caffe2/pull/391
Differential Revision: D4943212
Pulled By: Yangqing
fbshipit-source-id: 04d3595fa2d89fe57aed5b6a7a91a95114a82a1b
Summary:
Top-level makefile had `make` hardcoded, resulting in slow build and the following message when following installation instructions:
warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Replacing this recursive make command with the variable MAKE fixes the issue.
Closes https://github.com/caffe2/caffe2/pull/324
Differential Revision: D4920978
Pulled By: Yangqing
fbshipit-source-id: 1e75ab41786e52d1b7abcc2c46ad1088880d8c1d
Summary:
This script will attempt to determine files that will be useful for building with the correct python version. Currently on macOS with various python installations CMake fails to determine the correct location of python libraries.
Closes https://github.com/caffe2/caffe2/pull/163
Reviewed By: Yangqing
Differential Revision: D4594954
Pulled By: bwasti
fbshipit-source-id: c2b750ee9608a02fad4ce2f2293f5fa54dc7011c
(1) nccl submodule, cnmem submodule
(2) mpi ops fallback test
(3) a bit more blob interface
(4) fixed tests
(5) caffe2.python.io -> caffe2.python.dataio to avoid name conflicts
(6) In the build system autogen __init__.py instead of having manual
rules just to copy over an empty __init__.py.
