mirror of
https://github.com/pytorch/pytorch.git
synced 2025-10-21 05:34:18 +08:00
5bcbbf5373
Summary: *Context:* https://github.com/pytorch/pytorch/issues/53406 added a lint for trailing whitespace at the ends of lines. However, in order to pass FB-internal lints, that PR also had to normalize the trailing newlines in four of the files it touched. This PR adds an OSS lint to normalize trailing newlines. The changes to the following files (made in 54847d0adb9be71be4979cead3d9d4c02160e4cd) are the only manually-written parts of this PR: - `.github/workflows/lint.yml` - `mypy-strict.ini` - `tools/README.md` - `tools/test/test_trailing_newlines.py` - `tools/trailing_newlines.py` I would have liked to make this just a shell one-liner like the other three similar lints, but nothing I could find quite fit the bill. Specifically, all the answers I tried from the following Stack Overflow questions were far too slow (at least a minute and a half to run on this entire repository): - [How to detect file ends in newline?](https://stackoverflow.com/q/38746) - [How do I find files that do not end with a newline/linefeed?](https://stackoverflow.com/q/4631068) - [How to list all files in the Git index without newline at end of file](https://stackoverflow.com/q/27624800) - [Linux - check if there is an empty line at the end of a file [duplicate]](https://stackoverflow.com/q/34943632) - [git ensure newline at end of each file](https://stackoverflow.com/q/57770972) To avoid giving false positives during the few days after this PR is merged, we should probably only merge it after https://github.com/pytorch/pytorch/issues/54967. Pull Request resolved: https://github.com/pytorch/pytorch/pull/54737 Test Plan: Running the shell script from the "Ensure correct trailing newlines" step in the `quick-checks` job of `.github/workflows/lint.yml` should print no output and exit in a fraction of a second with a status of 0. That was not the case prior to this PR, as shown by this failing GHA workflow run on an earlier draft of this PR: - https://github.com/pytorch/pytorch/runs/2197446987?check_suite_focus=true In contrast, this run (after correcting the trailing newlines in this PR) succeeded: - https://github.com/pytorch/pytorch/pull/54737/checks?check_run_id=2197553241 To unit-test `tools/trailing_newlines.py` itself (this is run as part of our "Test tools" GitHub Actions workflow): ``` python tools/test/test_trailing_newlines.py ``` Reviewed By: malfet Differential Revision: D27409736 Pulled By: samestep fbshipit-source-id: 46f565227046b39f68349bbd5633105b2d2e9b19
91 lines
3.0 KiB
Python
91 lines
3.0 KiB
Python
## @package diagnose_protobuf
|
|
# Module scripts.diagnose_protobuf
|
|
"""Diagnoses the current protobuf situation.
|
|
|
|
Protocol buffer needs to be properly installed for Caffe2 to work, and
|
|
sometimes it is rather tricky. Specifically, we will need to have a
|
|
consistent version between C++ and python simultaneously. This is a
|
|
convenience script for one to quickly check if this is so on one's local
|
|
machine.
|
|
|
|
Usage:
|
|
[set your environmental variables like PATH and PYTHONPATH]
|
|
python scripts/diagnose_protobuf.py
|
|
"""
|
|
|
|
import os
|
|
import re
|
|
from subprocess import Popen, PIPE
|
|
|
|
# Get python protobuf version.
|
|
try:
|
|
import google.protobuf
|
|
python_version = google.protobuf.__version__
|
|
python_protobuf_installed = True
|
|
except ImportError:
|
|
print("DEBUG: cannot find python protobuf install.")
|
|
python_protobuf_installed = False
|
|
|
|
if os.name == 'nt':
|
|
protoc_name = 'protoc.exe'
|
|
else:
|
|
protoc_name = 'protoc'
|
|
|
|
try:
|
|
p = Popen([protoc_name, '--version'], stdout=PIPE, stderr=PIPE)
|
|
out, err = p.communicate()
|
|
except:
|
|
print('DEBUG: did not find protoc binary.')
|
|
print('DEBUG: out: ' + out)
|
|
print('DEBUG: err: ' + err)
|
|
native_protobuf_installed = False
|
|
else:
|
|
if p.returncode:
|
|
print('DEBUG: protoc returned a non-zero return code.')
|
|
print('DEBUG: out: ' + out)
|
|
print('DEBUG: err: ' + err)
|
|
native_protobuf_installed = False
|
|
else:
|
|
tmp = re.search(r'\d\.\d\.\d', out)
|
|
if tmp:
|
|
native_version = tmp.group(0)
|
|
native_protobuf_installed = True
|
|
else:
|
|
print('DEBUG: cannot parse protoc version string.')
|
|
print('DEBUG: out: ' + out)
|
|
native_protobuf_installed = False
|
|
|
|
PYTHON_PROTOBUF_NOT_INSTALLED = """
|
|
You have not installed python protobuf. Protobuf is needed to run caffe2. You
|
|
can install protobuf via pip or conda (if you are using anaconda python).
|
|
"""
|
|
|
|
NATIVE_PROTOBUF_NOT_INSTALLED = """
|
|
You have not installed the protoc binary. Protoc is needed to compile Caffe2
|
|
protobuf source files. Depending on the platform you are on, you can install
|
|
protobuf via:
|
|
(1) Mac: using homebrew and do brew install protobuf.
|
|
(2) Linux: use apt and do apt-get install libprotobuf-dev
|
|
(3) Windows: install from source, or from the releases here:
|
|
https://github.com/google/protobuf/releases/
|
|
"""
|
|
|
|
VERSION_MISMATCH = """
|
|
Your python protobuf is of version {py_ver} but your native protoc version is of
|
|
version {native_ver}. This will cause the installation to produce incompatible
|
|
protobuf files. This is bad in general - consider installing the same version.
|
|
""".format(py_ver=python_version, native_ver=native_version)
|
|
|
|
# Now, give actual recommendations
|
|
if not python_protobuf_installed:
|
|
print(PYTHON_PROTOBUF_NOT_INSTALLED)
|
|
|
|
if not native_protobuf_installed:
|
|
print(NATIVE_PROTOBUF_NOT_INSTALLED)
|
|
|
|
if python_protobuf_installed and native_protobuf_installed:
|
|
if python_version != native_version:
|
|
print(VERSION_MISMATCH)
|
|
else:
|
|
print('All looks good.')
|