Summary:
The lint was originally added in https://github.com/pytorch/pytorch/issues/54974, but at the time I didn't realize that these other Markdown files also each have a table of contents:
- `GLOSSARY.md`
- `torch/csrc/jit/OVERVIEW.md`
- `torch/csrc/jit/docs/serialization.md`
- `torch/fx/OVERVIEW.md`
This PR adds those files to the lint, and also changes the rule from using a fixed list of filenames to a `git grep` command that finds all Markdown files containing this magic comment:
```md
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56487
Test Plan: The "Lint / toc" job in GitHub Actions.
Reviewed By: janeyx99
Differential Revision: D27884885
Pulled By: samestep
fbshipit-source-id: 5462437502b17fba93abf5098e21754bf566a4fe
Summary:
This is an automated pull request to update the first-party submodule for [pytorch/tensorpipe](https://github.com/pytorch/tensorpipe).
New submodule commit: 416a9d8a4a
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56259
Test Plan: Ensure that CI jobs succeed on GitHub before landing.
Reviewed By: pbelevich
Differential Revision: D27881993
Pulled By: beauby
fbshipit-source-id: e7d8cefe89c6fb09b59e3ef57da05a7ab0a3cb16
Summary:
We should iterate all pages of the branches API. Otherwise, even using "pytorch/vision" would fail to find master.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56138
Reviewed By: heitorschueroff
Differential Revision: D27872346
Pulled By: ailzhang
fbshipit-source-id: 55881558f7980b1fb08b0d08ed6687a38df06edd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56410
Changes:
- Move create_tcp_store() helper function to common file
- Update test_jit_c10d to retry TCP Store creation in case allocated port becomes used
fixes https://github.com/pytorch/pytorch/issues/55053
Test Plan: Imported from OSS
Reviewed By: heitorschueroff
Differential Revision: D27869560
Pulled By: H-Huang
fbshipit-source-id: f4a6613049bb25e6f6f194214379a380968bb19c
Summary:
`is_variable` spits out a deprecation warning during the build (if it's
still something that needs to be tested we can ignore deprecated
warnings for the whole test instead of this change).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56305
Pulled By: driazati
Reviewed By: ezyang
Differential Revision: D27834218
fbshipit-source-id: c7bbea7e9d8099bac232a3a732a27e4cd7c7b950
Summary:
[First ShellCheck release in over a year!](https://github.com/koalaman/shellcheck/releases/tag/v0.7.2) I'm thankful for doing https://github.com/pytorch/pytorch/issues/55109 at the beginning of this month, because otherwise `master` would have just suddenly started failing a few hours ago.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56445
Test Plan:
CI. You can also run `shellcheck` locally; for instance, if you're on Mac and [installed it with Homebrew](https://github.com/koalaman/shellcheck/tree/v0.7.2#installing):
```sh
brew upgrade shellcheck
rm -r .extracted_scripts ; tools/extract_scripts.py --out=.extracted_scripts
tools/run_shellcheck.sh .jenkins/pytorch .extracted_scripts
```
Reviewed By: janeyx99
Differential Revision: D27874084
Pulled By: samestep
fbshipit-source-id: 3bd871a368fe03aecd559e2f55bce36af49cfa27
Summary:
This cuts out caffe2's old backtrace generation in favor of the one already in c10.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56198
Pulled By: driazati
Reviewed By: nikithamalgifb
Differential Revision: D27868282
fbshipit-source-id: aa9b9691271eaa3f95baab48773ffefebd924ae2
Summary:
Temporary fix to give people extra time to finish the deprecation.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56401
Reviewed By: xw285cornell, drdarshan
Differential Revision: D27862196
Pulled By: albanD
fbshipit-source-id: ed460267f314a136941ba550b904dee0321eb0c6
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56248
`info` error code for QR decomposition only indicates wrong parameters,
when everything is implemented correctly it will never be nonzero so we
don't need to check it for CPU path.
For MAGMA `checkMagmaInternalError` is added that checks for failed
memory allocations internal to MAGMA.
Test Plan: Imported from OSS
Reviewed By: ngimel
Differential Revision: D27850414
Pulled By: mruberry
fbshipit-source-id: ddda1209008f879f24c9ad08739e10c28b194d18
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56386
The diff resolves bug around incorrect handler resolution:
_create_static_handler pointed towards etcd, and _create_etcd_handler pointed towards static.
Test Plan:
buck test mode/dev-nosan //caffe2/test/distributed:test_launcher
Added test_launcher to the ci/cd tests
Reviewed By: cbalioglu
Differential Revision: D27858897
fbshipit-source-id: 440155789958c091ce5755e7c9524e4bb704203a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/55946
As `ddp_gpu_size` field of `SyncBatchNorm` will always be 1 for GPU modules, remove this field and the relevant code.
ghstack-source-id: 126883498
Test Plan: waitforbuildbot
Reviewed By: zhaojuanmao
Differential Revision: D27746021
fbshipit-source-id: b4518c07e6f0c6943fbd7a7548500a7d4337126c
Summary: `Redirects` was renamed to `Std` in `torch.distributed.elastic.multiprocessing.api`. Pointed out by a user in https://github.com/pytorch/elastic/issues/147.
Test Plan: N/A just doc change
Reviewed By: tierex
Differential Revision: D27866614
fbshipit-source-id: 9fb901aae7ebe11cde13000a1c118de527f34400
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56391
Previously we only support keeping output quantized for tensor output, this PR adds support
for list and dict (values) as well
Test Plan: Imported from OSS
Reviewed By: vkuzo
Differential Revision: D27860327
fbshipit-source-id: e770160ced47a7173abff5505ec620bd2b1a0b01
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56406
Been hard to reproduce
https://github.com/pytorch/pytorch/issues/50840, adding some debug log to get a
better sense of the issue.
ghstack-source-id: 126874222
Test Plan: waitforbuildbot
Reviewed By: rohan-varma
Differential Revision: D27863328
fbshipit-source-id: e6f125b77cfb636b90598eb54395609654f5e139
Summary:
Partial fix for https://github.com/pytorch/pytorch/issues/56357
Changes the `fuseLoops` API to the following form:
```
static bool fuseLoops(const std::vector<For*>& loops, For** fused);
```
Also, adds a new API to check for loop-carried dependences:
```
static bool hasLoopCarriedDependence(For* loop);
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56353
Reviewed By: bertmaher
Differential Revision: D27856214
Pulled By: navahgar
fbshipit-source-id: 443557088692585657faee296602c547a00117dd
Summary:
https://github.com/pytorch/pytorch/issues/54268 removed `test_run_mypy` since now we're running `mypy` as its own job in GitHub Actions, but previously we used this `set_cwd` context manager in that test to ensure that we picked up the `mypy` config correctly. However, for some reason, we have not been doing that in `test_doc_examples`, which has been succeeding in CI for a while despite being broken.
Specifically, [`run_test.py` changes the working directory to `test/` before running test files](48aaea3359/test/run_test.py (L534-L535)), which is contrary to [what `CONTRIBUTING.md` instructs developers to do](48aaea3359/CONTRIBUTING.md (python-unit-testing)). As a result, in CI, `test/test_type_hints.py` has been passing in CI, but if you run it locally from the root of the repo, this you get this error:
```
F
======================================================================
FAIL: test_doc_examples (__main__.TestTypeHints)
Run documentation examples through mypy.
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_type_hints.py", line 127, in test_doc_examples
self.fail(f"mypy failed:\n{stdout}")
AssertionError: mypy failed:
test/generated_type_hints_smoketest.py:851: error: Name 'tensor' is not defined [name-defined]
test/generated_type_hints_smoketest.py:853: error: Name 'tensor' is not defined [name-defined]
Found 2 errors in 1 file (checked 1 source file)
----------------------------------------------------------------------
Ran 1 test in 1.416s
FAILED (failures=1)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56388
Test Plan:
Before this PR, the first of the following two commands should fail (since that is essentially what is run in CI), but the second should fail:
```
python test/run_test.py -i test_type_hints
python test/test_type_hints.py
```
After this PR, both commands should succeed.
Reviewed By: driazati
Differential Revision: D27860173
Pulled By: samestep
fbshipit-source-id: efb82fffd7ccb04d0331824b40bdef7bbc319c98
Summary:
This guards some deprecated usages of the Protobuf API behind an `#ifdef` (this is how onnx does it as well)
](https://our.intern.facebook.com/intern/diff/27803121/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56186
Pulled By: driazati
Reviewed By: bertmaher, dzhulgakov
Differential Revision: D27803121
fbshipit-source-id: 2d3a348ec1ab9879a0d8f2dff17c5444fd4baf2c
Summary:
Since we're using specific VS, we don't need to specify VC version.
In fact, the VC version is not used in CI now.
Why I make this change now?
I'm writing a robot to update the vs_install.ps1 (https://github.com/pytorch/pytorch/pull/56261/) every 2 weeks.
It will submit a PR to check if the latest VS is compatible with PyTorch automatically.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56230
Reviewed By: bdhirsh
Differential Revision: D27856647
Pulled By: ezyang
fbshipit-source-id: b46f2bdf35ab5841fded470e23bbf7a01d5f60f4
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56181
Need to change to size_t vs size_t:
Reviewed By: ezyang
Differential Revision: D27800849
fbshipit-source-id: 25f744128eb8750c382dc967a99af3c9f16247d9
Summary:
As this diff shows, currently there are a couple hundred instances of raw `noqa` in the codebase, which just ignore all errors on a given line. That isn't great, so this PR changes all existing instances of that antipattern to qualify the `noqa` with respect to a specific error code, and adds a lint to prevent more of this from happening in the future.
Interestingly, some of the examples the `noqa` lint catches are genuine attempts to qualify the `noqa` with a specific error code, such as these two:
```
test/jit/test_misc.py:27: print(f"{hello + ' ' + test}, I'm a {test}") # noqa E999
test/jit/test_misc.py:28: print(f"format blank") # noqa F541
```
However, those are still wrong because they are [missing a colon](https://flake8.pycqa.org/en/3.9.1/user/violations.html#in-line-ignoring-errors), which actually causes the error code to be completely ignored:
- If you change them to anything else, the warnings will still be suppressed.
- If you add the necessary colons then it is revealed that `E261` was also being suppressed, unintentionally:
```
test/jit/test_misc.py:27:57: E261 at least two spaces before inline comment
test/jit/test_misc.py:28:35: E261 at least two spaces before inline comment
```
I did try using [flake8-noqa](https://pypi.org/project/flake8-noqa/) instead of a custom `git grep` lint, but it didn't seem to work. This PR is definitely missing some of the functionality that flake8-noqa is supposed to provide, though, so if someone can figure out how to use it, we should do that instead.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56272
Test Plan:
CI should pass on the tip of this PR, and we know that the lint works because the following CI run (before this PR was finished) failed:
- https://github.com/pytorch/pytorch/runs/2365189927
Reviewed By: janeyx99
Differential Revision: D27830127
Pulled By: samestep
fbshipit-source-id: d6dcf4f945ebd18cd76c46a07f3b408296864fcb
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56070
**Summary**
Currently, we're returning copies instead of alias on mobile GPU (Metal/Vulkan). As suggested by ailzhang , we could use the JIT pass - `RemoveTensorMutation` to ban mutations ahead of time. I've tested two scenarios as shown below. They both work fine on mobile.
- view
```
class Model (torch.nn.Module):
def forward(self, x):
y = x.view(-1)
z = torch.tensor(2.0).float()
y.add_(z)
return x
m = Model()
x = torch.rand(2, 3)
y = m(x)
```
- transpose
```
class Model (torch.nn.Module):
def forward(self, x):
y = x.transpose(1, 2)
z = torch.tensor(2.0).float()
x.add_(z)
return y
m = Model()
x = torch.rand(1, 2, 3)
y = m(x)
```
As we're adding more ops, we should add more tests to cover all the alias ops - https://github.com/pytorch/pytorch/blob/master/tools/autograd/gen_inplace_or_view_type.py#L31-L80
**Next step**
Synced offline with eellison. Since mutation removal is also being used in ONNX, Static runtime, some jit optimizations, Torch -> TVM, etc, instead of inventing something new, we would continue to make it better in cases where it fails.
Although this JIT pass could work for most of the mobile models, there are cases that it can't cover. What we're going to do next is to implement stub ops for GPU models to let them run on server side, such that users can compare results to see if there is any discrepancy.
ghstack-source-id: 126802123
Test Plan:
- Sandcastle
- CircleCI
Reviewed By: raziel
Differential Revision: D27692683
fbshipit-source-id: 9d1be8a6c0a276032b1907807a54fbe2afd882f9
Summary:
They are unused, unrelated to vectorization, and confusing for code
readers (each of them have 2 overloads that are actually used).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56313
Reviewed By: bdhirsh
Differential Revision: D27854290
Pulled By: ezyang
fbshipit-source-id: 14945ceac39a3f19e5d0f8d762b17f8c2172b966
