See the comment [here](https://github.com/pytorch/pytorch/issues/132014#issuecomment-2379547400) (cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov @XilunWu @rec) - this PR updates `_unsafe_set_version_counter` to accept a list of tensors, for overhead-sensitive users (e.g. distributed) who need to hide VC bumps from autograd on a large list of tensors without wanting to suffer the overhead of going from python->C++ separately for every tensor in the list.
I left the binding in pybind, and used a `std::vector`. if we **really** need to optimize overhead even further, we could write a manual cpython binding.
I use this updated API in the next PR to fix FSDP2, so that it properly hides the VC of all `all_gather_buffer` tensors in its call to `split_with_sizes_copy.out(all_gather_buffers)`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/137921
Approved by: https://github.com/awgu, https://github.com/albanD
Fixes#112594
docstring updated.
Here are the output to with the number before and after.
1) torch/autograd/forward_ad.py
Before :
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:23 in public function `enter_dual_level`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:23 in public function `enter_dual_level`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:42 in public function `exit_dual_level`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:42 in public function `exit_dual_level`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:62 in public function `make_dual`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:62 in public function `make_dual`:
D400: First line should end with a period (not 'a')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:128 in public class `UnpackedDualTensor`:
D204: 1 blank line required after class docstring (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:128 in public class `UnpackedDualTensor`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:128 in public class `UnpackedDualTensor`:
D209: Multi-line docstring closing quotes should be on a separate line
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:134 in public function `unpack_dual`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:165 in public class `dual_level`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:165 in public class `dual_level`:
D400: First line should end with a period (not 't')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:199 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:202 in public method `__exit__`:
D105: Missing docstring in magic method
15
```
After:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:205 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/forward_ad.py:208 in public method `__exit__`:
D105: Missing docstring in magic method
3
```
2) torch/autograd/functional.py
Before:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:262 in public function `vjp`:
D202: No blank lines allowed after function docstring (found 1)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:262 in public function `vjp`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:262 in public function `vjp`:
D400: First line should end with a period (not 'e')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:262 in public function `vjp`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:359 in public function `jvp`:
D202: No blank lines allowed after function docstring (found 1)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:359 in public function `jvp`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:359 in public function `jvp`:
D400: First line should end with a period (not 'f')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:359 in public function `jvp`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:584 in public function `jacobian`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:841 in public function `hessian`:
D202: No blank lines allowed after function docstring (found 1)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:841 in public function `hessian`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:973 in public function `vhp`:
D202: No blank lines allowed after function docstring (found 1)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:973 in public function `vhp`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:973 in public function `vhp`:
D400: First line should end with a period (not 'e')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:973 in public function `vhp`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:1076 in public function `hvp`:
D202: No blank lines allowed after function docstring (found 1)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:1076 in public function `hvp`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:1076 in public function `hvp`:
D400: First line should end with a period (not 'r')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:1076 in public function `hvp`:
D401: First line should be in imperative mood; try rephrasing (found 'Function')
20
```
After:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/functional.py:1 at module level:
D100: Missing docstring in public module
1
```
3) torch/autograd/profiler_legacy.py
Before:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:27 in public class `profile`:
D400: First line should end with a period (not 'd')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:29 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:62 in public method `config`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:74 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:86 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:103 in public method `__repr__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:108 in public method `__str__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:117 in public method `table`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:141 in public method `export_chrome_trace`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:148 in public method `export_stacks`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:154 in public method `key_averages`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:161 in public method `total_average`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:170 in public method `self_cpu_time_total`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:170 in public method `self_cpu_time_total`:
D400: First line should end with a period (not 'f')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:180 in private nested function `_get_record_key`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:180 in private nested function `_get_record_key`:
D400: First line should end with a period (not 'd')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:180 in private nested function `_get_record_key`:
D401: First line should be in imperative mood (perhaps 'Return', not 'Returns')
18
```
After:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:29 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:62 in public method `config`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:74 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:86 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:103 in public method `__repr__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:108 in public method `__str__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:117 in public method `table`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:141 in public method `export_chrome_trace`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:148 in public method `export_stacks`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:154 in public method `key_averages`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_legacy.py:161 in public method `total_average`:
D102: Missing docstring in public method
12
```
4) torch/autograd/gradcheck.py
Before:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:27 in public class `GradcheckError`:
D204: 1 blank line required after class docstring (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:27 in public class `GradcheckError`:
D400: First line should end with a period (not '`')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:258 in private function `_get_numerical_jacobian`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:258 in private function `_get_numerical_jacobian`:
D400: First line should end with a period (not 'f')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:258 in private function `_get_numerical_jacobian`:
D401: First line should be in imperative mood (perhaps 'Compute', not 'Computes')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:308 in public function `get_numerical_jacobian`:
D401: First line should be in imperative mood; try rephrasing (found 'Deprecated')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:459 in public function `get_numerical_jacobian_wrt_specific_input`:
D103: Missing docstring in public function
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:488 in private function `_get_analytical_jacobian_forward_ad`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:488 in private function `_get_analytical_jacobian_forward_ad`:
D400: First line should end with a period (not 't')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:488 in private function `_get_analytical_jacobian_forward_ad`:
D401: First line should be in imperative mood (perhaps 'Compute', not 'Computes')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:816 in public function `get_analytical_jacobian`:
D103: Missing docstring in public function
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:1944 in public function `gradcheck`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:1944 in public function `gradcheck`:
D400: First line should end with a period (not 'l')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:2133 in public function `gradgradcheck`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:2133 in public function `gradgradcheck`:
D400: First line should end with a period (not 's')
16
```
After:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:463 in public function `get_numerical_jacobian_wrt_specific_input`:
D103: Missing docstring in public function
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/gradcheck.py:820 in public function `get_analytical_jacobian`:
D103: Missing docstring in public function
3
```
5) torch/autograd/function.py
Before:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:27 in public class `FunctionCtx`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:29 in public method `save_for_backward`:
D401: First line should be in imperative mood (perhaps 'Save', not 'Saves')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:88 in public method `save_for_forward`:
D401: First line should be in imperative mood (perhaps 'Save', not 'Saves')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:141 in public method `mark_dirty`:
D401: First line should be in imperative mood (perhaps 'Mark', not 'Marks')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:177 in public method `mark_shared_storage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:185 in public method `mark_non_differentiable`:
D401: First line should be in imperative mood (perhaps 'Mark', not 'Marks')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:217 in public method `set_materialize_grads`:
D401: First line should be in imperative mood (perhaps 'Set', not 'Sets')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:276 in public class `BackwardCFunction`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:277 in public method `apply`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:291 in public method `apply_jvp`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:308 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:322 in private method `forward`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:322 in private method `forward`:
D400: First line should end with a period (not 's')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:322 in private method `forward`:
D401: First line should be in imperative mood; try rephrasing (found 'This')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:384 in private method `backward`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:384 in private method `backward`:
D400: First line should end with a period (not 'e')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:384 in private method `backward`:
D401: First line should be in imperative mood (perhaps 'Define', not 'Defines')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:416 in private method `jvp`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:416 in private method `jvp`:
D400: First line should end with a period (not 'e')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:416 in private method `jvp`:
D401: First line should be in imperative mood (perhaps 'Define', not 'Defines')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:439 in public class `Function`:
D400: First line should end with a period (not '`')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:472 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:482 in public method `__call__`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:505 in public method `vmap`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:505 in public method `vmap`:
D400: First line should end with a period (not 'h')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:505 in public method `vmap`:
D401: First line should be in imperative mood (perhaps 'Define', not 'Defines')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:536 in public method `apply`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:564 in public function `once_differentiable`:
D103: Missing docstring in public function
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:612 in public function `traceable`:
D401: First line should be in imperative mood (perhaps 'Mark', not 'Marks')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:626 in public class `InplaceFunction`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:627 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:741 in public class `NestedIOFunction`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:761 in public method `backward`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:768 in public method `forward`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:775 in public method `save_for_backward`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:780 in public method `saved_tensors`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:784 in public method `mark_dirty`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:787 in public method `mark_non_differentiable`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:790 in public method `forward_extended`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:793 in public method `backward_extended`:
D102: Missing docstring in public method
41
```
After:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:27 in public class `FunctionCtx`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:177 in public method `mark_shared_storage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:276 in public class `BackwardCFunction`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:277 in public method `apply`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:291 in public method `apply_jvp`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:308 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:471 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:481 in public method `__call__`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:536 in public method `apply`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:564 in public function `once_differentiable`:
D103: Missing docstring in public function
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:626 in public class `InplaceFunction`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:627 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:741 in public class `NestedIOFunction`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:761 in public method `backward`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:768 in public method `forward`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:775 in public method `save_for_backward`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:780 in public method `saved_tensors`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:784 in public method `mark_dirty`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:787 in public method `mark_non_differentiable`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:790 in public method `forward_extended`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/function.py:793 in public method `backward_extended`:
D102: Missing docstring in public method
22
```
6) torch/autograd/profiler_util.py
Before:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:26 in public class `EventList`:
D400: First line should end with a period (not ')')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:28 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:46 in public method `__str__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:70 in private method `_populate_cpu_children`:
D202: No blank lines allowed after function docstring (found 1)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:70 in private method `_populate_cpu_children`:
D205: 1 blank line required between summary line and description (found 0)
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:70 in private method `_populate_cpu_children`:
D401: First line should be in imperative mood (perhaps 'Populate', not 'Populates')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:166 in public method `self_cpu_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:179 in public method `table`:
D401: First line should be in imperative mood (perhaps 'Print', not 'Prints')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:210 in public method `export_chrome_trace`:
D401: First line should be in imperative mood (perhaps 'Export', not 'Exports')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:266 in public method `supported_export_stacks_metrics`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:273 in public method `export_stacks`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:354 in private function `_format_time`:
D400: First line should end with a period (not 't')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:354 in private function `_format_time`:
D401: First line should be in imperative mood (perhaps 'Define', not 'Defines')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:365 in private function `_format_time_share`:
D400: First line should end with a period (not 't')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:365 in private function `_format_time_share`:
D401: First line should be in imperative mood (perhaps 'Define', not 'Defines')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:373 in private function `_format_memory`:
D400: First line should end with a period (not 'g')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:373 in private function `_format_memory`:
D401: First line should be in imperative mood (perhaps 'Return', not 'Returns')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:408 in public method `cpu_time`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:412 in public method `cuda_time`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:416 in public method `privateuse1_time`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:420 in public class `Interval`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:421 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:425 in public method `elapsed_us`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:435 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:488 in public method `append_kernel`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:504 in public method `set_cpu_parent`:
D400: First line should end with a period (not 't')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:518 in public method `self_cpu_memory_usage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:526 in public method `self_cuda_memory_usage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:534 in public method `self_privateuse1_memory_usage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:542 in public method `self_cpu_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:550 in public method `cuda_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:567 in public method `self_cuda_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:579 in public method `cpu_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:586 in public method `self_privateuse1_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:598 in public method `privateuse1_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:615 in public method `key`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:618 in public method `__repr__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:659 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:687 in public method `add`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:726 in public method `__iadd__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:729 in public method `__repr__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:763 in public class `StringTable`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:764 in public method `__missing__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:773 in public class `MemRecordsAcc`:
D400: First line should end with a period (not 'l')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:775 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:783 in public method `in_interval`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:846 in private function `_build_table`:
D401: First line should be in imperative mood (perhaps 'Print', not 'Prints')
48
```
After :
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:28 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:46 in public method `__str__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:166 in public method `self_cpu_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:266 in public method `supported_export_stacks_metrics`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:273 in public method `export_stacks`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:408 in public method `cpu_time`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:412 in public method `cuda_time`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:416 in public method `privateuse1_time`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:420 in public class `Interval`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:421 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:425 in public method `elapsed_us`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:435 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:488 in public method `append_kernel`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:518 in public method `self_cpu_memory_usage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:526 in public method `self_cuda_memory_usage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:534 in public method `self_privateuse1_memory_usage`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:542 in public method `self_cpu_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:550 in public method `cuda_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:567 in public method `self_cuda_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:579 in public method `cpu_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:586 in public method `self_privateuse1_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:598 in public method `privateuse1_time_total`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:615 in public method `key`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:618 in public method `__repr__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:659 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:687 in public method `add`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:726 in public method `__iadd__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:729 in public method `__repr__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:763 in public class `StringTable`:
D101: Missing docstring in public class
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:764 in public method `__missing__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:775 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/profiler_util.py:783 in public method `in_interval`:
D102: Missing docstring in public method
33
```
7) torch/autograd/grad_mode.py
Before:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:73 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:78 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:82 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:133 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:137 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:182 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:187 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:190 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:193 in public method `clone`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:198 in public class `inference_mode`:
D400: First line should end with a period (not 'e')
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:250 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:257 in public method `__new__`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:262 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:266 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:269 in public method `clone`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:301 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:306 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:309 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:312 in public method `clone`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:354 in private class `_unsafe_preserve_version_counter`:
D400: First line should end with a period (not '!')
21
```
After:
```
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:1 at module level:
D100: Missing docstring in public module
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:73 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:78 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:82 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:133 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:137 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:182 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:187 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:190 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:193 in public method `clone`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:250 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:257 in public method `__new__`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:262 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:266 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:269 in public method `clone`:
D102: Missing docstring in public method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:301 in public method `__init__`:
D107: Missing docstring in __init__
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:306 in public method `__enter__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:309 in public method `__exit__`:
D105: Missing docstring in magic method
/home/ubuntu/Desktop/Docathon/pytorch/torch/autograd/grad_mode.py:312 in public method `clone`:
D102: Missing docstring in public method
19
```
@svekars @kit1980 @subramen
Pull Request resolved: https://github.com/pytorch/pytorch/pull/113266
Approved by: https://github.com/aaronenyeshi, https://github.com/soulitzer, https://github.com/kit1980
This PR implements the feature described in #107036 for `no_grad`, `enable_grad` and `inference_mode`.
Users can still use the above as before but they can also use them without parentheses.
For example:
```python
import torch
a = torch.ones(1, requires_grad=True)
def do_something():
print(2 * a)
with torch.no_grad():
do_something() # tensor([2.])
torch.no_grad()(do_something)() # tensor([2.])
torch.no_grad(do_something)() # tensor([2.])
do_something() # tensor([2.], grad_fn=<MulBackward0>)
```
For `inference_mode`, decorating without parenthesis is equivalent to decorating with the default `mode=True`, similiar to how dataclasses behave (https://docs.python.org/3/library/dataclasses.html#module-contents)
Closes#107036
Pull Request resolved: https://github.com/pytorch/pytorch/pull/107086
Approved by: https://github.com/albanD
Fixes#104985
Implemented `set_multithreading_enabled` C++ function to directly alter state rather than using `MultithreadingEnabled` class, which was automatically resetting the state when the object was destroyed. This behavior more closely aligns with set_grad_enabled which does work as expected. This allows us to change python class `set_multithreading_enabled` to act as both a function and context manager.
I also added a getter: `torch._C.is_multithreading_enabled`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/105291
Approved by: https://github.com/albanD
There are some I can't easily switch due to reasons like:
- Dynamo modelling the guard
- BC concerns (for torch.autograd.set_multithreading_enabled)
Test Plan:
- existing tests
Pull Request resolved: https://github.com/pytorch/pytorch/pull/102642
Approved by: https://github.com/albanD
tldr; this should fix some minor perf regressions that were caused by adding more as_strided() calls in aot autograd.
This PR adds a new context manager, `torch.autograd._set_view_replay_enabled()`.
Context: AOT Autograd has special handling for "outputs that alias graph intermediates". E.g. given this function:
```
def f(x):
y = torch.mul(x, 2)
out = y.view(-1)
return out
```
AOT Autograd will do the following:
```
def fn_to_compile(x):
y = torch.mul(x, 2)
out = y.view(-1)
# return the graph intermediate
return y, out
compiled_fn = compile(fn_to_compile)
def wrapper(x):
y, out = compiled_fn(x)
# regenerate the alias of the graph intermediate
return out._view_func(y)
```
What's annoying is that `out._view_func()` will result in a `.as_strided` call, because `out` is an ordinary runtime tensor. This (likely?) caused a perf regression, because when running the backward, out `as_strided_backward()` is slower than our `view_backward()`.
In this PR, I added some TLS for instructing autograd to do view replay instead of as_strided, even when given a normal tensor. I'm definitely interested in thoughts from autograd folks (cc @albanD @soulitzer). A few points that I want to bring up:
(1) One reason that this API seems generally useful to me is because of the case where you `torch.compile()` a function, and you pass in two inputs that alias each other, and mutate one of the inputs. Autograd is forced to add a bunch of as_strided() calls into the graph when this happens, but this would give users an escape hatch for better compiled perf in this situation
(2) To be fair, AOT Autograd probably won't need this TLS in the long term. There's a better (more complicated) solution, where AOT Autograd manually precomputes the view chain off of graph intermediates during tracing, and re-applies them at runtime. This is kind of complicated though and feels lower priority to implement immediately.
(3) Given all of that I made the API private, but lmk what you all think.
This is a followup of https://github.com/pytorch/pytorch/pull/92255.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92588
Approved by: https://github.com/ezyang, https://github.com/albanD
Copied the type hints from the other context managers.
Not sure how to add type hints for `clone` since it returns the same class. The `Self` type isn't introduced until Python 3.11 and mypy just recently added support for it. Could also use `"inference_mode"` with quotes to avoid using it before it's declared, or `from __future__ import annotations` to allow its use without quotes. Or we could just skip it.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94223
Approved by: https://github.com/albanD
This is a new version of #15648 based on the latest master branch.
Unlike the previous PR where I fixed a lot of the doctests in addition to integrating xdoctest, I'm going to reduce the scope here. I'm simply going to integrate xdoctest, and then I'm going to mark all of the failing tests as "SKIP". This will let xdoctest run on the dashboards, provide some value, and still let the dashboards pass. I'll leave fixing the doctests themselves to another PR.
In my initial commit, I do the bare minimum to get something running with failing dashboards. The few tests that I marked as skip are causing segfaults. Running xdoctest results in 293 failed, 201 passed tests. The next commits will be to disable those tests. (unfortunately I don't have a tool that will insert the `#xdoctest: +SKIP` directive over every failing test, so I'm going to do this mostly manually.)
Fixes https://github.com/pytorch/pytorch/issues/71105
@ezyang
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82797
Approved by: https://github.com/ezyang
Summary:
This fixes the case when `torch.inference_mode` is called with `mode=False` (disabled). When used as a decorator, it ignored the argument and enabled inference mode anyway.
`_DecoratorContextManager` is changed so that a new instance is a copy instead of a new instance with default parameters.
I also added more tests to cover this case.
Current behaviour:
```python
>>> import torch
>>> x = torch.ones(1, 2, 3, requires_grad=True)
>>> torch.inference_mode(mode=False)
... def func(x):
... return x * x
...
>>> out = func(x)
>>> out.requires_grad
False
```
New behaviour (fixed):
```python
>>> import torch
>>> x = torch.ones(1, 2, 3, requires_grad=True)
>>> torch.inference_mode(mode=False)
... def func(x):
... return x * x
...
>>> out = func(x)
>>> out.requires_grad
True
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/68617
Reviewed By: mrshenli
Differential Revision: D32958434
Pulled By: albanD
fbshipit-source-id: 133c69970ef8bffb9fc9ab5142dedcffc4c32945
Summary:
Adds a note explaining the difference between several often conflated mechanisms in the autograd note
Also adds a link to this note from the docs in `grad_mode` and `nn.module`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58513
Reviewed By: gchanan
Differential Revision: D28651129
Pulled By: soulitzer
fbshipit-source-id: af9eb1749b641fc1b632815634eea36bf7979156
Summary:
Fixes https://github.com/pytorch/pytorch/issues/56608
- Adds binding to the `c10::InferenceMode` RAII class in `torch._C._autograd.InferenceMode` through pybind. Also binds the `torch.is_inference_mode` function.
- Adds context manager `torch.inference_mode` to manage an instance of `c10::InferenceMode` (global). Implemented in `torch.autograd.grad_mode.py` to reuse the `_DecoratorContextManager` class.
- Adds some tests based on those linked in the issue + several more for just the context manager
Issues/todos (not necessarily for this PR):
- Improve short inference mode description
- Small example
- Improved testing since there is no direct way of checking TLS/dispatch keys
-
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58045
Reviewed By: agolynski
Differential Revision: D28390595
Pulled By: soulitzer
fbshipit-source-id: ae98fa036c6a2cf7f56e0fd4c352ff804904752c
Summary:
Change from self to self._class_() in _DecoratorManager to ensure a new object is every time a function is called recursively
Fixes https://github.com/pytorch/pytorch/issues/44531
Pull Request resolved: https://github.com/pytorch/pytorch/pull/44633
Reviewed By: agolynski
Differential Revision: D23783601
Pulled By: albanD
fbshipit-source-id: a818664dee7bdb061a40ede27ef99e9546fc80bb
