Joël Tang a6a3f2e06b [MPS] Fixes GELU, LeakyRELU and MISH on non-contiguous tensors (#123049) ...

Fixes GELU, LeakyRELU and MISH activation functions on non-contiguous tensors (for instance, when a transpose operation was applied on the tensors prior to the MPS operator), forward and backward passes.

I also extended tests on the 3 activation functions to check: full-precision and half-precision, contiguous and non-contiguous, and several dims of tensors: scalars, 1D, empty, 2D, > 3D.

I had issues with Mish and GELU activations when asserting the gradients vs. CPU with sum() on some cases, so I reverted to the previous setup by setting a gradient parameter on .backwards().
This PR also fixes an issue with LeakyRELU on empty tensors.

Fixes #98212 huggingface/transformers#22468 huggingface/transformers#19353
Pull Request resolved: https://github.com/pytorch/pytorch/pull/123049
Approved by: https://github.com/kulinseth

2024-04-21 00:12:32 +00:00

..

conda

PyTorch -> C++17 (#98209) (#100557)

2023-05-19 00:49:08 +00:00

src

[MPS] Fixes GELU, LeakyRELU and MISH on non-contiguous tensors (#123049)

2024-04-21 00:12:32 +00:00

tools

Run C++ tests on CI with run_test.py (#99956)

2023-05-09 21:24:12 +00:00

CMakeLists.txt

[Intel GPU] oneDNN GPU GEMM support (#117202)

2024-04-17 23:06:38 +00:00