183 Commits

Author SHA1 Message Date
1f74e082e2 only compare attributes for meta tensors (#72508)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/72508

Todo:

- [x] document this behavior
- [x] add tests

Test Plan: Imported from OSS

Reviewed By: zou3519

Differential Revision: D34262452

Pulled By: ezyang

fbshipit-source-id: bc5c9653d5c3ad5c6efccc9c8e0efc0d28e15104
(cherry picked from commit 233142c88e4cff02825c7e233aba9411a6df3e9f)
2022-02-17 02:33:08 +00:00
d4d0ab71b3 use torch.testing.assert_equal in TestCase.assertEqual (#67796)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/67796

Supersedes #58981.

cc mruberry

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D33542994

Pulled By: mruberry

fbshipit-source-id: 527099f5fdc154fd95ee48cd19f0a85eeec43443
(cherry picked from commit 1a58915e2cfde5c48ad77198a917872a03fd1b72)
2022-01-27 08:33:55 +00:00
928ca95ff0 fix TensorLikePair origination (#70304)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/70304

Without this patch `TensorLikePair` will try to instantiate everything although it should only do so for tensor-likes. This is problematic if it is used before a different pair that would be able to handle the inputs but never gets to do so, because `TensorLikePair` bails out before.

```python
from torch.testing._comparison import assert_equal, TensorLikePair, ObjectPair

assert_equal("a", "a", pair_types=(TensorLikePair, ObjectPair))
```

```
ValueError: Constructing a tensor from <class 'str'> failed with
new(): invalid data type 'str'.
```

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D33542995

Pulled By: mruberry

fbshipit-source-id: 77a5cc0abad44356c3ec64c7ec46e84d166ab2dd
2022-01-12 06:44:00 -08:00
802dd2b725 change sparse COO comparison strategy in assert_close (#68728)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/68728

This removes the ability for `assert_close` to `.coalesce()` the tensors internally. Additionally, we now also check `.sparse_dim()`. Sparse team: please make sure that is the behavior you want for all sparse COO comparisons in the future. #67796 will temporarily keep BC by always coalescing, but in the future `TestCase.assertEqual` will no longer do that.

cc nikitaved pearu cpuhrsch IvanYashchuk

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D33542996

Pulled By: mruberry

fbshipit-source-id: a8d2322c6ee1ca424e3efb14ab21787328cf28fc
2022-01-12 06:43:50 -08:00
8d05174def make meta tensor data access error message for expressive in assert_close (#68802)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/68802

Without this patch, the error message of comparing meta tensors looks like this after #68722 was merged:

```python
>>> t = torch.empty((), device="meta")
>>> assert_close(t, t)
NotImplementedError: Could not run 'aten::abs.out' with arguments from the 'Meta' backend. [...]
[...]
The above exception was the direct cause of the following exception:
[...]
RuntimeError: Comparing

TensorLikePair(
    id=(),
    actual=tensor(..., device='meta', size=()),
    expected=tensor(..., device='meta', size=()),
    rtol=1.3e-06,
    atol=1e-05,
    equal_nan=False,
    check_device=True,
    check_dtype=True,
    check_layout=True,
    check_stride=False,
    check_is_coalesced=True,
)

resulted in the unexpected exception above. If you are a user and see this message during normal operation please file an issue at https://github.com/pytorch/pytorch/issues. If you are a developer and working on the comparison functions, please except the previous error and raise an expressive `ErrorMeta` instead.
```

Thus, we follow our own advice and turn it into an expected exception until #68592 is resolved:

```python
>>> t = torch.empty((), device="meta")
>>> assert_close(t, t)
ValueError: Comparing meta tensors is currently not supported
```

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D33542999

Pulled By: mruberry

fbshipit-source-id: 0fe1ddee15b5decdbd4c5dd84f03804ca7eac95b
2022-01-12 06:43:47 -08:00
b652887ad7 improve documentation of comparison internals (#68977)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/68977

Follow-up to #68722 to address the review comments that were left open before merge.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D33542998

Pulled By: mruberry

fbshipit-source-id: 23c567cd328f83ae4df561ac8ee6c40c259408c9
2022-01-12 06:42:30 -08:00
61ea2fc35e Fix device type / dtype handling for parametrized test names (#65217)
Summary:
This PR absolves `_TestParametrizer`s (e.g. `ops`, `modules`, `parametrize`) of the responsibility of adding device type (e.g. `'cpu'`, `'cuda'`, etc.) / dtype (e.g. 'float32') to generated test names. This fixes repeated instances of the device string being added to generated test names (e.g. `test_batch_norm_training_True_cuda_track_running_stats_True_cuda_affine_True_cuda`).

The responsibility for placing device / dtype suffixes is now handled by `instantiate_device_type_tests()` instead so it is added a single time. It will place `<device>_<dtype>` at the end of the test name unconditionally, maintaining the current naming convention.

As part of this work, I also tightened the semantics through some additional error case handling:
* Composing multiple decorators that each try to handle the same parameter will error out with a nice message. This includes the case to trying to compose `modules` + `ops`, as they each try to handle `dtype`. Similarly, `ops` + `dtypes` is forbidden when both try to handle `dtype`. This required changes in the following test files:
  * `test/test_unary_ufuncs.py`
  * `test/test_foreach.py`
* The `modules` / `ops` decorators will now error out with a nice message if used with `instantiate_parametrized_tests()` instead of `instantiate_device_type_tests()`, since they're not (currently) written to work outside of a device-specific context.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/65217

Reviewed By: mruberry

Differential Revision: D32627303

Pulled By: jbschlosser

fbshipit-source-id: c2957228353ed46a0b7da8fa1a34c67598779312
2021-11-29 19:02:23 -08:00
3315c4b31e add instructions for unhandled exceptions in assert_close (#68722)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/68722

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D32684446

Pulled By: mruberry

fbshipit-source-id: 04fe5730721d24e44692cdc9bb327484356ead3f
2021-11-28 21:35:53 -08:00
f99f5ee088 add support for None in assert_close (#67795)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/67795

Closes #61035.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D32532207

Pulled By: mruberry

fbshipit-source-id: 6a2b4245e0effce4ddea7d89eca63e3b163951a7
2021-11-19 12:38:25 -08:00
0809553cf0 refactor assert_close to be more modular (#67794)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/67794

This change is needed to conveniently use the same comparison mechanism for our internal testsuite (see #67796). The reworked version is on par with the previous version except for the ability to pass a custom message as callable. Before we converted everything to a tensor so it was fairly easy to provide consistent mismatch diagnostics to the callable. Now, with arbitrary `Pair`'s that are used for comparison that is no longer viable.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D32532206

Pulled By: mruberry

fbshipit-source-id: dc847fba6a795c1766e01bc3e88b680a68287b1e
2021-11-19 12:37:16 -08:00
f3e2fefe09 Actually enable PYTORCH_RETRY_TEST_CASES for linux tests (#68486)
Summary:
After realizing that CUDA mem leaks were not rerun, I realized I forgot to pass the env var as a Docker variable.

What a noob mistake.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/68486

Reviewed By: seemethere

Differential Revision: D32501718

Pulled By: janeyx99

fbshipit-source-id: 9918d626e90bea1562a3094c6eb12cb7d86dbf6a
2021-11-17 11:50:48 -08:00
885a8e53ba replace onlyOnCPUAndCUDA with onlyNativeDeviceTypes (#65201)
Summary:
Reference https://github.com/pytorch/pytorch/issues/53849

Replace `onlyOnCPUandCUDA` with `onlyNativeDeviceTypes` which includes `cpu, cuda and meta`.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/65201

Reviewed By: mrshenli

Differential Revision: D31299718

Pulled By: mruberry

fbshipit-source-id: 2d8356450c035d6a314209ab51b2c237583920fd
2021-11-01 09:22:34 -07:00
a72a6365c9 disallow requires_grad=True in make_tensor for integral inputs (#67149)
Summary:
per title

Pull Request resolved: https://github.com/pytorch/pytorch/pull/67149

Reviewed By: albanD

Differential Revision: D31928613

Pulled By: ngimel

fbshipit-source-id: 4491954c4fcd4a4e3121155d4451cc7370c27a0b
2021-10-26 16:19:28 -07:00
8a65047acc [skip ci] Set test owners for everything considered with module: tests (#66865)
Summary:
Action following https://github.com/pytorch/pytorch/issues/66232

cc mruberry

Pull Request resolved: https://github.com/pytorch/pytorch/pull/66865

Reviewed By: anjali411

Differential Revision: D31771147

Pulled By: janeyx99

fbshipit-source-id: 8bebe5ac2098364ef1ee93b590abb5f4455b0f89
2021-10-20 09:37:03 -07:00
f9c2dc860d make layout check optional in torch.testing.assert_close() (#65419)
Summary:
In case the inputs have a different layout, `assert_close(..., check_layout=False)` converts them to strided before comparison. This is helpful if you just want to compare the values of sparse COO / CSR tensor against a strided reference.

This keeps BC, since the default `check_layout=True` was the old, hard-coded behavior.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/65419

Reviewed By: H-Huang

Differential Revision: D31133629

Pulled By: mruberry

fbshipit-source-id: ca8918af81fb0e0ba263104836a4c2eeacdfc7e6
2021-09-28 23:23:41 -07:00
b7ec7d760d Generic test parametrization functionality (#60753)
Summary:
This PR plays around with implementation & usage of a `parametrize` decorator for test parametrization similar to `pytest.mark.parametrize`, based on previous work introducing a `_TestParametrizer` class. It works with the internal `DeviceTest` hierarchy & composes with `dtype`, `skip*`, and other decorators. Basic usage is demonstrated in `test/test_blah.py`:

```python
import unittest
from itertools import product
from torch.testing._internal.common_device_type import (
    instantiate_device_type_tests, deviceCountAtLeast, ops)
from torch.testing._internal.common_methods_invocations import op_db
from torch.testing._internal.common_utils import (
    TestCase, run_tests, parametrize, instantiate_parametrized_tests, subtest)

class TestBlah(TestCase):
    parametrize("x", range(5))
    def test_default_names(self, x):
        print('Passed in:', x)

    # Use default names but add an expected failure.
    parametrize("x", [subtest(0, decorators=[unittest.expectedFailure]),
                       *range(1, 5)])
    def test_default_names_expected_failure(self, x):
        if x == 0:
            raise RuntimeError('Boom')
        print('Passed in:', x)

    parametrize("bias", [False, True], name_fn=lambda b: 'bias' if b else 'no_bias')
    def test_custom_names(self, bias):
        print('Passed in:', bias)

    parametrize("bias", [subtest(True, name='bias'),
                          subtest(False, name='no_bias')])
    def test_custom_names_alternate(self, bias):
        print('Passed in:', bias)

    parametrize("x,y", [(1, 2), (1, 3), (1, 4)])
    def test_two_things_default_names(self, x, y):
        print('Passed in:', x, y)

    parametrize("x", [1, 2, 3])
    parametrize("y", [4, 5, 6])
    def test_two_things_composition(self, x, y):
        print('Passed in:', x, y)

    parametrize("x", [subtest(0, decorators=[unittest.expectedFailure]),
                       *range(1, 3)])
    parametrize("y", [4, 5, subtest(6, decorators=[unittest.expectedFailure])])
    def test_two_things_composition_expected_failure(self, x, y):
        if x == 0 or y == 6:
            raise RuntimeError('Boom')
        print('Passed in:', x, y)

    parametrize("x", [1, 2])
    parametrize("y", [3, 4])
    parametrize("z", [5, 6])
    def test_three_things_composition(self, x, y, z):
        print('Passed in:', x, y, z)

    parametrize("x", [1, 2], name_fn=str)
    parametrize("y", [3, 4], name_fn=str)
    parametrize("z", [5, 6], name_fn=str)
    def test_three_things_composition_custom_names(self, x, y, z):
        print('Passed in:', x, y, z)

    parametrize("x,y", product(range(2), range(3)))
    def test_two_things_product(self, x, y):
        print('Passed in:', x, y)

    parametrize("x,y", [subtest((1, 2), name='double'),
                         subtest((1, 3), name='triple'),
                         subtest((1, 4), name='quadruple')])
    def test_two_things_custom_names(self, x, y):
        print('Passed in:', x, y)

    parametrize("x,y", [(1, 2), (1, 3), (1, 4)], name_fn=lambda x, y: '{}_{}'.format(x, y))
    def test_two_things_custom_names_alternate(self, x, y):
        print('Passed in:', x, y)

class TestDeviceBlah(TestCase):
    parametrize("x", range(10))
    def test_default_names(self, device, x):
        print('Passed in:', device, x)

    parametrize("x,y", [(1, 2), (3, 4), (5, 6)])
    def test_two_things(self, device, x, y):
        print('Passed in:', device, x, y)

    deviceCountAtLeast(1)
    def test_multiple_devices(self, devices):
        print('Passed in:', devices)

    ops(op_db)
    parametrize("flag", [False, True], lambda f: 'flag_enabled' if f else 'flag_disabled')
    def test_op_parametrized(self, device, dtype, op, flag):
        print('Passed in:', device, dtype, op, flag)

instantiate_parametrized_tests(TestBlah)
instantiate_device_type_tests(TestDeviceBlah, globals())

if __name__ == '__main__':
    run_tests()
```

Generated tests:
```
TestBlah.test_custom_names_alternate_bias
TestBlah.test_custom_names_alternate_no_bias
TestBlah.test_custom_names_bias
TestBlah.test_custom_names_no_bias
TestBlah.test_default_names_expected_failure_x_0
TestBlah.test_default_names_expected_failure_x_1
TestBlah.test_default_names_expected_failure_x_2
TestBlah.test_default_names_expected_failure_x_3
TestBlah.test_default_names_expected_failure_x_4
TestBlah.test_default_names_x_0
TestBlah.test_default_names_x_1
TestBlah.test_default_names_x_2
TestBlah.test_default_names_x_3
TestBlah.test_default_names_x_4
TestBlah.test_three_things_composition_custom_names_1_3_5
TestBlah.test_three_things_composition_custom_names_1_3_6
TestBlah.test_three_things_composition_custom_names_1_4_5
TestBlah.test_three_things_composition_custom_names_1_4_6
TestBlah.test_three_things_composition_custom_names_2_3_5
TestBlah.test_three_things_composition_custom_names_2_3_6
TestBlah.test_three_things_composition_custom_names_2_4_5
TestBlah.test_three_things_composition_custom_names_2_4_6
TestBlah.test_three_things_composition_x_1_y_3_z_5
TestBlah.test_three_things_composition_x_1_y_3_z_6
TestBlah.test_three_things_composition_x_1_y_4_z_5
TestBlah.test_three_things_composition_x_1_y_4_z_6
TestBlah.test_three_things_composition_x_2_y_3_z_5
TestBlah.test_three_things_composition_x_2_y_3_z_6
TestBlah.test_three_things_composition_x_2_y_4_z_5
TestBlah.test_three_things_composition_x_2_y_4_z_6
TestBlah.test_two_things_composition_expected_failure_x_0_y_4
TestBlah.test_two_things_composition_expected_failure_x_0_y_5
TestBlah.test_two_things_composition_expected_failure_x_0_y_6
TestBlah.test_two_things_composition_expected_failure_x_1_y_4
TestBlah.test_two_things_composition_expected_failure_x_1_y_5
TestBlah.test_two_things_composition_expected_failure_x_1_y_6
TestBlah.test_two_things_composition_expected_failure_x_2_y_4
TestBlah.test_two_things_composition_expected_failure_x_2_y_5
TestBlah.test_two_things_composition_expected_failure_x_2_y_6
TestBlah.test_two_things_composition_x_1_y_4
TestBlah.test_two_things_composition_x_1_y_5
TestBlah.test_two_things_composition_x_1_y_6
TestBlah.test_two_things_composition_x_2_y_4
TestBlah.test_two_things_composition_x_2_y_5
TestBlah.test_two_things_composition_x_2_y_6
TestBlah.test_two_things_composition_x_3_y_4
TestBlah.test_two_things_composition_x_3_y_5
TestBlah.test_two_things_composition_x_3_y_6
TestBlah.test_two_things_custom_names_alternate_1_2
TestBlah.test_two_things_custom_names_alternate_1_3
TestBlah.test_two_things_custom_names_alternate_1_4
TestBlah.test_two_things_custom_names_double
TestBlah.test_two_things_custom_names_quadruple
TestBlah.test_two_things_custom_names_triple
TestBlah.test_two_things_default_names_x_1_y_2
TestBlah.test_two_things_default_names_x_1_y_3
TestBlah.test_two_things_default_names_x_1_y_4
TestBlah.test_two_things_product_x_0_y_0
TestBlah.test_two_things_product_x_0_y_1
TestBlah.test_two_things_product_x_0_y_2
TestBlah.test_two_things_product_x_1_y_0
TestBlah.test_two_things_product_x_1_y_1
TestBlah.test_two_things_product_x_1_y_2
TestDeviceBlahCPU.test_default_names_x_0_cpu
TestDeviceBlahCPU.test_default_names_x_1_cpu
TestDeviceBlahCPU.test_default_names_x_2_cpu
TestDeviceBlahCPU.test_default_names_x_3_cpu
TestDeviceBlahCPU.test_default_names_x_4_cpu
TestDeviceBlahCPU.test_default_names_x_5_cpu
TestDeviceBlahCPU.test_default_names_x_6_cpu
TestDeviceBlahCPU.test_default_names_x_7_cpu
TestDeviceBlahCPU.test_default_names_x_8_cpu
TestDeviceBlahCPU.test_default_names_x_9_cpu
TestDeviceBlahCPU.test_multiple_devices_cpu
TestDeviceBlahCPU.test_op_parametrized_<opname>_<variant>_cpu_uint8_flag_enabled_cpu
TestDeviceBlahCPU.test_two_things_x_1_y_2_cpu
TestDeviceBlahCPU.test_two_things_x_3_y_4_cpu
TestDeviceBlahCPU.test_two_things_x_5_y_6_cpu
TestDeviceBlahMETA.test_default_names_x_0_meta
TestDeviceBlahMETA.test_default_names_x_1_meta
TestDeviceBlahMETA.test_default_names_x_2_meta
TestDeviceBlahMETA.test_default_names_x_3_meta
TestDeviceBlahMETA.test_default_names_x_4_meta
TestDeviceBlahMETA.test_default_names_x_5_meta
TestDeviceBlahMETA.test_default_names_x_6_meta
TestDeviceBlahMETA.test_default_names_x_7_meta
TestDeviceBlahMETA.test_default_names_x_8_meta
TestDeviceBlahMETA.test_default_names_x_9_meta
TestDeviceBlahMETA.test_multiple_devices_meta
TestDeviceBlahMETA.test_op_parametrized_<opname>_<variant>_meta_uint8_flag_enabled_meta
TestDeviceBlahMETA.test_two_things_x_1_y_2_meta
TestDeviceBlahMETA.test_two_things_x_3_y_4_meta
TestDeviceBlahMETA.test_two_things_x_5_y_6_meta
```

Caveats:
* `parametrize` decorators cannot be "stacked" yet; each one overwrites the previous. This will change to either:
  * Allow stacking of multiple decorators
  * Error out with a nice error message if multiple decorators are specified

The PR introduces `instantiate_parametrized_tests()` in addition to `instantiate_device_type_tests()`. The former should be used for non-device-specific tests, and the latter should be used for device-specific tests, as usual. Both of these support the `parametrize` decorator. Only the latter supports the `ops` decorator (no change here- this was already the case).

Pull Request resolved: https://github.com/pytorch/pytorch/pull/60753

Reviewed By: saketh-are

Differential Revision: D30606615

Pulled By: jbschlosser

fbshipit-source-id: a34f36d643f68a6e221f419d9bb3e1ae1d84dd65
2021-09-14 19:52:59 -07:00
26b7ff5aea deprecate dtype getters from torch.testing namespace (#63554)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63554

Following https://github.com/pytorch/pytorch/pull/61840#issuecomment-884087809, this deprecates all the dtype getters publicly exposed in the `torch.testing` namespace. The reason for this twofold:

1. If someone is not familiar with the C++ dispatch macros PyTorch uses, the names are misleading. For example `torch.testing.floating_types()` will only give you `float32` and `float64` skipping `float16` and `bfloat16`.
2. The dtype getters provide very minimal functionality that can be easily emulated by downstream libraries.

We thought about [providing an replacement](https://gist.github.com/pmeier/3dfd2e105842ad0de4505068a1a0270a), but ultimately decided against it. The major problem is BC: by keeping it, either the namespace is getting messy again after a new dtype is added or we need to somehow version the return values of the getters.

Test Plan: Imported from OSS

Reviewed By: H-Huang

Differential Revision: D30662206

Pulled By: mruberry

fbshipit-source-id: a2bdb10ab02ae665df1b5b76e8afa9af043bbf56
2021-09-07 08:58:51 -07:00
eafe33c995 remove componentwise comparison of complex values in torch.testing.assert_close (#63841)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63841

Closes #61906.

cc ezyang gchanan

Test Plan: Imported from OSS

Reviewed By: ezyang

Differential Revision: D30633526

Pulled By: mruberry

fbshipit-source-id: ddb5d61838cd1e12d19d0093799e827344382cdc
2021-08-30 12:38:44 -07:00
401bbb2aa0 remove componentwise comparison of complex values in TestCase.assertEqual (#63572)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63572

Addresses #61906. Issue will be fixed later in the stack when `torch.testing.assert_close` got the same treatment.

cc ezyang gchanan

Test Plan: Imported from OSS

Reviewed By: ezyang

Differential Revision: D30633527

Pulled By: mruberry

fbshipit-source-id: c2002a4998a7a75cb2ab83f87190bde43a9d4f7c
2021-08-30 12:36:45 -07:00
d37636901e [Doc] make_tensor to torch.testing module (#63925)
Summary:
This PR aims to add `make_tensor` to the `torch.testing` module in PyTorch docs.

TODOs:

* [x] Add examples

cc: pmeier mruberry brianjo

Pull Request resolved: https://github.com/pytorch/pytorch/pull/63925

Reviewed By: ngimel

Differential Revision: D30633487

Pulled By: mruberry

fbshipit-source-id: 8e5a1f880c6ece5925b4039fee8122bd739538af
2021-08-30 12:25:40 -07:00
b1154cc774 enable equal_nan for complex values in isclose (#63571)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/63571

Test Plan: Imported from OSS

Reviewed By: malfet, ngimel

Differential Revision: D30560127

Pulled By: mruberry

fbshipit-source-id: 8958121ca24e7c139d869607903aebbe87bc0740
2021-08-25 22:05:49 -07:00
1022443168 Revert D30279364: [codemod][lint][fbcode/c*] Enable BLACK by default
Test Plan: revert-hammer

Differential Revision:
D30279364 (b004307252)

Original commit changeset: c1ed77dfe43a

fbshipit-source-id: eab50857675c51e0088391af06ec0ecb14e2347e
2021-08-12 11:45:01 -07:00
b004307252 [codemod][lint][fbcode/c*] Enable BLACK by default
Test Plan: manual inspection & sandcastle

Reviewed By: zertosh

Differential Revision: D30279364

fbshipit-source-id: c1ed77dfe43a3bde358f92737cd5535ae5d13c9a
2021-08-12 10:58:35 -07:00
f16c73b9f3 Improve error messages of torch.testing.assert_close for sparse inputs (#61583)
Summary:
This utilizes the feature introduced in https://github.com/pytorch/pytorch/issues/60091 to modify the header of the error message.

Before:

```python
AssertionError: Tensor-likes are not equal!

Mismatched elements: 1 / 2 (50.0%)
Greatest absolute difference: 1 at index 1
Greatest relative difference: 0.3333333432674408 at index 1

The failure occurred for the values.
```

After:

```python
AssertionError: Sparse COO values of tensor-likes are not equal!

Mismatched elements: 1 / 2 (50.0%)
Greatest absolute difference: 1 at index 1
Greatest relative difference: 0.3333333432674408 at index 1
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/61583

Reviewed By: malfet

Differential Revision: D30014797

Pulled By: cpuhrsch

fbshipit-source-id: 66e30645e94de5c8c96510822082ff9aabef5329
2021-07-30 11:23:26 -07:00
3d6aa3a2f6 Enable torch.isclose to suppport bool tensors (#61271)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/60533

Pull Request resolved: https://github.com/pytorch/pytorch/pull/61271

Reviewed By: zhxchen17

Differential Revision: D29737618

Pulled By: SplitInfinity

fbshipit-source-id: 45314bc7e0b9a28c10700455b1e6267c0db3eefc
2021-07-21 18:50:14 -07:00
8ad584823f add shortcircuit in isclose for zero tolerances (#61529)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/61412.

Large integers gave false positives, because the comparison always takes place in floating point dtypes. This happens, because their integer precision is lower than the range of an integer dtype with the same number of bits.

For non-extremal values, `isclose` is defined by [this equation]:

```python
abs(a - b) <= atol + rtol * abs(b)
```

For `rtol == 0 and atol==0`, this is equivalent to `a == b`. This PR goes for the low hanging fruit and adds a shortcut for this case that falls back to an actual equality check.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/61529

Reviewed By: gchanan

Differential Revision: D29707534

Pulled By: mruberry

fbshipit-source-id: 71b8c4901e9cd4f366442437e52032b0d3002b4a
2021-07-16 12:48:16 -07:00
736bb26746 use rand over empty in flaky test (#61710)
Summary:
Fixes https://github.com/pytorch/pytorch/pull/61694#issuecomment-880641635. cc krshrimali.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/61710

Reviewed By: anjali411

Differential Revision: D29719660

Pulled By: mruberry

fbshipit-source-id: 589574a039ad431acc7d095d452f0b3e52260208
2021-07-16 10:50:05 -07:00
682ebc1dd1 remove UsageError in favor of ValueError (#61031)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61031

See https://github.com/pytorch/pytorch/pull/58916#issuecomment-868519515.

Test Plan: Imported from OSS

Reviewed By: iramazanli

Differential Revision: D29626810

Pulled By: mruberry

fbshipit-source-id: 25ddf26815f9ef82b8234d7dac811a6a13a53c54
2021-07-09 11:28:33 -07:00
09c90b3589 relax type equality constraint (#60638)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60638

Initial proposal in https://github.com/pytorch/pytorch/pull/58981#issuecomment-866690334. Opposed to the proposal, this PR only allows relaxing the type equality constraint to a common superclass constraint, for example `torch.Tensor` vs `torch.nn.Parameter`. Inputs that do not share a common superclass will still fail.

Test Plan: Imported from OSS

Reviewed By: soulitzer

Differential Revision: D29626811

Pulled By: mruberry

fbshipit-source-id: 1916c3b710d38889de7ce57eb0770c76cbbb8166
2021-07-09 11:27:32 -07:00
29ecb9f90b Don't check stride by default (#60637)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60637

We now have ~three out of three~  four out of four datapoints that `check_stride` will be `partial`'ed to `False`:

- `torch` test suite: https://github.com/pytorch/pytorch/pull/58981#discussion_r639514081
- `torchvision` test suite: https://github.com/pytorch/pytorch/issues/56544#issuecomment-845352605
- `kornia`: 9041c42b41/test/utils.py (L25)
- `torch.fft`: https://github.com/pytorch/pytorch/pull/60304#pullrequestreview-687882323

Given that the strides in most cases are in implementation detail, IMO we should change the default to `False`. In cases were matching strides is a requirement for closeness / equality it can always set to `True` manually.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D29556355

Pulled By: mruberry

fbshipit-source-id: 0029a44280d8f4369fbdb537dce3202eeee4b1d9
2021-07-07 09:55:36 -07:00
e2a3f4b560 Use maximum of tolerances in case of mismatching dtypes (#60636)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60636

See https://github.com/pytorch/pytorch/pull/58981#issuecomment-866654600.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D29556352

Pulled By: mruberry

fbshipit-source-id: 36e97e0f338df5d17a94af078f172c668ef51ecb
2021-07-07 09:55:34 -07:00
5f18ba7075 upcast to most precise dtype within their category before the comparison (#60536)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60536

`torch.isclose` does not do this bool tensors, which results in a test failure since subtraction (`abs(actual - expected)`) is not supported for them (see #58981). Since the `dtype` is already checked at this point, we can safely move the upcasting before `torch.isclose` is invoked.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D29556356

Pulled By: mruberry

fbshipit-source-id: 4c65fad4f06cf402d6aab9dde5b127235766d5e0
2021-07-07 09:55:32 -07:00
5ac87cde30 tests for diagnostics in callable msg in torch.testing.assert_close (#60254)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60254

Before we only tested that the correct error message is returned if `msg` is passed as callable. This adds tests that make sure that

- the inputs passed to the callable are the same inputs passed to `torch.assert_close` and
- the `diagnostics` namespace has the same attributes and types as documented.

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D29556354

Pulled By: mruberry

fbshipit-source-id: 9793c6d86fda842b6329381fc03b945eee878464
2021-07-07 09:55:30 -07:00
76d9e680d7 update docstring examples of torch.testing.assert_close (#60163)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60163

Changes to the default error message in case of mismatching values need to be reflected in the examples given in the docstring. Normally this should be enforced by a [`doctest`](https://docs.python.org/3/library/doctest.html). mruberry do you know why we don't have such a check?

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D29556353

Pulled By: mruberry

fbshipit-source-id: 8dbc3f566f429618811b542a059d9abde9a6530b
2021-07-07 09:55:29 -07:00
9979289037 Improve error messages of torch.testing.assert_close in case of mismatching values (#60091)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60091

Closes #58383. (1) and (2) are implemented. (3) was rejected. No consensus was reached on (4) and (5).

Improvements:

- Instead of calling everything "Tensors" we now use "Scalars" and "Tensor-likes" depending on the shape. Plus, we now internally have the option to adapt this identifier for example to report "Imaginary components of complex tensor-likes", which is even more expressive.
- The reported conditions "not close" and "not equal" are now determined based on `rtol` and `atol`.
- The number of mismatched elements and the offending indices are only reported in case the inputs are not scalar
- The allowed `rtol` and `atol` is only reported if `> 0`

**Example 1**

```python
torch.testing.assert_close(1, 3, rtol=0, atol=1)
```

Before:

```
AssertionError: Tensors are not close!

Mismatched elements: 1 / 1 (100.0%)
Greatest absolute difference: 2 at 0 (up to 1 allowed)
Greatest relative difference: 0.6666666865348816 at 0 (up to 0 allowed)
```

After:

```
AssertionError: Scalars are not close!

Absolute difference: 2 (up to 1 allowed)
Relative difference: 0.6666666865348816
```

**Example 2**

```python
torch.manual_seed(0)
t = torch.rand((2, 2), dtype=torch.complex64)
torch.testing.assert_close(t, t + complex(0, 1))
```

Before:

```
AssertionError: Tensors are not close!

Mismatched elements: 4 / 4 (100.0%)
Greatest absolute difference: 1.0000000596046448 at (0, 0) (up to 1e-05 allowed)
Greatest relative difference: 0.8833684352411922 at (0, 1) (up to 1.3e-06 allowed)

The failure occurred for the imaginary part.
```

After:

```
AssertionError: Imaginary components of tensor-likes are not close!

Mismatched elements: 4 / 4 (100.0%)
Greatest absolute difference: 1.0000000596046448 at index (0, 0) (up to 1e-05 allowed)
Greatest relative difference: 0.8833684352411922 at index (0, 1) (up to 1.3e-06 allowed)
```

Test Plan: Imported from OSS

Reviewed By: ngimel

Differential Revision: D29556357

Pulled By: mruberry

fbshipit-source-id: 559d4a19ad4fc069b2b4f8cb5fc2f6058621e33d
2021-07-07 09:54:09 -07:00
db1dd9e7e0 add support for quantized tensors in torch.testing.assert_close (#58926)
Summary:
This adds support for quantized tensors the same way torch.testing._internal.common_utils.TestCase.assertEqual does:

bf269fdc98/torch/testing/_internal/common_utils.py (L1314-L1341)

- `.qscheme()` is checked for equality
- `.q_scale` and `q_zero_point` are checked for equality (see comment below) for `.qscheme() == torch.per_tensor_affine`
- `.q_per_channel_scales`, `q_per_channel_zero_points`, and `q_per_channel_axis` are checked for equality (see comment below) for `.qscheme() == torch.per_tensor_affine`
- values are checked with the default checks after a `.int_repr().to(torch.int32)` call

Pull Request resolved: https://github.com/pytorch/pytorch/pull/58926

Reviewed By: jerryzh168

Differential Revision: D29483532

Pulled By: mruberry

fbshipit-source-id: 003fde7e21cf844778a879c3de0a7c84d13877bd
2021-06-30 21:43:02 -07:00
44b3dc4eac resolve conjugate bit in torch.testing.assert_close (#60522)
Summary:
We need to resolve the conjugate bit for complex tensors, because otherwise we may not be able to access the imaginary component:

```python
>>> torch.tensor(complex(1, 1)).conj().imag
RuntimeError: view_as_real doesn't work on unresolved conjugated tensors.  To resolve the conjugate tensor so you can view it as real, use self.resolve_conj(); however, be warned that the resulting tensor will NOT alias the original.
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/60522

Reviewed By: ngimel

Differential Revision: D29353095

Pulled By: mruberry

fbshipit-source-id: c36eaf883dd55041166f692f7b1d35cd2a34acfb
2021-06-30 01:31:30 -07:00
7e619b9588 First step to rearrange files in tools folder (#60473)
Summary:
Changes including:
- introduced `linter/`, `testing/`, `stats/` folders in `tools/`
- move appropriate scripts into these folders
- change grepped references in the pytorch/pytorch repo

Next step
- introduce `build/` folder for build scripts

Pull Request resolved: https://github.com/pytorch/pytorch/pull/60473

Test Plan:
- CI (this is important b/c pytorch/test-infra also rely on some script reference.
- tools/tests/

Reviewed By: albanD

Differential Revision: D29352716

Pulled By: walterddr

fbshipit-source-id: bad40b5ce130b35dfd9e59b8af34f9025f3285fd
2021-06-24 10:13:58 -07:00
6ea22672c4 add support for sparse tensors in torch.testing.assert_close (#58844)
Summary:
This adds support for sparse tensors the same way `torch.testing._internal.common_utils.TestCase.assertEqual` does:

5c7dace309/torch/testing/_internal/common_utils.py (L1287-L1313)

- Tensors are coalesced before comparison.
- Indices and values are compared individually.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/58844

Reviewed By: zou3519

Differential Revision: D29160250

Pulled By: mruberry

fbshipit-source-id: b0955656c2c7ff3db37a1367427ca54ca14f2e87
2021-06-23 21:59:01 -07:00
7d39608a29 split TestAsserts by functionality (#58919)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58919

Instead of having one large TestAsserts test case, we split of tests for
self-contained functionality like container or complex checking into
separate test cases. That makes it a lot easier to keep an overview over
what is tested.

Test Plan: Imported from OSS

Reviewed By: anjali411

Differential Revision: D29259407

Pulled By: mruberry

fbshipit-source-id: 9769cb6d56c1a3790280542db398cb247986b09a
2021-06-21 20:44:23 -07:00
14b0191d1f make assert_equal an example how to partial torch.testing.assert_close (#58918)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58918

~Instead of a distinct `torch.testing.assert_close` and `torch.testing.assert_equal`, this makes `torch.testing.assert_equal` a special case of `torch.testing.assert_close` for `rtol=atol=0`. In this case the closeness definition `abs(actual - expected) <= atol + rtol * abs(expected)` boils down to `abs(actual - expected) <= 0`. Since `abs(x)` can never be `<0`, this is equivalent to `abs(a - b) == 0` and this again boils down to `a == b`.~

Following https://github.com/pytorch/pytorch/pull/58918#issuecomment-860642057 and some offline discussions, we opted to use `assert_equal` as an example how to `partial` it.

This makes maintaing the module a lot easier, because we don't need to keep two functions in sync.

Test Plan: Imported from OSS

Reviewed By: anjali411

Differential Revision: D29259404

Pulled By: mruberry

fbshipit-source-id: fa1a1fa93672a7ed1c5f0e4beb0dcd45b5c14fce
2021-06-21 20:44:21 -07:00
64aec8d2ca [testing] OpInfoHelper tool (#58698)
Summary:
Fixes: https://github.com/pytorch/pytorch/issues/57577

Usage:
Add OpInfo entry to `common_methods_invocations` with `dtypes=_DYNAMIC_DYTPES`
Eg.
```
OpInfo('atan2',
        dtypes=_DYNAMIC_DTYPES,
        sample_inputs_func=sample_inputs_atan2,)
```

Run the helper with `python -m torch.testing._internal.opinfo_helper`

Output
```
OpInfo(atan2,
       # hint: all_types + (torch.bool,),
       dtypes=[torch.float32, torch.float64, torch.uint8, torch.int8, torch.int16, torch.int32, torch.int64, torch.bool],
       # hint: all_types + (torch.bool, torch.bfloat16, torch.float16),
       dtypesIfCUDA=[torch.float32, torch.float64, torch.uint8, torch.int8, torch.int16, torch.int32, torch.int64, torch.bool, torch.bfloat16, torch.float16],
       sample_inputs_func=sample_inputs_atan2)
```

Output without CUDA (run with `$ CUDA_VISIBLE_DEVICES=-1 python -m torch.testing._internal.opinfo_helper`)
```
UserWarning: WARNING: CUDA is not available, information pertaining to CUDA could be wrong
  warnings.warn("WARNING: CUDA is not available, information pertaining to CUDA could be wrong")
OpInfo(atan2,
       # hint: all_types + (torch.bool,),
       dtypes=[torch.float32, torch.float64, torch.uint8, torch.int8, torch.int16, torch.int32, torch.int64, torch.bool],
       sample_inputs_func=sample_inputs_atan2)
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/58698

Reviewed By: H-Huang

Differential Revision: D29160668

Pulled By: mruberry

fbshipit-source-id: 707370a83b451b02ad2fe539775c8c50ecf90be8
2021-06-16 17:17:03 -07:00
2e26976ad3 Disallow versionless Python shebangs (#58275)
Summary:
Some machines don't have a versionless `python` on their PATH, which breaks these existing shebangs.

I'm assuming that all the existing versionless `python` shebangs are meant to be `python3` and not `python2`; please let me know if my assumption was incorrect for any of these.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/58275

Test Plan: CI.

Reviewed By: zhouzhuojie

Differential Revision: D28428143

Pulled By: samestep

fbshipit-source-id: 6562be3d12924db72a92a0207b060ef740f61ebf
2021-05-14 08:26:02 -07:00
9148f19e85 enable support for nested containers in torch.testing.assert(equal|close) (#57270)
Summary:
In contrast to the initial opinion in https://github.com/pytorch/pytorch/issues/55385, there are legitimate use cases for nested containers. One such example is the [output of `LSTM`'s](https://pytorch.org/docs/stable/generated/torch.nn.LSTM):

```python
output: Tuple[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]] = torch.nn.LSTM()(input)
assert_close(output, expected)
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/57270

Reviewed By: albanD

Differential Revision: D28249303

Pulled By: mruberry

fbshipit-source-id: 75caa4414cc184ff0ce4cfc0dd5aafddfad42bcf
2021-05-12 15:37:42 -07:00
8824f49e68 Split test_testing.py::TestAsserts for multiple devices (#56365)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56365

Follow-up to https://github.com/pytorch/pytorch/pull/54784#discussion_r614156172. Instead of having one large testcase where most methods are decorated with `onlyCPU`, this factors out all tests that actually need another device into a separate test case.

Test Plan: Imported from OSS

Reviewed By: walterddr, albanD

Differential Revision: D28247529

Pulled By: mruberry

fbshipit-source-id: 946e7694b70e736941565f29b5dd459ed7fbca4e
2021-05-11 19:47:56 -07:00
71ca3e99af Only use actually mismatched elements for reporting in torch.testing (#57923)
Summary:
Redo of https://github.com/pytorch/pytorch/issues/57135 out of stack

 ---

Currently all values are used for the reported absolute and relative differences. This usually works fine, but breaks down for the extremals:

```python
torch.testing.assert_close(torch.tensor([1.0, 0.0]), torch.tensor([2.0, 0.0]))
```

```
[...]
Greatest absolute difference: 1.0 at 0 (up to 1e-05 allowed)
Greatest relative difference: nan at 1 (up to 1.3e-06 allowed)
```

Although the second element is matching it is listed as offender for the greatest relative difference. The `NaN` stems from the `0 / 0` division.

To overcome this, we should only use the values that were considered a mismatch for the reported stats.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/57923

Reviewed By: ngimel

Differential Revision: D28317316

Pulled By: mruberry

fbshipit-source-id: 4c604493bbe13b37f41225ea9af9e839a7304161
2021-05-10 20:58:47 -07:00
0dd0151c64 add torch.testing to docs (#57247)
Summary:
Redo of https://github.com/pytorch/pytorch/issues/56373 out of stack.

 ---

To reviewers: **please be nitpicky**. I've read this so often that I probably missed some typos and inconsistencies.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/57247

Reviewed By: albanD

Differential Revision: D28247402

Pulled By: mruberry

fbshipit-source-id: 71142678ee5c82cc8c0ecc1dad6a0b2b9236d3e6
2021-05-07 09:16:39 -07:00
126ea1ccad relax type equality constraint for scalars (#57532)
Summary:
Currently we require type equality for `torch.testing.assert_(equal|close)`:

3db45bcb91/torch/testing/_asserts.py (L509-L513)

That means `assert_equal(1, 1.0)` will correctly fail. Although the type of a scalar is similiar to a dtype of a tensor, `assert_equal(1, 1.0, check_dtype=False)` will also fail while `assert_equal(torch.as_tensor(1), torch.as_tensor(1.0), check_dtype=False)` will pass.

To make the interface more consistent, this PR relaxes the type equality constraint, by disabling it in case both inputs are scalars.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/57532

Reviewed By: ngimel

Differential Revision: D28242428

Pulled By: mruberry

fbshipit-source-id: b643c77f48b64fc2c8a43925120d2b634ec336b5
2021-05-05 22:42:51 -07:00
5c68072ee8 add support for complex input to torch.testing.assert_(equal|close) (#57162)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/57162

Reviewed By: ngimel

Differential Revision: D28141902

Pulled By: mruberry

fbshipit-source-id: fd35e73e10167e3e44da4daf6582183bc4a0de7f
2021-05-02 16:13:12 -07:00
805129f957 enable support for custom error messages in torch.testing (#55890)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/55890

Proof-of-concept for https://github.com/pytorch/pytorch/pull/55145#issuecomment-817297273

With this the user is able to pass a custom error message to `assert_(equal|close)` which will be used in case the values mismatch. Optionally, a callable can be passed which will be called with mismatch diagnostics and should return an error message:

```python
def make_msg(a, b, info):
    return (
        f"Argh, we found {info.total_mismatches} mismatches! "
        f"That is {info.mismatch_ratio:.1%}!"
    )

torch.testing.assert_equal(torch.tensor(1), torch.tensor(2), msg=make_msg)
```

If you imagine `a` and `b` as the outputs of binary ufuncs, the error message could look like this:

```python
def make_msg(input, torch_output, numpy_output, info):
    return (
        f"For input {input} torch.binary_op() and np.binary_op() do not match: "
        f"{torch_output} != {numpy_output}"
    )

torch.testing.assert_equal(
    torch.binary_op(input),
    numpy.binary_op(input),
    msg=lambda a, b, info: make_msg(input, a, b, info),
)
```

This should make it much easier for developers to find out what is actually going wrong.

Test Plan: Imported from OSS

Reviewed By: albanD

Differential Revision: D27903842

Pulled By: mruberry

fbshipit-source-id: 4c82e3d969e9a621789018018bec6399724cf388
2021-04-24 23:37:44 -07:00