Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25563
Before, for binary ops, name inference occurred after shape checks. This
defeats the purposes for names because the names are supposed to tell
the user that i.e. their tensors are misaligned or that they are adding
incompatible tensors.
This PR changes TensorIterator so that names are computed before shape checks and
propagated after the binary ops are finished. In order to support this,
this PR makes the following changes:
- adds a `names_` field to TensorIterator, similar to `shape_`. This is
necessary to hold the output names, that are computed in
`compute_names`, until they are used in `propagate_names_to_outputs()`.
Test Plan: Imported from OSS
Differential Revision: D17158869
Pulled By: zou3519
fbshipit-source-id: 0caa90f7a93e4d9bdb2549cd330cc3abd2258868
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25568
Test Plan
- new test [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D17159069
Pulled By: zou3519
fbshipit-source-id: fbc185ea5865b128508451096b742ac18e467670
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25566
masked_select returns a tensor with None names. However, it broadcasts
its inputs so we need to perform a check that they are broadcastable.
Test Plan: - new tests [namedtensor ci]
Differential Revision: D17159071
Pulled By: zou3519
fbshipit-source-id: ad201f3f73bc54163ede1ba3d906d2409ebef475
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25431
I put the name propagation logic in a central place, `make_reduction`,
that creates a TensorIterator for the reduction. This lets us implement
name inference rules for mean, std, var, std_mean, and var_mean.
Test Plan
- new tests [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D17123577
Pulled By: zou3519
fbshipit-source-id: 2d47080a40da0c4bcabbb3df71ffa8fbeb7a14c6
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25345
Test Plan
- New tests [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D17101486
Pulled By: zou3519
fbshipit-source-id: 58e803b042056ee6abab8551517f74078f2b81d5
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25177
Test Plan
- new tests [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D17051452
Pulled By: zou3519
fbshipit-source-id: 7259cdb7ba7f480035528cf3c60ef6d051e42db5
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25123
The approach is different for CPU and CUDA. In particular:
- in CPU, I added a name inference rule to bmm_out
- in CUDA, bmm calls THCTensor_(baddbmm) so I added a name inference
rule to that.
When one calls baddbmm on CPU or CUDA, it'll error out with NYI due to
named_guard: True on it in native_functions.yaml. I'm not planning on
implementing baddbmm soon because it's a little tricky to add it to CPU
and bmm is more commonly used function.
Test Plan
- new tests [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D16998073
Pulled By: zou3519
fbshipit-source-id: 8dc01898964318717911f28eebd6cdfffc7dfcf2
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24474
torch.dot is a little weird. It ignores the names of its inputs to be
consistent with the rest of our matrix multiplication functions.
I've written the implementation using a helper function that is also
used by other matrix multiplication functions so that it is easy to
change the behavior.
Test Plan
- new tests [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D16915802
Pulled By: zou3519
fbshipit-source-id: 628a6de1935357022cc92f4d23222736a70bb070
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24471
mv(Tensor[M, N], Tensor[O]) ignores the names of N and O and returns a
tensor with names [M].
Test Plan: - new tests [namedtensor ci]
Differential Revision: D16915805
Pulled By: zou3519
fbshipit-source-id: d7d47903f249f85ef3be8a188d51993834bf5f55
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24469
tensor.expand(*sizes) returns a tensor with names equal to tensor.names
plus unnamed padding in the beginning dimensions.
For example, Tensor[H, W].expand(10, 2, 128, 128) -> Tensor[None, None,
H, W].
Test Plan: - new tests [namedtensor ci]
Differential Revision: D16915804
Pulled By: zou3519
fbshipit-source-id: 77ac97f42e9959d7f6d358c5286e3dc27488e33d
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24306
Featuring:
- a new way of writing name inference tests. At some point I'll migrate
the older tests over.
- The out= variants aren't implemented. This is because they are a
little weird: the output gets resized, but I haven't throught through
what semantics that should have.
Test Plan: - new tests [namedtensor ci]
Differential Revision: D16915801
Pulled By: zou3519
fbshipit-source-id: 29ae2ee414c7d98e042965458c5dccef7ddbd4dd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24921
Let `unnamed = torch.randn(1, 1, 1)` and `named = torch.randn(1, 1,
names=('N', 'C'))`.
Previously, there was a bug where `unnamed + named` would error out.
This happened because `unify_from_right(unnamed.opt_names(),
named.opt_names())` would return `named.names()`, which was propagated
to the output tensor. However, the output tensor has dim 3, but
`names.names()` only has 2 elements, so the code would throw an error.
The solution implemented in this PR is to stop trying to do premature
optimization. If all inputs to an operation doesn't have names, then
don't run name inference. However, if any inputs do, then materialize
the names and run name inference.
It's possible to make this more efficient for the case where some inputs
are named and some aren't, but we should benchmark these cases
and determine if it is necessary for it to be more efficient.
Test Plan: - new tests [namedtensor ci]
Differential Revision: D16930710
Pulled By: zou3519
fbshipit-source-id: 0de73c803c8b0f9a1c2d80684b9a47cccba91cbc
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24885
Store a static pre-allocated vector of names. When one calls
`default_names`, it gives a const reference to some amount of these
names.
Also make clearer the maximum number of dimensions we support for named
tensors. Right now it is 64 but that number is easy to change. 64
follows some internal pytorch maximum number of dimensions;
TensorIterator reduce ops have a limit of 64 dims.
Test Plan: - new tests [namedtensor ci]
Differential Revision: D16915803
Pulled By: zou3519
fbshipit-source-id: 931741b199456f8976882b82f25ab5af6dcd108b
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24087
Added name inference rules for softmax and log_softmax.
Added the overloads for Dimname dim to softmax and log_softmax.
Test Plan: - [namedtensor ci]
Differential Revision: D16763391
Pulled By: zou3519
fbshipit-source-id: 676a14666d42441eb7d3c9babef7461c7b78d290
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24028
Previously, torch.abs(tensor, out=out) would ignore the names of the
`out` tensor and overwrite them with the names of `tensor`.
This patch changes the behavior to the following:
1) If `out` does not have names, then overwite them with `tensor.names`.
2) If `out` does have names, then check that `out.names` equals
`tensor.names`.
This patch also includes the following clean ups:
- renamed `default_names` to `FIXME_default_names` because it is
inefficient and needs to be fixed.
- Renamed impl::internal_get_names / impl::internal_has_names to
impl::get_names / impl::set_names. Devs should feel free to use them, so
I removed the internal_ prefix.
- Moved internal_set_names to NamedTensor.{h, cpp}. These functions
still have the internal_ prefix because their use requires caution.
Test Plan: - [namedtensor ci]
Differential Revision: D16763387
Pulled By: zou3519
fbshipit-source-id: 57dcc7c759246def0db2746d1dca8eddd5e90049
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23973
Without loss of generality, I describe the API for `tensor.view_names`.
`tensor.names_` has an analogous API.
`tensor.view_names(*names)` returns a view on tensor with named dims `names`.
`names` must be of length `tensor.dim()`; otherwise, if '*' is in `names`,
then it (known as the "glob") is expanded greedily to be equal to the
corresponding names from `tensor.names`.
For example,
```
>>> x = torch.empty(2, 3, 5, 7, names=('N', 'C', 'H', 'W'))
>>> x.view_names('*', 'height', 'width').names
('N', 'C', 'height', 'width')
>>> x.view_names('batch', '*', 'width').names
('batch', 'C', 'H', 'width')
```
tensor.view_names(**rename_map) returns a view on tensor that has
renamed dims as specified in the mapping `rename_map`.
For example,
```
>>> x = torch.empty(2, 3, 5, 7, names=('N', 'C', 'H', 'W'))
>>> x.view_names(W='width', H='height').names
('N', 'C', 'height', 'width')
```
These are different(!!!) from the C++ API, which only allows the
following:
- tensor.view_names(optional<DimnameList>)
C++ API parity for named tensors is not important right now; I am
punting that to the future.
Test Plan: - [namedtensor ci]
Differential Revision: D16710916
Pulled By: zou3519
fbshipit-source-id: 7cb8056c0fb4c97b04c3a2d1dd0f737e0a67ce34
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23962
This change should make the semantics clearer.
`tensor.names_(names)` sets tensor.names to be `names`.
`tensor.view_names(names)` returns a view of the tensor with names
`names`.
Test Plan
- [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D16710915
Pulled By: zou3519
fbshipit-source-id: c82fa9812624d03c86f7be84b0a460e3c047aaa0
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23804
`output = tensor.align_to(names)` returns a view of `tensor` such that
`output.names = names`. Dimensions with the same names in `tensor` and
`output` have the same sizes; dimensions with new names have size 1.
The following must be true for this operation to succeed:
1) tensor.names must be a subsequence (not necessarily contiguous) of `names`
2) Aligning tensor.names to names must not change the absolute position from the
right of any unnamed dimension.
In practice, these constraints mean that aligning cannot transpose
names.
Some examples:
- Tensor[C].align_to(C) -> Tensor[C]
- Tensor[N].align_to([N, C]) -> Tensor[N, C]
- Tensor[H, W].align_to([N, H, W, C]) -> Tensor[N, H, W, C]
- Tensor[None].align_to([N, None]) -> Tensor[N, None]
- Tensor[N].align_to([N, None None]) -> Tensor[N, None, None]
Examples of error cases:
- Tensor[W, H].align_to([N, H, W, C]) -> Error (not a subsequence)
- Tensor[None, H].align_to([None, H, W]) -> Error (would change the
absolute position from the right of a None dimension)
`torch.align_tensors(*tensors)` aligns the named dimensions of each
tensor according to the alignment rules so that they can be used in an
operation. More concretely, it aligns each tensor to the
longest names among the names of the tensors in `tensors`.
This allows users to emulate "broadcasting by names", which is one of
the things named tensors tries to enable. Here is an example:
```
imgs: Tensor[N, C, H, W]
scale: Tensor[N]
// Doesn't work because we do broadcasting by alignment by default
imgs * scale
// Does work
imgs, scale = torch.align_tensors(imgs, scale)
imas * scale
```
Future:
- Consider allowing broadcasting by names by default.
Test Plan:
- The diff looks pretty large but more than half of it is testing.
- new tests [namedtensor ci]
Differential Revision: D16657927
Pulled By: zou3519
fbshipit-source-id: e2f958bf5146c8ee3b694aba57d21b08e928a4e6
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24108
`torch.empty_like(tensor)` and `tensor.clone()` both propagate names to
the output tensor.
As a part of this change, I fixed the empty(..., names=) overload to
include the `memory_format` argument in the normal `empty` declaration
in native_functions.yaml.
Test Plan: - [namedtensor ci]
Differential Revision: D16763392
Pulled By: zou3519
fbshipit-source-id: c7b2bc058d26a515a5fd8deef22c2acb290c8816
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24107
In the short term, we implement this by having overloads for each of
these functions. In the long term, the plan is to move DimnameList to
TensorOptions so that we do not have to duplicate work.
Also fixes the implementation of empty. If there are no names, we should
just return an unnamed tensor instead of telling the user we don't
support their backend/layout.
Test Plan: - [namedtensor ci]
Differential Revision: D16763393
Pulled By: zou3519
fbshipit-source-id: 7324a6b157187d4f74abc5459052f3323a417412
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24202
tensor.set_names(names) is the out-of-place variant of
tensor.set_names_(names). This naming is probably confusing so I am
taking any and all suggestions.
Test Plan: - run tests [namedtensor ci]
Differential Revision: D16773014
Pulled By: zou3519
fbshipit-source-id: 61024303c1a34db631cc4cb2c53757345e40d72c
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24106
Test Plan
- Code reading. assertTensorDataAndNamesEqual isn't used in this commit
but it'll be used in future commits.
- [namedtensor ci]
Test Plan: Imported from OSS
Differential Revision: D16763390
Pulled By: zou3519
fbshipit-source-id: 170e27ebc4d79aca939c5d101489b20faedc6133
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24105
tensor.set_names(names) is the out-of-place variant of
tensor.set_names_(names). This naming is probably confusing so I am
taking any and all suggestions.
Test Plan: - run tests [namedtensor ci]
Differential Revision: D16763388
Pulled By: zou3519
fbshipit-source-id: 4b2fb3acc0514515e7ca805dbc5c3d4a9bd96317
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23746
`torch.empty_like(tensor)` and `tensor.clone()` both propagate names to
the output tensor.
As a part of this change, I fixed the empty(..., names=) overload to
include the `memory_format` argument in the normal `empty` declaration
in native_functions.yaml.
Test Plan: - [namedtensor ci]
Differential Revision: D16647821
Pulled By: zou3519
fbshipit-source-id: 43b261f3456b6bf5fca7b6313e659b259a2ba66d
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23743
In the short term, we implement this by having overloads for each of
these functions. In the long term, the plan is to move DimnameList to
TensorOptions so that we do not have to duplicate work.
Test Plan: - [namedtensor ci]
Differential Revision: D16647820
Pulled By: zou3519
fbshipit-source-id: c6c53c5f26a86b730cbc4d4eb69907ac0e08fc65
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23801
Test Plan
- Code reading. assertTensorDataAndNamesEqual isn't used in this commit
but it'll be used in future commits.
- [namedtensor ci]
gh-metadata: pytorch pytorch 23801 gh/zou3519/90/head
Test Plan: Imported from OSS
Differential Revision: D16667816
Pulled By: zou3519
fbshipit-source-id: 66519cd5d17bda4c4304a1bc6e2a03ae59d49e39
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23624
tensor.set_names(names) is the out-of-place variant of
tensor.set_names_(names). This naming is probably confusing so I am
taking any and all suggestions.
Test Plan:
- run tests [namedtensor ci]
gh-metadata: pytorch pytorch 23624 gh/zou3519/86/head
Differential Revision: D16621830
Pulled By: zou3519
fbshipit-source-id: f8a3837d3a370b41210e938369348dcbb4aee53a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23623
This is a quick, not-user-facing check for if pytorch was built with BUILD_NAMEDTENSOR=1.
Test Plan:
- run tests [namedtensor ci]
gh-metadata: pytorch pytorch 23623 gh/zou3519/85/head
Differential Revision: D16621829
Pulled By: zou3519
fbshipit-source-id: d7e1161dc176bab2c1f953265722daeba1e63102
Summary:
`is_pinned` was moved to native_functions.yaml, disabling it for named
tensors. This PR re-enables its usage for named tensors.
I wrote a named inference rule for torch.clone(), but something happened
to it. Disable it for now so we can get the namedtensor ci to be green.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23597
Test Plan: - run tests [namedtensor ci]
Differential Revision: D16581771
Pulled By: zou3519
fbshipit-source-id: 498018cdc55e269bec80634b8c0a63ba5c72914b
