8cde4c4d22
Summary: As part of the Variable/Tensor merge work: https://github.com/pytorch/pytorch/issues/13638, we make the following changes in this PR: 1. Remove the `Variable::Impl` class and the `DifferentiableViewImpl` class 2. Change all `Variable.data()` call sites to either use `Variable` directly, or use `Variable.tensor_data()` 3. Remove `Variable.data()` API 3. Add `Variable.variable_data()` that matches `tensor.data` in Python API, which creates a new `Variable` that shares the same storage and tensor metadata with the original `Variable`, but with a completely new autograd history. After this PR, Variable doesn't wrap a Tensor internally anymore, and both Variable and Tensor use the same TensorImpl class as its `impl_`. The only difference is that Variable always has AutogradMeta in its TensorImpl, but Tensor doesn't. **Note that this PR is BC-breaking in the following use cases:** **Use Case 1:** Previously, `x.data = y` works even if `x` and `y` are of different TensorImpl type (e.g. `x` is a CPU dense tensor whose impl is of type TensorImpl, while `y` is a CPU sparse tensor whose impl is of type SparseTensorImpl). However, after this PR, `x.data = y` doesn't work anymore if `x` and `y` are of different TensorImpl type, because the underlying implementation `variable.set_data(tensor)` no longer works if `variable` and `tensor` have different TensorImpl type. **Use Case 2:** If a tensor `x`'s `grad` is sparse, accumulating dense gradients to `x` will change the tensor that `x.grad` is pointing to. This is better illustrated with the following example: ```python params = torch.tensor([1.5, 1.5]).requires_grad_() with torch.no_grad(): # Change gradient to a sparse tensor params.grad = torch.sparse_coo_tensor(torch.tensor([[1, 1]]).long(), torch.tensor([1., 1.])) grad_saved = params.grad params.backward(torch.tensor([1.5, 1.5])) assert id(grad_saved) == id(params.grad) # This will fail after this PR ``` The assertion in the last line will fail after this PR, because adding dense gradients to sparse gradients will change the `params.grad` tensor reference. Pull Request resolved: https://github.com/pytorch/pytorch/pull/17072 Differential Revision: D14075257 Pulled By: yf225 fbshipit-source-id: 0e681df641270dea586042dd26db59f2e76b5957
Autograd
Autograd is a hotspot for PyTorch performance, so most of the heavy lifting is implemented in C++. This implies that we have to do some shuffling between Python and C++; and in general, we want data to be in a form that is convenient to manipulate from C++.
Our general model is that for any key data type that autograd manipulates,
there are two implementations: a C++ type and a Python object type. For
example, consider variables in autograd: we have both Variable in variable.h
(the C++ type) and THPVariable in python_variable.h (the Python type.)
(By the way, THP stands for TorcH Python, not to be confused with THPP, TorcH
C++). Variable contains the payload of a variable, while THPVariable just
contains a shared_ptr reference to Variable, as well as references to other
Python objects which the Python runtime needs to know about. A lot of
data accessor implementations in python_variable.cpp simply reach through
to the underlying Variable and return the appropriate value.
The most complicated application of this principle is Function, which also supports users implementing custom behavior in Python. We have the following classes:
Functioninfunction.h, the C++ type.THPFunctioninpython_function.h, the Python object type. Inpython_function.cpp, you can see the boilerplate that tells the Python interpreter about this object.PyFunctioninpython_function.h, a subclass ofFunctionwhich forwardsapplyto a PythonTHPFunction. (NOT a Python object, despite its name!)
Outside of PyFunction, the C++ objects largely avoid referencing Python
objects (there are a few exceptions, like pyobj in Variable, and
PyFunction, whose whole point is to let C++ call into Python). And pyobj
in Function to ensure uniqueness of the associated python wrapper (if it exists).