mirror of
https://github.com/pytorch/pytorch.git
synced 2025-10-20 21:14:14 +08:00
5883523c1d
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/62030 Remove dtype tracking from Python Storage interface, remove all the different `<type>Storage` classes except for `ByteStorage`, and update serialization accordingly, while maintaining as much FC/BC as possible Fixes https://github.com/pytorch/pytorch/issues/47442 * **THE SERIALIZATION FORMAT IS FULLY FC/BC.** We worked very hard to make sure this is the case. We will probably want to break FC at some point to make the serialization structure of tensors make more sense, but not today. * There is now only a single torch.ByteStorage class. Methods like `Tensor.set_` no longer check that the dtype of storage is appropriate. * As we no longer know what dtype of a storage is, we've **removed** the size method from Storage, replacing it with nbytes. This is to help catch otherwise silent errors where you confuse number of elements with number of bytes. * `Storage._new_shared` takes a `nbytes` kwarg and will reject previous positional only calls. `Storage._new_with_file` and `_set_from_file` require explicit element size arguments. * It's no longer possible to convert storages to different types using the float/double/etc methods. Instead, do the conversion using a tensor. * It's no longer possible to allocate a typed storage directly using FloatStorage/DoubleStorage/etc constructors. Instead, construct a tensor and extract its storage. The classes still exist but they are used purely for unpickling. * The preexisting serialization format stores dtype with storage, and in fact this dtype is used to determine the dtype of the tensor overall. To accommodate this case, we introduce a new TypedStorage concept that exists only during unpickling time which is used to temporarily store the dtype so we can construct a tensor. **If you overrode the handling of pickling/unpickling, you MUST add handling for TypedStorage** or your serialization code will degrade to standard file-based serialization. Original pull request: https://github.com/pytorch/pytorch/pull/59671 Reviewed By: soulitzer, ngimel Differential Revision: D29466819 Pulled By: ezyang fbshipit-source-id: 4a14e5d3c2b08e06e558683d97f7378a3180b00e
39 lines
1.0 KiB
Python
39 lines
1.0 KiB
Python
"""Adds docstrings to Storage functions"""
|
|
|
|
import torch._C
|
|
from torch._C import _add_docstr as add_docstr
|
|
|
|
|
|
storage_classes = [
|
|
'ByteStorageBase',
|
|
]
|
|
|
|
|
|
def add_docstr_all(method, docstr):
|
|
for cls_name in storage_classes:
|
|
cls = getattr(torch._C, cls_name)
|
|
try:
|
|
add_docstr(getattr(cls, method), docstr)
|
|
except AttributeError:
|
|
pass
|
|
|
|
|
|
add_docstr_all('from_file',
|
|
"""
|
|
from_file(filename, shared=False, size=0) -> Storage
|
|
|
|
If `shared` is `True`, then memory is shared between all processes.
|
|
All changes are written to the file. If `shared` is `False`, then the changes on
|
|
the storage do not affect the file.
|
|
|
|
`size` is the number of elements in the storage. If `shared` is `False`,
|
|
then the file must contain at least `size * sizeof(Type)` bytes
|
|
(`Type` is the type of storage). If `shared` is `True` the file will be
|
|
created if needed.
|
|
|
|
Args:
|
|
filename (str): file name to map
|
|
shared (bool): whether to share memory
|
|
size (int): number of elements in the storage
|
|
""")
|