Summary:
Make it a bit easier to run the tests anywhere/avoid skipping the tests by using buffers instead of temporary files.
[Er, still figuring out how the sync tooling works, I'll send this against github once the first diff is landed]
Test Plan: buck2 test
Reviewed By: fluckydog232
Differential Revision: D44818261
Pull Request resolved: https://github.com/pytorch/pytorch/pull/98798
Approved by: https://github.com/ezyang
Summary: IL generates massive function names: which meant that the pickle opcode used is BINUNICODE instead of the short version -- and then it would silently get skipped while pickling with protocol 4.
Differential Revision: D44815351
Pull Request resolved: https://github.com/pytorch/pytorch/pull/98674
Approved by: https://github.com/ezyang
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Test Plan: S271899
Reviewed By: lisroach
Differential Revision: D36402110
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78973
Approved by: https://github.com/osalpekar
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/72237
add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.
https://www.internalfb.com/intern/diff/D35423079/ was reverted due to this refactor changing the name of where most of the implementation components of PackageExporter/PackageImporter come from like ModuleActionType_ etc.
This diff also changes the import paths where these components come from to point to the correct file compared to D35423079
Test Plan: Imported from OSS
Reviewed By: malfet
Differential Revision: D35423079
Pulled By: PaliC
fbshipit-source-id: 31abc4364d5fd007911cfb67cf36ebfac5d786f4
(cherry picked from commit 023b0d1445e0b1e1bb7a03c660cd62eb9d26d2a6)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/65101
As title. Previously this was guarded against for implementation
simplicity, as we didn't really think there was a use case for saving a
mangled module name directly.
But people started doing stuff like:
```
exporter.save_module(my_imported_obj.__module__)
```
which implicitly passes along the mangled module name.
This PR makes it so that given `PackageImporter` instance can always
import modules that it created, and changes `PackageExporter` to
properly demangle the resulting module name when writing the package to
the export archive.
Differential Revision:
D30975712
D30975712
Test Plan: Imported from OSS
Pulled By: suo
fbshipit-source-id: d9e849bf651713890e72dccdcef74fa52d377149
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61148
Changes `__import__` processing to silently skip cases where the `__import__` statement cannot be parsed. Adds failed imports to a list retrievable by `PackageExporter.failed_dunder_import_list()`.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D29559680
Pulled By: Lilyjjo
fbshipit-source-id: 2513d0b9ef271c85cadc3f5a013fbd8c8de80b46
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61145
Remove 'verbose' mode from PackageExporter as people have complained that it is not useful.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D29559681
Pulled By: Lilyjjo
fbshipit-source-id: eadb1a3a25fadc64119334a09bf1fa4b355b1edd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/57341
Require that users be explicit about what they are going to be
interning. There are a lot of changes that are enabled by this. The new
overall scheme is:
PackageExporter maintains a dependency graph. Users can add to it,
either explicitly (by issuing a `save_*` call) or explicitly (through
dependency resolution). Users can also specify what action to take when
PackageExporter encounters a module (deny, intern, mock, extern).
Nothing (except pickles, tho that can be changed with a small amount
of work) is written to the zip archive until we are finalizing the
package. At that point, we consult the dependency graph and write out
the package exactly as it tells us to.
This accomplishes two things:
1. We can gather up *all* packaging errors instead of showing them one at a time.
2. We require that users be explicit about what's going in packages, which is a common request.
Differential Revision: D28114185
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Pulled By: suo
fbshipit-source-id: fa1abf1c26be42b14c7e7cf3403ecf336ad4fc12
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/57340
This API was only used within our own implementation. I couldn't find
any uses anywhere else. Removing it to reduce our overall surface area,
and also because the semantics are unclear in a world where
serialization is deferred to close() time.
Differential Revision: D28114188
Test Plan: Imported from OSS
Reviewed By: anjali411
Pulled By: suo
fbshipit-source-id: 6da53f20518885c7f4359e00e174f5e911906389
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/55996
**Sumamary**
This commit modifies `PackageExporter.save_module` so that the `module`
argument can be either a string (`str`) or a module
(`types.ModuleType`).
**Test Plan**
This commit adds a unit test similar to `TestSaveLoad.test_save_module`
that tests that calling `save_module` with a module object works.
**Fixes**
This commit fixes#55939.
Test Plan: Imported from OSS
Reviewed By: jamesr66a, huiguoo
Differential Revision: D27771781
Pulled By: SplitInfinity
fbshipit-source-id: 57c8cf45575bb8dcfca711759fadfff72efb35e7
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54303
**Summary**
Creating temporary files can cause problem in fbcode. This commit
updates the packaging tests so that exporters write to a memory
buffer when tests run in fbcode.
**Test Plan**
Continuous integration.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D27180839
Pulled By: SplitInfinity
fbshipit-source-id: 75689d59448de2cd1595ef0ecec69e1bbcf9a96f
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53749
Split up tests into cases that cover specific functionality. Goals:
1. Avoid the omnibus test file mess (see: test_jit.py) by imposing early
structure and deliberately avoiding a generic TestPackage test case.
2. Encourage testing of individual APIs and components by example.
3. Hide the fake modules we created for these tests in their own folder.
You can either run the test files individually, or still use
test/test_package.py like before.
Also this isort + black formats all the tests.
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D26958535
Pulled By: suo
fbshipit-source-id: 8a63048b95ca71f4f1aa94e53c48442686076034
