Enable pytest for a few unique files. pytest runs tests in a different order than unittest (but still a consistent ordering with respect to itself) and some tests change global state, causing other tests to fail.
`test_transpose_non_contiguous` in `test_torchinductor.py` gets impacted from some other test but I'm not sure which one, so my solution is to reset the metrics before the rest of the test is run.
`test_register_patterns` in `test_quantize_fx.py` adds extra keys to global variables, so remove them when the test is done via unittest's `addCleanUp` which also works on pytest.
pytest doesn't really have an equivalent for `load_tests` so change it to be like `test_jit` that imports all the classes. I also attempted to dynamically import them, but I failed.
`test_public_api_surface` in `test_fx.py` checks for a backwards compatibility classification. There is a different test in test_fx that results in `fuser_utils` being imported. pytest runs this test before `test_public_api_surface` while unittest runs it after, so pytest sees `fuser_utils` when crawling through the modules.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/96397
Approved by: https://github.com/huydhn
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53749
Split up tests into cases that cover specific functionality. Goals:
1. Avoid the omnibus test file mess (see: test_jit.py) by imposing early
structure and deliberately avoiding a generic TestPackage test case.
2. Encourage testing of individual APIs and components by example.
3. Hide the fake modules we created for these tests in their own folder.
You can either run the test files individually, or still use
test/test_package.py like before.
Also this isort + black formats all the tests.
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D26958535
Pulled By: suo
fbshipit-source-id: 8a63048b95ca71f4f1aa94e53c48442686076034
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51676
We offer the ability to access the importer from within packaged modules by doing
`import resources`. This behavior is nice (and more powerful than the
importlib resources API), but I think `resources` is too common a name
(pip has a package for it)
Change to `import torch_package_importer` but open to bikeshedding
Test Plan: Imported from OSS
Reviewed By: jamesr66a
Differential Revision: D26620314
Pulled By: suo
fbshipit-source-id: 0942c99f02c0f55f5f3a1b2566961018b796bdd4
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53233
**Summary**
This commit adds a `deny` method to `PackageExporter` that allows
modules to be prohibited during the packaging process. A dependency on a
module matching the names or globs that `deny` was called with will
cause an exception to be raised.
**Test Plan**
This commit adds unit tests to `PackagingTest` for this new method:
`test_deny` and `test_deny_glob`.
**Fixes**
This commit fixes#53217.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D26834010
Pulled By: SplitInfinity
fbshipit-source-id: 469b5c6741bcc6dab77e352f41db38fa1e0dae12
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53232
**Summary**
This commit adds an optional `allow_empty` argument to
`PackageExporter.mock` and `PackageExporter.extern` that allows certain
patterns for mocked modules and extern modules to be marked ones that
*must* be matched during the packaging process. If a mock or extern
module with `allow_empty=False` is not matched while packaging, an error
is thrown.
**Test Plan**
This commit adds two new test cases to `PackagingTest`,
`test_extern_glob_allow_empty` and `test_mock_glob_allow_empty` that
test this new flag. Existing tests already tests `allow_empty=True`.
**Fixes**
This commit fixes#53217.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D26834011
Pulled By: SplitInfinity
fbshipit-source-id: 9cf4ea56079ae210d6cfa8604218849eb5cde5f4
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51976
FX serializes things by serializing Python code as a string and exec'ing
it on load. This accomplishes one goal (we don't have to pickle the
graph object directly) but breaks the pickle abstraction in ways that
are not composable with `torch.package`.
In particular:
1. `forward` is serialized by saving Python code. On load, it's
installed
by `exec`ing that code. This `exec` call needs to have the right
importer installed, otherwise it will not import modules from the
`torch.package` but instead import from the Python environment.
2. Any types/functions used are emitted as `import` statement in the
generated Python code. These are effectively dynamic dependencies of the
`GraphModule` being saved, and need to be registered as such so that the
`PackageImporter` will package them.
To address these, this PR introduces a new protocol for the
importer/exporter: `__reduce_package__`.
A class can implement `__reduce_package__` to customize how it is placed
in the importer/exproter. It functions very similarly to `__reduce__`,
except:
- `__reduce_package__` takes one argument, which is the
`PackageExporter`
instance. Users can use this instance to save stuff to the package to
implement their serialization. `__reduce__` takes no args.
- Only the 2-element tuple version of the return value for `__reduce__`
is supported (this could be extended if necessary).
- When the reduction function is called on load, an additional argument
is added to the beginning of the args tuple. This is the
`PackageImporter`
instance doing the loading.
The `__reduce_package__` protocol is defined using `persistent_id` and
`persistent_load`, which ensures that we can still use the cpickle
implementation of the pickler by default.
Pull Request resolved: #51971
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26340591
Pulled By: suo
fbshipit-source-id: 5872a7d22e832056399a7372bae8a57807717882
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52176
Added tooling to print out zipfile structure for PackageExporter and PackageImporter.
API looks like:
```
exporter.print_file_structure("sss" /*only include files with this in the path*/)
importer3.print_file_structure(False /*don't print storage*/, "sss" /*only include files with this in the path*/)
```
The output looks like this with the storage hidden by default:
```
─── resnet.zip
├── .data
│ ├── extern_modules
│ └── version
├── models
│ └── models1.pkl
└── torchvision
└── models
├── resnet.py
└── utils.py
```
The output looks like this with the storage being printed out:
```
─── resnet_added_attr_test.zip
├── .data
│ ├── 94574437434544.storage
│ ├── 94574468343696.storage
│ ├── 94574470147744.storage
│ ├── 94574470198784.storage
│ ├── 94574470267968.storage
│ ├── 94574474917984.storage
│ ├── extern_modules
│ └── version
├── models
│ └── models1.pkl
└── torchvision
└── models
├── resnet.py
└── utils.py
```
If the output is filtered with the string 'utils' it'd looks like this:
```
─── resnet_added_attr_test.zip
└── torchvision
└── models
└── utils.py
```
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D26429795
Pulled By: Lilyjjo
fbshipit-source-id: 4fa25b0426912f939c7b52cedd6e217672891f21
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51975
See comments in code.
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26340592
Pulled By: suo
fbshipit-source-id: 61b16bafad15e19060710ad2d8487c776d672847
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52320
as title
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26468416
Pulled By: suo
fbshipit-source-id: 890eecea76426918daff900402fbcbc149e48535
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51568
The default behavior of inspect.getfile doesn't work on classes imported
from PackageImporter, because it returns the following.
sys.modules[kls.__module__].__file__
Looking in `sys.modules` is hard-coded behavior. So, patch it to first
check a similar registry of PackageImported modules we maintain.
Test Plan: Imported from OSS
Reviewed By: yf225
Differential Revision: D26201236
Pulled By: suo
fbshipit-source-id: aaf5d7ee8ca0155619c8185e64f70a30152ac567
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50838
Similar to `torch.save` and `torch.jit.save`, accept a IO-like object
instead of just a file.
Test Plan: Imported from OSS
Reviewed By: nikithamalgifb
Differential Revision: D25982719
Pulled By: suo
fbshipit-source-id: 42f3665932bbaa6897215002d116df6338edae50
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50049
Rationale and implementation immortalized in a big comment in
`torch/package/mangling.md`.
This change also allows imported modules to be TorchScripted
Test Plan: Imported from OSS
Reviewed By: pbelevich
Differential Revision: D25758625
Pulled By: suo
fbshipit-source-id: 77a99dd2024c76716cfa6e59c3855ed590efda8b
Summary:
fix https://github.com/pytorch/pytorch/issues/50448.
This replaces all `test/*.py` files with run_tests(). This PR does not address test files in the subdirectories because they seems unrelated.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50451
Reviewed By: janeyx99
Differential Revision: D25899924
Pulled By: walterddr
fbshipit-source-id: f7c861f0096624b2791ad6ef6a16b1c4895cce71
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49066
This PR tweaks mock_module and extern_module. They are now renamed
mock and extern, and now only edit the package when a module matching
the pattern specified is required through dependency analysis.
save_extern_module and save_mock_module are added to explicitly modify
the package, but should not be needed by most users of the API unless they
are overriding require_package.
mock and extern now use bazel-style glob matching rules
(https://docs.bazel.build/versions/master/be/functions.html#glob).
i.e. `torch.**` matches `torch` and `torch.bar` but not `torchvision`.
mock and extern also now take an exclude list to filter out packages
that should not apply to the action.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D25413935
Pulled By: zdevito
fbshipit-source-id: 5c06b417bee94ac8e72c13985b5ec42fcbe00817
Summary:
There's some code which uses `six.PY3`, similar to:
```python
if six.PY3:
print("Python 3+ code")
else:
print "Python 2 code"
```
Where:
```python
PY3 = sys.version_info[0] == 3
```
When run on Python 4, this will run the Python 2 code! Instead, use `six.PY2` and avoid `six.PY3`.
---
Similarly, there's some `sys.version_info[0] == 3` checks, better done as `sys.version_info[0] >= 3`.
---
Also, it's better to avoid comparing the `sys.version` string, as it makes assumptions that each version component is exactly one character long, which will break in Python 3.10:
```pycon
>>> sys.version
'3.8.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53) \n[Clang 6.0 (clang-600.0.57)]'
>>> sys.version < "3.3"
False
>>> fake_v3_10 = '3.10.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53) \n[Clang 6.0 (clang-600.0.57)]'
>>> fake_v3_10 < "3.3"
True
```
---
Finally, I think the intention here is to skip when the Python version is < 3.6:
```python
unittest.skipIf(sys.version_info[0] < 3 and sys.version_info[1] < 6, "dict not ordered")
```
However, it will really skip for Python 0.0-0.5, 1.0-1.5 and 2.0-2.5. It's best to compare to the `sys.version_info` tuple and not `sys.version_info[1]`:
```python
unittest.skipIf(sys.version_info < (3, 6), "dict not ordered")
```
---
Found using https://github.com/asottile/flake8-2020:
```console
$ pip install -U flake8-2020
$ flake8 --select YTT
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/32389
Reviewed By: zou3519
Differential Revision: D24424662
Pulled By: ezyang
fbshipit-source-id: 1266c4dbcc8ae4d2e2e9b1d7357cba854562177c
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/45686
This uses an online graphviz viewer. The code is simpler, and
since it embeds all the data in the url you can just click the url
from your terminal.
Test Plan: Imported from OSS
Reviewed By: ZolotukhinM
Differential Revision: D24059157
Pulled By: zdevito
fbshipit-source-id: 94d755cc2986c4226180b09ba36f8d040dda47cc
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/45214
When in verbose mode the package exporter will produce an html visualization
of dependencies of a module to make it easier to trim out unneeded code,
or debug inclusion of things that cannot be exported.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D23873525
Pulled By: zdevito
fbshipit-source-id: 6801991573d8dd5ab8c284e09572b36a35e1e5a4
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/45015
torch.package allows you to write packages of code, pickled python data, and
arbitrary binary and text resources into a self-contained package.
torch.package.PackageExporter writes the packages and
torch.package.PackageImporter reads them.
The importers can load this code in a hermetic way, such that code is loaded
from the package rather than the normal python import system. This allows
for the packaging of PyTorch model code and data so that it can be run
on a server or used in the future for transfer learning.
The code contained in packages is copied file-by-file from the original
source when it is created, and the file format is a specially organized
zip file. Future users of the package can unzip the package, and edit the code
in order to perform custom modifications to it.
The importer for packages ensures that code in the module can only be loaded from
within the package, except for modules explicitly listed as external using :method:`extern_module`.
The file `extern_modules` in the zip archive lists all the modules that a package externally depends on.
This prevents "implicit" dependencies where the package runs locally because it is importing
a locally-installed package, but then fails when the package is copied to another machine.
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D23824337
Pulled By: zdevito
fbshipit-source-id: 1247c34ba9b656f9db68a83e31f2a0fbe3bea6bd
