This PR addresses a minor typo in the file `test/quantization/fx/test_model_report_fx.py`:
- Corrected the word "paramter" to "parameter" for better readability and accuracy.
While it's a small change, correcting such typographical errors contributes to maintaining the overall quality and professionalism of the codebase.
Thank you for your time and consideration in reviewing this PR. I'm happy to make any further adjustments if needed.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/157646
Approved by: https://github.com/yewentao256, https://github.com/ezyang
This PR is part of a series attempting to re-submit https://github.com/pytorch/pytorch/pull/134592 as smaller PRs.
In quantization tests:
- Add and use a common raise_on_run_directly method for when a user runs a test file directly which should not be run this way. Print the file which the user should have run.
- Raise a RuntimeError on tests which have been disabled (not run)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/154728
Approved by: https://github.com/ezyang
Constant time access of first value in collection. This is a constant time operation instead of converting the item to a list to get the first item which is linear. The rule is turned on which automatically autofixes and enforces this.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/115507
Approved by: https://github.com/malfet
Applies the remaining flake8-comprehension fixes and checks. This changes replace all remaining unnecessary generator expressions with list/dict/set comprehensions which are more succinct, performant, and better supported by our torch.jit compiler. It also removes useless generators such as 'set(a for a in b)`, resolving it into just the set call.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94676
Approved by: https://github.com/ezyang
I applied some flake8 fixes and enabled checking for them in the linter. I also enabled some checks for my previous comprehensions PR.
This is a follow up to #94323 where I enable the flake8 checkers for the fixes I made and fix a few more of them.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94601
Approved by: https://github.com/ezyang
Summary: This adds the capability to generate a QConfigMapping based on
the suggestions of the ModelReport API for the user to use. The only
dependency of this feature is that the calibration is run before the
generation of the QConfigMapping and there is no dependency on the
report generation other than that the observers cannot be removed before
this is called. This maps module fqns to EqualizationQConfigs instead of regular
QConfigs.
Example Usage (after callibration):
```
quantization_mapping = mod_report.generate_qconfig_mapping()
equalization_mapping = mod_report.generate_equalization_mapping()
prepared_model = quantize_fx.prepare_fx(model, mapping, example_input, _equalization_config=equalization_mapping)
quantized_model = quantize_fx.convert_fx(prepared)
```
This was tested by ensuring that the suggestions generated in the QConfigMapping are:
1. Correct according to the set backend and data passed through
2. Able to be prepared and converted as a proper config (is a valid config)
The test for this is a part of the TestFxModelReportClass test suite.
Test Plan: python test/test_quantization.py TestFxModelReportClass.test_equalization_mapping_generation
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83698
Approved by: https://github.com/jerryzh168
Summary: This adds the capability to generate a QConfigMapping based on
the suggestions of the ModelReport API for the user to use. The only
dependency of this feature is that the callibration is run before the
generation of the QConfigMapping and there is no dependency on the
report generation other than that the observers cannot be removed before
this is called.
Example Usage (after callibration):
```
mapping = mod_report.generate_qconfig_mapping()
prepared_model = quantize_fx.prepare_fx(model, mapping, example_input)
quantized_model = quantize_fx.convert_fx(prepared)
```
This was tested by ensuring that the suggestions generated in the
QConfigMapping are:
1. Correct according to the set backend and data passed through
2. Able to be prepared and converted as a proper config (is a valid
config)
The test for this is a part of the TestFxModelReportClass test suite.
Test Plan: python test/test_quantization.py TestFxModelReportClass.test_qconfig_mapping_generation
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83688
Approved by: https://github.com/jerryzh168
Summary: This creates the framework in the ModelReport API for the
generation of QConfigs by the ModelReport instance based on suggestions.
This functionality will eventually be added into the report generation
or be something that complements it, but for now it will be an
independent call for API stability and to be able to better modularize
the features as it stabilizes.
This also adds the framework for the relavent test function and a note
in the README at what future changes are planned for this new method in
the ModelReport API.
Test Plan: python test/test_quantization.py TestFxModelReportClass.test_qconfig_generation
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83091
Approved by: https://github.com/HDCharles
Summary: This adds the capability to visualize the table of information
in the ModelReportVisualizer. This allows the user to filter based on
module name pattern match or feature name pattern match and the
implemented method `generate_table_visualization` prints out the table
in a string format that is easy to parse.
Expected Usage
```
mod_rep_visualizer.generate_table_visualization()
```
Can also pass in optional filters as well if needed.
The tests for this were just visual inspection for two reasons:
1.) This method does not return anything, it just generates the
visualization
2.) All the data to create the table visualization is gotten from
`generate_filtered_tables` which is already tested, so testing all that
for this again would be redundant.
Example Printed Output
```
Tensor Level Information
idx layer_fqn input_activation_global_max input_activation_global_min input_weight_channel_axis input_weight_threshold outlier_detection_channel_axis outlier_detection_ratio_threshold outlier_detection_reference_percentile weight_global_max weight_global_min
----- ------------- ----------------------------- ----------------------------- --------------------------- ------------------------ -------------------------------- ----------------------------------- ---------------------------------------- ------------------- -------------------
1 block1.linear 1.9543 -1.33414 1 0.5 1 3.5 0.95 0.380521 -0.568476
2 block2.linear 1.81486 0 1 0.5 1 3.5 0.95 0.521438 -0.0256195
Channel Level Information
idx layer_fqn channel constant_batch_counts input_activation_per_channel_max input_activation_per_channel_min input_weight_channel_comparison_metrics input_weight_equalization_recommended outlier_detection_batches_used outlier_detection_is_sufficient_batches outlier_detection_percentile_ratios outliers_detected weight_per_channel_max weight_per_channel_min
----- ------------- --------- ----------------------- ---------------------------------- ---------------------------------- ----------------------------------------- --------------------------------------- -------------------------------- ----------------------------------------- ------------------------------------- ------------------- ------------------------ ------------------------
1 block1.linear 0 0 1.9543 -1.33414 0.956912 True 1 True 1.77489 False 0.300502 -0.568476
2 block1.linear 1 0 1.14313 -0.756184 1.04378 True 1 True 2.07887 False 0.336131 -0.261025
3 block1.linear 2 0 0.653274 -0.937748 1.10837 True 1 True 1.00712 False 0.380521 -0.183536
4 block2.linear 0 0 1.81486 0 0.542731 True 1 True 1.78714 False 0.13552 -0.0256195
5 block2.linear 1 0 1.72578 0 0.505475 True 1 True 1.40475 False 0.485536 0.352621
6 block2.linear 2 0 1.7284 0 0.909304 True 1 True 1.40392 False 0.521438 0.0906605
```
Test Plan: Visual Test
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81973
Approved by: https://github.com/jerryzh168
Summary: This adds the ability to generate and display the collected
statistics in a table format for the ModelReportVisualizer. The output
of this is a dictionary containing two keys, mapping to a tensor stats
table and channel stats table respectively.
The two ways you can filter is by module_fqn, by only including modules
with the `module_fqn_filter` substring, or by feature filter, which only includes
features that contain the `feature_filter` substring.
Expected Use:
```
table_dict = mod_rep_visualizer.generate_filtered_tables()
tensor_table = table_dict[ModelReportVisualizer.TABLE_TENSOR_KEY]
channel_table = table_dict[ModelReportVisualizer.TABLE_CHANNEL_KEY]
```
Headers for the Tensor level info:
```
idx layer_fqn feature_1 feature_2 feature_3 .... feature_n
---- --------- --------- --------- --------- ---------
```
Headers for the channel level info:
```
idx layer_fqn channel feature_1 feature_2 feature_3 .... feature_n
---- --------- ------- --------- --------- --------- ---------
```
The reason we split this up into two tables is because with the design
where everything is in one table, it is ambiguous and easy to mix up
whether a tensor level stat is actually tensor level stat or might be a
per channel stat since we would have a row for each channel.
Also changed some of the framework to abstract out the finding of the
tables to the actual visualization to make the API much easier for the
user to digest and parse.
Test Plan: python test/test_quantization.py TestFxModelReportVisualizer.test_generate_table
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81673
Approved by: https://github.com/jerryzh168
Summary: The current implementation of the InputWeightEqualization
detector broke when it was tested on MobileNetV2, and the reason for
this is that it wasn't able to properly handle groups in Conv layers,
and there also had to be some minor reshaping of the weights to handle
this as well.
In addition, the output was correspondingly tuned so that instead of
giving on output for each channel on each layer, it gives a single
suggestion per module and just lets it know how many of the channels
could benefit from input-weight equalization, and suggests it if it's
more than half.
There was also the realization that the test class didn't do a good job
of testing different dimensions for the batch vs. height vs. width, so
this was updated to be more comprehensive as well.
Test Plan: python test/test_quantization TestFxDetectInputWeightEqualization
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81971
Approved by: https://github.com/jerryzh168
Summary: Added the functionality to be able to get the feature names and
module_fqns from the ModelReportVisualizer class. The purpose of this
addition is so that users can see the exact set of module_fqns or
feature names that they can filter based on, and use this information to
perform their filtering.
Test Plan: python test/test_quantization.py
TestFxModelReportVisualizer.test_get_modules_and_features
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81647
Approved by: https://github.com/andrewor14
Summary: We created a ModelReportVisualizer class, and the primary
way it is envisioned that it is accessed is:
```
model_report_visualizer = model_reporter.generate_visualizer()
```
This method only works after reports have been generated and it takes in
the generated reports and reformats them to be ordered by module, into
the format required by the ModelReportVisualization. It then generates
the visualizer instance and returns that to the user.
Test Plan: python test/test_quantization.py TestFxModelReportClass.test_generate_visualizer
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81589
Approved by: https://github.com/andrewor14
Summary: This introduces the skeleton for the ModelReportVisualizer
class. This class helps visualize the information generated by the
ModelReport class `generate_report()` output. This class aims to provide
visualizations in a table, plot (line graph) and histogram view.
This also introduces an empty test class for testing visualizations. As
implementations start occuring for this class, tests will also be
approrpriately added.
This includes the high level descriptions for each of the methods as
well. Expected use cases will be added to the class description in a
future commit as that gets finalized.
Test Plan: python test/test_quantization.py TestFxModelReportVisualizer
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81523
Approved by: https://github.com/andrewor14
Summary: Currently, the ModelReport API only takes in detectors at the
beginning and for each of its methods, you have to pass in the model
each time, which doesn't really make sense because:
1. you will always want to be working on the same model
2. passing in a different model could break things, so more
fault-tolerant if we keep the model internally and make calls on it
Therefore, now the model will be passed in in intialization, and will
just be used for the rest of the operations with the local link.
All the ModelReport tests have been adjusted to account for this, and
this change must pass all the tests to ensure a successful API
transition.
If you wish to see how the updated API looks, the Expected Usage in the
ModelReport clas description has been updated to reflect the changes.
The README has also been updated with these changes as well.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81588
Approved by: https://github.com/jerryzh168
Summary: Currently, all the detectors have pretty accurate naming
schemes that give an idea of what they do. However, since now there are
more and more detectors being developed, there is a need to make sure
that the naming scheme for detectors are consistent for their keys.
This updates the keys of the returned dictionary keys to better
highlight if something is an activation stat or weight stat, etc.
Test Plan:
python test/test_quantization.py TestFxModelReportDetector
python test/test_quantization.py TestFxModelReportObserver
python test/test_quantization.py TestFxModelReportDetectDynamicStatic
python test/test_quantization.py TestFxModelReportClass
python test/test_quantization.py TestFxDetectInputWeightEqualization
python test/test_quantization.py TestFxDetectOutliers
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81587
Approved by: https://github.com/jerryzh168
Summary: Currently the InputWeightEqualizationDetector has a
multi-layered output.
Example
```
{'block1.linear': {'channel_axis_selected': 1,
'channel_comparison_metrics': tensor([0.8736, 0.6594, 0.2916], grad_fn=<DivBackward0>),
'input_range_info': {'global_max': tensor(9.),
'global_min': tensor(-10.),
'per_channel_max': tensor([9., 9., 9.]),
'per_channel_min': tensor([-10., -10., -10.])},
'input_weight_equalization_recommended': [True,
False,
False],
'threshold': 0.8,
'weight_range_info': {'global_max': tensor(0.5618, grad_fn=<UnbindBackward0>),
'global_min': tensor(-0.2211, grad_fn=<UnbindBackward0>),
'per_channel_max': tensor([0.3764, 0.5618, 0.2894], grad_fn=<NotImplemented>),
'per_channel_min': tensor([-0.2211, 0.2213, 0.2228], grad_fn=<NotImplemented>)}},
}
```
With all the levels, it can be hard to parse the information for
anything, especially the planned visualization feature where the data
has to be reorganized. Therefore, to make it standardized across all
detectors, all outputs will be limited to one level.
The new format is:
```
{'block1.linear': { 'channel_axis_selected': 1,
'channel_comparison_metrics': tensor([0.5705, 0.9457, 0.8891], grad_fn=<DivBackward0>),
'activation_global_max': tensor(9.),
'activation_global_min': tensor(-10.),
'activation_per_channel_max': tensor([9., 9., 9.]),
'activation_per_channel_min': tensor([-10., -10., -10.]),
'input_weight_equalization_recommended': [False, True, True],
'threshold': 0.8,
'weight_global_max': tensor(0.4258, grad_fn=<UnbindBackward0>),
'weight_global_min': tensor(-0.4958, grad_fn=<UnbindBackward0>),
'weight_per_channel_max': tensor([0.1482, 0.3285, 0.4258], grad_fn=<NotImplemented>),
'weight_per_channel_min': tensor([-0.1517, -0.4958, -0.3027], grad_fn=<NotImplemented>)},
}
```
The README will also be updated to reflect this change.
Test Plan: python test/test_quantization.py TestFxDetectInputWeightEqualization
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81586
Approved by: https://github.com/jerryzh168
Summary: Currently, the PerChannelDetector has a multi-layered output.
Example:
```
{'backend': 'qnnpack',
'per_channel_status': {'block1.linear': {'per_channel_supported': True,
'per_channel_used': False},
'block2.linear': {'per_channel_supported': True,
'per_channel_used': False}}}
```
The issue with this is that when it comes to future features such as
visualizations where we need to go through this dictionary, it can be
hard because of the variable number of layers.
This changes the output format of the PerChannelDetector to have a
standard format.
Ex.)
```
{'block1.linear': {'backend': 'qnnpack',
'per_channel_supported': True,
'per_channel_used': False},
'block2.linear': {'backend': 'qnnpack',
'per_channel_supported': True,
'per_channel_used': False}}
```
Test Plan: python test/test_quantization.py TestFxModelReportDetector
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81585
Approved by: https://github.com/HDCharles
Summary: The current Outlier detector does a good job of finding whether
data distributions passing through layers have outliers. However,
suppose we have a completely constant channel. The outlier detector
would not detect it as an outlier, but that is still something we want
to highlight because a constant channel usually is a result of a bad
configuration or something really wrong with the data.
To address this there are two additions to the outlier detector that
this commit makes:
- The first is to add whether there are any constant batches at all and
let the user know in the text report
- The second is to let the user know the number of total constant
batches found for each channel, so they can figure out if there are any
unnecessary channels present.
The exisiting outlier detector tests were modified to do a quick check
for this feature.
Test Plan: python test/test_quantization.py TestFxDetectOutliers
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81249
Approved by: https://github.com/andrewor14
Summary: The outlier detector has a feature where it's able to notify
the user if below the whole set of batches that passed through were used
in Outlier calculation, which mainly happens as a result of 0-errors.
This changes the code so that instead of comparing against a value like
30 as we were before, we now let the user pass in an optional fractional
value and if the ratio of the batches used was below that value, the
detector alerts the user.
Test Plan: python test/test_quantization.py TestFxDetectOutliers
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81174
Approved by: https://github.com/andrewor14
Summary: This adds the implementation for the report generation for the
Outlier Detector class. This includes both the generation of a
dictionary containing each module that had an observer attached and any
relavent stats collected by the observer that can help shed light on
outlier relavent data or computed metrics. It also includes a string
denoting specific modules that had outliers and gives a bit of insight
into what channels they are contained in.
This contains both the implementation for the report generation for the
outlier detector as well as a test class to test the report generation
functionality.
Test Plan: python test/test_quantization.py TestFxDetectOutliers
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80937
Approved by: https://github.com/andrewor14
Summary: This adds the implementation for observer insertion point
selection for the OutlierDetector. For this detector, the insertion
points are to insert a ModelReportObserver before any leaf level module
to study the distribution of data that passes into the module to detect
outliers.
This commit contains the implementation of the observer insertion as
well as the relavent test case. Some code from the
InputWeightEqualization was abstracted and made more modular so the same
helper function could be used for multiple outlier class tests.
As a part of the work for this, there was testing done to determine what
a good default ratio threshold and reference percentile would be, and
the work to determine this (based on a normal distribution) was then
analyzed to find good paramters.
We still want to keep thresholds and reference percentile as something
the user can input because these were based on a normal distribution,
and it can definately vary depending on the type of data a user has.
Test Plan: python test/test_quantization.py TestFxDetectOutliers
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80880
Approved by: https://github.com/andrewor14
Summary: This adds the class framework for the ModelReport
OutlierDetector. This detector will be in charge of looking at
activation data and figuring out whether there are significant oultiers
present in them. It will average this data across batches to make a
recommendation / warning if significant outliers are found.
This commit contains just the class framework and a base test class.
Implementations will follow in following commits.
Test Plan: python test/test_quantization.py TestFxDetectOutliers
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80743
Approved by: https://github.com/HDCharles
Summary: This adds the implementation for the InputWeightEqualization
detector. This includes both the implementation and the relavent test
cases. This detector is meant to be added to initialize a ModelReport
instance and it will keep track of the necessary statistics to decide if
for certain layers of interest (linear and conv for now), it makes sense
to use input weight equalization and gives the suggestion to the user.
This includes the implementation and subsequent tests for the report
generation functionality of the detector. The full detector should now
be fleshed out and complete with this addition. This included
modifications to the ModelReportObserver class as well to capture min
and max per channel values. In addition, instead of passing in the
observer class to instantiate, the detectors now pass the ModelReport
instance the observer instance that they themselves instantiate.
Test Plan: python test/test_quantization.py TestFxDetectInputWeightEqualization
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80191
Approved by: https://github.com/HDCharles, https://github.com/andrewor14
Summary: This adds the implementation for the InputWeightEqualization
detector. This includes both the implementation and the relavent test
cases. This detector is meant to be added to initialize a ModelReport
instance and it will keep track of the necessary statistics to decide if
for certain layers of interest (linear and conv for now), it makes sense
to use input weight equalization and gives the suggestion to the user.
This implements the functionality of adding observer points for the
input-weight equalization detector and contains the relavent tests for
this functionality. The full Detector functionality will be fleshed out
in a later commit.
Test Plan: python test/test_quantization.py TestFxDetectInputWeightEqualization
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79962
Approved by: https://github.com/HDCharles, https://github.com/andrewor14
Summary: This adds the framework (method signatures and descriptors) for
the InputWeightEqualization Detector. There is no code implemenation yet
so the test suite for this is a simple pass. This Detector will be used
to determine whether input weight equalization should be recommended.
Test Plan: python test/test_quantization.py TestFxDetectInputWeightEqualization
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79916
Approved by: https://github.com/HDCharles
Summary: The ModelReport class in model_report.py combines the
functionality of the detectors and the ModelReportObserver. It creates
an end-to-end system where a user can pass in a prepared Graph Model to
insert the ModelReportObservers, then after the user callibrates their
model, the callibrated model can then be used by the ModelReport class
to generate reports based on what the user wished to gather information
on.
This contains the implementation and the tests for the generate_report
method which is used on a callibrated fx model to generate reports based
on data collected by the inserted observers during the callibration
phase and also potentially remove those observers if desired.
This also addresses and fixes a revert issue that has been fixed.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80054
Approved by: https://github.com/HDCharles
Summary: The ModelReport class in model_report.py combines the
functionality of the detectors and the ModelReportObserver. It creates
an end-to-end system where a user can pass in a prepared Graph Model to
insert the ModelReportObservers, then after the user callibrates their
model, the callibrated model can then be used by the ModelReport class
to generate reports based on what the user wished to gather information
on.
This contains the implementation and tests for the
prepare_detailed_calibration method which is used on a prepared fx model
to insert the desired observers for the different detectors.
This also fixes a revert issue with the applied fix.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80053
Approved by: https://github.com/HDCharles
Summary: The ModelReport class in model_report.py combines the
functionality of the detectors and the ModelReportObserver. It creates
an end-to-end system where a user can pass in a prepared Graph Model to
insert the ModelReportObservers, then after the user callibrates their
model, the callibrated model can then be used by the ModelReport class
to generate reports based on what the user wished to gather information
on.
This contains the init method and the signatures and docs for each
of the proposed helper functions.
This also address and fixes a revert issue.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80052
Approved by: https://github.com/HDCharles
Summary: The ModelReport class in model_report.py combines the
functionality of the detectors and the ModelReportObserver. It creates
an end-to-end system where a user can pass in a prepared Graph Model to
insert the ModelReportObservers, then after the user callibrates their
model, the callibrated model can then be used by the ModelReport class
to generate reports based on what the user wished to gather information
on.
This contains the implementation and the tests for the generate_report
method which is used on a callibrated fx model to generate reports based
on data collected by the inserted observers during the callibration
phase and also potentially remove those observers if desired.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79792
Approved by: https://github.com/HDCharles
Summary: The ModelReport class in model_report.py combines the
functionality of the detectors and the ModelReportObserver. It creates
an end-to-end system where a user can pass in a prepared Graph Model to
insert the ModelReportObservers, then after the user callibrates their
model, the callibrated model can then be used by the ModelReport class
to generate reports based on what the user wished to gather information
on.
This contains the implementation and tests for the
prepare_detailed_calibration method which is used on a prepared fx model
to insert the desired observers for the different detectors.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79752
Approved by: https://github.com/HDCharles
Summary: The ModelReport class in model_report.py combines the
functionality of the detectors and the ModelReportObserver. It creates
an end-to-end system where a user can pass in a prepared Graph Model to
insert the ModelReportObservers, then after the user callibrates their
model, the callibrated model can then be used by the ModelReport class
to generate reports based on what the user wished to gather information
on.
This contains the init method and the signatures and docs for each
of the proposed helper functions.
Test Plan: python test/test_quantization.py TestFxModelReportClass
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79595
Approved by: https://github.com/andrewor14
Summary: the goal is to add a base class to the model report detectors
so that they can contain a lot more specific information compared to the
primary model report class related to the observers and where they are
inserted etc.
Since this is just a base class, the testing will be with the
implemenations of the classes that derive from the base class
The two current detector methods were turned into Detector classes and
the tests were modified to reflect this, but the same functionality was
tested.
As a result, _detector.py was changed to detector.py
Test Plan: python test/test_quantization.py TestFxModelReportDetector
python test/test_quantization.py TestFxModelReportDetectDynamicStatic
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79671
Approved by: https://github.com/andrewor14
