I got feedback from two users that they were unsure of if the warning
meant anything. The warning now clearly states that there is a
performance degredation and to file us an issue about it.
This gives us coverage on all the batch rules for the cifar10 dp
example. We're slightly slower than opacus though, the numbers on my
machine are:
- 4 it/s for functorch vmap+grad
- 4.2 it/s for opacus
The differential should be investigated and I am also not sure if the
benchmarks are comparing the right things.
Problem: writing plumbing is repetitive and time consumign
Solution:
- run `gen_plumbing.py add.Tensor ~/pytorch/build/aten/src/ATen/RegistrationDeclarations.h`
- copy and paste the output into a source file.
In the long-run we don't want to be checking this codegen into the
codebase. However I haven't figured out what the design for the
long-term codegen should actually look like; how does one specify that
we want to *insert* some user-defined code into the middle of a
function? There are a few ideas:
Idea 1: ADD_TENSOR_PLUMBING_BEGIN and ADD_TENSOR_PLUMBING_END macros
```
ADD_TENSOR_PLUMBING_BEGIN
// your C++ logic here
ADD_TENSOR_PLUMBING_END
```
Idea 2: big .yaml file
```
- func: add.Tensor
// your C++ logic here
- func: add.Scalar
// your C++ logic here
```
PyTorch forward mode AD doesn't support a lot of operations yet. I've
verified that `jvp` can compose with `vmap`, but unfortunately `jvp`
doesn't compose with itself.
We have a problem where our tests fail everytime we rebase to the most
recent version of PyTorch. It would be nice to distinguish between
"PyTorch broke a previously passing test" vs "PyTorch added a new test
that would have already failed on PyTorch"
The solution that this PR introduces is for functorch to maintain a
"lagging" OpInfo database. The lagging database needs to be updated
every once in a while with new OpInfos from pytorch/pytorch. This makes
it so that functorch does not randomly get new OpInfo tests.
Updates make_functional to use the new improved variants. The new
variants are superior in every way so we're replacing the previous
variants with this.
If someone wants the older variants, they can be found at:
- make_functional_with_buffers_deprecated_v1
- make_functional_deprecated_v1
make_functional*_v2 is superior to the older make_functional. This PR
has all of our examples use it.
This PR also adds a "combine_state_for_ensemble(models)" API.
Coming soon: We're probably going to break BC on make_functional and
replace it with make_functional_v2. That's the nice thing about being a
prototype, we don't have to worry about BC too much.
