mirror of
https://github.com/pytorch/pytorch.git
synced 2025-10-23 23:04:52 +08:00
3d421b3137
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/46244 - What does the generated binding code do? The Python binding codegen produces code that takes the input list of PyObjects, finds the matching ATen C++ function using PythonArgParser, converts the PyObjects into C++ types and calls the ATen C++ function: ``` +--------+ parsing +------------------------+ binding +-----------------------+ | PyObjs | ---------> | PythonArgParser Output | ---------> | Cpp Function Dispatch | +--------+ +------------------------+ +-----------------------+ ``` - Are Python arguments 1-1 mapped to C++ arguments? Python arguments might be reordered, packed, unpacked when binding to C++ arguments, as illustrated below: ``` // Binding - Reorder & Packing // aten::empty.names(int[] size, *, Dimname[]? names, ScalarType? dtype=None, Layout? layout=None, Device? device=None, bool? pin_memory=None, MemoryFormat? memory_format=None) -> Tensor Python Args Cpp Args ----------------------------------------------------------- 0: size size 1: names names 2: memory_format -------+ 3: dtype -----+-|--> options 4: layout / | 5: device / +--> memory_format 6: pin_memory / 7: requires_grad -+ // Binding - Unpacking // aten::max.names_dim(Tensor self, Dimname dim, bool keepdim=False) -> (Tensor values, Tensor indices) Python Args Cpp Args ----------------------------------------------------------- +----> max /-----> max_values 0: input / self 1: dim / dim 2: keepdim / keepdim 3: out -----+ ``` - Why do we want to rewrite the python binding codegen? The old codegen takes Declarations.yaml as input. It doesn't distinguish between Python arguments and C++ arguments - they are all mixed together as a bag of non-typed dict objects. Different methods process these arg objects and add new attributes for various different purposes. It's not so obvious to figure out the semantics of these attributes. The complicated binding logic happens implicitly and scatteredly. ``` +--------------------+ | Native Functions | +--------------------+ | | v +--------------------+ | Cpp Signatures | +--------------------+ | | v +--------------------+ | Declarations.yaml | +--------------------+ | +-------------------------------------+ | +-------> | PythonArgParser Schema | | | +-------------------------------------+ | | . | | . v | . +--------------------+ +-------------------------------------+ | NonTyped Args Objs | --> | PythonArgParser -> Cpp Args Binding | +--------------------+ +-------------------------------------+ | . | . | . | +-------------------------------------+ +-------> | Cpp Function Dispatch | +-------------------------------------+ ``` This PR leverages the new immutable data models introduced in the new aten codegen. It introduces dedicated data models for python schema. This way, we can not only avoid subtle Declaration.yaml conversions but also decouple the generation of python schema, python to c++ binding and c++ function call. The ultimate state will be like the following diagram: ``` +-------------------+ +-------------------------------------+ +-------> | Python Signatures | --> | PythonArgParser Schema | | +-------------------+ +-------------------------------------+ | | . | | . | | . +------------------+ | +-------------------------------------+ | Native Functions | +-------> | PythonArgParser -> Cpp Args Binding | +------------------+ | +-------------------------------------+ | | . | | . | | . | +-------------------+ +-------------------------------------+ +-------> | Cpp Signatures | --> | Cpp Function Dispatch | +-------------------+ +-------------------------------------+ ``` This PR has migrated the core binding logic from tools/autograd/gen_python_functions.py to tools/codegen/api/python.py. It produces the byte-for-byte same results (tested with #46243). Will migrate the rest of gen_python_functions.py in subsequent PRs. Test Plan: Imported from OSS Reviewed By: bhosmer Differential Revision: D24388874 Pulled By: ljk53 fbshipit-source-id: f88b6df4e917cf90d868a2bbae2d5ffb680d1841
56 lines
1.6 KiB
Bash
Executable File
56 lines
1.6 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
set -ex
|
|
|
|
ignore_warning() {
|
|
# Invert match to filter out $1.
|
|
set +e
|
|
grep -v "$1" doxygen-log.txt > temp.txt
|
|
set -e
|
|
mv temp.txt doxygen-log.txt
|
|
}
|
|
|
|
command -v doxygen >/dev/null 2>&1 || { echo >&2 "doxygen is not supported. Aborting."; exit 1; }
|
|
|
|
pushd "$(dirname "$0")/../../.."
|
|
|
|
cp torch/_utils_internal.py tools/shared
|
|
|
|
python -m tools.codegen.gen
|
|
|
|
python tools/setup_helpers/generate_code.py \
|
|
--declarations-path build/aten/src/ATen/Declarations.yaml \
|
|
--native-functions-path aten/src/ATen/native/native_functions.yaml \
|
|
--nn-path aten/src
|
|
|
|
popd
|
|
|
|
# Run doxygen and log all output.
|
|
doxygen "$(dirname $0)" 2> original-doxygen-log.txt
|
|
cp original-doxygen-log.txt doxygen-log.txt
|
|
|
|
# Uncomment this if you need it for debugging; we're not printing this
|
|
# by default because it is confusing.
|
|
# echo "Original output"
|
|
# cat original-doxygen-log.txt
|
|
|
|
# Filter out some warnings.
|
|
ignore_warning "warning: no uniquely matching class member found for"
|
|
ignore_warning "warning: explicit link request to 'Item' could not be resolved"
|
|
ignore_warning "warning: Included by graph for 'types.h' not generated, too many nodes"
|
|
|
|
# Count the number of remaining warnings.
|
|
warnings="$(grep 'warning:' doxygen-log.txt | wc -l)"
|
|
|
|
echo "Treating all remaining warnings as errors"
|
|
|
|
if [[ "$warnings" -ne "0" ]]; then
|
|
echo "Failing Doxygen test because the following warnings were treated fatally:"
|
|
cat doxygen-log.txt
|
|
echo "Please fix these warnings. To run this test locally, use docs/cpp/source/check-doxygen.sh"
|
|
rm -f doxygen-log.txt original-doxygen-log.txt
|
|
exit 1
|
|
fi
|
|
|
|
rm -f doxygen-log.txt original-doxygen-log.txt
|