Add a lint rule for torch/csrc/util/pybind.h include (#82552)

We define specializations for pybind11 defined templates
(in particular, PYBIND11_DECLARE_HOLDER_TYPE) and consequently
it is important that these specializations *always* be #include'd
when making use of pybind11 templates whose behavior depends on
these specializations, otherwise we can cause an ODR violation.

The easiest way to ensure that all the specializations are always
loaded is to designate a header (in this case, torch/csrc/util/pybind.h)
that ensures the specializations are defined, and then add a lint
to ensure this header is included whenever pybind11 headers are
included.

The existing grep linter didn't have enough knobs to do this
conveniently, so I added some features.  I'm open to suggestions
for how to structure the features better.  The main changes:

- Added an --allowlist-pattern flag, which turns off the grep lint
  if some other line exists.  This is used to stop the grep
  lint from complaining about pybind11 includes if the util
  include already exists.

- Added --match-first-only flag, which lets grep only match against
  the first matching line.  This is because, even if there are multiple
  includes that are problematic, I only need to fix one of them.
  We don't /really/ need this, but when I was running lintrunner -a
  to fixup the preexisting codebase it was annoying without this,
  as the lintrunner overall driver fails if there are multiple edits
  on the same file.

I excluded any files that didn't otherwise have a dependency on
torch/ATen, this was mostly caffe2 and the valgrind wrapper compat
bindings.

Note the grep replacement is kind of crappy, but clang-tidy lint
cleaned it up in most cases.

See also https://github.com/pybind/pybind11/issues/4099

Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82552
Approved by: https://github.com/albanD

torch/csrc/utils.cpp

@@ -4,6 +4,7 @@
 #include <torch/csrc/python_headers.h>
 #include <torch/csrc/utils/invalid_arguments.h>
 #include <torch/csrc/utils/python_strings.h>
+#include <torch/csrc/utils/python_tuples.h>
 
 #include <torch/csrc/Export.h>
 
@@ -294,3 +295,58 @@ error:
 
 } // namespace gdb
 } // namespace torch
+
+namespace pybind11 {
+namespace detail {
+
+bool type_caster<at::Tensor>::load(handle src, bool) {
+  PyObject* obj = src.ptr();
+  if (THPVariable_Check(obj)) {
+    value = THPVariable_Unpack(obj);
+    return true;
+  }
+  return false;
+}
+
+handle type_caster<at::Tensor>::cast(
+    const at::Tensor& src,
+    return_value_policy /* policy */,
+    handle /* parent */) {
+  return handle(THPVariable_Wrap(src));
+}
+
+bool type_caster<at::IntArrayRef>::load(handle src, bool) {
+  PyObject* source = src.ptr();
+  auto tuple = PyTuple_Check(source);
+  if (tuple || PyList_Check(source)) {
+    // NOLINTNEXTLINE(bugprone-branch-clone)
+    const auto size =
+        tuple ? PyTuple_GET_SIZE(source) : PyList_GET_SIZE(source);
+    v_value.resize(size);
+    for (const auto idx : c10::irange(size)) {
+      PyObject* obj =
+          tuple ? PyTuple_GET_ITEM(source, idx) : PyList_GET_ITEM(source, idx);
+      if (THPVariable_Check(obj)) {
+        v_value[idx] = THPVariable_Unpack(obj).item<int64_t>();
+      } else if (PyLong_Check(obj)) {
+        // use THPUtils_unpackLong after it is safe to include
+        // python_numbers.h
+        v_value[idx] = THPUtils_unpackLong(obj);
+      } else {
+        return false;
+      }
+    }
+    value = v_value;
+    return true;
+  }
+  return false;
+}
+handle type_caster<at::IntArrayRef>::cast(
+    at::IntArrayRef src,
+    return_value_policy /* policy */,
+    handle /* parent */) {
+  return handle(THPUtils_packInt64Array(src.size(), src.data()));
+}
+
+} // namespace detail
+} // namespace pybind11