gracefully handle tokenize.TokenError in funcname parser. Adds support for non-Python source (#148737)

This change allows defining python functions in non-python source and having them be able to compiled by torch.compile. The existing implementation already returns None for the case where the file couldn't be read, so returning None (by making an empty funcname cache) makes sense for the case of non-python source code too.

Example [basilisp](https://github.com/basilisp-lang/basilisp):
```clojure
(import torch)
(import [torch.nn.functional :as F])
(torch/rand 10)

(defn f {:decorators [torch/compile]} [x]
  (* (F/relu x) x))

(f (-> (torch/randn 100)
       (.cuda)))
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/148737
Approved by: https://github.com/williamwen42
This commit is contained in:
cat-state
2025-03-11 19:49:24 +00:00
committed by PyTorch MergeBot
parent 73c8068cf8
commit abcec55532

View File

@ -31,7 +31,7 @@ def _add_file(filename: str) -> None:
try:
with tokenize.open(filename) as f:
tokens = list(tokenize.generate_tokens(f.readline))
except OSError:
except (OSError, tokenize.TokenError):
cache[filename] = {}
return