This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
This PR introduces inductor lite mode for opt-in optimizations and numeric correctness guarantees.
Different from default mode that applies all possible fusions, lite mode gives the control back to user and provides guarantee on numeric correctness. Specifically, this mode:
- **Fallback by Default**: Fallback for ALL nodes by default, unless users explicitly mark node for inductor fusion.
- **Selective Decomposition**: Skip decomposition for all nodes except for user marked nodes.
- **Regional inductor compile**
- Skip dead code elimination
- Skip buffer reues
- Skip reorder passes, such as reorder for peak memory, reorder for compute comm overlap, and reorder_for_reducing_graph_partitions.
- Skip all pre-grad, joint-graph, and post-grad passes.
## Example: Flex Attention
```python
import torch
import torch.fx.traceback as fx_traceback
from torch.nn.attention.flex_attention import create_block_mask, flex_attention
def _squared(score, b, h, m, n):
return score * score
def mask_mod(b, h, q, k):
return q >= 0
a, b = 12, 64
block_mask = create_block_mask(mask_mod, None, None, a * b, a * b, device="cuda")
def fn(x):
x = torch.sin(x)
with fx_traceback.annotate({"compile_with_inductor": 0}):
x = flex_attention(x, x, x, block_mask=block_mask, score_mod=_squared)
return torch.cos(x)
x = torch.randn(1, 1, a * b, b, dtype=torch.bfloat16, device="cuda", requires_grad=True)
opt_fn = torch.compile(fn, mode="lite", fullgraph=True,)
opt_fn(x)
```
[code diff](https://www.internalfb.com/intern/diffing/?paste_number=2027441476)
[default mode tlp](https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/.tmpYAzDxX/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000) vs [lite mode tlp](https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/.tmpnnuh1W/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000)
## Numerics
Inductor lite mode provides bitwise equivalence with `aot_eager` backend on torchtitan llama3-8b and DeepSeek v3. https://github.com/pytorch/torchtitan/pull/2005close: #167012
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167115
Approved by: https://github.com/ezyang
This adds a workflow to run full set of UTs on default and distributed configs on ROCm MI3xx CI runners, to _eventually_ assess if the CI capacity can handle the PR-based workload for trunk.yml. The plan was to keep this workflow in unstable as we test out this new CI capacity, so it wouldn't impact PR merges. However, since upstream maintainers have indicated that, as of today, even unstable workflows will block PR merges, we are going with branch push-based triggers to at least pipeclean this workflow on the new CI capacity.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167587
Approved by: https://github.com/jeffdaily
The seq_nr doesn't always increment for gradient accumulation nodes, and they might be copying annotation from forward nodes.
I'm just going to skip copying the custom meta for any gradient accumulation nodes and give them a special tag e.g. node.meta["is_gradient_acc"]=True
Example repro for deepseek torchtitan (without using DTensor): https://gist.github.com/yushangdi/aae13ea382732f31d0fdfb3ffeda12c8
(side note: if you want some more hints on these gradient acc node: 1) they have torch.ops.aten.add.Tensor op, not add.default. 2) they have the highest seq_nr(s) )
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167572
Approved by: https://github.com/mlazos
Summary:
Added a new flag called "use_legacy_dynamo_graph_capture" which defaults to True and only False with the updated test_strict_export_v2.py
In addiotion to this flag, we also use legacy tracer when the following features are used:
1. dynamic shape
2. preserve module call signature
3. retracing.
4. draft mode.
Test Plan:
test_strict_export_v2.py
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167236
Approved by: https://github.com/tugsbayasgalan
# Summary
See title ;)
## Design
Currently once you install there is no going back in the same python process, this need not be the case, cc @mikaylagawarecki's work on being able to grab original impl. I'll leave for follow up.
Okay I added an open reg, but I really want the backends to be found so some weird typing but we get
<img width="523" height="197" alt="Screenshot 2025-11-07 at 3 30 32 PM" src="https://github.com/user-attachments/assets/586de943-bbed-40cf-abd1-131f747a4cf1" />
## Overheads:
<img width="799" height="735" alt="Screenshot 2025-11-07 at 2 35 04 PM" src="https://github.com/user-attachments/assets/f9217f31-3e42-4816-8fb3-29ea8b49d735" />
First call to forward -> majority of time is spent in jit for FA
First call to backward, 3sec interestingly it doesn't appear that with_stack gets events in the backwards loop @albanD is this expected?
<img width="948" height="385" alt="Screenshot 2025-11-07 at 2 35 50 PM" src="https://github.com/user-attachments/assets/a40bacd0-3fb0-4bd8-b33e-bec8fb3f36c0" />
Getting form Pt op to impl is about 43 us which is dwarfed by other cpu overheads
<img width="1227" height="649" alt="Screenshot 2025-11-07 at 2 37 41 PM" src="https://github.com/user-attachments/assets/51da0615-facd-41e1-a6e2-fb7778079ab6" />
Just invoking the jit object from cutesl is 100s of us
<img width="545" height="414" alt="Screenshot 2025-11-07 at 2 38 19 PM" src="https://github.com/user-attachments/assets/d20345a0-6c47-4dcb-892f-9ef9894a1cf5" />
### Example usage
```Py
#!/usr/bin/env python3
"""Minimal FA4 smoke test for scaled dot product attention."""
from __future__ import annotations
import sys
from jsonargparse import CLI
import torch
import torch.nn.functional as F
from torch.nn.attention import (
install_flash_attention_impl,
sdpa_kernel,
SDPBackend,
)
def _map_dtype(kind: str) -> torch.dtype:
return torch.bfloat16 if kind == "bf16" else torch.float16
# To infinity and beyond
install_flash_attention_impl("FA4")
@sdpa_kernel([SDPBackend.FLASH_ATTENTION])
def main(
module_path: str = "flash_attn.cute.interface",
batch: int = 4,
seq: int = 81292,
heads: int = 16,
head_dim: int = 128,
device: int = 0,
dtype: str = "bf16"
) -> None:
if not torch.cuda.is_available():
sys.exit("CUDA is required for FA4 smoke testing")
torch.cuda.set_device(device)
dtype = _map_dtype(dtype)
generator = torch.Generator(device="cuda").manual_seed(0)
q = torch.randn(
batch,
heads,
seq,
head_dim,
device="cuda",
dtype=dtype,
requires_grad=True,
generator=generator,
)
k = torch.randn(
batch,
heads,
seq,
head_dim,
device="cuda",
dtype=dtype,
requires_grad=True,
generator=generator,
)
v = torch.randn(
batch,
heads,
seq,
head_dim,
device="cuda",
dtype=dtype,
requires_grad=True,
generator=generator,
)
from transformer_nuggets.utils.benchmark import profiler
with profiler("sdpa_FA4", with_stack=False):
for _ in range(3):
out = F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False)
loss = out.real.sum()
loss.backward()
print("Scaled dot product attention output norm:", out.norm().item())
print("dq norm:", q.grad.norm().item())
if __name__ == "__main__":
CLI(main)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167348
Approved by: https://github.com/albanD, https://github.com/malfet
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
This was intersting to debug:
(1) Python 3.14 ships with a lazy way for retrieving annotations. The annotations field can be a callable that lazily evaluates it
(2) On the dynamo side, `SET_FUNCTION_ATTRIBUTE` needs to handle an extra flag value (0x10)
(3) The decorator `functools.wraps` used extensively in the codebase (e.g. `make_fx`, `functionalize`) doesn't copy the `__annotations__` attribute by default. To correctly retrieve an annotatin, we need to walk on the chain of `__wrapped__` objects and retrieve the attribute from the first function. Fortunately, there are functions on the stdlib to do this.
Fixes:
```
'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_fx_out_op_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_fx_transpose_simple_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_optional_tensorlist2_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_fx_multi_out_op_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_fx_reapply_views_simple_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_fx_simple_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_nonfunctional_output_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_opt_tensor_list_cpu', 'test/functorch/test_eager_transforms.py::TestFunctionalizeCPU::test_functionalize_optional_tensorlist1_cpu'
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167573
Approved by: https://github.com/williamwen42
As titled, this PR adds manual bucketing pass to SimpleFSDP. Users will need to parse FQNs they wanted to bucket together using `module_bucket_plans`. Then, `_manual_bucket_collectives` will get the node of the subgraphs correspond to each `bucket_module`, and bucket bucketable (FSDP-style) AG/RS together. `_manual_reorder_graph` reorders them for overlapping.
For detailed performance, see this torchtitan PR: https://github.com/pytorch/torchtitan/pull/1881.
There are a few todo items isted in torchtitan PR. Let's start with this PR that implements FSDP+TP+llama3 manual bucketing. I will fix/add the rest in follow up PRs.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/165487
Approved by: https://github.com/ezyang
# Summary
See title ;)
## Design
Currently once you install there is no going back in the same python process, this need not be the case, cc @mikaylagawarecki's work on being able to grab original impl. I'll leave for follow up.
Okay I added an open reg, but I really want the backends to be found so some weird typing but we get
<img width="523" height="197" alt="Screenshot 2025-11-07 at 3 30 32 PM" src="https://github.com/user-attachments/assets/586de943-bbed-40cf-abd1-131f747a4cf1" />
## Overheads:
<img width="799" height="735" alt="Screenshot 2025-11-07 at 2 35 04 PM" src="https://github.com/user-attachments/assets/f9217f31-3e42-4816-8fb3-29ea8b49d735" />
First call to forward -> majority of time is spent in jit for FA
First call to backward, 3sec interestingly it doesn't appear that with_stack gets events in the backwards loop @albanD is this expected?
<img width="948" height="385" alt="Screenshot 2025-11-07 at 2 35 50 PM" src="https://github.com/user-attachments/assets/a40bacd0-3fb0-4bd8-b33e-bec8fb3f36c0" />
Getting form Pt op to impl is about 43 us which is dwarfed by other cpu overheads
<img width="1227" height="649" alt="Screenshot 2025-11-07 at 2 37 41 PM" src="https://github.com/user-attachments/assets/51da0615-facd-41e1-a6e2-fb7778079ab6" />
Just invoking the jit object from cutesl is 100s of us
<img width="545" height="414" alt="Screenshot 2025-11-07 at 2 38 19 PM" src="https://github.com/user-attachments/assets/d20345a0-6c47-4dcb-892f-9ef9894a1cf5" />
### Example usage
```Py
#!/usr/bin/env python3
"""Minimal FA4 smoke test for scaled dot product attention."""
from __future__ import annotations
import sys
from jsonargparse import CLI
import torch
import torch.nn.functional as F
from torch.nn.attention import (
install_flash_attention_impl,
sdpa_kernel,
SDPBackend,
)
def _map_dtype(kind: str) -> torch.dtype:
return torch.bfloat16 if kind == "bf16" else torch.float16
# To infinity and beyond
install_flash_attention_impl("FA4")
@sdpa_kernel([SDPBackend.FLASH_ATTENTION])
def main(
module_path: str = "flash_attn.cute.interface",
batch: int = 4,
seq: int = 81292,
heads: int = 16,
head_dim: int = 128,
device: int = 0,
dtype: str = "bf16"
) -> None:
if not torch.cuda.is_available():
sys.exit("CUDA is required for FA4 smoke testing")
torch.cuda.set_device(device)
dtype = _map_dtype(dtype)
generator = torch.Generator(device="cuda").manual_seed(0)
q = torch.randn(
batch,
heads,
seq,
head_dim,
device="cuda",
dtype=dtype,
requires_grad=True,
generator=generator,
)
k = torch.randn(
batch,
heads,
seq,
head_dim,
device="cuda",
dtype=dtype,
requires_grad=True,
generator=generator,
)
v = torch.randn(
batch,
heads,
seq,
head_dim,
device="cuda",
dtype=dtype,
requires_grad=True,
generator=generator,
)
from transformer_nuggets.utils.benchmark import profiler
with profiler("sdpa_FA4", with_stack=False):
for _ in range(3):
out = F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False)
loss = out.real.sum()
loss.backward()
print("Scaled dot product attention output norm:", out.norm().item())
print("dq norm:", q.grad.norm().item())
if __name__ == "__main__":
CLI(main)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167348
Approved by: https://github.com/albanD
Fixes#164663
## Issue
The torch model with multiple layers that is wrapped with fsdp2 registers pre and post forward hooks in a group using `_MultiHandler`. This becomes an issue during the context manager of the tracker where the hooks are reset and replaced. The hooks are all using the same fsdp state pointer so one reset will reset all. So when the output layer was modified with a new pre and post forward hook it would delete the previous layer's initialization causing `KeyError` for the Norm layer as it is nonexistent.
## The Fix
Check to see if there are multiple `_MultiHandler` objects and `RemoveHandler` objects and only execute the remove hook once.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/165662
Approved by: https://github.com/sanketpurandare
In pip 25.3 and newer, use of --no-use-pep517 has been removed (https://pip.pypa.io/en/stable/news/). In builds with pip 25.2, a warning message notes:
> DEPRECATION: Building 'torchvision' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. A possible replacement is to use the standardized build interface by setting the `--use-pep517` option, (possibly combined with `--no-build-isolation`), or adding a `pyproject.toml` file to the source tree of 'torchvision'. Discussion can be found at https://github.com/pypa/pip/issues/6334
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167096
Approved by: https://github.com/atalman
* Trigger this workflow on every completed run of `docker-builds.yml`
* Uses `ubuntu-latest` for downloading artifacts from `docker-build` workflow run
* Uses `linux.rocm.gfx942.docker-cache` to cache docker images as tarballs for MI3xx CI
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167554
Approved by: https://github.com/jeffdaily
Resolves issues running the dynamo cpython math.fma tests.
Though math.fma is enabled to perform a multiply add in dynamo, torch.addcmul is currently used which doesn't guarantee the user request for fma. It was decided to not use inductor fma prim as it would break the contract of using aten/core ir in dynamo output - otherwise export=True may have issues.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167217
Approved by: https://github.com/guilhermeleobas
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
Logging propagate_op_sharding_non_cached is a compromise between
- logging in DTensor.__init__ to catch ALL DTensor usage
- sparing the overhead in a latency-senstitive region like
DTensor.__init__
- and 'real' DTensor usage should incur at least one call to sharding
propagation
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167374
Approved by: https://github.com/zpcore
## Problems
When exporting a compiled model with nested input like below
```python
import torch
from torch.export import export, Dim
def test_export_compiled_model_with_nested_dynamic_shapes():
"""Test exporting a compiled model with nested dict inputs and dynamic shapes."""
print("Running test_export_compiled_model_with_nested_dynamic_shapes...")
class M(torch.nn.Module):
def forward(self, data_batch):
return data_batch["a1"] + data_batch["a2"]
m = M()
compiled_m = torch.compile(m)
example_args = ({
"a1": torch.ones(3, 3),
"a2": torch.ones(3, 3),
},)
dynamic_shapes = ({
"a1": {0: Dim.DYNAMIC},
"a2": {0: Dim.DYNAMIC},
},)
try:
ep = export(compiled_m, example_args, dynamic_shapes=dynamic_shapes, strict=True)
gm = ep.module()
result_exported = gm(*example_args)
result_compiled = compiled_m(*example_args)
assert torch.allclose(result_exported, result_compiled), "Results don't match!"
print("✓ test_export_compiled_model_with_nested_dynamic_shapes PASSED")
return True
except Exception as e:
print(f"✗ test_export_compiled_model_with_nested_dynamic_shapes FAILED")
print(f"Error: {e}")
import traceback
traceback.print_exc()
return False
def test_export_compiled_model_with_kwargs_dynamic_shapes():
"""Test exporting a compiled model with kwargs and dynamic shapes."""
print("\nRunning test_export_compiled_model_with_kwargs_dynamic_shapes...")
class M(torch.nn.Module):
def forward(self, a1, a2):
return a1 + a2
m = M()
compiled_m = torch.compile(m)
example_args = ()
example_kwargs = {
"a1": torch.ones(3, 3),
"a2": torch.ones(3, 3),
}
dynamic_shapes = {
"a1": {0: Dim.DYNAMIC},
"a2": {0: Dim.DYNAMIC},
}
try:
ep = export(compiled_m, example_args, kwargs=example_kwargs, dynamic_shapes=dynamic_shapes, strict=True)
gm = ep.module()
result_exported = gm(**example_kwargs)
result_compiled = compiled_m(**example_kwargs)
assert torch.allclose(result_exported, result_compiled), "Results don't match!"
print("✓ test_export_compiled_model_with_kwargs_dynamic_shapes PASSED")
return True
except Exception as e:
print(f"✗ test_export_compiled_model_with_kwargs_dynamic_shapes FAILED")
print(f"Error: {e}")
import traceback
traceback.print_exc()
return False
if __name__ == "__main__":
print("Testing export of compiled models with dynamic shapes\n")
print("=" * 70)
results = []
results.append(test_export_compiled_model_with_nested_dynamic_shapes())
results.append(test_export_compiled_model_with_kwargs_dynamic_shapes())
print("\n" + "=" * 70)
print(f"\nResults: {sum(results)}/{len(results)} tests passed")
if all(results):
print("✓ All tests passed!")
else:
print("✗ Some tests failed")
exit(1)
```
It will report
```
======================================================================
Running test_export_compiled_model_with_nested_dynamic_shapes...
✗ test_export_compiled_model_with_nested_dynamic_shapes FAILED
Error: Detected mismatch between the structure of `inputs` and `dynamic_shapes`: `inputs[0]` is a <class 'tuple'>, but `dynamic_shapes[0]` is a <class 'dict'>
For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#dynamic-shapes-validation
The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
Traceback (most recent call last):
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 614, in _tree_map_with_path
return tree_map_with_path(f, tree, *dynamic_shapes, is_leaf=is_leaf)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 2055, in tree_map_with_path
all_keypath_leaves = keypath_leaves + [treespec.flatten_up_to(r) for r in rests]
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 2055, in <listcomp>
all_keypath_leaves = keypath_leaves + [treespec.flatten_up_to(r) for r in rests]
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 1188, in flatten_up_to
helper(self, tree, subtrees)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 1185, in helper
helper(subspec, subtree, subtrees)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 1141, in helper
raise ValueError(
ValueError: Node type mismatch; expected <class 'tuple'>, but got <class 'dict'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/chzhu/infinitrain/test_exprot.py", line 25, in test_export_compiled_model_with_nested_dynamic_shapes
ep = export(compiled_m, example_args, dynamic_shapes=dynamic_shapes, strict=True)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/__init__.py", line 311, in export
raise e
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/__init__.py", line 277, in export
return _export(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1163, in wrapper
raise e
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1129, in wrapper
ep = fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/exported_program.py", line 124, in wrapper
return fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 2255, in _export
ep = _export_for_training(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1163, in wrapper
raise e
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1129, in wrapper
ep = fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/exported_program.py", line 124, in wrapper
return fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
export_artifact = export_func(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1415, in _strict_export
gm_torch_level = _export_to_torch_ir(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 785, in _export_to_torch_ir
_check_dynamic_shapes(combined_args, dynamic_shapes)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 1031, in _check_dynamic_shapes
_tree_map_with_path(check_shape, combined_args, dynamic_shapes, tree_name="inputs")
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 686, in _tree_map_with_path
_compare(tree_spec, other_tree_spec, [])
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 677, in _compare
_compare(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 652, in _compare
raise_mismatch_error(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 634, in raise_mismatch_error
raise UserError(
torch._dynamo.exc.UserError: Detected mismatch between the structure of `inputs` and `dynamic_shapes`: `inputs[0]` is a <class 'tuple'>, but `dynamic_shapes[0]` is a <class 'dict'>
For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#dynamic-shapes-validation
The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
Running test_export_compiled_model_with_kwargs_dynamic_shapes...
✗ test_export_compiled_model_with_kwargs_dynamic_shapes FAILED
Error: When `dynamic_shapes` is specified as a dict, its top-level keys must be the arg names ['kwargs'] of `inputs`, but here they are ['a1', 'a2']. Since here `inputs` is a list/tuple enclosing a single dict, maybe you just forgot to enclose `dynamic_shapes` in a list/tuple?
For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#dynamic-shapes-validation
The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
Traceback (most recent call last):
File "/home/chzhu/infinitrain/test_exprot.py", line 62, in test_export_compiled_model_with_kwargs_dynamic_shapes
ep = export(compiled_m, example_args, kwargs=example_kwargs, dynamic_shapes=dynamic_shapes, strict=True)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/__init__.py", line 311, in export
raise e
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/__init__.py", line 277, in export
return _export(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1163, in wrapper
raise e
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1129, in wrapper
ep = fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/exported_program.py", line 124, in wrapper
return fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 2255, in _export
ep = _export_for_training(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1163, in wrapper
raise e
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1129, in wrapper
ep = fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/exported_program.py", line 124, in wrapper
return fn(*args, **kwargs)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
export_artifact = export_func(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 1415, in _strict_export
gm_torch_level = _export_to_torch_ir(
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/_trace.py", line 785, in _export_to_torch_ir
_check_dynamic_shapes(combined_args, dynamic_shapes)
File "/home/chzhu/infinitrain/build/infinitrain/environments/development-venv/lib/python3.10/site-packages/torch/export/dynamic_shapes.py", line 1007, in _check_dynamic_shapes
raise UserError(
torch._dynamo.exc.UserError: When `dynamic_shapes` is specified as a dict, its top-level keys must be the arg names ['kwargs'] of `inputs`, but here they are ['a1', 'a2']. Since here `inputs` is a list/tuple enclosing a single dict, maybe you just forgot to enclose `dynamic_shapes` in a list/tuple?
For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#dynamic-shapes-validation
The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
======================================================================
```
## Torch Version
(reproducible nightly version)
## Other Behavior
The model can export regularly when we test without compiling the model
```python
import torch
from torch.export import export, Dim
def test_export_compiled_model_with_nested_dynamic_shapes():
"""Test exporting a compiled model with nested dict inputs and dynamic shapes."""
print("Running test_export_compiled_model_with_nested_dynamic_shapes...")
class M(torch.nn.Module):
def forward(self, data_batch):
return data_batch["a1"] + data_batch["a2"]
m = M()
example_args = ({
"a1": torch.ones(3, 3),
"a2": torch.ones(3, 3),
},)
dynamic_shapes = ({
"a1": {0: Dim.DYNAMIC},
"a2": {0: Dim.DYNAMIC},
},)
try:
ep = export(m, example_args, dynamic_shapes=dynamic_shapes, strict=True)
gm = ep.module()
result_exported = gm(*example_args)
result_compiled = m(*example_args)
assert torch.allclose(result_exported, result_compiled), "Results don't match!"
print("✓ test_export_compiled_model_with_nested_dynamic_shapes PASSED")
return True
except Exception as e:
print(f"✗ test_export_compiled_model_with_nested_dynamic_shapes FAILED")
print(f"Error: {e}")
import traceback
traceback.print_exc()
return False
def test_export_compiled_model_with_kwargs_dynamic_shapes():
"""Test exporting a compiled model with kwargs and dynamic shapes."""
print("\nRunning test_export_compiled_model_with_kwargs_dynamic_shapes...")
class M(torch.nn.Module):
def forward(self, a1, a2):
return a1 + a2
m = M()
example_args = ()
example_kwargs = {
"a1": torch.ones(3, 3),
"a2": torch.ones(3, 3),
}
dynamic_shapes = {
"a1": {0: Dim.DYNAMIC},
"a2": {0: Dim.DYNAMIC},
}
try:
ep = export(m, example_args, kwargs=example_kwargs, dynamic_shapes=dynamic_shapes, strict=True)
gm = ep.module()
result_exported = gm(**example_kwargs)
result_compiled = m(**example_kwargs)
assert torch.allclose(result_exported, result_compiled), "Results don't match!"
print("✓ test_export_compiled_model_with_kwargs_dynamic_shapes PASSED")
return True
except Exception as e:
print(f"✗ test_export_compiled_model_with_kwargs_dynamic_shapes FAILED")
print(f"Error: {e}")
import traceback
traceback.print_exc()
return False
if __name__ == "__main__":
print("Testing export of compiled models with dynamic shapes\n")
print("=" * 70)
results = []
results.append(test_export_compiled_model_with_nested_dynamic_shapes())
results.append(test_export_compiled_model_with_kwargs_dynamic_shapes())
print("\n" + "=" * 70)
print(f"\nResults: {sum(results)}/{len(results)} tests passed")
if all(results):
print("✓ All tests passed!")
else:
print("✗ Some tests failed")
exit(1)
```
## Root Cause
This is because of a side effect of torch.compile(model). When the model is being compiled, the input signature will become (*args, **kwargs) automatically. In the above example, the `data_batch` will be added into `args` in combined_args [here](dc011d3203/torch/export/dynamic_shapes.py (L720)), and it will look like
```
{'args': ({'a1': tensor([[1., 1., 1.]... 1., 1.]]), 'a2': tensor([[1., 1., 1.]... 1., 1.]])},)}
```
Without the compiling, the combined args will look like
```
{'data_batch': {'a1': tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]]), 'a2': tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])}}
```
Thus causing the mismatch when we use treemap to match the dynamic shape with the input argos
The error is also reproducible when we setup kwargs as example argos (see the 2nd test above)
## Fix
Proposed fix: In [_combine_args](dc011d3203/torch/export/dynamic_shapes.py (L720)) we explicitly flatten out the kwargs and args into combined args.
## Side Effects
There are 2 existing tests that assume this behavior and
1. add `args` explicitly to dynamic shapes
2. wrap args into nested format in dynamic_shape
I have modified those test to make args and dynamic_shapes to be in consistent format.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166358
Approved by: https://github.com/angelayi
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
Summary: When cudagraphs-parittion is enabled, we have seen an increase of cold compile time in the vllm benchmark (see https://github.com/vllm-project/vllm/issues/27080). After some profiling, we found Triton compilation time increased the most. Further investigation reveals it was caused by duplicated Triton kernels not being shared among different partitions. This PR fixes the issue by reusing the Trition kernel source code cache at the top-level PythonWrapperCodegen.
In theory we could further reduce the compilation time by completely skipping compiling duplicated partitions. That can come as a furture improvement.
Some vllm benchmarking data,
```
VLLM_USE_STANDALONE_COMPILE=0 VLLM_DISABLE_COMPILE_CACHE=1 vllm bench latency -O.cudagraph_mode=PIECEWISE -O.use_inductor_graph_partition=True --model meta-llama/Meta-Llama-3.1-8
```
Before:
```
torch.compile takes 69.18 s in total
```
After:
```
torch.compile takes 26.81 s in total
```
As a refrence, this is the compile time when turning off inductor graph partition. Looks like we still have some gap to close.
```
VLLM_USE_STANDALONE_COMPILE=0 VLLM_DISABLE_COMPILE_CACHE=1 vllm bench latency -O.cudagraph_mode=PIECEWISE -O.use_inductor_graph_partition=False --model meta-llama/Meta-Llama-3.1-8B
torch.compile takes 19.41 s in total
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167132
Approved by: https://github.com/eellison
ghstack dependencies: #167131
Summary: When cudagraphs partition and autotune_at_compile_time are enabled, currently each subgraph will generate its own auto-tuning code block and run them once by one. This PR improves it by only generating one auto-tuning code block at the main graph level and execute it once time to auto-tune all the kernels.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167131
Approved by: https://github.com/eellison
Avoids data-dependent errors for out-of-bounds & redundant slice checks.
The sharding logic that immediately depends on this only checks for redundant slices, and is saying: "it's safe to reuse the input placements if a) the slicing dimension isn't sharded, or b) the slice is redundant, so just pretend this op didn't happen".
This has a slight effect on output placements, when a slice is performed on a shared dim, and dynamic shapes are involved (size/start/end/step). Now if the slice isn't obviously redundant, we won't immediately consider the input placements valid (even if they could be for very particular runtime shapes), and select strategies valid for the general case - in this case I guess unsharding the slicing dim.
For backed symbols, we could choose to recompile when the redundant case is hit, by switching to `guard_or_false`, but it's not obvious how desirable this is.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166990
Approved by: https://github.com/laithsakka
This PR enables XPU devices in test_analysis.py.
For performance reason, it skips some slow tests, so a full scope should be enabled by using:
```
export PYTORCH_TEST_WTH_SLOW=1
```
**PR Stack:**
- https://github.com/pytorch/pytorch/pull/166840 : This PR enables the tests, ignores the tests that failed
- https://github.com/pytorch/pytorch/pull/166839 : This fixed the bug and enable the full tests for xpu
**Some skipped test time:**
```
test_augment_trace_against_flop_counter_maxat0_xpu_float16 [49.0863s]
test_augment_trace_against_flop_counter_maxat0_xpu_float32 [18.2268s]
test_augment_trace_against_flop_counter_maxat1_xpu_float16 [85.6549s]
test_augment_trace_against_flop_counter_maxat1_xpu_float32 [329.0832s]
test_augment_trace_against_flop_counter_maxat2_xpu_float16 [24.4825s]
test_augment_trace_against_flop_counter_maxat2_xpu_float32 [19.0688s]
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166840
Approved by: https://github.com/guangyey, https://github.com/jansel
Fixed a command injection vulnerability in PreCompiled Header (PCH) compilation where extra_cflags were passed to subprocess with shell=True, allowing arbitrary command execution through malicious compiler flags.
Changed subprocess.check_output(pch_cmd, shell=True) to use shlex.split() to safely parse the command without shell interpretation. This prevents shell metacharacters (;, |, &, etc.) in extra_cflags from being executed as shell commands.
Added test case test_pch_command_injection that verifies:
1. PCH compilation attempts with malicious payloads in extra_cflags
2. Shell commands embedded in flags are not executed
3. Exploit file is not created, proving no shell execution occurred
Note: On RHEL/Fedora and other systems with versioned GCC compilers, the test depends on https://github.com/pytorch/pytorch/pull/167501 being merged first, otherwise the test will be skipped due to GCC detection issues.
Fixes#167480
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167502
Approved by: https://github.com/malfet
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
Allocate ErrorMessages buffer associated with MPSStream and introduce `c10:🤘:report_error` method(and `TORCH_REPORT_ERROR` macro), that can be used to preserve up to `c10:🤘:error_message_count` messages
Add test that detects those
As results attempt to run something like
```python
import torch
x=torch.rand(10, 1, 10, device='mps')
y=x[:, [1]]
torch.mps.synchonize()
```
will raise `torch.AcceleratorError: index 1 is out of bounds for dimension 0 with size 1`
Fixes https://github.com/pytorch/pytorch/issues/111669
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166615
Approved by: https://github.com/manuelcandales, https://github.com/dcci
ghstack dependencies: #167444, #167445
The function was only returning True for compilers named 'c++', but failed to detect g++, gcc, g++-13, g++-14, etc. This fixes the detection to work for any GCC variant by checking for both COLLECT_GCC and 'gcc version' in the compiler output.
The previous implementation used os.path.basename() on the resolved compiler path and only checked if it exactly matched 'c++'. This caused false negatives for versioned GCC installations and direct g++ usage.
The fix simplifies the logic: if both COLLECT_GCC is present in the output (indicating GCC toolchain) and 'gcc version' appears in the version string, it's GCC.
Fixes#167499
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167501
Approved by: https://github.com/ezyang, https://github.com/malfet
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
Summary:
When PT2 sees an operation like `torch.empty_strided(8, device="cuda")` it returns a fake tensor on a specific device index, such as `"cuda:0"`. This is implemented via a list of multi-index devices in the fake tensor constructor. If a device supports indices but is not in this list, we can hit fake tensor propagation errors as `"cuda"` is not considered to be the same device as `"cuda:0"`.
This PR adds the `"mtia"` device to this list, to resolve some fake tensor propagation errors we're seeing with the full Inductor backend. (Internal task: T243176905)
Test Plan: Tests are stacked on the internal diff D86605248.
Reviewed By: StellarrZ
Differential Revision: D86605248
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167457
Approved by: https://github.com/eellison, https://github.com/mlazos
Summary: fix some type errors + instead of manually creating a filelock when dumping dcache's imc to file we simply use an odc (since this is the intended behavior of odc, anyways)
Test Plan:
```
buck test fbcode//mode/opt caffe2/test/inductor:caching
```
Differential Revision: D86345594
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167136
Approved by: https://github.com/aorenste
Prior to this PR, the IValue <-> StableIValue conversion for `DeviceObjType` (aka c10::Device) was to pack it into the leading bits of the StableIValue (which is a uint64_t)
After this PR, the IValue <-> StableIValue conversion for `DeviceObjType` expects DeviceType to be packed into the upper 32 bits of StableIValue and DeviceIndex to be packed into the lower 32 bits
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166579
Approved by: https://github.com/janeyx99
Make the PyObject preservation scheme thread-safe with free threaded (nogil) Python. The general idea is:
* Python Tensor and Storage objects always hold a strong reference to their underlying c10 object
* c10 objects hold a strong reference to their Python objects if there's at least one other reference to the c10 object
This is implemented in `intrusive_ptr`:
* The top most bit (`kHasPyObject`) from the weakref count is now used to indicate if the `intrusive_ptr_target` has an associated PyObject. So `kHasPyObject` is one bit, the weakref count is now 31 bits and the strong refcount remains 32 bits.
* When the reference count increases from one to two and `kHasPyObject` is set, we incref the associated Python object to ensure that it's kept alive.
* When the reference count decreases from two to one (i.e., there are no C++ reference to the `intrusive_ptr_target` other than from the Python object), we decre the associated Python object to break the cycle.
Other benefits:
* We can delete a lot of the copypasta from Python internal `subtype_dealloc`
* This fixes the weakref and GC bugs we had in the previous scheme. Python weakrefs on Tensors and Storages should just work as expected now.
Risks:
* Extra branch for reference count operations on `intrusive_ptr<TensorImpl>`, `intrusive_ptr<StorageImpl>`, and the generic `intrusive_ptr<intrusive_ptr_target>` even when we're not using Python.
* It's a big change
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166342
Approved by: https://github.com/albanD
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
Preferred solution to #166380
Changes:
- Moved summary print to bottom of CMakeLists.txt
- Fix the problem 'add_compile_options' should be called before targets defined, so opted for `append_cxx_flag_if_supported` and `append_c_flag_if_supported` ( new ).
- Added extra verbosity so it can be seen when linker script added.
( unfortunately linker script has to be added per-target rather than globally due to ninja/cmake depdendency tracking ).
Also move summary print to bottom of CMakeLists.txt and improve logging
Pull Request resolved: https://github.com/pytorch/pytorch/pull/166407
Approved by: https://github.com/Aidyn-A, https://github.com/atalman
Summary:
https://github.com/pytorch/pytorch/pull/166609 updates `node.is_impure` to consider a submodule as impure if submodule contains impure node. This in turn changes `graph.eliminate_dead_code()` function behavior, which does not eliminate nodes with side effects, see [pytorch documentation](https://docs.pytorch.org/docs/stable/fx.html#torch.fx.Graph.eliminate_dead_code)
> Remove all dead code from the graph, based on each node’s number of users, and whether the nodes have any side effects.
While this is correct that a submodule containing side-effectful ops is side-effectful and should not be dead code eliminated, some customers rely on the dead code elimination to eliminate submodules that contain impure ops which is the behavior before #166609 fix.
Due to production environment constraints, we have to revert https://github.com/pytorch/pytorch/pull/166609 and move the side-effectful submodule check logic to `const_fold.py`, which will correctly **not** const-fold a submodule that contains impure ops.
NOTE other call sites that use `node.is_impure()` to make decisions are still incorrectly eliminating side-effectful submodules, but we can't safely change that today.
## This pr
- move `_subgraph_has_impure_op` into `fx/experimental/const_fold.py`, check and prevent const-folding an impure submodule
- added a note in `node.is_impure` to highlight the incorrect behavior and context in case people go looking in the future.
Test Plan: run test_fx_const_fold and all tests pass
Differential Revision: D86641994
Pull Request resolved: https://github.com/pytorch/pytorch/pull/167443
Approved by: https://github.com/jfix71
This seems to generate a several-microsecond performance improvement in the detach benchmark I've been using.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci
[ghstack-poisoned]
Incremental addition of C++ fast path, taking advantage of the DTensor
dispatch key to let us work with IValues without Python conversion for
the fast path. I tried to just port unwrap_to_op_info to C++, but that
didn't get much of a win; the nice thing here seems to be the fusion
of unwrap_to_op_info with recompute_comparison_key.
This + the following PR appear to reduce DTensor dispatch time for
detach() from 43-46 usec (possibly 40-43 usec, don't have firm number
written down due to noise) to something like 33-36 usec (using a
benchmark similar to the one on #160580).
[ghstack-poisoned]
Another incremental step: don't deal with custom ops on the critical
path, let the dispatcher do that. Take control of calling into Python
on the critical path as well.
[ghstack-poisoned]
Intended to make it easier to reuse this logic for processing operator arguments as IValues in following PR(s).
Testing: python test/test_python_dispatch.py (broke during development, seems to work now)
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
It appears to be unused based on `cd torch; rg has_symints`.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
compute_global_tensor_info is on the hot path for DTensor.{from,to}_local. More incremental progress toward C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Next PR writes a C++ implementation. Seems good to have tests first.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Next PR writes a C++ implementation. Seems good to have tests first.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Move the entirety of `__new__` into C++, saving a layer of disable_dynamo and making progress toward all-C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Move the entirety of `__new__` into C++, saving a layer of disable_dynamo and making progress toward all-C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Move the entirety of `__new__` into C++, saving a layer of disable_dynamo and making progress toward all-C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Move the entirety of `__new__` into C++, saving a layer of disable_dynamo and making progress toward all-C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Move the entirety of `__new__` into C++, saving a layer of disable_dynamo and making progress toward all-C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
Move the entirety of `__new__` into C++, saving a layer of disable_dynamo and making progress toward all-C++.
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
Differential Revision: [D81530102](https://our.internmc.facebook.com/intern/diff/D81530102)
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
Differential Revision: [D81530102](https://our.internmc.facebook.com/intern/diff/D81530102)
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
Differential Revision: [D81530102](https://our.internmc.facebook.com/intern/diff/D81530102)
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
Differential Revision: [D81530102](https://our.internmc.facebook.com/intern/diff/D81530102)
cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
[ghstack-poisoned]
I initially didn't see good results porting this, but it was apparently because of pybind11 function calling overhead. (pybind11's object-handling primitives seem fine enough.) I'm interested in setting up nanobind, but this demonstrates it's not blocking.
[ghstack-poisoned]
These don't seem necessary to get reasonable errors:
- func was already passed to get_alias_info earlier in the calling function
- all we require for args and outs is the ability to index into them, which will cleanly throw if it fails anyway
[ghstack-poisoned]
See new code comment for explanation. In short, global cache seems wrong because of monkey patching but this use case is checking for a programming error IIUC.
[ghstack-poisoned]
They're usually/always used for sizes & strides, which is what SymDimVector is for (saves heap allocation of the list itself). Had to patch OptionalArray because there's a bunch of generated code that wants to convert these to SymIntArrayRef and the generator isn't easy to patch to manually wrap. Clear but small improvement in perf on "detach DTensor in a loop" benchmark; we aren't heap-allocating symdimlists anymore, though there's still some cost with destroying these because SymInt needs cleanup.
[ghstack-poisoned]
They're usually/always used for sizes & strides, which is what SymDimVector is for (saves heap allocation of the list itself). Had to patch OptionalArray because there's a bunch of generated code that wants to convert these to SymIntArrayRef and the generator isn't easy to patch to manually wrap. Clear but small improvement in perf on "detach DTensor in a loop" benchmark; we aren't heap-allocating symdimlists anymore, though there's still some cost with destroying these because SymInt needs cleanup.
[ghstack-poisoned]
They're usually/always used for sizes & strides, which is what SymDimVector is for (saves heap allocation of the list itself). Had to patch OptionalArray because there's a bunch of generated code that wants to convert these to SymIntArrayRef and the generator isn't easy to patch to manually wrap.
[ghstack-poisoned]
It is cheap to do an exact check against Tensor and much faster when it works (PyType_IsSubtype does not have this fastpath, I checked [source](9ee0214b5d/Objects/typeobject.c (L2889))). Spot-checked in perf on detach-DTensor-in-a-loop benchmark; small win but clear.
[ghstack-poisoned]
py::args::size() calls PyTuple_GetSize. Compiler can't know the two calls will always return the same result, so we have to consolidate them ourselves.
[ghstack-poisoned]
These seem to have been costing us 5-10 usec per detach (out of ~~95 usec total). If they need to ship let's talk about requirements and how we can make this more efficient given that we would prefer if an entire DTensor op could finish in 10 usec.
[ghstack-poisoned]
This seems to have been an especially slow one because of the repeated pybind access (schema is a pybind, as is arguments, and then we hit each argument). It's still ~~1% of total benchmark runtime because of the repeated single pybind function call, but that's a lot better.
[ghstack-poisoned]
This seems to be a (very very roughly) ~8% improvmeent on DTensor benchmark very similar to the benchmark from #160580 (120ish ms -> 110ish ms)
[ghstack-poisoned]
As the new comment explains, pybinding DispatchKeySet is expensive; one native function avoids the number of DispatchKeySets we incur pybind overhead for to 1.
Reviewers please advise on a better location for the new torch._C function; I just threw it in Module.cpp because I didn't know where to put it.
[ghstack-poisoned]
If SymInt::maybe_as_int() returns non-empty, then we get an inline
fast path. The philosophy here (as with the previous PR) is to
preserve performance in the "plain old ints" case.
Observed time spent in SymInt functions in computeStorageNBytes to
drop (and not cost shift elsewhere in the function) after this change,
profiling detach() using code similar to the benchmark from #160580
and Linux perf.
[ghstack-poisoned]
symbools are not identical with Py_True or PyFalse, so we can do those cheap checks first and at least get plain old bools to go fast
[ghstack-poisoned]
If I am reading [THPUtils_checkIndex](cf94cadbee/torch/csrc/utils.cpp (L28)) correctly, it already checks torch::is_symint before hitting __index__ or __int__.
[ghstack-poisoned]
If we want them interned, we should intern at callsites. (The numpy reference has bit rotted; see b222eb66c7 (diff-6bdb6105198083838f51c57b55b3a49472ed23043bb40018f1ea41138e687163))
Profiling a simple torchdispatch benchmark with perf before/after seems to show that time spent copying std::strings and interning Python strings is gone, though there is some noise and the improvement is very small.
[ghstack-poisoned]
This assertion was expensive because of is_traceable_wrapper_subclass. Finding a cheap check to run first that's likely to let us skip the rest seems to improve things significantly.
[ghstack-poisoned]
This assertion was expensive because of is_traceable_wrapper_subclass. Finding a cheap check to run first that's likely to let us skip the rest seems to improve things significantly.
[ghstack-poisoned]
Scanning a list of pybind enums with `in` is slow. See NOTE in code for full explanation.
This is a significant optimization; will be updating the torchdispatch/return_and_correct_aliasing portion of this stack with benchmark and results soonish.
[ghstack-poisoned]
- Empty containers are Falsey
- Hoist cheap checks first
- Microbenchmarked single-element set access method
Benchmark code:
```
import timeit
to_test = [
('list(x)', 'x = set([3])'),
('x[0]', 'x = [3]'),
('list(x)[0]', 'x = set([3])'),
('next(iter(x))', 'x = set([3])'),
]
for (stmt, setup) in to_test:
res = timeit.timeit(stmt=stmt, setup=setup)
print(f"Time for `{stmt}`: {res}")
```
Result with Python 3.13 on Mac (with excess digits manually trimmed; directionally matches result on Linux)
```
Time for `list(x)`: 0.03418
Time for `x[0]`: 0.00852
Time for `list(x)[0]`: 0.03561
Time for `next(iter(x))`: 0.02278
```
FWIW, I was surprised by this result, so I guess I'm glad I wrote the benchmark!
[ghstack-poisoned]
- Empty containers are Falsey
- Hoist cheap checks first
- Microbenchmarked single-element set access method
Benchmark code:
```
import timeit
to_test = [
('list(x)', 'x = set([3])'),
('x[0]', 'x = [3]'),
('list(x)[0]', 'x = set([3])'),
('next(iter(x))', 'x = set([3])'),
]
for (stmt, setup) in to_test:
res = timeit.timeit(stmt=stmt, setup=setup)
print(f"Time for `{stmt}`: {res}")
```
Result with Python 3.13 on Mac (with excess digits manually trimmed; directionally matches result on Linux)
```
Time for `list(x)`: 0.03418
Time for `x[0]`: 0.00852
Time for `list(x)[0]`: 0.03561
Time for `next(iter(x))`: 0.02278
```
FWIW, I was surprised by this result, so I guess I'm glad I wrote the benchmark!
[ghstack-poisoned]
- Empty containers are Falsey
- Hoist cheap checks first
- Microbenchmarked single-element set access method
Benchmark code:
```
import timeit
to_test = [
('list(x)', 'x = set([3])'),
('x[0]', 'x = [3]'),
('list(x)[0]', 'x = set([3])'),
('next(iter(x))', 'x = set([3])'),
]
for (stmt, setup) in to_test:
res = timeit.timeit(stmt=stmt, setup=setup)
print(f"Time for `{stmt}`: {res}")
```
Result with Python 3.13 on Mac (with excess digits manually trimmed; directionally matches result on Linux)
```
Time for `list(x)`: 0.03418
Time for `x[0]`: 0.00852
Time for `list(x)[0]`: 0.03561
Time for `next(iter(x))`: 0.02278
```
FWIW, I was surprised by this result, so I guess I'm glad I wrote the benchmark!
[ghstack-poisoned]
@ -18,6 +18,8 @@ Please report security issues using https://github.com/pytorch/pytorch/security/
All reports submitted through the security advisories mechanism would **either be made public or dismissed by the team within 90 days of the submission**. If advisory has been closed on the grounds that it is not a security issue, please do not hesitate to create an [new issue](https://github.com/pytorch/pytorch/issues/new?template=bug-report.yml) as it is still likely a valid issue within the framework.
**Note on crashes and out of bounds access**: PyTorch is a computational framework that performs operations on behalf of the caller. Like many low-level libraries, PyTorch generally does not validate all inputs to every function—the responsibility for providing valid arguments lies with the calling code. While crashes and out of bounds memory access should be reported as bugs, they are generally not considered security vulnerabilities in PyTorch's threat model.
Please refer to the following page for our responsible disclosure policy, reward guidelines, and those things that should not be reported:
"In typelist::count_if<Condition, TypeList>, the Condition argument must be a condition type trait, i.e. have a static constexpr bool ::value member.");
static_assert(
is_instantiation_of<typelist,TypeList>::value,
"In typelist::count_if<Condition, TypeList>, the TypeList argument must be typelist<...>.");
"In typelist::true_for_any_type<Condition, TypeList>, the Condition argument must be a condition type trait, i.e. have a static constexpr bool ::value member.");
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
