pytorch

mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 21:14:14 +08:00

Files

Shuqiang Zhang c0861d092c [PGNCCL] Add an API to get the status/error code at the PG level (#144498 )

Summary:
This PR is basically a replacement of
https://github.com/pytorch/pytorch/pull/140087, which caused some perf
drop due to frequent TCPStore check in watchdog thread. The fix is to move the
tcpstore check in monitoring thread

If unhealthy, the user should be able to get the type of errors, e.g.,
timeout,nccl error or remote error.

This API is applied to PG level, compared to the
work.get_future_result() API which is applied to Work Level.
Error detection at PG level is much more convenient for users to handle
the PG failure as a whole, e.g, restarting the PG.

Error handling at the work level is still useful for users to attach
work specific context and debug the RC of the specific failing
work/collective

Note it is critical for all ranks in the PG to be notified about an
error as soon as it occurs, so we introduce an errorType of
REMOTE_ERROR, which is 'broadcasted' from a src rank (which detects a
local error) to all other ranks in the PG, the broadcast is done through
TCPStore currently

Tags:

Pull Request resolved: https://github.com/pytorch/pytorch/pull/144498
Approved by: https://github.com/kwen2501

2025-01-24 16:47:32 +00:00

aoti_abi_check

[AOTI] Fix complex64 not defined (#132810 )

2024-08-08 18:08:23 +00:00

aoti_inference

[AOTInductor] Add standalone test for compilation from ExportedProgram (#142327 )

2024-12-10 06:50:09 +00:00

api

[ROCm][CI] upgrade CI to ROCm 6.3 (#142152 )

2025-01-09 17:14:16 +00:00

c10d

[PGNCCL] Add an API to get the status/error code at the PG level (#144498 )