frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-21 05:34:18 +08:00
Code Issues Packages Projects Releases Wiki Activity

c10d/logging: add C10D_LOCK_GUARD (#134131)

Browse Source 
This adds logs if we can't acquire locks in NCCLUtils and ProcessGroupNCCL for 30s.

This is motivated by some deadlocks were seeing and it's unclear if it's in NCCL or on the PyTorch side of things.

This required replacing most `std::mutex` with `std::timed_mutex` and `std::condition_variable_any` as appropriate.

Test plan:

existing CI for regressions

will add unit tests on `C10D_LOCK_GUARD`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/134131
Approved by: https://github.com/c-p-i-o, https://github.com/fduwjj
This commit is contained in:
Tristan Rice
2024-08-28 01:40:42 +00:00
committed by PyTorch MergeBot
parent c45ca8092d
commit f33bcbe5fd
14 changed files with 183 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
test/cpp/c10d/ProcessGroupNCCLErrorsTest.cpp
View File

@ -180,7 +180,7 @@ class ProcessGroupNCCLNoHeartbeatCaught
: ProcessGroupNCCLTimedOutErrors(store, rank, size, opts),
hasMonitorThreadCaughtError_(false) {}
std::mutex& getWatchdogMutex() {
std::timed_mutex& getWatchdogMutex() {
return workMetaListMutex_;
}
@ -413,7 +413,7 @@ TEST_F(ProcessGroupNCCLErrorsTest, testNCCLErrorsNoHeartbeat) {
work = pg.allreduce(tensors_);
{
// Now run all reduce with errors.
std::lock_guard<std::mutex> lock(pg.getWatchdogMutex());
std::lock_guard<std::timed_mutex> lock(pg.getWatchdogMutex());
LOG(INFO) << "Lock watchdog thread.";
// Wait long enough before monitor thread throws exceptions.
std::this_thread::sleep_for(