[gloo] Make it possible for gloo TCPStore to take over an existing socket fd (#103478)

Summary:
This diff allows the `TCPStore` server associated with a gloo process group to listen on an existing socket already bound to a port.

Without the functionality in this diff, canonical initialization of a gloo `ProcessGroup` is fundamentally racy: 1) ask the OS for a free port by creating a socket bound to port 0, 2) close the socket, 3) attempt to initialize a `TCPStore` server that listens on the previously free port. Of course, the problem is that in between steps 2 and 3, another process on the host may have claimed the port, causing `TCPStore` and overall process group initialization to fail. With this diff, it is now possible for users to completely avoid this race (see unit test for how this can be achieved).

Test Plan:
Added new unit test:
  buck2 test caffe2/test/distributed:store

Differential Revision: D46622317

Pull Request resolved: https://github.com/pytorch/pytorch/pull/103478
Approved by: https://github.com/H-Huang
This commit is contained in:
Jon Maltiel Swenson
2023-06-16 17:15:54 +00:00
committed by PyTorch MergeBot
parent 2bc56bec07
commit 0da38409a0
7 changed files with 94 additions and 4 deletions

View File

@ -34,6 +34,11 @@ struct TCPStoreOptions {
// A boolean value indicating whether multiple store instances can be
// initialized with the same host:port pair.
bool multiTenant = false;
// If specified, and if isServer is true, the underlying TCPServer will take
// over the bound socket associated to this fd. This option is useful to avoid
// port assignment races in certain scenarios.
c10::optional<int> masterListenFd = c10::nullopt;
};
class TORCH_API TCPStore : public Store {