frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 21:14:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

Highlight NCCL all_reduce and all_gather requirements (#18741)

Browse Source 
Summary:
See #18689
Pull Request resolved: https://github.com/pytorch/pytorch/pull/18741

Differential Revision: D14726874

Pulled By: mrshenli

fbshipit-source-id: a92404c653e3c62fc23fa3ccacfb3b2959b2e307
This commit is contained in:
Shen Li
2019-04-03 09:06:09 -07:00
committed by Facebook Github Bot
parent 2658190def
commit c0ad6747a9
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
torch/distributed/distributed_c10d.py
View File

@ -300,7 +300,10 @@ def init_process_group(backend,
build-time configurations, valid values include ``mpi``, ``gloo``,
and ``nccl``. This field should be given as a lowercase string
(e.g., ``"gloo"``), which can also be accessed via
:class:`Backend` attributes (e.g., ``Backend.GLOO``).
:class:`Backend` attributes (e.g., ``Backend.GLOO``). If using
multiple processes per machine with ``nccl`` backend, each process
must have exclusive access to every GPU it uses, as sharing GPUs
between processes can result in deadlocks.
init_method (str, optional): URL specifying how to initialize the
process group.
world_size (int, optional): Number of processes participating in