Frank Lin 249e65b92d Graph-Safe RNG State Exchange for Tensor Parallelism (#114068) ...

See #113541

The PR allows for registering and controlling multiple RNG states using indices, ensuring cudagraph-safe operations, and includes both C++ and Python API changes to support this functionality.

cc  @eellison @anijain2305 @jansel @ezyang @ptrblck @csarofeen @mcarilli
Pull Request resolved: https://github.com/pytorch/pytorch/pull/114068
Approved by: https://github.com/ezyang, https://github.com/eqy, https://github.com/xuzhao9

2024-03-27 01:14:38 +00:00

..

amp_examples.rst

[AMP] Use generic autocast in example, specify dtype (#79579)

2022-06-17 21:32:51 +00:00

autograd.rst

Document complex optimizer semantic behavior (#121667)

2024-03-16 00:43:47 +00:00

broadcasting.rst

Fixes docs (#51439)

2021-01-31 22:00:26 -08:00

cpu_threading_runtimes.svg

Update CPU threading doc (#33083)

2020-02-11 14:13:51 -08:00

cpu_threading_torchscript_inference.rst

Upgrade MKL-DNN to DNNL v1.2 (#32422)

2020-03-26 22:07:59 -07:00

cpu_threading_torchscript_inference.svg

Lint trailing newlines (#54737)

2021-03-30 13:09:52 -07:00

cuda.rst

Graph-Safe RNG State Exchange for Tensor Parallelism (#114068)

2024-03-27 01:14:38 +00:00

ddp.rst

Update DDP dynamo debug docs (#118295)

2024-01-29 14:58:26 +00:00

extending.func.rst

Fix the example in the extending.func.rst (#109279)

2023-09-14 17:29:39 +00:00

extending.rst

[BE] add parentheses to kwargs unpacking func(*args, **(kwargs or {})) (#115026)

2023-12-03 20:03:26 +00:00

faq.rst

Update faq.rst so OOM section mentions checkpoint (#62709)

2021-08-05 07:40:08 -07:00

fsdp.rst

[docs] start a new FSDP notes doc (#117323)

2024-01-22 15:46:35 +00:00

gradcheck.rst

Add first draft of gradcheck note (#55966)

2021-04-27 14:33:42 -07:00

hip.rst

Tweak formatting of note on macros (#89598)

2022-11-28 20:42:30 +00:00

large_scale_deployments.rst

Move ThreadLocalDebugInfo to c10 (#37774)

2020-05-11 19:27:41 -07:00

modules.rst

Replace master with main in links and docs/conf.py (#100176)

2023-05-02 18:20:32 +00:00

mps.rst

update mps note with more details (#78669)

2022-06-02 20:53:19 +00:00

multiprocessing.rst

[draft] Update Multiprocessing best practices with CPU device (#103229)

2023-06-25 06:26:40 +00:00

numerical_accuracy.rst

[DOCS][CUDA] Update TF32 docs for sm90 (#111337)

2023-10-19 09:36:13 +00:00

randomness.rst

Changed cublas repdocubility URL (#121534)

2024-03-08 23:46:21 +00:00

serialization.rst

Document file format returned by torch.save (#118719)

2024-02-03 02:11:44 +00:00

windows.rst

Remove remaining THC code (#69039)

2021-12-08 12:18:08 -08:00