frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 21:14:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updated A quick guide on how to add and cache dependencies on PyTorch CI (markdown)

Huy Do
2023-03-24 15:38:44 -07:00
parent 48ab5802ff
commit 7183868c10
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
A-quick-guide-on-how-to-add-and-cache-dependencies-on-PyTorch-CI.md

@ -1,12 +1,12 @@
### Context
With the high volume of pull requests to PyTorch and the extensive scope of PyTorch test suite, PyTorch CI runs thousand of build and test jobs daily across multiple platforms including Linux (CPU, CUDA, ROCm), Windows (CPU, CUDA), MacOS (x86_64, M1), Android, and iOS. This makes stability the most important of the CI. Because stability includes many areas and means many things, we will focus only on CI dependencies in this wiki. Specifically, this is about how to add and cache dependencies safely and reliably.
Most softwares are built on top of others and requires their dependencies to be in place to work. PyTorch CI is no difference. Over the years, some of the most common issues about them are:
Most softwares are built on top of others and requires their dependencies to be in place to work. PyTorch CI is no difference. Over the years, some of the most common issues about setting up dependencies are:
* A dependency is not pinned. It's updated upstream and unexpectedly break stuffs.
* A dependency is not cached and is setup from scratch every time a job requiring it is run. This usually means downloading and installing something, i.e. a Conda package, from somewhere, i.e. the Internet. Doing this multiple time and it's bound to fail flakily in some cases.
Like doing a science experiment, their solutions would be 1) to run the CI jobs in a control manner with fixed parameters and reproducible results and 2) to have all the dependencies ready. These are achieved by:
Like doing a science experiment, the solutions would be 1) to run the CI jobs in a control manner with fixed parameters and reproducible results and 2) to have all the dependencies close at hand. They are achieved by:
1. Pin all CI dependencies to specific versions and update them explicitly.
1. Put all Linux dependencies into Docker images.