mirror of
https://github.com/pytorch/pytorch.git
synced 2025-10-20 21:14:14 +08:00
Updated Docker image build on CircleCI (markdown)
@ -4,8 +4,6 @@ For those who used to work with the docker image build process, we’ve migrated
|
||||
|
||||
|
||||
The config of docker image now live at https://github.com/pytorch/pytorch/tree/master/.circleci/docker
|
||||
And we have weekly jobs to rebuild all the images: https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml#L5695-L5751 (this link might not work if we ever update config.yml file, so please search for “docker_build_job” to be sure)
|
||||
|
||||
and you can find existing images (both permanent and weekly) at [http://docker.pytorch.org](http://docker.pytorch.org/) which will be updated hourly by this job: https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml#L5752-L5776 (for same reason as above, you can search “ecr_gc_job” to be sure)
|
||||
|
||||
|
||||
@ -13,7 +11,9 @@ and you can find existing images (both permanent and weekly) at [http://docker.p
|
||||
|
||||
### How to trigger new build process if I don’t want to wait for a week?
|
||||
|
||||
Comment out or remove https://github.com/pytorch/pytorch/blob/cb9029df9db397f66da234d21b5ddafd6be6e602/.circleci/generate_config_yml.py#L120-L127, so your PR can/will trigger build job. **REMEMBER** to run [regenerate.sh](https://github.com/pytorch/pytorch/blob/master/.circleci/regenerate.sh) when you’re inside .circle directory after you update the file.
|
||||
New images are automatically built when any changes are committed to `.circleci/docker`. Developer images are retained for 2 weeks, so if your builds are older than 2 weeks you will need to rebuild your images if you intend to merge the PR associated with docker image changes.
|
||||
|
||||
New images will automatically be passed down through to dependent jobs.
|
||||
|
||||
### Where to find tags for newly created/pushed images?
|
||||
|
||||
@ -24,12 +24,6 @@ Comment out or remove https://github.com/pytorch/pytorch/blob/cb9029df9db397f66d
|
||||
We have ecr_gc_job job (you can search for it in config.yml) that runs every hour to purge old images. Currently, we need temporary images for 1 day, and weekly builds for 2 weeks. And we will keep image with tags defined in https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/workflows-ecr-gc.yml#L13 forever.
|
||||
code for the purge job is https://github.com/pytorch/pytorch/blob/master/.circleci/ecr_gc_docker/gc.py
|
||||
|
||||
### How to use new images?
|
||||
|
||||
1. locate new image tag at http://docker.pytorch.org/pytorch.html
|
||||
2. test with new images, you need to update https://github.com/pytorch/pytorch/blob/master/.circleci/cimodel/data/pytorch_build_definitions.py#L16 (or search for DOCKER_IMAGE_VERSION) and regenerate config.yml
|
||||
3. update https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/workflows-ecr-gc.yml#L13 and land changes to make sure the images won’t be purged
|
||||
|
||||
### The production images disappeared, what should I do?
|
||||
|
||||
Sometimes there is a bug in ecr_gc_job and it deletes Docker images it shouldn't. All Docker images are also saved to S3 with a one month retention period, so there's a chance they may still be there. You can use this script to recover in that case:
|
||||
@ -82,7 +76,6 @@ Note this instruction provides guidance to add a new **base** docker image. If y
|
||||
- Test your image by building it locally.
|
||||
- Add a repo in AWS ECR to hold your image. It requires access to PyTorch's AWS account to do this step.
|
||||
- Trigger a new build process as described above.
|
||||
- Remove https://github.com/pytorch/pytorch/blob/master/.circleci/verbatim-sources/workflows-ecr-gc.yml#L2-L8 so that docker_hub_index_job can be run on your PR. http://docker.pytorch.org/ will then display images pushed into the newly created repo.
|
||||
- **WAIT UNTIL ALL BUILD JOBS TO FINISH** and make sure all new images have been uploaded. Save the tag of the new images.
|
||||
- Run regenerated.sh with the new tag and update your PR.
|
||||
|
||||
|
Reference in New Issue
Block a user