mirror of
https://github.com/huggingface/trl.git
synced 2025-10-20 10:03:51 +08:00
🧱 PyPI publishing workflow (#3976)
This commit is contained in:
Quentin Gallouédec
committed by
GitHub
GitHub
parent
ab984fabac
commit
7ae16d3234
43
.github/workflows/publish.yml
vendored
Normal file
43
.github/workflows/publish.yml
vendored
Normal file
@ -0,0 +1,43 @@
|
||||
name: Publish to PyPI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- v*-release
|
||||
paths:
|
||||
- "VERSION"
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Read version
|
||||
id: get_version
|
||||
run: echo "version=$(cat VERSION)" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Debug - Show version.txt content
|
||||
run: echo "Version is ${{ steps.get_version.outputs.version }}"
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: "3.x"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install build twine
|
||||
|
||||
- name: Build package
|
||||
run: python -m build
|
||||
|
||||
- name: Publish to PyPI
|
||||
if: ${{ !contains(steps.get_version.outputs.version, 'dev') }}
|
||||
env:
|
||||
TWINE_USERNAME: __token__
|
||||
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
|
||||
run: |
|
||||
python -m twine upload dist/*
|
||||
309
CONTRIBUTING.md
309
CONTRIBUTING.md
@ -456,312 +456,3 @@ Warnings play a critical role in guiding users toward resolving potential issues
|
||||
```
|
||||
|
||||
By following this classification, you ensure that warnings, information, and exceptions are used appropriately, providing clear guidance to the user without cluttering the system with unnecessary messages.
|
||||
|
||||
|
||||
## Making a release
|
||||
|
||||
> [!NOTE]
|
||||
> VERSION needs to be formatted following the `v{major}.{minor}.{patch}` convention. We need to follow this convention to be able to retrieve versioned scripts.
|
||||
|
||||
#### 0. Prerequisites
|
||||
|
||||
- Dependencies:
|
||||
- twine: `pip install build twine`
|
||||
- Create an account in (and join the `trl` project):
|
||||
- PyPI: https://pypi.org/
|
||||
- Test PyPI: https://test.pypi.org/
|
||||
|
||||
### Major/Minor Release
|
||||
|
||||
#### 1. Ensure your local repository is up to date with the upstream repository
|
||||
|
||||
```bash
|
||||
git checkout main
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
> [!WARNING]
|
||||
> Do not merge other pull requests into `main` until the release is done. This is to ensure that the release is stable and does not include any untested changes. Announce internally (#trl-internal) to other maintainers that you are doing a release and that they must not merge PRs until the release is done.
|
||||
|
||||
#### 2. Create a release branch from main
|
||||
|
||||
```bash
|
||||
git checkout -b release-v{major}.{minor}
|
||||
```
|
||||
|
||||
#### 3. Change the version in the following files
|
||||
|
||||
- `.github/workflows/tests_latest.yml`:
|
||||
```diff
|
||||
- with: { ref: v{major}.{minor-1}-release }
|
||||
+ with: { ref: v{major}.{minor}-release }
|
||||
```
|
||||
- `CITATION.cff`
|
||||
```diff
|
||||
- version: "{major}.{minor-1}"
|
||||
+ version: "{major}.{minor}"
|
||||
```
|
||||
- `trl/__init__.py`
|
||||
```diff
|
||||
- __version__ = "{major}.{minor}.0.dev0"
|
||||
+ __version__ = "{major}.{minor}.0"
|
||||
```
|
||||
- `setup.cfg`
|
||||
```diff
|
||||
- version = {major}.{minor}.0.dev0
|
||||
+ version = {major}.{minor}.0
|
||||
```
|
||||
|
||||
#### 4. Commit and push these changes
|
||||
|
||||
```shell
|
||||
git add .github/workflows/tests_latest.yml CITATION.cff trl/__init__.py setup.cfg
|
||||
git commit -m 'Release: {major}.{minor}'
|
||||
git push origin release-v{major}.{minor}
|
||||
```
|
||||
|
||||
#### 5. Create a pull request
|
||||
|
||||
from `release-v{major}.{minor}` to `main`, named `Release: v{major}.{minor}`, wait for tests to pass, and request a review.
|
||||
|
||||
#### 6. Once the pull request is approved, merge it into `main`
|
||||
|
||||
#### 7. Add a tag in git to mark the release
|
||||
|
||||
```shell
|
||||
git checkout main
|
||||
git pull origin main
|
||||
git tag -a v{major}.{minor}.0 -m 'Adds tag v{major}.{minor}.0 for PyPI'
|
||||
git push origin v{major}.{minor}.0
|
||||
```
|
||||
|
||||
#### 8. Create a branch `v{major}.{minor}-release` for future patch releases.
|
||||
|
||||
```shell
|
||||
git checkout -b v{major}.{minor}-release
|
||||
git push origin v{major}.{minor}-release
|
||||
```
|
||||
|
||||
This ensures that future patch releases (`v{major}.{minor}.1`, `v{major}.{minor}.2`, etc.) can be made separately from `main`.
|
||||
|
||||
#### 9. Create the wheels for your release
|
||||
|
||||
These are the artifacts that will be uploaded to PyPI and installed by users via `pip install trl`.
|
||||
|
||||
Clean previous builds:
|
||||
|
||||
```shell
|
||||
rm -rf build dist
|
||||
```
|
||||
|
||||
At the root of your repo, run
|
||||
|
||||
```bash
|
||||
python -m build .
|
||||
```
|
||||
|
||||
This will create a folders named `dist` with the new versions of your package.
|
||||
|
||||
#### 10. Upload the package to PyPI Test
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Do not skip this step. It is important to test the package before uploading it to the main PyPI server.
|
||||
|
||||
```shell
|
||||
twine upload dist/* -r testpypi
|
||||
```
|
||||
|
||||
Then in a fresh environment containing all dependencies you need, try to install your new package from the PyPI test server.
|
||||
|
||||
```bash
|
||||
pip install -i https://test.pypi.org/simple/ trl
|
||||
```
|
||||
|
||||
You might get errors for missing dependencies since the PyPI test server does not contain all packages like PyPI does. To make sure you have everything you can do:
|
||||
|
||||
```bash
|
||||
pip install trl
|
||||
pip uninstall trl
|
||||
```
|
||||
|
||||
(the second line will remove trl but keep all its dependencies).
|
||||
|
||||
Also make sure you can actually use the package! Run the following line:
|
||||
|
||||
```bash
|
||||
python -c "from trl import *"
|
||||
```
|
||||
|
||||
along with anything that tests:
|
||||
|
||||
- the core feature of your package
|
||||
- the new features you’re adding in the release
|
||||
|
||||
#### 11. Publish on PyPI
|
||||
|
||||
> [!WARNING]
|
||||
> This can't be reverted. Make sure you have tested everything before doing this step.
|
||||
|
||||
```shell
|
||||
twine upload dist/*
|
||||
```
|
||||
|
||||
#### 12. Create a GitHub Release
|
||||
|
||||
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
|
||||
2. Click **Draft a new release**.
|
||||
3. Select the `v{major}.{minor}.0` tag you just created in step 7.
|
||||
4. Add a title (`v{major}.{minor}.0`) and a short description of what’s new.
|
||||
5. Click **Publish Release**.
|
||||
|
||||
#### 13. Bump to dev version
|
||||
|
||||
1. Create a branch `bump-dev-version-{major}.{minor+1}` from `main` and checkout to it.
|
||||
|
||||
```shell
|
||||
git checkout -b bump-dev-version-{major}.{minor+1}
|
||||
```
|
||||
|
||||
2. Change the version in the following files:
|
||||
1. `trl/__init__.py`
|
||||
```diff
|
||||
- __version__ = "{major}.{minor}.0"
|
||||
+ __version__ = "{major}.{minor+1}.0.dev0"
|
||||
```
|
||||
2. `setup.cfg`
|
||||
```diff
|
||||
- version = {major}.{minor}.0
|
||||
+ version = {major}.{minor+1}.0.dev0
|
||||
```
|
||||
|
||||
3. Commit and push these changes
|
||||
|
||||
```shell
|
||||
git add trl/__init__.py setup.cfg
|
||||
git commit -m '⬆️ Bump dev version'
|
||||
git push origin bump-dev-version-{major}.{minor+1}
|
||||
```
|
||||
|
||||
4. Create a pull request from `bump-dev-version-{major}.{minor+1}` to `main`, named `⬆️ Bump dev version`, and request urgent review.
|
||||
|
||||
5. Once the pull request is approved, merge it into `main`.
|
||||
|
||||
6. The codebase is now ready for the next development cycle, inform the team in the #trl-internal channel.
|
||||
|
||||
|
||||
## Making a patch release
|
||||
|
||||
#### 1. Ensure your local repository is up to date with the upstream repository
|
||||
|
||||
```bash
|
||||
git checkout v{major}.{minor}-release
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
#### 2. Cherry-pick the changes you want to include in the patch release
|
||||
|
||||
```bash
|
||||
git cherry-pick <commit-hash-0>
|
||||
git cherry-pick <commit-hash-1>
|
||||
...
|
||||
```
|
||||
|
||||
#### 3. Change the version in the following files
|
||||
|
||||
- `trl/__init__.py`
|
||||
```diff
|
||||
- __version__ = "{major}.{minor}.{patch-1}"
|
||||
+ __version__ = "{major}.{minor}.{patch}"
|
||||
```
|
||||
- `setup.cfg`
|
||||
```diff
|
||||
- version = {major}.{minor}.{patch-1}
|
||||
+ version = {major}.{minor}.{patch}
|
||||
```
|
||||
|
||||
#### 4. Commit and push these changes
|
||||
|
||||
```shell
|
||||
git add trl/__init__.py setup.cfg
|
||||
git commit -m 'Release: {major}.{minor}.{patch}'
|
||||
git push origin v{major}.{minor}-release
|
||||
```
|
||||
|
||||
#### 5. Wait for the CI to pass
|
||||
|
||||
#### 6. Add a tag in git to mark the release
|
||||
|
||||
```shell
|
||||
git tag -a v{major}.{minor}.{patch} -m 'Adds tag v{major}.{minor}.{patch} for PyPI'
|
||||
git push origin v{major}.{minor}.{patch}
|
||||
```
|
||||
|
||||
#### 7. Create the wheels for your release
|
||||
|
||||
These are the artifacts that will be uploaded to PyPI and installed by users via `pip install trl`.
|
||||
|
||||
Clean previous builds:
|
||||
|
||||
```shell
|
||||
rm -rf build dist
|
||||
```
|
||||
|
||||
At the root of your repo, run
|
||||
|
||||
```bash
|
||||
python -m build .
|
||||
```
|
||||
|
||||
This will create a folders named `dist` with the new versions of your package.
|
||||
|
||||
#### 8. Upload the package to PyPI Test
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Do not skip this step. It is important to test the package before uploading it to the main PyPI server.
|
||||
|
||||
```shell
|
||||
twine upload dist/* -r testpypi
|
||||
```
|
||||
|
||||
Then in a fresh environment containing all dependencies you need, try to install your new package from the PyPI test server.
|
||||
|
||||
```bash
|
||||
pip install -i https://test.pypi.org/simple/ trl
|
||||
```
|
||||
|
||||
You might get errors for missing dependencies since the PyPI test server does not contain all packages like PyPI does. To make sure you have everything you can do:
|
||||
|
||||
```bash
|
||||
pip install trl
|
||||
pip uninstall trl
|
||||
```
|
||||
|
||||
(the second line will remove trl but keep all its dependencies).
|
||||
|
||||
Also make sure you can actually use the package! Run the following line:
|
||||
|
||||
```bash
|
||||
python -c "from trl import *"
|
||||
```
|
||||
|
||||
along with anything that tests:
|
||||
|
||||
- the core feature of your package
|
||||
- the new features you’re adding in the release
|
||||
|
||||
#### 9. Publish on PyPI
|
||||
|
||||
> [!WARNING]
|
||||
> This can't be reverted. Make sure you have tested everything before doing this step.
|
||||
|
||||
```shell
|
||||
twine upload dist/*
|
||||
```
|
||||
|
||||
#### 10. Create a GitHub Release
|
||||
|
||||
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
|
||||
2. Click **Draft a new release**.
|
||||
3. Select the `v{major}.{minor}.{patch}` tag you just created in step 7.
|
||||
4. Add a title (`v{major}.{minor}.{patch}`) and a short description of what’s new.
|
||||
5. Click **Publish Release**.
|
||||
|
||||
167
RELEASE.md
Normal file
167
RELEASE.md
Normal file
@ -0,0 +1,167 @@
|
||||
# Making a release
|
||||
|
||||
> [!NOTE]
|
||||
> VERSION needs to be formatted following the `v{major}.{minor}.{patch}` convention. We need to follow this convention to be able to retrieve versioned scripts.
|
||||
|
||||
## Major/Minor Release
|
||||
|
||||
### 1. Ensure your local repository is up to date with the upstream repository
|
||||
|
||||
```bash
|
||||
git checkout main
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
> [!WARNING]
|
||||
> Do not merge other pull requests into `main` until the release is done. This is to ensure that the release is stable and does not include any untested changes. Announce internally (#trl-internal) to other maintainers that you are doing a release and that they must not merge PRs until the release is done.
|
||||
|
||||
### 2. Create a release branch from main
|
||||
|
||||
```bash
|
||||
git checkout -b release-v{major}.{minor}
|
||||
```
|
||||
|
||||
### 3. Change the version in the following files
|
||||
|
||||
- `.github/workflows/tests_latest.yml`:
|
||||
|
||||
```diff
|
||||
- with: { ref: v{major}.{minor-1}-release }
|
||||
+ with: { ref: v{major}.{minor}-release }
|
||||
```
|
||||
|
||||
- `CITATION.cff`
|
||||
|
||||
```diff
|
||||
- version: "{major}.{minor-1}"
|
||||
+ version: "{major}.{minor}"
|
||||
```
|
||||
|
||||
- `VERSION`
|
||||
|
||||
```diff
|
||||
- {major}.{minor}.0.dev0
|
||||
+ {major}.{minor}.0
|
||||
```
|
||||
|
||||
### 4. Commit and push these changes
|
||||
|
||||
```shell
|
||||
git add .github/workflows/tests_latest.yml CITATION.cff VERSION
|
||||
git commit -m 'Release: {major}.{minor}'
|
||||
git push origin release-v{major}.{minor}
|
||||
```
|
||||
|
||||
### 5. Create a pull request
|
||||
|
||||
from `release-v{major}.{minor}` to `main`, named `Release: v{major}.{minor}`, wait for tests to pass, and request a review.
|
||||
|
||||
### 6. Once the pull request is approved, merge it into `main`
|
||||
|
||||
It will automatically publish the new version of the package on PyPI.
|
||||
|
||||
### 7. Add a tag in git to mark the release
|
||||
|
||||
```shell
|
||||
git checkout main
|
||||
git pull origin main
|
||||
git tag -a v{major}.{minor}.0 -m 'Adds tag v{major}.{minor}.0 for PyPI'
|
||||
git push origin v{major}.{minor}.0
|
||||
```
|
||||
|
||||
### 8. Create a branch `v{major}.{minor}-release` for future patch releases
|
||||
|
||||
```shell
|
||||
git checkout -b v{major}.{minor}-release
|
||||
git push origin v{major}.{minor}-release
|
||||
```
|
||||
|
||||
This ensures that future patch releases (`v{major}.{minor}.1`, `v{major}.{minor}.2`, etc.) can be made separately from `main`.
|
||||
|
||||
### 9. Create a GitHub Release
|
||||
|
||||
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
|
||||
2. Click **Draft a new release**.
|
||||
3. Select the `v{major}.{minor}.0` tag you just created in step 7.
|
||||
4. Add a title (`v{major}.{minor}.0`) and a short description of what’s new.
|
||||
5. Click **Publish Release**.
|
||||
|
||||
### 10. Bump to dev version
|
||||
|
||||
1. Create a branch `bump-dev-version-{major}.{minor+1}` from `main` and checkout to it.
|
||||
|
||||
```shell
|
||||
git checkout -b bump-dev-version-{major}.{minor+1}
|
||||
```
|
||||
|
||||
2. Change the version in file `VERSION`:
|
||||
|
||||
```diff
|
||||
- {major}.{minor}.0
|
||||
+ {major}.{minor+1}.0.dev0
|
||||
```
|
||||
|
||||
3. Commit and push these changes
|
||||
|
||||
```shell
|
||||
git add VERSION
|
||||
git commit -m '⬆️ Bump dev version'
|
||||
git push origin bump-dev-version-{major}.{minor+1}
|
||||
```
|
||||
|
||||
4. Create a pull request from `bump-dev-version-{major}.{minor+1}` to `main`, named `⬆️ Bump dev version`, and request urgent review.
|
||||
|
||||
5. Once the pull request is approved, merge it into `main`.
|
||||
|
||||
6. The codebase is now ready for the next development cycle, inform the team in the #trl-internal channel.
|
||||
|
||||
## Making a patch release
|
||||
|
||||
### 1. Ensure your local repository is up to date with the upstream repository
|
||||
|
||||
```bash
|
||||
git checkout v{major}.{minor}-release
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
### 2. Cherry-pick the changes you want to include in the patch release
|
||||
|
||||
```bash
|
||||
git cherry-pick <commit-hash-0>
|
||||
git cherry-pick <commit-hash-1>
|
||||
...
|
||||
```
|
||||
|
||||
### 3. Change the version in the file `VERSION`
|
||||
|
||||
```diff
|
||||
- {major}.{minor}.{patch-1}
|
||||
+ {major}.{minor}.{patch}
|
||||
```
|
||||
|
||||
### 4. Commit and push these changes
|
||||
|
||||
```shell
|
||||
git add VERSION
|
||||
git commit -m 'Release: {major}.{minor}.{patch}'
|
||||
git push origin v{major}.{minor}-release
|
||||
```
|
||||
|
||||
### 5. Wait for the CI to pass
|
||||
|
||||
The CI will automatically publish the new version of the package on PyPI.
|
||||
|
||||
### 6. Add a tag in git to mark the release
|
||||
|
||||
```shell
|
||||
git tag -a v{major}.{minor}.{patch} -m 'Adds tag v{major}.{minor}.{patch} for PyPI'
|
||||
git push origin v{major}.{minor}.{patch}
|
||||
```
|
||||
|
||||
#### 7. Create a GitHub Release
|
||||
|
||||
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
|
||||
2. Click **Draft a new release**.
|
||||
3. Select the `v{major}.{minor}.{patch}` tag you just created in step 7.
|
||||
4. Add a title (`v{major}.{minor}.{patch}`) and a short description of what’s new.
|
||||
5. Click **Publish Release**.
|
||||
@ -1,6 +1,6 @@
|
||||
[metadata]
|
||||
name = trl
|
||||
version = 0.22.0.dev0
|
||||
version = file: VERSION
|
||||
description = Train transformer language models with reinforcement learning.
|
||||
long_description = file: README.md
|
||||
long_description_content_type = text/markdown
|
||||
|
||||
@ -12,13 +12,20 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
__version__ = "0.22.0.dev0"
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from .import_utils import OptionalDependencyNotAvailable, _LazyModule, is_diffusers_available
|
||||
|
||||
|
||||
# Read version from VERSION file
|
||||
_version_file = Path(__file__).parent.parent / "VERSION"
|
||||
try:
|
||||
with open(_version_file, encoding="utf-8") as f:
|
||||
__version__ = f.read().strip()
|
||||
except FileNotFoundError:
|
||||
__version__ = "unknown"
|
||||
|
||||
_import_structure = {
|
||||
"scripts": ["DatasetMixtureConfig", "ScriptArguments", "TrlParser", "get_dataset", "init_zero_verbose"],
|
||||
"data_utils": [
|
||||
|
||||
Reference in New Issue
Block a user