frozenleaves/trl
1
0
Fork 0
mirror of https://github.com/huggingface/trl.git synced 2025-10-20 18:43:52 +08:00
Code Issues Packages Projects Releases Wiki Activity

🧱 PyPI publishing workflow (#3976)

Browse Source
This commit is contained in:
Quentin Gallouédec
2025-08-29 12:52:25 -07:00
committed by GitHub
parent ab984fabac
commit 7ae16d3234
6 changed files with 221 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
.github/workflows/publish.yml vendored Normal file
View File

@ -0,0 +1,43 @@
name: Publish to PyPI
on:
push:
branches:
- main
- v*-release
paths:
- "VERSION"
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Read version
id: get_version
run: echo "version=$(cat VERSION)" >> $GITHUB_OUTPUT
- name: Debug - Show version.txt content
run: echo "Version is ${{ steps.get_version.outputs.version }}"
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.x"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install build twine
- name: Build package
run: python -m build
- name: Publish to PyPI
if: ${{ !contains(steps.get_version.outputs.version, 'dev') }}
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
run: |
python -m twine upload dist/*

309
CONTRIBUTING.md
View File

@ -456,312 +456,3 @@ Warnings play a critical role in guiding users toward resolving potential issues
``` ```
By following this classification, you ensure that warnings, information, and exceptions are used appropriately, providing clear guidance to the user without cluttering the system with unnecessary messages. By following this classification, you ensure that warnings, information, and exceptions are used appropriately, providing clear guidance to the user without cluttering the system with unnecessary messages.
## Making a release
> [!NOTE]
> VERSION needs to be formatted following the `v{major}.{minor}.{patch}` convention. We need to follow this convention to be able to retrieve versioned scripts.
#### 0. Prerequisites
- Dependencies:
- twine: `pip install build twine`
- Create an account in (and join the `trl` project):
- PyPI: https://pypi.org/
- Test PyPI: https://test.pypi.org/
### Major/Minor Release
#### 1. Ensure your local repository is up to date with the upstream repository
```bash
git checkout main
git pull origin main
```
> [!WARNING]
> Do not merge other pull requests into `main` until the release is done. This is to ensure that the release is stable and does not include any untested changes. Announce internally (#trl-internal) to other maintainers that you are doing a release and that they must not merge PRs until the release is done.
#### 2. Create a release branch from main
```bash
git checkout -b release-v{major}.{minor}
```
#### 3. Change the version in the following files
- `.github/workflows/tests_latest.yml`:
```diff
- with: { ref: v{major}.{minor-1}-release }
+ with: { ref: v{major}.{minor}-release }
```
- `CITATION.cff`
```diff
- version: "{major}.{minor-1}"
+ version: "{major}.{minor}"
```
- `trl/__init__.py`
```diff
- __version__ = "{major}.{minor}.0.dev0"
+ __version__ = "{major}.{minor}.0"
```
- `setup.cfg`
```diff
- version = {major}.{minor}.0.dev0
+ version = {major}.{minor}.0
```
#### 4. Commit and push these changes
```shell
git add .github/workflows/tests_latest.yml CITATION.cff trl/__init__.py setup.cfg
git commit -m 'Release: {major}.{minor}'
git push origin release-v{major}.{minor}
```
#### 5. Create a pull request
from `release-v{major}.{minor}` to `main`, named `Release: v{major}.{minor}`, wait for tests to pass, and request a review.
#### 6. Once the pull request is approved, merge it into `main`
#### 7. Add a tag in git to mark the release
```shell
git checkout main
git pull origin main
git tag -a v{major}.{minor}.0 -m 'Adds tag v{major}.{minor}.0 for PyPI'
git push origin v{major}.{minor}.0
```
#### 8. Create a branch `v{major}.{minor}-release` for future patch releases.
```shell
git checkout -b v{major}.{minor}-release
git push origin v{major}.{minor}-release
```
This ensures that future patch releases (`v{major}.{minor}.1`, `v{major}.{minor}.2`, etc.) can be made separately from `main`.
#### 9. Create the wheels for your release
These are the artifacts that will be uploaded to PyPI and installed by users via `pip install trl`.
Clean previous builds:
```shell
rm -rf build dist
```
At the root of your repo, run
```bash
python -m build .
```
This will create a folders named `dist` with the new versions of your package.
#### 10. Upload the package to PyPI Test
> [!IMPORTANT]
> Do not skip this step. It is important to test the package before uploading it to the main PyPI server.
```shell
twine upload dist/* -r testpypi
```
Then in a fresh environment containing all dependencies you need, try to install your new package from the PyPI test server.
```bash
pip install -i https://test.pypi.org/simple/ trl
```
You might get errors for missing dependencies since the PyPI test server does not contain all packages like PyPI does. To make sure you have everything you can do:
```bash
pip install trl
pip uninstall trl
```
(the second line will remove trl but keep all its dependencies).
Also make sure you can actually use the package! Run the following line:
```bash
python -c "from trl import *"
```
along with anything that tests:
- the core feature of your package
- the new features you’re adding in the release
#### 11. Publish on PyPI
> [!WARNING]
> This can't be reverted. Make sure you have tested everything before doing this step.
```shell
twine upload dist/*
```
#### 12. Create a GitHub Release
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
2. Click **Draft a new release**.
3. Select the `v{major}.{minor}.0` tag you just created in step 7.
4. Add a title (`v{major}.{minor}.0`) and a short description of what’s new.
5. Click **Publish Release**.
#### 13. Bump to dev version
1. Create a branch `bump-dev-version-{major}.{minor+1}` from `main` and checkout to it.
```shell
git checkout -b bump-dev-version-{major}.{minor+1}
```
2. Change the version in the following files:
1. `trl/__init__.py`
```diff
- __version__ = "{major}.{minor}.0"
+ __version__ = "{major}.{minor+1}.0.dev0"
```
2. `setup.cfg`
```diff
- version = {major}.{minor}.0
+ version = {major}.{minor+1}.0.dev0
```
3. Commit and push these changes
```shell
git add trl/__init__.py setup.cfg
git commit -m '⬆️ Bump dev version'
git push origin bump-dev-version-{major}.{minor+1}
```
4. Create a pull request from `bump-dev-version-{major}.{minor+1}` to `main`, named `⬆️ Bump dev version`, and request urgent review.
5. Once the pull request is approved, merge it into `main`.
6. The codebase is now ready for the next development cycle, inform the team in the #trl-internal channel.
## Making a patch release
#### 1. Ensure your local repository is up to date with the upstream repository
```bash
git checkout v{major}.{minor}-release
git pull origin main
```
#### 2. Cherry-pick the changes you want to include in the patch release
```bash
git cherry-pick <commit-hash-0>
git cherry-pick <commit-hash-1>
...
```
#### 3. Change the version in the following files
- `trl/__init__.py`
```diff
- __version__ = "{major}.{minor}.{patch-1}"
+ __version__ = "{major}.{minor}.{patch}"
```
- `setup.cfg`
```diff
- version = {major}.{minor}.{patch-1}
+ version = {major}.{minor}.{patch}
```
#### 4. Commit and push these changes
```shell
git add trl/__init__.py setup.cfg
git commit -m 'Release: {major}.{minor}.{patch}'
git push origin v{major}.{minor}-release
```
#### 5. Wait for the CI to pass
#### 6. Add a tag in git to mark the release
```shell
git tag -a v{major}.{minor}.{patch} -m 'Adds tag v{major}.{minor}.{patch} for PyPI'
git push origin v{major}.{minor}.{patch}
```
#### 7. Create the wheels for your release
These are the artifacts that will be uploaded to PyPI and installed by users via `pip install trl`.
Clean previous builds:
```shell
rm -rf build dist
```
At the root of your repo, run
```bash
python -m build .
```
This will create a folders named `dist` with the new versions of your package.
#### 8. Upload the package to PyPI Test
> [!IMPORTANT]
> Do not skip this step. It is important to test the package before uploading it to the main PyPI server.
```shell
twine upload dist/* -r testpypi
```
Then in a fresh environment containing all dependencies you need, try to install your new package from the PyPI test server.
```bash
pip install -i https://test.pypi.org/simple/ trl
```
You might get errors for missing dependencies since the PyPI test server does not contain all packages like PyPI does. To make sure you have everything you can do:
```bash
pip install trl
pip uninstall trl
```
(the second line will remove trl but keep all its dependencies).
Also make sure you can actually use the package! Run the following line:
```bash
python -c "from trl import *"
```
along with anything that tests:
- the core feature of your package
- the new features you’re adding in the release
#### 9. Publish on PyPI
> [!WARNING]
> This can't be reverted. Make sure you have tested everything before doing this step.
```shell
twine upload dist/*
```
#### 10. Create a GitHub Release
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
2. Click **Draft a new release**.
3. Select the `v{major}.{minor}.{patch}` tag you just created in step 7.
4. Add a title (`v{major}.{minor}.{patch}`) and a short description of what’s new.
5. Click **Publish Release**.

167
RELEASE.md Normal file
View File

@ -0,0 +1,167 @@
# Making a release
> [!NOTE]
> VERSION needs to be formatted following the `v{major}.{minor}.{patch}` convention. We need to follow this convention to be able to retrieve versioned scripts.
## Major/Minor Release
### 1. Ensure your local repository is up to date with the upstream repository
```bash
git checkout main
git pull origin main
```
> [!WARNING]
> Do not merge other pull requests into `main` until the release is done. This is to ensure that the release is stable and does not include any untested changes. Announce internally (#trl-internal) to other maintainers that you are doing a release and that they must not merge PRs until the release is done.
### 2. Create a release branch from main
```bash
git checkout -b release-v{major}.{minor}
```
### 3. Change the version in the following files
- `.github/workflows/tests_latest.yml`:
```diff
- with: { ref: v{major}.{minor-1}-release }
+ with: { ref: v{major}.{minor}-release }
```
- `CITATION.cff`
```diff
- version: "{major}.{minor-1}"
+ version: "{major}.{minor}"
```
- `VERSION`
```diff
- {major}.{minor}.0.dev0
+ {major}.{minor}.0
```
### 4. Commit and push these changes
```shell
git add .github/workflows/tests_latest.yml CITATION.cff VERSION
git commit -m 'Release: {major}.{minor}'
git push origin release-v{major}.{minor}
```
### 5. Create a pull request
from `release-v{major}.{minor}` to `main`, named `Release: v{major}.{minor}`, wait for tests to pass, and request a review.
### 6. Once the pull request is approved, merge it into `main`
It will automatically publish the new version of the package on PyPI.
### 7. Add a tag in git to mark the release
```shell
git checkout main
git pull origin main
git tag -a v{major}.{minor}.0 -m 'Adds tag v{major}.{minor}.0 for PyPI'
git push origin v{major}.{minor}.0
```
### 8. Create a branch `v{major}.{minor}-release` for future patch releases
```shell
git checkout -b v{major}.{minor}-release
git push origin v{major}.{minor}-release
```
This ensures that future patch releases (`v{major}.{minor}.1`, `v{major}.{minor}.2`, etc.) can be made separately from `main`.
### 9. Create a GitHub Release
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
2. Click **Draft a new release**.
3. Select the `v{major}.{minor}.0` tag you just created in step 7.
4. Add a title (`v{major}.{minor}.0`) and a short description of what’s new.
5. Click **Publish Release**.
### 10. Bump to dev version
1. Create a branch `bump-dev-version-{major}.{minor+1}` from `main` and checkout to it.
```shell
git checkout -b bump-dev-version-{major}.{minor+1}
```
2. Change the version in file `VERSION`:
```diff
- {major}.{minor}.0
+ {major}.{minor+1}.0.dev0
```
3. Commit and push these changes
```shell
git add VERSION
git commit -m '⬆️ Bump dev version'
git push origin bump-dev-version-{major}.{minor+1}
```
4. Create a pull request from `bump-dev-version-{major}.{minor+1}` to `main`, named `⬆️ Bump dev version`, and request urgent review.
5. Once the pull request is approved, merge it into `main`.
6. The codebase is now ready for the next development cycle, inform the team in the #trl-internal channel.
## Making a patch release
### 1. Ensure your local repository is up to date with the upstream repository
```bash
git checkout v{major}.{minor}-release
git pull origin main
```
### 2. Cherry-pick the changes you want to include in the patch release
```bash
git cherry-pick <commit-hash-0>
git cherry-pick <commit-hash-1>
...
```
### 3. Change the version in the file `VERSION`
```diff
- {major}.{minor}.{patch-1}
+ {major}.{minor}.{patch}
```
### 4. Commit and push these changes
```shell
git add VERSION
git commit -m 'Release: {major}.{minor}.{patch}'
git push origin v{major}.{minor}-release
```
### 5. Wait for the CI to pass
The CI will automatically publish the new version of the package on PyPI.
### 6. Add a tag in git to mark the release
```shell
git tag -a v{major}.{minor}.{patch} -m 'Adds tag v{major}.{minor}.{patch} for PyPI'
git push origin v{major}.{minor}.{patch}
```
#### 7. Create a GitHub Release
1. Go to the repo’s [releases section](https://github.com/huggingface/trl/releases) on GitHub.
2. Click **Draft a new release**.
3. Select the `v{major}.{minor}.{patch}` tag you just created in step 7.
4. Add a title (`v{major}.{minor}.{patch}`) and a short description of what’s new.
5. Click **Publish Release**.

1
VERSION Normal file
View File

@ -0,0 +1 @@
0.22.0.dev0

2
setup.cfg
View File

@ -1,6 +1,6 @@
[metadata] [metadata]
name = trl name = trl
version = 0.22.0.dev0 version = file: VERSION
description = Train transformer language models with reinforcement learning. description = Train transformer language models with reinforcement learning.
long_description = file: README.md long_description = file: README.md
long_description_content_type = text/markdown long_description_content_type = text/markdown

11
trl/__init__.py
View File

@ -12,13 +12,20 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
__version__ = "0.22.0.dev0" from pathlib import Path
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .import_utils import OptionalDependencyNotAvailable, _LazyModule, is_diffusers_available from .import_utils import OptionalDependencyNotAvailable, _LazyModule, is_diffusers_available
# Read version from VERSION file
_version_file = Path(__file__).parent.parent / "VERSION"
try:
with open(_version_file, encoding="utf-8") as f:
__version__ = f.read().strip()
except FileNotFoundError:
__version__ = "unknown"
_import_structure = { _import_structure = {
"scripts": ["DatasetMixtureConfig", "ScriptArguments", "TrlParser", "get_dataset", "init_zero_verbose"], "scripts": ["DatasetMixtureConfig", "ScriptArguments", "TrlParser", "get_dataset", "init_zero_verbose"],
"data_utils": [ "data_utils": [