MNT Remove Python 3.8 since it's end of life (#2135)

The end of life of Python 3.8 has arrived:

https://devguide.python.org/versions/

Therefore, Python 3.8 is removed from CI.

By default, Python 3.11 is now used.

Python 3.12 should be added to the CI matrix now, but that's for a
separate PR.

Also fixed:

The workflow tried to build on top of docker/README.md because globbing
was too broadly defined.

Reduce unnecessary steps to hopefully get disk space usage down, as
GitHub action currently fails with not enough disk space.
This commit is contained in:
Benjamin Bossan
2024-10-22 16:43:53 +02:00
committed by GitHub
parent 7717550a08
commit 095e86c036
12 changed files with 29 additions and 45 deletions

View File

@ -20,7 +20,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: 3.11
- name: Install requirements
run: |

View File

@ -4,7 +4,7 @@ on:
pull_request:
paths:
# Run only when DockerFile files are modified
- "docker/**"
- "docker/*/Dockerfile"
jobs:
get_changed_files:
name: "Build all modified docker images"
@ -18,7 +18,7 @@ jobs:
id: changed-files
uses: tj-actions/changed-files@1c8e6069583811afb28f97afeaf8e7da80c6be5c #v42
with:
files: docker/**
files: docker/*/Dockerfile
json: "true"
- name: Run step if only the files listed above change
if: steps.changed-files.outputs.any_changed == 'true'

View File

@ -17,7 +17,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
python-version: "3.11"
cache: "pip"
cache-dependency-path: "setup.py"
- name: Install dependencies
@ -34,7 +34,7 @@ jobs:
# TODO: remove 'fail-fast' line once timeout issue from the Hub is solved
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.9", "3.10", "3.11"]
os: ["ubuntu-latest", "macos-12", "windows-latest"]
runs-on: ${{ matrix.os }}
steps:

View File

@ -1,6 +1,6 @@
# PEFT Docker images
Here we store all PEFT Docker images used in our testing infrastructure. We use python 3.8 for now on all our images.
Here we store all PEFT Docker images used in our testing infrastructure. We use python 3.11 for now on all our images.
- `peft-cpu`: PEFT compiled on CPU with all other HF libraries installed on main branch
- `peft-gpu`: PEFT complied for NVIDIA GPUs wih all other HF libraries installed on main branch

View File

@ -4,7 +4,7 @@
# Use base conda image to reduce time
FROM continuumio/miniconda3:latest AS compile-image
# Specify py version
ENV PYTHON_VERSION=3.8
ENV PYTHON_VERSION=3.11
# Install apt libs - copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
RUN apt-get update && \
apt-get install -y curl git wget software-properties-common git-lfs && \

View File

@ -4,7 +4,7 @@
# Use base conda image to reduce time
FROM continuumio/miniconda3:latest AS compile-image
# Specify py version
ENV PYTHON_VERSION=3.8
ENV PYTHON_VERSION=3.11
# Install apt libs - copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
RUN apt-get update && \
apt-get install -y curl git wget software-properties-common git-lfs && \

View File

@ -4,7 +4,7 @@
# Use base conda image to reduce time
FROM continuumio/miniconda3:latest AS compile-image
# Specify py version
ENV PYTHON_VERSION=3.8
ENV PYTHON_VERSION=3.11
# Install apt libs - copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
RUN apt-get update && \
apt-get install -y curl git wget software-properties-common git-lfs && \

View File

@ -4,7 +4,7 @@
# Use base conda image to reduce time
FROM continuumio/miniconda3:latest AS compile-image
# Specify py version
ENV PYTHON_VERSION=3.8
ENV PYTHON_VERSION=3.11
# Install apt libs - copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
RUN apt-get update && \
apt-get install -y curl git wget software-properties-common git-lfs && \

View File

@ -4,23 +4,18 @@
# Use base conda image to reduce time
FROM continuumio/miniconda3:latest AS compile-image
# Specify py version
ENV PYTHON_VERSION=3.8
ENV PYTHON_VERSION=3.11
# Install apt libs - copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
# Install audio-related libraries
RUN apt-get update && \
apt-get install -y curl git wget software-properties-common git-lfs && \
apt-get install -y curl git wget software-properties-common git-lfs ffmpeg libsndfile1-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists*
# Install audio-related libraries
RUN apt-get update && \
apt install -y ffmpeg
RUN apt install -y libsndfile1-dev
RUN git lfs install
# Create our conda env - copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
RUN conda create --name peft python=${PYTHON_VERSION} ipython jupyter pip
RUN python3 -m pip install --no-cache-dir --upgrade pip
# Below is copied from https://github.com/huggingface/accelerate/blob/main/docker/accelerate-gpu/Dockerfile
# We don't install pytorch here yet since CUDA isn't available
@ -35,25 +30,20 @@ FROM nvidia/cuda:12.2.2-devel-ubuntu22.04 AS build-image
COPY --from=compile-image /opt/conda /opt/conda
ENV PATH /opt/conda/bin:$PATH
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-c"]
RUN source activate peft && \
python3 -m pip install --no-cache-dir bitsandbytes optimum auto-gptq
# Add autoawq for quantization testing
RUN source activate peft && \
python3 -m pip install --no-cache-dir https://github.com/casper-hansen/AutoAWQ/releases/download/v0.2.4/autoawq-0.2.4-cp38-cp38-linux_x86_64.whl
RUN source activate peft && \
python3 -m pip install --no-cache-dir https://github.com/casper-hansen/AutoAWQ_kernels/releases/download/v0.0.6/autoawq_kernels-0.0.6-cp38-cp38-linux_x86_64.whl
# Install apt libs
RUN apt-get update && \
apt-get install -y curl git wget && \
apt-get clean && \
rm -rf /var/lib/apt/lists*
# Add eetq for quantization testing
RUN source activate peft && \
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-c"]
RUN source activate peft && \
python3 -m pip install --no-cache-dir bitsandbytes optimum auto-gptq && \
# Add autoawq for quantization testing
python3 -m pip install --no-cache-dir https://github.com/casper-hansen/AutoAWQ/releases/download/v0.2.6/autoawq-0.2.6-cp311-cp311-linux_x86_64.whl && \
python3 -m pip install --no-cache-dir https://github.com/casper-hansen/AutoAWQ_kernels/releases/download/v0.0.8/autoawq_kernels-0.0.8-cp311-cp311-linux_x86_64.whl && \
# Add eetq for quantization testing
python3 -m pip install git+https://github.com/NetEase-FuXi/EETQ.git
# Activate the conda env and install transformers + accelerate from source
@ -65,17 +55,13 @@ RUN source activate peft && \
torchao \
git+https://github.com/huggingface/transformers \
git+https://github.com/huggingface/accelerate \
peft[test]@git+https://github.com/huggingface/peft
peft[test]@git+https://github.com/huggingface/peft \
# Add aqlm for quantization testing
aqlm[gpu]>=1.0.2 \
# Add HQQ for quantization testing
hqq
# Add aqlm for quantization testing
RUN source activate peft && \
pip install aqlm[gpu]>=1.0.2
# Add HQQ for quantization testing
RUN source activate peft && \
pip install hqq
RUN source activate peft && \
pip freeze | grep transformers
RUN echo "source activate peft" >> ~/.profile

View File

@ -16,7 +16,7 @@ rendered properly in your Markdown viewer.
# Installation
Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 PEFT. 🤗 PEFT is tested on **Python 3.8+**.
Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 PEFT. 🤗 PEFT is tested on **Python 3.9+**.
🤗 PEFT is available on PyPI, as well as GitHub:

View File

@ -54,7 +54,7 @@ setup(
packages=find_packages("src"),
package_data={"peft": ["py.typed", "tuners/boft/fbd/fbd_cuda.cpp", "tuners/boft/fbd/fbd_cuda_kernel.cu"]},
entry_points={},
python_requires=">=3.8.0",
python_requires=">=3.9.0",
install_requires=[
"numpy>=1.17",
"packaging>=20.0",
@ -76,7 +76,7 @@ setup(
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.11",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
],
)

View File

@ -2963,10 +2963,8 @@ class PeftHqqGPUTests(unittest.TestCase):
assert cc_matrix.min() > 0.97
# TODO: unskip the tests once https://github.com/casper-hansen/AutoAWQ/issues/466 is fixed
@require_torch_gpu
@require_auto_awq
@pytest.mark.skip(reason="Needs https://github.com/casper-hansen/AutoAWQ/issues/466 to be fixed first")
class PeftAwqGPUTests(unittest.TestCase):
r"""
Awq + peft tests