frozenleaves/peft
1
0
Fork 0
mirror of https://github.com/huggingface/peft.git synced 2025-10-20 15:33:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

FIX: Generation nightly CI failing due to gemma (#2580)

Browse Source 
For a month now, nightly CI has failed with dozens of tests causing this
error:

> RuntimeError: Offset increment outside graph capture encountered
unexpectedly.

(link: https://github.com/huggingface/peft/actions/runs/14850392078/job/41692748031)

It turns out that https://github.com/huggingface/peft/pull/2458, which
added a gemma model to the test suite, is most likely the culprit. Since
that commit, on nightly CI (with GPU), when transformers generates with
gemma, which uses torch.compile, an error can be triggered. For some
reason, this has a side effect on other tests that then results in the
error quoted above.

As is, there is no solution for the gemma issue. To still allow the
tests to run and help discover potential issues, this PR skips the
corresponding gemma tests, which should allow the other tests to pass
again.

I could confirm locally that these tests only fail when the gemma tests
are run in the same session. Hopefully, this generalizes to the CI
environment.

---------

Co-authored-by: githubnemo <githubnemo@users.noreply.github.com>
This commit is contained in:
Benjamin Bossan
2025-06-11 18:01:13 +02:00
committed by GitHub
parent a8b9a6cecc
commit 759bb70ace
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
Makefile
View File

@ -31,8 +31,12 @@ tests_core_multi_gpu:
tests_core_single_gpu: tests_core_single_gpu:
python -m pytest -m single_gpu_tests tests/test_common_gpu.py $(if $(IS_GITHUB_CI),--report-log "core_single_gpu.log",) python -m pytest -m single_gpu_tests tests/test_common_gpu.py $(if $(IS_GITHUB_CI),--report-log "core_single_gpu.log",)
# exclude gemma tests, as generation fails with torch.compile, these failures
# trigger side effects that make other tests fail with 'RuntimeError: Offset
# increment outside graph capture encountered unexpectedly.'
# TODO re-enable gemma once/if it is fixed
tests_common_gpu: tests_common_gpu:
python -m pytest tests/test_decoder_models.py $(if $(IS_GITHUB_CI),--report-log "common_decoder.log",) python -m pytest tests/test_decoder_models.py -k "not gemma" $(if $(IS_GITHUB_CI),--report-log "common_decoder.log",)
python -m pytest tests/test_encoder_decoder_models.py $(if $(IS_GITHUB_CI),--report-log "common_encoder_decoder.log",) python -m pytest tests/test_encoder_decoder_models.py $(if $(IS_GITHUB_CI),--report-log "common_encoder_decoder.log",)
python -m pytest tests/test_gptqmodel.py $(if $(IS_GITHUB_CI),--report-log "gptqmodel_gpu.log",) python -m pytest tests/test_gptqmodel.py $(if $(IS_GITHUB_CI),--report-log "gptqmodel_gpu.log",)