mirror of
https://github.com/huggingface/peft.git
synced 2025-10-20 06:53:46 +08:00
41921013f5
Introduction of a method evaluation suite. We generally face the problem that there is little knowledge on what PEFT methods perform best. To this end we decided to build an evaluation suite that has defined tasks, shared hyper-parameters and can be extended with new tasks and new method configurations over time. For the sake of comparison we've not decided to incorporate user-submitted results but we encourage users to inspect the results, suggest new experiments and improve the configuration of methods if they're deemed unfavorable. As of now there's only one task based on the MetaMathQA dataset which has the benefit of being complex while still fitting on a consumer GPU. Notable changes in this squash: * Add default training params The experiment specific training params use the default training params but can override any parameter from it if needed. However, this way it's easier to make a change to all experiments (say, I want to change the base model, I don't need to change each individual training_parameters.json). * Add possibility to change attn implementation However, both flash attention 2 and flex attention are slower on my system. Thus, stay with default None (-> SDPA). * Refactor to use GenerationConfig Allows to more easily use, say, static cache, which is the new default, as it's faster (apart from the first pass) * Better parsing of answers E.g. 1/2 == 0.5 * Keep adapter file by default after train run But add --clean to delete it. Keeping the adapter can be useful if the user wants to run further tests with the trained model. --------- Co-authored-by: Benjamin Bossan <benjamin.bossan@gmail.com>
146 lines
2.0 KiB
Plaintext
146 lines
2.0 KiB
Plaintext
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
pip-wheel-metadata/
|
|
share/python-wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
|
|
# PyInstaller
|
|
# Usually these files are written by a python script from a template
|
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
*.manifest
|
|
*.spec
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
htmlcov/
|
|
.tox/
|
|
.nox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
*.cover
|
|
*.py,cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
|
|
# Translations
|
|
*.mo
|
|
*.pot
|
|
|
|
# Django stuff:
|
|
*.log
|
|
local_settings.py
|
|
db.sqlite3
|
|
db.sqlite3-journal
|
|
|
|
# Flask stuff:
|
|
instance/
|
|
.webassets-cache
|
|
|
|
# Scrapy stuff:
|
|
.scrapy
|
|
|
|
# Sphinx documentation
|
|
docs/_build/
|
|
|
|
# PyBuilder
|
|
target/
|
|
|
|
# Jupyter Notebook
|
|
.ipynb_checkpoints
|
|
|
|
# IPython
|
|
profile_default/
|
|
ipython_config.py
|
|
|
|
# pyenv
|
|
.python-version
|
|
|
|
# pipenv
|
|
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
|
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
|
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
|
# install all needed dependencies.
|
|
#Pipfile.lock
|
|
|
|
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
|
|
__pypackages__/
|
|
|
|
# Celery stuff
|
|
celerybeat-schedule
|
|
celerybeat.pid
|
|
|
|
# SageMath parsed files
|
|
*.sage.py
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
env/
|
|
venv/
|
|
ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
|
|
# Spyder project settings
|
|
.spyderproject
|
|
.spyproject
|
|
|
|
# Rope project settings
|
|
.ropeproject
|
|
|
|
# mkdocs documentation
|
|
/site
|
|
|
|
# mypy
|
|
.mypy_cache/
|
|
.dmypy.json
|
|
dmypy.json
|
|
|
|
# Pyre type checker
|
|
.pyre/
|
|
|
|
# VSCode
|
|
.vscode
|
|
|
|
# IntelliJ
|
|
.idea
|
|
|
|
# Mac .DS_Store
|
|
.DS_Store
|
|
|
|
# More test things
|
|
wandb
|
|
|
|
# method_comparison logs
|
|
method_comparison/MetaMathQA/cancelled_results/
|
|
method_comparison/MetaMathQA/temporary_results/
|