Introduction of a method evaluation suite.
We generally face the problem that there is little knowledge on what PEFT methods perform best. To this end we decided to build an evaluation suite that has defined tasks, shared hyper-parameters and can be extended with new tasks and new method configurations over time.
For the sake of comparison we've not decided to incorporate user-submitted results but we encourage users to inspect the results, suggest new experiments and improve the configuration of methods if they're deemed unfavorable.
As of now there's only one task based on the MetaMathQA dataset which has the benefit of being complex while still fitting on a consumer GPU.
Notable changes in this squash:
* Add default training params
The experiment specific training params use the default training params
but can override any parameter from it if needed. However, this way it's
easier to make a change to all experiments (say, I want to change the
base model, I don't need to change each individual
training_parameters.json).
* Add possibility to change attn implementation
However, both flash attention 2 and flex attention are slower on my
system. Thus, stay with default None (-> SDPA).
* Refactor to use GenerationConfig
Allows to more easily use, say, static cache, which is the new default,
as it's faster (apart from the first pass)
* Better parsing of answers
E.g. 1/2 == 0.5
* Keep adapter file by default after train run
But add --clean to delete it.
Keeping the adapter can be useful if the user wants to run further tests
with the trained model.
---------
Co-authored-by: Benjamin Bossan <benjamin.bossan@gmail.com>
