Resolves#2695
For some PEFT methods, there was a bit of a mess when it comes to how
the init_weights argument was set in test_custom_models.py. The default
kwargs for the tests should be that the PEFT method is initialized as an
identity transform, and for specific tests we want to disable that. Note
that most PEFT methods are initialized by default to be identity
transforms, which is why the argument does not need to be set
explicitly, but it's not true for all PEFT methods.
With this PR, SHiRA, C3A, and FourierFT are now initialized to be
consistent with this. This made it possible to remove some extra
handling of those methods which was intermingled with certain tests.
Moreover, test_custom_models.py now uses the set_init_weights_false
helper function where appropriate.
While working on this, I also cleaned up a bit the docs for the
init_weights arguments of these PEFT methods where appropriate.
I added some clarifying comments.
For test_unload_adapter, I simplified a config type check and
rewrote it to load the base model only once.
---------
Co-authored-by: githubnemo <githubnemo@users.noreply.github.com>
Also fix a minor import nit where `TrainableTokensWrapper` was not
added to `utils/__init__.py`. Fixed the corresponding imports as well.
Another housekeeping job is to move hub_online_once to testing_utils.py since it has
grown to be used in a lot of places and testing_utils.py is the better place to keep
such utilities.
* align xpu behavior w/ CUDA in lorafa
for lorafa and randlora: i can see peft requirement torch >=1.13, and in 1.13, torch already has a device agnostic torch.autocast, switch to use the device agnostic API to also cover xpu
clean codes in tests folder to use device agnostic clean cache API. Before this PR, some test cases use device-agnostic clean cache API, some use torch.cuda.xx; after this PR, all use device-agnostic clean cache API
enable gptqmodel multi-device test case on XPU, enable torchao test cases on XPU
* randlora default dtype to bfloat16, align CUDA behavior
* refine randlora&vblora test, refine bnb test skip message
* enable torchao tests on XPU, all passed on torchao 0.11.0
* use accelerate utils
Supports torch AO quantization. Currently supported:
- int8_weight_only
- int8_dynamic_activation_int8_weight
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* allow bitsandbytes integration test selection
* fix typo: mutli -> multi
* enable tests to run on >2 GPUs
* fix for >3 GPUs, due to artidoro/qlora #186
* fix formatting
Some tests would currently fail with torch.compile, not because there is
anything wrong with how PEFT works with compiled models, but simply
because of the way the tests are written. This is because when models
are compiled, the keys of the state dict change. Tests have now been
adapted to unwrap the compiled model first before getting the state
dict.
Note that the mentioned issue does not affect saving and loading,
because save_pretrained is already called on the original module, so
there is no issue with mismatched keys.
Also fixed the docstring of get_peft_model_state_dict.
