frozenleaves/peft
1
0
Fork 0
mirror of https://github.com/huggingface/peft.git synced 2025-10-20 15:33:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add prompt tuning experiment with sample vocab (#2824)

Browse Source 
A new initialization method was added to prompt tuning in #2815. This PR
adds an experiment config for this method to the MetaMathQA benchmark.

Testing locally, this got a test accuracy of 36%, compared to 25% with
random initialization.
This commit is contained in:
Benjamin Bossan
2025-10-13 16:54:45 +02:00
committed by GitHub
parent 25f97e663a
commit 6392935921
2 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
method_comparison/MetaMathQA/experiments/prompt_tuning/llama-3.2-3B-sample_vocab-lr_0.001/adapter_config.json Normal file
View File

@ -0,0 +1,17 @@
{
"auto_mapping": null,
"base_model_name_or_path": null,
"inference_mode": false,
"num_attention_heads": 24,
"num_layers": 28,
"num_transformer_submodules": 1,
"num_virtual_tokens": 200,
"peft_type": "PROMPT_TUNING",
"prompt_tuning_init": "SAMPLE_VOCAB",
"prompt_tuning_init_text": null,
"revision": null,
"task_type": "CAUSAL_LM",
"token_dim": 3072,
"tokenizer_kwargs": null,
"tokenizer_name_or_path": null
}

6
method_comparison/MetaMathQA/experiments/prompt_tuning/llama-3.2-3B-sample_vocab-lr_0.001/training_params.json Normal file
View File

@ -0,0 +1,6 @@
{
"optimizer_kwargs": {
"lr": 1e-3
}
}