The multi-task prompt tuning of PEFT library

[lhyscau]

I can't find the code of the paper MULTITASK PROMPT TUNING ENABLES PARAMETER-EFFICIENT TRANSFER LEARNING from the author?Does the code complement of the peft just refer to the paper and written by some expert?

[BenjaminBossan]

I'm not sure I get your question, are you asking for examples of the code being used? Here is one:

https://github.com/huggingface/peft/blob/main/examples/conditional_generation/multitask_prompt_tuning.ipynb

If you're asking where the code is in peft, it's here:

https://github.com/huggingface/peft/tree/main/src/peft/tuners/multitask_prompt_tuning

[lhyscau]

I would like to know if the implementation of the multi task prompt tuning code of the peft library is written directly based on the original paper? Because I can't find the implementation code for this paper

[BenjaminBossan]

Yes, it should be based on the paper, but I'm not sure if everything from the paper has been added. Please check PR #400, where this was added.

[lhyscau]

There is a question about multitask_prompt_tuning. If word_embedding can be updated (according to the class prompt_tuning inherited by multitask_prompt_tuning in the code), will the word_embedding of the model be permanently changed after execution, so that it is not the original one? What I understand is that the checkpoint for weight saving is saved based on the selected word embedding, which is indices.

Or does it read word_embedding into memory, but the code does not write the changed word_embedding in memory back to SSD?

I look at the forward in multitask_prompt_tuning. Every time I upload indices, I take the vector of shared_matrix from word_embedding.

[BenjaminBossan]

If I understand your question correctly, you're asking if the use of multitask prompt tuning results in the original embedding layer to be modified. From my understanding, that is not the case, the original embeddings are left untouched. Instead, a new embedding layer is introduced which is being trained.

Pinging @mayank31398 in case there is more to add.