You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today we’re releasing the next step: QDoRA. This is just as memory efficient and scalable as FSDP/QLoRA, and critically is also as accurate for continued pre-training as full weight training. We think that this is likely to be the best way for most people to train1 language models. We’ve ran preliminary experiments on Llama 2, and completed some initial ones on Llama 3. The results are extremely promising.
Only linear and DoRA only supports linear and Conv2d layers at the momement.
DoRA introduces a bigger overhead than pure LoRA, so it is recommended to merge weights for inference.
DoRA should work with weights quantized with bitsandbytes (“QDoRA”). However, issues have been reported when using QDoRA with DeepSpeed Zero2.
Would be great to use this method to optimize cost-efficient and assumably even better fine-tuning!
The text was updated successfully, but these errors were encountered:
s-smits
changed the title
[Feature Request] Support for QDoRA: Efficient quantized fine-tuning close to fp16
[Feature Request] Support for QDoRA: Efficient quantized fine-tuning
Apr 23, 2024
https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html
Impelementation Answer.AI:
AnswerDotAI/fsdp_qlora#51
Caveats (from HF implementation):
Would be great to use this method to optimize cost-efficient and assumably even better fine-tuning!
The text was updated successfully, but these errors were encountered: