Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lora训练llama 貌似不支持? #51

Open
wind91725 opened this issue Apr 15, 2023 · 3 comments
Open

lora训练llama 貌似不支持? #51

wind91725 opened this issue Apr 15, 2023 · 3 comments

Comments

@wind91725
Copy link

如题:命令如下:
python pretrain.py --pretrained_model_path models/llama-7b.bin --dataset_path datasets/ceshi --spm_model_path /u01/wangcheng/llm/llama/tokenizer.model --config_path models/llama/7b_config.json --output_model_path models/llama_zh_7b --world_size 5 --data_processor lm --total_steps 300000 --save_checkpoint_steps 5000 --batch_size 24 --use_lora --lora_dropout 0.05

只运行到Using distributed mode for training. 就结束了?

@Daniel-1997
Copy link

我和你遇到了用样的问题,请问你解决了吗?

@hepj987
Copy link

hepj987 commented Jun 9, 2023

用最新版本的项目是可以用lora训练的,只是很奇怪,看他介绍lora是分两步走的,第一步--pretrained_model_path models 和--use_lora --lora_dropout 0.05只训练一个lora权重,第二步通过--lora_pretrained_model_path --pretrained_model_path models --use_lora --lora_dropout 0.05加载训练好的lora权重再训练。

@hepj987
Copy link

hepj987 commented Jun 9, 2023

用最新版本的项目是可以用lora训练的,只是很奇怪,看他介绍lora是分两步走的,第一步--pretrained_model_path models 和--use_lora --lora_dropout 0.05只训练一个lora权重,第二步通过--lora_pretrained_model_path --pretrained_model_path models --use_lora --lora_dropout 0.05加载训练好的lora权重再训练。

但是按他这个流程走,lora训练的时候 loss不降 acc也不提升

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants