-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to reproduce the same training process when using "train_from" #2006
Comments
This is roughly what I intended with #1826, but it's not compatible with all the changes we introduced in 2.0. |
Thanks for your efforts. |
Regarding this issue I implemented the following: |
I made an attempt for this feature in this PR #2520. The idea is to skip to the saved text line in each corpus when a training is resumed. |
Dear,
When the model training was forced to stop due to an accident. I use the opt "train_from" to continue training from the checkpoint. But the result is different from the tranining from start to finish without stopping:
Note that i fixed all random seed.
So it is very convenient that If a reproduction mechanism can be added into the code base.
Any help will be greatly appreciated.
The text was updated successfully, but these errors were encountered: