-
Notifications
You must be signed in to change notification settings - Fork 239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increase EMA-parameter during training #82
Comments
@Benjamin-Hansson just to clarify - when did you observe this spiking loss? Was it when the EMA parameter was not updated, or the other way around? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi, I noticed that the EMA-parameter (called beta in the code, τ in the paper) is not updated during training. In the paper they describe that they increase τ from the start value to 1 during training: "Specifically, we set τ = 1 − (1 − τbase) · (cos(πk/K) + 1)/2 with k the current training step and K the maximum number of training steps."
This makes a huge difference to the validation loss at the end of the training.
The text was updated successfully, but these errors were encountered: