-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] [MoE] Typo in Token Drop policy's default value #812
Comments
Thanks for the fix, @passaglia! We've already got an internal MR that's been reviewed to fix this issue, so it should be synced to GitHub soon. Thanks again! |
Great, thank you @yanring ! I'll close this issue once the GitHub repo is updated. |
Fixed in 7968fd6 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
In the new MoE Token Drop code, the code and documentation expect drop_policy to be either prob or position, but the current default argument is probs.
https://github.com/NVIDIA/Megatron-LM/blob/db3a3f79d1cda60ea4b3db0ceffcf20c5760e11d/megatron/core/transformer/moe/moe_utils.py#L272C5-L272C16
@yanring
Proposed fix
#811
The text was updated successfully, but these errors were encountered: