Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] The argument --no-position-embedding should be fixed #826

Open
Hoonly opened this issue May 14, 2024 · 0 comments
Open

[BUG] The argument --no-position-embedding should be fixed #826

Hoonly opened this issue May 14, 2024 · 0 comments

Comments

@Hoonly
Copy link

Hoonly commented May 14, 2024

Describe the bug
from the file Megatron-LM/megatron/training/arguments.py

    group.add_argument('--no-position-embedding',
                       action='store_false',
                       help='Disable position embedding. Deprecated: use --position-embedding-type',
                       dest='add_position_embedding')

I can see that this argument is Deprecated, but if we only use --position-embedding-type=rope, the add_position_embedding will accept default value True, conflict to the rope

To Reproduce

Expected behavior

Stack trace/logs

Environment (please complete the following information):

  • Megatron-LM commit ID 0d983e6
  • PyTorch version 2.3.0a0+ebedce2
  • CUDA version V12.3.107
  • NCCL version 2.19.4

Proposed fix
If you have a proposal for how to fix the issue state it here or link to a PR.

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant