Asking for the meaning of "rotary_percentage" in Rotary Position Embedding.py #8588
Unanswered
YOONSEOKHEO
asked this question in
Q&A
Replies: 1 comment
-
+1 Can we get an idea of what this does? I know it modifies the head dimension but a paper or something of how this applies would be greatly appreciated |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi.
I am digging into somewhere in Nemo nowadays.
I cannot follow up any cases for downscaling the "rotary_dim" by using "config.rotary_percentage".
Here is the URL:
NeMo/nemo/collections/nlp/modules/common/megatron/language_model.py
Line 565 in d2283e3
Does anyone share the concept of this variable and its process?
I am very appreciating your comments in advance.
Beta Was this translation helpful? Give feedback.
All reactions