Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expansion factor choices #14

Open
zhaoyanlyu opened this issue Sep 11, 2023 · 0 comments
Open

Expansion factor choices #14

zhaoyanlyu opened this issue Sep 11, 2023 · 0 comments

Comments

@zhaoyanlyu
Copy link

Thank you for the clear and well-executed implementation.

Following up on this issue: #11

May I kindly ask why you chose to expand the token-mixing MLP while bottlenecking the channel-mixing MLP? Is there a particular reason behind this design, or is it simply because this setup provides the best performance?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant