New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
flash-attention is not running, although is_flash_attn_2_available() returns true #30547
Comments
Yeah, so the PR to integrate Phi-3 with transformers has already been merged here. There hasn't been a stable release yet, which is why there's a difference between the pip version and when you install directly from source. So, you have to do the latter for now. That being said, I also see the message Maybe @gugarosa can help, since this was his PR. |
Hi! I get a lot of warning like the one below, it does not crash but it does not seem to ever finish as well. Has anyone ran into something similar and was able to solve it? (I'm using a cuda with version 12.1 and pytorch 2.3) "/usr/local/cuda-12/include/cusparse.h:254:20: note: declared here |
i am trying to use Phi-3-128k model, getting this problem .modeling_phi3:You are not running the flash-attention implementation, expect numerical differences. has anyone faced this error. How to solve this |
System Info
transformers
version: 4.40.1Who can help?
@Narsil
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
error: [Errno 2] No such file or directory: 'c:\users\79314\anaconda3\envs\comfyuitest\lib\site-packages\transformers-4.41.0.dev0py3.11.egg\transformers\models\deprecated\trajectory_transformer\pycache\convert_trajectory_transformer_original_pytorch_checkpoint_to_pytorch.cpython-311.pyc.1368162759184'
Expected behavior
How to run with flash-attention?
My curiosity:
Is there another reason limiting flash-attention,except is_flash_attn_2_available()
Why I can't find "You are not running the flash-attention implementation, expect numerical differences." Which file contains these message?
How can I build the latest transformers?
The text was updated successfully, but these errors were encountered: