Correct check for SDPA in Vision Language Models

[zucchini-nlp]

System Info

In current implementation of VLMs, the "_supports_sdpa" attribute checks and activates SDPA attention only for the language model. For example in Llava

It should also check and if available use SDPA attention for vision tower. Current implementations of the most common vision tower, CLIP, do not support SDPA (this PR adds sdpa for clip)

We can raise a warning for composite models if only one part support sdpa, but other does not. So that the user knows what is happening in the background.

Verified models

• BLIP-2
• InstructBLIP
• KOSMOS-2
• LLaVa
• LLaVa-NeXT
• Idefics
• Idefics2

[NielsRogge]

Edited your issue to include a list of models to check ;) feel free to expand