Hi @fxmarty I can take a look at this issue. Of I can ask questions if necessary. Or has anyone taken it already?

does someone know if longT5 and all T5 models are blocked by bias support in flash attention ?

Dao-AILab/flash-attention#617

Hi @davidan5 are you working on the implementation?

@ENate I was trying to understand the status and have an estimation of the code change to see if I can contribute.

I see.

I'm interested in taking a look at this for the Mistral model if that's still needed. Otherwise, please let me know if there are any other models that still need some work. Thanks

Is LongT5 still open?

Mistral is already covered! LongT5 if it is like T5 and has attention bias that might not be supported

Oh yea, looks like you added support for Mistral/Mixtral last month.

It doesn't seem to be supported for BERT yet (I think someone else is working on FA2 but not SDPA), so I'll take a crack at it. It looks like there is a config for relative position embeddings for BERT, so I'll just have it fallback to the original attention for configs using relative position embeddings.

@ArthurZucker - Please let me know if you know if someone else is already working on SDPA for BERT and I can look for something else to do. Thanks!

Not sure anyone is working on that but bert is already so small that I doubt it will have a lot of impact on perf!

@ArthurZucker for the T5 family of models, attention bias is required, so flash-attention won't work for now but torch SDPA can still use the memory efficient kernel from xformers, right? I did some benchmarking with Chronos models (based on T5 architecture) here (amazon-science/chronos-forecasting#33) and there's a clear speedup when using torch SDPA.

@abdulfatir That's correct

I can open a PR for T5 with SDPA then. Are there specific things that I should know of or a reference that can look at?

@abdulfatir For sure, some specific things that are good to know:

pytorch/pytorch#108108 (is_causal=True may not do what you expect)
pytorch/pytorch#110213 (You need

example of a PR: #29108

FYI going forward we should rather use

Hey @abdulfatir just wanted to check in if you are still working on dropping a PR to add SDPA support for T5? It would tremendously help accelerating diffusion models that use T5.

@sayakpaul sorry, I was on vacation. Will look into this now and maybe open a PR in a couple of days. I didn't know that there were diffusion models using the T5 arch. Pretty cool!

Amongst the open ones that are available, the most notable ones are:

• PixArt-Alpha
• PixArt-Sigma
• SD3 (will soon be open)
• DeepFloyd IF

So, it will be huge :)

@fxmarty @ArthurZucker @sayakpaul I have opened a PR #30375 for T5. I still have a couple of questions due to some tests failing. Let's discuss those on the PR.

Out of interest, is there any merit in implementing SDPA/FA2 for the DeBERTa family of models? Parametrically, the models are quite small but have relatively high memory costs due to (among other things) the disentangled attention mechanism - however I'm uncertain whether SDPA/FA2 would have a material effect on these costs.

There was a PR to refactor the DeBERTa modelling scripts (#22105), which appears to have been set aside for now, and I am unsure whether any potential implementations for DeBERTa were gated by that or waiting for it to avoid clashes. There are two tangentially related PRs (#27734, relating to DeBERTa and #28802, relating to BERT, but potentially usable as a starting point) but both appear to have been stuck awaiting some final review or merging for a month or two now.

I think there's still significant utility in the DeBERTa and other BERT family models given their lightweight size and still-leading functionality for text classification, although of course the limelight is held by larger text-completion/multi-modal models, especially with the rapid release of Phi-3, Llama-3 etc. over the past few weeks.

While most of DeBERTa's use-cases employ short, batched input strings which diminishes the effect of memory efficient optimizations, nevertheless there are tasks, particularly entailment detection between complex sentences, which can create large input strings and very high memory costs.

@His-Wardship - Thanks for your input. We have finally managed to merge #28802 (SDPA for BERT), so hopefully this will unlock other SDPA implementations for other models. I plan to open up a PR for RoBERTa and do some testing to see how if there are any performance gains there.

Regarding DeBERTa, I'm somewhat skeptical whether or not SDPA has support for its disentangled attention mechanism. I think the disentangled attention mechanism requires some extra operations that may not be supported in SDPA.