Update to transformers 4.39.1+

[vidyasiv]

Feature request

If an update is already planned, could you please let us know the expected timeline. We had begun working on this and found we needed a more changes (that could also affect other models) than we were comfortable making at once.

Motivation

In order to enable CohereForAI/c4ai-command-r-v01 an update is needed to v4.39.1

Your contribution

Part of the upgrade work can be found (very cohereforai specific) : https://github.com/vidyasiv/optimum-habana/tree/cohereforai (only works for batch size=1 for run_generation.py) and higher batch sizes fail as it needs further changes.
Contributors: @skavulya and @vidyasiv

[regisss]

Transformers v4.40 should be released soon so I'll work on directly jumping from 4.38 to 4.40

[vidyasiv]

Thanks @regisss , do you have a rough estimate of when that could be i.e end of April or earlier etc?

[regisss]

I'll push a branch compatible with Transformers v4.40 next week. If there is no major regression, it should be merged a few days after.

[regisss]

Upgraded to Transformers v4.40 in #1027.