Support cuDNN 9.0

[kmaehashi]

Tasks

• Read Release Notes (cuTENSOR / cuSPARSELt / cuDNN / NCCL).
• Fix code and CI to support the new vesrion.
  • FlexCI: Update .pfnci/schema.yaml and .pfnci/matirx.yaml.
  • Library Installer: Update cupyx/tools/install_library.py.
• Backport the above PR.
• Fix documentation.
  • Update Requirements section in the Installation Guide
• Backport the above PR.
• Implement new features using the version, if any.

[mnicely]

@kmaehashi I wanted to let you know that we have the new cuDNN Frontend that may make integration and fusions easier. Let me know if you have any questions

[kmaehashi]

Thanks for the info, @mnicely! It's great to see the stable release of an official high-level library with Python binding!

Actually, CuPy's cuDNN wrappers haven't been updated for years - it has been maintained to support new cuDNN releases but not adopting new features. They were originally added for use by Chainer and I think most users today yare not relying on the CuPy cuDNN features. It's indicated by that there are no bug reports nor feature requests recently.

Now that cuDNN wrappers are officially provided, I think it would make sense to deprecate cuDNN support in CuPy.

cc/ @leofang

[leofang]

Now that cuDNN wrappers are officially provided, I think it would make sense to deprecate cuDNN support in CuPy.

cc: @acc-mu3n for vis (for context in the private Gitter discussion)

[leofang]

Hi @honnibal you mentioned in the past that spaCy uses cuDNN through CuPy. Is it still the case? We wanna estimate the impact of dropping support of cuDNN in CuPy, but oddly we can't find any trace of cuDNN in the spaCy codebase (either its addition or removal!), and we are quite puzzled. Could you help us understand better?

cc: @jakirkham for vis