-
Notifications
You must be signed in to change notification settings - Fork 223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEA] Log the shuffle manager class when shuffle manager is misconfigured #10172
Comments
I am planning to work on this. |
For the context regarding the classloader. It is hard to reproduce but we have seen cases triggering
although the ShuffleManager class was correctly specified. In those cases rapidsShuffleManager is not instance of RapidsShuffleManagerLike. Thus. presumably classOf[RapidsShuffleManagerLike] is different than RapidsShuffleManagerLike reachable via org.apache.spark.SparkEnv.get.shuffleManager.getClass.getSuperclass.getInterfaces |
Is your feature request related to a problem? Please describe.
Instead of just reporting "Cannot initialize ..."
spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuShuffleEnv.scala
Lines 106 to 113 in 47047a9
The text was updated successfully, but these errors were encountered: