Temporarily disable UI Tab for Spark 4.0 and above

[pan3793]

🔍 Description

Spark 4.0 migrated from javax.servlet to jakarta.servlet in SPARK-47118, which breaks the Kyuubi UI tab.

Describe Your Solution 🔧

Temporarily disable UI Tab for Spark 4.0 and above.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Build Kyuubi with Spark 3.5 and successfully launch the Spark engine with Spark 4.0.0 preview1.

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[yikf]

Hi pan, Why does this break the kyuubi ui and is it a known issue with the preview version?

In general, will the kyuubi spark engine tab continue to not support spark 4.0 and above in the future, or will it only be temporary as we now need to test the preview version and this known issue has not yet been fixed?

[pan3793]

@yikf This is a temporary workaround, eventually, we may need to introduce a shim layer to dynamically bind to Spark UI classes (javax.servlet or jakarta.servlet). Please refer to SPARK-47118 for more context

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 71.42857% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 58.51%. Comparing base (02e1aa3) to head (2abad8c).

Files	Patch %	Lines
...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6390      +/-   ##
============================================
- Coverage     58.55%   58.51%   -0.04%     
  Complexity       24       24              
============================================
  Files           653      653              
  Lines         39974    39976       +2     
  Branches       5489     5490       +1     
============================================
- Hits          23406    23393      -13     
- Misses        14070    14079       +9     
- Partials       2498     2504       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

Thanks, merged to master/1.9

[pan3793]

Reverted because I have addressed the compatibility issues.