Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document running Spark jobs on a Kerberised cluster #356

Closed
Tracked by #530
Jimvin opened this issue Feb 13, 2024 · 2 comments · May be fixed by #358
Closed
Tracked by #530

Document running Spark jobs on a Kerberised cluster #356

Jimvin opened this issue Feb 13, 2024 · 2 comments · May be fixed by #358
Assignees
@razvan @Jimvin

Comments

@Jimvin
Copy link
Member

Jimvin commented Feb 13, 2024

As a Spark developer I would like to be able to run Spark jobs on a cluster where one or more services have Kerberos enabled. There are additional setup required to get a Spark job to run with Kerberos, including providing keytab, Kerberos configuration, service configuration and Spark context config.

We should document how to run an example Spark job that connects to a kerberos-enabled HDFS and Hive service.
@adwk67
Copy link
Member

adwk67 commented Feb 13, 2024 •
edited

A spark-job that uses the secret operator to generate keytabs for named users can be found here: https://github.com/stackabletech/hdfs-topology-provider/pull/5/files#diff-49100e41e40194b8d1f7aba8218bec678d9a9817efd5346b4609a92269721264 (this should indicate what needs to be implemented in the spark-k8s-operator)

@Jimvin Jimvin self-assigned this Feb 13, 2024
@razvan razvan self-assigned this Feb 16, 2024
@razvan razvan mentioned this issue Feb 16, 2024
Draft
@lfrancke lfrancke mentioned this issue Feb 20, 2024
Closed
@sbernauer
Copy link
Member

Duplicate of stackabletech/issues#530, closing this as well. This is included in the end-to-end-security demo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razvan razvan

@Jimvin Jimvin

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

docs/kerberos example stackabletech/spark-k8s-operator
4 participants
@razvan @adwk67 @Jimvin @sbernauer