Enable support for aws eks rbac and aws iam msk #287

creed123 · 2021-06-15T09:41:39Z

This pull request aims to:

Enable kafkadrop ui running as a pod in an aws eks cluster to connect to iam authenicated aws msk. It makes use of iam roles scoped to a service-account to generate temporary credentials to connect to an iam authenticated msk. More on this can be found here:
https://docs.amazonaws.cn/en_us/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html
https://aws.amazon.com/blogs/big-data/securing-apache-kafka-is-easy-and-familiar-with-iam-access-control-for-amazon-msk/

Additional environment variables need to be specified:
--KAFKA_IAM_ENABLED=true
--KAFKA_SASL_MECHANISM=AWS_MSK_IAM
--KAFKA_SECURITY_PROTOCOL=SASL_SSL
--KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;'
--KAFKA_IS_SECURED=true

fabioformosa · 2021-06-16T16:10:23Z

What are the differences respect to this other PR to address the same target?#275

Have you pushed the docker image of your forked project in dockerhub?

fabioformosa · 2021-06-16T17:00:01Z

Using your PR, I get

Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config 
at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110)
at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:134) 
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:740)

any idea?

creed123 · 2021-06-17T06:36:56Z

Using your PR, I get

Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config 
at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110)
at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:134) 
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:740)

any idea?

@fabioformosa We need to specify a semi colon at the end of the
--KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;'
Can check after making the change?

creed123 · 2021-06-17T06:41:37Z

What are the differences respect to this other PR to address the same target?#275

Have you pushed the docker image of your forked project in dockerhub?

This MR aims to use the role scoped to a service account for a pod in an eks cluster if such a role exists. #275 by default would use the role assigned to an ec2 instance.

fabioformosa · 2021-06-18T07:49:15Z

@fabioformosa We need to specify a semi colon at the end of the
--KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;'
Can check after making the change?

@creed123 Do you mean maybe software.amazon.msk.auth.iam.IAMLoginModule required; ?

Yes, I've tried. It solves but now I get:

ERROR 1 [| kafdrop-admin] o.a.k.c.NetworkClient: [AdminClient clientId=kafdrop-admin] Connection to node 
(b-2.xxx.eu-south-1.amazonaws.com/xxxx:9098) failed authentication due to: Access denied
WARN 1 [| kafdrop-admin] o.a.k.c.a.i.AdminMetadataManager : [AdminClient clientId=kafdrop-admin] Metadata update
 failed due to authentication error org.apache.kafka.common.errors.SaslAuthenticationException: Access denied

My eks automatically created a IAM Role, I gave fullAdminAccess to this Role to try to solve. Same error.

In your view, what can it be the cause?

creed123 · 2021-06-25T07:15:38Z

@fabioformosa We need to specify a semi colon at the end of the
--KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;'
Can check after making the change?

@creed123 Do you mean maybe software.amazon.msk.auth.iam.IAMLoginModule required; ?

Yes, I've tried. It solves but now I get:
ERROR 1 [| kafdrop-admin] o.a.k.c.NetworkClient: [AdminClient clientId=kafdrop-admin] Connection to node 
(b-2.xxx.eu-south-1.amazonaws.com/xxxx:9098) failed authentication due to: Access denied
WARN 1 [| kafdrop-admin] o.a.k.c.a.i.AdminMetadataManager : [AdminClient clientId=kafdrop-admin] Metadata update
 failed due to authentication error org.apache.kafka.common.errors.SaslAuthenticationException: Access denied
My eks automatically created a IAM Role, I gave fullAdminAccess to this Role to try to solve. Same error.

In your view, what can it be the cause?

@fabioformosa Can you check if the role attached to your ec2 instance has the correct permissions?

yortch · 2021-07-28T17:05:16Z

In case it helps, I verified this works from OpenShift (Kubernetes) cluster using IRSA (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). The only additional change I had to make was to provide AWS_REGION as an environment variable for AWS STS (Security Token Service) to work. I do believe this could work just with the pom.xml changes added to this pull request. For convenience I published my image to quay.io: https://quay.io/yortch/kafdrop and this is how I built it:

git clone https://github.com/creed123/kafdrop.git
cd kafdrop
mvn clean package
mvn assembly:single docker:build
docker run -d --rm obsidiandynamics/kafdrop:3.28.0-SNAPSHOT
docker login quay.io
#provide quay.io credentials
docker ps -l 
#get container ID and replace it below
docker commit <container_id> quay.io/yortch/kafdrop:3.28.0-SNAPSHOT
docker push quay.io/yortch/kafdrop:3.28.0-SNAPSHOT
#subsequently made quay.io repository public

tooptoop4 · 2021-10-27T13:26:35Z

gentle ping

mfinger-incontact · 2022-05-24T16:38:02Z

Gentle ping X2

kirkokada · 2022-05-31T22:19:16Z

gentle ping x3

davideicardi · 2022-06-01T07:29:45Z

I will be happy to merge this PR, but someone need to resolve the conflicts and review it.

mfinger-incontact · 2022-06-30T16:52:20Z

I will be happy to merge this PR, but someone need to resolve the conflicts and review it. <

I'm happy to help out with the conflicts, but I guess I need to become a contributor over here.

davideicardi · 2022-06-30T17:10:17Z

I will be happy to merge this PR, but someone need to resolve the conflicts and review it. <

I'm happy to help out with the conflicts, but I guess I need to become a contributor over here.

@mfinger-incontact
I think it is better to just fork the repository (or this branch) and apply the same changes, resolve conflicts then create a new PullRequest

aberenshtein · 2022-10-31T12:52:42Z

@mfinger-incontact gentle ping

Harshil Nori and others added 11 commits June 15, 2021 11:49

Added support for iam enabled msk using custom role attached to k8s pod

e4496fc

Dockerfile added

24d2a53

Renamed Dockerfile

ed63e4c

Removed unwanted client.properties. Additional formatting changes

2a979d7

Additional formatting changes

98f2206

Pom changes

a018aa0

Updated dockerfile

93bd4a9

Updated pom properties.

041a47d

Updated yml with is secured

1360e6b

Updated yml with is secured

e0788a6

Updated yml with updated properties file

bc78e82

creed123 changed the title ~~Added support for aws msk iam using oidc identity provider authentication~~ Enable kafkadrop ui running within aws eks cluster to connect to iam authenticated msk Jun 15, 2021

creed123 changed the title ~~Enable kafkadrop ui running within aws eks cluster to connect to iam authenticated msk~~ Enable support for aws eks rbac and aws iam msk Jun 15, 2021

creed123 mentioned this pull request Jun 15, 2021

Kafdrop UI has issue using IAM access control to address encryption,authentication and authorisation #283

Closed

Added warn log

77e97a2

richardwu mentioned this pull request Apr 21, 2023

Add support for AWS MSK IAM #513

Open

bluemonk3y mentioned this pull request Jun 6, 2023

AWS MSK IAM support specmesh/specmesh-build#149

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable support for aws eks rbac and aws iam msk #287

Enable support for aws eks rbac and aws iam msk #287

creed123 commented Jun 15, 2021 •

edited

fabioformosa commented Jun 16, 2021

fabioformosa commented Jun 16, 2021

creed123 commented Jun 17, 2021

creed123 commented Jun 17, 2021

fabioformosa commented Jun 18, 2021

creed123 commented Jun 25, 2021

yortch commented Jul 28, 2021

tooptoop4 commented Oct 27, 2021

mfinger-incontact commented May 24, 2022

kirkokada commented May 31, 2022

davideicardi commented Jun 1, 2022

mfinger-incontact commented Jun 30, 2022

davideicardi commented Jun 30, 2022

aberenshtein commented Oct 31, 2022

Enable support for aws eks rbac and aws iam msk #287

Are you sure you want to change the base?

Enable support for aws eks rbac and aws iam msk #287

Conversation

creed123 commented Jun 15, 2021 • edited

fabioformosa commented Jun 16, 2021

fabioformosa commented Jun 16, 2021

creed123 commented Jun 17, 2021

creed123 commented Jun 17, 2021

fabioformosa commented Jun 18, 2021

creed123 commented Jun 25, 2021

yortch commented Jul 28, 2021

tooptoop4 commented Oct 27, 2021

mfinger-incontact commented May 24, 2022

kirkokada commented May 31, 2022

davideicardi commented Jun 1, 2022

mfinger-incontact commented Jun 30, 2022

davideicardi commented Jun 30, 2022

aberenshtein commented Oct 31, 2022

creed123 commented Jun 15, 2021 •

edited