Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eventhub Entra ID authentication fails with RBAC permissions granted on consumer group #35337

Open
dlindblom opened this issue Apr 24, 2024 · 19 comments · May be fixed by #35626
Open

Eventhub Entra ID authentication fails with RBAC permissions granted on consumer group #35337

dlindblom opened this issue Apr 24, 2024 · 19 comments · May be fixed by #35626
Assignees
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Event Hubs Messaging Messaging crew needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@dlindblom
Copy link

  • Package Name: azure-eventhub
  • Package Version: 5.11.6
  • Operating System: Linux
  • Python Version: 3.12.2

Describe the bug
When connecting to Eventhub to read data from consumer group authentication fails when using AD/Entra Authentication is used, and RBAC permissions assigned to consumer group per design for Event hub and following least access principle and avoid risk consuming application read from wrong consumer group causing problems for other consumers.
Ref: https://learn.microsoft.com/en-us/azure/event-hubs/authorize-access-azure-active-directory.
Error message: "Unauthorized access. 'Listen' claim(s) are required to perform this operation"

To Reproduce
Steps to reproduce the behavior:
Create Azure Eventhub
Create Consumer Group inside Eventhub
Aassign RBAC Role of Azure Event Hubs Data receiver to Consumer group for Service Principle
Use AD Service Principle Authentication in Python SDK for Eventhub and try to read Consumer Group.

Expected behavior
Successful authentication towards Event hub and Consumer Group, and allow Data receiver to only read from Consumer Group granted permission for.

Screenshots
INFO - 2024-04-03T10:17:14+0000 - connection_async - work_async: b'Cannot get initial delivery count' (b'/project/src/vendor/azure-uamqp-c/src/link.c':b'link_frame_received':343)
INFO - 2024-04-03T10:17:14+0000 - receiver - _state_changed: Receiver link failed to open - expecting to receive DETACH frame.
INFO - 2024-04-03T10:17:14+0000 - receiver - _detach_received: Received Link detach event: b'amqp:unauthorized-access'
Link: b'receiver-link-069af3dd-8783-414c-99f6-7434b4cfb924'
Description: b'Unauthorized access. 'Listen' claim(s) are required to perform this operation. Resource: 'sb://XXXXXX.servicebus.windows.net/XYZ/consumergroups/ABC/partitions/1'. TrackingId:402635ccdfb648218ee8a5effc4e33c3_G21, SystemTracker:gateway5, Timestamp:2024-04-03T10:17:14'
Details: None
Retryable: False
Connection: b'EHReceiver-e90c7302-4ad8-4126-acbf-ea8423d7a9f5-partition1'
DEBUG - 2024-04-03T10:17:14+0000 - connection_async - work_async: Deallocating cError
WARNING - 2024-04-03T10:17:14+0000 - receiver - get_state: LinkDetach('ErrorCodes.UnauthorizedAccess: Unauthorized access. 'Listen' claim(s) are required to perform this operation. Resource: 'sb://XXXXXX.servicebus.windows.net/XYZ/consumergroups/ABC/partitions/1'. TrackingId:402635ccdfb648218ee8a5effc4e33c3_G21, SystemTracker:gateway5, Timestamp:2024-04-03T10:17:14')
DEBUG - 2024-04-03T10:17:14+0000 - receiver - destroy: Destroying cMessageReceiver
DEBUG - 2024-04-03T10:17:14+0000 - receiver - destroy: Destroying cLink
INFO - 2024-04-03T10:17:14+0000 - client_async - close_async: CBS session pending b'EHReceiver-e90c7302-4ad8-4126-acbf-ea8423d7a9f5-partition1'.

Additional context

@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Event Hubs needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team. labels Apr 24, 2024
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kasun04.

@kashifkhan
Copy link
Member

Thank you for the feedback @dlindblom . We will investigate and get back to you asap.

@kashifkhan kashifkhan added Messaging Messaging crew and removed Service Attention This issue is responsible by Azure service team. needs-team-attention This issue needs attention from Azure service team or SDK team labels Apr 24, 2024
@kashifkhan kashifkhan self-assigned this Apr 24, 2024
@github-actions github-actions bot added needs-team-attention This issue needs attention from Azure service team or SDK team needs-team-triage This issue needs the team to triage. labels Apr 24, 2024
@xiangyan99 xiangyan99 removed the needs-team-triage This issue needs the team to triage. label Apr 24, 2024
@swathipil swathipil assigned swathipil and unassigned kashifkhan Apr 29, 2024
@swathipil
Copy link
Member

Hi @dlindblom - Based on prior internal issues, the recommendation from the service team has been to assign the role at the namespace level, as assigning at the consumer group level does not work consistently.

We would suggest that you report this issue with the documentation by following the directions under the Feedback section at the bottom of the page.

Additional Note: It looks like you are using uamqp as the underlying AMQP stack, which is no longer supported. We highly recommend that the default pure Python AMQP stack is used, as it includes all latest bug fixes and support for new features. You can do this by removing uamqp_transport=True from arguments that are passed in during client creation.

Thanks!

@swathipil swathipil added issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Apr 30, 2024
Copy link

Hi @dlindblom. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

@dlindblom
Copy link
Author

Hi, Thx for feedback on AMQP Stack. But the suggestion on documentation is not suitable approach. The assignment at consumer group level have been recommended general approach by Microsoft and is working fine except for the Python SDK.
/unresolve

@dlindblom dlindblom reopened this May 2, 2024
@github-actions github-actions bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. labels May 2, 2024
@swathipil
Copy link
Member

Hi @dlindblom - Would you be able to provide a list of other SDKs you've tested with assignment at the consumer group level that have worked?

@swathipil swathipil added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels May 6, 2024
Copy link

github-actions bot commented May 6, 2024

Hi @dlindblom. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@dlindblom
Copy link
Author

@swathipil I been informed that it works with .Net SDK

@github-actions github-actions bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels May 7, 2024
@swathipil
Copy link
Member

swathipil commented May 7, 2024

@dlindblom - Can you confirm that you are testing with the same Event Hub, consumer group, and credentials when receiving from both the Python and .NET SDKs, and .NET is consistently working while Python returns the above error?

@swathipil swathipil added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels May 7, 2024
Copy link

github-actions bot commented May 7, 2024

Hi @dlindblom. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@dlindblom
Copy link
Author

@swathipil I been informed that it works with .Net SDK

@github-actions github-actions bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels May 8, 2024
@dlindblom
Copy link
Author

@swathipil confirmed by Microsoft engineering

@swathipil
Copy link
Member

Hi @dlindblom - We've gotten confirmation from the service team that permission should be granted at either the namespace level or Event Hub level. They will be updating documentation to reflect this.

Can you let us know the steps that you took to grant permission at the consumer group level?

@swathipil swathipil added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels May 9, 2024
Copy link

github-actions bot commented May 9, 2024

Hi @dlindblom. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@dlindblom
Copy link
Author

@swathipil https://learn.microsoft.com/en-us/azure/event-hubs/authorize-access-azure-active-directory#resource-scope

"Before you assign an Azure role to a security principal, determine the scope of access that the security principal should have. Best practices dictate that it's always best to grant only the narrowest possible scope.

The following list describes the levels at which you can scope access to Event Hubs resources, starting with the narrowest scope:

Consumer group: At this scope, role assignment applies only to this entity. Currently, the Azure portal doesn't support assigning an Azure role to a security principal at this level.
..."

It's mention Azure portal doesn't support assigning it at that level, and that is true from default navigation, but Microsoft dev team have advised that adding /consumergroup_name in the URL it's possible in Azure Portal as well and the navigation is pending to be fixed.
We usually deploy permissions using Graph API as Infrastructure As Code, but using the Portal for some manual assignment.

@github-actions github-actions bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels May 10, 2024
@swathipil
Copy link
Member

Hi @dlindblom - Thanks for the information and your patience! I was able to confirm from our side that the .NET SDK is able to receive when the role is assigned at the consumer group level. I was also able to reproduce the "Unauthorized access" error in the Python SDK. We are currently looking into this and will keep you updated.

In the meantime, would you be able to provide the steps you are taking to deploy permissions using the Graph API?

@swathipil swathipil added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels May 14, 2024
Copy link

Hi @dlindblom. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

Copy link

Hi @dlindblom, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

@github-actions github-actions bot added the no-recent-activity There has been no recent activity on this issue. label May 21, 2024
@dlindblom
Copy link
Author

dlindblom commented May 30, 2024

@swathipil example using PowerShell:
New-AzRoleAssignment -ObjectId <objectId> -RoleDefinitionName <roleName> -Scope /subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/Microsoft.EventHub/namespaces/<eventHubNameSpace>/eventhubs/<eventhub>/consumergroups/<consumergroupName>
Using GraphAPI should be similar, using same Scope
Azure Portal can also be used if one refer to the Scope in the URI

@github-actions github-actions bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. no-recent-activity There has been no recent activity on this issue. labels May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Event Hubs Messaging Messaging crew needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
4 participants