CSE native cluster deployment fails with ACCESS_TO_RESOURCE_IS_FORBIDDEN

[ab9876543210]

Describe the bug

We are on VCD 10.3.3.20027910 and CSE 3.1.4, and are having issues deploying native clusters with limited-privilege (customized vApp Author + CSE rights; we are calling this "Orchestrator") users. The same operation succeeds with higher-privilege role (customized Organization Administrator + CSE rights).
Before upgrading to CSE 3.1.4, the Orchestrator role could create CSE clusters just fine. All CSE rights were/are assigned to it, along with necessary vApp rights.

Currently, deployments are failing with this error:

Error adding control plane node: failure on creating nodes ['mstr-xzsd']
Error:Status code: 403/ACCESS_TO_RESOURCE_IS_FORBIDDEN, [ 8068d1d1-bd06-4f9a-b9ce-a25c588ee2be ] Either you need some or all of the following rights [Base] to perform operations [VAPP_VM_VIEW] for 8afc3401-9a9a-4e57-933d-be534cec5f1f or the target entity is invalid. (request id: 8068d1d1-bd06-4f9a-b9ce-a25c588ee2be)

This looks like an error on CSE end only, as the user can create vApps just fine in the same Org-vDC where we are attempting to create CSE clusters.

It seems the same error of #1360 which was for TKGm clusters, and supposedly fixed on 3.1.4 ..

Reproduction steps

...

Expected behavior

Native cluster deployments should succeed using an account which has the necessary vApp/CSE clusters creation rights.

Additional context

Limited privilege role rights:

name                                         id
----                                         --
vApp: Use Console                            urn:vcloud:right:0efcfea8-d79f-3556-8275-0ba6e18ce05b
vApp: Sharing                                urn:vcloud:right:1222edb5-adb4-32c4-be70-0e88aef1f03b
Organization Network: View                   urn:vcloud:right:194c71a1-3d68-3156-b789-6a6384028b78
vApp Template / Media: View                  urn:vcloud:right:1aa46727-6192-365d-b571-5ce51beb3b48
Organization vDC Network: View Properties    urn:vcloud:right:2c8d98ef-4acc-3be4-9214-fcb9682b7a19
vApp: Create / Reconfigure                   urn:vcloud:right:2dc8abec-2e0d-3789-a5f9-ce0453160b53
vApp Template / Media: Edit                  urn:vcloud:right:3eedbfb4-c4a3-373d-b4b5-d76ca363ab50
vApp Template / Media: Copy                  urn:vcloud:right:444def42-24a8-33b5-a780-13af93b52fac
Catalog: Add vApp from My Cloud              urn:vcloud:right:4886663f-ae31-37fc-9a70-3dbe2f24a8c5
vApp: Copy                                   urn:vcloud:right:4965b0e7-9ed8-371d-8b08-fc716d20bf4b
vApp: Edit VM Properties                     urn:vcloud:right:5250ab79-8f50-33f9-8af5-015cb39c380b
vApp: Power Operations                       urn:vcloud:right:580860cd-55bc-322d-ac39-4f9d8e3e1cd2
vApp: VM Boot Options                        urn:vcloud:right:69bc6569-6b5b-3fdf-abaf-e0d16ae0e2d7
vApp: View ACL                               urn:vcloud:right:6bdadad3-1e25-3a4a-9d39-4927676e09dc
{cse}:CSE NATIVE DEPLOY RIGHT                urn:vcloud:right:6f68a446-f1ee-4125-88f4-1756ffb77a4a
vApp: Edit VM CPU                            urn:vcloud:right:729a3828-8b63-31b2-88db-f56612a06722
vApp Template: Checkout                      urn:vcloud:right:ab08b301-7f06-33a6-8f0c-eb8bdaa782d6
vApp: Snapshot Operations                    urn:vcloud:right:af90833f-5014-3fa5-b7f7-f2e653ec200b
vApp: Manage VM Password Settings            urn:vcloud:right:b2bb3262-8724-3775-ab39-f8713782c856
vApp: Edit Properties                        urn:vcloud:right:c2a29357-1b2a-3f9d-9cd6-de3d525d49f3
vApp: Edit VM Memory                         urn:vcloud:right:c6c827dc-fc42-33a8-844f-8ab5a91f8a6c
vApp: Edit VM Hard Disk                      urn:vcloud:right:cd02b5f8-c54a-334a-b782-5d31a1d77d85
vApp: Delete                                 urn:vcloud:right:df05c07f-c537-3777-8d9b-a9cfe8d49014
Catalog: View Published Catalogs             urn:vcloud:right:f01671e6-dfad-379d-b8e2-0d18e37ce993
vApp: Edit VM Network                        urn:vcloud:right:f24fffde-f953-3976-9f2b-8b355b25881d
Organization vDC Compute Policy: View        urn:vcloud:right:f3633840-37d7-3214-968d-297834656d98
Catalog: View Private and Shared Catalogs    urn:vcloud:right:fa4ce8f8-c640-3b65-8fa5-a863b56c3d51
Organization vDC Named Disk: Create          urn:vcloud:right:438e45e9-9389-3e29-9073-638b36921a2a
Organization vDC Named Disk: Delete          urn:vcloud:right:1e5ad20d-1023-34d1-b073-1ea30bce3854
Organization vDC Named Disk: Edit Properties urn:vcloud:right:7bbee458-b3c5-3252-ba5a-b1781b1c7b92
Organization vDC Named Disk: View Properties urn:vcloud:right:fd036ae5-b78b-3c9f-8f28-a7f6b33d0d92
Organization vDC Named Disk: Change Owner    urn:vcloud:right:5ddb661d-caf0-3680-9a74-59d4b06137f3
cse:nativeCluster: View                      urn:vcloud:right:fb2a4048-075b-43af-bf48-2287315a2a85
cse:nativeCluster: Modify                    urn:vcloud:right:bfab5226-aa1b-4351-9bbf-99d86790000c
cse:nativeCluster: Full Access               urn:vcloud:right:7a4b9fcf-e852-4b99-9be2-9b424a37b5b9
cse:nativeCluster: Administrator View        urn:vcloud:right:34254421-eae7-4609-9d83-918bcdc46a35
cse:nativeCluster: Administrator Full access urn:vcloud:right:a25c9745-92a8-4dd9-a69d-22e573b09346

[ab9876543210]

As further info, here the logs from server-container-debug.log:

22-11-21 14:44:39 | cluster_service_2_x:897 - _create_cluster_async | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | DEBUG :: Setting metadata on cluster vApp 'obs-test-2211P11345-2'
22-11-21 14:44:45 | cluster_service_2_x:917 - _create_cluster_async | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | DEBUG :: Creating control plane node for cluster 'obs-test-2211P11345-2' (urn:vcloud:entity:cse:nativeCluster:cf236302-a5fa-46f2-b699-7cd77b66ddb3)
22-11-21 14:44:45 | cluster_service_2_x:2172 - _update_task | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | DEBUG :: Sending behavior response:{'type': 'BEHAVIOR_RESPONSE', 'headers': {'taskId': '06321893-3cf3-48e2-84eb-bd559de0517b', 'entityId': 'urn:vcloud:entity:cse:nativeCluster:cf236302-a5fa-46f2-b699-7cd77b66ddb3', 'contentType': 'application/vnd.vmware.vcloud.task+json'}, 'payload': '{"status": "running", "operation": "Creating control plane node for cluster \'obs-test-2211P11345-2\' (urn:vcloud:entity:cse:nativeCluster:cf236302-a5fa-46f2-b699-7cd77b66ddb3)"}'}
22-11-21 14:44:45 | mqtt_publisher:116 - send_response | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | DEBUG :: publish return (rc, msg_id): (0, 37210)
22-11-21 14:44:48 | cluster_service_2_x:2593 - _add_nodes | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | DEBUG :: Found sizing policy with name System Default on the VDC FCH-ITR_ISN7
22-11-21 14:44:48 | cluster_service_2_x:2675 - _add_nodes | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | ERROR :: Status code: 403/ACCESS_TO_RESOURCE_IS_FORBIDDEN, [ 0b2e9e0d-f83b-4b8a-b45e-bcd500e818c7 ] Either you need some or all of the following rights [Base] to perform operations [VAPP_VM_VIEW] for 8afc3401-9a9a-4e57-933d-be534cec5f1f or the target entity is invalid. (request id: 0b2e9e0d-f83b-4b8a-b45e-bcd500e818c7)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/container_service_extension/rde/backend/cluster_service_2_x.py", line 2632, in _add_nodes
    task = vapp.add_vms(specs, power_on=False)
  File "/usr/local/lib/python3.7/site-packages/pyvcloud/vcd/vapp.py", line 1032, in add_vms
    EntityType.RECOMPOSE_VAPP_PARAMS.value, params)
  File "/usr/local/lib/python3.7/site-packages/pyvcloud/vcd/client.py", line 1537, in post_linked_resource
    media_type, extra_headers=extra_headers)
  File "/usr/local/lib/python3.7/site-packages/pyvcloud/vcd/client.py", line 1520, in post_resource
    extra_headers=extra_headers)
  File "/usr/local/lib/python3.7/site-packages/pyvcloud/vcd/client.py", line 1267, in _do_request
    _objectify_response(response, objectify_results))
  File "/usr/local/lib/python3.7/site-packages/pyvcloud/vcd/client.py", line 1278, in _response_code_to_exception
    raise AccessForbiddenException(sc, request_id, objectify_response)
pyvcloud.vcd.exceptions.AccessForbiddenException: Status code: 403/ACCESS_TO_RESOURCE_IS_FORBIDDEN, [ 0b2e9e0d-f83b-4b8a-b45e-bcd500e818c7 ] Either you need some or all of the following rights [Base] to perform operations [VAPP_VM_VIEW] for 8afc3401-9a9a-4e57-933d-be534cec5f1f or the target entity is invalid. (request id: 0b2e9e0d-f83b-4b8a-b45e-bcd500e818c7)
22-11-21 14:44:48 | cluster_service_2_x:940 - _create_cluster_async | Request Id: 74202936-ba0c-4022-8891-0b421aee2a40 | ERROR :: failure on creating nodes ['mstr-vmg7']