Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Policy using resource: iam-policy recieves UnrecognizedClientException during dry-run in GovCloud #7610

Closed
iamperson347 opened this issue Aug 4, 2022 · 5 comments · Fixed by #9475
Labels

Comments

@iamperson347
Copy link

Describe the bug

I'm receiving an when running the policy outlined below in dry-run mode. It seems to go in successfully when run without dry-run, but I'm not sure if they actual policy is working or not.

This is occuring in AWS GovCloud so the issue might be specific to GovCloud regions.

What did you expect to happen?

dry-run works successfully. Policy works.

Cloud Provider

Amazon Web Services (AWS)

Cloud Custodian version and dependency information

Custodian:   0.9.17
Python:      3.8.0 (default, Dec  9 2021, 17:53:27)
             [GCC 8.4.0]
Platform:    posix.uname_result(sysname='Linux', nodename='<removed>', release='4.15.0-2075-aws-fips', version='#80-Ubuntu SMP Tue Jun 14 01:57:31 UTC 2022', machine='x86_64')
Using venv:  True
Docker: False
Installed:

argcomplete==2.0.0
attrs==21.4.0
boto3==1.24.10
botocore==1.27.10
docutils==0.17.1
importlib-metadata==4.11.4
importlib-resources==5.7.1
jmespath==1.0.0
jsonschema==4.6.0
pyrsistent==0.18.1
python-dateutil==2.8.2
pyyaml==6.0
s3transfer==0.6.0
six==1.16.0
tabulate==0.8.9
typing-extensions==4.2.0
urllib3==1.26.9
zipp==3.8.0

Policy

- name: iam-policy-account-audit-policy
    resource: iam-policy
    description: |
      Cloud Custodian IAM Policy account:* Audit
    mode:
      type: periodic
      role: arn:aws-us-gov:iam::{account_id}:role/<removed>
      schedule: "rate(1 day)"      
    filters:
      - type: has-allow-all
      - type: used
    actions:   
      - type: notify
        template: default.html
        template_format: "html"
        priority_header: "5"
        subject: "Cloud Custodian - Security Audit: AWS IAM Policy HasAllowAll"
        to:
          - <email removed>
        transport:
          type: sqs
          queue: <sqs queue removed>

Relevant log/traceback output

2022-08-04 09:30:03,594: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 526, in resources
    resources = self.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 582, in augment
    return self.source.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/resources/iam.py", line 394, in augment
    return universal_augment(self.manager, super().augment(resources))
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/tags.py", line 91, in universal_augment
    resource_tag_results = client.get_resources(
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
2022-08-04 09:30:03,599: custodian.commands:ERROR Error while executing policy iam-policy-account-audit-policy, continuing
Traceback (most recent call last):
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/commands.py", line 301, in run
    policy()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/policy.py", line 1236, in __call__
    resources = PullMode(self).run()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 526, in resources
    resources = self.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 582, in augment
    return self.source.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/resources/iam.py", line 394, in augment
    return universal_augment(self.manager, super().augment(resources))
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/tags.py", line 91, in universal_augment
    resource_tag_results = client.get_resources(
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid

Extra information or context

No response

@kapilt
Copy link
Collaborator

kapilt commented Aug 4, 2022

which region specifically? this seems like an issue on the aws side, you probably want to inquire to aws support or your TAM. specifically we've already done the iam call, but then we're getting an error using the resource group tagging api with the same credentials.

@iamperson347
Copy link
Author

iamperson347 commented Aug 4, 2022 via email

@kapilt
Copy link
Collaborator

kapilt commented Aug 8, 2022

regarding seeing api calls, the easiest is probably via the builtin aws sdk instrumentation capabilities, ie.
https://boto3.amazonaws.com/v1/documentation/api/1.14.0/guide/sdk-metrics.html

an alternative, you can optionally enable X-ray tracing via the cli (--trace xray) but given this is a failure scenario its not clear if that's viable here.

@kapilt
Copy link
Collaborator

kapilt commented Aug 8, 2022

i typically use https://vector.dev with csm as a sink/log of calls here's the config file i use for vector, and then i just set the AWS_CSM* env vars up.

[sources.aws_csm]
type = "socket"
address = "127.0.0.1:31000"
max_length = 102_400
mode = "udp"

[transforms.parse]
type = "remap"
inputs = [ "aws_csm" ]
source = """
. = parse_json!(.message)
del(.SessionToken)
"""

[transforms.filter]
type = "filter"
inputs = ["parse"]
condition = '.Type != "ApiCallAttempt" && .Service != "SSO"'


[sinks.console]
inputs = ["filter"]
type = "console"
encoding = "json"

@bshutterkion
Copy link

bshutterkion commented Apr 8, 2024

I also want to add my experience, as I see this issue, but only with govcloud regions.

Running custodian version 0.9.35

Having a policy as simple as what's below, returns the same botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid:

---
policies:
  - name: iam-all-policies
    resource: aws.iam-policy
    description: |
      Identify All IAM Policies.

custodian run ../UnrecognizedClientException/iam-policy-all.yml -s . --verbose --debug > debug_output.txt 2>&1

2024-04-08 16:29:37,659: custodian.commands:DEBUG Loaded file ../UnrecognizedClientException/iam-policy-all.yml. Contains 1 policies
2024-04-08 16:29:37,663: custodian.aws:DEBUG using default region:us-gov-west-1 from boto
2024-04-08 16:29:38,994: custodian.output:DEBUG Storing output with <LogFile file://./iam-all-policies/custodian-run.log>
2024-04-08 16:29:38,998: custodian.policy:DEBUG Running policy:iam-all-policies resource:aws.iam-policy region:us-gov-west-1 c7n:0.9.35
2024-04-08 16:29:39,001: custodian.cache:DEBUG expired 1 stale cache entries
2024-04-08 16:29:40,253: custodian.output:DEBUG metric:PolicyException Count:1 policy:iam-all-policies restype:aws.iam-policy
2024-04-08 16:29:40,253: custodian.output:DEBUG metric:ApiCalls Count:2 policy:iam-all-policies restype:aws.iam-policy
2024-04-08 16:29:40,253: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/lib/python3.12/site-packages/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 543, in resources
    resources = self.augment(resources)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 599, in augment
    return self.source.augment(resources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/resources/iam.py", line 396, in augment
    return universal_augment(self.manager, super().augment(resources))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/tags.py", line 92, in universal_augment
    resource_tag_results = client.get_resources(
                           ^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
Traceback (most recent call last):
  File "/lib/python3.12/site-packages/c7n/cli.py", line 363, in main
    command(config)
  File "/lib/python3.12/site-packages/c7n/commands.py", line 128, in _load_policies
    return f(options, list(policies))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/commands.py", line 307, in run
    policy()
  File "/lib/python3.12/site-packages/c7n/policy.py", line 1357, in __call__
    resources = mode.run()
                ^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 543, in resources
    resources = self.augment(resources)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 599, in augment
    return self.source.augment(resources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/resources/iam.py", line 396, in augment
    return universal_augment(self.manager, super().augment(resources))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/tags.py", line 92, in universal_augment
    resource_tag_results = client.get_resources(
                           ^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
> /lib/python3.12/site-packages/botocore/client.py(1009)_make_api_call()
-> raise error_class(parsed_response, operation_name)
(Pdb) 

The same user can run awscli commands fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants