[Integration][AWS] Add new integration #555

shalev007 · 2024-04-15T15:25:53Z

Description

Ocean AWS Integration 🥇
basically the same exporter we have today only

is OSS
can use multi account
utilises the ocean framework

What - AWS Integration
Why - Better, faster, more accounts, less installations
How -

Type of change

Please leave one option from the following and delete the rest:

New Integration (non-breaking change which adds a new integration)

Screenshots

API Documentation

Provide links to the API documentation used for this integration. WIP

yairsimantov20

Shallow comments

integrations/aws/CHANGELOG.md

integrations/aws/Dockerfile

integrations/aws/config.yaml

integrations/aws/overrides.py

integrations/aws/pyproject.toml

integrations/aws/utils.py

integrations/aws/.port/resources/blueprints.json

integrations/aws/.port/resources/port-app-config.yml

integrations/aws/.port/spec.yaml

integrations/aws/.port/resources/blueprints.json

integrations/aws/overrides.py

integrations/aws/utils.py

integrations/aws/main.py

integrations/aws/utils.py

integrations/aws/main.py

yairsimantov20

Try to make the code more simple

integrations/aws/.port/spec.yaml

integrations/aws/config.yaml

integrations/aws/main.py

integrations/aws/utils.py

yairsimantov20 · 2024-04-21T09:43:38Z

integrations/aws/utils.py

+ self.enabled_regions = []
+
+ async def updateEnabledRegions(self):
+ session = aioboto3.Session(self.access_key_id, self.secret_access_key, self.session_token)


i think it should know to detect those

The session uses by default it's default region, but I would like to scan all of the regions available for each account

yairsimantov20 · 2024-04-21T09:44:38Z

integrations/aws/utils.py

+ def isRole(self):
+ return self.session_token is not None
+
+ async def createSession(self, region: str) -> aioboto3.Session:
+ if self.isRole():
+ return aioboto3.Session(self.access_key_id, self.secret_access_key, self.session_token, region)
+ else:
+ return aioboto3.Session(aws_access_key_id=self.access_key_id, aws_secret_access_key=self.secret_access_key, region_name=region)


why dont you work only with the role?

Left it as an option to run the integration outside of AWS since I initially developed it to use a user credentials, it supports both ways though

cant you run login with the cli?

I could, but running this integration on-prem might not use a cli

yairsimantov20 · 2024-04-21T09:44:56Z

integrations/aws/utils.py

+ else:
+ return aioboto3.Session(aws_access_key_id=self.access_key_id, aws_secret_access_key=self.secret_access_key, region_name=region)
+
+ async def createSessionForEachRegion(self) -> AsyncIterator[aioboto3.Session]:


why is that async?

aioboto3 Session returns a coroutine

yairsimantov20 · 2024-04-24T11:06:04Z

integrations/aws/aws_credentials.py

+ async def create_session_for_each_region(self) -> AsyncIterator[aioboto3.Session]:
+ for region in self.enabled_regions:
+ yield self.create_session(region)


can you please test a multiple regions account?

Yes, tested it, why?

yairsimantov20 · 2024-04-24T11:28:17Z

integrations/aws/utils.py

+ def isRole(self):
+ return self.session_token is not None
+
+ async def createSession(self, region: str) -> aioboto3.Session:
+ if self.isRole():
+ return aioboto3.Session(self.access_key_id, self.secret_access_key, self.session_token, region)
+ else:
+ return aioboto3.Session(aws_access_key_id=self.access_key_id, aws_secret_access_key=self.secret_access_key, region_name=region)


cant you run login with the cli?

integrations/aws/.port/spec.yaml

integrations/aws/aws_credentials.py

yairsimantov20

more comments

yairsimantov20 · 2024-04-24T12:10:06Z

integrations/aws/utils.py

+def validate_request(request: Request) -> None:
+ """
+ Validates the request by checking for the presence of the API key in the request headers.
+ """
+ api_key = request.headers.get('x-port-aws-ocean-api-key')
+ if not api_key:
+ raise ValueError("API key not found in request headers")
+ if not ocean.integration_config.get("aws_api_key"):
+ raise ValueError("API key not found in integration config")
+ if api_key != ocean.integration_config.get("aws_api_key"):
+ raise ValueError("Invalid API key")


I think there is better way to validate the request

https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html

I'm using api-destinations service to send the requests to the ocean integration, So I have no control over the requst, The only auth types avaiable to me were
Oauth, user/password or an api-key

Lets talk about this

integrations/aws/utils.py

yairsimantov20 · 2024-04-24T12:14:33Z

integrations/aws/main.py

+ resource = await describe_single_resource(resource_type, identifier, account_id, body.get("awsRegion"))
+ for resource_config in matching_resource_configs:
+ blueprint = resource_config.port.entity.mappings.blueprint.strip('"')
+ if not resource: # Resource probably deleted


can you please show me an example for delete event?

an event with an AWS S3 Bucket name arrived but the the bucket does not exist on AWS

can you show me how it looks like?

yairsimantov20 · 2024-04-24T12:15:07Z

integrations/aws/main.py

+ resource_type = body.get("resource_type")
+ identifier = body.get("identifier")
+ account_id = body.get("accountId")


is there a reason for any of them to not exist?

Yes, in AWS as a user you can create custom events, all of the events of our exporter need to be in the same format in order for our bot to handle them, so if an event body was not formatted correctly we cannot process it.

please take a look at how we currently format the events in our docs (any cloudformation example) https://docs.getport.io/build-your-software-catalog/sync-data-to-catalog/cloud-providers/aws/examples/

integrations/aws/main.py

integrations/aws/session_manager.py

yairsimantov20 · 2024-04-24T13:29:23Z

integrations/aws/session_manager.py

+ if account['Id'] == self._application_account_id:
+ # Skip the current account as it is already added
+ # Replace the Temp account details with the current account details
+ self._aws_accessible_accounts[0] = account
+ continue


Can we talk about this?

Sure let's talk

integrations/aws/session_manager.py

yairsimantov20 · 2024-04-26T15:31:42Z

integrations/aws/main.py

+
+@ocean.on_resync()
+async def resync_all(kind: str) -> ASYNC_GENERATOR_RESYNC_TYPE:
+ await update_available_access_credentials()


why are u doing it only here?

My initial approach was to periodically refresh the credentials, the resync all method that runs before any other resync method seems like a nice compromise, it seems redundant to me to update on evry type of sync, WDYT?

for now do it in every reync
nothing promise you it will stay that way
you can update the credentials once and then set it to the event parameters that you have already done this to prevent future calls

if "CREDENTIALS_CACHE" in event.attributes: return fetch event.attributes["CREDENTIALS_CACHE"] = X

integrations/aws/pyproject.toml

integrations/aws/overrides.py

integrations/aws/main.py

yairsimantov20 · 2024-04-26T15:35:06Z

integrations/aws/main.py

+ except Exception as e:
+ logger.error(f"Failed to process event from aws error: {e}", error=e)
+ return {"ok": False}


not really
think about when we will work over ocean SAAS it can be very helpful to monitor

integrations/aws/main.py

yairsimantov20 · 2024-04-26T15:42:05Z

integrations/aws/main.py

+ blueprint=blueprint,
+ )
+ resource.update({KIND_PROPERTY: resource_type, ACCOUNT_ID_PROPERTY: account_id, REGION_PROPERTY: body.get("awsRegion")})
+ await ocean.register_raw(resource_config.kind, [_fix_unserializable_date_properties(resource)])


This is very problematic as ocean dont let you pass the resource config itself and therefor you might trigger the same resource parsing multiple times

yairsimantov20 · 2024-04-26T15:44:03Z

integrations/aws/main.py

+ resource.update(
+ {
+ KIND_PROPERTY: resource_type,
+ ACCOUNT_ID_PROPERTY: account_id,
+ REGION_PROPERTY: region,
+ IDENTIFIER_PROPERTY: identifier,
+ }
+ )


please remove anything that can already be found in the playload

resources that are not using cloudcontrol do not always use Identifier as their identifier key, for example
in EC2 case it is listed as InstanceId

yairsimantov20 · 2024-04-26T15:46:23Z

integrations/aws/utils.py

+ response = await getattr(client, describe_method)()
+ next_token = response.get(marker_param)
+ for resource in response.get(list_param, []):
+ resource.update({KIND_PROPERTY: kind, ACCOUNT_ID_PROPERTY: account_id, REGION_PROPERTY: region, IDENTIFIER_PROPERTY: resource.get('Identifier')})


lets talk about this i want to see an exmple for the data

integrations/aws/utils.py

yairsimantov20 · 2024-04-26T15:48:21Z

integrations/aws/utils.py

+ if kind == ResourceKindsWithSpecialHandling.ACM:
+ async with session.client('acm') as acm:
+ response = await acm.describe_certificate(CertificateArn=identifier)
+ return response.get('Certificate', {})


? you are still using .get

yairsimantov20

comments

integrations/aws/overrides.py

yairsimantov20 · 2024-05-01T08:51:11Z

integrations/aws/main.py

+ if response.status_code <= 299:
+ response.status_code = status.HTTP_500_INTERNAL_SERVER_ERROR


can u check if there are status codes that might close the aws pipeline for the webhooks?

I don't know what did you mean in

close pipeline for webhooks

but according to AWS documentation:
Events associated with error codes 409, 429, and 5xx are retried.
Events associated with error codes 1xx, 2xx, 3xx, and 4xx (excluding 429) aren't retried.

integrations/aws/main.py

yairsimantov20 · 2024-05-01T08:52:35Z

integrations/aws/main.py

+ await ocean.register_raw(
+ resource_config.kind, [fix_unserializable_date_properties(resource)]
+ )


it is problematic that you might issue the same kind multiple times

I don't think so since they have the same identifier, BTW the Azure integration does this the same way exactly

Added unique to get_matching_kinds_from_config so that no kinds will be duplicated

integrations/aws/aws/utils.py

integrations/aws/aws/aws_credentials.py

integrations/aws/.port/resources/port-app-config.yml

github-actions bot added the size/XXL label Apr 15, 2024

yairsimantov20 requested changes Apr 16, 2024

View reviewed changes

yairsimantov20 reviewed Apr 16, 2024

View reviewed changes

integrations/aws/.port/resources/blueprints.json Show resolved Hide resolved

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch from dc1024e to da49c5e Compare April 16, 2024 15:56

Tankilevitch requested changes Apr 16, 2024

View reviewed changes

Tankilevitch requested changes Apr 17, 2024

View reviewed changes

Tankilevitch reviewed Apr 17, 2024

View reviewed changes

integrations/aws/main.py Outdated Show resolved Hide resolved

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch 2 times, most recently from 9ceb5c6 to 973f6ac Compare April 18, 2024 16:43

yairsimantov20 requested changes Apr 21, 2024

View reviewed changes

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch 3 times, most recently from 39eded1 to 543b9cf Compare April 24, 2024 09:38

yairsimantov20 reviewed Apr 24, 2024

View reviewed changes

yairsimantov20 requested changes Apr 24, 2024

View reviewed changes

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch from 543b9cf to 3c52b8d Compare April 24, 2024 14:04

yairsimantov20 reviewed Apr 25, 2024

View reviewed changes

integrations/aws/session_manager.py Outdated Show resolved Hide resolved

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch from 14887c8 to 9f428ac Compare April 25, 2024 17:06

yairsimantov20 reviewed Apr 26, 2024

View reviewed changes

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch 3 times, most recently from 420330f to 334b773 Compare April 30, 2024 14:07

yairsimantov20 requested changes May 1, 2024

View reviewed changes

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch from aa9bd75 to 77c3d9c Compare May 5, 2024 11:27

Tankilevitch changed the title ~~PORT 7056 aws exporter code in ocean~~ [Integration][AWS aws exporter code in ocean May 6, 2024

Tankilevitch changed the title ~~[Integration][AWS aws exporter code in ocean~~ [Integration][AWS] Add new integration May 6, 2024

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch from 0872d18 to db83a0d Compare May 8, 2024 08:05

Shalev Avhar added 27 commits May 29, 2024 19:04

feat: rename account kind

00b867d

feat: remove config

66c924f

feat: move function

80e8e4f

feat: rename live events keys

9d2824d

feat: use cached sessions

c58d481

feat: use conventions

16ecd40

fix: remove try-catch

e545b8e

fix: inherit ocean exception

c47701b

fix: account mapping and syncing

bf4d9a8

feat: rename default name

54fa784

fix: change matching kinds function

175f5c4

fix: live events key

4aff277

feat: update kinds

9505e4a

feat: fix formatting issues

b24adbc

feat: upgrade ocean

707bca8

test: change dev version for demo deploy

50ead17

feat: update dep

590e8a2

feat: upgrade tag

e96b432

fix: health check endpoint

bda8bf8

feat: upgrade tag

be527f1

feat: support resync specific kinds

6e3ea13

feat: add how to get keys to description

968a006

feat: update ocean version

0177299

feat: remove any trace of cloudResource

5a28c69

fix: update lockfile

5a7ef9a

fix: formatting issue

a4550e6

feat: delete config yaml

f3b1e6f

shalev007 force-pushed the PORT-7056-aws-exporter-code-in-ocean branch from 027e5e0 to f3b1e6f Compare May 29, 2024 16:04

shalev007 merged commit 0205024 into main May 30, 2024
5 checks passed

shalev007 deleted the PORT-7056-aws-exporter-code-in-ocean branch May 30, 2024 08:09

		if response.status_code <= 299:
		response.status_code = status.HTTP_500_INTERNAL_SERVER_ERROR

[Integration][AWS] Add new integration #555

[Integration][AWS] Add new integration #555

Conversation

shalev007 commented Apr 15, 2024 • edited

Description

Type of change

Screenshots

API Documentation

yairsimantov20 left a comment

Choose a reason for hiding this comment

yairsimantov20 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yairsimantov20 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yairsimantov20 Apr 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yairsimantov20 Apr 26, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yairsimantov20 Apr 26, 2024 • edited

Choose a reason for hiding this comment

yairsimantov20 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shalev007 commented Apr 15, 2024 •

edited

yairsimantov20 Apr 24, 2024 •

edited

yairsimantov20 Apr 26, 2024 •

edited

yairsimantov20 Apr 26, 2024 •

edited