Aggregation CFN template for Data Exports #787
Conversation
This commit adds a new cloudformation template to support CUR 2.0 in Data Exports with the same functionality as `cur-aggregation.yaml`
Updated comments
| Fn::Sub: "arn:${AWS::Partition}:s3:::${ResourcePrefix}-${DestinationAccountId}-shared" | ||
| StorageClass: STANDARD | ||
| Id: ReplicationRule1 | ||
| Prefix: "" |
There was a problem hiding this comment.
Need to manage this carefully. Please can you test creating athena table with 2 aggregated exports?
There was a problem hiding this comment.
+1 it should be leading to data folder to exclude metadata folder replication
There was a problem hiding this comment.
Should it be a filter on the ReplicationConfiguration? Or how else would I exclude metadata folder replication?
There was a problem hiding this comment.
you can try both. whatever works the best. I would suggest using the right prefix would be optimal as you know the path. Also this means that we need to use replication even if we have a local CUR (or alternatively prohibit cur to write /metadata/).
There was a problem hiding this comment.
As discussed, we will have to use local replication, because: 1) S3 replication itself doesn't support exclusion 2) We can't exclude metadata folder in Athena 3) SCAD won't accept reduced bucket write permissions without metadata folder
@yprikhodko I had it in the version you shared with me, but it wasn't in the version that was present on Well Architected Labs website. It also kept causing errors for me when I tried to upload the template if I didn't specify anything in that field so I ended up just removing it. We can add it back in if you'd like and know how to make it not error.
|
Made updates suggested by Yuriy and Iakov
|
@yprikhodko and @iakov-aws have you seen this error before? I'm getting it when trying to run the CFN template outside of us-east-1 All other resources besides SourceS3 seem to be deploying correctly. 
|
|
This might happen if you delete s3 in one region and then try to recreate in another one too soon with the same name. You can try changing prefix or wait a bit more. |
| res = client.create_export( | ||
| Export=export | ||
| ) | ||
| print(json.dumps(res)) |
There was a problem hiding this comment.
do you need to delete old one as well?
There was a problem hiding this comment.
Delete is handled in next block
There was a problem hiding this comment.
in case when you change the name on update and you have to create the new CUR, you need to delete old. Correct ?
| ########################################################################### | ||
| # Glue Database for CID framework | ||
| ########################################################################### | ||
| CIDDatabase: |
There was a problem hiding this comment.
This can be a blocker. I struggle to understand the full flow if CID database is defined here
Additional Changes
merge main
This commit adds a new cloudformation template to support CUR 2.0 in Data Exports with the same functionality as
cur-aggregation.yamlBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.