S3 Lambda Notification

[eglijin]

Does anyone know why creating a stack with a lambda function and a s3 bucket subscription generates an extra Python based lambda function in the stack? it's even present in the integ.s3.ts.

[peterwoodworth]

@eglijin I'm not super sure which specific constructs you're using are, but it's likely the CDK is creating this additional lambda under the hood to help one of the constructs you're using work.
Can you link me to the integ.s3.ts you see this in, and/or let me know which specific constructs you're using? Thanks

[eglijin]

@peterwoodworth
here's the code for the inter.s3.ts

import * as s3 from '@aws-cdk/aws-s3';
import * as cdk from '@aws-cdk/core';
import { S3EventSource } from '../lib';
import { TestFunction } from './test-function';

class S3EventSourceTest extends cdk.Stack {
  constructor(scope: cdk.App, id: string) {
    super(scope, id);

    const fn = new TestFunction(this, 'F');
    const bucket = new s3.Bucket(this, 'B', {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    });
    const importedBucket = s3.Bucket.fromBucketArn(this, 'Imported', bucket.bucketArn);

    fn.addEventSource(new S3EventSource(importedBucket, {
      events: [s3.EventType.OBJECT_CREATED],
      filters: [{ prefix: 'subdir/' }],
    }));
  }
}

const app = new cdk.App();
new S3EventSourceTest(app, 'lambda-event-source-s3');
app.synth();

And here's the cf json it generates, notice the ZipFile based python code, I'm really confused about why it's there.

{
  "Resources": {
    "FServiceRole3AC82EE1": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": "lambda.amazonaws.com"
              }
            }
          ],
          "Version": "2012-10-17"
        },
        "ManagedPolicyArns": [
          {
            "Fn::Join": [
              "",
              [
                "arn:",
                {
                  "Ref": "AWS::Partition"
                },
                ":iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
              ]
            ]
          }
        ]
      }
    },
    "FC4345940": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "ZipFile": "exports.handler = async function handler(event) {\n    console.log('event:', JSON.stringify(event, undefined, 2));\n    return { event };\n}"
        },
        "Role": {
          "Fn::GetAtt": [
            "FServiceRole3AC82EE1",
            "Arn"
          ]
        },
        "Handler": "index.handler",
        "Runtime": "nodejs14.x"
      },
      "DependsOn": [
        "FServiceRole3AC82EE1"
      ]
    },
    "B08E7C7AF": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "Tags": [
          {
            "Key": "aws-cdk:auto-delete-objects",
            "Value": "true"
          }
        ]
      },
      "UpdateReplacePolicy": "Delete",
      "DeletionPolicy": "Delete"
    },
    "BPolicy3F02723E": {
      "Type": "AWS::S3::BucketPolicy",
      "Properties": {
        "Bucket": {
          "Ref": "B08E7C7AF"
        },
        "PolicyDocument": {
          "Statement": [
            {
              "Action": [
                "s3:DeleteObject*",
                "s3:GetBucket*",
                "s3:List*"
              ],
              "Effect": "Allow",
              "Principal": {
                "AWS": {
                  "Fn::GetAtt": [
                    "CustomS3AutoDeleteObjectsCustomResourceProviderRole3B1BD092",
                    "Arn"
                  ]
                }
              },
              "Resource": [
                {
                  "Fn::GetAtt": [
                    "B08E7C7AF",
                    "Arn"
                  ]
                },
                {
                  "Fn::Join": [
                    "",
                    [
                      {
                        "Fn::GetAtt": [
                          "B08E7C7AF",
                          "Arn"
                        ]
                      },
                      "/*"
                    ]
                  ]
                }
              ]
            }
          ],
          "Version": "2012-10-17"
        }
      }
    },
    "BAutoDeleteObjectsCustomResource6224D839": {
      "Type": "Custom::S3AutoDeleteObjects",
      "Properties": {
        "ServiceToken": {
          "Fn::GetAtt": [
            "CustomS3AutoDeleteObjectsCustomResourceProviderHandler9D90184F",
            "Arn"
          ]
        },
        "BucketName": {
          "Ref": "B08E7C7AF"
        }
      },
      "DependsOn": [
        "BPolicy3F02723E"
      ],
      "UpdateReplacePolicy": "Delete",
      "DeletionPolicy": "Delete"
    },
    "CustomS3AutoDeleteObjectsCustomResourceProviderRole3B1BD092": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": "lambda.amazonaws.com"
              }
            }
          ]
        },
        "ManagedPolicyArns": [
          {
            "Fn::Sub": "arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
          }
        ]
      }
    },
    "CustomS3AutoDeleteObjectsCustomResourceProviderHandler9D90184F": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "AssetParametersbe270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824S3Bucket09A62232"
          },
          "S3Key": {
            "Fn::Join": [
              "",
              [
                {
                  "Fn::Select": [
                    0,
                    {
                      "Fn::Split": [
                        "||",
                        {
                          "Ref": "AssetParametersbe270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824S3VersionKeyA28118BE"
                        }
                      ]
                    }
                  ]
                },
                {
                  "Fn::Select": [
                    1,
                    {
                      "Fn::Split": [
                        "||",
                        {
                          "Ref": "AssetParametersbe270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824S3VersionKeyA28118BE"
                        }
                      ]
                    }
                  ]
                }
              ]
            ]
          }
        },
        "Timeout": 900,
        "MemorySize": 128,
        "Handler": "__entrypoint__.handler",
        "Role": {
          "Fn::GetAtt": [
            "CustomS3AutoDeleteObjectsCustomResourceProviderRole3B1BD092",
            "Arn"
          ]
        },
        "Runtime": "nodejs12.x",
        "Description": {
          "Fn::Join": [
            "",
            [
              "Lambda function for auto-deleting objects in ",
              {
                "Ref": "B08E7C7AF"
              },
              " S3 bucket."
            ]
          ]
        }
      },
      "DependsOn": [
        "CustomS3AutoDeleteObjectsCustomResourceProviderRole3B1BD092"
      ]
    },
    "ImportedNotificationsDB5DE386": {
      "Type": "Custom::S3BucketNotifications",
      "Properties": {
        "ServiceToken": {
          "Fn::GetAtt": [
            "BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691",
            "Arn"
          ]
        },
        "BucketName": {
          "Fn::Select": [
            0,
            {
              "Fn::Split": [
                "/",
                {
                  "Fn::Select": [
                    5,
                    {
                      "Fn::Split": [
                        ":",
                        {
                          "Fn::GetAtt": [
                            "B08E7C7AF",
                            "Arn"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        "NotificationConfiguration": {
          "LambdaFunctionConfigurations": [
            {
              "Events": [
                "s3:ObjectCreated:*"
              ],
              "Filter": {
                "Key": {
                  "FilterRules": [
                    {
                      "Name": "prefix",
                      "Value": "subdir/"
                    }
                  ]
                }
              },
              "LambdaFunctionArn": {
                "Fn::GetAtt": [
                  "FC4345940",
                  "Arn"
                ]
              }
            }
          ]
        },
        "Managed": false
      },
      "DependsOn": [
        "ImportedAllowBucketNotificationsTolambdaeventsources3F74160805710A6B67"
      ]
    },
    "ImportedAllowBucketNotificationsTolambdaeventsources3F74160805710A6B67": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Fn::GetAtt": [
            "FC4345940",
            "Arn"
          ]
        },
        "Principal": "s3.amazonaws.com",
        "SourceAccount": {
          "Ref": "AWS::AccountId"
        },
        "SourceArn": {
          "Fn::GetAtt": [
            "B08E7C7AF",
            "Arn"
          ]
        }
      }
    },
    "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleB6FB88EC": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": "lambda.amazonaws.com"
              }
            }
          ],
          "Version": "2012-10-17"
        },
        "ManagedPolicyArns": [
          {
            "Fn::Join": [
              "",
              [
                "arn:",
                {
                  "Ref": "AWS::Partition"
                },
                ":iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
              ]
            ]
          }
        ]
      }
    },
    "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleDefaultPolicy2CF63D36": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyDocument": {
          "Statement": [
            {
              "Action": [
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
              ],
              "Effect": "Allow",
              "Resource": "*"
            }
          ],
          "Version": "2012-10-17"
        },
        "PolicyName": "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleDefaultPolicy2CF63D36",
        "Roles": [
          {
            "Ref": "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleB6FB88EC"
          }
        ]
      }
    },
    "BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Description": "AWS CloudFormation handler for \"Custom::S3BucketNotifications\" resources (@aws-cdk/aws-s3)",
        "Code": {
          "ZipFile": "import boto3  # type: ignore\nimport json\nimport logging\nimport urllib.request\n\ns3 = boto3.client(\"s3\")\n\nEVENTBRIDGE_CONFIGURATION = 'EventBridgeConfiguration'\n\nCONFIGURATION_TYPES = [\"TopicConfigurations\", \"QueueConfigurations\", \"LambdaFunctionConfigurations\"]\n\ndef handler(event: dict, context):\n  response_status = \"SUCCESS\"\n  error_message = \"\"\n  try:\n    props = event[\"ResourceProperties\"]\n    bucket = props[\"BucketName\"]\n    notification_configuration = props[\"NotificationConfiguration\"]\n    request_type = event[\"RequestType\"]\n    managed = props.get('Managed', 'true').lower() == 'true'\n    stack_id = event['StackId']\n\n    if managed:\n      config = handle_managed(request_type, notification_configuration)\n    else:\n      config = handle_unmanaged(bucket, stack_id, request_type, notification_configuration)\n\n    put_bucket_notification_configuration(bucket, config)\n  except Exception as e:\n    logging.exception(\"Failed to put bucket notification configuration\")\n    response_status = \"FAILED\"\n    error_message = f\"Error: {str(e)}. \"\n  finally:\n    submit_response(event, context, response_status, error_message)\n\ndef handle_managed(request_type, notification_configuration):\n  if request_type == 'Delete':\n    return {}\n  return notification_configuration\n\ndef handle_unmanaged(bucket, stack_id, request_type, notification_configuration):\n  external_notifications = find_external_notifications(bucket, stack_id)\n\n  if request_type == 'Delete':\n    return external_notifications\n\n  def with_id(notification):\n    notification['Id'] = f\"{stack_id}-{hash(json.dumps(notification, sort_keys=True))}\"\n    return notification\n\n  notifications = {}\n  for t in CONFIGURATION_TYPES:\n    external = external_notifications.get(t, [])\n    incoming = [with_id(n) for n in notification_configuration.get(t, [])]\n    notifications[t] = external + incoming\n\n  if EVENTBRIDGE_CONFIGURATION in notification_configuration:\n    notifications[EVENTBRIDGE_CONFIGURATION] = notification_configuration[EVENTBRIDGE_CONFIGURATION]\n  elif EVENTBRIDGE_CONFIGURATION in external_notifications:\n    notifications[EVENTBRIDGE_CONFIGURATION] = external_notifications[EVENTBRIDGE_CONFIGURATION]\n\n  return notifications\n\ndef find_external_notifications(bucket, stack_id):\n  existing_notifications = get_bucket_notification_configuration(bucket)\n  external_notifications = {}\n  for t in CONFIGURATION_TYPES:\n    external_notifications[t] = [n for n in existing_notifications.get(t, []) if not n['Id'].startswith(f\"{stack_id}-\")]\n\n  if EVENTBRIDGE_CONFIGURATION in existing_notifications:\n    external_notifications[EVENTBRIDGE_CONFIGURATION] = existing_notifications[EVENTBRIDGE_CONFIGURATION]\n\n  return external_notifications\n\ndef get_bucket_notification_configuration(bucket):\n  return s3.get_bucket_notification_configuration(Bucket=bucket)\n\ndef put_bucket_notification_configuration(bucket, notification_configuration):\n  s3.put_bucket_notification_configuration(Bucket=bucket, NotificationConfiguration=notification_configuration)\n\ndef submit_response(event: dict, context, response_status: str, error_message: str):\n  response_body = json.dumps(\n    {\n      \"Status\": response_status,\n      \"Reason\": f\"{error_message}See the details in CloudWatch Log Stream: {context.log_stream_name}\",\n      \"PhysicalResourceId\": event.get(\"PhysicalResourceId\") or event[\"LogicalResourceId\"],\n      \"StackId\": event[\"StackId\"],\n      \"RequestId\": event[\"RequestId\"],\n      \"LogicalResourceId\": event[\"LogicalResourceId\"],\n      \"NoEcho\": False,\n    }\n  ).encode(\"utf-8\")\n  headers = {\"content-type\": \"\", \"content-length\": str(len(response_body))}\n  try:\n    req = urllib.request.Request(url=event[\"ResponseURL\"], headers=headers, data=response_body, method=\"PUT\")\n    with urllib.request.urlopen(req) as response:\n      print(response.read().decode(\"utf-8\"))\n    print(\"Status code: \" + response.reason)\n  except Exception as e:\n      print(\"send(..) failed executing request.urlopen(..): \" + str(e))\n"
        },
        "Handler": "index.handler",
        "Role": {
          "Fn::GetAtt": [
            "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleB6FB88EC",
            "Arn"
          ]
        },
        "Runtime": "python3.7",
        "Timeout": 300
      },
      "DependsOn": [
        "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleDefaultPolicy2CF63D36",
        "BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleB6FB88EC"
      ]
    }
  },
  "Parameters": {
    "AssetParametersbe270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824S3Bucket09A62232": {
      "Type": "String",
      "Description": "S3 bucket for asset \"be270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824\""
    },
    "AssetParametersbe270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824S3VersionKeyA28118BE": {
      "Type": "String",
      "Description": "S3 key for asset version \"be270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824\""
    },
    "AssetParametersbe270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824ArtifactHash76F8FCF2": {
      "Type": "String",
      "Description": "Artifact hash for asset \"be270bbdebe0851c887569796e3997437cca54ce86893ed94788500448e92824\""
    }
  }
}

[peterwoodworth]

I see,

The S3EventSource construct will create a Custom Resource under the hood. The CDK might use custom resources when functionalities are not yet offered by CloudFormation. Custom Resources require a handler, so the function you're seeing here is the handler for the Custom Resource which provisions bucket notifications.

This is the description of the class that gets called

aws-cdk/packages/@aws-cdk/aws-s3/lib/notifications-resource/notifications-resource-handler.ts

Lines 15 to 16 in 8cb5dff

	* A Lambda-based custom resource handler that provisions S3 bucket
	* notifications for a bucket.

Without this handler, bucket notifications wouldn't work!

Hope this answers your question

[namedgraph]

@peterwoodworth can you explain to me why BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleDefaultPolicy2CF63D36 allows s3:GetBucketNotification/s3:PutBucketNotification to all resources and not just the specific bucket?

Because of this, my CF template cannot go through the AWS Marketplace approval 🙄 Can I somehow change * to the bucket ARN?

[namedgraph]

Looks like this might be the issue for it: #5925

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.