(AWS Lambda) Service: AWSLambdaInternal; Status Code: 404; Error Code: ProvisionedConcurrencyConfigNotFoundException

[golesmn]

Describe the bug

We have some Lambda functions with scheduled auto-scaling in our sandbox environment. We use auto-scaling to scale down the provisioned concurrency to 0 outside of office hours. However, when we try to update the stack outside of office hours, the update fails due to insufficient provisioned concurrency.

If we check the provisioned concurrency outside office hours using cli its the result

Expected Behavior

Deploy the stack successfully.

Current Behavior

Lambda provisioned concurrency not set and deployment failed.

Reproduction Steps

• Make a scalable target for a lambda
• Set the scheduled provisioned concurrency and set min and max to 0 for first schedule and 1 for other schedule.
• Let the first schedule run and make sure the current provisioned concurrency is 0
• Deploy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.133.0

Framework Version

No response

Node.js Version

16

OS

linux

Language

Python

Language Version

Python3.9

Other information

No response

[pahud]

Thank you for your report.

Can you share full minimal code snippets that represents this issue so we could see more details about your CDK code?

[golesmn]

@pahud

def lambda_alias(
    scope: Construct,
    function_name: str,
    function: aws_lambda.Function,
    removal_policy: RemovalPolicy,
    concurrent_execution_value: int,
    scheduled_autoscaling_configs: dict[str, Any],
    autoscaling: bool,
) -> aws_lambda.Alias:
    current_version = function.current_version
    alias = aws_lambda.Alias(
        scope=scope,
        id=f"{function_name}FunctionAlias",
        alias_name=get_alias_name(function_name),
        version=current_version,
        provisioned_concurrent_executions=concurrent_execution_value,
    )
    alias.apply_removal_policy(removal_policy)

    if autoscaling:
        scaling_target = aws_applicationautoscaling.ScalableTarget(
            scope=scope,
            id=f"{function_name}ScalableTarget",
            service_namespace=aws_applicationautoscaling.ServiceNamespace.LAMBDA,
            min_capacity=0,
            max_capacity=concurrent_execution_value,
            resource_id=f"function:{function_name}:{get_alias_name(function_name)}",
            scalable_dimension="lambda:function:ProvisionedConcurrency",
        )
        scaling_target.node.add_dependency(alias)

        scaling_down_starting_hour = str(
            scheduled_autoscaling_configs.get("scaleInStartingHour")
        )
        scaling_down_ending_hour = str(
            scheduled_autoscaling_configs.get("scaleInEndingHour")
        )

        night_schedule = aws_applicationautoscaling.Schedule.cron(
            hour=scaling_down_starting_hour, minute="0"
        )
        scaling_target.scale_on_schedule(
            id=f"{function_name}ScheduledDownScaling",
            schedule=night_schedule,
            max_capacity=0,
            min_capacity=0,
            time_zone=TimeZone.EUROPE_BERLIN,
        )
        morning_schedule = aws_applicationautoscaling.Schedule.cron(
            hour=scaling_down_ending_hour,
            minute="45",
        )
        scaling_target.scale_on_schedule(
            id=f"{function_name}ScheduledUpScaling",
            schedule=morning_schedule,
            min_capacity=concurrent_execution_value,
            max_capacity=concurrent_execution_value,
            time_zone=TimeZone.EUROPE_BERLIN,
        )

    return alias

here concurrent_execution_value is 0 for night schedule and > 1 for morning schedule.

[nmussy]

ProvisionedConcurrencyConfigNotFoundException seems like a CloudFormation error, and I'm not sure there's a lot that should be done in the CDK code base for this specific use-case. What you should be able to do is use custom resources to update the concurrency on the fly, scaling up the capacity before the update and lowering it after it's done

[golesmn]

@nmussy

ProvisionedConcurrencyConfigNotFoundException seems like a CloudFormation error, and I'm not sure there's a lot that should be done in the CDK code base for this specific use-case. What you should be able to do is use custom resources to update the concurrency on the fly, scaling up the capacity before the update and lowering it after it's done

Yeah we were thinking the same. But I was looking if we could do it without scaling up before deployment. But it seems we can't :)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[nmussy]

@nmussy

ProvisionedConcurrencyConfigNotFoundException seems like a CloudFormation error, and I'm not sure there's a lot that should be done in the CDK code base for this specific use-case. What you should be able to do is use custom resources to update the concurrency on the fly, scaling up the capacity before the update and lowering it after it's done

Yeah we were thinking the same. But I was looking if we could do it without scaling up before deployment. But it seems we can't :)

Yeah, it's one of those unfortunate things with CloudFormation. When you do come up with the solution, feel free to leave it here in case someone stumbles on the same issue in the future 👍