Aggressive Retry Behavior with Non-Existent SQS Queue for S3 Source

[Shadow6363]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When the queue for an aws_s3 source is deleted, the source retries at an extremely high frequency seemingly without any backoff. This has separately exceeded the limits of an AWS VPC DNS resolver with lookups and once that potential issue was addressed, it instead exceeded the account-level AWS SQS API limits that cannot be increased.
The retry frequency ends up sufficient to elevate the service's CPU triggering upward scaling, which just further increases the volume until the upper autoscaling limit is hit.

We would expect the behavior to instead be similar to when the queue is just empty or if a special case is needed, we would expect there to be a randomized, exponential backoff, with an eventual retry limit.

Configuration

[sources.s3_logs_ads]
type = "aws_s3"
region = "us-east-1"
compression = "gzip"
sqs.queue_url = "https://sqs.us-east-1.amazonaws.com/000000000000/vector-logs-ads-bucket"

Version

0.29.1

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response