-
Notifications
You must be signed in to change notification settings - Fork 359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support configuring a poll interval in s3 Kafka Connect Source #1229
Comments
Hi @jessebye , Would a backoff also be useful for you? I am thinking if there are no new objects to slow down the polling interval up to a given max threshold. |
@stheppi Backoff would be nice to have. I could see that being useful in prod configurations too. However, the main issue we'd like to address is our dev environment costs. Currently it's about $300 USD per day to keep a few s3 connectors active because they poll every 40ms or so. If we could configure that to poll every 60s, it would cut our bills dramatically. |
What is the expected behaviour?
We'd like to be able to configure a polling interval for the s3 Kafka Connect source so we can reduce AWS i/o charges incurred by frequent polling.
What was observed?
There is no way to configure the polling interval. I'm not sure what the default is, but it's very frequent and causing high AWS bills in our development environments.
The text was updated successfully, but these errors were encountered: