-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request.Send() doesn't close open connections, leading to 'too many open files' error #4695
Comments
Hi @colmsnowplow, This is quite an old issue. I apologize for not getting to this sooner. If this is still an issue with the Go SDK v2, please open a new issue there and we will take a look. Thanks again, |
Comments on closed issues are hard for our team to see. |
Describe the bug
My appliciation uses a fork of the kinsumer library, which uses this package to get data from kinesis. We have run into 'too many open files' errors in production on readfrom kinesis. (In case it's relevant, running on ECS with default setting of 1024 max network ports).
From digging into the code, I believe this happens when we process events from the stream very quickly, leading to many pulls from kinesis in a short period of time. It seems that this library doesn't close connections once a shard iterator is done pulling records from the shard, and when many of these calls are made before the connection expires, we run into this issue.
We are pulling data from multiple shards concurrently, but I believe that the issue is not caused by concurrent requests being made, but many sequential requests.
I believe this occurs because:
At this point, it would be very useful to me if someone more familiar with the codebase could tell me whether the above explanation of the scenario seems valid, or explain some other explanation I could investigate.
At this point I haven't found the time to produce a reproduction, but I can find the time to do so if I'm not barking up the wrong tree.
I would also be interested to hear opinions on how to solve or get around the issue in a sustainable way (beyond increasing the amount of connections available on the box, which we are doing), assuming that it is valid and reproducable.
Expected Behavior
I expect not to run into "too many open files" errors when making many subsequent GetRecords requests.
Current Behavior
Error:
Reproduction Steps
I'm yet to find a full reproduction but I believe it can be done as follows:
GetRecords
on each, immediately ack, and callGetRecords
again immediately - do this in a loopPossible Solution
I think this can be solved if the connection we open when making the request in
GetRecords()
is closed, either when the last record is acked, or when we call a request on the next shard iterator.Alternatively, if
GetRecords()
returned something which allows me to close the request manually, I could handle it in code.Additional Information/Context
No response
SDK version used
v1.40.22 - relevant code seems equivlent in latest
Environment details (Version of Go (
go version
)? OS name and version, etc.)1.17
The text was updated successfully, but these errors were encountered: