-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Amazon S3: Mismatch when reading HTTP header from GCS #8790
Labels
Comments
gouyelliot
changed the title
Amazon S3: Mismatch when reading HTTP header when using GCS
Amazon S3: Mismatch when reading HTTP header using GCS
May 3, 2024
gouyelliot
changed the title
Amazon S3: Mismatch when reading HTTP header using GCS
Amazon S3: Mismatch when reading HTTP header from GCS
May 3, 2024
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Describe the bug
While configuring our FluentBit instance, I reached a situation when the Amazon S3 Output Plugin would block for several minutes when trying to send the files.
I'm using Google Cloud Storage with a HMAC key (set with env vars), using the
endpoint
configuration set to https://storage.googleapis.com.Here is the log when the plugin is trying to send the data:
After digging in the source code, I found that the problem comes from the header_lookup function, which get the value of a header from the HTTP response.
Turns out that Google have a custom HTTP header named
x-goog-stored-content-length
, which is matched by theheader_lookup
instead of theContent-Length
header here, resulting in the client trying to read from the socket again, and timing out after 4 minutes.Here a example of HTTP response payload from GCS:
To Reproduce
Here my current config
Expected behavior
The HTTP client should not use the
x-goog-stored-content-length
header as the content length of the request.I'll try to create a PR next week, the bug is actually no hard to fix !
Your Environment
The text was updated successfully, but these errors were encountered: