Replies: 1 comment
-
Hi, |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, |
Beta Was this translation helpful? Give feedback.
-
Our team has encountered OOM crashed of vector running on k8s several times,
after reading the document about buffering model,
we got confused because we have all the buffer settings to be default
(we didn't actually write these in the config)
With a Kafka source which has average event size to be ~300 bytes, we expected the max memory usage of vector should be around
300 bytes * 500 # = 150 kb
and when buffer is full, source should stop consuming message from Kafka
but the memory usage reached over 1Gb, and messages are still consumed from Kafka until the pod was OOM killed instead
So we did a experiment, set
buffer.max_events
to 1 and have a http sink that does not exist, and vector still consumed more than 10000 events from Kafkaafter digging into the source code, I found that sender has its own memory buffer:
vector/src/topology/builder.rs
Line 61 in 98c54ad
it can buffer thousands of events before sending to sink memory buffer, and this buffer size seems to be unconfigurable
We also got many logs like this
WARN sink{component_kind="sink" component_id=sink component_type=aws_kinesis_streams component_name=sink}:request{request_id=1977859}: vector::sinks::util::retries: Retrying after error. error=TransientError: connection closed before message completed internal_log_rate_limit=true
Seems that requests were retried over and over again while events were still consumed from the Kafka source
Would this be the reason that we got such high memory usage?
As my current understanding there are at least 3 buffers for a event to go through before sending out
source buffer => sink buffer => sink batching buffer
where do the events waited for retrying stand?
And it would be really helpful if there are a page describes all the settings that could affect the memory usage of a vector process, thanks🙏
Beta Was this translation helpful? Give feedback.
All reactions