Replies: 2 comments
-
I would take a look at some of the librdkafka options available here: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md |
Beta Was this translation helpful? Give feedback.
0 replies
-
I had similar experiences, as discussed here: #19278 Solution is to have multiple sources/sinks on single server, or spread load over multiple smaller servers. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
My pipeline configuration consists of a source node and sink node, from source kafka to sink kafka. the configuration is as shown in the figure. Due to the large traffic from the source kafka, it was found that there was a lag between the source and the sink kafka. When I run vector on a host with 16 CPUs and 32G memory, the configured in this pipeline writes to the sink kafka at a speed of 1.1Gib/s, and the CPU usage is only 25% at this time. When the number of copies of the same pipeline configuration is increased to 4, the writing the sink kafka speed can reach 2.9Gib/s, and the CPU usage reaches 90% at this time. I want to know how to in one configure kafka sink to improve write speed and concurrency?
BTW: I have tried increasing the number of sink partitions but it has no effect. and setting environmental VECTOR_EXPERIMENTAL_REQUEST_BUILDER_CONCURRENCY to increase the number has also no effect
Beta Was this translation helpful? Give feedback.
All reactions