Seeking Recommendations for Alerting Based on Vector Metrics in Kubernetes with Prometheus

[nikolasj]

ello Vector maintainers and community,

I've been using Vector within my Kubernetes cluster for log processing and forwarding. I have Prometheus set up to collect metrics from Vector. While I have a basic understanding of the metrics provided, I'm looking for recommendations or best practices when it comes to alerting based on these metrics to ensure the health and performance of Vector.

Here are some of the metrics I've been collecting:

vector_buffer_byte_size
vector_buffer_events
vector_buffer_max_byte_size
vector_buffer_max_event_size
vector_buffer_received_bytes_total
Could anyone share their experience or recommendations on setting up alerts for these (or other relevant) Vector metrics?

Are there any specific thresholds or patterns that the community has found to be particularly indicative of issues or noteworthy situations?

Any advice, reference configurations, or links to best practices would be greatly appreciated. Thank you in advance!

[arch-xtof]

Hello, I am using Vector to process 10s of millions of logs per day. So far, the only alert I have created with Vector metrics is for log delivery rate. It is calculated the following way:

<sent_events_within_period>
/
<received_events_within_period_that_are_expected_to_be_fully_forwarded>

Here is an example query

sum by (region) (increase(vector_component_sent_events_total{app="vector-gelf", component_type="loki"}[5m])) 
/ 
sum by (region) (increase(vector_component_sent_events_total{app="vector-gelf", component_kind="transform", component_name="filter_logs"}[5m]))