-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vector metric vector_open_files not showing correct data and missing description in documentation #20431
Comments
Thanks @ShahroZafar . I do see that the metric is undocumented. It should, as you note, measure the number of files the
When the file is rotated, does it match one of the As an aside, you could try increasing |
No. The
We have |
Ah, I missed that you had
However, I believe the |
A note for the community
Problem
We are using file source to fetch pod logs and push them to kafka using the kafka sink. We want a mechanism which we can be certain that vector is not losing or laging far behind.
To do so, we saw that there is a metric in vector
vector_open_files
which is not mentioned in the documentation but actually exists. We assume that this metric is at any given time how many files are open by vector for reading.Our configuration is such that at any given time vector agent can be reading at max 2 files (file and the 2nd file created due to rotation containing copy of first file). However in the graph we see that the metric value reaches 3 from time to time. Also when a file is rotated and vector detects it, ideally it should complete reading that file and the vector_open_files should drop to 1.
Our main blocker shifting to vector is a way using which we can absolutely be sure that vector is upto the speed and it not lagging far behind. Also a mechanism using which we can get insights that no data is being lost while file reading.
Configuration
Currently we are testing vector at 20k requests per seconds. Our actual application can have logs produced at about 200k requests per seconds.
We haven't chose the kubernetes_logs source at the moment since we don't want any enrichment
Version
0.37.1
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered: