New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Time.monotonic makes it impossible to line up distributed traces #361
Comments
What are you looking for to solve this issue? There are a couple of ways to do this.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're using viztracer for lightweight tracing when training pytorch models. Running in a datacenter, all of the clocks are synchronized to within some small number of ms. Since viztracer uses only the monotonic clock during tracing (absolutely the correct answer), traces from different machines will have wildly different timestamps. Since we can't force the traces to start at the same moment, the
--align_combine
feature gets them to within seconds of each other (some improvement!) but I think we can do better.It would be keen to have an option (or update the default) to calculate the offset between the system time and monotonic time during trace save, and offset the timestamp by that difference. That way, we will project the monotonic time into global time (+/- the error of the system clock), and be able to compare traces that have been combined.
If it's something you're interested in, I can look into making a PR.
The text was updated successfully, but these errors were encountered: