Important metrics to be recorded for athens

[manugupt1]

Metrics can be used to measure how Athens is performing. We should have metrics for both Olympus and Proxy.

Some metrics can be storage, latency, how much data has been transferred

Along with that, we want to see how much do we want to measure. For example,
A) Disk usage / day.
B) Errors / minute
C) Time to process a request / second

and so on.

It will be great if we can have a clear idea of the metrics that we want and create them as sub tickets for this one and start measuring one.

[ghost]

Related to: #360

[michalpristas]

i'm thinking of

• Request - count (total, failed, succ), duration (ms),
• Errors - count,
• Disk usage
• network usage
• For workers number of in/out work items and time spent in a queue
• Cache hit rate (because it's fun to watch)

I imagine a PR for a bullet point, so it's not so huge

[timraymond]

I'll give this a shot. As discussed in Slack on 9/21/18 (discussion began roughly here), we're going to try to do this with opencensus.io, and more specifically, the ochttp plugin. There's some question as to how it can be integrated with Buffalo, which is part of what I'll be investigating as I look into this. Initial goals are going to be getting Go runtime stats and RED-style metrics[1] around HTTP handlers. This should coincide with roughly the first two bullet points on @michalpristas 's list.

[1] Stands for (R)ate, (E)rrors, (D)uration. This article does a decent job at introducing it, along with its cousin, USE (Utilization, Saturation, Errors).

[arschles]

@timraymond still interested in giving this a shot?

[timraymond]

@arschles Apologies, I had to take a step back due to some family issues that came up back in November. If someone else wants to pick this up, please feel free to do so.

Fortunately, things are starting to settle down again, so I might be able to contribute soon :)

[bndw]

Would adding per-module metrics be of use to anyone else? I'd like to be able to audit which packages are regularly accessed (think a counter with fields like name, version).

[abursavich]

@bndw, I would vote against any per-module metrics because they could have unbounded cardinality, which usually isn't ideal for real-time monitoring systems and may be more suited to (offline) log aggregation.

[bndw]

@abursavich Good point, I agree that the general import all the modules use case probably makes per-module metrics a bad idea.

I was looking at this through a corp lens where module usage is more constrained.

[arschles]

@bndw @abursavich do you think that a simple counter for top-level paths is still too much cardinality?

[bndw]

@arschles I think too much is subjective. Personally, our use-case needs metrics around package usage. Ideally Athens can provide that without us having to maintain a fork.

[arschles]

I totally lost track of this one. Sorry @bndw - I agree that too much is subjective. I would say that Athens by default emits a "medium" amount of cardinality, and then you can turn it up as needed, via a config variable. Not sure about this, but would you be open to having Athens emit more metrics as the log level goes up?

[ghost]

I imagine a PR for a bullet point, so it's not so huge

agreed, and maybe just the dashboard itself with whoever starts the first one.

[arschles]

@robjloranger we already have a prometheus stats exporter. would you be cool with expanding our prometheus output (possibly based on the log level) and then using the built-in prometheus dashboard for #360?

[linzhp]

Can I propose two more metrics?

• vcsLister.List
• goGetFetcher.Fetch

These two operations both depend on external services and are both expensive. It would be helpful to track their latency and count (success, failure).

For implementation, both operations call os/exec. Is there any existing View that I can use?