Support Prometheus via plugin

[alexey-pelykh]

Sending stats and access logs to https://prometheus.io/ would be a great addition. For inspiration, there's https://github.com/gmm42/tinyproxy_exporter/blob/master/tinyproxy_exporter

[abhinavsingh]

Thank you, I'll look into it :)

[magmax]

Hi there!

I'd like to try to fix this, if you don't mind.

which is the best approach? Perhaps to give support on core to store metrics, so plugins can send their own metrics there and be served from the prometheus plugin?

[abhinavsingh]

@magmax Thank you, that would be great and much appreciated.

Indeed giving support directly in the core would be the best way to implement it. I think proxy.py in-built eventing mechanism might be a good fit here. We currently use eventing for CDT integration. Here is a little example of using eventing mechanism in standalone fashion and a little abstract about why we built eventing feature within proxy.py

1. Eventing can be enabled by passing --enable-events flag
2. Just as with CDT integration, each plugin/core/others can publish metrics into a single queue
3. A subscriber can receive these published metrics and send to upstream prometheus or other endpoints

[magmax]

Nice, @abhinavsingh !

I still have some problems with this XD

First of all, Prometheus works on "pull" method instead of "push". That means that each resource won't send data to the server; it is the Prometheus server which will ask for data whenever it wants. That means the prometheus plugin will require to add an endpoint to the webserver.

So... This is the plan:

1. I will start by enabling events and allow core and plugins to send metrics.
2. Then, I can create the plugin to consume those metrics and render them in screen.
3. Finally, I can show them in an endpoint on the web server.

I've thought in avoiding to depend on prometheus-client library on core, what means I will require to duplicate part of its functionality: creating some types of metrics to allow communication between emitter, core and prometheus-plugin.

[abhinavsingh]

@magmax Whoops, correct. Subscriber can simply accumulate and store metrics, while an endpoint indeed will be necessary from where prometheus can pull the data. While onto it, you can define an abstract class (to write/read metrics and associated basic types) and write a single implementation of your choice :)

• in-memory
• on disk (files/sqlite/time-series db)
• in cache (redis, memcache, ..)

I've thought in avoiding to depend on prometheus-client library on core

Indeed. We can start with adding a couple of metric types. Flow wise, event emitters will emit these types, subscribers will accumulate these types and store.

You can tie all this functionality under 2 flags:

1. --enable-metrics (which internally can also --enable-events) -- This flag essentially starts metric storage collection
2. --prometheus-path -- This flag simply adds an endpoint at provided /path/ which can be polled by prometheus

And 1), 2) can both be provided independently. E.g even if --prometheus-path is provided without --enable-metrics, endpoint will keep returning empty responses. The 2nd part (--prometheus-path) itself can be its own plugin. And it's perfectly ok to use prometheus-client within this plugin file. This file will read out of proxy.py core metrics and convert them into prometheus objects.

PS: While I do understand the concept, I have not used prometheus as much as I would have liked, so I'll have to bank upon yours and community expertise for final integration here :)