[Jsonnet template] Support for multiple prometheuses statefulsets.

[Sjd-Risca]

What is missing?

I'm looking forward for a way to deploy multiple prometheuses statefulsets via the provided jsonnet template and the prometheus-operator.

Why do we need it?

If it could be possible to have multiple prometheuses instances groups, then it would be easy to arbitrarily shard the monitoring stack. This is a little different from the current automatic sharding option since this way I could have some instances dedicated only to certain targets/rules via proper namespace/label selector. Applying the sharding on the current jsonnet template is really easy, not so easy instead to create a partitioning via multiple prometheus instances.

I would expect some benefits such as:

• easier scaling of monitoring workload;
• management improvement such as instances dedicated to different teams or different operational profiles.

Environment

• kube-prometheus version:

  Latest: e7fa2f0b3c7606296dba118cc1759e8055f7215b

Anything else we need to know?:

Obviously the same consideration about the sharding and replica are still valid about proper metrics inspection and rules evaluation (such as prometheus federation or thanos backend).
It could also still be desired to have the option of the automatic sharding inside of the specific prometheus statefulset itself.

[PaintedShepherd]

You can already do this, just apply a new Prometheus Resource.
Like here: https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-prometheus.yaml

[Sjd-Risca]

@PaintedShepherd : sorry, but I think that the description of my issue was not clear enough (my fault, I will fix it soon). At current time I'm already applying multiple Prometheus Resources, but I would like a way to set it easily with the current definition at prometheus.libsonnet.
By now I have done an ugly and heavy rewrite of it, but I'd like something more cleaner and immediate to use.