Probe CRD Add all SD variants

[chipzzz]

Component(s)

Probe CRD

What is missing? Please describe.

Probe CRD is missing all regular SD variants that are available in the regular scrape configuration. Users that utilize blackbox exporter and probe targets using http_sd_config or consul_sd_config and others cannot use probe CRD in lieu of the scrape_config.

It was mentioned here that this would be covered by scrapeConfig CRD, however, I believe that still only works for external kubernetes clusters and not external endpoints.

#5269

Describe alternatives you've considered.

Custom SD that provides list to configmap - defeats the purpose.

Environment Information.

Environment

Kubernetes Version: latest
Prometheus-Operator Version:
latest

[slashpai]

It should be possible with new ScrapeConfig CRD https://prometheus-operator.dev/docs/user-guides/scrapeconfig/

API:https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1alpha1.ScrapeConfig

[slashpai]

ScrapeConfig should work for the scenario, have you tried it?

[chipzzz]

@slashpai ,how exactly would that work? the probe selector is defined at prometheus level. Probe CRD itself requires static_configs which would replace what you would define in scrape_config(non-crd) if you were using blackbox exporter +scrapeconfig(non-crd) to probe targets. There's no probe crd selector in the scrapeConfig CRD.

Am i missing something? Which component of the ScrapeConfig CRD should I be using to probe targets via Probe CRD like I would with regular scrape_config(non-crd) + balckbox exporter+blackbox modules ?

[slashpai]

IIUC what you are looking for is to observe the services registered with external consul. In that case you would need ScrapeConfig only and Probe won't be needed. Note that ScrapeConfig is v1alpha1 API

Is that what you are looking for or something else

[chipzzz]

@slashpai no, i'm not trying to scrape metrics.. blackbox exporter is used for probing endpoints. I.e hit www.google.com and get a response. Probe CRD was designed to replace some of the probing config that we would be otherwise placed in prometheus scrape config. Probe crd currently only allows static_configs and ingress , it doesn't allow http_sd_configs https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#probe

so it does not allow dynamic querying of and endpoint and getting a list of hosts back you have to list the endpoints statically, same with consul doesn't allow to hit consul endpoints dynamically and probe them. I would have to list them manually via static_configs.

[slashpai]

ok I understand now. AFAIK we didn't have any plans to add more SD's too probe as we are developing more on ScrapeConfig.

May be @simonpasquier will have more context why other sd's were not supported in Probe.

[simonpasquier]

Now that we have the ScrapeConfig CRD, I don't think that we need/want to add more service discoveries to Probes.

[chipzzz]

@simonpasquier , For the missing SD's in probe how will ScrapeConfig CRD enable that? Probe CRD replaces the config that you would have in additional scrape configs, and where you would have setup the SD in. Those SD's are not available in Probe CRD

[simonpasquier]

My point is that you can use the ScrapeConfig CRD to scrape targets such as blackbox exporter.

[chipzzz]

@simonpasquier , yes but i'm just trying to probe them i.e probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC. Not scrape them.