Dashboard with external prometheus

[arichard42]

Hi,

I am deploying rook-ceph-cluster using an external kube-prometheus-stack.

The relevant values I use to deploy the operator are :

monitoring:
  enabled: true

And the cluster relevant values :

monitoring:
  enabled: true
  createPrometheusRules: true
  rulesNamespaceOverride: monitoring
  prometheusRule:
    labels:
      release: prometheus
cephObjectStores:
  - name: ceph-objectstore
    spec:
      gateway:
        port: 80
        instances: 2

I am currently experiencing two well identified problems with dashboard performance graph :

1. Disk utilization

Some graphs are showing empty are N/A indication for some disk io statistics.

For example one Cluster/Hosts/Overall Performance, the AVG Disk Utilization frame show N/A.
Stats about latencies on Cluster/OSDs/Overall Performance are also empty.

Looking on the prometheus/grafana side, it appears this is because prometheus have replaced the instance label with a host:port indication, copying the original instance label in an exported_instance label.

This is because the relevant serviceMonitor/rook-ceph/rook-ceph-mgr generated by rook operator do not have the "honor_labels: true" indication.

Editing manually the serviceMonitor/rook-ceph/rook-ceph-mgr resource solves the problem, but this is not a long term solution as this resource is generated internaly by the operator at the next operator or cluster update.

2. Object store with two instances

When I deploy the object store with two instances, the two instances are using the same name and id :

ceph orch ps --daemon_type=rgw
NAME                  HOST                 PORTS  STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION    IMAGE ID      
rgw.ceph-objectstore  kub4.equation.intra         running (3w)     0s ago   3w        -        -  <unknown>  8c1697a0a924  
rgw.ceph-objectstore  kub5.equation.intra         running (6w)     0s ago   6w        -        -  <unknown>  798f1b1e71ca  

and most of the object gateway dashboard shows no data with an error "found duplicate series for the match group".

Looking at a cephadm (non-rook) deployed cluter, the instances get different ids :

ceph orch ps --daemon_type=rgw
NAME                       HOST                      PORTS  STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  
rgw.test.linuxagr2.gxvxyf  linuxagr2.equation.intra  *:80   running (63m)     2m ago  63m    85.9M        -  18.2.2   3183778aa140  bf83fe3c2c27  
rgw.test.linuxagr.zxbvhr   linuxagr.equation.intra   *:80   running (63m)     2m ago  63m    90.5M        -  18.2.2   3183778aa140  307f520d3dd5  

The only solution so far is to have a single gw instance.

[BlaineEXE]

Regarding the RGW instances in the dashboard, I think @rkachach has done some investigation into that.

Regarding the monitoring labels, you can provide additional labels on the mgr using this mechanism: https://rook.io/docs/rook/latest-release/CRDs/Cluster/ceph-cluster-crd/#annotations-and-labels

[BlaineEXE]

I also don't fully understand what you mean here:

Looking on the prometheus/grafana side, it appears this is because prometheus have replaced the instance label with a host:port indication, copying the original instance label in an exported_instance label.

If you could show us in an example what you mean, it could help us get more clear about what you're trying to explain and how it is proving to be a problem. It seems that another means of addressing this may be to alter the metrics to not use host:port.

[rkachach]

As @BlaineEXE explained I was looking into the no-unique RGW instance id issue for a while. It's a known issue which is being tracked by #13674, unfortunately at this moment we don't have a definitive solution yet. As for the point nº 1 if you are sure that's a rook issue I'd recommend opening a BUG report and providing as much detailed information as you can.

[arichard42]

I just tested on rook-ceph 1.14.5 and the disk utilization issue on ceph dashboard is still present. You may easily see that by displaying the ceph-dashboard/host overview dashboard from grafana : when the bug is present, the AVG disk utilization panel shows N/A on rook and the percentage of disk utilization on a bare metal ceph cluster managed by cephadm.

When deploying a bare metal ceph cluster using cephadm, the prometheus instance is configured with :

scrape_configs:
- job_name: ceph
  honor_labels: true
  ...
scrape_configs:
- job_name: ceph-exporter
  honor_labels: true
  ...

When looking at some prometheus values like ceph_disk_occupation, the instance is the host name of the device :

ceph_disk_occupation{ceph_daemon="osd.3", device="/dev/dm-3", device_ids="nvme0n2=VMware_Virtual_NVMe_Disk_VMware_NVME_0000", devices="nvme0n2", instance="linuxagr.equation.intra", job="ceph"}
ceph_disk_occupation{ceph_daemon="osd.4", device="/dev/dm-3", device_ids="nvme0n2=VMware_Virtual_NVMe_Disk_VMware_NVME_0000", devices="nvme0n2", instance="linuxagr2.equation.intra", job="ceph"}
ceph_disk_occupation{ceph_daemon="osd.5", device="/dev/dm-2", device_ids="nvme0n2=VMware_Virtual_NVMe_Disk_VMware_NVME_0000", devices="nvme0n2", instance="linuxagr3.equation.intra", job="ceph"}

Under a rook-ceph cluster managed cluster, the prometheus instance is configured by the servicemonitors resources :

kubectl -n rook-ceph get servicemonitors.monitoring.coreos.com
NAME                 AGE
rook-ceph-exporter   119d
rook-ceph-mgr        181d

Theses resources are dynamically created by rook-ceph-operator :

kubectl -n rook-ceph get servicemonitors.monitoring.coreos.com rook-ceph-mgr -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2023-12-01T15:02:57Z"
  generation: 18
  labels:
    release: prometheus
    team: rook
  name: rook-ceph-mgr
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: ceph.rook.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CephCluster
    name: rook-ceph
    uid: 34ef9f0e-1532-4fba-a812-15f38f175c82
  resourceVersion: "262192918"
  uid: 91371c58-a30e-4873-b7e6-8c64f6c37e66
spec:
  endpoints:
  - interval: 10s
    path: /metrics
    port: http-metrics
  namespaceSelector:
    matchNames:
    - rook-ceph
  selector:
    matchLabels:
      app: rook-ceph-mgr
      rook_cluster: rook-ceph

and the effect is that the prometheus scape config generated do not have the "honorLabels: true" setting

When looking at the prometheus values ceph_disk_occupation, the instance is replaced by a pod address and the original instance (hostname) is put in the field exported_instance :

ceph_disk_occupation_human{ceph_daemon=~"osd.0"}
ceph_disk_occupation_human{ceph_daemon="osd.0", container="mgr", device="/dev/sdb", endpoint="http-metrics", exported_instance="kub2.equation.intra", instance="10.240.30.58:9283", job="rook-ceph-mgr", namespace="rook-ceph", pod="rook-ceph-mgr-b-58687cfc6f-pkvz9", service="rook-ceph-mgr"}

This is the normal prometheus behavior when honorlabels is false (and is the default value for this field).

A simple solution to this is to dynamically modify the servicemonitors resources in order to add "honorlabels: true" to the endpoint :

kubectl -n rook-ceph patch servicemonitors.monitoring.coreos.com rook-ceph-exporter --type='json' -p='[{"op": "add", "path": "/spec/endpoints/0/honorLabels", "value": true}]'
kubectl -n rook-ceph patch servicemonitors.monitoring.coreos.com rook-ceph-mgr --type='json' -p='[{"op": "add", "path": "/spec/endpoints/0/honorLabels", "value": true}]'

Probably the rook-ceph-operator code must be modified in order to add this when generating the servicemonitors resources.

Alain RICHARD

[travisn]

@arichard42 If you add the labels in the cephcluster CR such as this, does it solve this issue for you?

spec:
  labels:
    exporter: 
      honorLabels: "true"
    monitoring:
      honorLabels: "true"

[arichard42]

Hi @travisn,

If I add this label, the effect is that the generated ServiceMonitor get this label 👍

kubectl -n rook-ceph get servicemonitors.monitoring.coreos.com rook-ceph-mgr -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    honorLabels: "true"
    release: prometheus
    team: rook
  name: rook-ceph-mgr
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: ceph.rook.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CephCluster
    name: rook-ceph
spec:
  endpoints:
  - interval: 10s
    path: /metrics
    port: http-metrics
  namespaceSelector:
    matchNames:
    - rook-ceph
  selector:
    matchLabels:
      app: rook-ceph-mgr
      rook_cluster: rook-ceph

and this doesn't work as this has nothing to do with the required "spec.endpoints.honorLabels: true" that is necessary to solve the problem.

Regards