deprecation of host_name in favour of k8s_node_name for K8s

[prashant-shahi]

We recently deprecated host_name for Kubernetes in the favour of k8s_node_name.
Now, users must have two separate dashboards to monitor K8ss nodes or non-K8s nodes.

cc @ankitnayan

[srikanthccv]

If there were to be one single dashboard, there should be an attribute which is common across the environments. There is no such attribute since host.name mean different in a different context. We can't add the host.name as k8s.node.name in k8s because it gives a wrong impression that host.name is the node name when in practice, it is not. To make it more explicit, let's say we use downward API or something to set the host.name with node and enrich the host metrics. While this solves the problem of having a single dashboard for host metrics, it introduces additional complications.

1. Incorrect representation of host.name.
2. Correlation with another telemetry i.e if I see a spike in host metrics CPU and want to see traces, the host.name hack from the host metrics doesn't translate to traces because there is no host.name but only k8s.node.name.

[prashant-shahi]

@srikanthccv In case of hybrid environments with hostmetrics dashboard to monitor VMs, we would still see pod name of OtelAgent as the host.name attribute.

Is there any proper way to separate it other than adding filter in each widgets to skip one with k8s.* labels?

[srikanthccv]

In case of hybrid environments with hostmetrics dashboard to monitor VMs, we would still see pod name of OtelAgent as the host.name attribute.

Is this under the assumption that we will use a single dashboard for all environments?

Is there any proper way to separate it other than adding filter in each widgets to skip one with k8s.* labels

I didn't understand this question.

[prashant-shahi]

Is this under the assumption that we will use a single dashboard for all environments?

Not necessarily. It can have two dashboards. But in this case, K8s dashboard would be all good, but the other one would have otelAgent pod names.

I didn't understand this question.

related to above. question being, if not single dashboard, how to go about separating it?