You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We're setting up a single collector pod in a StatefulSet configured to monitor "all the rest" of our OTEL components ... this collector is called the otel-collector-cluster-agent, and it uses the TargetAllocator to monitor for ServiceMonitor jobs that have specific labels. We currently have two different ServiceMonitors - one for collecting otelcol.* metrics from the collectors, and one for collecting opentelemtry_.* metrics from the Target Allocators.
We are seeing metrics reported from the TargetAllocator pods duplicated into DataPoints that refer to the right, and wrong ServiceMonitor:
Metric #14
Descriptor:
-> Name: opentelemetry_allocator_targets
-> Description: Number of targets discovered.
-> Unit:
-> DataType: Gauge
NumberDataPoints #0
Data point attributes:
-> container: Str(ta-container)
-> endpoint: Str(targetallocation)
-> job_name: Str(serviceMonitor/otel/otel-collector-collectors/0) <<<< WRONG
-> namespace: Str(otel)
-> pod: Str(otel-collector-cluster-agent-targetallocator-bdf46447d-6stx7)
-> service: Str(otel-collector-cluster-agent-targetallocator)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-05-02 16:16:00.917 +0000 UTC
Value: 36.000000
NumberDataPoints #1
Data point attributes:
-> container: Str(ta-container)
-> endpoint: Str(targetallocation)
-> job_name: Str(serviceMonitor/otel/otel-collector-target-allocators/0) <<<< RIGHT
-> namespace: Str(otel)
-> pod: Str(otel-collector-cluster-agent-targetallocator-bdf46447d-6stx7)
-> service: Str(otel-collector-cluster-agent-targetallocator)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-05-02 16:16:00.917 +0000 UTC
When we delete the otel-collector-collectors ServiceMonitor, the behavior does not change... which is wild... however, if we delete the entire stack and namespace and then re-create it without the second ServiceMonitor, then the data is correct... until we create the second Service Monitor, then it goes bad again.
We should see datapoints for opentelemetry_.* metrics that only come from the target allocator pods and are attributed once .. meaning, one DataPoint per target pod, and that's it:
Metric #14
Descriptor:
-> Name: opentelemetry_allocator_targets
-> Description: Number of targets discovered.
-> Unit:
-> DataType: Gauge
Value: 36.000000
NumberDataPoints #0
Data point attributes:
-> container: Str(ta-container)
-> endpoint: Str(targetallocation)
-> job_name: Str(serviceMonitor/otel/otel-collector-target-allocators/0) <<<< RIGHT
-> namespace: Str(otel)
-> pod: Str(otel-collector-cluster-agent-targetallocator-bdf46447d-6stx7)
-> service: Str(otel-collector-cluster-agent-targetallocator)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-05-02 16:16:00.917 +0000 UTC
Component(s)
collector
What happened?
Component(s)
receiver/prometheus
What happened?
Description
We're setting up a single collector pod in a StatefulSet configured to monitor "all the rest" of our OTEL components ... this collector is called the
otel-collector-cluster-agent
, and it uses the TargetAllocator to monitor forServiceMonitor
jobs that have specific labels. We currently have two differentServiceMonitors
- one for collectingotelcol.*
metrics from the collectors, and one for collectingopentelemtry_.*
metrics from the Target Allocators.We are seeing metrics reported from the TargetAllocator pods duplicated into
DataPoints
that refer to the right, and wrong ServiceMonitor:When we delete the
otel-collector-collectors
ServiceMonitor, the behavior does not change... which is wild... however, if we delete the entire stack and namespace and then re-create it without the second ServiceMonitor, then the data is correct... until we create the second Service Monitor, then it goes bad again.Steps to Reproduce
This creates a
/scrape_configs
that looks like this:Expected Result
We should see datapoints for
opentelemetry_.*
metrics that only come from the target allocator pods and are attributed once .. meaning, oneDataPoint
per target pod, and that's it:Collector version
0.98.0
Environment information
Environment
OS: BottleRocket 1.19.2
OpenTelemetry Collector configuration
No response
Log output
No response
Additional context
No response
Kubernetes Version
1.28
Operator version
0.98.0
Collector version
0.98.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: