Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unit "bytes" not a suffix of metric "process_runtime_dotnet_gc_allocations_size_bytes_total" #5502

Closed
RoyLi-Pvt opened this issue Apr 3, 2024 · 4 comments · Fixed by #5646
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package

Comments

@RoyLi-Pvt
Copy link

RoyLi-Pvt commented Apr 3, 2024

Bug Report

List of all OpenTelemetry NuGet packages and version using:

OpenTelemetry.Exporter.Console" Version="1.8.0"
OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.8.0"
OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.8.0-rc.1"
OpenTelemetry.Extensions.Hosting" Version="1.8.0"
OpenTelemetry.Instrumentation.AspNetCore" Version="1.7.1"
OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.0.0-beta.10"
OpenTelemetry.Instrumentation.GrpcNetClient" Version="1.7.0-beta.1"
OpenTelemetry.Instrumentation.Http" Version="1.7.1"
OpenTelemetry.Instrumentation.Process" Version="1.0.0-alpha.6"
OpenTelemetry.Instrumentation.Runtime" Version="1.7.0"

Runtime version:

net8.0

Symptom

Prometheus pull error msg : unit "bytes" not a suffix of metric "process_runtime_dotnet_gc_allocations_size_bytes_total"
Prometheus Build Info:
Version 2.51.1
Rev 855b5ac4b80956874eb1790a04c92327f2f99e38

Init Code:

  Telemetry.WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation()
            .AddProcessInstrumentation()
            .AddMeter("Microsoft.AspNetCore.Hosting")
            .AddMeter("Microsoft.AspNetCore.Routing")
            .AddMeter("Microsoft.AspNetCore.Diagnostics")
            .AddMeter("Microsoft.AspNetCore.Server.Kestrel")
            .AddMeter("Microsoft.AspNetCore.Http.Connections")
            .AddMeter("Microsoft.Extensions.Diagnostics.HealthChecks")
            .SetMaxMetricStreams(500)
            .SetMaxMetricPointsPerMetricStream(200)
            .AddPrometheusExporter();
    });

This error occurred after upgrading the version of OpenTelemetry.Exporter.Prometheus.AspNetCore to 1.8.0-rc1 and no longer occurred after downgrading to 1.7.0-rc1.

@RoyLi-Pvt RoyLi-Pvt added the bug Something isn't working label Apr 3, 2024
@vishweshbankwar vishweshbankwar added the pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package label Apr 4, 2024
@xiang17
Copy link
Contributor

xiang17 commented Apr 11, 2024

This is the same issue as open-telemetry/opentelemetry-dotnet-contrib#1617.

You can configure the Prometheus exporter to disable the _total suffix with this option: AddPrometheusExporter(o => o.DisableTotalNameSuffixForCounters = true). The option is included in releases starting with 1.8.0-beta.1.

@hacst
Copy link

hacst commented Apr 17, 2024

@xiang17 Could you (or someone else familiar) clarify whether this is a problem with AddRuntimeInstrumentation of the contrib that should be fixed there, or something to be fixed in the core implementation here? Having the workaround with the parameter is good to know, but it is non-default and the update breaking previously working exports is quite concerning.

All of this becoming only visible when prometheus fails to get metrics for a service after update is not ideal either. Shouldn't the implementation behind AddPrometheusExporter prevent us from (silently) producing output prometheus cannot handle?

@bIgBV
Copy link

bIgBV commented May 6, 2024

Agreed, we also ran into this issue after deploying a service change which included the AddRuntimeInstrumentation call. Ideally, if it is a known breaking issue, wouldn't it be better to set the DisableTotalNameSuffixForCounters to true by default?

@kamisoft-fr
Copy link

This is the same issue as open-telemetry/opentelemetry-dotnet-contrib#1617.

You can configure the Prometheus exporter to disable the _total suffix with this option: AddPrometheusExporter(o => o.DisableTotalNameSuffixForCounters = true). The option is included in releases starting with 1.8.0-beta.1.

Same problem here with npgsql metrics, this workarround does not work in this case:

it transforms "db_client_commands_bytes_written_total" in "db_client_commands_bytes_written"

so we still get unit "bytes" not a suffix of metric "db_client_commands_bytes_written"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants