tracing.md

OpenTelemetry Tracing

Gubernator supports OpenTelemetry for generating detailed traces of application behavior and sending to Jaeger Tracing server.

Enabling Jaeger Exporter

Jaeger export is enabled by default and sends traces to localhost port 6831/udp.

Configure with environment variables.

OTEL_EXPORTER_JAEGER_PROTOCOL=http/thrift.binary
OTEL_EXPORTER_JAEGER_ENDPOINT=http://<jaeger-server>:14268/api/traces

See OpenTelemetry configuration spec. for a complete list of available environment variables and example.conf for possible environment variable examples.

Why not set OTEL_EXPORTER_JAEGER_AGENT_HOST?

This setting works by exporting UDP datagrams to the remote host, instead of localhost. This will work, but runs the risk of data loss. If a span is exported with sufficient data, it will exceed the network interface's MTU size and the datagram will be dropped. This is only a problem when sending to a remote host. The loopback interface MTU is typically much larger and doesn't encounter this problem.

When exporting via HTTP there is no MTU limit. This effectively bypasses the Jaeger Agent and sends directly to Jaeger's collector.

HoneyComb.io

Use the following to send trace data to honeycomb.io directly. For most production workloads you should setup Refinery

OTEL_TRACES_SAMPLER=always_on
OTEL_EXPORTER_OTLP_PROTOCOL=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<your-api-key>

Sampling

Because Gubernator generates a trace for each request, the tracing volume could exceed Jaeger's resources. In production, it is recommended to set OTEL_TRACES_SAMPLER=parentbased_traceidratio sampler type and OTEL_TRACES_SAMPLER_ARG to a decimal number between 0 (none) and 1 (all) for the proportion of traces to be sampled.

OTEL_TRACES_SAMPLER=always_on
OTEL_TRACES_SAMPLER_ARG=1.0

Distributed Traces

OpenTelemetry defines capabilities for clients to send trace ids to downstream services. That service will link the client span with the server span. When the client and server both send traces to the same Jaeger server, the trace will appear with the two spans linked in the same view.

When sending gRPC requests to Gubernator, be sure to use the otelgrpc interceptor to propagate the client's trace context to the server so it can add spans.

See gRPC section and cmd/gubernator-cli/main.go for usage examples.

gRPC

If using Gubernator's Golang gRPC client, the client must be created like so:

    import (
        "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
        "google.golang.org/grpc"
    )

    // ...

    opts := []grpc.DialOption{
        grpc.WithBlock(),
        grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
        grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
    }

    endpoint := "<your-endpoint-url>"
    conn, err := grpc.DialContext(ctx, endpoint, opts...)

HTTP

If using HTTP, the tracing ids must be propagated in HTTP headers. This is typically done using OpenTelemetry instrumentation, such as otelhttp.

See OpenTelemetry registry for instrumentation using many other HTTP frameworks.

Gubernator Standlone

When deployed as a standalone daemon, Gubernator's gRPC service will receive embedded trace ids in requests from the client's context object.

For this to work, the client must be configured to embed tracing ids.

Gubernator Module

When embedded into a dependent codebase as a Go module, most all Gubernator functions create spans linked to the trace ids embedded into the context object.

Follow the same steps to configure your codebase as the Gubernator standalone, above.

Holster Tracing Library

Read more at: https://github.com/mailgun/holster/blob/master/tracing/README.md