Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracer logs are written to disk instead of stdout/stderr in kubernetes #5362

Open
mbyiounderdog opened this issue Mar 27, 2024 · 0 comments
Open

Comments

@mbyiounderdog
Copy link

Describe the bug

We are using automatic instrumentation for the dotnet dd-trace library in Kubernetes. There is some other issue with how we are doing GRPC tracing (separate bug). It is causing the tracing library to write many logging messages to the disk, which eventually causes our pods to be killed since it is using hundreds of Mi of ephemeral storage.

The tracing log messages should not be written to a file. They should be written to stdout/stderr so they can be collected and managed automatically by Kubernetes and the Datadog Agent.

Are we missing some configuration that would let us configure that? If not, I'm sure this issue is affecting many Datadog customers. Perhaps they haven't noticed it yet if they just didn't happen to trigger log messages.

To Reproduce
Steps to reproduce the behavior:

  1. Setup the tracing library with GRPC
  2. Encounter some issue with how traces are being used (we are still investigating this)
  3. See that a pod's ephemeral storage is very high.

Expected behavior
Log messages should be written to stdout/stderr so they can be managed correctly.

Runtime environment (please complete the following information):

  • Instrumentation mode: automatic injection via Datadog admission webhook
  • Tracer version: reproduced with 2.46.0 and 2.49.0
  • OS: Alpine Linux
  • CLR: .Net 7.0

Additional context
Running in AWS EKS, with the latest Datadog agent/cluster deployed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant