Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prometheus-kube-stack] Impossible to disable some alerts from PrometheusRules group #4539

Closed
ellipsis-me opened this issue May 14, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@ellipsis-me
Copy link

ellipsis-me commented May 14, 2024

Describe the bug a clear and concise description of what the bug is.

Hi team, how could I disable a this exlusive alert etcdHighNumberOfFailedGRPCRequests from etcd group?
I tried to satisfy the condition, but It isn't working, the condition on template is {{- if not (.Values.defaultRules.disabled.etcdHighNumberOfFailedGRPCRequests | default false) }} and I tried to disable in values like:

defaultRules:
  disabled:
    etcdHighNumberOfFailedGRPCRequests: true

And when I run helm template again, the alert continue beeing generated inside the group.

What's your helm version?

version.BuildInfo{Version:"v3.14.2", GitCommit:"c309b6f0ff63856811846ce18f3bdc93d2b4d54b", GitTreeState:"clean", GoVersion:"go1.21.7"}

What's your kubectl version?

Client Version: v1.28.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.24.13+rke2r1

Which chart?

kube-prometheus-stack

What's the chart version?

58.5.2

What happened?

The condition inside the group to disable some alerts isn't working.

What you expected to happen?

Disable some alerts from a PrometheusRules group.

How to reproduce it?

Thying to disable other alerts based on the condition inside the PrometheusRules like:

defaultRules:
  disabled:
    etcdHighNumberOfFailedGRPCRequests: true
    otherAlert: true

Enter the changed values of values.yaml?

defaultRules:
disabled:
etcdHighNumberOfFailedGRPCRequests: true

Enter the command that you execute and failing/misfunctioning.

helm template -f values.yaml --debug .

Anything else we need to know?

An diff between my disabled (true or false) generates the same helm templates with no changes:
2024-05-15_16-16

That's a really good function to be fixed, because currently We need to disable the entire PrometheusRules group and outside the installation intall an edited PrometheusRule without the alert we don't want.

@ellipsis-me ellipsis-me added the bug Something isn't working label May 14, 2024
@machine424
Copy link
Contributor

machine424 commented May 16, 2024

I think the disabled dict you added at the beginning of the default values.yaml was overridden by the empty one at the end:

## Disabled PrometheusRule alerts
disabled: {}
# KubeAPIDown: true
# NodeRAIDDegraded: true

It's better to keep the default values.yaml as it is, provide your own values file and leave Helm take care of the merging (in most cases it works well).

@ellipsis-me
Copy link
Author

You are right, thanks, using this disabled is working now! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants