flux alerts

[TheKangaroo]

I would like to add some alerts for flux.
As discussed here, my alerts rely on a custom kube-state-metrics config, so I'm not sure if this is something that's helpful for others.
Basically, I rely on the kube-state-metrics config from their monitoring-example repo.

Perhaps it's possible to add a usage description to the alerts?
If so, let me know and I'll send you a PR :)

[jeff-french]

@TheKangaroo If the custom KSM config means the rules aren't a good fit here, would mind sharing them in a Gist (or anywhere else)? I'm about to write alerts for Flux in the week or so and would love to have a jump start!

[TheKangaroo]

Sure these are basically our alerts (helm template). We'll improve them over time but for now we started with this.
As I said earlier, they rely on the kube-state-metrics config in the flux monitoring-example repo.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: flux.rules
  labels:
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring
spec:
  groups:
  - name: flux.rules
    rules:
    - alert: FluxKustomizationFailing
      annotations:
        description: Flux Kustomization {{`{{`}} $labels.name {{`}}`}} in namespace {{`{{`}} $labels.namespace {{`}}`}} failed.
        runbook_url: {{ .Values.defaultRules.runbookUrl }}/{{ .Template.Name }}
        summary: Errors while reconcile Flux Kustomization(s)
      expr: gotk_resource_info{customresource_kind=~"Kustomization",
        ready="False"}
      for: 5m
      labels:
        severity: warning
{{- if .Values.defaultRules.additionalRuleLabels }}
{{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }}
{{- end }}
    - alert: FluxHelmReleaseFailing
      annotations:
        description: Flux HelmRelease {{`{{`}} $labels.name {{`}}`}} in namespace {{`{{`}} $labels.namespace {{`}}`}} failed.
        runbook_url: {{ .Values.defaultRules.runbookUrl }}/{{ .Template.Name }}
        summary: Errors while reconcile Flux HelmRelease(s)
      expr: gotk_resource_info{customresource_kind=~"HelmRelease",
        ready="False"}
      for: 5m
      labels:
        severity: warning
{{- if .Values.defaultRules.additionalRuleLabels }}
{{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }}
{{- end }}
    - alert: FluxSourceFailing
      annotations:
        description: Flux Source {{`{{`}} $labels.name {{`}}`}} in namespace {{`{{`}} $labels.namespace {{`}}`}} failed.
        runbook_url: {{ .Values.defaultRules.runbookUrl }}/{{ .Template.Name }}
        summary: Errors while reconcile Flux Source(s)
      expr: gotk_resource_info{customresource_kind=~"GitRepository|HelmRepository|Bucket|OCIRepository",
        ready="False"}
      for: 5m
      labels:
        severity: warning
{{- if .Values.defaultRules.additionalRuleLabels }}
{{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }}
{{- end }}
    - alert: FluxResourceSuspended
      annotations:
        description: Flux Resource {{`{{`}} $labels.name {{`}}`}} in namespace {{`{{`}} $labels.namespace {{`}}`}} suspended.
        runbook_url: {{ .Values.defaultRules.runbookUrl }}/{{ .Template.Name }}
        summary: Flux Resource(s) are suspended for an extended period of time.
      expr: gotk_resource_info{suspended="true"}
      for: 2h
      labels:
        severity: none
{{- if .Values.defaultRules.additionalRuleLabels }}
{{ toYaml .Values.defaultRules.additionalRuleLabels | indent 8 }}
{{- end }}

We send alerts for failing resources like GitRepo, Kustomization, HelmCharts and HelmReleases.
We added the "suspended" alert with a timeout of 2h in case someone is troubleshooting something and forgets to resume a flux resource after that.

[jeff-french]

Awesome! Thanks for sharing!