Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Invalid memory address on Kubernetes v1.24 #336

Open
mattups opened this issue Jul 14, 2023 · 3 comments
Open

[BUG] Invalid memory address on Kubernetes v1.24 #336

mattups opened this issue Jul 14, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@mattups
Copy link

mattups commented Jul 14, 2023

Describe the bug
Deployed a fresh new install with Helm, chart v1.3.4 attempting to migrate from a v1.20 Kubernetes cluster.

The deployment is successful and pods are running but log is filled with these messages:

{"caller":"transport.go:66","host":"172.22.221.200:30511","method":"POST","status":0,"took":"250.26µs","ts":"2023-07-14T13:31:34.772640823Z","uri":"/mes"}
--
Fri, Jul 14 2023 3:31:34 pm | 2023/07/14 13:31:34 http: panic serving 10.42.1.0:42280: runtime error: invalid memory address or nil pointer dereference
Fri, Jul 14 2023 3:31:34 pm | goroutine 260 [running]:
Fri, Jul 14 2023 3:31:34 pm | net/http.(*conn).serve.func1()
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/net/http/server.go:1854 +0xbf
Fri, Jul 14 2023 3:31:34 pm | panic({0xce5d20, 0x1712b30})
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/runtime/panic.go:890 +0x263
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus/alertmanager/template.(*Template).ExecuteTextString(0xc000a3b0e0?, {0xde4eb6, 0x1d}, {0xc87400, 0xc000a3b140})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/go/pkg/mod/github.com/prometheus/[email protected]/template/template.go:101 +0x53
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/card.(*templatedCard).executeTemplate(0xc0004cf1d0, {0xc000a3b0e0, {0x16d3c00, 0x1}, {0xc00098c200, 0x3f}, 0x0})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/card/templated_card.go:69 +0x214
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/card.(*templatedCard).Convert(0xc0009f6e10?, {0x114cb90?, 0xc000978f00?}, {0xc000a3b0e0, {0x16d3c00, 0x1}, {0xc00098c200, 0x3f}, 0x0})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/card/templated_card.go:34 +0x14c
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/card.loggingMiddleware.Convert({{0x1146080?, 0xc0002066f0?}, {0x1146340?, _}}, {_, _}, {0xc000a3b0e0, {0x16d3c00, 0x1}, {0xc00098c200, ...}, ...})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/card/converter.go:92 +0x24c
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/service.simpleService.Post({{0x11476c0, 0xc000224e00}, 0xc000206780, {0xc00001e410, 0xca}}, {0x114cb90?, 0xc000978d80?}, {0xc000a3b0e0, {0x16d3c00, 0x1}, ...})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/service/service.go:45 +0x188
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/service.loggingService.Post({{0x1146080?, 0xc000206210?}, {0x1147700?, 0xc0002067e0?}}, {0x114cb90?, 0xc000978d80?}, {0xc000a3b0e0, {0x16d3c00, 0x1}, {0xc00098c200, ...}, ...})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/service/logging.go:33 +0x143
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/transport.handleRoute({0x115afe8, 0xc00024e500}, {0x11476e0, 0xc000224e60}, {0x1146080, 0xc000206210})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/transport/transport.go:131 +0x3d0
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/transport.addRoute.func1({0x115afe8?, 0xc00024e500?})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/transport/transport.go:81 +0x33
Fri, Jul 14 2023 3:31:34 pm | github.com/labstack/echo/v4.WrapMiddleware.func1.1.1({0x114c8d0?, 0xc000978d20}, 0x40dfe7?)
Fri, Jul 14 2023 3:31:34 pm | /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:932 +0xe7
Fri, Jul 14 2023 3:31:34 pm | net/http.HandlerFunc.ServeHTTP(0x100?, {0x114c8d0?, 0xc000978d20?}, 0x0?)
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/net/http/server.go:2122 +0x2f
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/transport.opencensusMiddleware.func1.1({0x114c8d0?, 0xc000978d20?}, 0x176b101?)
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/transport/transport.go:54 +0x38
Fri, Jul 14 2023 3:31:34 pm | net/http.HandlerFunc.ServeHTTP(0x114cb90?, {0x114c8d0?, 0xc000978d20?}, 0x176b178?)
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/net/http/server.go:2122 +0x2f
Fri, Jul 14 2023 3:31:34 pm | go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP(0xc0003e1540, {0x114b220, 0xc000a3b020}, 0x40dc08?)
Fri, Jul 14 2023 3:31:34 pm | /home/runner/go/pkg/mod/[email protected]/plugin/ochttp/server.go:92 +0x264
Fri, Jul 14 2023 3:31:34 pm | github.com/labstack/echo/v4.WrapMiddleware.func1.1({0x115afe8?, 0xc00024e500})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:933 +0x124
Fri, Jul 14 2023 3:31:34 pm | github.com/prometheus-msteams/prometheus-msteams/pkg/transport.kitLoggerMiddleware.func1.1({0x115afe8, 0xc00024e500})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/work/prometheus-msteams/prometheus-msteams/pkg/transport/transport.go:74 +0xfe
Fri, Jul 14 2023 3:31:34 pm | github.com/labstack/echo/v4.(*Echo).add.func1({0x115afe8, 0xc00024e500})
Fri, Jul 14 2023 3:31:34 pm | /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:575 +0x51
Fri, Jul 14 2023 3:31:34 pm | github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc00050efc0, {0x114bd30?, 0xc000525a40}, 0xc0008c0800)
Fri, Jul 14 2023 3:31:34 pm | /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:662 +0x3d1
Fri, Jul 14 2023 3:31:34 pm | net/http.serverHandler.ServeHTTP({0x1149538?}, {0x114bd30, 0xc000525a40}, 0xc0008c0800)
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/net/http/server.go:2936 +0x316
Fri, Jul 14 2023 3:31:34 pm | net/http.(*conn).serve(0xc0009423f0, {0x114cb90, 0xc000598240})
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/net/http/server.go:1995 +0x612
Fri, Jul 14 2023 3:31:34 pm | created by net/http.(*Server).Serve
Fri, Jul 14 2023 3:31:34 pm | /opt/hostedtoolcache/go/1.20.2/x64/src/net/http/server.go:3089 +0x5ed
Fri, Jul 14 2023 3:31:35 pm | {"alert":{"receiver":"mes-msteams","status":"resolved","alerts":[{"status":"resolved","labels":{"alertname":"MESDBServerCpuUsage","instance":"redacted_hostname","monitor":"example","service":"mes","severity":"warning"},"annotations":{"description":"CPU Usage is more than 80%\\n VALUE = 80.34616302694333\\n LABELS = map[instance:<REDACTED>]","summary":"<REDACTED> CPU usage (instance <REDACTED>)"},"startsAt":"2023-07-14T02:33:01.276Z","endsAt":"2023-07-14T03:01:46.276Z","generatorURL":"http://<REDACTED>/graph?g0.expr=100+-+%28avg+by%28instance%29+%28rate%28windows_cpu_time_total%7Bjob%3D%22mes-db-prod-node%22%2Cmode%3D%22idle%22%7D%5B2m%5D%29%29+%2A+100%29+%3E+80&g0.tab=1","fingerprint":"8730831f4fed40e4"},{"status":"resolved","labels":{"alertname":"MESDBServerCpuUsage","instance":"<REDACTED>","monitor":"example","service":"mes","severity":"warning"},"annotations":{"description":"CPU Usage is more than 80%\\n VALUE = 80.34635020641699\\n LABELS = map[instance:<REDACTED>]","summary":"<REDACTED> CPU usage (instance <REDACTED>)"},"startsAt":"2023-07-14T12:27:31.276Z","endsAt":"2023-07-14T12:34:16.276Z","generatorURL":"http://<REDACTED>/graph?g0.expr=100+-+%28avg+by%28instance%29+%28rate%28windows_cpu_time_total%7Bjob%3D%22mes-db-prod-node%22%2Cmode%3D%22idle%22%7D%5B2m%5D%29%29+%2A+100%29+%3E+80&g0.tab=1","fingerprint":"679fb0fadb6cd018"}],"groupLabels":{"alertname":"MESDBServerCpuUsage"},"commonLabels":{"alertname":"MESDBServerCpuUsage","monitor":"example","service":"mes","severity":"warning"},"commonAnnotations":{},"externalURL":"http://<REDACTED>:9093","version":"4","groupKey":"{}/{service=~\"^(?:mes)$\"}:{alertname=\"MESDBServerCpuUsage\"}","truncatedAlerts":0},"caller":"converter.go:86","card":{"@context":"","@type":"","title":"","text":"","summary":"","themeColor":""},"escaped_underscores":true,"template_file":"","took":"31.301µs","ts":"2023-07-14T13:31:35.2
  | 67584704Z"}
Fri, Jul 14 2023 3:31:35 pm | {"caller":"transport.go:66","host":"172.22.221.200:30511","method":"POST","status":0,"took":"286.659µs","ts":"2023-07-14T13:31:35.267699865Z","uri":"/mes"}
Fri, Jul 14 2023 3:31:35 pm | 2023/07/14 13:31:35 http: panic serving 10.42.1.0:39551: runtime error: invalid memory address or nil pointer dereferenc

Also, when calling a receiver with cURL I do have an empty response message:

❯ curl -X POST -d @prom.json http://prometheus-msteams/connector
curl: (52) Empty reply from server

Version
Helm Chart v1.3.4, image is quay.io/prometheusmsteams/prometheus-msteams:v1.5.2

To Reproduce
Install the chart via Rancher 2.7.1 on an RKE1 cluster, Kubernetes version 1.24

Expected behavior
Running without nil pointer reference errors.

Additional context
Same deployment with same values file works fine on RKE1 Kubernetes v1.20 on Rancher 2.5

@mattups mattups added the bug Something isn't working label Jul 14, 2023
@Jnewbon
Copy link

Jnewbon commented Aug 4, 2023

Hi,

I found out that this error is produced when the executables working directory does not contain the card template.

Check the working directory contains the provided template file and is readable by the user running the service

@mattups
Copy link
Author

mattups commented Oct 19, 2023

Circling back on this...
So apparently there is not shell to exec into the container, so I cannot verify that.
I'm also passing now the card template via values file:

customCardTempalate: |
  {{ define "teams.card" }}
  {
    "@type": "MessageCard",
    "@context": "http://schema.org/extensions",
    "themeColor": "{{- if eq .Status "resolved" -}}2DC72D
                  {{- else if eq .Status "firing" -}}
                      {{- if eq .CommonLabels.severity "critical" -}}8C1A1A
                      {{- else if eq .CommonLabels.severity "warning" -}}FFA500
                      {{- else -}}808080{{- end -}}
                  {{- else -}}808080{{- end -}}",
    "summary": "{{ .CommonAnnotations.summary }}",
    "title": "Prometheus Alert ({{ .Status }})",
    "sections": [ {{$externalUrl := .ExternalURL}}
    {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}
      {
        "activityTitle": "[{{ $alert.Annotations.description }}]({{ $externalUrl }})", 
        "facts": [
          {{- range $key, $value := $alert.Annotations }}
          {
            "name": "{{ reReplaceAll "_" "\\\\_" $key }}",
            "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
          },
          {{- end -}}
          {{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}
          {
            "name": "{{ reReplaceAll "_" "\\\\_" $key }}",
            "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
          }
          {{- end }}
        ],
        "markdown": true
      }
      {{- end }}
    ]
  }
  {{ end }}

But the error remains. Am I missing something from the docs maybe?

@mattups
Copy link
Author

mattups commented Oct 19, 2023

Ok it's solved.
In the end the problem was that the charts was deployed via Rancher (not Fleet).
The volume mounts were ok, ConfigMaps as well and so on but in the end it didn't work.

Solved by using Helm CLI and the default installation command.

For context, this was happening with Rancher v2.7.1 as well with v2.7.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants