x509: certificate signed by unknown authority error

[laith-leo]

Describe the bug
I'm getting the error (in the prometheus-msteams pod logs) bellow when the alert condition is met.

`
2022/08/21 03:45:10 [ERR] POST https://mycompany.webhook.office.com/webhookb2/XXXXXXX request failed: Post "https://mycompany.webhook.office.com/webhookb2/XXXXXXX@XXXXXXX-XXXXXXX/IncomingWebhook/XXXXXXXe": x509: certificate signed by unknown authority

{"caller":"transport.go:133","err":"http client failed: Post "https://mycompany.webhook.office.com/webhookb2/XXXXXXX\": POST https://mycompany.webhook.office.com/webhookb2/XXXXXXX giving up after 1 attempt(s): Post "https://mycompany.webhook.office.com/webhookb2/XXXXXX\": x509: certificate signed by unknown authority","ts":"2022-08-21T03:08:16.90243817Z"} {"caller":"transport.go:66","host":"prometheus-msteams.mon.svc.cluster.local:2000","method":"POST","status":500,"took":"1.087262237s","ts":"2022-08-21T03:08:16.90243817Z","uri":"/alertmanager-mywork"}`
XXXXXX is to replace the original webhook

Version
prometheus-msteams1.5.1
K8s version: v1.21.13-gke.900
Helm version: v3.9.3

To Reproduce
Helming prometheus-msteams with kube-prometheus stack#the latest as of today

Additional context:
Tests in another environments are working just fine and can see the alerts in MS Teams using the same charts and same k8S server version.

[kbudde]

hi @laith-leo,
just to be sure: you replaced the URL in the logs, correct?

The error message is quite clear. The server you are talking to has an unknown certificate. If it would be office.com, you would see issues from all your installations.
But as it's just one instance, it's most likely an issue there.

Please recheck the URL to your webhook, maybe there's a type somewhere in it.
Maybe you have a proxy for outgoing traffic from your kubernetes instance or some service mesh magic?

I would test it with another pod with openssl:
openssl s_client -connect mycompany.webhook.office.com:443
shows me a valid certificate chain:

Certificate chain
 0 s:C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = *.webhook.office.com
   i:C = US, O = DigiCert Inc, CN = DigiCert Cloud Services CA-1
 1 s:C = US, O = DigiCert Inc, CN = DigiCert Cloud Services CA-1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA