Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Certificate Request failure due to missing secret in cert-manager #1254

Open
vcanuel opened this issue Mar 23, 2024 · 2 comments
Open

Certificate Request failure due to missing secret in cert-manager #1254

vcanuel opened this issue Mar 23, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@vcanuel
Copy link

vcanuel commented Mar 23, 2024

Hi,

We have observed that there are certificate requests failing for the past 12 days due to a missing secret.

Below is the error message from the event log: Failed to create CertificateRequest: certificaterequests.cert-manager.io "k8ssandra-operator-serving-cert-1" already exists.

NAME                                              APPROVED   DENIED   READY   ISSUER                                               REQUESTOR                                         AGE
k8ssandra-operator-cass-operator-serving-cert-1   True                False   k8ssandra-operator-cass-operator-selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager   70d
k8ssandra-operator-serving-cert-1                 True                False   k8ssandra-operator-selfsigned-issuer                 system:serviceaccount:cert-manager:cert-manager   70d

The specific error for k8ssandra-operator-serving-cert-1 is as follows: Referenced secret k8ssandra-operator/k8ssandra-operator-serving-cert-gj7jj not found: secrets "k8ssandra-operator-serving-cert-gj7jj" not found.

Interestingly, there is a k8ssandra-operator-serving-cert-jnrk4 (created 12 days ago, coinciding with the start of the error).

Do you have any suggestions on how to resolve this issue?

Thank you.

  • Helm charts version info
    cert-manager cert-manager 2 2024-01-13 12:50:00.362899 +0100 CET deployed cert-manager-v1.13.3 v1.13.3
    k8ssandra-operator k8ssandra-operator 3 2024-03-23 18:19:49.737575 +0100 CET deployed k8ssandra-operator-1.13.0 1.13.0
    prometheus-grafana k8ssandra-operator 1 2024-01-16 12:13:12.095415 +0100 CET deployed kube-prometheus-stack-55.8.2 v0.70.0

  • Helm charts user-supplied values
    N.A

  • Kubernetes version information:
    Client Version: v1.29.0
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.27.8-gke.1067004

  • Kubernetes cluster kind:
    K8s on GKE
@vcanuel vcanuel added the bug Something isn't working label Mar 23, 2024
@vcanuel vcanuel mentioned this issue Mar 23, 2024
Closed
@burmanm
Copy link
Contributor

burmanm commented Mar 25, 2024 •
edited

Are there some additional tools or stuff installed to the cluster? I'm not able to even replicate the naming of the Secret objects:

➜  cass-operator git:(master) ✗ kubectl get certificate
NAME                                        READY   SECRET                                             AGE
k8ssandra-cass-operator-serving-cert        True    k8ssandra-cass-operator-webhook-server-cert        44s
k8ssandra-k8ssandra-operator-serving-cert   True    k8ssandra-k8ssandra-operator-webhook-server-cert   44s
➜  cass-operator git:(master) ✗

➜  cass-operator git:(master) ✗ kubectl get certificaterequest
NAME                                          APPROVED   DENIED   READY   ISSUER                                           REQUESTOR                                         AGE
k8ssandra-cass-operator-serving-cert-1        True                True    k8ssandra-cass-operator-selfsigned-issuer        system:serviceaccount:cert-manager:cert-manager   4m7s
k8ssandra-k8ssandra-operator-serving-cert-1   True                True    k8ssandra-k8ssandra-operator-selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager   4m7s
➜  cass-operator git:(master) ✗

➜  cass-operator git:(master) ✗ kubectl get secret     
NAME                                               TYPE                                  DATA   AGE
k8ssandra-cass-operator-webhook-server-cert        kubernetes.io/tls                     3      48s
k8ssandra-k8ssandra-operator-token                 kubernetes.io/service-account-token   3      48s
k8ssandra-k8ssandra-operator-webhook-server-cert   kubernetes.io/tls                     3      48s
sh.helm.release.v1.k8ssandra.v1                    helm.sh/release.v1                    1      48s
➜  cass-operator git:(master) ✗

➜  cass-operator git:(master) ✗ helm list -A
NAME            NAMESPACE               REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
k8ssandra       k8ssandra-operator      1               2024-03-25 17:27:55.707636 +0200 EET    deployed        k8ssandra-operator-1.13.0       1.13.0     
➜  cass-operator git:(master) ✗

@vcanuel
Copy link
Author

vcanuel commented Mar 25, 2024

This cluster is purely dedicated to K8ssandra.
I have installed a prometheus operator in the same namespace as recommended here https://docs.k8ssandra.io/tasks/monitor/prometheus-grafana/ . Everything was going fine since many months.

kubectl get secret -n k8ssandra-operator | grep k8ssandra                                                                                                    in zsh at 16:38:15
k8ssandra-operator-cass-operator-serving-cert-xrp2v                 Opaque                                1      14d
k8ssandra-operator-cass-operator-webhook-server-cert                kubernetes.io/tls                     3      72d
k8ssandra-operator-serving-cert-jnrk4                               Opaque                                1      14d
k8ssandra-operator-token                                            kubernetes.io/service-account-token   3      57d
k8ssandra-operator-webhook-server-cert                              kubernetes.io/tls                     3      72d
k8ssandra-reaper-ui                                                 Opaque                                2      72d
prod-k8ssandra-medusa-key                                           Opaque                                1      72d
sh.helm.release.v1.k8ssandra-operator.v1                            helm.sh/release.v1                    1      72d
sh.helm.release.v1.k8ssandra-operator.v2                            helm.sh/release.v1                    1      57d
sh.helm.release.v1.k8ssandra-operator.v3                            helm.sh/release.v1                    1      46h
kubectl get certificaterequest -n k8ssandra-operator                                                                                                         in zsh at 16:39:58
NAME                                              APPROVED   DENIED   READY   ISSUER                                               REQUESTOR                                         AGE
k8ssandra-operator-cass-operator-serving-cert-1   True                False   k8ssandra-operator-cass-operator-selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager   72d
k8ssandra-operator-serving-cert-1                 True                False   k8ssandra-operator-selfsigned-issuer                 system:serviceaccount:cert-manager:cert-manager   72d
 kubectl get certificate -n k8ssandra-operator                                                                                                                in zsh at 16:40:01
NAME                                            READY   SECRET                                                 AGE
k8ssandra-operator-cass-operator-serving-cert   True    k8ssandra-operator-cass-operator-webhook-server-cert   72d
k8ssandra-operator-serving-cert                 True    k8ssandra-operator-webhook-server-cert                 72d
 helm list -A                                                                                                                                                 in zsh at 16:40:51
NAME                    NAMESPACE               REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
cert-manager            cert-manager            2               2024-01-13 12:50:00.362899 +0100 CET    deployed        cert-manager-v1.13.3            v1.13.3    
k8ssandra-operator      k8ssandra-operator      3               2024-03-23 18:19:49.737575 +0100 CET    deployed        k8ssandra-operator-1.13.0       1.13.0     
prometheus-grafana      k8ssandra-operator      1               2024-01-16 12:13:12.095415 +0100 CET    deployed        kube-prometheus-stack-55.8.2    v0.70.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
K8ssandra
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vcanuel @burmanm