Kubernetes > 1.22

[tawalaya]

Problem

Hi, we are running a recent version of Kubernetes, and for some reason, scrapandre does not pick up any pods. I've looked into it a bit, and it might be related to the kube-sync crate. It is using a fairly old version of k8s-openapi that only supports Kubernetes 1.22. Looking into this a bit more, there may be an issue with the kube-sync in the near future as the k8s-openapi removed some of the used features in favor of the kube library (API clients) see here.

Solution

Thus, it might be better to migrate away from the kube-sync crate to the kube crate to stay up to date with newer Kubernetes releases.

Alternatives

Update the kube-sync to work with newer versions of Kubernetes.

[mmadoo]

I have a cluster on k8s 1.26.8 with scaphandre docker image with tag dev and it works well.

[tawalaya]

We are running v1.29. It could also be a different reason why the pods don't show up. However, the deprecation of the kube-openapi API will be an issue sooner or later.

[tawalaya]

The error i see in the logs is this btw:

scaphandre::exporters: watching kubernetes...
isahc::client: send; method=GET uri=https://10.96.0.1:443/api/v1/namespaces//pods?
isahc::handler: handler;
isahc::handler: handler; id=0
isahc::handler:   Trying 10.96.0.1:443...
isahc::handler: Connected to 10.96.0.1 (10.96.0.1) port 443 (#0)
isahc::handler: ALPN: offers h2,http/1.1
isahc::handler: TLSv1.3 (OUT), TLS handshake, Client hello (1):
isahc::handler: TLSv1.3 (IN), TLS handshake, Server hello (2):
isahc::handler: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
isahc::handler: TLSv1.3 (IN), TLS handshake, Request CERT (13):
isahc::handler: TLSv1.3 (IN), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (IN), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (IN), TLS handshake, Finished (20):
isahc::handler: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Finished (20):
isahc::handler: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
isahc::handler: ALPN: server accepted h2
isahc::handler: Server certificate:
isahc::handler:  subject: CN=kube-apiserver
isahc::handler:  start date: Jan  5 14:52:55 2024 GMT
isahc::handler:  expire date: Jan  4 14:57:56 2025 GMT
isahc::handler:  subjectAltName: host "10.96.0.1" matched cert's IP address!
isahc::handler:  issuer: CN=kubernetes
isahc::handler:  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
isahc::handler: using HTTP/2
isahc::handler: h2h3 [:method: GET]
isahc::handler: h2h3 [:path: /api/v1/namespaces//pods?]
isahc::handler: h2h3 [:scheme: https]
isahc::handler: h2h3 [:authority: 10.96.0.1]
isahc::handler: h2h3 [accept: */*]
isahc::handler: h2h3 [accept-encoding: deflate, gzip]
isahc::handler: h2h3 [authorization: Bearer OMITTED]
isahc::handler: h2h3 [user-agent: curl/8.0.1-DEV isahc/1.7.2]
isahc::handler: h2h3 [content-length: 0]
isahc::handler: Using Stream ID: 1 (easy handle 0x563c2768d6d0)
isahc::handler: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
isahc::handler: Connection #0 to host 10.96.0.1 left intact
scaphandre::exporters: Failed getting pods list, despite client seems ok. Couldn't build http client (isahc)
scaphandre::exporters: First check done on pods.