Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sharding with a deployment with '--resources=pods' and '--node=""' does not fetch pending pods #2353

Open
carlosrmendes opened this issue Mar 27, 2024 · 10 comments · May be fixed by #2359 or #2388
Open

sharding with a deployment with '--resources=pods' and '--node=""' does not fetch pending pods #2353

carlosrmendes opened this issue Mar 27, 2024 · 10 comments · May be fixed by #2359 or #2388
Assignees
@CatherineF-dev
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@carlosrmendes
Copy link

What happened: a deployment with --resources=pods and --node="" does not fetch pending pods

What you expected to happen: pending pods be fetched by ksm

How to reproduce it (as minimally and precisely as possible):

  • create a ksm deploy with --resources=pods and --node="" and -v=10
  • force another pod to be in a pending state
  • check the logs for the api request like https://<kubernetes_api_ip>:443/api/v1/pods?allowWatchBookmarks=true&fieldSelector=spec.nodeName%3D%22%22&resourceVersion=3998681&timeout=5m23s&timeoutSeconds=323&watch=true
  • 'kubect proxy' and try to make the same request
  • make the same request without the &fieldSelector=spec.nodeName%3D%22%22 and confirm that on the response, the pending pods does not have any nodeSelectot key with an empty value
  • kube-state-metrics version: v2.10.1
  • Kubernetes version (use kubectl version): k3s v1.24.10
@carlosrmendes carlosrmendes added the kind/bug Categorizes issue or PR as related to a bug. label Mar 27, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Mar 27, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@CatherineF-dev
Copy link
Contributor

Thanks for reporting this.

Could you help paste your pending pod yaml, if it's available? Want to see nodeName value on the yaml

@CatherineF-dev
Copy link
Contributor

/assign @CatherineF-dev

@carlosrmendes
Copy link
Author

The pending pod doesn't even have the nodeName key, probably the key is only set after the pod is scheduled to a node.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-03-27T19:56:40Z"
  generateName: pending-test-
  labels:
    app.kubernetes.io/name: pending-test
    controller-revision-hash: pending-test-675ffcc546
    statefulset.kubernetes.io/pod-name: pending-test-0
  name: pending-test-0
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: pending-test
    uid: c5ed648f-afaa-44ef-9652-40b4633ca5b9
  resourceVersion: "3998134"
  uid: a04b2548-6977-4749-8375-96489b1aa42a
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: pending-test
    resources:
      requests:
        memory: 64Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-ndgdz
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: pending-test-0
  preemptionPolicy: PreemptLowerPriority
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: pending-test
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-ndgdz
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-03-27T19:56:40Z"
    message: '0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes
      are available: 1 Insufficient memory.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable

@CatherineF-dev
Copy link
Contributor

Wondering whether apiserver fieldSelector supports -nodeName which means nodeName doesn't exist

@CatherineF-dev
Copy link
Contributor

I tried again and verified --field-selector spec.nodeName= can get unscheduled pods.

--node="" is translated into spec.nodeName=

  1. nothing: get all pods
  2. --node="A": --field-selector spec.nodeName=A get pods on node A
  3. --node="": --field-selector spec.nodeName= get pods with no node assigned
kubectl get pods --field-selector spec.nodeName= -A

NAMESPACE   NAME                          READY   STATUS    RESTARTS   AGE
default     helloworld-7f5cd44b59-zhgzj   0/1     Pending   0          87s

kube_pod_status_scheduled{namespace="default",pod="helloworld-7f5cd44b59-zhgzj",uid="26018553-ceb6-427f-be1a-088032edac7a",condition="false"} 1

@carlosrmendes
Copy link
Author

with kubectl works, the Pending pods are returned, but making an http request to the kubernetes api (same way that ksm do) does not:

image

@carlosrmendes
Copy link
Author

carlosrmendes commented Apr 3, 2024 •
edited

I figured it out that the problem is the --node="" translation to url parameter fieldSelector=spec.nodeName%3D%22%22, without the encoded quotes works: fieldSelector=spec.nodeName%3D

@CatherineF-dev CatherineF-dev linked a pull request Apr 3, 2024 that will close this issue
Open
@CatherineF-dev
Copy link
Contributor

@carlosrmendes does it mean we need to fix this for KSM? Or just your local query doesn't work.

@CatherineF-dev
Copy link
Contributor

I can reproduce it that --node="" doesn't work as expected.

/usr/lib/go-1.22/bin/go  run main.go  --resources=pods --node=  --kubeconfig ~/.kube/config

curl localhost:8080/metrics | grep kube_pod_status

This was referenced May 9, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CatherineF-dev CatherineF-dev

Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Milestone
No milestone
3 participants
@carlosrmendes @k8s-ci-robot @CatherineF-dev