Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NNCP apply takes *very long* time to apply; nmstatectl does not #1242

Open
k8scoder192 opened this issue May 28, 2024 · 2 comments
Open

NNCP apply takes *very long* time to apply; nmstatectl does not #1242

k8scoder192 opened this issue May 28, 2024 · 2 comments

Comments

@k8scoder192
Copy link

k8scoder192 commented May 28, 2024 •
edited
Loading

What happened:
Apply of a nncp takes over 60 seconds. If I exec into the node's nmstate-handler pod and use nmstatectl to create an interface (vlan or bridge) it takes seconds.

What you expected to happen:
Apply of nncp to apply quickly

How to reproduce it (as minimally and precisely as possible):
Apply a yaml such as this

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: svc-uc-v108-dal4-qz4-sr7-rk047-s04
spec:
  desiredState:
    interfaces:
    - ipv4:
        dhcp: false
        enabled: false
      mac-address: 02:10:2c:2c:7e:52
      name: svc-uc-v108
      state: up
      type: vlan
      vlan:
        base-iface: bond1
        id: 108
  nodeSelector:
    kubernetes.io/hostname: dal4-qz4-sr7-rk047-s04

Anything else we need to know?:
nmstate version --> kubernetes-nmstate-handler:v0.80.0

Environment:

  • NodeNetworkState on affected nodes (use kubectl get nodenetworkstate <node_name> -o yaml):
apiVersion: nmstate.io/v1beta1
kind: NodeNetworkState
metadata:
  creationTimestamp: "2023-10-20T21:26:46Z"
  generation: 1
  name: dal4-qz4-sr7-rk048-s47
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: dal4-qz4-sr7-rk048-s47
    uid: 0e2aece7-b94c-4d1e-94d8-cfa8d55c08a3
  resourceVersion: "428891868"
  uid: cdd70d0c-0b6d-482c-a8e0-25c017e4e0c7
status:
  currentState:
    dns-resolver:
      config:
        search: null
        server: null
      running:
        search: []
        server: []
    interfaces:
    - accept-all-mac-addresses: false
      ethtool:
        feature:
          esp-hw-offload: false
          esp-tx-csum-hw-offload: false
          highdma: true
          rx-gro: true
          rx-gro-list: false
          rx-lro: false
          rx-udp-gro-forwarding: false
          rx-vlan-filter: true
          rx-vlan-hw-parse: true
          tx-checksum-ip-generic: true
          tx-esp-segmentation: false
          tx-generic-segmentation: true
          tx-gre-csum-segmentation: true
          tx-gre-segmentation: true
          tx-gso-list: false
          tx-ipxip4-segmentation: true
          tx-ipxip6-segmentation: true
          tx-nocache-copy: false
          tx-scatter-gather-fraglist: false
          tx-sctp-segmentation: false
          tx-tcp-ecn-segmentation: true
          tx-tcp-mangleid-segmentation: true
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
          tx-udp_tnl-csum-segmentation: true
          tx-udp_tnl-segmentation: true
      ipv4:
        address:
        - ip: 198.19.0.17
          prefix-length: 22
        enabled: true
      ipv6:
        address:
        - ip: fe80::849c:86ff:feed:378b
          prefix-length: 64
        enabled: true
      link-aggregation:
        mode: 802.3ad
        options:
          ad_actor_sys_prio: 65535
          ad_actor_system: "00:00:00:00:00:00"
          ad_select: stable
          ad_user_port_key: 0
          all_slaves_active: dropped
          arp_all_targets: any
          arp_interval: 0
          arp_validate: none
          downdelay: 0
          lacp_rate: fast
          lp_interval: 1
          miimon: 1000
          min_links: 0
          primary_reselect: always
          updelay: 0
          use_carrier: true
          xmit_hash_policy: layer3+4
        port:
        - enp175s0f0np0
        - enp175s0f1np1
      mac-address: 86:9C:86:ED:37:8B
      max-mtu: 65535
      min-mtu: 68
      mptcp:
        address-flags: []
      mtu: 1500
      name: bond1
      state: ignore
      type: bond
    - accept-all-mac-addresses: false
      ethtool:
        feature:
          rx-checksum: true
          rx-gro: true
          rx-gro-list: false
          rx-udp-gro-forwarding: false
          tx-checksum-ip-generic: true
          tx-generic-segmentation: true
          tx-gso-list: true
          tx-nocache-copy: false
          tx-scatter-gather-fraglist: true
          tx-sctp-segmentation: true
          tx-tcp-ecn-segmentation: true
          tx-tcp-mangleid-segmentation: true
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
      ipv4:
        enabled: false
      ipv6:
        address:
        - ip: fe80::2c9c:bff:fe94:9164
          prefix-length: 64
        enabled: true
      mac-address: 2E:9C:0B:94:91:64
      max-mtu: 65535
      min-mtu: 68
      mtu: 1500
      name: cilium_vxlan
      state: ignore
      type: vxlan
      vxlan:
        destination-port: 8472
        id: 0
        learning: false
    - accept-all-mac-addresses: false
      bridge:
        options:
          group-addr: 01:80:C2:00:00:00
          group-forward-mask: 0
          group-fwd-mask: 0
          hash-max: 4096
          mac-ageing-time: 300
          multicast-last-member-count: 2
          multicast-last-member-interval: 100
          multicast-membership-interval: 26000
          multicast-querier: false
          multicast-querier-interval: 25500
          multicast-query-interval: 12500
          multicast-query-response-interval: 1000
          multicast-query-use-ifaddr: false
          multicast-router: auto
          multicast-snooping: true
          multicast-startup-query-count: 2
          multicast-startup-query-interval: 3124
          stp:
            enabled: false
            forward-delay: 15
            hello-time: 2
            max-age: 20
            priority: 32768
          vlan-default-pvid: 1
          vlan-protocol: 802.1q
        port: []
      ethtool:
        feature:
          highdma: true
          rx-gro: true
          rx-gro-list: false
          rx-udp-gro-forwarding: false
          tx-checksum-ip-generic: true
          tx-esp-segmentation: true
          tx-fcoe-segmentation: true
          tx-generic-segmentation: true
          tx-gre-csum-segmentation: true
          tx-gre-segmentation: true
          tx-gso-list: true
          tx-gso-partial: true
          tx-gso-robust: true
          tx-ipxip4-segmentation: true
          tx-ipxip6-segmentation: true
          tx-nocache-copy: false
          tx-scatter-gather-fraglist: true
          tx-sctp-segmentation: true
          tx-tcp-ecn-segmentation: true
          tx-tcp-mangleid-segmentation: true
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-tunnel-remcsum-segmentation: true
          tx-udp-segmentation: true
          tx-udp_tnl-csum-segmentation: true
          tx-udp_tnl-segmentation: true
          tx-vlan-hw-insert: true
          tx-vlan-stag-hw-insert: true
      ipv4:
        address:
        - ip: 172.17.0.1
          prefix-length: 16
        enabled: true
      ipv6:
        enabled: false
      mac-address: 02:42:F3:B6:47:0A
      max-mtu: 65535
      min-mtu: 68
      mptcp:
        address-flags: []
      mtu: 1500
      name: docker0
      state: ignore
      type: linux-bridge
    - accept-all-mac-addresses: false
      ethernet:
        auto-negotiation: true
        duplex: full
        speed: 10000
        sr-iov:
          total-vfs: 0
          vfs: []
      ethtool:
        coalesce:
          rx-usecs: 1
          tx-usecs: 0
        feature:
          esp-hw-offload: true
          esp-tx-csum-hw-offload: true
          hw-tc-offload: false
          l2-fwd-offload: false
          rx-all: false
          rx-checksum: true
          rx-gro: true
          rx-gro-list: false
          rx-hashing: true
          rx-lro: false
          rx-ntuple-filter: false
          rx-udp-gro-forwarding: false
          rx-vlan-filter: true
          rx-vlan-hw-parse: true
          tx-checksum-ip-generic: true
          tx-checksum-sctp: true
          tx-esp-segmentation: true
          tx-generic-segmentation: true
          tx-gre-csum-segmentation: true
          tx-gre-segmentation: true
          tx-gso-partial: true
          tx-ipxip4-segmentation: true
          tx-ipxip6-segmentation: true
          tx-nocache-copy: false
          tx-tcp-mangleid-segmentation: false
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
          tx-udp_tnl-csum-segmentation: true
          tx-udp_tnl-segmentation: true
          tx-vlan-hw-insert: true
        pause:
          autoneg: true
          rx: true
          tx: true
        ring:
          rx: 512
          rx-max: 4096
          tx: 512
          tx-max: 4096
      ipv4:
        address:
        - ip: 198.18.0.17
          prefix-length: 25
        enabled: true
      ipv6:
        address:
        - ip: fe80::3eec:efff:fe60:532e
          prefix-length: 64
        enabled: true
      mac-address: 3C:EC:EF:60:53:2E
      max-mtu: 9710
      min-mtu: 68
      mptcp:
        address-flags: []
      mtu: 1500
      name: eno1
      state: ignore
      type: ethernet
    - accept-all-mac-addresses: false
      ethernet:
        auto-negotiation: true
        sr-iov:
          total-vfs: 0
          vfs: []
      ethtool:
        coalesce:
          rx-usecs: 1
          tx-usecs: 0
        feature:
          esp-hw-offload: true
          esp-tx-csum-hw-offload: true
          hw-tc-offload: false
          l2-fwd-offload: false
          rx-all: false
          rx-checksum: true
          rx-gro: true
          rx-gro-list: false
          rx-hashing: true
          rx-lro: false
          rx-ntuple-filter: false
          rx-udp-gro-forwarding: false
          rx-vlan-filter: true
          rx-vlan-hw-parse: true
          tx-checksum-ip-generic: true
          tx-checksum-sctp: true
          tx-esp-segmentation: true
          tx-generic-segmentation: true
          tx-gre-csum-segmentation: true
          tx-gre-segmentation: true
          tx-gso-partial: true
          tx-ipxip4-segmentation: true
          tx-ipxip6-segmentation: true
          tx-nocache-copy: false
          tx-tcp-mangleid-segmentation: false
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
          tx-udp_tnl-csum-segmentation: true
          tx-udp_tnl-segmentation: true
          tx-vlan-hw-insert: true
        pause:
          autoneg: true
          rx: true
          tx: true
        ring:
          rx: 512
          rx-max: 4096
          tx: 512
          tx-max: 4096
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 3C:EC:EF:60:53:2F
      max-mtu: 9710
      min-mtu: 68
      mtu: 1500
      name: eno2
      state: ignore
      type: ethernet
    - accept-all-mac-addresses: false
      controller: bond1
      ethernet:
        auto-negotiation: true
        duplex: full
        speed: 100000
        sr-iov:
          total-vfs: 0
          vfs: []
      ethtool:
        coalesce:
          adaptive-rx: true
          adaptive-tx: true
          rx-frames: 128
          rx-usecs: 8
          tx-frames: 128
          tx-usecs: 8
        feature:
          hw-tc-offload: true
          rx-all: false
          rx-checksum: true
          rx-fcs: false
          rx-gro: true
          rx-gro-list: false
          rx-hashing: true
          rx-lro: false
          rx-ntuple-filter: false
          rx-udp-gro-forwarding: false
          rx-udp_tunnel-port-offload: true
          rx-vlan-filter: true
          rx-vlan-hw-parse: true
          tx-checksum-ip-generic: true
          tx-generic-segmentation: true
          tx-gre-csum-segmentation: true
          tx-gre-segmentation: true
          tx-gso-partial: true
          tx-ipxip4-segmentation: true
          tx-ipxip6-segmentation: true
          tx-nocache-copy: false
          tx-tcp-mangleid-segmentation: false
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
          tx-udp_tnl-csum-segmentation: true
          tx-udp_tnl-segmentation: true
          tx-vlan-hw-insert: true
          tx-vlan-stag-hw-insert: true
        pause:
          autoneg: false
          rx: true
          tx: true
        ring:
          rx: 1024
          rx-max: 8192
          tx: 1024
          tx-max: 8192
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 86:9C:86:ED:37:8B
      max-mtu: 9978
      min-mtu: 68
      mtu: 1500
      name: enp175s0f0np0
      state: ignore
      type: ethernet
    - accept-all-mac-addresses: false
      controller: bond1
      ethernet:
        auto-negotiation: true
        duplex: full
        speed: 100000
        sr-iov:
          total-vfs: 0
          vfs: []
      ethtool:
        coalesce:
          adaptive-rx: true
          adaptive-tx: true
          rx-frames: 128
          rx-usecs: 8
          tx-frames: 128
          tx-usecs: 8
        feature:
          hw-tc-offload: true
          rx-all: false
          rx-checksum: true
          rx-fcs: false
          rx-gro: true
          rx-gro-list: false
          rx-hashing: true
          rx-lro: false
          rx-ntuple-filter: false
          rx-udp-gro-forwarding: false
          rx-udp_tunnel-port-offload: true
          rx-vlan-filter: true
          rx-vlan-hw-parse: true
          tx-checksum-ip-generic: true
          tx-generic-segmentation: true
          tx-gre-csum-segmentation: true
          tx-gre-segmentation: true
          tx-gso-partial: true
          tx-ipxip4-segmentation: true
          tx-ipxip6-segmentation: true
          tx-nocache-copy: false
          tx-tcp-mangleid-segmentation: false
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
          tx-udp_tnl-csum-segmentation: true
          tx-udp_tnl-segmentation: true
          tx-vlan-hw-insert: true
          tx-vlan-stag-hw-insert: true
        pause:
          autoneg: false
          rx: true
          tx: true
        ring:
          rx: 1024
          rx-max: 8192
          tx: 1024
          tx-max: 8192
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 86:9C:86:ED:37:8B
      max-mtu: 9978
      min-mtu: 68
      mtu: 1500
      name: enp175s0f1np1
      state: ignore
      type: ethernet
    - accept-all-mac-addresses: false
      ethtool:
        feature:
          rx-gro: true
          rx-gro-list: false
          rx-udp-gro-forwarding: false
          tx-generic-segmentation: true
          tx-gso-list: true
          tx-sctp-segmentation: true
          tx-tcp-ecn-segmentation: true
          tx-tcp-mangleid-segmentation: true
          tx-tcp-segmentation: true
          tx-tcp6-segmentation: true
          tx-udp-segmentation: true
      ipv4:
        address:
        - ip: 127.0.0.1
          prefix-length: 8
        enabled: true
      ipv6:
        address:
        - ip: ::1
          prefix-length: 128
        enabled: true
      mac-address: "00:00:00:00:00:00"
      mtu: 65536
      name: lo
      state: ignore
      type: loopback
    routes:
      config:
      - destination: 10.203.92.0/23
        next-hop-address: 198.19.0.30
        next-hop-interface: bond1
        table-id: 254
      - destination: 0.0.0.0/0
        next-hop-address: 198.18.0.1
        next-hop-interface: eno1
        table-id: 254
      running:
      - destination: 10.203.92.0/23
        next-hop-address: 198.19.0.30
        next-hop-interface: bond1
        table-id: 254
      - destination: 0.0.0.0/0
        next-hop-address: 198.18.0.1
        next-hop-interface: eno1
        table-id: 254
  handlerNmstateVersion: nmstatectl 2.2.13
  hostNetworkManagerVersion: 1.36.6
  lastSuccessfulUpdateTime: "2024-05-02T16:12:49Z"
  • Problematic NodeNetworkConfigurationPolicy:
    Any but see above yaml, here is the actual output from the nncp
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  annotations:
    nmstate.io/webhook-mutating-timestamp: "1716582405477407436"
  creationTimestamp: "2024-05-24T20:26:45Z"
  generation: 1
  name: svc-uc-v108-dal4-qz4-sr7-rk047-s04
  resourceVersion: "525584546"
  uid: dae7d032-d339-4bbd-b7d9-ac1d6d42e2a6
spec:
  desiredState:
    interfaces:
    - ipv4:
        dhcp: false
        enabled: false
      mac-address: 02:10:2c:2c:7e:52
      name: svc-uc-v108
      state: up
      type: vlan
      vlan:
        base-iface: bond1
        id: 108
  nodeSelector:
    kubernetes.io/hostname: dal4-qz4-sr7-rk047-s04
status:
  conditions:
  - lastHeartbeatTime: "2024-05-28T16:03:32Z"
    lastTransitionTime: "2024-05-28T16:03:32Z"
    message: 1/1 nodes successfully configured
    reason: SuccessfullyConfigured
    status: "True"
    type: Available
  - lastHeartbeatTime: "2024-05-28T16:03:32Z"
    lastTransitionTime: "2024-05-28T16:03:32Z"
    reason: SuccessfullyConfigured
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2024-05-28T16:03:32Z"
    lastTransitionTime: "2024-05-28T16:03:32Z"
    reason: ConfigurationProgressing
    status: "False"
    type: Progressing
  lastUnavailableNodeCountUpdate: "2024-05-28T16:03:32Z"
  • kubernetes-nmstate image (use kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'):
kubernetes-nmstate-handler:v0.80.0
  • NetworkManager version (use nmcli --version)
nmcli tool, version 1.36.6
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.10", GitCommit:"7e54d50d3012cf3389e43b096ba35300f36e0817", GitTreeState:"clean", BuildDate:"2022-08-17T18:32:54Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.10", GitCommit:"7e54d50d3012cf3389e43b096ba35300f36e0817", GitTreeState:"clean", BuildDate:"2022-08-17T18:26:59Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g. from /etc/os-release):
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Kernel

5.15.0-1020
  • Others:
    nmstate pod log:

kubectl logs nmstate-handler-9zkz6 -n nmstate

{"level":"info","ts":"2024-05-28T15:48:59.214Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- bridge:\n    options:\n      stp:\n        enabled: false\n    port:\n    - name: usb-int-v53\n      vlan:\n        mode: trunk\n        trunk-tags:\n        - id-range:\n            max: 4094\n            min: 2\n  ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:20:34:a8:07:57\n  name: usb-int-br53\n  state: up\n  type: linux-bridge\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:True Reason:ConfigurationProgressing Message:Applying desired state LastHeartbeatTime:2024-05-28 15:48:59.21462073 +0000 UTC m=+3231.903715825 LastTransitionTime:2024-05-28 15:48:59.21462073 +0000 UTC m=+3231.903715825} {Type:Failing Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214621417 +0000 UTC m=+3231.903716506 LastTransitionTime:2024-05-28 15:48:59.214621417 +0000 UTC m=+3231.903716506} {Type:Available Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214623185 +0000 UTC m=+3231.903718275 LastTransitionTime:2024-05-28 15:48:59.214623185 +0000 UTC m=+3231.903718275} {Type:Pending Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214623966 +0000 UTC m=+3231.903719056 LastTransitionTime:2024-05-28 15:48:59.214623966 +0000 UTC m=+3231.903719056} {Type:Aborted Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214624079 +0000 UTC m=+3231.903719168 LastTransitionTime:2024-05-28 15:48:59.214624079 +0000 UTC m=+3231.903719168}]}","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:48:59.222Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:48:59.306Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 0, unknown: 1}, progressing: {true: 1, false: 0, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 0, unknown: 1}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:48:59.306Z","logger":"policyconditions","msg":"SetPolicyProgressing"}
{"level":"info","ts":"2024-05-28T15:49:03.879Z","logger":"probe","msg":"Running 'ping' probe"}
{"level":"info","ts":"2024-05-28T15:49:04.581Z","logger":"probe","msg":"Running 'dns' probe"}
{"level":"info","ts":"2024-05-28T15:49:05.185Z","logger":"probe","msg":"Running 'api-server' probe"}
I0528 15:49:06.236434       1 request.go:601] Waited for 1.046322626s due to client-side throttling, not priority and fairness, request: GET:https://172.16.0.1:443/apis/objectbucket.io/v1alpha1?timeout=32s
{"level":"info","ts":"2024-05-28T15:49:07.048Z","logger":"probe","msg":"Running 'node-readiness' probe"}
{"level":"info","ts":"2024-05-28T15:49:07.063Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"nmstate","nodenetworkconfigurationpolicy":"/usb-int-br53-dal4-qz4-sr7-rk047-s04","output":"setOutput: dns-resolver: {}\nroute-rules: {}\nroutes: {}\ninterfaces:\n- name: usb-int-br53\n  type: linux-bridge\n  state: up\n  mac-address: 02:20:34:a8:07:57\n  ipv4:\n    enabled: false\n    dhcp: false\n  bridge:\n    options:\n      stp:\n        enabled: false\n    port:\n    - name: usb-int-v53\n      vlan:\n        mode: trunk\n        trunk-tags:\n        - id-range:\n            min: 2\n            max: 4094\novs-db: {}\novn: {}\n\n \n"}
{"level":"info","ts":"2024-05-28T15:49:07.063Z","logger":"enactmentconditions","msg":"NotifySuccess","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.067Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- bridge:\n    options:\n      stp:\n        enabled: false\n    port:\n    - name: usb-int-v53\n      vlan:\n        mode: trunk\n        trunk-tags:\n        - id-range:\n            max: 4094\n            min: 2\n  ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:20:34:a8:07:57\n  name: usb-int-br53\n  state: up\n  type: linux-bridge\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.067145399 +0000 UTC m=+3239.756240528 LastTransitionTime:2024-05-28 15:49:07.067145399 +0000 UTC m=+3239.756240528} {Type:Failing Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.067145008 +0000 UTC m=+3239.756240137 LastTransitionTime:2024-05-28 15:49:07.067145008 +0000 UTC m=+3239.756240137} {Type:Available Status:True Reason:SuccessfullyConfigured Message:successfully reconciled LastHeartbeatTime:2024-05-28 15:49:07.067144298 +0000 UTC m=+3239.756239442 LastTransitionTime:2024-05-28 15:49:07.067144298 +0000 UTC m=+3239.756239442} {Type:Pending Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.06714591 +0000 UTC m=+3239.756241040 LastTransitionTime:2024-05-28 15:48:59 +0000 UTC} {Type:Aborted Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.067146601 +0000 UTC m=+3239.756241728 LastTransitionTime:2024-05-28 15:48:59 +0000 UTC}]}","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.082Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.082Z","logger":"controllers.NodeNetworkConfigurationPolicy.forceNNSRefresh","msg":"forcing NodeNetworkState refresh after NNCP applied","node":"dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.319Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.320Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.335Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/cos-dp-br53-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.421Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"cos-dp-br53-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.421Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.435Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v108-dal4-qz4-sr7-rk046-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.518Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v108-dal4-qz4-sr7-rk046-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.518Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.533Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/default-br90-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.620Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"default-br90-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.620Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.635Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v39-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.721Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v39-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.721Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.738Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/usb-int-br106-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.819Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-br106-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.819Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.835Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-br37-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.920Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-br37-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.920Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.935Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v37-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.050Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v37-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.050Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.069Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-vsi-br107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.157Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-vsi-br107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.157Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.173Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v108-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.263Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v108-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.263Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.279Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/default-br90-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.366Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"default-br90-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.366Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.382Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-br43-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.466Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-br43-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.466Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.482Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-vsi-v107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.567Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-vsi-v107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.567Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.587Z","logger":"enactmentconditions","msg":"Reset","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:08.590Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:10:2c:2c:7e:52\n  name: svc-uc-v108\n  state: up\n  type: vlan\n  vlan:\n    base-iface: bond1\n    id: 108\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[]}","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:08.598Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.358Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:10:2c:2c:7e:52\n  name: svc-uc-v108\n  state: up\n  type: vlan\n  vlan:\n    base-iface: bond1\n    id: 108\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[]}","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.367Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.474Z","logger":"enactmentconditions","msg":"NotifyProgressing","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.477Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:10:2c:2c:7e:52\n  name: svc-uc-v108\n  state: up\n  type: vlan\n  vlan:\n    base-iface: bond1\n    id: 108\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:True Reason:ConfigurationProgressing Message:Applying desired state LastHeartbeatTime:2024-05-28 15:49:09.477075435 +0000 UTC m=+3242.166170578 LastTransitionTime:2024-05-28 15:49:09.477075435 +0000 UTC m=+3242.166170578} {Type:Failing Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477076702 +0000 UTC m=+3242.166171846 LastTransitionTime:2024-05-28 15:49:09.477076702 +0000 UTC m=+3242.166171846} {Type:Available Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477077817 +0000 UTC m=+3242.166172945 LastTransitionTime:2024-05-28 15:49:09.477077817 +0000 UTC m=+3242.166172945} {Type:Pending Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477079183 +0000 UTC m=+3242.166174311 LastTransitionTime:2024-05-28 15:49:09.477079183 +0000 UTC m=+3242.166174311} {Type:Aborted Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477079649 +0000 UTC m=+3242.166174778 LastTransitionTime:2024-05-28 15:49:09.477079649 +0000 UTC m=+3242.166174778}]}","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.485Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.565Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 0, unknown: 1}, progressing: {true: 1, false: 0, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 0, unknown: 1}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.565Z","logger":"policyconditions","msg":"SetPolicyProgressing"}

I killed the pod and a new one properly started up. Tried the apply for another vlan interface and it still took > 60s to apply

I attached the full logs of nmstate-handler-9zkz6
logs--svc-uc-v108-apply.txt
@qinqon
Copy link
Member

qinqon commented May 30, 2024

kubernetes-nmstate do some probes before commit the changes to ensure that it do not break the cluster, that's where the time goes on.

@k8scoder192
Copy link
Author

k8scoder192 commented May 30, 2024 •
edited
Loading

@qinqon did you look at the full logs? It's NOT due to the probes. The probes come back extremely quick. Something else is wrong.

Edit
I also noticed this is only happening on 1 worker node. If I look at the logs of the other nmstate-handler pods, they aren't constantly reporting "enactment updated at the node: true". Something is triggering this pod to constantly perform state changes like below for all the interfaces

journalctl -S today -u NetworkManager.service -f (output on the node I see the problem on)

May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9313] device (usb-int-v87): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9321] device (usb-int-v87): Activation: starting connection 'usb-int-v87' (74ec521d-7a67-4ade-82ae-19f6892e645c)
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9326] device (usb-int-v87): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9342] device (usb-int-v87): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9371] device (usb-int-v87): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0251] device (usb-int-br87): attached bridge port usb-int-v87
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0251] device (usb-int-v87): Activation: connection 'usb-int-v87' enslaved, continuing activation
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0256] device (usb-int-v87): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0281] device (usb-int-v87): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0283] device (usb-int-v87): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0287] device (usb-int-v87): Activation: successful, device activated.
May 31 18:12:42 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179162.9356] checkpoint[0x55bcd4c8d450]: destroy /org/freedesktop/NetworkManager/Checkpoint/292
May 31 18:12:42 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179162.9390] audit: op="checkpoint-destroy" arg="/org/freedesktop/NetworkManager/Checkpoint/292" pid=2292492 uid=0 result="success"
May 31 18:12:46 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179166.5941] audit: op="checkpoint-create" arg="/org/freedesktop/NetworkManager/Checkpoint/293" pid=2292893 uid=0 result="success"
May 31 18:12:46 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179166.5968] audit: op="checkpoint-adjust-rollback-timeout" arg="/org/freedesktop/NetworkManager/Checkpoint/293" pid=2292893 uid=0 result="success"
May 31 18:12:47 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179167.3864] audit: op="connection-update" uuid="c504a0fa-730b-4182-8b98-f404ab44c196" name="usb-int-br67" args="connection.timestamp" pid=2292893 uid=0 result="success"
May 31 18:12:47 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179167.5521] audit: op="device-reapply" interface="usb-int-br67" ifindex=177998 pid=2292893 uid=0 result="success"
May 31 18:12:47 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179167.5563] audit: op="checkpoint-adjust-rollback-timeout" arg="/org/freedesktop/NetworkManager/Checkpoint/293" pid=2292893 uid=0 result="succe

When I look at NNCP for that node, they are constantly going from "Available SuccessfullyConfigured" to blank status then back to "Available SuccessfullyConfigured". When that happens I see the above journalctl output for that particular interface.

I really need to figure out why k8s nmstate is behaving this way. This node's nmstate-handler pod restarted this week. The other node's pods have been up for 200+ days. I'm afraid to bound the other node's pods because of this odd behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@qinqon @k8scoder192