Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nodes getting deleted after a rate-limited cache refresh #2880

Closed
okushchenko opened this issue Dec 9, 2022 · 5 comments · May be fixed by #4535
Closed

Nodes getting deleted after a rate-limited cache refresh #2880

okushchenko opened this issue Dec 9, 2022 · 5 comments · May be fixed by #4535
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@okushchenko
Copy link
Contributor

What happened:

After adding a bunch of new nodes to our Kubernetes cluster, I observed that some of them failed to get registered. Based on the log line below, it seems like cloud-controller-manager failed to refresh the VM cache due to a rate limit, and deleted a k8s node afterwards, because it wasn't in the cache.

INFO  [2022-12-09T16:39:57.990980639Z] sigs.k8s.io/cloud-provider-azure/azure_vmssflex_cache.go:204: Could not find node (node-foo) in the existing cache. Forcely freshing the cache to check again... (pid: 10)
ERROR [2022-12-09T16:39:58.2093317Z] sigs.k8s.io/cloud-provider-azure/util.go:137: Send.sendRequest: error in parsing response body string "\"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"TooManyRequests\\\",\\r\\n        \\\"message\\\": \\\"{\\\\\\\"operationGroup\\\\\\\":\\\\\\\"HighCostGet3Min\\\\\\\",\\\\\\\"startTime\\\\\\\":\\\\\\\"2022-12-09T16:39:58.2035018+00:00\\\\\\\",\\\\\\\"endTime\\\\\\\":\\\\\\\"2022-12-09T16:41:00+00:00\\\\\\\",\\\\\\\"allowedRequestCount\\\\\\\":140,\\\\\\\"measuredRequestCount\\\\\\\":866}\\\",\\r\\n        \\\"target\\\": \\\"HighCostGet3Min\\\"\\r\\n      }\\r\\n    ],\\r\\n    \\\"innererror\\\": {\\r\\n      \\\"internalErrorCode\\\": \\\"TooManyRequestsReceived\\\"\\r\\n    },\\r\\n    \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n    \\\"message\\\": \\\"The server rejected the request because too many requests have been received for this subscription.\\\"\\r\\n  }\\r\\n}\"": json: cannot unmarshal string into Go value of type map[string]interface {}, Skip retrying regional host (pid: 10)
ERROR [2022-12-09T16:39:58.209414301Z] sigs.k8s.io/cloud-provider-azure/azure_vmssflex_cache.go:86: ListVmssFlexVMsWithoutInstanceView failed: &{false 429 0001-01-01 00:00:00 +0000 UTC "{\r\n  \"error\": {\r\n    \"details\": [\r\n      {\r\n        \"code\": \"TooManyRequests\",\r\n        \"message\": \"{\\\"operationGroup\\\":\\\"HighCostGet3Min\\\",\\\"startTime\\\":\\\"2022-12-09T16:39:58.2035018+00:00\\\",\\\"endTime\\\":\\\"2022-12-09T16:41:00+00:00\\\",\\\"allowedRequestCount\\\":140,\\\"measuredRequestCount\\\":866}\",\r\n        \"target\": \"HighCostGet3Min\"\r\n      }\r\n    ],\r\n    \"innererror\": {\r\n      \"internalErrorCode\": \"TooManyRequestsReceived\"\r\n    },\r\n    \"code\": \"OperationNotAllowed\",\r\n    \"message\": \"The server rejected the request because too many requests have been received for this subscription.\"\r\n  }\r\n}"} (pid: 10)
ERROR [2022-12-09T16:39:58.209452998Z] sigs.k8s.io/cloud-provider-azure/azure_vmssflex_cache.go:190: failed to refresh vmss flex VM cache for vmssFlexID /subscriptions/0000000-0000-0000-0000-00000000/resourceGroups/resource-group-name/providers/Microsoft.Compute/virtualMachineScaleSets/flex-scale-set-name (pid: 10)

What you expected to happen:

I'd expect that when cloud-controller-manager is getting rate-limited while refreshing the VM cache, it'd not try to delete nodes afterwards, as some of the cache is out-of-date.

How to reproduce it (as minimally and precisely as possible):

  1. Run some client that does a lot of VM lists besides cloud-controller-manager.
  2. Try to add a new node to your k8s cluster.
  3. Observe that the node gets registered and then immediately de-registered by CCM, after it fails to refresh the VM cache.

Anything else we need to know?:

Environment:

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.801", GitCommit:"3af12a99be420abe240c12af3eea549e2b8ba9ad", GitTreeState:"clean", BuildDate:"2022-11-11T10:59:47Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

Note that we use Flex VM Scale Sets, Kubernetes 1.24, and https://github.com/kubernetes-sigs/cloud-provider-azure/releases/tag/v1.25.5.
@okushchenko okushchenko added the kind/bug Categorizes issue or PR as related to a bug. label Dec 9, 2022
@zmyzheng
Copy link
Contributor

zmyzheng commented Dec 9, 2022

Hi @okushchenko , VMSS Flex will be officially supported starting from v1.26.0 (which is going to be release this month). In the github ticket you created, I notice you are using v1.25.5. I think the latest VMSS Flex code might not be cherry picked to the v1.25.5 version so it is possible the 1.25.5 is not stable for VMSS Flex.

Before the official release of v1.26.0, we have built a stable version of Cloud Controller Manager (CCM) and Cloud Node Manager (CNM) and stored them into an Azure Container Registry:

CCM: selfhostedk8s.azurecr.io/azure-cloud-controller-manager:d8d65f1
CNM: selfhostedk8s.azurecr.io/azure-cloud-node-manager:d8d65f1

Could you please try the above images and let us if this can mitigate the issue?

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 9, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 8, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale May 8, 2023
@gcampbell12 gcampbell12 mentioned this issue Sep 5, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improvements to reduce rate limiting with flex scalesets gcampbell12/cloud-provider-azure
4 participants
@okushchenko @k8s-ci-robot @zmyzheng @k8s-triage-robot