Resetting the client before ARM throttling happens to avoid throttling #1997

chmill-zz · 2022-07-08T02:53:57Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

ARM throttling is different than Azure RP throttling. Described here https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling

To avoid the instance level throttling per hour we should be able to reset the client to reset which instance we are communicating with on the ARM side. This will avoid ARM throttling.

Today we stick with the same instance and can hit throttling unnecessarily.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

linux-foundation-easycla · 2022-07-08T02:54:01Z

The committers listed above are authorized under a signed CLA.

✅ login: chmill-zz (d53bd69, 8d986ec)

netlify · 2022-07-08T02:54:02Z

✅ Deploy Preview for kubernetes-sigs-cloud-provide-azure ready!

Name	Link
🔨 Latest commit	`8d986ec`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/62c79c4757848f0009f92215
😎 Deploy Preview	https://deploy-preview-1997--kubernetes-sigs-cloud-provide-azure.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-07-08T02:54:03Z

✅ Deploy Preview for warm-cascaron-bb779f ready!

Name	Link
🔨 Latest commit	`8d986ec`
🔍 Latest deploy log	https://app.netlify.com/sites/warm-cascaron-bb779f/deploys/62c79c47bae3b6000a33239f
😎 Deploy Preview	https://deploy-preview-1997--warm-cascaron-bb779f.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

k8s-ci-robot · 2022-07-08T02:54:05Z

Welcome @chmill-zz!

It looks like this is your first PR to kubernetes-sigs/cloud-provider-azure 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cloud-provider-azure has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2022-07-08T02:54:05Z

Hi @chmill-zz. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2022-07-08T02:54:08Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chmill-zz
To complete the pull request process, please assign nilo19 after the PR has been reviewed.
You can assign the PR to them by writing /assign @nilo19 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nilo19 · 2022-07-12T02:55:11Z

pkg/azureclients/diskclient/azure_diskclient.go

@@ -470,3 +476,11 @@ func (page DiskListPage) Values() []compute.Disk {
 }
 return *page.dl.Value
 }
+
+func recreateClientChecker(response *http.Response) {
+ needToRecrateClient := armclient.RecreateClientDueToArmLimits(response)


typo: recrate->recreate

nilo19 · 2022-07-12T02:56:08Z

pkg/azureclients/diskclient/azure_diskclient.go

@@ -470,3 +476,11 @@ func (page DiskListPage) Values() []compute.Disk {
 }
 return *page.dl.Value
 }
+
+func recreateClientChecker(response *http.Response) {


Can we extract this into util.go?

I think to put in this in util.go I'd need to call InitializeCloudFromConfig because I wouldn't know the client we're using at the time of the calls and just reset them all. I'm not confident on this approach or impact though. Or possibly NewCloud is better to call?

nilo19 · 2022-07-12T02:56:19Z

/ok-to-test

coveralls · 2022-07-12T03:02:27Z

Coverage decreased (-0.2%) to 79.543% when pulling 8d986ec on chmill-zz:handleArmThrottling into a756b8e on kubernetes-sigs:master.

nilo19 · 2022-07-13T01:12:43Z

@chmill-zz the test coverage drops a little. Is it possible to enrich it?

lzhecheng · 2022-07-28T08:08:45Z

/retest

bridgetkromhout · 2022-11-16T21:03:00Z

/retest

bridgetkromhout · 2022-11-17T20:07:13Z

/retest

k8s-ci-robot · 2022-12-13T16:22:55Z

@chmill-zz: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2023-12-18T13:24:22Z

@chmill-zz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb`
pull-cloud-provider-azure-check	`8d986ec`	link	true	`/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-unit	`8d986ec`	link	true	`/test pull-cloud-provider-azure-unit`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-capz`
pull-cloud-provider-azure-e2e-ccm-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-shared-probe-capz	`8d986ec`	link	true	`/test pull-cloud-provider-azure-e2e-ccm-vmss-shared-probe-capz`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-triage-robot · 2024-03-17T13:36:59Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-04-16T13:51:55Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-05-16T14:19:19Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2024-05-16T14:19:23Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chmill-zz added 2 commits July 1, 2022 13:37

starting point for handling arm throttling

d53bd69

arm limits for disk client scenario

8d986ec

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. labels Jul 8, 2022

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jul 8, 2022

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 8, 2022

k8s-ci-robot requested review from feiskyer and jwtty July 8, 2022 02:54

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 8, 2022

nilo19 reviewed Jul 12, 2022

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 12, 2022

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 13, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 17, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 16, 2024

k8s-ci-robot closed this May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resetting the client before ARM throttling happens to avoid throttling #1997

Resetting the client before ARM throttling happens to avoid throttling #1997

chmill-zz commented Jul 8, 2022

linux-foundation-easycla bot commented Jul 8, 2022 •

edited

netlify bot commented Jul 8, 2022 •

edited

netlify bot commented Jul 8, 2022 •

edited

k8s-ci-robot commented Jul 8, 2022

k8s-ci-robot commented Jul 8, 2022

k8s-ci-robot commented Jul 8, 2022

nilo19 Jul 12, 2022

nilo19 Jul 12, 2022

chmill-zz Jul 18, 2022

nilo19 commented Jul 12, 2022

coveralls commented Jul 12, 2022

nilo19 commented Jul 13, 2022

lzhecheng commented Jul 28, 2022

bridgetkromhout commented Nov 16, 2022

bridgetkromhout commented Nov 17, 2022

k8s-ci-robot commented Dec 13, 2022

k8s-ci-robot commented Dec 18, 2023

k8s-triage-robot commented Mar 17, 2024

k8s-triage-robot commented Apr 16, 2024

k8s-triage-robot commented May 16, 2024

k8s-ci-robot commented May 16, 2024

Resetting the client before ARM throttling happens to avoid throttling #1997

Resetting the client before ARM throttling happens to avoid throttling #1997

Conversation

chmill-zz commented Jul 8, 2022

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

linux-foundation-easycla bot commented Jul 8, 2022 • edited

netlify bot commented Jul 8, 2022 • edited

✅ Deploy Preview for kubernetes-sigs-cloud-provide-azure ready!

netlify bot commented Jul 8, 2022 • edited

✅ Deploy Preview for warm-cascaron-bb779f ready!

k8s-ci-robot commented Jul 8, 2022

k8s-ci-robot commented Jul 8, 2022

k8s-ci-robot commented Jul 8, 2022

nilo19 Jul 12, 2022

Choose a reason for hiding this comment

nilo19 Jul 12, 2022

Choose a reason for hiding this comment

chmill-zz Jul 18, 2022

Choose a reason for hiding this comment

nilo19 commented Jul 12, 2022

coveralls commented Jul 12, 2022

nilo19 commented Jul 13, 2022

lzhecheng commented Jul 28, 2022

bridgetkromhout commented Nov 16, 2022

bridgetkromhout commented Nov 17, 2022

k8s-ci-robot commented Dec 13, 2022

k8s-ci-robot commented Dec 18, 2023

k8s-triage-robot commented Mar 17, 2024

k8s-triage-robot commented Apr 16, 2024

k8s-triage-robot commented May 16, 2024

k8s-ci-robot commented May 16, 2024

linux-foundation-easycla bot commented Jul 8, 2022 •

edited

netlify bot commented Jul 8, 2022 •

edited

netlify bot commented Jul 8, 2022 •

edited