Client-Side Rate Limiting Issue with Consul Agents in Kubernetes Environment

[brainshell]

Overview of the Issue

We encountered a rate-limiting issue when scaling up many new nodes with Consul agents in our Kubernetes clusters. New agents use the Kubernetes Auth Method for login and exceed the consul server leader's default QPS and Burst settings of the Kubernetes client hitting the token review API. This issue is causing a significant bottleneck, limiting the number of new nodes that can be brought online at one time. The Consul Server is unable to complete the consul agent's logins once the Kubernetes client-go client-side rate limiting hits, and the leader gets stuck in an infinite loop of retries with increasing backoff time. The leader consul server will be trapped in this retry backoff loop until we remove the pressure, or a new leader is elected.

Resolving this issue will significantly improve the scalability and reliability of Consul services in Kubernetes environments. We would greatly appreciate assistance in addressing this problem.

---

Reproduction Steps

1. Create a cluster with 3 server nodes
2. Create >300 consul agents at once using Kubernetes Auth Method
3. Observe reaching the client-side rate limit log on consul server

Expected Behavior

The Consul agents should scale without hitting Kubernetes client-side rate limits, allowing for a smooth and efficient scaling process.

Actual Behavior

The leader consul server is reaching the default rate limits within the Kubernetes client-go package, preventing new nodes and their consul agents from coming online as expected. The leader gets stuck in an infinite loop of retries with increasing backoff time.

Workaround

The workaround for this problem is to manually delete or reduce the new nodes and consul agents to relieve pressure on the consul server. Once stuck in the infinite loop retrying with a long backoff, we must do a consul leave on the leader to reset the rate limit.

Proposed Resolution

A proposed resolution for this problem is exposing the QPS and Burst parameters for the Kubernetes auth method validator client as configurable variables for the server pods defaulted to the upstream defaults. We have tested increasing these parameters and confirmed that it eliminates the client-side rate-limiting issue and does not overwhelm the control plane. By modifying these limits in the code and rebuilding the Consul binary, we were able to eliminate the client-side rate limiting issue. Our testing with configuring QPS and Burst parameters of the validator client confirms that rate limiting no longer occurs for the consul server with sufficient QPS and Burst.

The proposed resolution does not remove the client-side rate limiting altogether but allows cluster operators to configure the QPS and Burst values according to their cluster size and performance. By default, the Kubernetes Auth Method validator client should still use the same settings as the default Kubernetes client-go library, so there will be no change in behavior for existing clusters. Cluster operators can adjust these settings based on their own testing and monitoring results and ensure that their control plane can handle the increased load without affecting other Kubernetes API clients.

Impact

The current rate limiting issue has the potential to impact service deployment and scaling within large Kubernetes clusters. Addressing this bug is crucial for maintaining operational efficiency and reliability for clusters that support burst workloads.

Consul info for both Client and Server

Client info

consul info
agent:
    check_monitors = 0
    check_ttls = 0
    checks = 0
    services = 0
build:
    prerelease =
    revision = 0e046bbb
    version = 1.13.2
    version_metadata =
consul:
    acl = enabled
    known_servers = 3
    server = false
runtime:
    arch = amd64
    cpu_count = 16
    goroutines = 53
    max_procs = 16
    os = linux
    version = go1.18.1
serf_lan:
    coordinate_resets = 0
    encrypted = true
    event_queue = 0
    event_time = 193
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 655959
    members = 6
    query_queue = 0
    query_time = 1

{
  "enable_central_service_config": true,
  "check_update_interval": "0s",
  "disable_host_node_id": false,
  "telemetry": {
    "prometheus_retention_time": "30s",
    "disable_hostname": true,
  },
  "limits": {
    "http_max_conns_per_client": -1,
    "rpc_max_conns_per_client": -1,
  },
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "tokens": {
      "agent": "REDACTED",
    },
  },
}

Server info

agent:
    check_monitors = 0
    check_ttls = 0
    checks = 0
    services = 0
build:
    prerelease =
    revision = 0e046bbb
    version = 1.13.2
    version_metadata =
consul:
    acl = enabled
    bootstrap = false
    known_datacenters = 6
    leader = true
    leader_addr = REDACTED:8300
    server = true
raft:
    applied_index = 9108175
    commit_index = 9108175
    fsm_pending = 0
    last_contact = 0
    last_log_index = 9108175
    last_log_term = 756
    last_snapshot_index = 9097562
    last_snapshot_term = 756
    latest_configuration = [REDACTED]
    latest_configuration_index = 0
    num_peers = 2
    protocol_version = 3
    protocol_version_max = 3
    protocol_version_min = 0
    snapshot_version_max = 1
    snapshot_version_min = 0
    state = Leader
    term = 756
runtime:
    arch = amd64
    cpu_count = 16
    goroutines = 483
    max_procs = 16
    os = linux
    version = go1.18.1
serf_lan:
    coordinate_resets = 0
    encrypted = true
    event_queue = 0
    event_time = 193
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 655955
    members = 6
    query_queue = 0
    query_time = 1
serf_wan:
    coordinate_resets = 0
    encrypted = true
    event_queue = 0
    event_time = 1
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 8662
    members = 18
    query_queue = 0
    query_time = 1

{
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "enable_token_replication": true,
    "enable_token_persistence": true
  },
  "enable_central_service_config": true,
  "leave_on_terminate": true,
  "telemetry": {
    "prometheus_retention_time": "30s",
    "disable_hostname": true
  },
  "limits": {
    "http_max_conns_per_client": -1,
    "rpc_max_conns_per_client": -1
  },
  "primary_datacenter": "REDACTED",
  "primary_gateways": [],
  "connect": {
    "enable_mesh_gateway_wan_federation": true
  },
  "bind_addr": "0.0.0.0",
  "bootstrap_expect": 3,
  "client_addr": "0.0.0.0",
  "connect": {
    "enabled": true
  },
  "datacenter": "REDACTED",
  "data_dir": "/consul/data",
  "domain": "consul",
  "ports": {
    "serf_lan": 8301,
    "grpc": 8503
  },
  "recursors": [],
  "retry_join": ["REDACTED-consul-server.consul-b.svc:8301"],
  "server": true,
  "telemetry": {
    "prometheus_retention_time": "1m"
  },
  "ca_file": "/consul/tls/ca/tls.crt",
  "cert_file": "/consul/tls/server/tls.crt",
  "key_file": "/consul/tls/server/tls.key",
  "verify_incoming_rpc": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ports": {
    "http": -1,
    "https": 8501
  },
  "ui_config": {
    "enabled": true
  }
}

Operating system and Environment details

• Consul Version: 1.13.2
• Consul-k8s Chart Version: 0.49.0
• Kubernetes Client-go Version: 1.18.2
• Kubernetes Cluster Version: 1.25.x

Log Fragments

Throttling request took 16m47.985873452s, request: POST:https://0.0.0.0/apis/authentication.k8s.io/v1/tokenreviews

Full logs Here: https://gist.github.com/brainshell/660ede2a1830c7d01083a8972ce7318d

Additional Context

• Identified locations in the source code with default rate limits.

Source Code

Validator Instantiation

consul/agent/consul/authmethod/kubeauth/k8s.go

Lines 103 to 113 in 0e046bb

	client, err := k8s.NewForConfig(&client_rest.Config{
	Host: config.Host,
	BearerToken: config.ServiceAccountJWT,
	Dial: transport.DialContext,
	TLSClientConfig: client_rest.TLSClientConfig{
	CAData: []byte(config.CACert),
	},
	ContentConfig: client_rest.ContentConfig{
	ContentType: "application/json",
	},
	})

Kube Client Config Defaults

// Config holds the common attributes that can be passed to a Kubernetes client on
// initialization.
type Config struct {
   ...   

   // QPS indicates the maximum QPS to the master from this client.
   // If it's zero, the created RESTClient will use DefaultQPS: 5
   QPS float32

   // Maximum burst for throttle.
   // If it's zero, the created RESTClient will use DefaultBurst: 10.
   Burst int
   
   ...
}

https://github.com/kubernetes/client-go/blob/6b7c68377979c821b73d98d1bd4c5a466034f491/rest/config.go#L116-L120