You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have searched in the issues and found no similar issues.
Describe the feature
When I perform a stress test on the cluster, tasks occasionally encounter errors like this:
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [closed=[CANCELLED], committed=[remote_addr=xxx/xxx:xxx]]
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
We can increase rss.rpc.executor.size to reduce the probability of problems.
In this time, I will see metrics grpc_server_executor_blocking_queue_size like this:
This may mean that rpc requests are not processed in a timely manner. But there are no indicators that directly prove this. So we should add rpc queued time and rpc process time.
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?
Yes I am willing to submit a PR!
The text was updated successfully, but these errors were encountered:
Code of Conduct
Search before asking
Describe the feature
When I perform a stress test on the cluster, tasks occasionally encounter errors like this:
In this time, I will see metrics
grpc_server_executor_blocking_queue_size
like this:This may mean that rpc requests are not processed in a timely manner. But there are no indicators that directly prove this. So we should add rpc queued time and rpc process time.
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: