Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add num of tasks running health validation #13

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

italux
Copy link

@italux italux commented Jun 4, 2021

Description

In some cases, I've seen that the number of running tasks are under the tasks.max configuration number which can leverage partitions to become without any consumer and to avoid this behavior I'm adding validation to the handle_task_healthcheck comparing the number of running tasks on the container obtained from get_connectors_health with the desired tasks.max

This validation is enabled by default but I've added a new param to disable the validation passing --failure-less-tasks False or setting HEALTHCHECK_FAILURE_LESS_TASKS=False

  • Logging samples with DEBUG level
2021-06-04 00:38:44.293 [   INFO] - Initializing healthcheck server...
2021-06-04 00:38:44.296 [   INFO] - Healthcheck server started at: http://localhost:18083
2021-06-04 00:38:44.296 [   INFO] - ------------------------------------------------
2021-06-04 00:38:45.137 [  DEBUG] - Starting new HTTP connection (1): localhost:8083
2021-06-04 00:38:46.202 [  DEBUG] - http://localhost:8083 "GET /connectors HTTP/1.1" 200 25
2021-06-04 00:38:46.206 [  DEBUG] - Starting new HTTP connection (1): localhost:8083
2021-06-04 00:38:47.071 [  DEBUG] - http://localhost:8083 "GET /connectors/ProtoElasticConnector/status HTTP/1.1" 200 481
2021-06-04 00:38:47.074 [  DEBUG] - Starting new HTTP connection (1): localhost:8083
2021-06-04 00:38:47.918 [  DEBUG] - http://localhost:8083 "GET /connectors/ProtoElasticConnector HTTP/1.1" 200 1472
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [  DEBUG] - Connector 'ProtoElasticConnector' state: RUNNING tasks running: '6' desired: '6'
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' task '0' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' task '1' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' task '2' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' task '3' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' task '4' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [   INFO] - Connector 'ProtoElasticConnector' task '5' is healthy in state: RUNNING
2021-06-04 00:38:47.919 [   INFO] - ------------------------------------------------

@italux italux marked this pull request as ready for review June 4, 2021 04:10
@italux
Copy link
Author

italux commented Jul 4, 2021

@devshawn Do you think this is useful?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant