New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI] DockerTests test500Readiness failing #108523
Comments
Pinging @elastic/es-delivery (Team:Delivery) |
Pinging @elastic/es-core-infra (Team:Core/Infra) |
This test is related to the readiness probe so adding the core/infra label to this. Not sure if any changes have been made in that area lately. Also muting for now. |
@rjernst As far as I can tell the test waits for ES to report green, then attempts to connect on the readiness port and fails. Is there any scenario in which the node would report as green but the readiness port would not be listening? Is there some kind of race condition here? |
Readiness is orthogonal to health. It waits for 2 conditions to be met:
If the cluster is green, a master node should be there. So that leaves file settings, but this test doesn't use file settings, so there shouldn't be any waiting. I'm attempting to gather more information but dumping the ES log file before we fail the test: |
Previously readiness waited only on a master node being elected. Recently it was also made to wait on file settings being applied. Yet the node may be fully started before those file settings are applied. The test expected readiness was ok after the node finishes starting. This commit retries the readiness check until it succeeds since readiness state will be updated async to the node finishing starting. closes elastic#108523
Previously readiness waited only on a master node being elected. Recently it was also made to wait on file settings being applied. Yet the node may be fully started before those file settings are applied. The test expected readiness was ok after the node finishes starting. This commit retries the readiness check until it succeeds since readiness state will be updated async to the node finishing starting. closes #108523
Previously readiness waited only on a master node being elected. Recently it was also made to wait on file settings being applied. Yet the node may be fully started before those file settings are applied. The test expected readiness was ok after the node finishes starting. This commit retries the readiness check until it succeeds since readiness state will be updated async to the node finishing starting. closes elastic#108523
Build scan:
https://gradle-enterprise.elastic.co/s/jfhv4jdyalmjm/tests/:qa:packaging:destructiveDistroTest.default-docker/org.elasticsearch.packaging.test.DockerTests/test500Readiness
Reproduction line:
Applicable branches:
main
Reproduces locally?:
Didn't try
Failure history:
Failure dashboard for
org.elasticsearch.packaging.test.DockerTests#test500Readiness
Failure excerpt:
The text was updated successfully, but these errors were encountered: