-
Notifications
You must be signed in to change notification settings - Fork 262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Responses do not arrive #495
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior
We still expect every message and repsponse to arrive correctly at the subscribed clients
Actual behavior
We have the problem that sometimes our service does not receive the responses to their requests anymore.
It always happens for the same service (which has the highest load), for this example I will refer to it as Service A. Service A receives a lot of messages via Websocket. Those requests are sent to another service with a response topic. At somepoint the responses do not arrive at ServiceA anymore. We can see in the logs that the other services still receive the requests and send the responses to the response topic. We can also see that the reponses arrive at the hiveMq-Broker. But Service B (while still sending requests) just do not receive those responses anymore.
We do not know what causes this issue but for some reasons the subscriptions of Service A are removed or get lost?
In order to fix this issue we need to restart Service A. After it booted, it subscribes to the topics again and receives all the messages. There are no changes done to the service other then restarting it, which indicates that we subscribe correctly. Also all the other services we use in our system still are able to receive and send messages.
To Reproduce
That is the funny part. We have no idea what causes the broker/Service to act like this. It mostly happend after a deployment when we receive a lot of requests at the same time. But it also appeared out of nowhere, without any warning. The issue always affects the same services. The whole system runs again after the restart of the service.
Steps
Reproducer code
Details
The text was updated successfully, but these errors were encountered: