[BUG] Prevent "The MessageProducer was closed due to an unrecoverable error" #817
Comments
|
Thanks for reporting this. Could you please provide more information, such as how to reproduce this issue? |
|
Ok. |
|
@carcaryn you can use this property to set the idle timeout |
|
@neuqlz @saragluna The BIG problem here is that jmsTemplate can no longer be used, which kind of breaks the idea of autoconfiguration - minimal setup that just works. Let's put it like this - if it falls on me to implement reconnect logic, I'll have to override the beans created by this jms starter - then it turns out I don't need said starter at all... So in fact, in my project, instead of using those neat @JmsListener and jmsTemplate features (all done with Apache Qpid), we had to revert to using EDIT: attached log excerpt - the root cause says |
|
@zoladkow, your proposal is very reasonable, now the ability of JMS starter is very single, just support set fixed idle-timeout for the connection. And don't support the other fancy features, like reconnect after revoke, never-expired and so on. It's indeed a problem that decreases the usability of the JMS starter. We will discuss it and consider adding these related features to the starter. Now you can set the idle-timeout very long to workaround. And also PR is very welcome. Please stay tuned. |
|
@neuqlz Is there any update on this issue. I have tried to setup idle timeout very high but still facing same issue. Could you please provide some inputs to handle this issue. |
|
@neuqlz Hi, we faced with the same issue. Any updates regarding JMS starter? |
|
Sorry, currently we don't have a concrete plan about this feature mentioned above. |
|
Is there a workaround for this issue? |
|
I disabled cache producers like this CachingConnectionFactory connectionFactory = (CachingConnectionFactory) jmsTemplate.getConnectionFactory(); connectionFactory.setCacheProducers(false); Work for me. In future i plan to implement reconect functionality. |
|
I tried disabling the JMS Cache Producer. It is also not working for me. In my work scenario I have 50 messages to send, but when I disable this cache producer only 2 messages sent and job remaining idle without any error or anything. I would need some one step up and check this ASAP. |
|
I am facing the same issue. After timeout I am getting the same exception and can't reuse the JmsTemplate. |
|
Hi @Arnab09 , sorry for the late response. Due to that we don't have plans to enhance the reconnect function recently, so the current workaround for this issue is to disable cache of MessageProducer like @DzianisShaplyka1984 said. |
|
Also want to mention that this can also occur when you modify a setting within Azure Service Bus. For example, I can reproduce it by doing the following scenarios for Topics: Scenario 1
Scenario 2
So while it isn't even the timeout but rather changes on the Azure side that can also cause the problem which is arguably worse... |
|
@welsh thanks for your reporting, we will look into this. |
Stuck with the same issue.. :( To solve this I created @Bean
public ConnectionFactory jmsConnectionFactory(AzureServiceBusJMSProperties busJMSProperties){
final String connectionString = busJMSProperties.getConnectionString();
final String clientId = busJMSProperties.getTopicClientId();
final int idleTimeout = busJMSProperties.getIdleTimeout();
final ServiceBusKey serviceBusKey = ConnectionStringResolver.getServiceBusKey(connectionString);
final String remoteUri = String.format("amqps://%s?amqp.idleTimeout=%d&amqp.traceFrames=true",
serviceBusKey.getHost(), idleTimeout);
final JmsConnectionFactory jmsConnectionFactory =
new JmsConnectionFactory(
serviceBusKey.getSharedAccessKeyName(),
serviceBusKey.getSharedAccessKey(),
remoteUri
);
jmsConnectionFactory.setClientID(clientId);
CachingConnectionFactory cachingConnectionFactory =
new CachingConnectionFactory(jmsConnectionFactory);
// set cache producers to FALSE here
cachingConnectionFactory.setCacheProducers(false);
return cachingConnectionFactory;
}Is there any better way of doing this or any workaround to set cache producers FALSE..? |
|
Hi I had/have the same issue with Sleuth. I resolved it with /**
* We need to disable cache producers because Azure Service Bus kills connections
* after some time. Sleuth uses BeanPostProcessor for wrapping a JMS
* ConnectionFactory and then there is no way how to get the original
* ConnectionFactory. This BeanPostProcessor with higher precedence disables
* cache producers before Sleuth wraps the ConnectionFactory.
*/
public class AzureServiceBusBraveBeanPostProcessor implements BeanPostProcessor, Ordered {
@Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
if (bean instanceof CachingConnectionFactory) {
CachingConnectionFactory cachingConnectionFactory = (CachingConnectionFactory) bean;
cachingConnectionFactory.setCacheProducers(false);
}
return bean;
}
/**
* It has to run before Sleuth/Brave TracingConnectionFactoryBeanPostProcessor
*
* @return
*/
@Override
public int getOrder() {
return 0;
}
}Do you plan fix this issue? It is open for more than one year. This is the real blocker (you have to mention this behaviour at least in your documentation) and disabling caching is not optimal. Is this real working supported library or only PoC? Because I feel it is only PoC. You should mention that this library is not production ready because it is not. |
|
Given that CachingConnectionFactory of Spring Jms doesn't expose the cached session, producer and consumer for us to remove when items are closed, thus we removed the usage of cache. |
|
@craigblackman2 Perhaps it's a misunderstanding. The "message processing" should not be understood as the job that is started as a result of receiving a message. Any time you see that a documentation makes a distinction between these tree steps: a.client receives a message, b.client processes the message, c.client acks/rejects the message, it's in fact still a single logical action that should be executed and finished ASAP. Even if you set the session or receive mode to CLIENT_ACKNOWLEDGE or PEEKLOCK or some such this needs to happen fast. If your solution expects to start a job, then any result (or error) of this job should be communicated as a separate message. If your solution expects to start and finish a job within the same transaction, then you need to find other ways to accomplish this. Also it might not be the best idea to have transactions hanging for so long. |
|
@superrdean It's not a "fancy feature" just a valid expectation that a starter package should not contain such a trivial and unreasonable limitation like a timeout exception that is not being handled. I mean there are workarounds, true, but those should not be required on the client code, which just has JmsTemplate instance provided by injection. @jialigit Which brings me to your problem summary. I mean, my biggest problem is that it is a STARTER and starters are meant to be just plug&play with bare minimum config required. And having some background job sending messages with undetermined intervals is a VERY COMMON use case. Heck, pooling should not be even needed for that. |
|
@zoladkow thanks for the post, yes I take your point. Long running transactions are not ideal and we will likely have to look at an alternative approach. @hui1110 I am coming to the conclusion that the connection eviction will not happen in an app where a |
|
@craigblackman2 There are two points that need to be confirmed with you. After add the following configuration in spring:
jms:
servicebus:
pool:
enabled: true
max-connections:
But when the job after receiving the message is completed, there will be the following exception: |
|
@craigblackman2 Can you help confirm the above two points? |
|
@hui1110 sorry for the delay. Point 1 - yes this is the case. Point 2 - yes to an extent - it does not have MessageProducer errors if a job is around 5 minutes but we still see other session closed errors as in the logs shared above if a job is 10 minutes or more. We will then see the Transaction is not declared Reference error. If a job lasts between 5 and 10 minutes, the sending succeeds but the JMS listener fails so the message for the second job is lost. |
|
Could you please update the status of this issue? |
|
This issue is for tracking the exception: |
Can we close it now?
How about creating a new issue? |
|
Sorry, I do not want to be a prude but how is this issue fixed? By this: #817 (comment) ? |
|
That's not being prude at all. This situation is just silly. Since Apache Qpid JMS client employed in this starter as a shortcut to have an easy win, while it does speak the right protocol AMQP, it does not recognize particular ASB specific states or error messages, it has no way to employ any sort of reasonable strategies. Conversely, the proper library for working with Azure Service Bus - Suggestion, @chenrujun, @hui1110, maybe, instead of blaming Spring JMS + qpid-jms-client combo for not exposing "session, producer and consumer" to mitigate the "unrecoverable" error, why not to reimplement this starter on top of edit: The reasonable thing would be to acknowledge that the current solution on top of just Apache Qpid JMS client has limitations with regard to particular behavior of ASB, which prevent a basic use case of this starter from working properly. As such, the decision should be made to employ a proper ASB client library, which would prevent such failures. After all spring boot starters are not meant to replace a full fledged library, they are not meant to result in the smallest dependency tree and lightest overall weight of resulting artifact. They are expected to take the original library and provide reasonable defaults with minimal coding needed on the client part. |
@hui1110 what's the fix when you say 'has been fixed'? |
Enable pools to avoid exceptions thrown when sending messages: |
Really? Where is it enabled? It is starting ridiculous. This is just a workaround (which all users of this library have to use). It is not described anywhere (only in one comment in this issue). If you think it is enough then it is really sad. If you think it is the solution then you should enable it by default. You should definitely write it to your reference documentation. It is completely missing there. It is really funny that someone have time to advocate this solution as fix (pretend it is no problem) but he/they do not have time to write it to reference documentation or enable it in autoconfiguration (I can understand that it is not backward compatible). Do you really want to solve this issue again and again? |
|
@Saljack we agree that using pool connection facotry for such case should not be announced as a fix, a fix should be able to enable send/receive messages normally and smoothly without exceptions or without extra configuration. Our current plan for this issue that:
|
stliu
changed the title
|
I had the same issue and I fixed it by enabling pool in spring boot More information |
Hi everyone,
I've this problem with Azure Jms Spring Library. After few times of innactivity, when I try to send a message to a topic, the library throw me this exception: "The MessageProducer was closed due to an unrecoverable error; nested exception is javax.jms.IllegalStateException: The MessageProducer was closed due to an unrecoverable error".
How can I prevent this problem? Is a configuration problem? I know that azure close connection after ten minutes of inactivity, but how can I resolve the MessageProducer error in my project?
Thanx
Project azure-servicebus-jms-spring-boot-starter
Version 2.1.7
The text was updated successfully, but these errors were encountered: