[QUERY] Service bus Delivery not on receive link error

[akshit-mehta-rubrik]

Query/Question
Trying to complete messages in service bus using ServiceBusReceiverClient and getting error - "Delivery not on receive link". wanted to know the meaning of error, root cause and resolution steps.

Messages received: 28K (500 at a time and 28K/500 total receiveMessage calls)
Messages to be completed: 28K one by one in a loop without custom delay
Message lock duration: 4 minutes
Message auto lock duration: 400 minutes

Expected result: 28K messages deleted from queue
Actual result: 2K messages were deleted and for remaining messages "Delivery not on receive link" error was thrown while completing. The number varies in each error and cannot be confirmed.

  private val receiver =
    new ServiceBusClientBuilder()
      .credential(
        fullyQualifiedNamespace,
        credentials
      )
    )
      .receiver()
      .maxAutoLockRenewDuration(
        400
      )
      .disableAutoComplete()
      .queueName(queueName)
      .buildClient()

// delete logic
def deleteMessages(messageList: List[QueueMessage]): Int = {
    var messagesDeleted = 0
    messageList.foreach { msg =>
      {
        val result =
          Try(
            receiver.complete(
              msg
                .originalMessage
                .asInstanceOf[ServiceBusReceivedMessage]
            )
          )
        result match {
          case Failure(e) =>
            scribe.error(
              s"Error occurred while completing message: ${msg.id}." + //gives id of message
                s" Error message - ",e
            )
          case Success(_) => messagesDeleted += 1
        }
      }
    }
    
    messagesDeleted
  }

Why is this not a Bug or a feature Request?
A clear explanation of why is this not a bug or a feature request?
currently it's not certain that issue is due to SDK or something is missed from my end.

Setup (please complete the following information if applicable):

• Application runs in Azure Databricks and uses scala 2.12.x. Driver node configuration -
• a) Standard Ds3v2 - 14gb, 4 cores, JVM memory: ~6Gb,
  b) Standard D4asv5 - 16gb, 4 cores, JVM memory: ~8Gb
  for simplicity of recreating this error we're just doing service bus receive and delete and no other operation.
• Approximate queue message size: ~500Bytes as seen from Service bus UI
• Library/Libraries: com-azure:azure-messaging-servicebus:7.13.3, 7.10.1

Debug logs

    com.azure.messaging.servicebus.ServiceBusException: Delivery not on receive link.
    	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.lambda$updateDisposition$52(ServiceBusReceiverAsyncClient.java:1478)
    	at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3811)
    	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
    	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
    	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
    	at reactor.core.publisher.Operators.error(Operators.java:198)
    	at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
    	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
    	at reactor.core.publisher.Operators.error(Operators.java:198)
    	at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
    	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
    	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
    	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    	at reactor.core.publisher.Mono.block(Mono.java:1765)
    	at com.azure.messaging.servicebus.ServiceBusReceiverClient.complete(ServiceBusReceiverClient.java:174)

   Caused by: java.lang.IllegalArgumentException: Delivery not on receive link.
    	at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDispositionImpl(ReceiverUnsettledDeliveries.java:322)
    	at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDisposition(ReceiverUnsettledDeliveries.java:153)
    	at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.updateDisposition(ServiceBusReactorReceiver.java:120)
    	at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.updateDisposition(ServiceBusReceiveLinkProcessor.java:133)
    	at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.updateDisposition(ServiceBusAsyncConsumer.java:69)
    	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.updateDisposition(ServiceBusReceiverAsyncClient.java:1455)
    	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.complete(ServiceBusReceiverAsyncClient.java:421)
    	at com.azure.messaging.servicebus.ServiceBusReceiverClient.complete(ServiceBusReceiverClient.java:174)

24/05/10 15:07:40 DEBUG ServiceBusReceiverAsyncClient: {"az.sdk.message":"Update started.","lockToken":"f4048348-c939-424d-bf8c-393e074132b8","entityPath":"<queueName>","sessionId":null,"dispositionStatus":"COMPLETED"}
24/05/10 15:07:40 WARN ServiceBusReactorReceiver: {"az.sdk.message":"Delivery not found to update disposition.","entityPath":"<queueName>","linkName":"<queueName>_d948be_1715353650225","lockToken":"f4048348-c939-424d-bf8c-393e074132b8"}
24/05/10 15:07:40 ERROR ServiceBusReactorReceiver: {"az.sdk.message":"Delivery not on receive link.","exception":"Delivery not on receive link.","entityPath":"<queueName>","linkName":"<queueName>_d948be_1715353650225"}
java.lang.IllegalArgumentException: Delivery not on receive link.
        at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDispositionImpl(ReceiverUnsettledDeliveries.java:322)
        at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDisposition(ReceiverUnsettledDeliveries.java:153)
        at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.updateDisposition(ServiceBusReactorReceiver.java:120)
        at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.updateDisposition(ServiceBusReceiveLinkProcessor.java:133)
        at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.updateDisposition(ServiceBusAsyncConsumer.java:69)
        at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.updateDisposition(ServiceBusReceiverAsyncClient.java:1455)
        at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.complete(ServiceBusReceiverAsyncClient.java:421)
        at com.azure.messaging.servicebus.ServiceBusReceiverClient.complete(ServiceBusReceiverClient.java:174)

Information Checklist

• Query Added
• Setup information Added
• Debug logs

[github-actions]

@anuchandy @conniey @lmolkova

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[anuchandy]

@akshit-mehta-rubrik are you looking into delete all the messages in the queue? We are developing a feature that can delete them in bulk, which is in beta state right now.