[BUG] StacklessClosedChannelException on Blob Downloads in Windows Azure Functions on Version 12.25.4

[SoulKa]

Describe the bug
I am getting io.netty.channel.StacklessClosedChannelException exceptions when downloading blobs using downloadToFile() in version 12.25.4. The same code works using version 12.25.3. It is running in an Azure Function. The download link is valid and the blob storage is available. I can reproduce the error. Probably an issue due to the netty upgrade in version 12.25.4?

Exception or Stack Trace

Exception: io.netty.channel.StacklessClosedChannelException
Stack: reactor.core.Exceptions$ReactiveException: io.netty.channel.StacklessClosedChannelException
	at reactor.core.Exceptions.propagate(Exceptions.java:396)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:98)
	at reactor.core.publisher.Mono.block(Mono.java:1742)
	at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:133)
	at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1208)
	at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1169)
	at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:1078)
	< CENSORED USER CLASSES >
	at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
	at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:22)
	at com.microsoft.azure.functions.worker.chain.FunctionExecutionMiddleware.invoke(FunctionExecutionMiddleware.java:19)
	at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
	at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invoke(JavaFunctionBroker.java:133)
	at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:124)
	at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:34)
	at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
	at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:44)
	at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:94)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)
		... 28 more
Caused by: io.netty.channel.StacklessClosedChannelException
	at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)
	Suppressed: io.netty.channel.StacklessClosedChannelException
		... 1 more
	Suppressed: io.netty.channel.StacklessClosedChannelException
		... 1 more
	Suppressed: io.netty.channel.StacklessClosedChannelException
		... 1 more

To Reproduce
Steps to reproduce the behavior:

1. Create an Azure Function that creates a blob client using new BlobClientBuilder().endpoint(blobSasUrl).buildClient() and a valid download URL to a blob with a valid SAS URL. Version 12.25.4 of the storage library has to be used.
2. Download an existing blob with client.downloadToFile(destinationPath, true)
3. Get the StacklessClosedChannelException

Code Snippet

var client = new BlobClientBuilder().endpoint(blobSasUrl).buildClient();
client.downloadToFile(destinationPath, true);

Expected behavior
The blob gets successfully downloaded as it was in version 12.25.3

Setup (please complete the following information):

• OS: Windows
• IDE: IntelliJ Ultimate
• Library/Libraries: com.azure:azure-sdk-bom:1.2.23
  • com.azure:azure-core-http-netty:1.14.2
  • com.azure:azure-storage-blob:12.25.4
• Java version: 17
• App Server/Environment: Azure Function running on Runtime 4.33.1.22394 and App Service Plan EP2 and EP3, both getting the same issue
• Frameworks: None, only Azure Functions SDK

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[github-actions]

@ibrahimrabab @ibrandes @seanmcc-msft

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[SoulKa]

I just ran the function locally on my mac. I could not reproduce the error there. It has to do with the Windows Function environment in combination with the new library...

[SoulKa]

Update: I noticed that the HTTP download of the blob simply does not progress. So the blob storage closes the TCP/HTTP connection after ~45s thus the exception.
It seems that this is the origin of the issue. I do not know the internals of the Azure netty or blob storage download implementation, but some change in 12.25.4 results in the freezing HTTP call.

Please tell me if I can provide any useful information or tests.

[SoulKa]

Is someone working on this? Azure Functions seem like a common use case for blob downloads. It would be bad if this issue persists in the newer versions.

[ibrahimrabab]

Hi @SoulKa
We are investigating this issue on our end and will get back to you on this thread as soon as we can with our findings. Thank you for your patience!