The function invocation context is missing the forwarding task property #2481

dzejsien · 2024-05-14T11:18:42Z

"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",

<TargetFramework>net8.0</TargetFramework>
<AzureFunctionsVersion>v4</AzureFunctionsVersion>

Packages:

    <PackageReference Include="Azure.Identity" Version="1.11.3" />
    <PackageReference Include="Azure.Messaging.EventHubs" Version="5.11.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventHubs" Version="6.3.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />

ISSUE:

I have 2 functions:

HCheck httpTrigger, which checks availability of integrated event hubs
EventHubTrigger

Startup:

var host = new HostBuilder()
    .ConfigureAppConfiguration((_, config) =>
        config
            .AddJsonFile("appsettings.json", true, true)
            .AddDefaultAzureAppConfiguration(...)
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true))
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
        services.Configure<LoggerFilterOptions>(options =>
        {
            ...
        });
        
        //Add healthchecks
        services
        .AddKeyedSingleton(serviceKey: EventHubServiceNames.DeadletterEventHub, (serviceProvider, serviceKey) =>
        {
            var configuration = serviceProvider.GetRequiredService<IConfiguration>();
            return new EventHubProducerClient(configuration["..."]);
        })
        .AddKeyedSingleton(serviceKey: EventHubServiceNames.OutputEventHub, (serviceProvider, serviceKey) =>
        {
            var configuration = serviceProvider.GetRequiredService<IConfiguration>();
            return new EventHubProducerClient(configuration["..."], configuration["..."]);
        })
        .AddKeyedSingleton(serviceKey: EventHubServiceNames.InputEventHub, (serviceProvider, serviceKey) =>
        {
            var configuration = serviceProvider.GetRequiredService<IConfiguration>();
            return new EventHubProducerClient(configuration["..."], configuration["..."]);
        })
        .AddHealthChecks()
        .AddAzureEventHub(clientFactory: serviceProvider => serviceProvider.GetRequiredKeyedService<EventHubProducerClient>(EventHubServiceNames.InputEventHub), name: "inEventHub")
        .AddAzureEventHub(clientFactory: serviceProvider => serviceProvider.GetRequiredKeyedService<EventHubProducerClient>(EventHubServiceNames.OutputEventHub), name: "outEventHub")
        .AddAzureEventHub(clientFactory: serviceProvider => serviceProvider.GetRequiredKeyedService<EventHubProducerClient>(EventHubServiceNames.DeadletterEventHub), name: "dlqEventHub");

       
    })
    .ConfigureLogging((hostingContext, logging) =>
    {
        logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
    })
    .Build();

var lifetime = host.Services.GetRequiredService<IHostApplicationLifetime>();
var loggerFactory = host.Services.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger("...");
lifetime.ApplicationStarted.Register(()
    =>
{
    logger.LogWarning("...");
});

await host.RunAsync();

When HC Function is executed, the ex is thrown, nevertheless HC returns Healthy.

Message: Exception while executing function: Functions.HealthCheck The function invocation context is missing the forwarding task property.

StackTrace:
Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception System.InvalidOperationException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Azure.WebJobs.Script.Grpc.DefaultHttpProxyService+<EnsureSuccessfulForwardingAsync>d__6.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel+<InvokeResponse>d__103.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:1099)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:101)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:82)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__3`1.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:225)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:53)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)

Please support.

The text was updated successfully, but these errors were encountered:

nnicksize · 2024-05-15T06:01:45Z

This happens for us too, but only sporadically. According to app insights around 1/10 of the time. It started yesterday at around 14:00 UTC (16:00+0200).
We also run on dotnet 8, isolated worker, and on an app service plan with no other services on it.

nnicksize · 2024-05-15T06:11:23Z

@fabiocav No direspect, however your change in patch 4.34.2, commit c7323b4, has changes in the forwarding service and the timeline of the error showing up kinda lines up. :) Thanks for looking at it!

Edit: Nevermind, we're running function host 4.34.1.1 which exactly doesn't yet include your fix I mentioned above.

cplankl · 2024-05-17T08:00:43Z

This happens in our Azure function too.

Running net8.0 in isolated worker. Function host is also 4.34.1.1. StackTrace is exactly the same as in the initial post.

Our health function looks like this. Nothing special in it.

    [Function("Health")]
    public static IActionResult Health(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)]
        HttpRequest req)
    {
        return new StatusCodeResult(StatusCodes.Status200OK);
    }

peterasamoah7 · 2024-05-21T08:34:59Z

We recently started seeing this in our function app as well. Any solution or reason why this is happening ?

smockr · 2024-05-22T05:29:25Z

We have also been seeing this for a just under a week now.
I have a vague memory of looking at this previously and seeing that 4.34.2 might fix it, but it seems like maybe that was only a change and has not been confirmed as a fix?

If there is a fix, is it possible to get an indication of a rollout timeline? This is a bit of an annoyance for us as it is failing http triggered requests that have otherwise succeeded, causing retries from third parties to those endpoints.

chrisrlewis · 2024-05-22T09:03:47Z

We are seeing this too, with the most basic 'Ping' function. Sporadically, not for every request.

Runtime version 4.34.1.1 - also would like an indication of when the fix will be rolled out.

patrick-elmore · 2024-05-22T13:33:20Z

We are seeing the same thing as well. The endpoint we are getting these errors on is a vanilla health check endpoint that just returns 200 response, almost identical to the code @cplankl posted.

Runtime version - 4.34.1.22669
dotnet-isolated
.net6.0

perum · 2024-05-22T20:30:29Z

Also seeing this and it's causing issues for alerting. Roughly 1 failure per hour, but still intermittent. 4.34 runtime, dotnet-isolated, net8.0. Please advise or get in a fix for this...

mtycholaz · 2024-05-24T04:39:47Z

I'm seeing this problem happen if I add a custom middleware to a basic function.

Code to reproduce

https://github.com/mtycholaz/azure-function-middleware

gsayem · 2024-05-25T19:48:14Z

Middleware moved to ConfigureFunctionsWebApplication.

var host = new HostBuilder()
    .ConfigureAppConfiguration((_, config) =>
        config
            .AddJsonFile("appsettings.json", true, true)
            .AddDefaultAzureAppConfiguration(...)
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true))
           .ConfigureFunctionsWebApplication(configureWorker =>
            {
               // The Middlewares
                configureWorker.UseMiddleware<CustomMiddleware>();
            })

ChoOo7 · 2024-05-27T10:28:25Z

@gsayem
Does moving the middleware helps to prevent errors ?
Thanks

cplankl · 2024-05-27T11:32:28Z

@gsayem
In my code I do not add custom middlewares:

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var configuration = new ConfigurationBuilder()
    .SetBasePath(Environment.CurrentDirectory)
    .AddEnvironmentVariables()
    .Build();

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

However, ConfigureFunctionsWebApplication() will probably register middlewares but it is also a recommendation from Microsoft to do so:

This example includes ASP.NET Core integration to improve performance and provide a familiar programming model when your app uses HTTP triggers. If you do not intend to use HTTP triggers, you can replace the call to ConfigureFunctionsWebApplication with a call to ConfigureFunctionsWorkerDefaults. If you do so, you can remove the reference to Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore from your project file. However, for the best performance, even for functions with other trigger types, you should keep the FrameworkReference to ASP.NET Core.

gsayem · 2024-05-27T15:47:27Z

@gsayem Does moving the middleware helps to prevent errors ? Thanks

I think yes, also my errors are gone when I moved my middleware.

gsayem · 2024-05-27T15:51:29Z

@gsayem In my code I do not add custom middlewares:
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var configuration = new ConfigurationBuilder()
    .SetBasePath(Environment.CurrentDirectory)
    .AddEnvironmentVariables()
    .Build();

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();
However, ConfigureFunctionsWebApplication() will probably register middlewares but it is also a recommendation from Microsoft to do so:

This example includes ASP.NET Core integration to improve performance and provide a familiar programming model when your app uses HTTP triggers. If you do not intend to use HTTP triggers, you can replace the call to ConfigureFunctionsWebApplication with a call to ConfigureFunctionsWorkerDefaults. If you do so, you can remove the reference to Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore from your project file. However, for the best performance, even for functions with other trigger types, you should keep the FrameworkReference to ASP.NET Core.

This should work, I tried your example and it's working as expected.

mtycholaz · 2024-05-30T00:24:33Z

@gsayem I tried out your recommendation. The error has gone away for me now. I don't fully understand the difference between ConfigureFunctionsWorkerDefaults and ConfigureFunctionsWebApplication but that seemed to have fixed it for me.

dzejsien added the bug label May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The function invocation context is missing the forwarding task property #2481

The function invocation context is missing the forwarding task property #2481

dzejsien commented May 14, 2024

nnicksize commented May 15, 2024 •

edited

nnicksize commented May 15, 2024 •

edited

cplankl commented May 17, 2024

peterasamoah7 commented May 21, 2024

smockr commented May 22, 2024

chrisrlewis commented May 22, 2024

patrick-elmore commented May 22, 2024

perum commented May 22, 2024

mtycholaz commented May 24, 2024

gsayem commented May 25, 2024

ChoOo7 commented May 27, 2024

cplankl commented May 27, 2024

gsayem commented May 27, 2024

gsayem commented May 27, 2024

mtycholaz commented May 30, 2024

The function invocation context is missing the forwarding task property #2481

The function invocation context is missing the forwarding task property #2481

Comments

dzejsien commented May 14, 2024

nnicksize commented May 15, 2024 • edited

nnicksize commented May 15, 2024 • edited

cplankl commented May 17, 2024

peterasamoah7 commented May 21, 2024

smockr commented May 22, 2024

chrisrlewis commented May 22, 2024

patrick-elmore commented May 22, 2024

perum commented May 22, 2024

mtycholaz commented May 24, 2024

gsayem commented May 25, 2024

ChoOo7 commented May 27, 2024

cplankl commented May 27, 2024

gsayem commented May 27, 2024

gsayem commented May 27, 2024

mtycholaz commented May 30, 2024

nnicksize commented May 15, 2024 •

edited

nnicksize commented May 15, 2024 •

edited