StackOverflowException after upgrading from .Net 6 to .Net 8

[Abhishek418b]

Hello,

I have upgraded 5 projects in my solution from .Net 6 to .Net 8 using upgrade assistant in Visual Studio.

When I run the solution, StackOverflowException is being thrown and Swagger is not loading. Looks like onchangetokenfired() method (This method is from dll code) is being called many times resulting in infinite loop.

However, we deployed same code to Azure App Service and it is working fine. So, looks like it is working in publish mode and not in debug mode in Visual Studio in local.

While I run the solution in VS Code via command "dotnet run" it is running successfully but to debug the code I've installed C# toolkit in VS Code same exception is being thrown.

Please help in fixing this issue in Visual Studio.

[KalleOlaviNiemitalo]

dotnet/runtime#79966 added the token.HasChanged check in the Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration<TState>.RegisterChangeTokenCallback(IChangeToken? token) method. That call shows up in the stack trace.

The following buggy program causes a stack overflow already in .NET 6.0. However, the resulting stack trace does not match the screen shot; it does not include the get_HasChanged accessor.

using System.Threading;
using Microsoft.Extensions.Primitives;

class Program
{
    static void Main(string[] args)
    {
        // BUG: This changeTokenProducer always returns a change token that
        // already indicates a change, so ChangeToken.OnChange calls it again
        // to get a new change token, and these calls overflow the stack.
        ChangeToken.OnChange(
            changeTokenProducer: CreateChangedChangeToken,
            changeTokenConsumer: () => {});
    }

    static IChangeToken CreateChangedChangeToken()
    {
        return new CancellationChangeToken(new CancellationToken(canceled: true));
    }
} 

The following buggy program causes a stack overflow in .NET 8.0 but not in .NET 6.0. The stack trace mostly matches the screen shot.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using Microsoft.Extensions.Primitives;

static class Program
{
    static CancellationTokenSource source = new CancellationTokenSource();

    static void Main(string[] args)
    {
        // BUG: This changeTokenProducer always returns a change token that
        // already indicates a change, so ChangeToken.OnChange calls it again
        // to get a new change token, and these calls overflow the stack.
        ChangeToken.OnChange(
            changeTokenProducer: ProduceChangeToken,
            changeTokenConsumer: ConsumeChangeToken);
    }

    static IChangeToken ProduceChangeToken()
    {
        ManualChangeToken manual = new ManualChangeToken();
        IChangeToken[] changeTokens =
        {
            manual,
        };
        IChangeToken result = new CompositeChangeToken(changeTokens);
        manual.SetChanged();
        return result;
    }

    static void ConsumeChangeToken()
    {
    }
} 

// A change token that can be manually set as changed,
// but does not have active callbacks; it calls the callbacks only when
// the HasChanged property is read or a new callback is registered.
internal sealed class ManualChangeToken : IChangeToken
{
    private List<Entry> callbacks = new List<Entry>();

    private bool hasChanged;

    public bool ActiveChangeCallbacks => false;

    public bool HasChanged
    {
        get
        {
            if (!Volatile.Read(ref this.hasChanged))
            {
                return false;
            }

            this.RunCallbacks();
            return true;
        }
    }

    public IDisposable RegisterChangeCallback(Action<object?> callback, object? state)
    {
        Entry entry = new Entry(this, callback, state);

        lock (this.callbacks)
        {
            this.callbacks.Add(entry);
        }

        _ = this.HasChanged;
        return entry;
    }

    internal void SetChanged()
    {
        Volatile.Write(ref this.hasChanged, true);
    }

    private void RunCallbacks()
    {
        Debug.Assert(this.hasChanged);

        while (true)
        {
            Entry entry;
            lock (this.callbacks)
            {
                if (this.callbacks.Count == 0)
                {
                    break;
                }
 
                entry = this.callbacks[this.callbacks.Count - 1];
                this.callbacks.RemoveAt(this.callbacks.Count - 1);
            }

            entry.Invoke();
        }
    }

    private sealed class Entry : IDisposable
    {
        private readonly ManualChangeToken changeToken;
        private readonly Action<object?> callback;
        private readonly object? state;

        internal Entry(ManualChangeToken changeToken, Action<object?> callback, object? state)
        {
            this.changeToken = changeToken;
            this.callback = callback;
            this.state = state;
        }

        public void Dispose()
        {
            lock (this.changeToken.callbacks)
            {
                this.changeToken.callbacks.Remove(this);
            }
        }

        internal void Invoke()
        {
            this.callback.Invoke(this.state);
        }
    }
}

So, it seems your program has a change token producer that returns change tokens that have already expired.
I suggest you set a breakpoint at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired()`, let it trigger until the stack trace starts looking too deep, and then check what the CompositeChangeToken is composed of.

Does your program include any custom IFileProvider or IChangeToken implementation?

[Abhishek418b]

Below is my Program.cs:

/// <summary>
/// Program class
/// </summary>
public class Program
{
    /// <summary>
    /// Main method
    /// </summary>
    /// <param name="args"></param>
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    /// <summary>
    /// Host builder
    /// </summary>
    /// <param name="args"></param>
    /// <returns></returns>
    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

As you can see, we don't have any custom code related to change tokens and no custom IFileProvider or IChangeToken implementation.

The OnChangeTokenFired() method is from dll decompiled code in which breakpoint can't be put directly.

[KalleOlaviNiemitalo]

Does that get a stack overflow on .NET 8.0 in Visual Studio, even with a no-op Startup class?

internal class Startup
{
    public void Configure() {}
}

[Abhishek418b]

Below is my Startup.cs:

public class Startup
{
    readonly string apiSpecificOrigins = "apiSpecificOrigins";
    readonly string[] headers = new string[] { "Content-Type", "Accept", "Authorization", "myInsightAuthorization" };
    readonly string[] methods = new string[] { "GET", "POST", "PUT", "DELETE", "OPTIONS" };
    readonly bool isDevelopment;
    readonly string serverRegion;
    /// <summary>
    /// Startup constructor
    /// </summary>
    /// <param name="configuration"></param>
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
        isDevelopment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") == "Development";
        serverRegion = configuration["ServerRegion"];
    }

    /// <summary>
    /// Configuration variable
    /// </summary>
    public static IConfiguration Configuration { get; private set; }

    /// <summary>
    /// Add methods to the container
    /// </summary>
    /// <param name="services"></param>
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();

        services.AddApiVersioning(options =>
        {
            options.ReportApiVersions = true;
        });
        
        services.AddOData().EnableApiVersioning();
        
        services.AddODataApiExplorer(options =>
        {
            options.GroupNameFormat = "'v'VVV";
            options.SubstituteApiVersionInUrl = true;
        });
        services.AddMvc(options =>
        {
            options.OutputFormatters.Insert(0, new CustomODataOutputFormatter(isDevelopment));
        });

        services.AddTransient<IConfigureOptions<SwaggerGenOptions>, ConfigureSwaggerOptions>();

        services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();

        services.AddSwaggerGen(
            options =>
            {
                options.DocumentFilter<SwaggerDocumentFilter>();

#if !DEBUG
options.DocumentFilter();
#endif
options.OperationFilter();
options.IncludeXmlComments(XmlCommentsFilePath);
});

        services.AddCors(options =>
        {
            options.AddPolicy(name: apiSpecificOrigins,
                              builder =>
                              {
                                  builder.WithOrigins(Configuration["AccessControlAllowOrigins"].Split(","))
                                         .WithHeaders(headers)
                                         .WithMethods(methods)
                                         .AllowCredentials();
                              });
        });



        AddAppInsights(services);

        services.AddDbContext<ReadOnlyContext>();
        services.AddDbContext<ReadWriteConnection>();

        services.AddScoped<IAdminUsersService, AdminUsersService>();
        services.AddScoped<IAdminRolesService, AdminRolesService>();
       
    }

    /// <summary>
    /// Http request pipeline
    /// </summary>
    /// <param name="app"></param>
    /// <param name="env"></param>
    /// <param name="modelBuilder"></param>
    /// <param name="provider"></param>
    public void Configure(IApplicationBuilder app,
                          IWebHostEnvironment env,
                          VersionedODataModelBuilder modelBuilder,
                          IApiVersionDescriptionProvider provider)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }
        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseCors(apiSpecificOrigins);

        app.Use(async (context, next) =>
        {
            if (context.Request.Method == "OPTIONS")
            {
                context.Response.StatusCode = 204;
                context.Response.Headers.Add("Access-Control-Allow-Origin", context.Request.Headers["Origin"]);
                context.Response.Headers.Add("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
                context.Response.Headers.Add("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS");
                context.Response.Headers.Add("Access-Control-Allow-Credentials", "true");
                return;
            }
            context.Response.Headers.Add("ServerRegion", serverRegion);
            await next();
        });

        app.UseAuthorization();

        app.UseEndpoints(routeBuilder =>
        {
            routeBuilder.EnableDependencyInjection();
            routeBuilder.Select().Filter().OrderBy().Expand().Count().MaxTop(100);
            routeBuilder.MapVersionedODataRoute("odata", "{version:apiVersion}", modelBuilder);
        });

        app.UseSwagger();

        app.UseSwaggerUI(options =>
        {

#if DEBUG
options.RoutePrefix = "swagger";
#else
options.RoutePrefix = String.Empty;
#endif

            foreach (var description in provider.ApiVersionDescriptions)
            {
                options.SwaggerEndpoint($"/swagger/{description.Name}/swagger.json", description.GroupName.ToUpperInvariant());
            }
        });
    }

    static string XmlCommentsFilePath
    {
        get
        {
            var basePath = PlatformServices.Default.Application.ApplicationBasePath;
            var fileName = typeof(Startup).GetTypeInfo().Assembly.GetName().Name + ".xml";
            return Path.Combine(basePath, fileName);
        }
    }
    private void AddAppInsights(IServiceCollection services)
    {
        string appInsightsKey = Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"];
        if (!string.IsNullOrEmpty(appInsightsKey))
        {
            // The following line enables Application Insights telemetry collection.
            services.AddApplicationInsightsTelemetry();
        }
    }
}

Which method code do u suggest to comment out and check if stack overflow is occurring?

[KalleOlaviNiemitalo]

I'd try removing App Insights first, but this is only a guess.

I wonder if Visual Studio is injecting something buggy via HostFactoryResolver.

[Abhishek418b]

I have tried removing App Insights but still issue persists. Please suggest if u have any other solution.

[KalleOlaviNiemitalo]

The OnChangeTokenFired() method is from dll decompiled code in which breakpoint can't be put directly.

AFAIK it's possible to set a breakpoint by method name even if you don't have the source code.

[Abhishek418b]

When I run the solution in release mode, the exception is occurring in AssemblyNameParser.cs.

[KalleOlaviNiemitalo]

I guess AssemblyNameParser does not cause the stack overflow, but recursion in other classes first consumes almost all the stack, and then something calls AssemblyNameParser which cannot work with so little stack remaining.