Messaging pacts - WithContentAsync - factory is invoked before provider state has been set

[DavidJFowler]

When verifying a messaging pact scenario using WithContentAsync, the factory Func is invoked before a POST to /provider-states has completed.

To reproduce, create a dotnet 7 console app. Add a FrameworkReference to Microsoft.AspNetCore.App in the .csproj file:

<ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>

add the following package refs:

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.11.0" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.4.2" />
    <PackageReference Include="PactNet" Version="4.5.0" />
  </ItemGroup>

Edit program.cs:

using Newtonsoft.Json.Serialization;
using Newtonsoft.Json;
using NUnit.Framework;
using PactNet;
using PactNet.Matchers;
using PactNet.Verifier;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using System.Net;
using Microsoft.AspNetCore.Http;

// create pact
var pactDir = Path.Join("..", "..", "..", "pacts");
var v3 = Pact.V3("Message Consumer", "Message Producer", new PactConfig
{
    PactDir = pactDir,
    DefaultJsonSettings = new JsonSerializerSettings
    {
        ContractResolver = new CamelCasePropertyNamesContractResolver()
    }
});

var messagePact = v3.WithMessageInteractions();

messagePact.ExpectsToReceive("some events")
    .Given("events exist")
    .WithJsonContent(Match.MinType(new { MessageText = "Hello World"}, 1))
    .Verify<ICollection<MyMessage>>(events => Assert.That(events, Is.Not.Empty));

// verify pact

// configure provider states handler
var isProviderStatesCalled = false;
const string pactProviderServiceUri = "http://127.0.0.1:9001";
var builder = WebApplication.CreateBuilder(new WebApplicationOptions());
builder.WebHost.UseUrls(pactProviderServiceUri);
await using var app = builder.Build();
app.MapPost("/provider-states", async context =>
{
    isProviderStatesCalled = true;
    context.Response.StatusCode = (int) HttpStatusCode.OK;
    await context.Response.WriteAsync(string.Empty);
});

await app.StartAsync();

var verifier = new PactVerifier(new PactVerifierConfig
{
    LogLevel = PactLogLevel.Debug
});

var defaultSettings = new JsonSerializerSettings
{
    ContractResolver = new CamelCasePropertyNamesContractResolver(),
    DefaultValueHandling = DefaultValueHandling.Ignore,
    NullValueHandling = NullValueHandling.Ignore,
    Formatting = Formatting.Indented
};

verifier.MessagingProvider("Message Producer", defaultSettings)
    .WithProviderMessages(scenarios =>
    {
        scenarios.Add("some events", scenarioBuilder =>
        {
            return scenarioBuilder.WithContentAsync(async () =>
            {
                var events = new List<MyMessage>
                {
                    new MyMessage
                    {
                        MessageText = "Hello World"
                    }
                };

                Assert.That(isProviderStatesCalled, Is.True);

                await Task.CompletedTask;

                return events;
            });
        });
    }).WithFileSource(new FileInfo(Path.Join(pactDir, "Message Consumer-Message Producer.json")))
    .WithProviderStateUrl(new Uri(new Uri(pactProviderServiceUri), "/provider-states"))
    .Verify();

await app.StopAsync();

Console.WriteLine($"isProviderStatesCalled {isProviderStatesCalled}");

public class MyMessage
{
    public string MessageText { get; set; }
}

On running the app, the assertion

Assert.That(isProviderStatesCalled, Is.True);

fails because the POST to provider-states has not been made.

Changing to WithContent works:

/* ... */

verifier.MessagingProvider("Message Producer", defaultSettings)
    .WithProviderMessages(scenarios =>
    {
        scenarios.Add("some events", scenarioBuilder =>
        {
            scenarioBuilder.WithContent(() =>
            {
                var events = new List<MyMessage>
                {
                    new MyMessage
                    {
                        MessageText = "Hello World"
                    }
                };

                Assert.That(isProviderStatesCalled, Is.True);

                return events;
            });
        });
    }).WithFileSource(new FileInfo(Path.Join(pactDir, "Message Consumer-Message Producer.json")))
    .WithProviderStateUrl(new Uri(new Uri(pactProviderServiceUri), "/provider-states"))
    .Verify();

[adamrodger]

You can see here that the call is awaited immediately upon being invoked:

https://github.com/pact-foundation/pact-net/blob/master/src/PactNet/Verifier/Messaging/MessageScenarioBuilder.cs#L63

And then it sets the resultant value as a sync factory internally. We know that's definitely worked because otherwise it throws an exception because the internal factory value is null.

I suspect this is more a problem with the example than the code itself, because firstly it's capturing variables inside lambdas, which is always fraught with danger, and secondly it's awaiting Task.CompletedTask which you wouldn't really do in real code.

The only other thing I can think of is because the variable is a dynamic so perhaps it's not awaiting properly, but I doubt that given it still sets the internal state properly.

[DavidJFowler]

Thanks @adamrodger, that makes some sense. I wasn't expecting the factory to be awaited immediately.

The way WithContentAsync is currently implemented, it cannot work with provider state. It would need to look like this:

        public async Task WithContentAsync(Func<Task<dynamic>> factory)
        {
            this.factory = () => factory().GetAwaiter().GetResult;
        }

or alternatively, declare MessageScenarioBuilder.factory as Func<Task<dynamic>> and await it at the verification stage

[adamrodger]

I'm not sure that's the case. Given the method is called and awaited immediately then the value must be available at that point:

public async Task WithContentAsync(Func<Task<dynamic>> factory)
{
    dynamic value = await factory();

    // the value is now known because the task must have resolved already

    this.factory = () => value; // this just re-wraps the value that already exists so it can be retrieved sync later on
}

So there's no way for the supplied factory method to somehow not have completed yet, because execution isn't passed back to the user code until the factory method completes due to the await point.

The problem is in your example - you aren't awaiting the setup part yourself, and thus the code proceeds to the verify part before the setup part completes. That's a fault with the example code, not with PactNet.

verifier.MessagingProvider("Message Producer", defaultSettings)
    .WithProviderMessages(scenarios =>
    {
        scenarios.Add("some events", scenarioBuilder =>
        {
            ////////// THIS IS NOT AWAITED ////////////
            return scenarioBuilder.WithContentAsync(async () =>
            {
                var events = new List<MyMessage>
                {
                    new MyMessage
                    {
                        MessageText = "Hello World"
                    }
                };

                Assert.That(isProviderStatesCalled, Is.True);

                await Task.CompletedTask;

                return events;
            });
        });
    }).WithFileSource(new FileInfo(Path.Join(pactDir, "Message Consumer-Message Producer.json")))
    .WithProviderStateUrl(new Uri(new Uri(pactProviderServiceUri), "/provider-states"))
    .Verify();

[DavidJFowler]

I'm not sure that's the case. Given the method is called and awaited immediately then the value must be available at that point:

@adamrodger that's the point. The method is called and awaited too soon - before the call has been made to the provider-states endpoint.

[adamrodger]

Ah ok I see the confusion - the API isn't intended to be a callback that's executed for each interaction during the verification. It's a one-time setup to create some canned responses which are analogous to what the service would publish at runtime.

So that means those factory methods are invoked once during the verifier setup and the responses kept effectively in a dictionary of magic string to canned response.

A callback feature is potentially possible, but it's not a feature that exists in PactNet currently. So yeah, this isn't a defect in the existing functionality, it's more a misunderstanding of how it's intended to work.

[DavidJFowler]

This still doesn't explain why the POST to the provider state url is made before the factory runs in a synchronous scenario, but runs after in an async scenario. Perhaps the provider state configuration should be removed entirely from messaging pacts, as otherwise it is very confusing and of no use when WithContentAsync is used.

Thanks very much for your help in explaining how this all works though.

[mefellows]

What's required to move this forward - is there a proposed change, documentation update etc.?

[DavidJFowler]

Hi @mefellows

Is it possible to update the documentation to show a working example of a messaging pact provider test using a state provider and WithContentAsync()?

This is my latest attempt. It is still failing because the POST to the provider state endpoint is not being executed until after the verification attempt. As previously, altering the code to use WithContent() fixes the problem.

App.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.1" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
    <PackageReference Include="PactNet" Version="4.5.0" />
  </ItemGroup>

</Project>

program.cs

using Newtonsoft.Json.Serialization;
using Newtonsoft.Json;
using NUnit.Framework;
using PactNet;
using PactNet.Matchers;
using PactNet.Verifier;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using System.Net;
using Microsoft.AspNetCore.Http;

// create pact
var pactDir = Path.Join("..", "..", "..", "pacts");
var v3 = Pact.V3("Message Consumer", "Message Producer", new PactConfig
{
    PactDir = pactDir,
    DefaultJsonSettings = new JsonSerializerSettings
    {
        ContractResolver = new CamelCasePropertyNamesContractResolver()
    }
});

var messagePact = v3.WithMessageInteractions();

messagePact.ExpectsToReceive("some events")
    .Given("events exist")
    .WithJsonContent(Match.MinType(new { MessageText = "Hello World" }, 1))
    .Verify<ICollection<MyMessage>>(events => Assert.That(events, Is.Not.Empty));

// verify pact

var messageSender = new MyMessageSender();

// configure provider states handler
const string pactProviderServiceUri = "http://127.0.0.1:9001";
var builder = WebApplication.CreateBuilder(new WebApplicationOptions());
builder.WebHost.UseUrls(pactProviderServiceUri);
await using var app = builder.Build();
app.MapPost("/provider-states", async context =>
{
    Console.WriteLine("Adding event");
    messageSender.Messages.Add(new MyMessage
    {
        MessageText = "Hello World"
    });
    context.Response.StatusCode = (int)HttpStatusCode.OK;
    await context.Response.WriteAsync(string.Empty);
});

await app.StartAsync();

using (var verifier = new PactVerifier(new PactVerifierConfig
       {
           LogLevel = PactLogLevel.Debug
       }))
{

    var defaultSettings = new JsonSerializerSettings
    {
        ContractResolver = new CamelCasePropertyNamesContractResolver(),
        DefaultValueHandling = DefaultValueHandling.Ignore,
        NullValueHandling = NullValueHandling.Ignore,
        Formatting = Formatting.Indented
    };

    try
    {
        verifier.MessagingProvider("Message Producer", defaultSettings)
        .WithProviderMessages(scenarios =>
        {
            scenarios.Add("some events", async scenarioBuilder =>
            {
                await scenarioBuilder.WithContentAsync(async () =>
                {
                    var events = await messageSender.GetMessagesAsync();

                    Assert.That(events.Any(), Is.True);

                    return events;
                });
            });
        }).WithFileSource(new FileInfo(Path.Join(pactDir, "Message Consumer-Message Producer.json")))
        .WithProviderStateUrl(new Uri(new Uri(pactProviderServiceUri), "/provider-states"))
        .Verify();
    }
    catch (Exception ex)
    {
        
        Console.WriteLine($"Verification failed {ex}");
    }
}


await app.StopAsync();

public class MyMessageSender
{
    public List<MyMessage> Messages { get; } = new();

    public async Task<IList<MyMessage>> GetMessagesAsync()
    {
        await Task.Delay(50);
        return Messages.ToList();
    }
}

public class MyMessage
{
    public string MessageText { get; init; }
}